Pith. sign in

REVIEW 3 major objections 5 minor 36 references

Instruction-tuned language models do not sample from the distributions they can describe: per-call outputs collapse to a single favorite answer, while a single describe-call recovers the target shape.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 02:50 UTC pith:4BNWHFIM

load-bearing objection A serious empirical paper that establishes a real describe-vs-sample asymmetry but overreaches on the alignment-causality claim, which rests on a confounded base-vs-instruct comparison. the 3 major comments →

arxiv 2607.25292 v1 pith:4BNWHFIM submitted 2026-07-28 cs.AI

Instruction-Tuned Language Models Cannot Sample from Distributions They Can Describe

Chaemin Jang , Dongman Lee , Jihee Kim This is my paper
classification cs.AI
keywords silicon samplingmode collapseinstruction tuningKNOWS/DOES splitpersona aggregationsampling primitivetotal-variation distanceprompt perturbation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that the per-call output of an instruction-tuned language model is not a draw from the distribution the prompt specifies. Asked to produce one answer per call on five non-trivial categorical targets, each model returns a single favored answer in over 94% of calls, and repeated persona–question pairs on a public-opinion benchmark return the identical answer on more than half of cells. Yet the same model, when asked to describe the distribution directly in one call, matches the target to within 0.05 total-variation distance. The authors call this gap the KNOWS/DOES split, trace it to a degenerate randomness primitive visible at the logit level, and argue that alignment training induces it, since base models fail far less often across three model families. They then show that asking for the distribution directly more than halves the error against human survey data, and that a same-cost prompt-perturbation method—PPA—cuts the same error by 21%.

Core claim

The paper's central claim is that instruction-tuned language models do not sample from distributions: whenever the task is to produce one output per call from a stated categorical target, the model concentrates on a single favorite answer, and this holds across temperature, top-p, and six prompt-level corrections. The collapse is visible at the logit level—top-two logit gaps exceed 14 nats on some targets, beyond any recoverable decoding temperature—and it originates in a degenerate randomness primitive: asked for a uniform random integer, the model returns 42 in 78% of calls, and the random number it writes inside chain-of-thought scaffolds is itself non-uniform. The same model can, in one

What carries the argument

The load-bearing object is the per-call sampling primitive: the next-token distribution the model actually uses when asked to emit one answer. The paper shows this primitive is degenerate—logits concentrate on a single option with top-two gaps of more than 14 nats at temperature zero, and the model's internal random-number generator is non-uniform (an integer 1–100 prompt returns 42 in 78% of calls). Chains of thought inherit the same defect: the random value u written inside an inverse-CDF scaffold is itself non-uniform. The KNOWS/DOES split is the counterpoint: switching the interface from 'sample one output per call' to 'describe the distribution in one call' (a list of N exemplars or JSO

Load-bearing premise

The causal claim that alignment training induces the collapse rests on comparing base models prompted with 3-shot in-context examples against instruction-tuned models prompted zero-shot with a chat template; if the base models' greater diversity comes from the different prompt format or from weaker instruction-following rather than from the absence of alignment training, the alignment-induced conclusion collapses.

What would settle it

Take one base model and its instruction-tuned counterpart, and run both with the exact same zero-shot chat template and the same five synthetic categorical targets; if the base model, now following instructions, still produces a spread of answers while the tuned model collapses, the alignment claim is supported. If the base model collapses under the identical template, the reported base-versus-instruct difference is a prompt-format artifact rather than evidence for alignment-induced collapse.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Any silicon-sampling pipeline that aggregates per-call persona responses inherits a degenerate sampling primitive; the paper shows that more than half of (persona, question) cells return the same answer across repeated calls on a 100-item public-opinion benchmark.
  • When only a population estimate is needed, a single call that asks the model to describe the response distribution directly more than halves total-variation error against human survey data (0.22 vs 0.46).
  • For applications that need one output per simulated respondent, PPA—randomizing option order, question phrasing, and persona position across calls—reduces the same error by 21% at no added API cost, and the improvement tracks the per-call entropy increase it induces.
  • Decoding controls cannot fix the failure: temperature-zero logit gaps exceed the range any API-available temperature can recover, and temperature and top-p sweeps capture under 20% of PPA's improvement.
  • The KNOWS/DOES split replicates across five additional instruction-tuned model families, and the paper's analysis shows that the describe pathway succeeds by counting rather than by drawing independent samples.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • I infer that the degenerate primitive likely affects any aggregate built from repeated per-call outputs of an instruction-tuned model—synthetic training data, judge ratings, or agent trajectories—since the mechanism (alignment-induced peaked logits) is not specific to survey questions.
  • The causal direction could be tested cleanly by fine-tuning a single base model on instruction-following data and comparing pre/post per-call diversity under an identical prompt template; the paper's base-vs-instruct comparison conflates training with prompt format, so this controlled test would settle whether alignment induces the collapse.
  • The describe-pathway success suggests a general design rule for LLM applications: whenever the goal is a distribution, ask for a count or probability specification and sample externally, rather than relying on repeated per-call draws.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reports that instruction-tuned language models, when repeatedly asked to produce one categorical outcome per call, fail dramatically to generate draws from the distribution named in the prompt. Across synthetic targets (skewed binary, mixture, three-way, bimodal, skewed 5-way), gpt-4o returns a single modal answer in more than 94% of calls, and a direct random-number probe shows a degenerate format-specific favorite (e.g., 42 for integers 1–100). The paper further claims this failure is induced by alignment training, based on a base-vs-instruct comparison across three 8B families; documents a KNOWS/DOES split in which a single-call description of the distribution is accurate where per-call sampling collapses; and applies these findings to OpinionQA, showing that a describe-style pipeline reduces TV-to-Pew by about half relative to persona aggregation, while a proposed Prompt-Perturbed Argyle (PPA) reduces TV by 21% at no increased API cost.

Significance. If the descriptive core is correct, the paper is an important, well-controlled demonstration that standard silicon-sampling pipelines rest on an invalid sampling primitive for at least some instruction-tuned models. The strengths are substantial: pre-registered item-selection rules; 200 calls per cell; arbitrary synthetic targets that cannot be memorized from training statistics; external Pew ground truth; demographically matched Argyle controls; cross-family replication; bootstrap robustness checks; and mechanism controls for PPA. The paper also ships enough detail in the appendices to reproduce the main analyses. However, the headline claims 'cannot sample' and 'induced by alignment training' are broader than the evidence supports, especially when the cross-family data in Appendix Q are taken into account. The central descriptive result for gpt-4o is credible; the global generalization and the causal attribution need revision.

major comments (3)
  1. [§3.4 and Appendix B] The causal claim that alignment training induces the sampling collapse is confounded by prompt format and instruction-following ability. Base models are prompted with a 3-shot in-context format because they 'do not reliably follow zero-shot instructions,' while instruct models receive zero-shot chat prompts. This varies training status together with prompting protocol. The 3-shot demonstrations may themselves induce distribution-matching behavior, or the base models' higher diversity may reflect off-task/ill-formed responses rather than genuine distributional sampling. Appendix B also reports elevated parse-failure rates on two instruct cells (156/200 and 118/200) and analyzes only valid parses, which can bias toward determinism if failures occur when the model hedges or produces diverse but non-canonical outputs. The descriptive collapse in §3.2 stands, but the abstract's and §7's 'indu
  2. [Abstract; §3.2 Table 1; Appendix Q Tables 28–31] The paper repeatedly states that instruction-tuned models 'cannot sample from distributions' and that 'every instruction-tuned model fails on every task we test,' but the paper's own Appendix Q contradicts this in degree. gpt-5.4 on int 1–100 has P(mode)=0.110 and coverage 59/100, which is not categorical collapse; V0 TV is below 0.20 in 4 of 25 cross-family (model, task) cells (e.g., gpt-5.4 fair coin TV=0.195, skewed binary TV=0.130; deepseek bimodal TV=0.220). If the intended claim is that some instruction-tuned models collapse under standard API settings, the title and abstract must be qualified accordingly. As written, the claim is falsified by the paper's own replication data.
  3. [§5.3, Eq. (3); Appendix O] The regression evidence for PPA's mechanism is weaker than the prose suggests. The coefficient on ΔH is robust in sign (positive in 1000/1000 bootstrap resamples), but Appendix O reports hold-out R² from 50/50 splits has mean 0.095, with only 81.4% of splits positive. The 'same slope' argument for temperature (ΔTV/ΔH ≈ 0.25 for both) is suggestive but based on a single temperature condition and may be a statistical artifact of dividing two small differences. This does not invalidate PPA's headline improvement, which is well established with n=100 items, but the mechanism claim should be presented with the acknowledged instability and not over-interpreted.
minor comments (5)
  1. [Table 16 (Appendix I)] The row label 'V8 list-of-N=200' appears to be a typo; V8 uses lists of N=20, not 200.
  2. [Appendix B, Table 9] The bimodal 5-way V8 cell reports 10/10 parse failures, leaving TV undefined. This is reported honestly, but the main text's 'both KNOWS interventions succeed on all five' (§3.5) should note the exception, since the claim relies on gpt-4o only.
  3. [§3.2 and Table 13] The abstract says the logit gap 'exceeds 14 nats on some targets.' The body shows the three-way target has a gap of 3.75 nats, so consider saying 'can exceed 14 nats' and report the distribution of gaps rather than only the maximum.
  4. [§3.1 and Appendix E] The TV>0.20 failure threshold is used throughout but defined only informally. Since it is a free parameter, state the threshold choice in the setup and report sensitivity to it, especially where it drives 'fails' counts in Tables 3 and 31.
  5. [Appendix Q, Table 28] The gpt-5.4 row shows P(mode)=0.110 with coverage 59/100, which the table labels a 'favored value.' Calling this a 'degenerate RNG primitive' is an overreach; a model with 59% coverage on 1–100 is not concentrating on a single answer.

Circularity Check

0 steps flagged

No circular derivation: claims rest on external comparisons and direct empirical tests, not on definitions or self-citations.

full rationale

The paper's central claims are empirical findings rather than derived results. The per-call collapse is measured against arbitrary synthetic target distributions and against external Pew ground truth via OpinionQA, so the evaluation is not defined in terms of the conclusion. The KNOWS/DOES split is supported by a ten-intervention comparison (V0-V9) and by cross-family replication; V8's success is explicitly shown to be a counting/specification operation (cross-call sequence-rank correlation >0.97), not a hidden sampling claim. The only in-sample statistical model is the §5.3 entropy regression, fit to the same 20 items it describes. This is a descriptive mechanism claim, and it is not load-bearing for the paper's headline results: PPA's 21% TV reduction is a direct paired comparison, and the regression is independently corroborated by the temperature analogue and bootstrap resampling. No load-bearing self-citation appears (the reference list contains no work by the present authors), and no uniqueness theorem or ansatz is imported from prior self-authored work. The base-vs-instruct protocol does differ in prompt format (3-shot for base models vs. zero-shot chat for instruct models), which is a potential confound for the alignment-causality claim; however, this is a validity threat, not a circular reduction, because the instruct-model collapse is established independently of that comparison and the causal attribution is an interpretation, not a definitional equivalence.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 1 invented entities

The central empirical claims rest on measurement assumptions (API logits, Pew ground truth, 200-call estimates) and on the comparability of base-vs-instruct prompting. The only fitted numerical parameter used for a mechanism claim is the entropy regression slope. No new physical or generative entity is introduced with independent falsifiable evidence; the 'degenerate sampling primitive' is an explanatory construct.

free parameters (2)
  • TV failure threshold = 0.20
    Hand-chosen threshold for counting a task as 'failed' in Tables 3, 16, 31; changing it changes how many models/tasks are said to fail, though the main collapse is much larger than the threshold.
  • Entropy-regression beta_DeltaH = 0.27 (95% CI [0.16, 0.38])
    OLS coefficient in Eq. 3 fit to 20 OpinionQA items; used to argue PPA's TV improvement is driven by per-call entropy increase. The point estimate is stable in bootstrap but the regression is fit to the same data it explains.
axioms (4)
  • domain assumption Base models prompted with 3-shot examples are the correct counterfactual for 'no alignment training'
    Appendix B uses 3-shot V0 on base models vs zero-shot chat on instruct models; the causal claim that alignment induces collapse depends on this comparability.
  • domain assumption Pew ATP response distributions are valid ground truth for opinion items
    Used as target in OpinionQA TV comparisons (§4); if ground truth is biased or mismatched, error estimates change.
  • domain assumption 200 API calls at temperature 1.0 provide reliable empirical distribution estimates for categorical outputs
    Main behavioral protocol; no CIs reported on many per-cell results, and parse failures are excluded.
  • domain assumption The API top-logprobs at temperature 0 reflect the model's internal next-token distribution
    Logit-level claims in §3.2/Appendix F rely on this being a faithful readout.
invented entities (1)
  • Degenerate sampling primitive no independent evidence
    purpose: Explains per-call collapse: the model's internal random-number generation is non-uniform and logits are sharply peaked, so every decoding temperature still concentrates probability on one option
    Inferred from behavioral and logit measurements; no independent prediction or direct neural measurement is provided, and the paper itself notes it is a trace/interpretation rather than a proven mechanism.

pith-pipeline@v1.3.0-alltime-deepseek · 21486 in / 12528 out tokens · 125555 ms · 2026-08-01T02:50:33.552517+00:00 · methodology

0 comments
read the original abstract

Silicon sampling uses language models as proxies for human survey respondents, treating each model call as an independent draw from the persona's response distribution. We show this draw does not exist: instruction-tuned models do not sample from distributions, they collapse to a single output. The same persona on the same question returns the same answer on more than half of items in a public-opinion benchmark. The collapse is sharp: the model's internal probabilities concentrate on a single option, and the failure is substantially amplified by instruction tuning: across three model families with materially different post-training pipelines, every instruction-tuned model fails on every task we test, while base models fail far less often. Strikingly, the same model that cannot sample from a distribution can describe it accurately in a single call. We call this gap the KNOWS/DOES split, and trace it to a degenerate sampling primitive visible in the logits and induced by alignment training. Exploiting this split, asking the model to describe the response distribution in one call more than halves the error against human survey data compared to persona aggregation. For applications that require per-persona outputs, we propose Prompt-Perturbed Argyle (PPA), which reduces the same error by 21% at no added cost.

Figures

Figures reproduced from arXiv: 2607.25292 by Chaemin Jang, Dongman Lee, Jihee Kim.

Figure 1
Figure 1. Figure 1: The KNOWS/DOES split on the skewed binary target. When asked to sample one outcome per call, the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 7 linked inside Pith

  1. [1]

    and Busby, Ethan C

    Argyle, Lisa P. and Busby, Ethan C. and Fulda, Nancy and Gubler, Joshua R. and Rytting, Christopher and Wingate, David , title =. Political Analysis , volume =

  2. [2]

    , title =

    Park, Joon Sung and O'Brien, Joseph and Cai, Carrie Jun and Morris, Meredith Ringel and Liang, Percy and Bernstein, Michael S. , title =. Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST) , year =

  3. [3]

    and Shaw, Aaron and Hill, Benjamin Mako and Cai, Carrie and Morris, Meredith Ringel and Willer, Robb and Liang, Percy and Bernstein, Michael S

    Park, Joon Sung and Zou, Carolyn Q. and Shaw, Aaron and Hill, Benjamin Mako and Cai, Carrie and Morris, Meredith Ringel and Willer, Robb and Liang, Percy and Bernstein, Michael S. , title =. arXiv preprint arXiv:2411.10109 , year =

    Pith/arXiv arXiv

  4. [4]

    Proceedings of the 40th International Conference on Machine Learning (ICML) , year =

    Santurkar, Shibani and Durmus, Esin and Ladhak, Faisal and Lee, Cinoo and Liang, Percy and Hashimoto, Tatsunori , title =. Proceedings of the 40th International Conference on Machine Learning (ICML) , year =

  5. [5]

    and Arriaga, Rosa I

    Aher, Gati V. and Arriaga, Rosa I. and Kalai, Adam Tauman , title =. Proceedings of the 40th International Conference on Machine Learning (ICML) , year =

  6. [6]

    and Filippas, Apostolos and Manning, Benjamin S

    Horton, John J. and Filippas, Apostolos and Manning, Benjamin S. , title =

  7. [7]

    arXiv preprint arXiv:2501.10970 , year =

    Calderon, Nitay and Reichart, Roi and Dror, Rotem , title =. arXiv preprint arXiv:2501.10970 , year =

    Pith/arXiv arXiv

  8. [8]

    and Dorff, Cassy and Kenkel, Brenton and Larson, Jennifer M

    Bisbee, James and Clinton, Joshua D. and Dorff, Cassy and Kenkel, Brenton and Larson, Jennifer M. , title =. Political Analysis , volume =

  9. [9]

    Machine bias:

    Boelaert, Julien and Coavoux, Samuel and Ollion,. Machine bias:. Sociological Methods and Research , volume =

  10. [10]

    Transactions of the Association for Computational Linguistics , volume =

    Tjuatja, Lindia and Chen, Valerie and Wu, Tongshuang and Talwalkar, Ameet and Neubig, Graham , title =. Transactions of the Association for Computational Linguistics , volume =

  11. [11]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    Hu, Tiancheng and Collier, Nigel , title =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

  12. [12]

    Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL) , year =

    Gligori. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL) , year =

    2024

  13. [13]

    Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL) , pages =

    Meister, Nicole and Guestrin, Carlos and Hashimoto, Tatsunori , title =. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL) , pages =

    2025

  14. [14]

    and Zenn, Johannes and Liu, Zhen and Liu, Weiyang and Bamler, Robert and Sch

    Xiao, Tim Z. and Zenn, Johannes and Liu, Zhen and Liu, Weiyang and Bamler, Robert and Sch. Flipping against all odds:. arXiv preprint arXiv:2506.09998 , year =

    Pith/arXiv arXiv

  15. [15]

    and Manning, Christopher D

    Zhang, Jiayi and Yu, Simon and Chong, Derek and Sicilia, Anthony and Tomz, Michael R. and Manning, Christopher D. and Shi, Weiyan , title =. arXiv preprint arXiv:2510.01171 , year =

    Pith/arXiv arXiv

  16. [16]

    Proceedings of the 12th International Conference on Learning Representations (ICLR) , year =

    Padmakumar, Vishakh and He, He , title =. Proceedings of the 12th International Conference on Learning Representations (ICLR) , year =

  17. [17]

    Ouyang, Long and Wu, Jeff and Jiang, Xu and Almeida, Diogo and Wainwright, Carroll L. and Mishkin, Pamela and Zhang, Chong and Agarwal, Sandhini and Slama, Katarina and Ray, Alex and Schulman, John and Hilton, Jacob and Kelton, Fraser and Miller, Luke and Simens, Maddie and Askell, Amanda and Welinder, Peter and Christiano, Paul and Leike, Jan and Lowe, R...

  18. [18]

    , title =

    Stiennon, Nisan and Ouyang, Long and Wu, Jeffrey and Ziegler, Daniel and Lowe, Ryan and Voss, Chelsea and Radford, Alec and Amodei, Dario and Christiano, Paul F. , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  19. [19]

    and Hatfield-Dodds, Zac and Mann, Ben and Amodei, Dario and Joseph, Nicholas and McCandlish, Sam and Brown, Tom and Kaplan, Jared , title =

    Bai, Yuntao and Kadavath, Saurav and Kundu, Sandipan and Askell, Amanda and Kernion, Jackson and Jones, Andy and Chen, Anna and Goldie, Anna and Mirhoseini, Azalia and McKinnon, Cameron and Chen, Carol and Olsson, Catherine and Olah, Christopher and Hernandez, Danny and Drain, Dawn and Ganguli, Deep and Li, Dustin and Tran-Johnson, Eli and Perez, Ethan an...

    Pith/arXiv arXiv

  20. [20]

    Improving alignment of dialogue agents via targeted human judgements , journal =

    Glaese, Amelia and McAleese, Nat and Tr. Improving alignment of dialogue agents via targeted human judgements , journal =

  21. [21]

    Proceedings of the 8th International Conference on Learning Representations (ICLR) , year =

    Holtzman, Ari and Buys, Jan and Du, Li and Forbes, Maxwell and Choi, Yejin , title =. Proceedings of the 8th International Conference on Learning Representations (ICLR) , year =

  22. [22]

    Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    Fan, Angela and Lewis, Mike and Dauphin, Yann , title =. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

  23. [23]

    and Blank, Idan A

    Mahowald, Kyle and Ivanova, Anna A. and Blank, Idan A. and Kanwisher, Nancy and Tenenbaum, Joshua B. and Fedorenko, Evelina , title =. Trends in Cognitive Sciences , volume =

  24. [24]

    arXiv preprint arXiv:2207.05221 , year =

    Kadavath, Saurav and Conerly, Tom and Askell, Amanda and Henighan, Tom and Drain, Dawn and Perez, Ethan and Schiefer, Nicholas and Hatfield-Dodds, Zac and DasSarma, Nova and Tran-Johnson, Eli and Johnston, Scott and El-Showk, Sheer and Jones, Andy and Elhage, Nelson and Hume, Tristan and Chen, Anna and Bai, Yuntao and Bowman, Sam and Fort, Stanislav and G...

    Pith/arXiv arXiv

  25. [25]

    Transactions on Machine Learning Research , year =

    Lin, Stephanie and Hilton, Jacob and Evans, Owain , title =. Transactions on Machine Learning Research , year =

  26. [26]

    arXiv preprint arXiv:2601.05414 , year =

    Zhao, Minda and Du, Yilun and Wang, Mengyu , title =. arXiv preprint arXiv:2601.05414 , year =

    Pith/arXiv arXiv

  27. [27]

    and Khashabi, Daniel and Hajishirzi, Hannaneh , title =

    Wang, Yizhong and Kordi, Yeganeh and Mishra, Swaroop and Liu, Alisa and Smith, Noah A. and Khashabi, Daniel and Hajishirzi, Hannaneh , title =. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

  28. [28]

    Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    Honovich, Or and Scialom, Thomas and Levy, Omer and Schick, Timo , title =. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

  29. [29]

    and Choi, Yejin , title =

    Liu, Alisa and Swayamdipta, Swabha and Smith, Noah A. and Choi, Yejin , title =. Findings of the Association for Computational Linguistics: EMNLP 2022 , pages =

    2022

  30. [30]

    Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages =

    Bonifacio, Luiz and Abonizio, Hugo and Fadaee, Marzieh and Nogueira, Rodrigo , title =. Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages =

  31. [31]

    Brown, Tom B. and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom and Child, Rewon and Ramesh, Aditya and Ziegler, Daniel M. and Wu, Jeffrey and W...

  32. [32]

    and Zhou, Denny , title =

    Wei, Jason and Wang, Xuezhi and Schuurmans, Dale and Bosma, Maarten and Ichter, Brian and Xia, Fei and Chi, Ed and Le, Quoc V. and Zhou, Denny , title =. Advances in Neural Information Processing Systems (NeurIPS) , volume =

  33. [33]

    Advances in Neural Information Processing Systems (NeurIPS) , volume =

    Kojima, Takeshi and Gu, Shixiang Shane and Reid, Machel and Matsuo, Yutaka and Iwasawa, Yusuke , title =. Advances in Neural Information Processing Systems (NeurIPS) , volume =

  34. [34]

    Proceedings of the 10th International Conference on Learning Representations (ICLR) , year =

    Xie, Sang Michael and Raghunathan, Aditi and Liang, Percy and Ma, Tengyu , title =. Proceedings of the 10th International Conference on Learning Representations (ICLR) , year =

  35. [35]

    Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages =

    Min, Sewon and Lyu, Xinxi and Holtzman, Ari and Artetxe, Mikel and Lewis, Mike and Hajishirzi, Hannaneh and Zettlemoyer, Luke , title =. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages =

    2022

  36. [36]

    and Morris, Meredith Ringel and Liang, Percy and Bernstein, Michael S

    Park, Joon Sung and Popowski, Lindsay and Cai, Carrie J. and Morris, Meredith Ringel and Liang, Percy and Bernstein, Michael S. , title =. Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (UIST) , year =