Pith. sign in

REVIEW 5 major objections 8 minor 32 references

MultiQ&A: An Analysis in Measuring Robustness via Automated Crowdsourcing of Question Perturbations and Answers

T0 review · 5 major / 8 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MultiQ&A claims that gpt-3.5-turbo answer cohorts stay largely consistent across 1.9 million rephrased questions, and that the disagreements quantify hallucinations for pre-deployment auditing.

desk verdict A large-scale, well-integrated robustness pipeline whose main results table contains a mathematically impossible row, so the headline claim is currently unverified. read the letter →

arxiv 2502.03711 v1 pith:B4EDHO6P submitted 2025-02-06 cs.CL cs.AIcs.LG

Nicole Cho , William Watson This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords LLMrobustnessquestionperturbationscrowdsourcinghallucinationensemblevotinganswerconsistencyansweringgpt-3.5-turbo
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MultiQ&A is a stress test for question-answering LLMs: it takes a question, rewrites it into several lexical variants, answers each variant with a separate model agent, and compares the answers. The paper applies this to 376,201 questions, generating 1.9 million perturbations and 2.3 million answers across extractive, multiple-choice, and abstractive QA. It reports that gpt-3.5-turbo is largely consistent under these perturbations, while worst-case robustness is weakest in free-form answering and collapses on arithmetic reasoning. The aim is to give institutions a scalable way to measure confidence, consistency, and hallucination before deployment.

What carries the argument

Three cooperating gpt-3.5-turbo agents: a Query Rewriter that produces v+1 paraphrases (including the identity) at temperature 1.0; independent Answer Generators that answer each paraphrase with task-specific prompts; and an Aggregator that clusters answers with Sentence-BERT, re-ranks them with a cross-encoder, and computes agreement metrics (Fleiss's kappa, entropy certainty, Gibbs' M2) plus supervised robustness metrics (worst-case $\Omega$, best-case O, plurality vote). The pipeline treats answer variability across paraphrases as the signal of instability.

What would settle it

A hand-labeled sample of a few hundred perturbed question pairs, asking whether the ground-truth answer remains the same for each pair, would settle it: if a nontrivial share of paraphrases have a different correct answer, the robustness numbers conflate meaning change with inconsistency.

Watch

Extended reading notes

Core claim

The central claim is that an ensemble of independently prompted gpt-3.5-turbo agents answering the same question in several phrasings remains relatively robust and consistent, and that the disagreements between agents can be quantified to expose hallucinations. The paper packages this as MultiQ&A, a five-step pipeline that perturbs, answers, quantifies, plurality-votes, and visualizes, measuring robustness with worst-case and best-case accuracy, Fleiss's kappa, entropy-based certainty, and Cronbach's alpha. Across extractive, multiple-choice, and abstractive tasks, the paper finds a clear ordering: extractive outperforms multiple choice outperforms abstractive, with MathQA as an outlier where consistency collapses. The system also logged 2,293 content-filter refusals, which it suggests can enrich adversarial datasets for safety testing.

Load-bearing premise

The whole measurement rests on the assumption that the rewrites preserve the original question's meaning, so that answer differences count as model inconsistency rather than new questions; the paper never independently verifies that assumption.

Editorial extensions

If this is right

  • If MultiQ&A is right, institutions can audit an LLM's consistency before deployment by perturbing their own real questions at scale.
  • The extractive greater than multiple choice greater than abstractive ordering implies that context and constrained choices are protective, and free-form generation is where hallucinations concentrate.
  • Plurality voting across paraphrases gives a practical ensemble answer that is usually close to or better than the baseline answer.
  • The MathQA outlier shows that arithmetic tasks need separate scrutiny, since model consistency is not uniform across domains.
  • Automated perturbation can also generate adversarial examples and content-filter triggers for safety evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the same model both rewrites and answers the questions, the reported 'robustness' may partly reflect gpt-3.5-turbo's own paraphrasing tendencies rather than intrinsic semantic stability; a cross-model rewriter-answerer split could test this.
  • The metrics are computed from a small cohort of six phrasings per question, so increasing the number of perturbations or using human-verified paraphrases could materially change the measured consistency.
  • The 2,293 content-filter refusals could serve as a focused probe of safety behavior under paraphrase, an angle the paper starts but does not develop.
  • The ordinal ranking by scenario suggests a testable recipe for deployment: give the model extractive context when possible, constrain choices when not, and add extra monitoring for generative and numeric tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. The paper introduces MultiQ&A, a pipeline in which a gpt-3.5-turbo Query Rewriter generates five lexical perturbations of each input question, a set of gpt-3.5-turbo Answer Generator agents answers the original and perturbed questions, and an Aggregator computes accuracy, worst/best-case robustness, plurality-vote accuracy, and agreement/reliability statistics. The pipeline is applied to 12 QA datasets across extractive, multiple-choice, and abstractive formats, yielding roughly 1.9 million perturbed questions and 2.3 million answers. The central claim is that gpt-3.5-turbo remains 'relatively robust and consistent under perturbations' and that MultiQ&A can serve as a scalable framework for institutional LLM stress-testing.

Significance. If the empirical claims held, MultiQ&A would be a useful large-scale robustness evaluation tool: it combines standard psychometric and agreement metrics, covers diverse QA formats, and ships a concrete agent-based pipeline. The paper's strengths are its scale, the breadth of datasets, and the explicitness of the metric definitions in §5. However, the central empirical claim rests on Table 4, which contains a row that is impossible under the paper's own definitions, and on two unverified assumptions: that perturbations preserve semantics and that the answering agents are independent of the perturbation generator. Until these are addressed, the paper does not support its headline conclusion that gpt-3.5-turbo is relatively robust and consistent under perturbations.

major comments (5)
  1. [§5.1, Table 4] The Abstractive TruthfulQA validation row reports Base=52.4, Mode=28.1, Worst=61.8. Under the definition in §5.1, Ω = (1/n) Σ_j min_i m(f(T_i(x_j)), y_j), and T_0 is the identity transformation, so for every sample the minimum over raters is no greater than the correctness indicator of the original question. Therefore Ω ≤ A must hold in every row. The paper itself states the stronger hierarchy Ω ≤ min(Ŷ, A) ≤ max(Ŷ, A) ≤ O. The reported Worst=61.8 exceeds both Base=52.4 and Mode=28.1, which is a mathematical impossibility, not a statistical anomaly. This row must be recomputed from raw per-question responses, and the audit trail released, before the robustness claim in Table 5 and the abstract can be evaluated.
  2. [§3.1, Table 3 caption] The load-bearing assumption that perturbed questions preserve the semantic intent of q0 is asserted but never validated. The Query Rewriter prompt asks for 'radically different ways' of rewriting, and Table 3's own caption concedes that 'variations can predispose gpt-3.5-turbo to hallucinations.' No human evaluation, entailment/paraphrase classifier, or post-hoc semantic consistency check is reported. Consequently, answer variability across perturbations may reflect changed question meaning rather than model inconsistency, which would invalidate the interpretation of Ω, O, and the agreement metrics as measuring robustness to meaning-preserving reformulations.
  3. [§3.2] The Answer Generator agents are described as 'independent,' but they are all instances of gpt-3.5-turbo, and the Query Rewriter that produces the perturbations is also gpt-3.5-turbo. Independence is only at the level of separate prompts and separate API calls; there is no evidence that the answers are conditionally independent given the question, nor that the perturbation-generation process is independent of the answer-generation process in a way that the robustness metrics require. The reported robustness scores should be framed as measuring the self-consistency of a single model under its own rewrites, not as evidence about model robustness generally. A concrete fix would be to cross-perturb with one model and answer with another, or to report agreement statistics across multiple base models.
  4. [§4.2, Table 4, §6.4] Two sampling issues affect the generalizability of the aggregate numbers. First, MathQA train is evaluated on only 693 of 29,800 samples, with no description of how this subset was selected or whether it is representative; the MathQA row is also an outlier on every metric, so this selection matters. Second, §6.4 reports 2,293 cases where gpt-3.5-turbo failed due to content filtering, but the paper does not state whether these cases were excluded from the per-dataset and aggregate metric computations, nor how their exclusion biases the reported robustness and agreement numbers. The paper should report the number of excluded samples per dataset split and recompute Table 5 under both inclusion and exclusion policies.
  5. [§4.3, Table 2] The abstractive QA scoring protocols are heterogeneous and may not be comparable across datasets: TruthfulQA is scored by semantic similarity with correct options, WikiQA by a cosine similarity threshold of 60%, SciQ by approximate Levenshtein distance, and SQuADv2/TriviaQA/HotpotQA by exact or fuzzy matching. These thresholds and matching functions are not justified, and the differences can dominate the reported accuracy and robustness numbers. Table 5's aggregation across abstractive datasets therefore mixes incompatible correctness definitions; the paper should either use one shared scoring protocol or clearly separate datasets by scoring method in the aggregate analysis.
minor comments (8)
  1. [§5.1] The notation for the plurality metric is confusing: Ŷ is defined as a sum of indicator values but is described as 'the ensemble answer generated by plurality voting'; it is actually the accuracy of the mode, not the mode itself. Please rename the quantity or clarify the definition.
  2. [§3.3] The 'answer-critic model' used for re-ranking is mentioned without a citation or specification. Please state the model, its training data, and how semantic alignment is measured.
  3. [§6.2 vs Table 4] The text states Cronbach's α ≈ 42.8% on the MathQA test split, but Table 4 reports α = 45.2 for MathQA test. Please reconcile the numbers.
  4. [§1, §7] The introduction says the study covers 12 QA datasets, while the conclusion says '13 datasets'; Table 1 lists 12 datasets, but SQuADv2 is used in both extractive and abstractive modes, so the counting should be made consistent.
  5. [§5.3] The verbal description of Cronbach's α reverses the roles of the variance terms: 'σ2_y is the variance in scores across v+1 raters for each sample' should be the item variance, and 'σ2_x is the variance in total correct responses per rater' should be the total score variance. Please align the prose with the formula.
  6. [§6.1] There is a typo: 'even with with significant prompt alterations' should read 'even with significant prompt alterations.'
  7. [Table 4] The split label 'Extn' appears to be a truncated form of 'Extractive'; please use the full label for clarity.
  8. [§4.3] For TruthfulQA abstractive scoring by 'semantic similarity with correct options,' the similarity measure and threshold are not specified; please provide them or cite the exact implementation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; the pipeline's metrics are direct external measurements, though Table 4 contains a non-circular internal contradiction that must be corrected.

full rationale

The central derivation chain is self-contained: every reported quantity is a direct statistic over collected answers and external ground-truth labels, with no fitted parameter later relabeled as a prediction and no metric defined in terms of the claim it supports. The identity perturbation T0 is included by construction and the paper explicitly states the resulting bound Ω ≤ A, so the worst-case index is a lower-bound measure rather than a hidden restatement of the baseline. Self-citations to the authors' prior work appear only in the related-work section and are not load-bearing for the robustness measurement. The same-model design (gpt-3.5-turbo both rewrites and answers) is a genuine validity limitation on the claimed agent independence, but it is not a definitional circularity: answer correctness is still judged against external datasets, and the paper itself concedes perturbations can induce hallucinations. A separate, non-circular correctness defect must be flagged: Table 4's Abstractive TruthfulQA validation row reports Base=52.4, Mode=28.1, Worst=61.8, which violates the paper's own Ω ≤ min(Y-hat, A) relationship from §5.1; no data generated by the stated pipeline can produce that row. This is an internal inconsistency, not a reduction of a prediction to its inputs, but it means the headline robustness claim cannot be assessed until the row is recomputed and the audit trail is released.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central metrics involve no fitted quantities, but the evaluation depends on hand-chosen design decisions (v, temperature, similarity thresholds) and several unvalidated domain assumptions about perturbation quality and the meaning of disagreement.

free parameters (4)
  • number_of_perturbations_v = 5
    Chosen by hand; robustness and agreement metrics depend on v+1 raters.
  • temperature_tau = 1.0
    Chosen to maximize diversity in perturbation and answer generation; affects answer distribution.
  • wikiqa_cosine_threshold = 0.60
    Correctness for abstractive WikiQA is determined by cosine similarity above 60%; chosen without sensitivity analysis.
  • levenshtein_match_threshold = unspecified
    SciQ abstractive answers matched by approximate Levenshtein distance, but cutoff is not given.
assumptions (4)
  • domain assumption LLM-generated perturbations preserve the semantic intent of the original question.
    Stated in §3.1; never validated; Table 3 caption admits variations can predispose to hallucinations.
  • domain assumption Disagreement across perturbed answers indicates hallucination or inconsistency.
    Implied by the framing in abstract and §5; alternative valid answers are not considered.
  • domain assumption Ground-truth labels in the 12 QA datasets are correct and unambiguous.
    Standard practice, but no manual verification here; some datasets (e.g., SQuADv2) have multiple valid answer spans.
  • domain assumption Standard psychometric formulas (Fleiss kappa, Cronbach alpha, entropy) apply to LLM raters as if they were human raters.
    Used in §5 without discussion of violations (e.g., raters are not independent random samples).

how reviews work

0 comments
Cite this review

Pith. "Pith review of MultiQ&A: An Analysis in Measuring Robustness via Automated Crowdsourcing of Question Perturbations and Answers." pith.science (2026). https://pith.science/paper/B4EDHO6P

@misc{pith2026250203711,
  author       = {Pith},
  title        = {Pith review of: MultiQ&A: An Analysis in Measuring Robustness via Automated Crowdsourcing of Question Perturbations and Answers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B4EDHO6P}},
  note         = {Machine review of arXiv:2502.03711}
}
read the original abstract

One critical challenge in the institutional adoption journey of Large Language Models (LLMs) stems from their propensity to hallucinate in generated responses. To address this, we propose MultiQ&A, a systematic approach for evaluating the robustness and consistency of LLM-generated answers. We demonstrate MultiQ&A's ability to crowdsource question perturbations and their respective answers through independent LLM agents at scale. Our experiments culminated in the examination of 1.9 million question perturbations and 2.3 million answers. Furthermore, MultiQ&A shows that ensembled LLMs, such as gpt-3.5-turbo, remain relatively robust and consistent under perturbations. MultiQ&A provides clarity in the response generation space, offering an effective method for inspecting disagreements and variability. Therefore, our system offers a potential framework for institutional LLM adoption with the ability to measure confidence, consistency, and the quantification of hallucinations.

Figures

Figures reproduced from arXiv: 2502.03711 by the authors.

Figure 1
Figure 1. System Overview for MultiQ&A: A single question [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 12 canonical work pages

  1. [7]

    In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), 6769–6781

    Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), 6769–6781. Online: Associa- tion for Computational Linguistics. Khashabi, D.; Min, S.; Khot, T.; Sabharwal, A.; Tafjord, O.; Clark, P.; and Hajishirzi, H

    work page 2020

  2. [8]

    In Findings of the Association for Computational Linguistics: EMNLP 2020, 1896–1907

    UNIFIEDQA: Crossing Format Boundaries with a Single QA System. In Findings of the Association for Computational Linguistics: EMNLP 2020, 1896–1907. Online: Association for Computational Linguistics. Kojima, T.; Gu, S. S.; Reid, M.; Matsuo, Y .; and Iwasawa, Y

    work page 2020

  3. [9]

    arXiv:2205.11916

    Large Language Models are Zero-Shot Reasoners. arXiv:2205.11916. Kuder, G.; and Richardson, M

    arXiv

  4. [10]

    In Proceed- ings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, 2251–2277

    Gradient-based Constrained Sampling from Language Models. In Proceed- ings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, 2251–2277. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics. Landis, J. R.; and Koch, G. G

    work page 2022

  5. [12]

    arXiv:2206.02336

    Making Large Language Models Better Reasoners with Step-Aware Verifier. arXiv:2206.02336. Liang, P.; Bommasani, R.; Lee, T.; Tsipras, D.; Soylu, D.; Yasunaga, M.; Zhang, Y .; Narayanan, D.; Wu, Y .; Kumar, A.; Newman, B.; Yuan, B.; Yan, B.; Zhang, C.; Cosgrove, C.; Manning, C. D.; R ´e, C.; Acosta-Navas, D.; Hudson, D. A.; Zelikman, E.; Durmus, E.; Ladhak...

    arXiv

  6. [13]

    arXiv:2211.09110

    Holistic Evaluation of Language Models. arXiv:2211.09110. Lin, S.; Hilton, J.; and Evans, O

    arXiv

  7. [14]

    What Makes Good In-Context Examples for GPT- 3? arXiv:2101.06804. Lord, F. M

    arXiv

  8. [16]

    arXiv:2108.12237

    Evaluating the Ro- bustness of Neural Language Models to Input Perturbations. arXiv:2108.12237. Nogueira, R.; and Cho, K

    arXiv

Show all 32 references
  1. [17]

    arXiv:1901.04085

    Passage Re-ranking with BERT. arXiv:1901.04085. Nye, M.; Andreassen, A. J.; Gur-Ari, G.; Michalewski, H.; Austin, J.; Bieber, D.; Dohan, D.; Lewkowycz, A.; Bosma, M.; Luan, D.; Sutton, C.; and Odena, A

    1901 arXiv

  2. [18]

    arXiv:2112.00114

    Show Your Work: Scratchpads for Intermediate Computation with Lan- guage Models. arXiv:2112.00114. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C. L.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; Schulman, J.; Hilton, J.; Kelton, F.; Miller, L.; Simens, ...

    arXiv

  3. [19]

    arXiv:2203.02155

    Training language models to follow instruc- tions with human feedback. arXiv:2203.02155. Rajpurkar, P.; Jia, R.; and Liang, P

    arXiv

  4. [20]

    arXiv:1806.03822

    Know What You Don’t Know: Unanswerable Questions for SQuAD. arXiv:1806.03822. Rajpurkar, P.; Zhang, J.; Lopyrev, K.; and Liang, P

    arXiv

  5. [22]

    In Pro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Processing

    Sentence-BERT: Sen- tence Embeddings using Siamese BERT-Networks. In Pro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Processing . Association for Computa- tional Linguistics. Shannon, C. E

    2019

  6. [23]

    In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceedings of the 2023 Confer- ence on Empirical Methods in Natural Language Process- ing, 3607–3625

    The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Models. In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceedings of the 2023 Confer- ence on Empirical Methods in Natural Language Process- ing, 3607–3625...

    2023

  7. [24]

    arXiv:2302.13971

    LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971. Wang, A.; Pruksachatkun, Y .; Nangia, N.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. R

    arXiv

  8. [25]

    arXiv preprint 1905.00537

    Su- perGLUE: A Stickier Benchmark for General-Purpose Lan- guage Understanding Systems. arXiv preprint 1905.00537. Wang, X.; Wei, J.; Schuurmans, D.; Le, Q.; Chi, E.; Narang, S.; Chowdhery, A.; and Zhou, D

    1905 arXiv

  9. [26]

    arXiv:2203.11171

    Self-Consistency Improves Chain of Thought Reasoning in Language Models. arXiv:2203.11171. Watson, W.; Cho, N.; Balch, T.; and Veloso, M

    arXiv

  10. [27]

    In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 7144–7159

    Hid- denTables and PyQTax: A Cooperative Game and Dataset For TableQA to Ensure Scale and Data Privacy Across a Myriad of Taxonomies. In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 7144–7159...

    2023

  11. [28]

    arXiv:2201.11903

    Chain-of- Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903. Wilcox, A. R

    arXiv

  12. [29]

    In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems , CHI EA ’22

    PromptChainer: Chaining Large Language Model Prompts through Visual Programming. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems , CHI EA ’22. New York, NY , USA: Association for Computing Machinery. ISBN 9781450391566. Yang, Y .; Yih, W....

    2022

  13. [31]

    In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2369–2380

    HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2369–2380. Brussels, Bel- gium: Association for Computational Linguistics. Yu, W.; Iter, D.; Wang, S.; Xu, Y .;...

    2018

  14. [32]

    arXiv:2110.07159

    Inter- preting the Robustness of Neural NLP Models to Textual Perturbations. arXiv:2110.07159. A Disclaimer This paper was prepared for informational purposes by the Artificial Intelligence Research group of JPMorgan Chase & Co. and its affiliates (”JPMorgan”) and is not a pro...

    arXiv

  15. [2015]

    In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , 2013–2018

    WikiQA: A Chal- lenge Dataset for Open-Domain Question Answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , 2013–2018. Lisbon, Por- tugal: Association for Computational Linguistics. Yang, Z.; Qi, P.; Zhang, S.; Bengio, Y .; Coh...

    2015

  16. [2016]

    arXiv:1606.05250

    SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv:1606.05250. Reimers, N.; and Gurevych, I

    arXiv

  17. [2017]

    arXiv e-prints , arXiv:1705.03551

    triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints , arXiv:1705.03551. Karpukhin, V .; Oguz, B.; Min, S.; Lewis, P.; Wu, L.; Edunov, S.; Chen, D.; and Yih, W.-t

    arXiv

  18. [2018]

    arXiv:1803.05457v1

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Chal- lenge. arXiv:1803.05457v1. Cohen, J

    arXiv

  19. [2019]

    MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based For- malisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 235...

    2019

  20. [2020]

    In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M.; and Lin, H., eds.,Advances in Neu- ral Information Processing Systems, volume 33, 1877–1901

    Language Mod- els are Few-Shot Learners. In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M.; and Lin, H., eds.,Advances in Neu- ral Information Processing Systems, volume 33, 1877–1901. Curran Associates, Inc. Cho, N.; Srishankar, N.; Cecchi, L.; and Watson, W

    1901

  21. [2021]

    arXiv:2005.11401

    Retrieval- Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401. Li, Y .; Lin, Z.; Zhang, S.; Fu, Q.; Chen, B.; Lou, J.-G.; and Chen, W

    2005 arXiv

  22. [2022]

    arXiv:2207.10342

    Lan- guage Model Cascades. arXiv:2207.10342. Fleiss, J. L

    arXiv

  23. [2023]

    arXiv:2304.13734

    The Internal State of an LLM Knows When It’s Lying. arXiv:2304.13734. Bisk, Y .; Zellers, R.; Bras, R. L.; Gao, J.; and Choi, Y

    arXiv

  24. [2024]

    arXiv:2410.05229

    GSM-Symbolic: Un- derstanding the Limitations of Mathematical Reasoning in Large Language Models. arXiv:2410.05229. Moradi, M.; and Samwald, M

    arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.