Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

The Non-Determinism of Small LLMs: Evidence of Low Answer Consistency in Repetition Trials of Standard Multiple-Choice Benchmarks

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Small LLMs answer the same multiple-choice question consistently only 50–80% of the time at low temperature, while 50B–80B models exceed 95%.

desk verdict Useful empirical map of small-model answer inconsistency, but the medium-model contrast is provisional until contamination is addressed; needs one solid revision, not a desk reject. read the letter →

arxiv 2509.09705 v1 pith:BD3J6CJS submitted 2025-09-05 cs.CL cs.AI

Claudio Pinhanez , Paulo Cavalin , Cassia Sanctos , Marcelo Grave , Yago Primerano This is my paper
classification cs.CLcs.AI
keywords answerconsistencynon-determinismsmalllanguagemodelsmultiple-choicebenchmarksinferencetemperatureMMLU-ReduxMedQAoracleguessing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that small language models (2B–8B parameters) are far less deterministic than their average scores suggest: when asked the same multiple-choice question ten times, they produce the same answer only about half to four-fifths of the time, even at low inference temperatures. Medium models (50B–80B) stay consistent on more than 95% of questions, pointing to a scale-linked reliability gap. The authors argue that accuracy on consistently answered questions tracks overall accuracy, so a model can decline to answer when it is not consistent and raise precision. The work matters because customer-service, medical, and safety-critical uses of LLMs assume that identical inputs get identical answers, an assumption these results call into question.

What carries the argument

The machinery is a formal definition of answer consistency as equivalence to an oracle guessing at a 0.99 success rate. For ten repetitions of a 4- or 5-choice question, this reduces to the rule: answer the same way in at least 9 of 10 trials (SURE) or not (UNSURE). The paper packages results as a pair, RWS | S/T—the share of SURE answers that are correct, and the share of questions that are SURE—and visualizes the temperature path in a consistency plot with quadrants labeled by reliability and consistency.

What would settle it

Repeat the 10-trial protocol on a question set verified to be absent from each model's training data. If medium models' S/T drops below 90% while small models stay in the 50–80% range, the scale gap is contamination; if small models rise above 90% at temperature 0.3, the reported inconsistency is a sampling artifact, not a capacity limit.

Watch

Extended reading notes

Core claim

The paper's central discovery is a quantified consistency gap: across 26 open-weight models on MMLU-Redux and MedQA, small models (2B–8B) achieve 0.99-consistency—defined as giving the same answer in at least 9 of 10 repetitions—on only 50–80% of questions at temperature 0.3, with the share falling as temperature rises. Medium models (50B–80B) reach S/T values of 87–99%. Accuracy among consistent answers (RWS) correlates with overall accuracy at low temperature (R² = 0.488 on MMLU-Redux, 0.831 on MedQA at t=0.3), so filtering out inconsistent questions can raise precision. Finetuned medical models can exceed their base models in both accuracy and consistency, as medllama3-v20 shows with 96%

Load-bearing premise

The results assume that benchmark contamination and rogue memorization are negligible; if medium models memorized MMLU-Redux, their near-perfect consistency reflects memory, not stable reasoning.

Editorial extensions

If this is right

  • Single-run evaluation scores for small models overstate their reliability; a model averaging 60% accuracy may flip answers on a large share of those it gets right.
  • Requiring 0.99-consistency before trusting an answer lets an application trade coverage for precision: on MedQA, Bio-Medical-Llama-3-8B improves RWS to 0.79 while covering only 60% of questions.
  • Low temperature (0.3) maximizes the share of consistently answered questions, while higher temperature (1.0) increases accuracy among those consistent answers on MMLU-Redux—a trade-off that flips for MedQA, where low temperature is best on both axes.
  • The 50–80% S/T band reproduces across the granite family, which was trained to avoid benchmark contamination, suggesting the inconsistency is not an artifact of a single model or benchmark.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If consistency tracks reliability, a cheap runtime gate could ask the model the same question two or three times and abstain when answers diverge; the paper's RWS|S/T curves give a direct way to price that gate's false-abstention rate.
  • The 9-of-10 threshold inherits the oracle's 0.99 success rate; choosing a stricter or laxer bar (e.g., 10-of-10 or 7-of-10) would compress or widen the reported consistency ranges and could be tuned per application risk tolerance.
  • A stronger test than identical repetition would be paraphrase consistency—asking semantically equivalent rewordings—which the paper lists as future work; if paraphrase consistency is lower than repetition consistency, the 50–80% band is an upper bound on real-world reliability.
  • The scale-linked gap suggests a testable hypothesis: intermediate sizes (e.g., 13B–30B) should interpolate between the 50–80% and >95% bands, providing a curve that could separate capacity effects from contamination effects.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies answer consistency of open-source LLMs when the same multiple-choice question is repeated 10 times. It proposes a definition of c-answer consistency through an oracle-guessing analogy, operationalizes it as a SURE/UNSURE classification (9 or 10 identical choices out of 10), and reports a two-number summary RWS | S/T for each model and temperature. Experiments cover MMLU-Redux and MedQA across 26 small (2B-8B) and medium (50B-80B) models at temperatures 0.3, 0.7, and 1.0. The central empirical claims are that small models produce consistent answers on only about 50%-80% of questions at low temperature, that medium models show much higher consistency (about 87%-99% S/T on MMLU-Redux), and that the accuracy of consistently answered questions (RWS) correlates with overall accuracy, especially at low temperature. The paper also introduces consistency plots as a visualization tool.

Significance. If the reported consistency range for small models is accurate, the paper provides practically important evidence that single-run benchmark scores can mask substantial trial-to-trial variability in small open LLMs. The RWS|S/T representation and the consistency plot are simple and potentially useful tools for reliability evaluation, and the use of open models avoids closed-API confounds such as caching. The paper explicitly reports machine-generated raw repetition data rather than fitting a model to a target. However, the headline scale-based conclusion and some of the aggregate statistics are currently not established because of data-reporting inconsistencies and because the medium-model S/T values, which carry the size comparison, are exactly the values most vulnerable to the benchmark-contamination confound that the paper itself acknowledges in Section 6.

major comments (4)
  1. [Tables 1-2] The empirical tables contain arithmetic inconsistencies that make the reported RWS/S/T values unreliable as printed. In Table 1, the Llama-3-8B row at t=0.3 lists SURE&right=43%, UNSURE&right=18%, UNSURE&wrong=29%, and SURE&wrong=104%, which is impossible and does not sum to 100%. In Table 2, BioMistral-7B at t=1.0 has SURE&right=7% and SURE&wrong=28%, so RWS should be 7/(7+28)=0.20, but the table reports 0.02. Because RWS and S/T are the paper's central quantities, the tables must be regenerated and checked before the empirical claims can be assessed.
  2. [Section 4.5] The aggregate S/T report is inconsistent with Table 1. The text states that for 'all 9 small models' at t=0.3 the average S/T is 69% with standard deviation 11%, implying a range of 58%-80%. The nine small-model S/T values in Table 1 at t=0.3 are 53, 79, 74, 78, 78, 63, 52, 78, and 24; their average is 64%, not 69%. The reported value is obtained only by silently dropping Granite-3.1-2b-base, the lowest cell. Similarly, Section 4.3 says the granite S/T range is 52%-75% after ignoring that model, but Table 1 shows 78% for two granite instruct models. If the outlier is excluded, the exclusion must be stated and justified; otherwise the aggregate and the '50%-80%' range need revision.
  3. [Sections 6-7] The paper's own limitations section concedes that benchmark contamination 'may have inflated the number of consistently corrected answers due to rogue memorization, especially for medium models.' This is not a peripheral caveat: the sole evidence for the claim that 'limited consistency is mostly an issue of small models' is the 96%-99% S/T of the three medium models in Table 1, and contamination would mechanically inflate exactly that quantity if the model has memorized the benchmark item. The paper does not provide a contamination-filtered recomputation, a memorization diagnostic, or any analysis separating stable generation from memorized reproduction. Without such a check, the size-based conclusion is confounded. A concrete test is needed, for example removing items that a model can reproduce from a masked or paraphrased prompt, or applying an existing contamination audit to MMLU-Redu
  4. [Section 3.1, Eqs. (1)-(3)] The derivation does not establish the stated equivalence between 'oracle guessing correctly at a 0.99 rate' and 'answering at least 9 of the 10 repetitions with the same choice.' The binomial calculation concerns the number of correct repetitions p, not the number of identical choices. Later, SURE is defined by identical answers, which may be wrong. The jump from 'correct with probability 0.99' to 'same letter 9 or 10 times' is informal, so the '0.99-consistency' label is not justified by the stated oracle model. Please either give a precise probabilistic model (e.g., a latent-answer model with probability c of emitting the same latent choice) or rename the threshold to avoid implying a false equivalence.
minor comments (5)
  1. [Introduction] Typo: 'MMLM-Redux' should be 'MMLU-Redux'.
  2. [Appendix A, Table 3] The reference mapping is garbled: Mixtral-8x7b-instruct is attributed to Qwen et al. (2025), while Qwen2-5-72b-instruct is attributed to Jiang et al. (2024). Please correct.
  3. [Section 4.1] The paper says top-K sampling was used but only specifies 'default top-K search inference parameters.' The actual top-k value should be stated, since it directly affects the degree of nondeterminism.
  4. [Section 5.1] The text says 'all the accuracy standard deviations are extremely small,' but Table 1 shows Mixtral-8x7b-instruct with stdev 0.093 and 0.105. Clarify whether this statement refers only to the MedQA results.
  5. [Section 4.4] The quadrant names (USEFUL, RELIABLE but inconsistent, etc.) are introduced after the plots are shown. Consider defining them formally in the text before the figures, or present them as illustrative labels rather than definitions.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the consistency metrics are direct measurements from repetition trials, and self-citations are peripheral.

full rationale

The paper's central chain is empirical: fixed prompts are run 10 times per question under top-K sampling at three temperatures, and the fraction of questions with 9/10 identical answers (S/T) and accuracy on those questions (RWS) are computed directly from the repetition data (Sections 3.2, 4, 5). No parameter is fitted to a target and then reported as a prediction; the correlation between RWS and average accuracy is a regression over measured model results, not a claim that is forced by construction. The 0.99 threshold is an explicitly stated operational decision (Section 3.1: 'a model is answer consistent when it is equivalent to an oracle guessing correctly at a 0.99 rate'), not a quantity derived from the data; whether 9/10 same-letter answers truly equals 0.99 oracle correctness is a validity question, not circularity. The hand-named consistency-plot quadrants (Section 4.4) are descriptive labels, not load-bearing derivations. The only self-citations are (a) Cavalin et al. 2024, used in Section 6 to name 'rogue memorization' as a possible contamination mechanism, and (b) IBM Granite Team 2024, used to justify that Granite models were trained to avoid benchmark contamination. Both are peripheral: the small-model inconsistency finding does not rest on Granite alone (Llama-3-8B, its instruct variant, and DeepSeek-7B show the same pattern), and the contamination caveat is explicitly acknowledged as a limitation ('may have inflated the number of consistently corrected answers due to rogue memorization ... especially for medium models'). That caveat weakens the medium-vs-small scale conclusion as a correctness/confound matter, but it does not make the derivation circular. No equation in the paper reduces to its own input, and no 'uniqueness theorem' is imported from the authors' prior work.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No numbers were fitted to data. The main hand-chosen quantities are the 0.99 consistency threshold and the 10-repetition design. The strongest unvalidated input is the contamination assumption, which the authors explicitly flag.

free parameters (3)
  • consistency threshold c = 0.99 (chosen by hand)
    Defines 0.99-consistency; operationalized as 9 or 10 identical answers out of 10. No derivation fixes this value.
  • number of repetitions M = 10
    Study design choice; the binomial threshold and SURE criterion depend on M=10.
  • temperature values = 0.3, 0.7, 1.0
    Hand-selected discretization of the temperature axis; results are only shown for these points.
assumptions (5)
  • standard math Repeated answers are independent and follow a binomial distribution with fixed success rate.
    Section 3.1 uses the binomial formula to compute oracle thresholds; independence is assumed.
  • ad hoc to paper Oracle guessing at success rate c is an appropriate model for answer consistency.
    Section 3.1 definition; the equivalence between 'consistently identical' and 'oracle correct' is asserted, not derived.
  • domain assumption Default top-K sampling with the stated prompt and max_new_tokens=3 is representative of standard evaluation practice.
    Section 4.1 sets inference parameters; no baseline with greedy decoding or other samplers is provided.
  • domain assumption Benchmark contamination and rogue memorization do not materially affect results.
    Authors state in Section 6 that contamination may have inflated consistent-correct answers, especially for medium models.
  • domain assumption MMLU-Redux and MedQA ground-truth labels are correct and the 10-repetition consistency measures reflect model behavior, not parsing artifacts.
    Sections 4.1 and 5 rely on letter parsing and benchmark labels.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Non-Determinism of Small LLMs: Evidence of Low Answer Consistency in Repetition Trials of Standard Multiple-Choice Benchmarks." pith.science (2026). https://pith.science/paper/BD3J6CJS

@misc{pith2026250909705,
  author       = {Pith},
  title        = {Pith review of: The Non-Determinism of Small LLMs: Evidence of Low Answer Consistency in Repetition Trials of Standard Multiple-Choice Benchmarks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BD3J6CJS}},
  note         = {Machine review of arXiv:2509.09705}
}
read the original abstract

This work explores the consistency of small LLMs (2B-8B parameters) in answering multiple times the same question. We present a study on known, open-source LLMs responding to 10 repetitions of questions from the multiple-choice benchmarks MMLU-Redux and MedQA, considering different inference temperatures, small vs. medium models (50B-80B), finetuned vs. base models, and other parameters. We also look into the effects of requiring multi-trial answer consistency on accuracy and the trade-offs involved in deciding which model best provides both of them. To support those studies, we propose some new analytical and graphical tools. Results show that the number of questions which can be answered consistently vary considerably among models but are typically in the 50%-80% range for small models at low inference temperatures. Also, accuracy among consistent answers seems to reasonably correlate with overall accuracy. Results for medium-sized models seem to indicate much higher levels of answer consistency.

Figures

Figures reproduced from arXiv: 2509.09705 by the authors.

Figure 2
Figure 2. Bar graph showing the proportions of SURE/UNSURE and right/wrong of the 3 medium mod￾els for the MMLU-Redux benchmark, 0.99-consistency, for 3 temperatures. scalable infrastructure with multiple GPUs through an available API. Technical details about the mod￾els are provided in section A of the Appendix. In terms of parameters, we employed the de￾fault top-K search inference parameters provided by transformers and ad… view at source ↗
Figure 3
Figure 3. Bar graph showing the proportions of SURE/UNSURE and right/wrong of the 3 granite models for the [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Consistency plots for the MMLU-Redux benchmark for small (top), medium (center), and granite (bottom) models. inference temperature. This helps visualizing how high levels of answer consistency and reliability are depicted in consistency plots. In the case of the granite models (bottom of figure 4), the same patterns of the small models (top of of figure 4) are seen, namely, paths which go right-to-left, bottom￾to-t… view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Scatter plot graphs showing the correlation be [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Bar graph showing the proportions of SURE/UNSURE and right/wrong of the 4 finetuned models for the MedQA benchmark, 0.99-consistency, for 3 temperatures. We considered the exact same infrastructure to run these evaluations, but given the smaller size of MedQA compared …
Figure 8
Figure 8. Figure 8: Bar graph showing the proportions of SURE/UNSURE and right/wrong of the 6 granite models for the [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 10
Figure 10. Figure 10: Consistency plot for the MedQA benchmark [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 9
Figure 9. Figure 9: Consistency plots for the MedQA benchmark [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 11
Figure 11. Figure 11: Scatter plot graphs showing the correlation [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Two Confounds in Cross-Model Value Comparison: Response Determinism and the Access Harness

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Cross-model value distances from single draws are inflated by response determinism and confounded by the deployment client; a repeated counterbalanced protocol plus flip/magnitude decomposition separates them.

Reference graph

Works this paper leans on

35 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [4]

    Samuel Ackerman, Ella Rabinovich, Eitan Farchi, and Ateret Anaby Tavor. 2024. https://aclanthology.org/2024.findings-emnlp.158 A novel metric for measuring the robustness of large language models in non-adversarial scenarios . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 2794--2802, Miami, Florida, USA. Association for C...

    work page 2024

  4. [5]

    deterministic

    Berk Atil, Sarp Aykent, Alexa Chittams, Lisheng Fu, Rebecca J. Passonneau, Evan Radcliffe, Guru Rajan Rajagopal, Adam Sloan, Tomasz Tudrej, Ferhan Ture, Zhe Wu, Lixinyu Xu, and Breck Baldwin. 2025. http://arxiv.org/abs/2408.04667 Non-determinism of "deterministic" llm settings

    arXiv 2025

  5. [6]

    Paulo Cavalin, Pedro Henrique Domingues, Claudio Pinhanez, and Julio Nogima. 2024. Fixing rogue memorization in many-to-one multilingual translators of extremely-low-resource languages by rephrasing training samples. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol...

    work page 2024

  6. [7]

    Yuxing Cheng, Yi Chang, and Yuan Wu. 2025. A survey on data contamination for large language models. arXiv preprint arXiv:2502.14425

    arXiv 2025

  7. [8]

    ContactDoctor. 2024. Bio-medical: A high-performance biomedical language model. https://huggingface.co/ContactDoctor/Bio-Medical-Llama-3-8B

    work page 2024

  8. [9]

    DeepSeek-AI and Aixin Liu et al . 2024. http://arxiv.org/abs/2412.19437 Deepseek-v3 technical report

    arXiv 2024

Show all 35 references
  1. [10]

    Abhimanyu Dubey and et al. 2024. http://arxiv.org/abs/2407.21783 The Llama 3 herd of models

    2024 arXiv

  2. [11]

    Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile van Krieken, and Pasquale Minervini...

    2025 arXiv

  3. [12]

    IBM Granite Team. 2024. https://github.com/ibm-granite/granite-3.0-language-models/ Granite 3.0 language models

    2024

  4. [13]

    Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexei Figueroa, Alexander Löser, Daniel Truhn, and Keno K

    Tianyu Han, Lisa C. Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexei Figueroa, Alexander Löser, Daniel Truhn, and Keno K. Bressem. 2025. http://arxiv.org/abs/2304.08247 Medalpaca -- an open-source collection of medical conversational ai models and training data

    2025 arXiv

  5. [14]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  6. [15]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024 arXiv

  7. [16]

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421

    2021

  8. [17]

    Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. 2024. http://arxiv.org/abs/2402.10373 Biomistral: A collection of open-source pretrained large language models for medical domains

    2024 arXiv

  9. [18]

    Noah Lee, Jiwoo Hong, and James Thorne. 2024. Evaluating the consistency of llm evaluators. arXiv preprint arXiv:2412.00543

    2024 arXiv

  10. [19]

    Wangyue Li, Liangzhi Li, Tong Xiang, Xiao Liu, Wei Deng, and Noa Garcia. 2024. https://aclanthology.org/2024.lrec-main.251 Can multiple-choice questions really be useful in detecting the abilities of LLM s? In Proceedings of the 2024 Joint International Conference on Computati...

    2024

  11. [20]

    Probe Medical. 2024. Medllama3 v20. https://huggingface .co/ProbeMedicalYonseiMAILab/medllama3-v20

    2024

  12. [21]

    Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. 2024. http://arxiv.org/abs/2410.05229 Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models

    2024 arXiv

  13. [22]

    Grigor Nalbandyan, Rima Shahbazyan, and Evelina Bakhturina. 2025. Score: Systematic consistency and robustness evaluation for large language models. arXiv preprint arXiv:2503.00137

    2025 arXiv

  14. [23]

    Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, Renqian Luo, Scott Mayer McKinney, Robert Osazuwa Ness, Hoifung Poon, Tao Qin, Naoto Usuyama, Chris White, and Eric Horvitz. 2023. http:/...

    2023 arXiv

  15. [24]

    Shuyin Ouyang, Jie M Zhang, Mark Harman, and Meng Wang. 2025. An empirical study of the non-determinism of chatgpt in code generation. ACM Transactions on Software Engineering and Methodology, 34(2):1--28

    2025

  16. [25]

    Aditya Patwardhan, Vivek Vaidya, and Ashish Kundu. 2024. Automated consistency analysis of llms. In 2024 IEEE 6th International Conference on Trust, Privacy and Security in Intelligent Systems, and Applications (TPS-ISA), pages 118--127. IEEE

    2024

  17. [26]

    Pouya Pezeshkpour and Estevam Hruschka. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.130 Large language models sensitivity to the order of options in multiple-choice questions . In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2006--2017...

    2024 doi

  18. [27]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025 arXiv

  19. [28]

    Mathieu Ravaut, Bosheng Ding, Fangkai Jiao, Hailin Chen, Xingxuan Li, Ruochen Zhao, Chengwei Qin, Caiming Xiong, and Shafiq Joty. 2024. How much are large language models contaminated? a comprehensive survey and the llmsanitize library. arXiv preprint arXiv:2404.00699

    2024 arXiv

  20. [29]

    Sara Mahdavi, Joelle Barral, Dale Webster, Greg S

    Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schaekermann, Amy Wang, Mohamed Amin, Sami Lachgar, Philip Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Aguera y ...

    2023 arXiv

  21. [30]

    Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. 2024. The good, the bad, and the greedy: Evaluation of llms should not ignore non-determinism. arXiv preprint arXiv:2407.10457

    2024 arXiv

  22. [31]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. http://arxiv.org/abs/2302.13971 Llama: ...

    2023 arXiv

  23. [32]

    Alan Mathison Turing. 1939. Systems of logic based on ordinals. Proceedings of the London Mathematical Society, Series 2, 45:161--228

    1939

  24. [33]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  25. [34]

    Haochun Wang, Sendong Zhao, Zewen Qiang, Nuwa Xi, Bing Qin, and Ting Liu. 2024. http://arxiv.org/abs/2402.01349 Beyond the answers: Reviewing the rationality of multiple choice question answering for the evaluation of large language models

    2024 arXiv

  26. [35]

    Fangyun Wei, Xi Chen, and Lin Luo. 2024. http://arxiv.org/abs/2403.07872 Rethinking generative large language model evaluation for semantic comprehension

    2024 arXiv

  27. [36]

    Cheng Xu, Shuhao Guan, Derek Greene, M Kechadi, et al. 2024. Benchmark data contamination of large language models: A survey. arXiv preprint arXiv:2406.04244

    2024 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.