Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

Evaluation is All You Need: Strategic Overclaiming of LLM Reasoning Capabilities Through Evaluation Design

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Evaluation design, not model quality, can move reasoning-benchmark scores by several percentage points.

desk verdict Useful empirical catalog of evaluation sensitivities for R1-Distill models, but the 'strategic overclaiming' attribution is asserted, not demonstrated, and the baseline statistics are fragile. read the letter →

arxiv 2506.04734 v2 pith:HXUPPZLR submitted 2025-06-05 cs.AI cs.CLcs.LG

Lin Sun , Weihong Lin , Jinzhu Wu , Yongfu Zhu , Xiaoqi Jian , Guangxiang Zhao , Change Jia , Linglin Zhang
show 3 more authors
Sai-er Hu Yuhan Wu Xiangzheng Zhang
This is my paper
classification cs.AIcs.CLcs.LG
keywords LLMevaluationbenchmarkreproducibilitydesignseedsensitivityoptionorderbiasDeepSeek-R1-Distillconfidenceintervalsreasoningmodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that benchmark scores for reasoning-focused LLMs are far less objective than they look: small, often unreported evaluation choices — how many times a question is sampled, which random seed is used, which version of a dataset, where an instruction sits, how multiple-choice options are ordered, and how the model is parallelized — can move scores by several percentage points. It demonstrates this on the DeepSeek-R1-Distill series and derivative open-source models across AIME24, AIME25, and GPQA Diamond. The authors argue that many published "improvements" over the base models are partly artifacts of favorable evaluation setups, and that the community should report confidence intervals, reproduce baseline scores under identical settings, and choose the sample size N from a statistically grounded formula. If the claim holds, comparisons between reasoning models need standardized protocols before gains can be trusted.

What carries the argument

The argument is carried by systematic controlled perturbation of seven evaluation variables, measured as absolute score differences against a control configuration (N=64, dynamic seeds, a fixed dataset version, instruction after the question, correct answer in option A, and a fixed tensor-parallelism setting). The quantitative instrument for the proposed standard is a central-limit-theorem sample-size formula, $N \geq (z_{\alpha/2} \, s / \epsilon)^2$, which estimates how many repeated samples are needed so the reported mean lies within a chosen error margin $\epsilon$ at confidence level $1-\alpha$. The paper uses this formula to show that the required N differs by model and benchmark, so fixed values like N=16, 32, or 64 are not universally justified.

What would settle it

Run the same control configuration (N=64, dynamic seed) on AIME24, AIME25, and GPQA Diamond for the four DeepSeek-R1-Distill models many times, say 30 independent reruns, and plot the spread of control scores. If that spread routinely covers the fluctuation magnitudes attributed to dataset version, seed, or option ordering, the claim that these evaluation choices are the main cause of multi-point shifts would be falsified.

Watch

Extended reading notes

Core claim

The central discovery is that evaluation design, not model capability, can account for a large share of apparent performance differences among reasoning models. Using controlled comparisons on four sizes of DeepSeek-R1-Distill and on other open-source reasoning models, the paper finds that switching AIME dataset versions changes scores by up to 3.9 percentage points, that seed choice alone can let a small model match or beat a larger one, and that option order and correct-answer placement in GPQA Diamond produce swings above 5 percentage points, with several models moving by 10 points or more. These magnitudes exceed the repeated-run baseline fluctuation the paper measures, leading the authors to conclude that many officially reported gains are partially attributable to evaluation setup. The paper's proposed remedy is a two-principle paradigm: transparency about every evaluation condition, and stability, meaning reporting statistically grounded confidence intervals rather than peak scores.

Load-bearing premise

All the comparison claims depend on treating a single repeated run of the control configuration as the baseline fluctuation; that one number has no confidence interval, so statements that a fluctuation "exceeds baseline" or that "over 75% of experiments" do so are not statistically supported.

Editorial extensions

If this is right

  • Reported benchmark gains of open-source reasoning models should not be trusted unless the evaluation script, dataset version, seed policy, N, option ordering, and hardware settings are disclosed.
  • Seed and option-order effects are large enough that a small model under a favorable configuration can appear to outperform a larger model on the same benchmark.
  • Benchmark scores that look like training improvements may instead be dataset-version or answer-position effects; direct comparison to baselines re-run under identical conditions is required.
  • A statistically chosen N, rather than a round number, should become the norm, with confidence intervals reported instead of peak scores.
  • Multiple-choice reasoning benchmarks should randomize or counterbalance option order, since correct-answer placement alone shifts results by over five points.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If these magnitudes generalize, leaderboard ranks among closely matched reasoning models may be within the noise of evaluation design; a controlled protocol could matter more than the next incremental training gain.
  • The option-order results suggest that GPQA Diamond partly measures position heuristics; a testable extension is to report position-balanced scores as the official metric.
  • The proposed N formula could be applied to any benchmark to compute minimum evaluation sizes; a practical audit is to re-run published leaderboard configurations and see how many reported gains survive.
  • The paper's framing implies evaluation design can be strategically tuned; a natural follow-up is an automatic audit that detects undocumented favorable configurations in released evaluation scripts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This manuscript reports a series of controlled evaluations of DeepSeek-R1-Distill models and derivative open-source reasoning models on AIME24, AIME25, and GPQA Diamond, varying the number of samples N, random seed strategy, dataset version, instruction position, option or answer ordering, and tensor parallelism. It finds that these incidental evaluation choices can move scores by up to several percentage points (e.g., 3.9 points on AIME25 dataset versions, over 5 points in GPQA option manipulations) and argues that many published performance improvements may be partially attributable to favorable evaluation design rather than genuine model gains. The authors propose an evaluation paradigm based on full transparency and statistically grounded stability, including a CLT-based method for choosing N and reporting confidence intervals.

Significance. If the empirical sensitivity results are correct, the paper makes a useful and timely contribution to LLM evaluation practice. The controlled comparisons cover many models, and the appendices include extensive per-run raw tables, which is a genuine reproducibility asset. The manuscript is also careful about documenting inference settings and reports baseline reruns. However, the paper's headline attribution claim—that reported performance gains are partially due to favorable evaluation setups—is not demonstrated by the data, and the proposed statistical protocol contains internal inconsistencies. These issues are fixable but currently weaken the central narrative.

major comments (3)
  1. [Section 1 and Table 4] The claim that 'many claimed performance gains in open-source models are partially attributable to favorable evaluation setups rather than genuine model improvement' is not established by the presented evidence. The paper reports fluctuation magnitudes for derivative models (Tables 11–13) but never audits a specific published result by rerunning the developer's original evaluation script and comparing it against a neutral configuration. The only direct comparison of official and control scores, Table 4, shows official GPQA Diamond results below the control group (e.g., 32B: 62.1 vs 67.4), which is opposite to the inflation narrative. The manuscript should either add concrete audits of specific claims or explicitly narrow the conclusion to sensitivity of benchmark scores to evaluation design.
  2. [Section 2.2 and Table 1] The baseline fluctuation is computed from a single repeated run, so statements such as 'over 75% of experiments exhibit deviations beyond the baseline fluctuation range' have no statistical content: a single-point baseline cannot define a range. The same issue affects the 'exceeds the baseline reference' statements in Sections 2.3, 2.4, and 2.7. Please rerun the control configuration multiple times (or model the sampling distribution) and report intervals, or replace the exceedance language with effect-size comparisons and confidence intervals.
  3. [Section 3.2, Eq. (2)–(3), Table 4] The 'Estimated Interval' is not derived in the text, and the reported intervals are internally inconsistent with the control-group means. For example, for DeepSeek-R1-Distill-Qwen-32B on AIME24 the interval is reported as 73.3±1 while the control mean is 71.8, and on AIME25 the interval is 53.9±1 while the control mean is 56.6. Moreover, because s in Eq. (3) is estimated from the same data used to construct the interval, the iterative procedure may stop prematurely; no convergence or coverage analysis is provided. The proposed N-calibration method should be validated experimentally (e.g., with split-sample or bootstrap checks) before being presented as a standard.
minor comments (6)
  1. [Table 4] Please define what the ± value represents (standard error, margin of error, or confidence-interval half-width) and state how s was computed; the notation and values are currently opaque.
  2. [Section 2.3 and Table 2] The sentence 'over 70% of evaluation results stabilize beyond N = 8' is ambiguous because Table 2 reports counts of experiments that have already stabilized at each minimum N; clarify the aggregation and whether the counts are cumulative.
  3. [Section 2.5] Given that baseline fluctuations can reach 1.4 percentage points (Table 1), instruction-position differences below 2 points may be within noise; consider reporting confidence intervals for these comparisons.
  4. [Appendix A.1, Table 5] The table uses footnote markers and unusual parameter formatting (e.g., the QwQ-32B row showing '1 3' and footnote references), which makes it hard to verify the exact values; please reformat for readability.
  5. [References] Several references are to Notion or Hugging Face URLs that may break or change, and one URL is split across lines; adding DOIs, version identifiers, or stable repository links would improve reproducibility.
  6. [Figures 1–7] The figures would benefit from error bars or per-condition point markers; currently the reader cannot assess the spread underlying the plotted fluctuations.

Circularity Check

0 steps flagged · score 1.0 of 10

No load-bearing circularity; the sensitivity results are controlled measurements, not derivations from fitted inputs or self-citations.

full rationale

This is an empirical sensitivity study rather than a derivation, and I find no circular step in its argument chain. The core evidence in Section 2 is a series of controlled comparisons where one evaluation variable is changed while the control configuration is held fixed; the reported fluctuations are measured absolute differences, not quantities defined as equal to their inputs. The N-selection proposal in Section 3.2 is a standard CLT-based sample-size calculation (Eqs. 1-3) and is not fitted to force Table 4; Table 4 is presented as a predictive check, and the stated failures are used as evidence that the appropriate N depends on model and benchmark. The paper's self-citations (TinyR1-32B-Preview, Light-R1, Zhao et al. 2025) concern evaluated derivative models or prior sensitivity findings, but the same sensitivity is demonstrated on non-self-cited models including DeepSeek-R1-Distill, QwQ-32B, Skywork, and OpenRS, so no load-bearing premise reduces to a self-citation. The most serious concern is evidentiary rather than circular: the headline attribution that reported gains are 'partially attributable to favorable evaluation setups' exceeds what the fluctuation magnitudes establish, because no specific official benchmark claim is reproduced under the original developer script versus the paper's configuration. That overreach is a validity and support gap, not a self-referential derivation.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No fitted parameters or invented entities. The paper's contribution is empirical, but it rests on statistical assumptions about i.i.d. sampling and the validity of the control configuration.

assumptions (4)
  • standard math Central Limit Theorem applies to pass@1 averaged scores across N independent repetitions.
    Invoked in Section 3.2 to derive the N formula. Assumes i.i.d. per-question outcomes and sufficient sample size.
  • domain assumption Evaluation repetitions with dynamic seeds are independent and identically distributed.
    Section 2.1 states that N-samples with dynamic seed can be considered theoretically well-randomized; this underpins the variance analysis.
  • domain assumption The control group configuration (N=64, dynamic seed, specific dataset versions) is a valid reference for comparing fluctuations.
    All fluctuation measurements are absolute differences from this control group; if this baseline is not stable or representative, the reported magnitudes are not meaningful.
  • domain assumption Official benchmark datasets and model-recommended inference parameters are appropriate for evaluating the central claim.
    The study uses official model cards and dataset versions as of April 26, 2025; results may depend on these choices.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluation is All You Need: Strategic Overclaiming of LLM Reasoning Capabilities Through Evaluation Design." pith.science (2026). https://pith.science/paper/HXUPPZLR

@misc{pith2026250604734,
  author       = {Pith},
  title        = {Pith review of: Evaluation is All You Need: Strategic Overclaiming of LLM Reasoning Capabilities Through Evaluation Design},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HXUPPZLR}},
  note         = {Machine review of arXiv:2506.04734}
}
read the original abstract

Reasoning models represented by the Deepseek-R1-Distill series have been widely adopted by the open-source community due to their strong performance in mathematics, science, programming, and other domains. However, our study reveals that their benchmark evaluation results are subject to significant fluctuations caused by various factors. Subtle differences in evaluation conditions can lead to substantial variations in results. Similar phenomena are observed in other open-source inference models fine-tuned based on the Deepseek-R1-Distill series, as well as in the QwQ-32B model, making their claimed performance improvements difficult to reproduce reliably. Therefore, we advocate for the establishment of a more rigorous paradigm for model performance evaluation and present our empirical assessments of the Deepseek-R1-Distill series models.

Figures

Figures reproduced from arXiv: 2506.04734 by the authors.

Figure 1
Figure 1. Score fluctuation ranges of the Deepseek-R1-Distill series models on relevant benchmarks [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. As the value of N increases, the fluctuation [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Under the 1-Seed-N setting, the evaluation [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Fluctuations in benchmark scores across dif [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Compared to other variables, changes in benchmark scores caused by instruction position are rel￾atively minor. However, they still introduce variations in evaluation stability. In certain model-benchmark com￾binations, placing the instruction before the question yields…
Figure 6
Figure 6. Figure 6: GPQA Diamond exhibits significant evaluation variability under option and correct answer biases. In [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: The impact of Tensor Parallelism (TP) varia [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Evaluating LLMs When They Do Not Know the Answer: Statistical Evaluation of Mathematical Reasoning via Comparative Signals

    cs.LG 2026-02 conditional novelty 6.0 of 10

    A one-step semiparametric estimator using pairwise comparison signals as control variates achieves the efficiency bound for estimating LLM accuracy on math benchmarks.

  2. Uncertainty Under the Curve: A Sequence-Level Entropy Area Metric for Reasoning LLM

    cs.AI 2025-08 conditional novelty 5.0 of 10

    Entropy Area Score sums token-level predictive entropy across a reasoning sequence, correlates with answer entropy, and selects SFT training data better than Pass Rate filtering in limited AIME experiments.

Reference graph

Works this paper leans on

27 extracted references · 11 canonical work pages · cited by 2 Pith papers

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Quy-Anh Dang and Chris Ngo. 2025. http://arxiv.org/abs/2503.16219 Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't

    arXiv 2025

  4. [4]

    DeepSeek-AI. 2025. http://arxiv.org/abs/2501.12948 DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    arXiv 2025

  5. [5]

    Andreas Hochlehnert, Hardik Bhatnagar, Vishaal Udandarao, Samuel Albanie, Ameya Prabhu, and Matthias Bethge. 2025. http://arxiv.org/abs/2504.07086 A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility

    arXiv 2025

  6. [6]

    HuggingFaceH4 . 2024. Huggingfaceh4/aime\_2024. https://huggingface.co/datasets/HuggingFaceH4/aime_2024

    work page 2024

  7. [7]

    Jiang Jinhao, Chen Zhipeng, Min Yingqian, Chen Jie, Cheng Xiaoxue, Wang Jiapeng, Tang Yiru, Sun Haoxiang, Deng Jia, Zhao Wayne Xin, Liu Zheng, Yan Dong, Xie Jian, Wang Zhongyuan, and Wen Ji-Rong. 2024. Enhancing LLM Reasoning with Reward-guided Tree Search . arXiv preprint arXiv:2411.11694

    arXiv 2024

  8. [8]

    He Jujie, Liu Jiacai, Liu Chris Yuhao, Yan Rui, Wang Chaojie, Cheng Peng, Zhang Xiaoyu, Zhang Fuxiang, Xu Jiacheng, Shen Wei, Li Siyuan, Zeng Liang, Wei Tianwen, Cheng Cheng, An Bo, Liu Yang, and Zhou Yahui. 2025 a . Skywork Open Reasoner 1 Technical Report . arXiv preprint arXiv:2505.22312

    arXiv 2025

Show all 27 references
  1. [9]

    He Jujie, Liu Jiacai, Liu Chris Yuhao, Yan Rui, Wang Chaojie, Cheng Peng, Zhang Xiaoyu, Zhang Fuxiang, Xu Jiacheng, Shen Wei, Li Siyuan, Zeng Liang, Wei Tianwen, Cheng Cheng, Liu Yang, and Zhou Yahui. 2025 b . Skywork Open Reasoner Series . https://capricious-hydrogen-41c.noti...

    2025

  2. [10]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention . In Proceedings of the ACM SIGOPS 29th Symposium on Operating ...

    2023

  3. [11]

    Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. 2025 a . DeepCoder: A Fully Open-Source 14B Coder at O3-mini Level . https://pretty-radio-b75...

    2025

  4. [12]

    Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica

    Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. 2025 b . DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RL . https://pretty-radio-b75.notion.site/DeepScale...

    2025

  5. [13]

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. http://arxiv.org/abs/2501.19393 s1: Simple test-time scaling

    2025 arXiv

  6. [14]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2023. http://arxiv.org/abs/2311.12022 GPQA: A Graduate-Level Google-Proof Q&A Benchmark

    2023 arXiv

  7. [15]

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2024. HybridFlow: A Flexible and Efficient RLHF Framework . arXiv preprint arXiv: 2409.19256

    2024 arXiv

  8. [16]

    Mingyang Song, Mao Zheng, Zheng Li, Wenjie Yang, Xuan Luo, Yue Pan, and Feng Zhang. 2025. http://arxiv.org/abs/2503.17287 FastCuRL: Curriculum Reinforcement Learning with Stage-wise Context Scaling for Efficient Training R1-like Reasoning Models

    2025

  9. [17]

    Lin Sun, Guangxiang Zhao, Xiaoqi Jian, Yuhan Wu, Weihong Lin, Yongfu Zhu, Change Jia, Linglin Zhang, Jinzhu Wu, Junfeng Ran, Sai er Hu, Zihan Jiang, Junting Zhou, Wenrui Liu, Bin Cui, Tong Yang, and Xiangzheng Zhang. 2025. http://arxiv.org/abs/2503.04872 TinyR1-32B-Preview: Bo...

    2025 arXiv

  10. [18]

    Qwen Team. 2025 a . https://qwenlm.github.io/blog/qwq-32b/ QwQ-32B: Embracing the Power of Reinforcement Learning

    2025

  11. [19]

    RUCAIBox STILL Team. 2025 b . https://github.com/RUCAIBox/Slow_Thinking_with_LLMs STILL-3-1.5B-preview: Enhancing Slow Thinking Abilities of Small Models through Reinforcement Learning

    2025

  12. [20]

    TinyR1 Team. 2025 c . https://huggingface.co/qihoo360/TinyR1-32B-Preview SuperDistillation Achieves Near-R1 Performance with Just 5\

    2025

  13. [21]

    Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, Haosheng Zou, Yongchao Deng, Shousheng Jia, and Xiangzheng Zhang. 2025. https://github.com/Qihoo360/Light-R1 Light-R1: Curriculum SFT, DPO and RL for Long COT from Scra...

    2025

  14. [22]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  15. [23]

    yentinglin . 2025. yentinglin/aime\_2025. https://huggingface.co/datasets/yentinglin/aime_2025

    2025

  16. [24]

    Min Yingqian, Chen Zhipeng, Jiang Jinhao, Chen Jie, Deng Jia, Hu Yiwen, Tang Yiru, Wang Jiapeng, Cheng Xiaoxue, Song Huatong, Zhao Wayne Xin, Liu Zheng, Wang Zhongyuan, and Wen Ji-Rong. 2024. Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning ...

    2024 arXiv

  17. [25]

    Guangxiang Zhao, Saier Hu, Xiaoqi Jian, Jinzhu Wu, Yuhan Wu, Change Jia, Lin Sun, and Xiangzheng Zhang. 2025. http://arxiv.org/abs/2502.12459 Stress Testing Generalization: How Minor Modifications Undermine Large Language Model Performance

    2025

  18. [26]

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2023. http://arxiv.org/abs/2309.03882 Large Language Models Are Not Robust Multiple Choice Selectors

    2023 arXiv

  19. [27]

    Zyphra. 2025. ZR1-1.5B: A small but powerful reasoning model for math and code

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.