Pith. sign in

REVIEW 3 major objections 4 minor 30 references

LLMs cannot spot math errors, even when allowed to peek into the solution

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read State-of-the-art LLMs cannot reliably spot a student's first math error even with the gold solution in hand; a rewritten 'corrected student solution' improves most models.

desk verdict Useful empirical study on error localization with corrected student solutions, but the title oversells and the key comparison rests on thin validation of the corrected references. read the letter →

arxiv 2509.01395 v1 pith:KEHSTKGY submitted 2025-09-01 cs.CL cs.AI

KV Aditya Srivatsa , Kaushal Kumar Maurya , Ekaterina Kochmar This is my paper
classification cs.CLcs.AI
keywords errorlocalizationmeta-reasoningmathwordproblemsfirststepcorrectedstudentsolutionstepwisereasoningLLMevaluationsemanticalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tests a specific meta-reasoning skill: given a student's step-by-step solution to a math word problem, can an LLM say which numbered step is the first mistake? It finds that state-of-the-art LLMs fail this test even when given the correct reference solution, with exact-step accuracy staying near 65% on the easier dataset and near 51% on the harder one—far below the same models' ability to solve the problems. Their proposal is to interpose a 'corrected student solution': the LLM first rewrites the student's faulty solution so it keeps the student's method and variables but reaches the correct answer, then compares the original against that rewrite to find the first divergence. This correction-based condition outperforms the gold-solution condition for most models on both datasets, and the paper's feature analysis shows that alignment between the student's reasoning and the reference is the strongest predictor of successful localization while the model's own problem-solving skill is nearly irrelevant.

What carries the argument

The corrected student solution (S′)—an LLM-generated rewrite of the student's faulty steps that fixes the error while preserving the student's approach, notation, and step structure—is the object that carries the argument. The error is located as the first step at which the student trace S and S′ diverge. A second piece of machinery, Semantic Recall, uses BERTScore recall between the reference and the student solution truncated just before the first error to quantify how well a reference is aligned with the student's reasoning; the paper uses it to explain why corrections help, and it is the top-ranked feature in the classifier that predicts whether a model will localize the error correctly.

What would settle it

Take a sample of student solutions from VtG where the corrected student solution has been human-verified as correct and stylistically aligned up to the first error step. If models given these verified corrections locate the exact error no better than models given the gold solution, then alignment is not the operative cause. A simpler check: replace LLM-generated corrections with human-written ones that preserve the student's method; if accuracy does not rise above the gold-solution baseline, the benefit comes from the generated text itself rather than from approach alignment.

Watch

Extended reading notes

Core claim

The central claim is that access to a correct answer does not give an LLM the ability to critique an incorrect solution: across six models and two datasets, adding the gold solution to the prompt raises first-error localization only modestly, and exact-match accuracy remains far below each model's accuracy on the underlying math problems. The constructive discovery is that what the model is shown matters more than how well the model can solve the problem. If the supplied reference is first rewritten into a 'corrected student solution'—matching the student's step order, notation, and approach up to the point where the real mistake occurs, and only then diverging—most models locate the first e

Load-bearing premise

The method defines the error as the first divergence between the student trace and an LLM-generated corrected version of that trace, so it assumes the correction is mathematically right and faithful to the student's approach in the steps before the true mistake; when that assumption fails, as it does for one of the tested models, the correction hurts rather than helps.

Editorial extensions

If this is right

  • If the central claim holds, exact error-step localization remains a distinct meta-reasoning bottleneck: even the best settings reach only about 65% on VtG and 51% on PRM800K, well below the same models' problem-solving accuracy.
  • Supplying a corrected student solution—rather than a gold solution—is the intervention that consistently raises exact-match accuracy for most models on both datasets, including the largest general-purpose model tested.
  • The gain depends on correction quality: when a model generates an incorrect correction that still forces the final answer to match (Qwen2.5-72B-Math), the corrected-solution condition falls below even the no-gold baseline, so corrections need verification before use.
  • Semantic alignment between the reference and the student's reasoning up to the first error is the strongest predictor of successful error localization, while the model's ability to solve the problem is nearly irrelevant—so error-localization performance cannot be read off from math-skill benchmarks.
  • Error types bias the predictions: misunderstanding-type errors tend to be located later than they occur, while missing-or-extra-variable errors tend to be located slightly early, so practical tutors should condition on error type.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the alignment principle generalizes, the same corrected-trace comparison should transfer to stepwise reasoning in other domains—code debugging, proofs, or multi-step science calculations—where a corrected trace can be diffed against the original.
  • The strong decoupling of problem-solving from error localization suggests that training data which only asks models to produce correct solutions will not build critique ability; an explicit objective that first repairs an erroneous trace and then names the error step is a natural test of that idea.
  • A fully automatic version of this method could replace LLM-generated corrections with corrections from a symbolic solver or deterministic step-diffing tool, separating the alignment benefit from the correction generator's reliability and making the first-divergence rule auditable.
  • For deployed tutoring systems, the Qwen result is a caution: an unverified LLM correction can push feedback toward the wrong step, so corrections should be checked, or flagged with uncertainty, before being shown as the basis of a hint.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies first-error-step localization in stepwise student math solutions using the VtG and PRM800K datasets. It evaluates six LLMs in three settings: no reference solution (w/o-S), dataset-provided gold solution (w-GS), and a proposed two-stage condition (w-Cor) in which the LLM first generates a corrected student solution aligned with the student's approach and then localizes the first error as the first divergence between the student trace and the corrected trace. The authors report that w-GS improves over w/o-S but leaves accuracy low, and that w-Cor yields further gains for most models on both datasets. They also present a feature-importance analysis suggesting that semantic alignment between the reference and the student solution is the most predictive feature, and a chi-square analysis arguing that problem-solving ability is weakly correlated with error-localization ability.

Significance. If the central claim were fully supported, the paper would offer a practical and inexpensive method for improving error localization in tutoring systems, and its evidence that meta-reasoning is distinct from problem-solving ability would be valuable. The study has notable strengths: six diverse models, two complementary datasets, a public code/data release, manual annotation with reported inter-annotator agreement, and a concrete failure analysis for Qwen2.5-72B-Math. The finding that gold references help but are insufficient is credible and likely reproducible. However, the main w-Cor result currently rests on a confounded comparison with w-GS and on an under-validated assumption about the quality of LLM-generated corrected solutions. These issues are fixable but are load-bearing for the paper's headline claim.

major comments (3)
  1. [§2, §3, Table 1] The w-Cor condition is not a controlled comparison with w-GS. In w-Cor, the model first generates a corrected solution S' = LLMθ(P(G,S)) and then uses Q,S,S' for localization; in w-GS it only sees Q,S,G in a single prompt. Thus w-Cor has an additional LLM call that performs its own reasoning about the problem and the student trace. The reported improvements over w-GS (e.g., +11.8 points for Llama3-70B on VtG, +8.9 for Llama3.1-405B on PRM800K) could be attributable to this extra computation rather than to the alignment property of S'. To support the paper's claim that aligned corrected solutions improve localization, an ablation should hold the number of stages fixed—for example, a w-GS condition with an intermediate 'restate/elaborate the gold solution' step, or a condition where the corrected solution is generated without access to the gold solution. Without such a control, Table 1 doe
  2. [§C.2, Table 9, Limitations] The w-Cor method assumes the generated correction S' is mathematically correct and aligned with S up to the student's first error. This assumption is validated only on 90 corrections (15 per model). At per-model n=15, a reported 93.3% correctness rate has a wide confidence interval, and the paper's own Qwen2.5-72B-Math result (69.6% correctness) shows the assumption can fail. When it fails, the first-divergence rule points at the wrong step. Since most w-Cor gains over w-GS in Table 1 are between 2 and 9 points, an unmeasured correction-failure rate in that range could erase or reverse the reported effect. The Limitations section explicitly concedes that generated corrections 'may still harbor inconsistencies in their intermediate steps' and that expert-annotated corrections were not used. The manuscript should either validate correctness/alignment on the full evaluation sets (or a subst
  3. [§4.1, Figure 2] The failure analysis for Qwen2.5-72B-Math exposes a compounding-risk issue: the correction stage and the localization stage use the same model, so errors in S' systematically mislead the subsequent localization. The example in Figure 2 shows the original first error remaining in the corrected solution and later contradictory deductions being introduced. This makes w-Cor's benefit model-dependent in a way the paper does not characterize beyond the 15-sample manual check. Since the proposed method is intended as a general recipe, the authors should report a per-model analysis of correction quality versus localization error, or explicitly discuss the dependency and provide guidance on when w-Cor should not be applied.
minor comments (4)
  1. [§4.1] The text says Qwen2.5-72B-Math 'drops in both w-GS and w-Cor compared to w/o-S,' but Table 1 shows that on PRM800K w-GS (28.50) is actually higher than w/o-S (21.86). The statement is true for VtG, and for w-Cor on PRM800K, but not for w-GS on PRM800K.
  2. [Table 11] The Llama3-70B block contains a duplicated 'w-GS' row; one of the rows appears to be intended for a different setting or should be removed.
  3. [Figure 2] The figure caption labels the model as 'Qwen2.5-Math,' while the text and Table 6 consistently refer to 'Qwen2.5-72B-Math.' Please make the naming consistent.
  4. [§4.2, §C.1] Semantic Recall for w-Cor is computed against a corrected solution that was itself generated from the student solution, so high recall may be partly by construction. The feature-importance result should be interpreted with this in mind, and a brief caveat in the main text would be appropriate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the w-Cor pipeline generates a corrected student solution without using the first-error label, and the central claim is an empirical benchmark result.

full rationale

The paper's central claim is empirical: adding a corrected student solution (w-Cor) improves first-error-step localization over gold-solution (w-GS) and no-solution (w/o-S) baselines. The target label E (first erroneous step) is not used to construct the corrected solution S': the generation prompt (Figure 9) takes only the problem, the gold solution, and the student solution, and the localization prompt (Figure 8) takes the problem, the student solution, and S'. E is therefore not fed into the pipeline, and no fitted parameter is renamed as a prediction. The only self-citation (Srivatsa and Kochmar, 2024) supplies feature-extraction implementations for the auxiliary Random Forest analysis in Section C.4 and is not load-bearing for the main result. The Limitations section explicitly concedes that LLM-generated corrections 'may still harbor inconsistencies in their intermediate steps' and that only 90 corrections were manually verified; this is a validity/robustness caveat, not a circular reduction, because the method does not define E in terms of S' by construction. No uniqueness theorem or ansatz is imported from the authors' prior work. Thus the derivation chain is self-contained and no circular step is present.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The study relies on benchmark annotations, gold-solution correctness, a semantic similarity metric, and the reliability of LLM-generated corrections. No free parameters or invented entities are introduced.

assumptions (4)
  • domain assumption The first-error-step annotations in VtG and PRM800K are accurate and consistent.
    The experiments treat dataset annotations as ground truth for evaluation.
  • domain assumption The gold solutions in VtG and PRM800K are correct and can serve as a reliable reference.
    The w-GS and correction-generation settings assume the gold solution is a trustworthy target.
  • domain assumption BERTScore recall is a valid proxy for semantic alignment between student and reference solutions.
    Used in Section C.1 to measure alignment and as a feature in the Random Forest analysis.
  • domain assumption The LLM-generated corrected student solution, when used as a reference, has the first error at the first divergence from the student trace.
    This is the mechanism behind w-Cor; the paper's own Qwen results show it can fail.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLMs cannot spot math errors, even when allowed to peek into the solution." pith.science (2026). https://pith.science/paper/KEHSTKGY

@misc{pith2026250901395,
  author       = {Pith},
  title        = {Pith review of: LLMs cannot spot math errors, even when allowed to peek into the solution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KEHSTKGY}},
  note         = {Machine review of arXiv:2509.01395}
}
read the original abstract

Large language models (LLMs) demonstrate remarkable performance on math word problems, yet they have been shown to struggle with meta-reasoning tasks such as identifying errors in student solutions. In this work, we investigate the challenge of locating the first error step in stepwise solutions using two error reasoning datasets: VtG and PRM800K. Our experiments show that state-of-the-art LLMs struggle to locate the first error step in student solutions even when given access to the reference solution. To that end, we propose an approach that generates an intermediate corrected student solution, aligning more closely with the original student's solution, which helps improve performance.

Figures

Figures reproduced from arXiv: 2509.01395 by the authors.

Figure 1
Figure 1. In this example, the corrected version of the original student solution (with Llama3-70B) makes the location of the student’s first mistake more apparent as compared to the reference (i.e., gold) solution. intelligent tutoring systems (Jia et al., 2024; Niño￾Rojas et al., 2024). Accurate error detection and categorization promote better personalized and ef￾fective feedback generation (Anderson et al., 1990; Hattie a… view at source ↗
Figure 2
Figure 2. Qwen2.5-72B-Math is often unable to rectify the first error step (i) in the student’s solution when generating the corrected solution (ii). Instead, additional erroneous deductions (iii) are made later in the solution to make sure that the final answer matches that of the gold (reference) solution. Note that the model can correctly solve the corresponding step in a standalone problem-solving setup (iv). solutions an… view at source ↗
Figure 3
Figure 3. Relative Importance of Features Toward Cor [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Distribution of ground-truth error types in [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 6
Figure 6. Figure 6: Prompt without solution (w/o-S) Solution Type Model VtG PRM800K Gold – 89.52 85.12 Corrected Llama3-70B 94.77 94.46 Llama3.1-405B 96.04 95.34 Llama3.1-70B 96.18 95.88 GPT-4o 95.86 93.79 Qwen2.5-72B-Math 95.03 87.66 LearnLM-1.5-Pro 94.98 92.98 [PITH_FULL_IMAGE:figures/…
Figure 7
Figure 7. Figure 7: Prompt with gold solution (w-GS) masters degree in a STEM field. The first annotator annotates all 90 samples and the second annotator annotates just the agreement set. With this, we estimate that the inter-annotator agreement is κ = 0.82 for correctness and 0.85 for s…
Figure 9
Figure 9. Figure 9: Corrected solution generation prompt coefficient) across models, datasets, and prompt types. We interpret the correlation in each case by the corresponding p-values and ϕ-coefficients. A [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Error-step distance distributions for VtG −15 −10 −5 0 5 10 15 0 5 10 15 20 −15 −10 −5 0 5 10 15 0 5 10 15 20 −15 −10 −5 0 5 10 15 0 5 10 15 20 −15 −10 −5 0 5 10 15 0 5 10 15 20 −15 −10 −5 0 5 10 15 0 5 10 15 20 −15 −10 −5 0 5 10 15 0 5 10 15 20 −15 −10 −5 0 5 10 15 0…
Figure 11
Figure 11. Figure 11: Error-step distance distributions for PRM800K [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

30 extracted references · 12 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report . arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [4]

    Anderson, C

    John R. Anderson, C. F. Boyle, Albert T. Corbett, and R. Pelletier. 1990. Cognitive Tutors: Lessons Learned . In Advances in Instructional Psychology, volume 4, pages 179--202. Lawrence Erlbaum Associates, Inc

    work page 1990

  5. [5]

    Carpenter, Elizabeth Fennema, Maria L

    Thomas P. Carpenter, Elizabeth Fennema, Maria L. Franke, Linda Levi, and Sandra B. Empson. 1989. Cognition and teaching: The role of content knowledge . In Jane Kilpatrick, William G. Martin, and David Schifter, editors, Teachers’ Knowledge and the Mathematics Classroom, pages 49--84. National Council of Teachers of Mathematics

    work page 1989

  6. [6]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training Verifiers to Solve Math Word Problems . Preprint, arXiv:2110.14168

    arXiv 2021

  7. [7]

    Jacob Cohen. 1988. Statistical Power Analysis for the Behavioral Sciences , 2 edition. Lawrence Erlbaum Associates, Hillsdale, NJ

    work page 1988

  8. [8]

    Nico Daheim, Jakub Macina, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.478 Stepwise Verification and Remediation of Student Reasoning Errors with Large Language Model Tutors . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8386--8411, Miami, Florida, U...

    doi:10.18653/v1/2024.emnlp-main.478 2024

Show all 30 references
  1. [9]

    Le, and Thai Le

    Cuong Dang, Dung D. Le, and Thai Le. 2024. https://arxiv.org/abs/2402.11469 A Curious Case of Searching for the Correlation between Training Data and Adversarial Robustness of Transformer Textual Models . Preprint, arXiv:2402.11469

    2024 arXiv

  2. [11]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  3. [12]

    John Hattie and Helen Timperley. 2007. The Power of Feedback . Review of Educational Research, 77(1):81--112

    2007

  4. [13]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. https://arxiv.org/abs/2103.03874 Measuring Mathematical Problem Solving With the MATH Dataset . Preprint, arXiv:2103.03874

    2021 arXiv

  5. [14]

    Qinjin Jia, Jialin Cui, Ruijie Xi, Chengyuan Liu, Parvez Rashid, Ruochi Li, and Edward Gehringer. 2024. On Assessing the Faithfulness of LLM-generated Feedback on Student Assignments . In Proceedings of the 17th International Conference on Educational Data Mining, pages 491--499

    2024

  6. [15]

    Peter Kincaid, Robert P

    J. Peter Kincaid, Robert P. Fishburne, Richard L. Rogers, and Brad S. Chissom. 1975. https://apps.dtic.mil/sti/pdfs/ADA006655.pdf Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel . Techn...

    1975

  7. [16]

    Hang Li, Tianlong Xu, Kaiqi Yang, Yucheng Chu, Yanling Chen, Yichi Song, Qingsong Wen, and Hui Liu. 2024. https://arxiv.org/abs/2412.16838 Ask-Before-Detection: Identifying and Mitigating Conformity Bias in LLM-Powered Error Detector for Math Word Problem Solutions . Preprint,...

    2024 arXiv

  8. [17]

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. https://arxiv.org/abs/2305.20050 Let's Verify Step by Step . Preprint, arXiv:2305.20050

    2023 arXiv

  9. [18]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report . arXiv preprint arXiv:2412.19437

    2024 arXiv

  10. [19]

    Jakub Macina, Nico Daheim, Sankalan Chowdhury, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.372 M ath D ial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems...

    2023 doi

  11. [20]

    Francisco Ni \ n o-Rojas, Diana Lancheros-Cuesta, Martha Tatiana Pamela Jim \'e nez-Valderrama, Gelys Mestre, and Sergio G \'o mez. 2024. Systematic Review: Trends in Intelligent Tutoring Systems in Mathematics Teaching and Learning . International Journal of Education in Math...

    2024

  12. [21]

    OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Pai...

    2024 arXiv

  13. [22]

    Rea and Richard A

    Louis M. Rea and Richard A. Parker. 1992. Designing and Conducting Survey Research: A Comprehensive Guide . Jossey--Bass, San Francisco, CA

    1992

  14. [23]

    Sidney Siegel. 1956. Nonparametric Statistics for the Behavioral Sciences . McGraw--Hill, Singapore

    1956

  15. [24]

    Kv Aditya Srivatsa and Ekaterina Kochmar. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.72 What Makes Math Word Problems Challenging for LLM s ? In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1138--1148, Mexico City, Mexico. Association...

    2024 doi

  16. [25]

    John Sweller, Jeroen JG van Merri \"e nboer, and Fred Paas. 1998. Cognitive architecture and instructional design . Educational Psychology Review, 10(3):251--296

    1998

  17. [26]

    LearnLM Team, Abhinit Modi, Aditya Srikanth Veerubhotla, Aliya Rysbek, Andrea Huber, Brett Wiltshire, Brian Veprek, Daniel Gillick, Daniel Kasenberg, Derek Ahmed, Irina Jurenka, James Cohan, Jennifer She, Julia Wilkowski, Kaiz Alarakyia, Kevin R. McKee, Lisa Wang, Markus Kunes...

    2024 arXiv

  18. [27]

    Nandan Thakur, Suleman Kazi, Ge Luo, Jimmy Lin, and Amin Ahmad. 2025. https://aclanthology.org/2025.naacl-long.14/ MIRAGE -Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systems . In Proceedings of the 2025 Conference of the Nations of the Ame...

    2025

  19. [28]

    Gladys Tyen, Hassan Mansoor, Victor Carbune, Peter Chen, and Tony Mak. 2024. https://doi.org/10.18653/v1/2024.findings-acl.826 LLM s cannot find reasoning errors, but can correct them given the error location . In Findings of the Association for Computational Linguistics: ACL ...

    2024 doi

  20. [29]

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. 2024. https://arxiv.org/abs/2409.12122 Qwen2.5-Math Technical Report: ...

    2024 arXiv

  21. [30]

    Zhongshen Zeng, Pengguang Chen, Shu Liu, Haiyun Jiang, and Jiaya Jia. 2024. https://arxiv.org/abs/2312.17080 MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation . Preprint, arXiv:2312.17080

    2024 arXiv

  22. [31]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://arxiv.org/abs/1904.09675 BERTScore: Evaluating Text Generation with BERT . Preprint, arXiv:1904.09675

    2020 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.