Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 28-task benchmark drawn from 23 NLP papers finds that state-of-the-art LLM agents fail most attempts to reimplement masked functions from research code, with best unit-test accuracy at 42.9%.

desk verdict A genuinely useful benchmark for LLM agent code reproduction, with a credible low-accuracy headline result, but the unit-test ground truth needs validation before the specific numbers can be trusted. read the letter →

arxiv 2506.17335 v1 pith:2AJHJH5Q submitted 2025-06-19 cs.SE cs.AI

Shuo Yan , Ruochen Li , Ziming Luo , Zimu Wang , Daoyang Li , Liqiang Jing , Kaiyu He , Peilin Wu
show 6 more authors
George Michalopoulos Yue Zhang Ziyang Zhang Mian Zhang Zhiyu Chen Xinya Du
This is my paper
classification cs.SEcs.AI
keywords LLMagentscodereproductionbenchmarkNLPresearchrepository-levelgenerationunittestingLLM-as-a-judgescientificreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LMR-BENCH asks whether LLM agents can do a foundational piece of scientific labor: read a peer-reviewed NLP paper and reimplement one of its core functions inside the original code repository, with the function masked out. The benchmark comprises 28 such tasks drawn from 23 papers at major NLP venues, spanning categories from training objectives to evaluation metrics. Across standard prompting and a coding-agent setting, the best models pass all associated unit tests on at most 42.9% of tasks, and agent-based runs score slightly lower than direct prompting. The paper concludes that state-of-the-art models still lack the abstract reasoning and cross-file integration needed to reproduce research code, and that LMR-BENCH can track this gap; a sympathetic reader would take that as the central claim.

What carries the argument

The central object is the benchmark itself, a set of 28 code-reproduction tasks that each pair a paper, a repository with masked target functions, a structured instruction, a golden implementation, and a small unit-test suite with task-specific checker functions. Masking the function forces the model to reconstruct an algorithm from its textual description, while the surrounding repository forces it to handle dependencies that span files. Two evaluation instruments carry the measurement: an automated unit-test harness executed in isolated container environments, and an LLM-as-a-judge pass that assigns each implementation to one of three categories: logically incorrect, logically correct but incorrectly implemented, or completely correct. Together they separate functional correctness from fidelity to the reference design, so the benchmark can distinguish 'understood the idea' from 'implemented it faithfully.'

What would settle it

Re-run the 28 tasks with the same models and golden functions but with a second, independently authored unit-test suite sampled from the same original datasets; if pass rates shift by more than the observed gaps between models, the benchmark's ranking is an artifact of the specific tests rather than a stable measure.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that code reproduction from NLP research papers is an unsolved capability for current LLM agents, and LMR-BENCH is offered as a reliable instrument for measuring it. Each of the 28 tasks masks one or more functions in a real repository, provides the paper, the codebase, and an instruction, and scores the generated function by running human-curated unit tests inside task-specific container environments alongside an LLM judge that sorts implementations into logically incorrect, logically correct but incorrectly implemented, or completely correct. The measured ceiling is 42.9% unit-test accuracy under standard prompting, with the agent setting lower, which the paper reads as evidence that even the strongest models fail most reproduction tasks. The error analysis attributes the largest share of agent failures (43.1%) to incomplete comprehension of the problem context, not to syntax errors alone, leading to the claim that the bottleneck is scientific reasoning and repository-level synthesis.

Load-bearing premise

The load-bearing premise is that the human-written unit tests and per-task checker functions are faithful, discriminative ground truth; if those tests are too lenient, too strict, or anchored to a single recorded run of the golden function, the reported pass rates do not measure reproduction ability.

Editorial extensions

If this is right

  • Current LLM agents cannot reliably reproduce code from NLP research papers: best unit-test accuracy is 42.9% under standard prompting, and the agent setting performs slightly worse.
  • The deficit spans understanding and execution: both LLM-as-a-judge and human evaluation find nontrivial shares of implementations that are logically correct but incorrectly implemented.
  • Repository structure affects outcomes: deeper, less branched directory hierarchies predict higher success, while higher branch factor predicts lower success in the agent runs.
  • The dominant failure mode is incomplete comprehension of the problem context (43.1% of agent errors), followed by brittle code generation (27.6%) and cross-file retrieval (13.8%).
  • Neither unit tests nor LLM-as-a-judge alone is sufficient; the two metrics agree with each other and with human judgment at about 62.5% exact agreement and are complementary.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the benchmark's three-test-per-task design with task-specific tolerances makes the 25–43% pass-rate range a coarse estimate; an independent replication with a second test suite would show how much of the ranking is test-specific.
  • Editorial inference: because many source repositories predate model training cutoffs, a pass could partly reflect memorization; a held-out split of papers published after the models' cutoffs would separate genuine reproduction from recall.
  • Editorial inference: the cross-file retrieval errors suggest an actionable design target—agents given pre-indexed, dependency-aware repository context should outperform agents that must discover the same dependencies from a flat prompt, and that is directly testable on LMR-BENCH.
  • Editorial inference: the logistic-regression result that deeper, less branched repositories aid success implies a scaffold-level intervention: flattening or annotating dependency chains in the repository context should move agent pass rates, a prediction the benchmark can test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents LMR-BENCH, a benchmark of 28 code-reproduction tasks derived from 23 NLP papers published in top-tier venues. In each task an LLM is given the paper, the repository with one or more masked functions, and instructions, and must implement the missing function. The authors evaluate six configurations (GPT-4o, GPT-4.1, and o4-mini under standard prompting and with OpenHands) using two metrics: unit-test accuracy on human-curated tests executed in Docker containers, and LLM-as-a-judge classifications into three correctness categories. They report unit-test accuracy of 25.0-42.9% and conclude that state-of-the-art models and agents cannot yet reproduce NLP research code, with failure modes including paper parsing, incomplete context comprehension, cross-file retrieval, and policy errors. They also report a logistic regression linking repository structure to success.

Significance. If the measurement is reliable, LMR-BENCH is a valuable and well-scoped benchmark: it is the only one in its comparison table that combines published papers, repository-level tasks, human-curated unit tests, task-specific Docker environments, and a source task of reproducing LLM/NLP papers. The concrete failure examples in Appendix F, the action analysis for OpenHands, and the error-cause taxonomy are informative, and the Docker-based execution is a real strength. The main uncertainty is not the utility of the benchmark but whether the headline accuracies are valid measures of reproduction ability, since the unit-test ground truth is small, task-specific, and not yet demonstrated to be deterministic or discriminative; the paper's own human-evaluation section does not currently resolve this concern.

major comments (4)
  1. [Section 3.2, Unit Test Evaluation Preparation] The central measurement, unit-test accuracy in Table 2, is built from approximately three hand-authored test cases per task with task-specific checker functions (e.g., value-difference tolerances and a BERTScore threshold) and golden outputs recorded during the reproduction phase. The paper does not establish that these suites are deterministic or discriminative: with three cases, a partially correct or even semantically wrong implementation can pass by chance, while a functionally correct implementation that differs in constant handling, numerical ordering, or stochastic behavior can fail (the latter possibility is acknowledged in Section 5.3). Because all headline conclusions are filtered through this measurement, please add a per-task account of test counts, checker thresholds, and golden-output stability, plus a mutation or fault-injection analysis quantifying false passes and false failures; without this, Table 2 does not yet measure reproduction ability.
  2. [Section 5.3 and Appendix D] The paper claims that its unit tests are validated by human evaluation, but the evidence is internally inconsistent and does not support the claim. The annotation protocol changes across the paper: 12 annotators in Section 3.2, four annotators in Section 5.3, and 30 specialists drawing a 'random sample of 40 papers' in Appendix D even though the benchmark contains 28 tasks. Table 2 also contradicts the text's ranking: for OpenHands (o4-mini), human 'Completely Correct' is 39.3% while unit-test accuracy is 35.7%, so unit tests do not 'achieve the highest accuracy' in that condition. Moreover, no ground-truth definition is given for a judge's 'accuracy', and no per-task comparison of human labels against unit-test outcomes is reported; the 62.5% agreement between the LLM judge and humans does not validate the unit tests. Please provide a consistent human-evaluation protocol with per-task confusion matrices between human judgments, unit tests, and LLM judge.
  3. [Section 4, Experimental Results] The capability conclusion depends on the assumption that the evaluated models have not memorized the public repositories from which the 28 tasks were constructed. All evaluated models were trained on public GitHub data, and the source papers and repositories are prominent, yet the paper reports no contamination analysis, no repository release dates, and no held-out variants. Because LMR-BENCH is intended to track progress over time, please report per-task contamination status and, where possible, evaluate on modified or renamed repositories; otherwise the conclusion about 'persistent limitations' may conflate generalization failure with memorization effects.
  4. [Section 3.3, LLM-as-a-Judge Evaluation] The paper presents LLM-as-a-judge as a complementary metric, but its validity is not established. Section 5.3 reports that it performs worst among the three evaluation methods and contradicts human judgments 9.5% of the time, and Appendix A shows the judge is asked to compare against the golden reference, which the paper itself acknowledges can penalize correct alternative implementations. There is no report of the judge model, sampling temperature, prompt variation, or per-task agreement with unit tests. Please provide these details and a calibration analysis before presenting the judge distribution as a reliable second metric.
minor comments (5)
  1. [Section 5.3] The paragraph reporting LLM-human agreement states 'They agreed 62.5% of the time' and then concludes 'the agreement exceeds 70%; these two numbers need to be reconciled with a precise definition of the agreement measure.
  2. [Figure 3] The figure caption contains the leftover annotation 'If you want to revise this figure please make a copy,' which appears to be an editing artifact and should be removed.
  3. [Figure 6] The caption refers to 'pass papers' and 'fail papers'; these should be 'passed tasks' and 'failed tasks,' and the y-axis label should be made explicit.
  4. [Table 2] The table reports no confidence intervals; with only 28 tasks, differences of 3.6 percentage points (e.g., between GPT-4.1 and o4-mini in unit-test accuracy) are within sampling noise, so bootstrap intervals or an equivalent uncertainty estimate should be added.
  5. [Section 5.1] The logistic regression treats each (task, model) observation as independent, but the 168 observations are clustered within 28 tasks; the reported standard errors may be too small, and a mixed-effects model or clustered standard errors would be more appropriate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LMR-BENCH evaluates models against external golden-recorded unit tests and human-curated checkers, so the headline accuracy does not reduce to the models' own outputs.

full rationale

LMR-BENCH's headline result—unit-test accuracies of 25–42.9%—is measured by executing candidate implementations in Docker containers and comparing outputs to golden outputs recorded by human annotators during a reproducibility pass (Section 3.2, 'Unit Test Evaluation Preparation'). The golden outputs and checker functions are external to the evaluated models; they were not fit to or derived from the model outputs reported in Table 2. The only LLM involvement in test creation is stated as 'LLMs are leveraged to support the creation of unit tests to enhance efficiency,' followed by 'all annotations undergo rigorous human review and refinement,' so the test ground truth is human-curated rather than defined by the models being scored. The LLM-as-a-judge metric is explicitly framed as a complementary similarity-to-reference measure and is not the load-bearing evidence for the paper's central claim; indeed Section 5.3 reports mismatches between LLM and human judgments. Self-citations (e.g., DSBench, MLR-Copilot, LLM4SR) appear only in related-work context and do not justify the benchmark's validity. The logistic regression in Section 5.1 is fitted to the same benchmark data for explanatory inference, not to produce the headline accuracy, and no fitted parameter is renamed as a prediction. The paper's stated limitations concern scalability and annotation cost, not a circular derivation. No equation or construction in the paper reduces the measured accuracy to the model outputs or to a self-citation, so no circular step is exhibited.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central measurement stands on three unpaid premises: the unit tests faithfully encode the papers' algorithms (thin, about three cases per task), the models are not contaminated by public repos (unchecked), and the 28 curated tasks generalize to NLP research reproduction (asserted, with selection criteria that exclude hard-to-reproduce repos). No parameters were fitted to produce the headline numbers, but the checker thresholds that set pass or fail are unreported.

free parameters (1)
  • Task-specific checker tolerances (e.g., BERTScore threshold for prompt engineering tasks) = unspecified
    These hand-chosen tolerances determine pass or fail on the unit-test metric, but their values are not reported, so the headline accuracies cannot be reconstructed from the text.
assumptions (4)
  • domain assumption The evaluated models (GPT-4o, GPT-4.1, o4-mini) were not effectively contaminated by the 23 source papers and repositories, which are public and predate the models' training data.
    No contamination analysis is provided in Section 4.1; memorized reference implementations would inflate measured reproduction ability, though the low accuracies suggest any such effect is not dominant.
  • domain assumption The 23 selected papers and 28 gold functions are representative of reproduction challenges in NLP research.
    Selection criteria in Section 3.2 (method-focused, reproducible, clear) exclude broken or underspecified repositories, so difficulty levels and failure modes may not generalize to all research code.
  • domain assumption Around three unit-test cases per task suffice to certify functional correctness.
    Section 3.2 states 'around 3 test cases' per task; for functions with several branches this is thin and may not distinguish correct from coincidentally-passing implementations.
  • ad hoc to paper Checker functions and tolerances faithfully map implementation differences to correctness judgments.
    Section 3.2 introduces task-specific checkers to absorb inherent variability in NLP implementations, but the thresholds are not published, so the mapping cannot be audited.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research." pith.science (2026). https://pith.science/paper/2AJHJH5Q

@misc{pith2026250617335,
  author       = {Pith},
  title        = {Pith review of: LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2AJHJH5Q}},
  note         = {Machine review of arXiv:2506.17335}
}
read the original abstract

Large language model (LLM) agents have demonstrated remarkable potential in advancing scientific discovery. However, their capability in the fundamental yet crucial task of reproducing code from research papers, especially in the NLP domain, remains underexplored. This task includes unique complex reasoning challenges in the intellectual synthesis of abstract concepts and the comprehension of code repositories with interdependent files. Motivated by this gap, we present LMR-BENCH, a benchmark designed to systematically evaluate the capability of LLM agents on code reproduction from Language Modeling Research. It consists of 28 code reproduction tasks derived from 23 research papers published in top-tier NLP venues over the past five years, spanning nine fundamental categories. Models are provided with a research paper, a code repository containing one or more masked functions, and instructions for implementing these functions. We conduct extensive experiments in standard prompting and LLM agent settings with state-of-the-art LLMs, evaluating the accuracy of unit tests and performing LLM-based evaluation of code correctness. Experimental results reveal that even the most advanced models still exhibit persistent limitations in scientific reasoning and code synthesis, highlighting critical gaps in LLM agents' ability to autonomously reproduce scientific research

Figures

Figures reproduced from arXiv: 2506.17335 by the authors.

Figure 1
Figure 1. Overview of LMR-BENCH. Given an NLP research paper and a corresponding codebase with masked functions, the LLM agent is tasked with reproducing the function, requiring its ability of scientific method understanding, abstract reasoning and cross-file understanding. debugging and issues resolving (e.g., SWE-bench (Jimenez et al., 2024), DebugBench (Tian et al., 2024)). While these benchmarks are valuable, they typical… view at source ↗
Figure 2
Figure 2. Question distribution in LMR-BENCH. bility of LLM research agents (Luo et al., 2025). PaperBench (Starace et al., 2025) assesses LLM agents to replicate 20 research papers from ICML 2024. However, this dataset requires reproduction entirely from scratch, which is far beyond existing agents’ ability and may lead to discrepancies com￾pared to human-curated codebases. In this paper, we build upon the concept of code ge… view at source ↗
Figure 3
Figure 3. Dataset annotation pipeline of LMR-BENCH. functions for reproduction; and (3) the definition of the target function, including the detailed de￾scription on its definition, input, output, and any additional steps required for implementation. The reproduction process involves two different setup: standard prompting and LLM agent settings. The output function is evaluated via a combination of unit tests and LLM-as-a-ju… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Dataset evaluation pipeline of LMR-BENCH. The agent is presented with a target paper and a masked test function. After reproduction, the test function is evaluated in two stages. First, an LLM judge assesses the code for correctness and alignment with the paper’s logic…
Figure 5
Figure 5. Figure 5: Action distribution in OpenHands agents with different backbone models. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Action counts comparison for pass papers [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Error distribution of OpenHands on LMR￾BENCH. code agent, with a limited context window, some￾times fail to recall or look up these dependencies. Consequently, it may attempt to redefine a function that already exists elsewhere, use a placeholder value for an unknown c…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?

    cs.CL 2026-06 unverdicted novelty 7.0 of 10

    NatureBench evaluates ten frontier AI coding agents on 90 tasks from Nature papers under web-search-disabled conditions and finds the strongest agent surpasses published SOTA on only 17.8% of tasks, succeeding mainly ...

Reference graph

Works this paper leans on

12 extracted references · 7 canonical work pages · cited by 1 Pith paper

  1. [1]

    Examine both implementations in detail , focusing on : - Logical correctness relative to the specification provided above - Handling of edge cases and error conditions - Subtle deviations such as off - by - one errors or missing checks

    work page

  2. [2]

    Classify your judgment into exactly one of the following categories :

    work page

  3. [3]

    Completely Correct : the implementation is fully faithful to the specification with no errors

    work page

  4. [4]

    InThe Twelfth In- ternational Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024

    Repobench: Benchmarking repository-level code auto-completion systems. InThe Twelfth In- ternational Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open- Review.net. Yuliang Liu, Xiangru Tang, Zefan Cai, Junjie Lu, Yichi Zhang, Yanjun Shao, Zexuan Deng, Helan Hu, Zengxian Yang, Kaikai An, Ruijun Huang, Shuzheng Si, She...

    arXiv 2024

  5. [5]

    OpenReview.net. Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Pat- wardhan. 2025. Paperbench: Evaluating ai’s ability to replicate AI research.CoRR, abs/2504.01848. 11 Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Y...

    arXiv 2025

  6. [6]

    Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo- level coding challenges. InProceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 13643–13658. Association for Computational Lin- guistics. Tianyi Zh...

    work page 2024

  7. [9]

    Incorrect Logic : the core algorithm deviates from the specification and produces wrong results

    work page

  8. [10]

    Logic Correct but Subtle Errors : the main algorithm matches the specification , but there are other implementation mistakes or omissions

    work page

Show all 12 references
  1. [12]

    instance_id

    For the chosen category , provide a concise rationale with two to four bullet points illustrating the key discrepancies or confirmations Output Format ( JSON ) : { category : <1 | 2 | 3 > , rationale : [ First key point ... , Second key point ... ] } B Prompt for Standard Prom...

    2023 arXiv

  2. [2021]

    Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N

    Evaluating large language models trained on code.CoRR, abs/2107.03374. Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Hua...

    2025 arXiv

  3. [2024]

    InForty-first In- ternational Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024

    Mlagentbench: Evaluating language agents on machine learning experimentation. InForty-first In- ternational Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenRe- view.net. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Day- iheng Liu, Lei Zhang, ...

    2024 arXiv

  4. [2025]

    Dong Huang, Qingwen Bu, Jie M

    Scholar inbox: Personalized paper recommen- dations for scientists.CoRR, abs/2504.08385. Dong Huang, Qingwen Bu, Jie M. Zhang, Michael Luck, and Heming Cui. 2023. Agentcoder: Multi-agent- based code generation with iterative testing and opti- misation.CoRR, abs/2312.13010. Qia...

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.