Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Artisan, an LLM agent that writes reproduction scripts, recreates 44 of 60 tables from software-engineering papers and flags 20 undisclosed errors in the papers or artifacts.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 02:35 UTC pith:WEVM5O2R

load-bearing objection Useful and transparent, but the 44/60 success claim mixes paper reproduction with artifact reproduction; the core ideas and the inconsistency findings are the real value. the 3 major comments →

arxiv 2602.10046 v2 pith:WEVM5O2R submitted 2026-02-10 cs.SE

Automated Table Reproduction via Code Generation

Doehyun Baek , Michael Pradel This is my paper
classification cs.SE
keywords artifact evaluationLLM agentcode generationtable reproductionreproducibilitybenchmarkpaper-artifact inconsistencyautomated judging
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to show that artifact evaluation—the labor-intensive, one-time check that a paper's code reproduces its results—can be automated by an LLM agent that writes a runnable script for each numeric table. The agent is given the paper, the table with its digits hidden, and the artifact URL; its script must regenerate the table from the artifact, and an automated judge rejects scripts that merely copy precomputed results. On a new benchmark of 60 tables from 23 software-engineering papers, the agent produces acceptable scripts for 44, outperforming a general-purpose LLM agent by a factor of 3.14x at an average cost of $0.45 and 48 minutes per table. The same process surfaced 20 previously undisclosed mismatches between papers and their artifacts. If these results hold, a large share of reproducibility checking could be run cheaply, repeatedly, and with executable evidence.

Core claim

Artisan's central claim is that reproducing a table of numeric results is best treated as a code-generation problem: the output is a script that anyone can run and audit, not a rating or a set of numbers. Two mechanisms carry the approach. First, table obfuscation replaces every digit in the target table with '?' so the agent is guided toward the right structure without seeing the expected values. Second, a two-tier judge executes the submitted script in a clean container and checks not only that the output matches the expected values but also, via an LLM-based method judge, that the script genuinely runs the artifact's computations rather than copying checked-in results or hard-coding answe

What carries the argument

The reproduction script is the central object: a self-contained program that regenerates a target table from the artifact and formats the output. The judging mechanism is the load-bearing component: an execution-based output judge runs the script in a fresh container and classifies the outcome as static error, runtime error, mismatched results, or success, giving partial feedback that reveals which values matched without leaking the rest; an LLM-based method judge then labels the script as copied results, last-mile reproduction, or full reproduction, accepting only the latter two. Table obfuscation (digits replaced by question marks) prevents the agent from hard-coding the expected table, an

Load-bearing premise

The benchmark's ground truth is the corrected version of each table, not the published numbers, so a script that matches the corrected table counts as a success even when it disagrees with what the paper actually prints.

What would settle it

Re-run the 44 accepted reproduction scripts and compare their output against the published table values in the paper itself; if substantially fewer than 44 match the printed numbers, the reported success rate measures reproduction of corrected tables rather than reproduction of the paper's results.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Artifact evaluation could shift from a one-time badge check to a repeatable, script-based audit that runs whenever the artifact or environment changes.
  • Researchers could run Artisan before submission to catch paper-artifact mismatches early, rather than discovering them during review or after publication.
  • The generated scripts give reviewers and readers executable evidence of what was reproduced, so a failure can be attributed to the paper or artifact rather than to the agent.
  • The 60-task benchmark provides a common yardstick for measuring how well different LLM agents reproduce software-engineering results, supporting systematic progress.
  • Because the cost is about $0.45 per table, screening all tables in a paper or a whole conference issue becomes economically feasible.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The benchmark scores success against corrected, artifact-derived tables rather than the numbers printed in the paper, so the 44/60 figure may overstate the agent's ability to reproduce the paper as published; the same runs are better read as detecting where the paper and artifact diverge.
  • Because the method judge is an LLM that does not execute scripts, the paper's own measurements show it lets about 10% of copied-results scripts through as successes and mislabels a small number of genuine scripts as copies; a judge with execution ability could close that gap.
  • The obfuscation and judging design could be extended beyond tables to figures, logs, or other quantitative artifacts, and the same agent loop could run continuously in a repository's CI to detect bit-rot in artifacts.
  • If the 3.14x gap over a general-purpose agent holds across other benchmarks, the gain may come less from the agent's planning than from the feedback loop that steers it; an ablation that varies only the feedback would test this.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents Artisan, an LLM agent that takes a research paper, a target numeric table, and an artifact URL, and produces an executable reproduction script. The agent is guided by table obfuscation, an artifact download mechanism, bash and formatting tools, and a two-tier judge that checks both the script's output and its reproduction method. To evaluate Artisan, the authors introduce Artisan-Bench, a set of 60 tasks from 23 software engineering papers with manually validated ground-truth scripts. In the main evaluation, Artisan with GPT-5.1 succeeds on 44/60 tasks (24 full and 20 last-mile reproductions), outperforming the best baseline (mini-swe-agent) by 3.14x, and the authors report that the process helped uncover 20 paper-artifact inconsistencies.

Significance. If the headline result held as stated, this would be a useful advance: it reframes artifact evaluation as a code-generation task, provides executable evidence, and introduces a manually validated benchmark in a domain where no comparable benchmark exists. The paper's strengths include public code and data, careful manual validation of task feasibility, ground-truth scripts as evidence, a clear ablation study isolating the output judge, method judge, and format tool, and a transparent failure analysis. However, the main quantitative claim is materially weakened by the choice of ground truth: for 23 of 60 tasks the target is the corrected artifact-derived table rather than the table printed in the paper, and 20 of the 44 successes are last-mile scripts. The contribution is therefore better characterized as reproducing artifact-derived expected tables on a curated, feasible subset, and the abstract overstates what is measured.

major comments (3)
  1. [§3.3, Table 3] The success metric conflates reproduction of the paper's published results with reproduction of artifact-derived corrected results. Section 3.3 states that for the 20 inconsistency cases plus 3 author-acknowledged differences, 'we use the corrected version of the table as the task in our benchmark' — 23 of 60 tasks. A further 15 tasks use only part of a table and 4 exclude non-deterministic results. Because the agent receives the paper PDF, including the original printed values, the execution judge in these tasks rejects the published values and steers the agent toward artifact-derived values. Thus the 44/60 headline and the 3.14x advantage do not measure reproduction of the results reported in the paper. Moreover, the 20 'discovered' inconsistencies are removed from the success count by construction, so '44/60' and '20 errors' are not independent outputs of the same evaluation. Please r
  2. [§4.1.4, Table 3] The abstract's '44/60 reproduction scripts' counts 20 last-mile reproduction scripts, which reuse checked-in raw data and do not execute the experimental pipeline. Last-mile reproduction is a defensible contribution, but it is not full reproduction. The headline should disclose the 24 full / 20 last-mile split. The comparison with baselines should also be reported for full reproduction alone: Table 3 shows 24 full reproductions for Artisan vs 9 for mini-swe-agent, so the 3.14x ratio depends on including last-mile scripts. The current abstract gives an inflated impression of pipeline-level reproduction.
  3. [§2.5.1, Fig. 4] The claim that the judging mechanism steers the agent 'without exposing' expected results is inaccurate as written. In a mismatched-results feedback, the output judge displays the correct value for entries that matched: Figure 4c shows 'Resolved 7,799,929', which is the expected value from Figure 4a. For the 23 corrected-table tasks, this feedback leaks artifact-derived values that are not present in the published paper. This may be a reasonable reward-shaping design, but it should be acknowledged and its influence should be quantified, e.g., by an ablation that hides matched-value hints, so that the agent's success is not attributable to this leakage.
minor comments (5)
  1. [§4.2.3] Typo: 'erronous' should be 'erroneous'.
  2. [§4.6 heading] The section heading appears truncated as 'Discovery of Paper-Artifact In'.
  3. [§3.2] Minor wording issue: 'one papers in each of fuzzing, mutation testing, ...' should be 'one paper in each of ...'.
  4. [§4.3] Minor wording issue: 'Artisan to be impose higher costs' should be 'Artisan imposes higher costs'.
  5. [§4.1.4] The paper should state more prominently that the final success classification is manual for scripts whose output matches: the method category is manually assigned, not solely by the automated method judge. This is not a flaw, but it should be explicit in the effectiveness description.

Circularity Check

0 steps flagged

No load-bearing circularity; minor self-reference from including authors' own paper [13] and pre-identified inconsistencies.

full rationale

Artisan is an empirical benchmark-and-agent paper; its effectiveness claims rest on executing scripts against manually ground-truthed targets from 23 external papers, not on a mathematical derivation. Most circularity patterns therefore do not apply. The main benchmark-design choice—using corrected tables as targets for 23/60 tasks (Section 3.3)—is transparent and affects construct validity (the metric mixes paper reproduction with artifact reproduction), but it is not circular: corrected targets were derived by human/artifact inspection, and the same judge is applied to all agents. The only self-reference is that the task set includes 'action [13]' (Bouzenia and Pradel 2024), co-authored by the present second author, and Table 6 reports five inconsistencies from that paper as 'confirmed by the authors'; this weakens the independence of part of the 20-error claim but is not load-bearing for the headline 44/60 reproduction result, which is computed from all tasks and baselines. Sections 3.3 and 4.6 also show the 20 inconsistencies were identified during benchmark construction and then 'encountered' by Artisan; while rhetorically presenting pre-known errors as a discovered benefit, this does not reduce the main success metric to its inputs. No circular step meets the quote-and-reduction bar.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

No new physical or mathematical entities are introduced. The paper's free parameters are evaluation budgets and selection thresholds; its axioms are the benchmark-construction choices that define what counts as a reproducible task.

free parameters (3)
  • step_limit = 30
    Chosen by authors; 16/60 Artisan runs hit this limit with GPT-5.1, so the 44/60 success count is bounded by this hand-picked budget.
  • cost_limit = USD 1 per task
    Chosen as a reasonable LLM budget; changes which models/tasks are feasible (DeepSeek only enables 20 tasks for $1).
  • reproducibility_budget = 8 hours per task
    Artifact selection excludes papers whose experiments exceed 8h; this determines the benchmark's scope and feasibility.
axioms (4)
  • ad hoc to paper Reproduction of a table is judged against the corrected table produced from the artifact, not the exact published table.
    Section 3.3: for the 20 inconsistency cases and 3 acknowledged-difference cases, the authors 'use the corrected version of the table as the task'; this redefines the reproduction target and is a central design choice.
  • domain assumption The selected 23 Docker-packaged, non-GPU, non-non-public-API, under-8-hour papers are representative of SE artifacts.
    Section 3.2 filters 114 badge-holding papers down to 23; generalizing success rates to all SE artifacts assumes this subset is representative.
  • domain assumption A single run per agent/model configuration is sufficient to estimate effectiveness.
    Table 3 reports one run per configuration with no confidence intervals, despite stochastic LLM agents; the ranking may depend on noise.
  • domain assumption Manual ground truth scripts establish that each task is reproducible.
    Section 3.4 describes author-written scripts for every task as executable evidence; the benchmark's feasibility rests on these not containing hidden errors.

pith-pipeline@v1.3.0-alltime-deepseek · 20889 in / 13548 out tokens · 130222 ms · 2026-08-03T02:35:04.941726+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Automated Table Reproduction via Code Generation." pith.science (2026). https://pith.science/paper/WEVM5O2R

@misc{pith2026260210046,
  author       = {Pith},
  title        = {Pith review of: Automated Table Reproduction via Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WEVM5O2R}},
  note         = {Machine review of arXiv:2602.10046}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Reproducibility is an important goal in computer science research, e.g., for artifact evaluation and to build upon experimental results of prior work. Recently, LLM agents are being used to automatically reproduce research results, but they fail to provide executable evidence of reproduction and do not consider the method of reproduction, which limits their usefulness. We present Artisan, an LLM agent that reproduces tables of numeric results, given a paper and its artifact. The approach is enabled by two key contributions: First, we frame the reproduction problem as a code generation task, enabling users to audit and re-run the resulting reproduction script independently of the agent. Second, we design automated judging mechanisms that steer the agent toward correct results without exposing them, while preventing shortcuts like copying precomputed results. To evaluate Artisan, we introduce Artisan-Bench, the first benchmark assessing the ability to generate code that reproduces research results. Artisan-Bench comprises 60 tasks derived from 23 software engineering papers. Our experiments show that Artisan is effective and efficient, with the added benefit of aiding the discovery of 20 new errors in either the paper or artifact.

Figures

Figures reproduced from arXiv: 2602.10046 by Doehyun Baek, Michael Pradel.

Figure 1
Figure 1. Figure 1: Overview of Artisan. In summary, this paper contributes the following: • Artisan, the first automated artifact evaluation approach leveraging code generation capabilities of an LLM agent guided by automated judging mechanism. • Artisan-Bench, the first benchmark that evaluates LLM agents’ capabilities in generating repro￾duction scripts on software engineering papers. • An empirical evaluation showing that… view at source ↗
Figure 2
Figure 2. Figure 2: Challenges in automated artifact evaluation. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Example obfuscation and execution outputs for Table 3 of ScType [ [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Example of the mismatched results feedback for Table 2 of Drosos et al. [ [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Examples of three kinds of reproduction methods. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Part of the README of Drosos et al. [17] artifact. # [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Tradeoff between effectiveness and cost. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Breakdown of wall-clock time by activity. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Part of the full reproduction scripts reproducing Table 3 Zid et al. [ [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. On the Security of Research Artifacts

    cs.CR 2026-05 conditional novelty 7.0

    41.6% of insecure code findings in research artifacts pose practical security risks; the SAFE framework distinguishes them with 84.8% accuracy and 84.6% F1-score.

  2. Are Researchers Being Replaced by Artificial Intelligence?

    cs.CY 2026-04 unverdicted novelty 3.0

    AI is shifting researchers from creators to curators of generated content, risking loss of intellectual ownership and genuine understanding of science.

Reference graph

Works this paper leans on

46 extracted references · 1 canonical work pages · cited by 2 Pith papers

  1. [1]

    Artifact Evaluation Track — ASE 2024 (Call for Artifacts)

    ASE 2024 2024. Artifact Evaluation Track — ASE 2024 (Call for Artifacts). ASE 2024. https://conf.researchr.org/track/ase- 2024/ase-2024-artifact-evaluation-track#Call-for-Artifacts

    2024

  2. [2]

    Artifact Evaluation — ICSE 2024

    ICSE 2024 2024. Artifact Evaluation — ICSE 2024 . ICSE 2024. https://conf.researchr.org/track/icse-2024/icse-2024- artifact-evaluation

    2024

  3. [3]

    Artifacts — FSE 2024 (Call for Artifacts)

    ESEC/FSE 2024 2024. Artifacts — FSE 2024 (Call for Artifacts) . ESEC/FSE 2024. https://2024.esec-fse.org/track/fse-2024- artifacts , Vol. 1, No. 1, Article . Publication date: February 2026. Artisan: Agentic Artifact Evaluation 19

    2024

  4. [5]

    Florian Angermeir, Maximilian Amougou, Mark Kreitz, Andreas Bauer, Matthias Linhuber, Davide Fucci, Fabiola Moyón Constante, Daniel Méndez, and Tony Gorschek. 2025. Reflections on the Reproducibility of Commercial LLM Perfor- mance in Empirical Software Engineering Studies. CoRR abs/2510.25506 (2025). https://doi.org/10.48550/ARXIV.2510. 25506 arXiv:2510.25506

    doi:10.48550/arxiv.2510 2025

  5. [6]

    2020.Artifact Review and Badging — Current

    Association for Computing Machinery. 2020.Artifact Review and Badging — Current. https://www.acm.org/publications/ policies/artifact-review-and-badging-current Version 1.1

    2020

  6. [7]

    Manos Athanassoulis, Peter Triantafillou, Raja Appuswamy, Rajesh Bordawekar, Badrish Chandramouli, Xuntao Cheng, Ioana Manolescu, Yannis Papakonstantinou, and Nesime Tatbul. 2022. Artifacts Availability & Reproducibility (VLDB 2021 Round Table). SIGMOD Rec. 51, 2 (2022), 74–77. https://doi.org/10.1145/3552490.3552511

    arXiv 2022

  7. [8]

    Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi

    Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y. Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi. 2025. Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. CoRR abs/2503.11926 (2025). https://doi.org/10.48550/ARXIV.2503.11926 arXiv:2503.11926

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.11926 2025

  8. [10]

    Ben Bogin, Kejuan Yang, Shashank Gupta, Kyle Richardson, Erin Bransom, Peter Clark, Ashish Sabharwal, and Tushar Khot. 2024. super: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, Yase...

    doi:10.18653/v1/2024 2024

  9. [11]

    Devanbu, and Michael Pradel

    Islem Bouzenia, Premkumar T. Devanbu, and Michael Pradel. 2025. RepairAgent: An Autonomous, LLM-Based Agent for Program Repair. In47th IEEE/ACM International Conference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 - May 6, 2025 . IEEE, 2188–2200. https://doi.org/10.1109/ICSE55347.2025.00157

    arXiv 2025

  10. [12]

    Islem Bouzenia, Bajaj Piyush Krishan, and Michael Pradel. 2024. DyPyBench: A Benchmark of Executable Python Software. Proc. ACM Softw. Eng. 1, FSE (2024), 338–358. https://doi.org/10.1145/3643742

    doi:10.1145/3643742 2024

  11. [14]

    Islem Bouzenia and Michael Pradel. 2025. You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects. Proc. ACM Softw. Eng. 2, ISSTA (2025), 1054–1076. https://doi.org/10.1145/3728922

    doi:10.1145/3728922 2025

  12. [15]

    Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Aleksander Madry, and Lilian Weng. 2025. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, A...

    2025

  13. [16]

    Jimenez, John Yang, Leyton Ho, Tejal Patwardhan, Kevin Liu, and Aleksander Madry

    Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Leyton Ho, Tejal Patwardhan, Kevin Liu, and Aleksander Madry. 2024. Introducing SWE-bench Verified. https://openai.com/index/introducing-swe-bench-verified/

    2024

  14. [17]

    Georgios-Petros Drosos, Thodoris Sotiropoulos, Diomidis Spinellis, and Dimitris Mitropoulos. 2024. Bloat beneath Python’s Scales: A Fine-Grained Inter-Project Dependency Analysis. Proc. ACM Softw. Eng. 1, FSE (2024), 2584–2607. https://doi.org/10.1145/3660821

    doi:10.1145/3660821 2024

  15. [19]

    Dominik Helm, Tobias Roth, Sven Keidel, Michael Reif, and Mira Mezini. 2024. Unimocg: Modular Call-Graph Algorithms for Consistent Handling of Language Features. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024, Vienna, Austria, September 16-20, 2024 , Maria Christakis and Michael Pradel (Eds.). A...

    arXiv 2024

  16. [20]

    Ben Hermann. 2022. What Has Artifact Evaluation Ever Done for Us? IEEE Secur. Priv. 20, 5 (2022), 96–99. https: //doi.org/10.1109/MSEC.2022.3184234

    arXiv 2022

  17. [21]

    Ben Hermann, Stefan Winter, and Janet Siegmund. 2020. Community expectations for research artifacts and evaluation processes. In ESEC/FSE ’20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020 , Prem Devanbu, Myra B. Cohen, and Thomas , Vol. 1, No. 1, A...

    arXiv 2020

  18. [22]

    David Heye, Karl Kindermann, Robin Decker, Johannes Lohmöller, Anastasiia Belova, Sandra Geisler, Klaus Wehrle, and Jan Pennekamp. 2025. Supporting Artifact Evaluation with LLMs: A Study with Pub- lished Security Research Papers. In Proceedings of the 2025 IEEE International Conference on Big Data . IEEE. https://www.comsys.rwth-aachen.de/publication/2025...

    2025

  19. [23]

    Jaemin Hong and Sukyoung Ryu. 2024. To Tag, or Not to Tag: Translating C’s Unions to Rust’s Tagged Unions. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ASE 2024, Sacramento, CA, USA, October 27 - November 1, 2024 , Vladimir Filkov, Baishakhi Ray, and Minghui Zhou (Eds.). ACM, 40–52. https://doi.org/10.114...

    arXiv 2024

  20. [24]

    Chuxuan Hu, Liyun Zhang, Yeji Lim, Aum Wadhwani, Austin Peters, and Daniel Kang. 2025. REPRO-Bench: Can Agentic AI Systems Assess the Reproducibility of Social Science Research?. In Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025 , Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad T...

    2025

  21. [25]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net. https://openreview.net/forum? id=VTF8yNQM66

    2024

  22. [26]

    Shriram Krishnamurthi. 2013. Artifact evaluation for software conferences. ACM SIGSOFT Softw. Eng. Notes 38, 3 (2013), 7–10. https://doi.org/10.1145/2464526.2464530

    arXiv 2013

  23. [27]

    Shriram Krishnamurthi and Jan Vitek. 2015. The real software crisis: repeatability as a core value. Commun. ACM 58, 3 (2015), 34–36. https://doi.org/10.1145/2658987

    work page doi:10.1145/2658987 2015

  24. [28]

    Gichan Lee, Hansae Ju, and Scott Uk-Jin Lee. 2024. NeuroJIT: Improving Just-In-Time Defect Prediction Using Neurophysiological and Empirical Perceptions of Modern Developers. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ASE 2024, Sacramento, CA, USA, October 27 - November 1, 2024 , Vladimir Filkov, Baishak...

    arXiv 2024

  25. [29]

    Myungho Lee, Jiseong Bak, Seokhyeon Moon, Yoonchan Jhi, and Hakjoo Oh. 2024. Effective Unit Test Generation for Java Null Pointer Exceptions. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ASE 2024, Sacramento, CA, USA, October 27 - November 1, 2024 , Vladimir Filkov, Baishakhi Ray, and Minghui Zhou (Eds.)....

    arXiv 2024

  26. [31]

    Liang, Carmen Badea, Christian Bird, Robert DeLine, Denae Ford, Nicole Forsgren, and Thomas Zimmermann

    Jenny T. Liang, Carmen Badea, Christian Bird, Robert DeLine, Denae Ford, Nicole Forsgren, and Thomas Zimmermann

  27. [32]

    Monte MacDiarmid, Benjamin Wright, Jonathan Uesato, Joe Benton, Jon Kutasov, Sara Price, Naia Bouscal, Sam Bowman, Trenton Bricken, Alex Cloud, Carson Denison, Johannes Gasteiger, Ryan Greenblatt, Jan Leike, Jack Lindsey, Vlad Mikulik, Ethan Perez, Alex Rodrigues, Drake Thomas, Albert Webson, Daniel Ziegler, and Evan Hubinger

  28. [33]

    Al Muttakin, Saikat Mondal, and Chanchal Roy. 2026. The State of Open Science in Software Engineering Research: A Case Study of ICSE Artifacts. arXiv:2601.02066 [cs.SE] https://arxiv.org/abs/2601.02066

    arXiv 2026

  29. [35]

    Get in Researchers; We’re Measuring Reproducibility

    Daniel Olszewski, Allison Lu, Carson Stillman, Kevin Warren, Cole Kitroser, Alejandro Pascual, Divyajyoti Ukirde, Kevin R. B. Butler, and Patrick Traynor. 2023. "Get in Researchers; We’re Measuring Reproducibility": A Reproducibility Study of Machine Learning Papers in Tier 1 Security Conferences. In Proceedings of the 2023 ACM SIGSAC Conference on Comput...

    2023

  30. [37]

    Tobias Roth, Julius Näumann, Dominik Helm, Sven Keidel, and Mira Mezini. 2024. AXA: Cross-Language Analysis through Integration of Single-Language Analyses. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ASE 2024, Sacramento, CA, USA, October 27 - November 1, 2024 , Vladimir Filkov, Baishakhi Ray, and Mingh...

    arXiv 2024

  31. [38]

    Minju Seo, Jinheon Baek, Seongyun Lee, and Sung Ju Hwang. 2025. Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning. CoRR abs/2504.17192 (2025). https://doi.org/10.48550/ARXIV.2504.17192 arXiv:2504.17192

    doi:10.48550/arxiv.2504.17192 2025

  32. [39]

    Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, and Arvind Narayanan

    Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, and Arvind Narayanan. 2024. CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark. Trans. Mach. Learn. Res. 2024 (2024). https://openreview.net/forum?id=BsMMc4MEGS

    2024

  33. [40]

    Yahui Song, Xiang Gao, Wenhua Li, Wei-Ngan Chin, and Abhik Roychoudhury. 2024. ProveNFix: Temporal Property- Guided Program Repair. Proc. ACM Softw. Eng. 1, FSE (2024), 226–248. https://doi.org/10.1145/3643737

    doi:10.1145/3643737 2024

  34. [41]

    Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. 2025. PaperBench: Evaluating AI’s Ability to Replicate AI Research. In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, ...

    2025

  35. [42]

    SWE-agent contributors. 2025. mini-swe-agent. https://github.com/SWE-agent/mini-swe-agent. GitHub repository, MIT License. Version v1.14.4

    2025

  36. [43]

    Aichernig, and Christian Burghard

    Felix Wallner, Bernhard K. Aichernig, and Christian Burghard. 2024. It’s Not a Feature, It’s a Bug: Fault-Tolerant Model Mining from Noisy Data. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024 . ACM, 29:1–29:13. https://doi.org/10.1145/3597503.3623346

    arXiv 2024

  37. [44]

    Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, and et al. 2025. OpenHands: An Open Platform for AI Software Developers as Generalist Agents....

    2025

  38. [45]

    Solved Issues

    You Wang, Michael Pradel, and Zhongxin Liu. 2025. Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study. CoRR abs/2503.15223 (2025). https://doi.org/10.48550/ARXIV.2503.15223 arXiv:2503.15223

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.15223 2025

  39. [46]

    Stefan Winter, Christopher Steven Timperley, Ben Hermann, Jürgen Cito, Jonathan Bell, Michael Hilton, and Dirk Beyer. 2022. A retrospective study of one decade of artifact evaluations. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore,...

    arXiv 2022

  40. [47]

    Yanzheng Xiang, Hanqi Yan, Shuyin Ouyang, Lin Gui, and Yulan He. 2025. SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers. CoRR abs/2504.00255 (2025). https://doi.org/10. 48550/ARXIV.2504.00255 arXiv:2504.00255

    Pith/arXiv arXiv doi:10.48550/arxiv.2504.00255 2025

  41. [48]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

  42. [49]

    Brian Zhang. 2024. Towards Finding Accounting Errors in Smart Contracts. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024 . ACM, 138:1–138:13. https://doi.org/10.1145/3597503.3639128

    arXiv 2024

  43. [50]

    Cyrine Zid, Fiorella Zampetti, Giuliano Antoniol, and Massimiliano Di Penta. 2024. A Study on the Pythonic Functional Constructs’ Understandability. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024 . ACM, 237:1–237:13. https://doi.org/10.1145/3597503.3639211 , Vol. 1, No. ...

    arXiv 2024

  44. [51]

    SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024 , Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, a...

    2024

  45. [2024]

    ACM Softw

    Can GPT-4 Replicate Empirical Software Engineering Research? Proc. ACM Softw. Eng. 1, FSE (2024), 1330–1353. https://doi.org/10.1145/3660767

    doi:10.1145/3660767 2024

  46. [2025]

    arXiv:2511.18397 [cs.AI] https: //arxiv.org/abs/2511.18397

    Natural Emergent Misalignment from Reward Hacking in Production RL. arXiv:2511.18397 [cs.AI] https: //arxiv.org/abs/2511.18397

    arXiv