Pith. sign in

REVIEW 5 major objections 4 minor 3 cited by

An LLM agent can reproduce human artifact-evaluation outcomes for 85% of real software artifacts, the paper claims, by modeling the evaluation as a state-aware workflow rather than a one-shot script.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 05:23 UTC pith:VGUQUSDU

load-bearing objection The abstract and body describe different systems and different results, and the headline BCR metric cannot actually measure badge agreement; the engineering work is real, but the paper needs major restructuring before it is publishable. the 5 major comments →

arxiv 2602.02235 v3 pith:VGUQUSDU submitted 2026-02-02 cs.SE

Agent-Based Software Artifact Evaluation

Zhaonan Wu , Yanjie Zhao , Zhenpeng Chen , Zheng Wang , Haoyu Wang This is my paper
classification cs.SE
keywords artifact evaluationLLM agentssoftware engineeringreproducibilityAE Graphautomationdependency modelingbadge consistency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Artifact evaluation is a standard but labor-intensive part of software engineering conferences, and this paper tries to show it can be automated end-to-end by a large language model agent. The proposed system, ArtifactCopilot, treats evaluation as a state-aware workflow: it converts README documentation into a dependency-aware AE Graph, normalizes containers so all commands run from the host, and uses two cooperating agents to plan and execute with automatic error recovery. Evaluated on 48 real artifacts from four SE conferences, the agent matches human-assigned badges in 85.42% of cases, needs only 0.11 human interventions per artifact, and costs about $0.09 per artifact—far outperforming a static script-generation baseline and a general-purpose interactive coding agent. If this holds, automated pre-screening could absorb the growing submission volume while leaving final judgment to human reviewers.

Core claim

The paper's central discovery is that the barriers to automating artifact evaluation are representational and environmental, not just a lack of model capability. After manually executing twelve artifacts, the authors identify three bottlenecks: READMEs interleave commands with prose so execution state is implicit; docker run -it and similar commands silently shift the execution context between host and container; and errors require iterative diagnosis and repair. ArtifactCopilot answers these with (1) an AE Graph whose nodes are commands and artifact files and whose edges are execution order and data dependencies, built by an LLM from the documentation; (2) an environment-normalization stage

What carries the argument

The AE Graph is the load-bearing representation: a directed graph in which command nodes and artifact nodes are connected by sequential-execution edges and artifact input/output edges, with each node holding execution state. It is constructed by an LLM from aggregated README, INSTALL, and SETUP files, then validated and refined during execution. Its role is to make implicit task dependencies explicit so the agent can query global state, isolate the blast radius of a failure, and continue from a checkpoint instead of restarting. Working alongside it are environment normalization (host-driven command injection into Docker containers, with custom entrypoints replayed through a detached shell) a

Load-bearing premise

The 85.42% agreement is measured against a ground truth built from the authors' manually extracted Golden Command Set and official conference badge outcomes, both treated as correct and complete—if official badges vary in strictness across venues and years, or the Golden Command Set omits commands that human reviewers actually ran, the headline number overstates automation quality.

What would settle it

An independent run of the same system on the same 48 artifacts, with a fresh set of human reviewers documenting every command they execute and every repair they make, would produce a new set of ground-truth labels; if the agent's agreement with those labels is substantially below 85.42%—or if the official badges disagree with the authors' functional labels—the claim does not survive.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the 85.42% figure holds, artifact evaluation can be pre-screened automatically at scale, reserving human review for borderline or high-risk cases.
  • The system's structured reports give reviewers a concrete map of which commands ran, failed, or were repaired, potentially making human evaluation faster and more consistent.
  • The AE Graph exposes underspecification in artifacts, so running the agent can give authors a concrete checklist of missing dependencies or ambiguous instructions before submission.
  • At roughly $0.09 per artifact, automated evaluation is cheap enough to run multiple times per artifact, enabling risk-based sampling rather than exhaustive manual checks.
  • The ablation results imply that any future automation system needs explicit dependency modeling, environment normalization, and task decomposition: skipping any one costs a large accuracy drop.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial observation: the paper's abstract reports a different headline result (a system called ArtifactGuide, 70.56% on 60 artifacts) than the body (ArtifactCopilot, 85.42% on 48 artifacts). These numbers and systems need reconciliation before the stronger figure is taken at face value.
  • A natural next step is to close the failure modes the paper itself identifies: integrating static code-level dependency analysis into the AE Graph so implicit imports and data-validity checks are caught before execution.
  • The approach could be turned into an author-facing pre-submission check: if an artifact passes ArtifactCopilot's run, it has at least satisfied an explicit command-based baseline, which would likely reduce reviewer intervention counts community-wide.
  • Given that the dataset excludes commercial software, private data, specialized hardware, and repositories larger than 10GB, the true scope is narrower than 'all artifacts'; measuring generalization to other venues and domains is the immediate empirical test.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper reports an end-to-end agent framework, ArtifactCopilot, that takes a paper PDF or a downloaded repository and produces an artifact-evaluation badge recommendation. The framework builds an AE Graph from README/documentation, uses a Planning Agent to construct and normalize a Docker-based execution environment, and uses an Evaluation Agent to execute commands, recover from errors, and determine a functional badge outcome. Evaluation on 48 artifacts from four SE venues reports an 85.42% Badge Consistency Rate (BCR), 0.11 human interventions per artifact, $0.091 average cost, and ablations showing large drops when the AE Graph, environment normalization, or hierarchical two-agent decomposition are removed. The abstract additionally advertises a separate scoring rubric called ArtifactGuide, a 70.56% agreement result over 60 artifacts, and a user study with 8 researchers; none of these appear in the body of the paper.

Significance. If the empirical claims were reliable, this would be a useful step toward semi-automated artifact evaluation: the AE Graph representation, environment normalization, and hierarchical planning/execution decomposition are sensible responses to real failure modes, and the ablation study and per-artifact cost analysis are informative. The paper also contributes a real-world dataset of 48 artifacts and a documented failure analysis. However, the headline metric does not measure what it claims, the abstract and body report different systems and different results, and a claimed user study is absent from the body. The current evidence does not support the central claim that ArtifactCopilot 'matches human artifact evaluation outcomes.'

major comments (5)
  1. [Abstract vs. §1, §5.2, Table 4] The abstract advertises ArtifactGuide, a structured scoring rubric, and reports 70.56% three-run mean exact badge agreement over 60 artifacts, plus a controlled user study with 8 experienced researchers. The body describes ArtifactCopilot with an AE Graph, reports 85.42% BCR on 48 artifacts, and contains no ArtifactGuide, no 60-artifact dataset, no three-run results, and no user-study section. These are different systems, different datasets, and different metrics; a reader cannot determine which result is the paper's actual claim.
  2. [§2 and §5.1 (BCR definition)] BCR is degenerate because the dataset contains no negative labels. Section 2 restricts the dataset to papers awarded Artifacts Functional or Artifacts Reusable badges and explicitly treats Reusable as functionally positive, so all 48 artifacts have a positive functional label. Consequently, BCR condition (2), 'the resulting evaluation outcome is consistent with the functional-level badge outcome,' is automatically true whenever the agent returns any positive outcome. An agent that executes all GCS commands and always predicts 'functional' cannot be penalized for false positives, even if a human reviewer would reject the artifact for incorrect outputs. The reported 85.42% therefore measures GCS command completion plus positive prediction, not agreement with human badge decisions.
  3. [§2 and §5.1 (ground truth)] The Golden Command Set is manually extracted by the authors from README files and is treated as the operational definition of successful execution, while official badges are used only as positive labels. The paper itself documents that badge criteria differ across venues and years (§2), and the GCS may omit commands that human reviewers actually ran or include commands without verifying their output semantics. The BCR is thus an agreement with an author-constructed command inventory, not with the official human evaluation process. The metric should be presented as a command-completion rate unless the GCS is independently validated against reviewer behavior.
  4. [Abstract vs. full text (user study)] The abstract claims that a controlled user study with 8 experienced researchers shows ArtifactCopilot reports improve reviewer confidence, help locate evidence, and clarify evaluation scope. No user study appears in the body: there is no protocol, no participant description, no measured outcomes, and no statistical analysis. This is a load-bearing missing piece of evidence and must either be added or removed from the abstract.
  5. [§3, §5.1, §5.2 (exploration/validation contamination)] The 12-artifact exploration set was used to identify challenges and inform system design, and it is then included in the headline 48-artifact BCR of 85.42%. The paper discloses this and reports a separate 86.11% on the 36 held-out artifacts, but the abstract and conclusion use the pooled figure. Since the design was shaped by the exploration set, the held-out-only result is the appropriate headline; reporting the pooled number as the main result overstates the evidence for the framework's general performance.
minor comments (4)
  1. [§4.4.2 vs. §6.3] Section 4.4.2 says each command is retried up to five times, while Section 6.3 says the retry threshold is k=3. Please reconcile the two numbers.
  2. [§4.4.2 and §6.3] The blocking-detection threshold of three minutes appears only in Section 6.3; Section 4.4.2 describes only 'sustained low CPU utilization across intervals.' Specify the threshold where the mechanism is introduced.
  3. [Table 4] Claude Code has no LCR or RASR entries. Clarify whether these metrics were not measured for this baseline or are not applicable.
  4. [§5.1 baselines] The LLM+Scripts baseline has zero interventions and $0.002 cost, but its BCR is 20.83%. A sentence explaining why the baseline is not trivially improved by adding a feedback loop would help the reader interpret the comparison.

Circularity Check

2 steps flagged

BCR degenerates to a command-completion rate because all 48 artifacts are labeled functionally positive; the headline 85.42% is not a badge-agreement rate.

specific steps
  1. self definitional [§2 Dataset Construction (functional normalization) + §5.1 BCR definition]
    "we normalize all evaluation outcomes to the Artifacts Functional level. As the Artifacts Reusable badge is awarded only when the functional criteria are satisfied and relies on subjective assessments of documentation quality, it is excluded from metric computation; artifacts awarded the Artifacts Reusable badge are thus treated as functionally positive. ... The primary outcome metric is the Badge Consistency Rate (BCR), defined as the proportion of artifacts for which both (1) all guideline-specified commands in the GCS are executed successfully, and (2) the resulting evaluation outcome is con"

    The dataset is restricted to papers that already received Functional or Reusable badges, and Reusable is relabeled as functionally positive. Therefore every artifact in the 48-artifact set has a positive functional label. Under this construction, BCR condition (2) cannot penalize false positives: any agent that returns a functional-level decision after completing the GCS commands matches the official label, even if a human reviewer would reject the artifact. Since §4.4.3 derives the Functional badge from successful execution of all graph nodes, the agent's positive decision is essentially the same event as condition (1). The reported 85.42% BCR thus reduces by construction to a GCS command-completion rate, not to agreement with human badge judgments. The headline claim that ArtifactCopilot

  2. other [§5.1 Experimental Setup vs. §6.3 Threats to Validity; §3 Preliminary Study]
    "We evaluate all 48 artifacts described in § 2, including 12 artifacts used during the exploratory study and 36 held out for validation. The exploration set informed system design decisions but involved no parameter tuning; results are therefore reported jointly to improve statistical robustness. ... the retry threshold of the error recovery mechanism (k=3) and the blocking detection interval of three minutes were selected based on pilot experiments."

    The exploration set was used to inform system design and, per §6.3, pilot experiments were used to select system parameters (k=3 and a three-minute blocking interval). Those same 12 exploration artifacts are then pooled into the headline 85.42% BCR. The paper honestly discloses this and separately reports 86.11% on the 36 held-out artifacts, so this is a mild design-set/pilot leakage rather than a definitional reduction. However, the headline number is not a purely out-of-sample prediction: part of the reported performance measures behavior on data that shaped the system and its pilot-selected parameters.

full rationale

The central circularity is in the BCR metric. Because §2 selects only Functional/Reusable artifacts and treats Reusable as functionally positive, there are no negative functional labels in the dataset. Condition (2) of the BCR definition is therefore satisfied for any agent that returns a functional decision after executing the GCS commands; combined with §4.4.3, which makes the Functional badge equivalent to successful execution of all graph nodes, the 85.42% headline reduces by construction to a command-completion rate. The claimed 'agreement with human artifact evaluation outcomes' is not actually measured. A second, milder circularity is the inclusion of the 12-artifact exploration set in the pooled headline; the paper discloses this and reports validation-only numbers separately, but the abstract and summary emphasize the pooled 85.42% figure. I also flag that the abstract's ArtifactGuide/70.56%/60-artifact result does not appear in the body; the body evaluates ArtifactCopilot/85.42% on 48 artifacts. That is a missing-support/reporting inconsistency rather than a definitional circularity. There are no load-bearing self-citations, imported uniqueness theorems, or ansatz-by-citation chains, and the held-out 36-artifact split does provide some genuine out-of-sample signal for the non-degenerate part of the pipeline. Overall, the paper's central empirical claim is partially circular by construction, warranting a score of 6.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The central result rests on the assumption that official badges and the author-built GCS are a valid ground truth, that the exploration-set-informed design can be fairly included in the headline number, and that the exclusions in dataset construction define the intended scope. No code, data, or prompts are shipped, so these assumptions cannot be checked.

free parameters (3)
  • retry threshold k = 3
    Number of attempts before marking a command failed; chosen based on pilot experiments (§4.4.2, §6.3) and directly affects recovery success rates.
  • blocking detection interval = 3 minutes
    Sustained low CPU utilization interval interpreted as stall (§4.4.2, §6.3); selected from pilot experiments and affects the system's ability to detect implicit blocking.
  • context limit / compression threshold = 100k tokens
    When accumulated context approaches the model limit, historical traces are summarized (§4.5); this hand-chosen threshold influences long-horizon execution stability.
axioms (4)
  • domain assumption Official ACM badge outcomes are a valid ground truth for functional reproducibility.
    Used as the reference label in BCR (§2, §5.1); the paper itself documents inconsistency across venues and years (§2), so this assumption is load-bearing.
  • domain assumption The manually constructed Golden Command Set correctly and completely captures the executable behavior required by each README.
    BCR condition (1) depends on this; annotator agreement is described (§2) but the GCS is not independently verified or shipped.
  • domain assumption LLM execution with temperature=0 and structured outputs is sufficiently deterministic for single-run evaluation.
    The paper acknowledges stochasticity in §6.3 but reports no multiple runs or variance for the headline BCR.
  • ad hoc to paper Artifacts excluded for commercial, private-data, hardware, or large-disk reasons are outside the intended claim.
    Exclusion criteria (§2) remove 52 of 100 candidates; the abstract's 60-artifact dataset suggests a different scope, so this assumption is not stable.
invented entities (1)
  • AE Graph no independent evidence
    purpose: Structured dependency-aware representation of evaluation tasks derived from README; used to model execution states, command dependencies, and artifact flows.
    Introduced in §4.2; its value is evidenced only by the paper's own ablation study, with no external benchmark or independent reproduction.

pith-pipeline@v1.3.0-alltime-deepseek · 17818 in / 12219 out tokens · 107251 ms · 2026-08-03T05:23:58.676801+00:00 · methodology

0 comments
read the original abstract

Artifact Evaluation (AE) has become a standard mechanism for linking software engineering papers to their supporting artifacts, but growing submission volume makes purely manual AE increasingly difficult to sustain. Although recent advances in LLM-based agents make AE automation increasingly plausible, current descriptive badge policies define badge semantics but provide no executable verification criteria, leaving both human reviewers and agents without a detailed basis for judgment. To address this gap, we construct ArtifactGuide, a structured scoring rubric grounded in ACM policy, expert-informed calibration, and artifact-based validation, and we design ArtifactCopilot, an agent collecting review evidence following a fixed sequence under ArtifactGuide and deriving the final badge decision from accumulated evidence. We evaluate our framework on 60 real artifacts from recent software engineering conferences using human-adjudicated badges as reference. The results show that ArtifactGuide improves the AE performance of coding agents over official ACM badge-policy prompts, increasing three-run mean exact badge agreement by 10.55 to 28.34 percentage points. Across all evaluated systems and prompting protocols, ArtifactCopilot achieves the highest badge-level agreement at 70.56% and is the only system that completed all repeated runs successfully while producing a review report in every run. A controlled user study with 8 experienced researchers suggests that ArtifactCopilot reports improve reviewer confidence, help reviewers locate evidence, and understand evaluation scope more clearly. Further analysis translates insights from automated AE into practical guidance for designing higher-quality artifacts with clearer review routes, more explicit claim-to-output links, and more concrete reuse paths.

Figures

Figures reproduced from arXiv: 2602.02235 by Haoyu Wang, Yanjie Zhao, Zhaonan Wu, Zheng Wang, Zhenpeng Chen.

Figure 1
Figure 1. Figure 1: Three commonly used artifact evaluation badges. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of dataset construction process. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Representative examples of automation failures and manual intervention during artifact evaluation. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: ArtifactCopilot workflow: from paper PDF to badge determination through structured modeling and hierarchical execution. environments and resolves environment-level decisions in advance, and an Evaluation Agent carries out commands, detects failures, and performs automated recovery during artifact evaluation. 4.1 Overview and Architecture ArtifactCopilot accepts two input modes: papers in PDF format or pre-… view at source ↗
Figure 5
Figure 5. Figure 5: Example transformation from README documentation to an AE Graph. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Intermediate artifacts produced by the Planning Agent. The Planning Agent enforces a host-centric unified scheduling model in which all commands are issued from the host and explicitly mapped to target containers through a fixed execution interface. The system leverages container execution APIs for host-driven command injection. Containers with customized entrypoints, however, exhibit execution behaviors t… view at source ↗
Figure 7
Figure 7. Figure 7: Prompt structure for the Planning Agent and Evaluation Agent, with static components placed before dynamic execution history to support cache reuse. • RQ3: How effectively does ArtifactCopilot automate artifact evaluation compared to repre￾sentative baselines? • RQ4: Which system components contribute most to ArtifactCopilot’s performance? 5.1 Experimental Setup Datasets and Baselines. We evaluate all 48 a… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. On the Security of Research Artifacts

    cs.CR 2026-05 conditional novelty 7.0

    41.6% of insecure code findings in research artifacts pose practical security risks; the SAFE framework distinguishes them with 84.8% accuracy and 84.6% F1-score.

  2. An Agentic Approach Towards Replication Package Quality Evaluation

    cs.SE 2026-06 unverdicted novelty 6.0

    A multi-agent system automates checks on replication package quality using 31 machine-verifiable criteria derived from 34 sources, showing 91.4% inter-run consistency and 75.4% agreement with manual review on five packages.

  3. An Agentic Approach Towards Replication Package Quality Evaluation

    cs.SE 2026-06 conditional novelty 5.0

    A multi-agent LLM pipeline can partially automate replication-package quality checks, with 91.4% inter-run consistency and 75.4% agreement with a manual baseline on five packages.

Reference graph

Works this paper leans on

44 extracted references · 1 canonical work pages · cited by 2 Pith papers

  1. [1]

    Association for Computing Machinery, New York, NY, USA

    2023.Proceedings of ESEC/FSE 2023. Association for Computing Machinery, New York, NY, USA. https://dl.acm.org/ doi/proceedings/10.1145/3611643

    doi:10.1145/3611643 2023

  2. [2]

    Association for Computing Machinery, New York, NY, USA

    2024.Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. Association for Computing Machinery, New York, NY, USA. doi:10.1145/3650212

    doi:10.1145/3650212 2024

  3. [3]

    Association for Computing Machinery, New York, NY, USA

    2024.Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. Association for Computing Machinery, New York, NY, USA. doi:10.1145/3691620

    doi:10.1145/3691620 2024

  4. [4]

    Association for Computing Machinery, New York, NY, USA

    2024.Proceedings of the 46th International Conference on Software Engineering. Association for Computing Machinery, New York, NY, USA. doi:10.1145/3597503

    doi:10.1145/3597503 2024

  5. [5]

    ICSE 2025 - Artifact Evaluation

    2025. ICSE 2025 - Artifact Evaluation. https://conf.researchr.org/track/icse-2025/icse-2025-artifact-evaluation Accessed: 2026-01-28

    2025

  6. [6]

    Association for Computing Machinery, New York, NY, USA

    2025.Proceedings of the 47th International Conference on Software Engineering. Association for Computing Machinery, New York, NY, USA. doi:10.5555/3767285

    work page doi:10.5555/3767285 2025

  7. [7]

    Florian Angermeir, Maximilian Amougou, Mark Kreitz, Andreas Bauer, Matthias Linhuber, Davide Fucci, Daniel Mendez, Tony Gorschek, et al. 2025. Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies.arXiv preprint arXiv:2510.25506(2025)

    arXiv 2025

  8. [8]

    Anthropic. 2025. Claude Code: AI coding agent for terminal & IDE. https://claude.com/product/claude-code. Accessed: 2025-01-23

    2025

  9. [9]

    Association for Computing Machinery. 2020. Artifact Review and Badging. https://www.acm.org/publications/policies/ artifact-review-and-badging-current. Accessed: 2026-01-23

    2020

  10. [10]

    Association for Computing Machinery. 2026. ACM Digital Library. https://dl.acm.org/. Accessed: 2026-01-27

    2026

  11. [11]

    Dirk Beyer and Stefan Winter. 2025. Artifact Evaluations for Stronger Research Results. InProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering. 1234–1237

    2025

  12. [12]

    Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2024. Repairagent: An autonomous, llm-based agent for program repair.arXiv preprint arXiv:2403.17134(2024)

    Pith/arXiv arXiv 2024

  13. [13]

    Islem Bouzenia and Michael Pradel. 2025. You name it, I run it: An LLM agent to execute tests of arbitrary projects. Proceedings of the ACM on Software Engineering2, ISSTA (2025), 1054–1076

    2025

  14. [14]

    Christian Braun, Alexander Lilienbeck, and Daniel Mentjukov. 2025. The Hidden Structure – Improving Legal Document Understanding Through Explicit Text Formatting. arXiv:2505.12837 [cs.CL] https://arxiv.org/abs/2505.12837

    Pith/arXiv arXiv 2025

  15. [15]

    CERN and OpenAIRE. 2026. Zenodo. https://zenodo.org/. Accessed: 2026-01-29

    2026

  16. [16]

    Bruce R Childers and Panos K Chrysanthis. 2017. Artifact evaluation: Is it a real incentive?. In2017 IEEE 13th international conference on e-science (e-Science). IEEE, 488–489

    2017

  17. [17]

    GitHub, Inc. 2026. GitHub. https://github.com/. Accessed: 2026-01-29

    2026

  18. [18]

    Odd Erik Gundersen and Sigbjørn Kjensmo. 2018. State of the art: Reproducibility in artificial intelligence. InProceedings of the AAAI conference on artificial intelligence, Vol. 32

    2018

  19. [19]

    Sean N Halpin. 2024. Inter-coder agreement in qualitative coding: Considerations for its use.American Journal of Qualitative Research8, 3 (2024), 23–43

    2024

  20. [20]

    Junda He, Christoph Treude, and David Lo. 2025. LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision, and the Road Ahead.ACM Transactions on Software Engineering and Methodology34, 5 (2025), 1–30. , Vol. 1, No. 1, Article . Publication date: February 2018. 20 Zhaonan Wu, Yanjie Zhao, Zhenpeng Chen, Zheng Wang, and Haoyu Wang

    2025

  21. [21]

    Ben Hermann, Stefan Winter, and Janet Siegmund. 2020. Community expectations for research artifacts and evaluation processes. InProceedings of the 28th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering. 469–480

    2020

  22. [22]

    Ruida Hu, Chao Peng, Xinchen Wang, Junjielong Xu, and Cuiyun Gao. 2025. Repo2Run: Automated Building Executable Environment for Code Repository at Scale.arXiv preprint arXiv:2502.13681(2025)

    arXiv 2025

  23. [23]

    ICSE 2024. 2024. ICSE 2024 - Artifact Evaluation. https://conf.researchr.org/track/icse-2024/icse-2024-artifact- evaluation. Accessed: 2024-04-21

    2024

  24. [24]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770(2023)

    Pith/arXiv arXiv 2023

  25. [25]

    Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al . 2025. Deepseek-v3. 2: Pushing the frontier of open large language models.arXiv preprint arXiv:2512.02556(2025)

    Pith/arXiv arXiv 2025

  26. [26]

    Mugeng Liu, Xiaolong Huang, Wei He, Yibing Xie, Jie M Zhang, Xiang Jing, Zhenpeng Chen, and Yun Ma. 2024. Research artifacts in software engineering publications: Status and trends.Journal of Systems and Software213 (2024), 112032

    2024

  27. [27]

    Al Muttakin, Saikat Mondal, and Chanchal Roy. 2026. The State of Open Science in Software Engineering Research: A Case Study of ICSE Artifacts.arXiv preprint arXiv:2601.02066(2026)

    arXiv 2026

  28. [28]

    Animesh Nandi, Atri Mandal, Shubham Atreja, Gargi B Dasgupta, and Subhrajit Bhattacharya. 2016. Anomaly detection using program control flow graph mining from execution logs. InProceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 215–224

    2016

  29. [29]

    Per Runeson, Emma Soderberg, and Martin Host. 2024. A conceptual framework and recommendations for open data and artifacts in empirical software engineering. InProceedings of the 1st IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering. 68–75

    2024

  30. [30]

    Damien Saucez, Luigi Iannone, and Olivier Bonaventure. 2019. Evaluating the artifacts of SIGCOMM papers.ACM SIGCOMM Computer Communication Review49, 2 (2019), 44–47

    2019

  31. [31]

    Harald Semmelrock, Tony Ross-Hellauer, Simone Kopeinik, Dieter Theiler, Armin Haberl, Stefan Thalmann, and Dominik Kowald. 2025. Reproducibility in machine-learning-based research: Overview, barriers, and drivers.AI Magazine46, 2 (2025), e70002

    2025

  32. [32]

    Mohammed Latif Siddiq, Arvin Islam-Gomes, Natalie Sekerak, and Joanna Santos. 2025. Large Language Models for Software Engineering: A Reproducibility Crisis.arXiv preprint arXiv:2512.00651(2025)

    arXiv 2025

  33. [33]

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. 2024. Openhands: An open platform for ai software developers as generalist agents.arXiv preprint arXiv:2407.16741(2024)

    Pith/arXiv arXiv 2024

  34. [34]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al . 2022. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems35 (2022), 24824–24837

    2022

  35. [35]

    Stefan Winter, Christopher S Timperley, Ben Hermann, Jürgen Cito, Jonathan Bell, Michael Hilton, and Dirk Beyer

  36. [36]

    Zhiqiang Xie, Minjie Wang, Zihao Ye, Zheng Zhang, and Rui Fan. 2022. Graphiler: Optimizing graph neural networks with message passing data flow graph.Proceedings of Machine Learning and Systems4 (2022), 515–528

    2022

  37. [37]

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering.Advances in Neural Information Processing Systems37 (2024), 50528–50652

    2024

  38. [38]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. React: Syner- gizing reasoning and acting in language models. InThe eleventh international conference on learning representations

    2022

  39. [39]

    Zhengmin Yu, Yuan Zhang, Ming Wen, Yinan Nie, Wenhui Zhang, and Min Yang. 2025. CXXCrafter: An LLM-Based Agent for Automated C/C++ Open Source Software Building.Proceedings of the ACM on Software Engineering2, FSE (2025), 2618–2640

    2025

  40. [40]

    Zhiqiang Yuan, Weitong Chen, Hanlin Wang, Kai Yu, Xin Peng, and Yiling Lou. 2024. Transagent: An llm-based multi-agent system for code translation.arXiv preprint arXiv:2409.19894(2024)

    Pith/arXiv arXiv 2024

  41. [41]

    Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. 2024. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges.arXiv preprint arXiv:2401.07339(2024)

    Pith/arXiv arXiv 2024

  42. [42]

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Autocoderover: Autonomous program improvement. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1592–1604

    2024

  43. [43]

    Noa Zilberman and Andrew W Moore. 2020. Thoughts about artifact badging. 60–63 pages. , Vol. 1, No. 1, Article . Publication date: February 2018

    2020

  44. [2022]

    InProceedings of the 30th ACM joint European software engineering conference and symposium on the foundations of software engineering

    A retrospective study of one decade of artifact evaluations. InProceedings of the 30th ACM joint European software engineering conference and symposium on the foundations of software engineering. 145–156