Pith. sign in

REVIEW 4 major objections 5 minor 53 references

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a terminal task's training value is set by where it falls relative to the solvers that will attempt it, not merely by whether it is solvable.

desk verdict Solid empirical recipe for selecting terminal tasks via solver disagreement or strong-pass/weak-fail; the revision-loop mechanism is overclaimed and needs a no-revision control. read the letter →

arxiv 2608.06352 v1 pith:HHYYNNP4 submitted 2026-08-06 cs.LG cs.CL

Fanzhe Meng , Guoxin Chen , Jiale Zhao , Shuang Sun , Zhiyu Lin , Wayne Xin Zhao , Ruihua Song , Ji-Rong Wen
show 1 more author
Kai Jia
This is my paper
classification cs.LGcs.CL
keywords terminalagentstasksynthesissolver-relativelearnabilitylearnablezonesynthetictrainingdataadversarialsolvercalibrationsupervisedfine-tuningverifiabletasks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the usefulness of a terminal task for training an agent depends on where the task sits relative to the solver that will learn from it, not merely on whether the task is executable or solvable. It introduces CalibForge, a task-synthesis system that treats solver pass/fail behavior as construction-time feedback: a candidate task is authored, probed, revised, and re-probed until it lands in a learnable zone—demonstrably solvable by a stronger solver while not uniformly solved by a weaker one. This matters because synthetic agent training data is typically filtered only for executability, which can yield tasks that are either too easy to teach anything or too broken to be solved. The paper reports that models trained on 5,431 such calibrated tasks improve by up to 24.71 points on Terminal-Bench 2.0 and also transfer gains to SWE-bench Pro and Doc2Repo, suggesting that solver-relative calibration is a practical construction target.

What carries the argument

The central object is the retention criterion Cγ, a predicate over verified solver outcomes. Multi-solver calibration requires $0 < \sum_{i=1}^K y_i < K$, meaning at least one solver passes and at least one fails; contrastive calibration requires $y_s = 1 \land y_w = 0$, meaning a designated stronger solver passes while a designated weaker solver fails. This criterion is used inside an author–solver loop: when outcomes do not satisfy Cγ, the authoring agent revises the instruction, environment, or verifier, revalidates, and re-probes, up to $R_{max} = 50$ rounds, so the learnable zone is a revision target rather than a one-pass filter.

What would settle it

Take a set of retained contrastive tasks and probe each one five times with the same stronger/weaker solver pair; if a substantial fraction of tasks switch from strong-pass/weak-fail to other outcome patterns, the rejection criterion is measuring single-attempt noise rather than stable task difficulty.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that task learnability is solver-relative and observable through verified solver outcomes, and that this observable status can drive task revision rather than just task filtering. A candidate that passes structural validation and self-solving may still sit outside the desired capability interval: only 19% of candidates initially satisfy the contrastive strong-pass/weak-fail relation, while feedback-driven revision and re-probing raise cumulative acceptance to 96%. Matched ablations on Qwen3-30B-A3B-Instruct show that retention criteria based on cross-solver disagreement and strong/weak contrast outperform authoring-plus-validation alone (22.47% on Terminal-Bench 2.0) and single-solver feedback (24.34%), reaching 29.21% and 31.09% for matched 1,300-task sets, while the full 5,431-task collection reaches 32.58% and 47.57% on two backbones.

Load-bearing premise

The load-bearing premise is that a single verified pass or fail from a 100-step, 30-minute solver attempt reliably tells whether a task is too easy, too hard, or learnable for the chosen solvers.

Editorial extensions

If this is right

  • Retaining tasks by cross-solver disagreement or a strong-pass/weak-fail relation yields more effective supervision than authoring-plus-validation or single-solver feedback, measured on Terminal-Bench 2.0.
  • Revision, not merely filtering, drives the effect: although only 19% of validated candidates initially satisfy the contrastive criterion, feedback-driven revision and re-probing raise cumulative acceptance to 96%.
  • Training on the full 5,431-task collection improves two Qwen backbones by up to 24.71 points on Terminal-Bench 2.0, with gains appearing across every reported category.
  • The learned supervision transfers out of distribution: the 30B model improves by 27.68 points on SWE-bench Pro and 30.04 points on Doc2Repo, while the 35B model improves by 3.03 and 3.85 points respectively.
  • The calibration budget shapes the final training set: most candidates are retained within five probes, but a long tail of recoverable tasks requires sustained probing, so the maximum number of rounds controls which tasks enter the collection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I infer that the stability of the learnable zone could be audited by probing each retained task multiple times with the same solver pair; substantial pass/fail flipping across repeats would indicate that the retention criterion is partly measuring sampling luck.
  • I infer that the learnable zone is likely solver-family-relative: swapping the stronger and weaker solvers for a different model family could shift which tasks are retained, meaning the zone is a property of the solver pair and not an absolute difficulty ranking.
  • I infer that the same adversarial author–solver loop could extend to other verifiable agent domains, such as code repair, web interaction, or robotics, wherever a verifier can certify a final state and where task value depends on being neither uniformly solved nor uniformly failed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CalibForge, a terminal-task synthesis system that authors candidate command-line tasks, validates them via structural checks and self-solving, and then revises them over up to 50 rounds through 'adversarial solver calibration.' Two retention criteria are introduced: multi-solver disagreement (Eq. 1) and contrastive strong-pass/weak-fail (Eq. 2). The authors distill SFT trajectories from 5,431 retained tasks using DeepSeek-V4-Pro and fine-tune Qwen3-30B-A3B-Instruct and Qwen3.5-35B-A3B. They report large gains over base models and over baselines on Terminal-Bench 2.0, SWE-bench Pro, and Doc2Repo, and attribute the gains to the calibration strategies via matched-task ablations.

Significance. The central idea---using verified solver behavior to place synthetic tasks in a 'learnable zone' between solvability and uniform solvability---is timely and, if operationalized reliably, would be a useful contribution to agent training-data construction. The paper has strong experimental hygiene in several respects: Table 3 matches task counts across variants, all baselines are re-distilled and retrained under a shared teacher/SFT protocol, a decontamination procedure is described, and the primary benchmark (Terminal-Bench 2.0) is external. The main weakness is that the calibration loop's behavior has not been separated from chance re-probing, which leaves the central mechanism under-evidenced; the transfer claims also rest partly on a single-run evaluation and a self-associated benchmark. These issues are addressable with additional controls and reporting.

major comments (4)
  1. [Section 3.4, Figures 8 and 9] The attribution of the 96% final acceptance to feedback-driven revision is not supported, because no no-revision control is reported. With the observed first-probe target rate p=0.19 from Figure 8, 50 independent re-probes of an unchanged candidate yield an eventual acceptance probability of 1-(1-0.19)^50 ≈ 0.99996, so the observed 96% is statistically compatible with a process in which revision does not change task difficulty. Since Eq. (2) is evaluated from a single pass/fail draw per solver per round, the contrastive calibration loop may simply be selecting candidates whose noise realizations happen to satisfy strong-pass/weak-fail. Please add a no-revision control (re-probing candidates for up to Rmax rounds without author revision) and report the acceptance curve, or provide an explicit stochastic model with outcome correlations that can separate chance retention from revision effects.
  2. [Section 3.4, Table 3] The retention criteria for the No Solver and Single Solver ablation variants are not specified. It is not stated whether Single Solver retains a candidate on a solver pass, a strong-pass/weak-fail relation, or some other rule, and the same is true for No Solver. Without this information, the comparison in Table 3 cannot be interpreted as isolating the calibration strategy: Single Solver may already include a teacher-solvability filter (DeepSeek-V4-Pro is both the authoring agent and the trajectory teacher), and the difference between Single Solver and Contrast Solver may reflect the weak-fail branch, the revision prompt, or the retention rule rather than the solver-relative learnable zone. Please state these criteria explicitly and, if needed, add a variant that controls for teacher-solvability filtering.
  3. [Section 3.1, Table 1] SWE-bench Pro is evaluated once, as stated in Section 3.1. The headline transfer claim includes a 27.68-point improvement on SWE-bench Pro for Qwen3-30B-A3B-Instruct, but a single evaluation run provides no variance estimate or significance test, and this is especially important for a 731-task benchmark where sampling noise is non-negligible. Please report multiple runs (or per-instance bootstrap intervals) for SWE-bench Pro, or relegate it to a supplementary result with an explicit caveat.
  4. [Section 3.1, Table 1; References] Doc2Repo is used as an out-of-distribution evaluation benchmark, but its primary citations (Chen et al., 2026b, and related Zhao et al., 2026b) have several authors in common with the present paper. The headline transfer gain of 30.04 points on Doc2Repo is therefore not fully independent. Please disclose the author overlap in the main text, describe any steps taken to prevent benchmark-specific fitting, and consider reporting the headline transfer results without Doc2Repo or with Doc2Repo clearly separated from fully external benchmarks.
minor comments (5)
  1. [Appendix B.2] In the table of Case C2, 'Difficulti of self-assessment' should be 'Difficulty of self-assessment.'
  2. [Figure 3] Figure 3 is labeled 'TB2 tasks solved (pass@3)' while Table 1 reports task accuracy; clarify the relationship between these metrics, for example whether pass@3 is the fraction of tasks solved in at least one of the three evaluation runs.
  3. [Algorithm 1] Algorithm 1's internal formatting, such as 'WIDESEARCHANDSPECIFY' and 'PROBEANDVERIFY', is difficult to read; please use standard pseudocode spacing and capitalization.
  4. [Section 3.1] The description of CalibForge-Eval says it follows the DeepSeek-V4 evaluation setting, but the exact scaffold version and the teacher's reasoning-effort configuration are not specified; a brief citation or appendix pointer would improve reproducibility.
  5. [Appendix B] The case studies in Appendix B are illustrative rather than systematic evidence for revision effects; please state this explicitly so they are not read as quantitative support for the calibration mechanism.

Circularity Check

1 steps flagged · score 2.0 of 10

Central training results are validated on external benchmarks; only minor self-citation via Doc2Repo, not load-bearing.

  1. other [Section 3.1 (Benchmarks and Evaluation); Table 1; Introduction]
    "To evaluate out-of-distribution transfer, we additionally use the 731-task public set of SWE-bench Pro ... and Doc2Repo (Chen et al., 2026b; Zhao et al., 2026b) for full-repository generation from natural-language specifications."

    Doc2Repo is cited to Chen et al. (2026b) and Zhao et al. (2026b), whose author lists overlap with the present paper (G. Chen, F. Meng, J. Zhao, W.X. Zhao, R. Song, J.-R. Wen, K. Jia). The headline transfer gains ('30.04 points on Doc2Repo') therefore include a benchmark produced by the same research group. This is a minor self-citation signal, but it is not load-bearing: the central training-data-quality claim is also supported by the external Terminal-Bench 2.0 and SWE-bench Pro evaluations, and no fitted constant or derivation step reduces to Doc2Repo.

full rationale

The CalibForge derivation chain is otherwise self-contained. The learnable-zone criteria (Eqs. 1-2) are operational definitions, not predictions; the downstream claims are tested on Terminal-Bench 2.0 (external) and SWE-bench Pro (external) under a shared distillation and SFT protocol with baselines. The contrastive-calibration funnel (19% to 96% acceptance) is a process statistic of a loop that by construction retains only tasks satisfying C_con, so it should not be read as independent evidence that revision improves tasks; that is a missing-control robustness concern, not a circular reduction. No fitted parameter is renamed as a prediction, and no uniqueness theorem or load-bearing self-citation is invoked. The only self-association is Doc2Repo, which adds minor but non-determinative support.

Assumptions & free parameters 6 free parameters · 5 assumptions · 1 invented entities

The ledger shows that the paper's contribution is empirical: no mathematical derivation is offered, and the method's effectiveness rests on several domain assumptions about solver behavior, distillation transfer, and decontamination. The main free parameters are the calibration loop limits, solver budgets, and SFT recipe; none of these are fit to the evaluation benchmarks, so circularity burden stays low. The invented entity 'learnable zone' is an operational construct with downstream falsifiability.

free parameters (6)
  • Maximum calibration rounds Rmax = 50
    Hand-chosen limit on the author-solver revision loop in Algorithm 1 and Section 3.1. If raised, the acceptance funnel in Section 3.4 (96% cumulative acceptance) and the task distribution would change.
  • Solver attempt budget per probe = 100 interaction steps and 30 minutes
    Each solver attempt in calibration is truncated at 100 steps and 30 minutes (Section 3.1). This directly determines which outcomes count as pass or fail and thus shapes the retained task set.
  • Number of solver models K in multi-solver calibration = 3 (DeepSeek-V4-Flash, GLM-5, Kimi K2.5)
    The retention criterion Cmulti requires 0 < sum y_i < K (Eq. 1), and K is a chosen design parameter, not derived from any principle.
  • Decontamination thresholds = 14-gram exact match; 5-shingle Jaccard 0.30 for instructions and 0.45 for verifier code
    Hand-set filtering thresholds in Appendix D. They determine which candidate tasks are excluded before training, which influences the composition of the training set and the benchmark scores.
  • SFT training recipe = 10 epochs, learning rate 1e-5, global batch size 128, context length 131072
    The final checkpoints after 10 epochs are reported (Appendix E). The reported gains could depend on this recipe, which is a shared but not optimized parameter.
  • Trajectory distillation filter = 200-step limit, one-hour timeout; drop trajectories for length, invalid tool calls, tokenizer-unsafe tokens
    This shared protocol in Section 3.1 controls for teacher budget across task sets but determines the SFT trajectory counts (e.g., multi-solver yields 2,425 trajectories in Table 3).
assumptions (5)
  • domain assumption Verified pass/fail outcomes from a single solver attempt are reliable enough to define task difficulty and retention.
    Used in Section 2.3 (Eq. 1 and 2); no repeated probing or variance estimates are reported, so a lucky or unlucky attempt can determine retention. This is the load-bearing premise for the 'learnable zone'.
  • ad hoc to paper A strong-pass/weak-fail relation is a useful curriculum target: tasks that a stronger solver solves and a weaker solver fails carry effective training signal for downstream students.
    The core design principle in Section 2.3 and Figure 1. Its validity is the paper's central empirical claim, but it is assumed as a target for task revision rather than proven from prior theory.
  • domain assumption Trajectory distillation from DeepSeek-V4-Pro transfers as useful supervision to Qwen student models.
    All task sets, including baselines, are distilled with DeepSeek-V4-Pro under CalibForge-Eval (Section 3.1). The paper relies on this setup to compare data sources, and transfer to Qwen is observed but assumed to be protocol-fair.
  • domain assumption The decontamination procedure removes all meaningful overlap between training tasks and the three evaluation benchmarks.
    Appendix D defines similarity thresholds and structural flags. If overlap remains, reported benchmark gains could be inflated.
  • domain assumption Terminal-Bench 2.0, SWE-bench Pro, and Doc2Repo measure capabilities that improved training should transfer to.
    Used as primary and OOD evaluations (Section 3.1). The analysis assumes these benchmarks are valid indicators of terminal-agent and software-engineering competence.
invented entities (1)
  • Solver-relative learnable zone independent evidence
    purpose: Conceptual target for task retention: a task is kept only if it has demonstrated solvability but is not uniformly solved, defined via Cmulti or Ccon.
    It has a falsifiable handle: tasks inside the zone are those satisfying Eq. 1 or Eq. 2, and the paper tests whether training on such tasks improves held-out benchmarks. This is a construct, not a physical entity, but it is the paper's central new operational notion.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks." pith.science (2026). https://pith.science/paper/HHYYNNP4

@misc{pith2026260806352,
  author       = {Pith},
  title        = {Pith review of: CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HHYYNNP4}},
  note         = {Machine review of arXiv:2608.06352}
}
read the original abstract

Training terminal agents requires executable and verifiable tasks that are not merely solvable, but appropriately challenging for learning. Executable validation establishes feasibility, yet does not reveal how a task behaves relative to a given solver setting. In this paper, we present CalibForge, an autonomous terminal-task synthesis system that uses verified solver behavior to revise candidate tasks through adversarial solver calibration. Multi-solver calibration targets disagreement within a heterogeneous solver pool, whereas contrastive solver calibration targets a designated strong-pass/weak-fail relation; both operationalize a solver-relative learnable zone anchored in demonstrated solvability. Using CalibForge, we construct 5,431 calibrated terminal tasks. Our ablations show that both strategies yield more effective supervision than authoring and validation alone or ordinary single-solver feedback. Models trained on the full collection achieve 32.58% and 47.57% on Terminal-Bench 2.0. The largest improvements over the corresponding base model reach 24.71 percentage points on Terminal-Bench 2.0, 27.68 points on SWE-bench Pro, and 30.04 points on Doc2Repo. Together, these results support solver-relative learnability as a practical target for constructing effective and transferable agent training data.

Figures

Figures reproduced from arXiv: 2608.06352 by the authors.

Figure 1
Figure 1. Overview of CalibForge. Top: Multi-solver and contrastive calibration target a solver￾relative learnable zone through cross-solver disagreement and a strong-pass/weak-fail relation, respectively; both outperform authoring and validation alone and single-solver feedback on Qwen3- 30B-A3B-Instruct. Bottom: Training on trajectories distilled from 5,431 calibrated tasks improves two Qwen backbones on Terminal-Bench 2.0,… view at source ↗
Figure 2
Figure 2. Overview of CalibForge. Starting from a clue, CalibForge authors a candidate task, subjects it to structural validation and self-solving, and then probes it using one of two adversarial solver-calibration strategies: multi-solver or contrastive calibration. The solvers’ pass/fail outcomes provide the retention signal, while feedback summaries and full interaction trajectories guide revisions at any stage: the author… view at source ↗
Figure 3
Figure 3. Per-category Terminal-Bench 2.0 tasks solved (pass@3) for (a) CalibForge-30B-A3B and (b) [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Domain composition of CalibForge-generated tasks and prior terminal-task datasets under [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Rank–frequency distribution of capability tags in the CalibForge task collection. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Per-task distributions of environment and verification statistics. Solid and dashed vertical [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Trajectory characteristics across task sets re-distilled under the identical teacher protocol: [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: First-probe solver outcomes and final retention under contrastive calibration. The left [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Progressive contrastive-calibration funnel. Width denotes the remaining candidates; [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 20 canonical work pages

  1. [1]

    arXiv preprint arXiv:2602.07274 , year=

    Termigen: High-fidelity environment and robust trajectory synthesis for terminal agents , author=. arXiv preprint arXiv:2602.07274 , year=

    arXiv

  2. [2]

    arXiv preprint arXiv:2602.21193 , year=

    On data engineering for scaling llm terminal capabilities , author=. arXiv preprint arXiv:2602.21193 , year=

    arXiv

  3. [3]

    arXiv preprint arXiv:2601.11868 , year=

    Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces , author=. arXiv preprint arXiv:2601.11868 , year=

    arXiv

  4. [4]

    arXiv preprint arXiv:2601.16443 , year=

    Endless Terminals: Scaling RL Environments for Terminal Agents , author=. arXiv preprint arXiv:2601.16443 , year=

    arXiv

  5. [5]

    arXiv preprint arXiv:2602.01244 , year=

    Large-Scale Terminal Agentic Trajectory Generation from Dockerized Environments , author=. arXiv preprint arXiv:2602.01244 , year=

    arXiv

  6. [6]

    arXiv preprint arXiv:2603.05344 , year=

    Building effective ai coding agents for the terminal: Scaffolding, harness, context engineering, and lessons learned , author=. arXiv preprint arXiv:2603.05344 , year=

    arXiv

  7. [7]

    A Self-Evolving Framework for Efficient Terminal Agents via Observational Context Compression

    A Self-Evolving Framework for Efficient Terminal Agents via Observational Context Compression , author=. arXiv preprint arXiv:2604.19572 , year=

    work page Pith review arXiv

  8. [8]

    arXiv preprint arXiv:2602.10999 , year=

    Cli-gym: Scalable cli task generation via agentic environment inversion , author=. arXiv preprint arXiv:2602.10999 , year=

    arXiv

Show all 53 references
  1. [9]

    arXiv preprint arXiv:2605.22535 , year=

    TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks , author=. arXiv preprint arXiv:2605.22535 , year=

    arXiv

  2. [10]

    arXiv preprint arXiv:2605.20876 , year=

    Terminal-World: Scaling Terminal-Agent Environments via Agent Skills , author=. arXiv preprint arXiv:2605.20876 , year=

    arXiv

  3. [11]

    arXiv preprint arXiv:2604.25727 , year=

    Toward Scalable Terminal Task Synthesis via Skill Graphs , author=. arXiv preprint arXiv:2604.25727 , year=

    arXiv

  4. [12]

    arXiv preprint arXiv:2606.23321 , year=

    Tmax: A simple recipe for terminal agents , author=. arXiv preprint arXiv:2606.23321 , year=

    arXiv

  5. [13]

    arXiv preprint arXiv:2606.22883 , year=

    CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents , author=. arXiv preprint arXiv:2606.22883 , year=

    arXiv

  6. [14]

    arXiv preprint arXiv:2603.03194 , year=

    BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing? , author=. arXiv preprint arXiv:2603.03194 , year=

    arXiv

  7. [15]

    arXiv preprint arXiv:2509.16941 , year=

    Swe-bench pro: Can ai agents solve long-horizon software engineering tasks? , author=. arXiv preprint arXiv:2509.16941 , year=

    arXiv

  8. [16]

    2026 , month = jan, howpublished =

    Qijia Shen and Jay Rainton and Aznaur Aliev and Ahmed Awelkair and Boyuan Ma and Zhiqi Huang and Yuzhen Mao and Wendong Fan and Philip Torr and Bernard Ghanem and Changran Hu and Urmish Thakker and Guohao Li , title =. 2026 , month = jan, howpublished =

    2026

  9. [17]

    arXiv preprint arXiv:2606.19348 , year=

    Deepseek-v4: Towards highly efficient million-token context intelligence , author=. arXiv preprint arXiv:2606.19348 , year=

  10. [18]

    doi:10.5281/zenodo.20953922 , url =

    2026 , version =. doi:10.5281/zenodo.20953922 , url =

    2026 doi

  11. [19]

    2025 , eprint=

    Qwen3 Technical Report , author=. 2025 , eprint=

    2025

  12. [20]

    arXiv preprint arXiv:2602.15763 , year=

    Glm-5: from vibe coding to agentic engineering , author=. arXiv preprint arXiv:2602.15763 , year=

    arXiv

  13. [21]

    5: Visual Agentic Intelligence , author=

    Kimi K2. 5: Visual Agentic Intelligence , author=. arXiv preprint arXiv:2602.02276 , year=

    arXiv

  14. [22]

    International Conference on Learning Representations , volume=

    Swe-bench: Can language models resolve real-world github issues? , author=. International Conference on Learning Representations , volume=

  15. [23]

    arXiv preprint arXiv:2603.24477 , year =

  16. [24]

    OpenThoughts-Agent team, Snorkel AI, Bespoke Labs , month = Feb, title =

  17. [25]

    Advances in Neural Information Processing Systems , volume=

    Intercode: Standardizing and benchmarking interactive coding with execution feedback , author=. Advances in Neural Information Processing Systems , volume=

  18. [26]

    The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=

    Swe-agent: Agent-computer interfaces enable automated software engineering , author=. The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=

  19. [27]

    arXiv preprint arXiv:2104.14337 , year=

    Dynabench: Rethinking benchmarking in NLP , author=. arXiv preprint arXiv:2104.14337 , year=

    arXiv

  20. [28]

    Advances in neural information processing systems , volume=

    Emergent complexity and zero-shot transfer via unsupervised environment design , author=. Advances in neural information processing systems , volume=

  21. [29]

    Advances in neural information processing systems , volume=

    Reflexion: Language agents with verbal reinforcement learning , author=. Advances in neural information processing systems , volume=

  22. [30]

    NeurIPS 2022 Foundation Models for Decision Making Workshop , year=

    React: Synergizing reasoning and acting in language models , author=. NeurIPS 2022 Foundation Models for Decision Making Workshop , year=

    2022

  23. [31]

    Agent Systems with Harness Engineering , author=

  24. [32]

    Expanding the Capability Frontier of

    Xuanzhong Chen and Zile Qiao and Guoxin Chen and Liangcai Su and Zhen Zhang and Xinyu Wang and Pengjun Xie and Fei Huang and Jingren Zhou and Yong Jiang and Ting Chen , booktitle=. Expanding the Capability Frontier of. 2026 , url=

    2026

  25. [33]

    International Conference on Learning Representations , volume=

    Agentbench: Evaluating llms as agents , author=. International Conference on Learning Representations , volume=

  26. [34]

    Advances in Neural Information Processing Systems , volume=

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments , author=. Advances in Neural Information Processing Systems , volume=

  27. [35]

    Transactions on Machine Learning Research , year=

    Core-bench: Fostering the credibility of published research through a computational reproducibility agent benchmark , author=. Transactions on Machine Learning Research , year=

  28. [36]

    International Conference on Learning Representations , volume=

    Openhands: An open platform for ai software developers as generalist agents , author=. International Conference on Learning Representations , volume=

  29. [37]

    arXiv preprint arXiv:2601.16206 , year=

    Llm-in-sandbox elicits general agentic intelligence , author=. arXiv preprint arXiv:2601.16206 , year=

    arXiv

  30. [38]

    Findings of the Association for Computational Linguistics: ACL 2026 , pages=

    Longcli-bench: A preliminary benchmark and study for long-horizon agentic programming in command-line interfaces , author=. Findings of the Association for Computational Linguistics: ACL 2026 , pages=

    2026

  31. [39]

    arXiv preprint arXiv:2606.28480 , year=

    TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents , author=. arXiv preprint arXiv:2606.28480 , year=

    arXiv

  32. [40]

    arXiv preprint arXiv:2604.13018 , year=

    Toward autonomous long-horizon engineering for ml research , author=. arXiv preprint arXiv:2604.13018 , year=

    arXiv

  33. [41]

    arXiv preprint arXiv:2607.08964 , year=

    Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading , author=. arXiv preprint arXiv:2607.08964 , year=

    arXiv

  34. [42]

    arXiv preprint arXiv:2605.14133 , year=

    ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents , author=. arXiv preprint arXiv:2605.14133 , year=

    arXiv

  35. [43]

    arXiv preprint arXiv:2607.14186 , year=

    NexForge: Scaling Executable Agent Tasks via Requirement-First Synthesis , author=. arXiv preprint arXiv:2607.14186 , year=

    arXiv

  36. [44]

    arXiv preprint arXiv:2604.28093 , year=

    What Makes a Good Terminal-Agent Benchmark Task: A Guideline for Adversarial, Difficult, and Legible Evaluation Design , author=. arXiv preprint arXiv:2604.28093 , year=

    arXiv

  37. [45]

    Peng, Xiaoxuan and Zhang, Kaiqi and Lu, Xinyu and Cao, Boxi and Lu, Yaojie and Lin, Hongyu and Han, Xianpei and Sun, Le , journal=

  38. [46]

    arXiv preprint arXiv:2606.03461 , year=

    What Makes Interaction Trajectories Effective for Training Terminal Agents? , author=. arXiv preprint arXiv:2606.03461 , year=

    arXiv

  39. [47]

    Immersion in the

    Zhao, Jiale and Chen, Guoxin and Meng, Fanzhe and Li, Minghao and Chen, Jie and Xu, Hui and Sun, Yongshuai and Zhao, Xin and Song, Ruihua and Zhang, Yuan and others , journal=. Immersion in the

  40. [48]

    arXiv preprint arXiv:2606.10728 , year=

    DeNovoSWE: Scaling Long-Horizon Environments for Generating Entire Repositories from Scratch , author=. arXiv preprint arXiv:2606.10728 , year=

    arXiv

  41. [49]

    arXiv preprint arXiv:2602.03411 , year=

    Swe-master: Unleashing the potential of software engineering agents via post-training , author=. arXiv preprint arXiv:2602.03411 , year=

  42. [50]

    arXiv preprint arXiv:2602.03419 , year=

    SWE-World: Building Software Engineering Agents in Docker-Free Environments , author=. arXiv preprint arXiv:2602.03419 , year=

  43. [51]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) , address=

    LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) , address=. 2024 , url=

    2024

  44. [52]

    arXiv preprint arXiv:2607.10891 , year=

    SETA: Scaling Environments for Terminal Agents , author=. arXiv preprint arXiv:2607.10891 , year=

    arXiv

  45. [53]

    2026 , howpublished =

    2026

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.