SyncMind: Measuring Agent Out-of-Sync Recovery in Collaborative Software Engineering

Chi Han; Heng Ji; Manling Li; Sha Li; Xingyao Wang; Xuehang Guo; Yangyi Chen

REVIEW 4 major objections 4 minor 38 references

SyncMind: Measuring Agent Out-of-Sync Recovery in Collaborative Software Engineering

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read State-of-the-art LLM coding agents are poor at recovering from 'out-of-sync' states in shared codebases: the best tested agent, Claude-3.5-Sonnet, reaches only 28.18% independent recovery and 33.70% with collaborator help, while…

desk verdict SyncBench is a real, reusable benchmark for agent out-of-sync recovery, but the headline SR numbers are internally inconsistent and lack error bars, so treat the quantitative claims as provisional. read the letter →

arxiv 2502.06994 v2 pith:4YJRCGSL submitted 2025-02-10 cs.SE cs.AIcs.CL

Xuehang Guo , Xingyao Wang , Yangyi Chen , Sha Li , Chi Han , Manling Li , Heng Ji This is my paper · ORCID

classification cs.SEcs.AIcs.CL

keywords out-of-syncrecoverycollaborativesoftwareengineeringLLMcodingagentsbenchmarkconstructionassistanceseekingresourceawarenesscommit-historyrollbackagentevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Software engineering is increasingly collaborative, and AI coding agents are usually evaluated in static environments where the codebase never changes under them. This paper argues that real collaboration is dynamic: while an agent works, teammates update the shared repository, so the agent's beliefs about the code can go out of sync with the actual state. To make this measurable, the authors formally define the out-of-sync problem (belief state diverging from world state), build SyncBench, a benchmark of 24,332 rollback-derived instances from 21 real GitHub repositories with executable tests, and evaluate seven LLM agents. The headline finding is that even the strongest agent recovers from out-of-sync states only about a quarter of the time, collaboration is rarely sought even when it helps, and resource constraints barely change agent behavior. If correct, this means today's coding agents lack the awareness and collaborative initiative that shared-codebase work requires.

What carries the argument

The object that carries the argument is the SyncBench out-of-sync instance, generated by a commit-history rollback proxy: for each extracted Python function, the pipeline takes the current state as ground truth $S_2$ and walks the repository's history backwards to find an older state $B_2$ where the function (Caller) or one of its imported dependencies (Callee) is stale and the unit tests fail. Multi-level filtering keeps only instances where the updated state passes, the rolled-back state fails, and parsing results are stable across runs. Agents then recover under a 30-turn protocol (extended to 50 turns in a resource-awareness condition) with three action types: exploring the repository by executing commands in an agentic coding environment, proposing a solution, and, in collaborative mode, asking a know-everything LLM-simulated collaborator. Success is judged by whether the agent's parsed test outputs exactly match the ground-truth outputs, and five metrics summarize the result: success rate, localization accuracy at file and function level, conditional success rate given localization, assistance-seeking rate, and recovery efficiency in time and expense.

What would settle it

Run the same seven agents on instances where the rolled-back state still passes all unit tests but produces different parsed outputs (the pass-to-pass divergence that the paper's own construction pipeline supports but SyncBench's released version filters out). If agents recover at rates far above 28% there, the benchmark's difficulty is an artifact of visible test failures and the claim that agents fail to resynchronize would be weakened; if they recover at similar low rates, the difficulty reflects genuine desynchronization.

Watch

Extended reading notes

Core claim

The paper's central claim is that current LLM agents are not reliable collaborators in evolving codebases because they cannot consistently detect, diagnose, and recover from out-of-sync states. An agent is out-of-sync when its belief state $B_k$ differs from the true repository state $S_k$ at time $T_k$ after a collaborator's update $U$ that the agent missed; recovery requires localizing the root cause, learning about the missing update, and updating the belief so that $B_n = S_n$. SyncBench operationalizes this by rolling a function (Caller tasks) or an imported dependency (Callee tasks) back through commit history until the tests fail, then asking the agent to make the parsed test outputs match the ground truth exactly. Across 300 evaluated instances and seven models, Claude-3.5-Sonnet reaches 28.18% independent success and 33.70% with collaborator assistance, all agents seek assistance at most 4.86% of the time, and an oracle-information upper bound of 86.33% shows large untapped headroom. The paper concludes that technical proficiency alone is insufficient: successful recovery requires proactive collaboration and adaptive resource use, both of which current agents largely lack.

Load-bearing premise

The load-bearing premise is that rolling one function or one imported dependency back through commit history, then scoring the agent on whether parsed test outputs exactly match the current state, faithfully stands in for a real collaborator whose understanding of a shared codebase has gone out of sync.

Editorial extensions

If this is right

Frontier coding agents are far from reliable on shared codebases: if the 28.18% independent recovery rate holds, roughly seven out of ten out-of-sync situations end in failure, so production systems cannot delegate maintenance of evolving repositories to these agents without human oversight.
Collaboration headroom is large: with oracle information the success rate jumps to 86.33%, so agent behavior rather than raw coding skill is a major bottleneck, and training or prompting for proactive, well-timed help-seeking could be a comparatively cheap lever.
Static software-engineering benchmarks fix the workspace, whereas SyncBench's dynamic rollback instances measure a distinct capability, so agent rankings on static tasks may not predict performance in live collaborative development.
Resource awareness cannot be assumed: tripling budgets and halving or doubling assistance costs produced negligible changes in behavior and success, meaning efficiency must be engineered explicitly rather than left to the model.
Agents that explore early and ask high-quality, targeted questions recover more often, giving concrete behavioral targets for the next generation of collaborative agents.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Implicit in the paper but not stated: the benchmark slices the phenomenon narrowly, since each instance rolls back one function or one dependency and is filtered to be test-visible, so the 28% figure should be read as the difficulty of single-edit, failure-visible desynchronization rather than of all collaboration failures; multi-file or silent-drift desynchronization may behave differently.
A testable extension suggested by the results: the oracle-information upper bound of 86.33% implies that most of the gap is initiative and situation awareness rather than code-generation skill, so prompting or fine-tuning agents to verify repository changes before editing could be evaluated directly on SyncBench's 300 instances.
Another consequence the paper leaves implicit: because tripling budgets and halving or doubling assistance costs barely moved behavior, simple resource mentions in prompts are evidently too weak a signal, and explicit cost accounting with real penalties for overspending is a natural next experiment on the same benchmark.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The benchmark is worth keeping, but the headline numbers need a second pass before you quote them. SyncBench is a genuine artifact: it operationalizes "out-of-sync" as a concrete, test-verifiable recovery task mined from git history, with Caller and Callee variants, and runs real agents inside Docker with executable ground-truth checks. The pipeline is open, the data is released, and the qualitative findings—agents recover poorly (best independent SR ≈ 28%) and rarely seek help (ASR ≤ 4.86%)—are likely to survive corrections. That's a real contribution to agent evaluation.

Now the soft spots. First, an internal arithmetic contradiction. Table 1 reports Llama-3.1-8B independent SR of 1.33% on Caller and 0.67% on Callee (150 instances each), which pools to ≈1.00%. Table C1, the source for the paper's overall narrative, reports the same quantity as 0.33%. That's not rounding; it's a genuine inconsistency in the headline metric. Second, there are no confidence intervals or significance tests anywhere. With only 300 total instances and per-repo samples as small as n=3 (whisper), the differences among mid-tier agents are plausibly sampling noise. The paper should either add uncertainty quantification or explicitly state that the relative ordering of mid-tier models is not statistically supported. Third, the resource-awareness claims in §4.7 rest on only two models (Llama-3.1-8B and 70B), yet the abstract generalizes to "a significant lack of resource awareness" across LLM agents. That's an overreach. The downsampled evaluation subset (≤15 instances per repo) also means the numbers describe the sampled distribution, not the full 24k-instance benchmark.

To be fair, the paper's own appendix acknowledges several of these limitations: the git-rollback proxy doesn't capture all live collaboration patterns, and the resource model is simplified. That honesty is good. The central design—task construction, filtering, and executable validation—is sound, and the benchmark is reusable. The inconsistency and missing error bars are fixable; they don't undermine the artifact.

Who this is for: anyone building or evaluating agents for real-world software collaboration. It deserves a serious referee; the bench itself is a contribution. But the revision needs to resolve the Table 1/C1 contradiction, add or explicitly decline uncertainty quantification, and narrow the resource-awareness generalization.

My recommendation: engage with it, but treat the specific numbers as provisional until cleaned up.

Referee Report

4 major / 4 minor

Summary. The paper introduces SyncMind, a framework that formalizes the notion of an LLM coding agent being out-of-sync with a shared codebase, together with SyncBench, a benchmark constructed by rolling Python functions or imported dependencies back to older failing commits in 21 GitHub repositories. The benchmark contains 24,332 raw instances, of which 300 are evaluated across seven LLMs under independent and collaborative recovery settings, with executable test-based verification, localization and conditional success metrics, assistance-seeking rate, and resource-aware recovery under time and budget constraints. The headline empirical findings are that recovery success is generally low (Claude-3.5-Sonnet reaches 28.18% independent and 33.70% collaborative SR), assistance-seeking is rare (ASR at most 4.86%), collaboration improves performance when it occurs, and varying budgets or time limits has little effect on the two models tested in the resource experiments.

Significance. If the quantitative results are taken at face value, this is a useful and timely contribution to agent evaluation in collaborative software engineering. The construction pipeline is open-source, the instances are derived from real repositories, verification uses executable ground-truth tests rather than subjective judgments, and the evaluation spans seven LLMs in two interaction modes with several complementary metrics. These are concrete strengths that go beyond a purely static code-repair benchmark. The credible parts of the paper would provide evidence that current agents are weak at recovering from a controlled single-edit desynchronization and rarely ask for help. The main value lies in the benchmark artifact and in the qualitative observation that agent behavior changes little under resource constraints, though the latter is currently established only for two models.

major comments (4)

[Table 1 vs Table C1, §3.4 and §4.2] There is an arithmetic inconsistency in the headline metric. For Llama-3.1-8B, Table 1 reports independent SR of 1.33% on Caller (150 instances) and 0.67% on Callee (150 instances), which pools to 1.00% (3 successes out of 300), yet Table C1 reports overall independent SR of 0.33%. Since Table C1 is the source for the performance-gap statements in §4.2 and for the abstract's lower-bound figures, this is not a rounding artifact. Please correct the affected row and re-derive the claims that depend on it, or explicitly explain why the two tables are not directly comparable.
[§4.2, §4.6, Tables C1 and B2] No confidence intervals, significance tests, or effect-size estimates are reported. With 300 total instances and per-repository samples as small as n=3 (whisper) and n=6 (scikit-learn), the mid-tier ordering (e.g., GPT-4o 4.00%, DeepSeek 7.33%, Llama-3.3-70B 16.33%) may plausibly be within sampling noise, and the repository-level analysis in §4.6 relies on very small counts. Please add binomial confidence intervals or an explicit uncertainty/ordering analysis, and restrict the qualitative model-ordering claims to what those intervals support.
[§4.7 and Tables C5–C7] The resource-awareness experiments are run on only two models, Llama-3.1-8B and Llama-3.1-70B, but the abstract and §4.7 conclude that existing LLM agents show a significant lack of resource awareness and adaptability. This generalization is not supported by the reported evidence. Please either extend the resource-constraint evaluation to all seven agents or explicitly scope the conclusion to the two evaluated models and remove the broader claim from the abstract.
[§3.1, §3.4, Appendix A] The benchmark operationalizes out-of-sync scenarios as a git rollback of one function (Caller) or one imported dependency (Callee) to an older failing commit, with success defined by exact parsed-test-output match. As the authors acknowledge in Appendix A, this may not capture multi-file, multi-agent, or non-test-visible desynchronization. This is not a circularity problem, but it is a validity threat to the abstract's wording about fundamental limitations of LLMs in CSE generally. Please add an analysis of how representative the rollback instances are of real out-of-sync events (e.g., distribution of change sizes, comparison with genuine concurrent-edit scenarios), or re-scope the abstract and conclusion to the simulated single-edit recovery setting.

minor comments (4)

[§B.2, example (2)] The example header says 'nwaskon/seaborn'; the repository name should be 'mwaskom/seaborn' as in Table B2.
[§C.1] The sentence 'comparable technical recovery capacities among Claude-3.5-Sonnet, GPT-4o mini, and GPT-4o mini' appears to contain a typo; the second mention should likely be 'GPT-4o'.
[§C.1] The phrase 'our evaluation results (Tab. C1-1)' should refer to a specific table number, either 'Table C1' or 'Table 1'.
[Equation (5) and Table C8] The text says recovery efficiency is inversely proportional to the reported Eff values; this wording is confusing because higher efficiency should mean lower resource use. Consider renaming the quantities to 'resource usage ratios' or clarifying the direction explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are empirical measurements against executable ground-truth tests, not derivations from the benchmark's own definitions.

full rationale

SyncMind's out-of-sync definition (§2.1) and SyncBench's construction (§3.1) define what counts as an out-of-sync instance—a git-rolled-back function or dependency whose tests fail—but the paper's headline results are measured agent behaviors on those instances: SR is computed by executing the agent's proposed repository and comparing parsed test outputs with the ground-truth commit (§3.4), ASR is measured from turn-level actions, and resource-awareness findings are measured under varied turn, budget, and cost settings (§4.7). None of these quantities is fitted from, or definitionally equal to, the benchmark's filtering criteria; the metric is not defined as the filter itself, so the construction and the finding remain separable. The oracle "upper-bound" experiment (§4.4) does inject ground-truth information by design, but the reported 86.33% is an empirical outcome of how well agents follow full instructions, not a forced consequence of the setup. The only same-group citation is OpenHands (Wang et al., 2024a), used as the agent environment; it is an external, code-released platform rather than a load-bearing self-citation, and the paper does not invoke any author-supplied uniqueness theorem or ansatz. Appendix A's stated limitations—Python-only repositories, simplified resource modeling, and simulation not capturing all live collaboration patterns—are honest scope restrictions and do not indicate a circular derivation. A separate arithmetic discrepancy between Table 1 and Table C1 for Llama-3.1-8B SR is a consistency and correctness concern, not a circularity one.

Assumptions & free parameters 5 free parameters · 5 assumptions · 2 invented entities

The central claims rest on a small number of domain assumptions about how out-of-sync is operationalized (single-function git rollback, exact-output success, simulated collaborator) plus hand-set experimental budgets. There are no fitted mathematical constants, but the 300-instance subset and resource settings are experimenter-chosen and shape the headline percentages.

free parameters (5)

Max recovery turns = 30 turns (50 in extension)
Set after pilot tests on Llama-3.1-70B and GPT-4o; directly bounds SR and ASR and is a resource constraint in all experiments.
Initial budget = $1000
Chosen after pilots because $3000 budget made SR worse; used as the standard resource limit and the basis for the resource-awareness result.
Action costs = $100 proposal, $100 assistance
Hand-set in pilot; determines how many solution and assistance actions agents can afford within 30 turns, so it shapes ASR and cost-sensitivity results.
Per-repo downsampling cap = 15 instances per repo (300 total)
Caps each repository's contribution to the evaluation subset, changing the effective task distribution relative to the full 24,332-instance pool.
Filtering criteria = at least 1 failure in B2, more than 1 pass in S1, identical parsing S1/Sn
Hand-selected quality thresholds that determine which commit-rollback instances enter SyncBench; affects task difficulty distribution.

assumptions (5)

domain assumption A single collaborator's belief state Bk deviating from world state Sk is a complete characterization of the out-of-sync problem.
Section 2.1 formalizes out-of-sync with three conditions; the paper assumes these conditions capture what real CSE collaborators experience.
domain assumption Rolling a function or dependency back through git history until unit tests fail produces realistic out-of-sync instances.
Section 3.1 Out-of-Sync Simulation constructs Caller and Callee this way; the realism of this proxy is not independently validated.
domain assumption An LLM given complete ground-truth context and response restrictions behaves like a know-everything human collaborator.
Section 3.3 defines know-everything collaborators; the only validation is the in-paper single-turn oracle experiment in Section 4.4.
domain assumption Exact match of parsed pytest outputs between agent solution and ground-truth state is a correct success criterion.
Section 3.4 Parsing Validation; a correct semantic fix that changes test output formatting would be counted as failure.
domain assumption The tested models are representative of 'existing LLM agents' for generalizing resource-awareness conclusions.
Section 4.7 resource experiments use only Llama-3.1-8B and Llama-3.1-70B; generalization to all LLM agents is assumed.

invented entities (2)

Agent out-of-sync state (Bk != Sk)
purpose: Formal target construct that SyncBench operationalizes and measures.
Defined in Section 2.1; no external falsifiable handle outside the benchmark itself.
Know-everything collaborator
purpose: LLM-simulated ideal human collaborator for collaborative recovery experiments.
Constructed in Section 3.3 with ground-truth context; its validity rests on the oracle single-turn experiment rather than external evidence.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SyncMind: Measuring Agent Out-of-Sync Recovery in Collaborative Software Engineering." pith.science (2026). https://pith.science/paper/4YJRCGSL

@misc{pith2026250206994,
  author       = {Pith},
  title        = {Pith review of: SyncMind: Measuring Agent Out-of-Sync Recovery in Collaborative Software Engineering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4YJRCGSL}},
  note         = {Machine review of arXiv:2502.06994}
}

read the original abstract

Software engineering (SE) is increasingly collaborative, with developers working together on shared complex codebases. Effective collaboration in shared environments requires participants -- whether humans or AI agents -- to stay on the same page as their environment evolves. When a collaborator's understanding diverges from the current state -- what we term the out-of-sync challenge -- the collaborator's actions may fail, leading to integration issues. In this work, we introduce SyncMind, a framework that systematically defines the out-of-sync problem faced by large language model (LLM) agents in collaborative software engineering (CSE). Based on SyncMind, we create SyncBench, a benchmark featuring 24,332 instances of agent out-of-sync scenarios in real-world CSE derived from 21 popular GitHub repositories with executable verification tests. Experiments on SyncBench uncover critical insights into existing LLM agents' capabilities and limitations. Besides substantial performance gaps among agents (from Llama-3.1 agent <= 3.33% to Claude-3.5-Sonnet >= 28.18%), their consistently low collaboration willingness (<= 4.86%) suggests fundamental limitations of existing LLM in CSE. However, when collaboration occurs, it positively correlates with out-of-sync recovery success. Minimal performance differences in agents' resource-aware out-of-sync recoveries further reveal their significant lack of resource awareness and adaptability, shedding light on future resource-efficient collaborative systems. Code and data are openly available on our project website: https://xhguo7.github.io/SyncMind/.

Figures

Figures reproduced from arXiv: 2502.06994 by the authors.

**Figure 1.** The Out-of-Sync Challenge. At Ti , Agent and Human work on respective tasks. During Agent’s task completion from Ti to Tk , Human updates <repo> at Tj that Agent is unaware of due to being occupied with its own task. This leads Agent to become out-of-sync at Tk as a result of Sk ̸= Bk. Collaborative systems—whether involving humans, AI agents, or both—boost efficiency and capabilities by combining complementary st… view at source ↗

**Figure 2.** Typical Causes of Out-of-Sync. Examples of out-of-sync scenarios in our benchmark. workspace states—a core challenge in the field (Yang et al., 2024b). While version control systems (Torvalds, 2005) can detect surface-level code conflicts, they cannot identify semantic inconsistencies that require manual resolution. This includes scenarios where agents must resolve dependency updates, modify existing functions to a… view at source ↗

**Figure 3.** SyncMind for Measuring Agent Out-of-Sync Recovery. Depending on agent’s initiative, an agent updates its world belief (B2 → Bn) by interacting with Env, proposing solutions, and seeking collaborator assistance. a task, the agent becomes out-of-sync at Tk (Ti < Tk) if any of the following conditions are satisfied: (1) Knowledge gap: ∃ Update U at time Tj (Ti < Tj < Tk) where the agent lacks knowledge of U. (2) State … view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Resource-Aware Out-of-Sync Recovery. We introduce resource-aware recovery by mapping resource consumption to each out-of-sync recovery task. 3. SyncBench: Agent Out-of-Sync Benchmark 3.1. Benchmark Construction Aligning with real-world out-of-sync scenarios, our bench…

**Figure 5.** Figure 5: Agent Out-of-Sync Benchmark Construction. A systematic benchmark construction approach (§3.1). testing environments tailored for our out-of-sync recovery tasks. Each source repository is packaged into a dedicated Docker image with complete codebase, dependencies, and v…

**Figure 6.** Figure 6: Influence of Collaborator Assistance. We quantify collaborator influence on agent out-of-sync recovery performance as ∆collaborator to unveil its +positive or -negative impact on certain aspects of agents’ recovery performance. 5 [PITH_FULL_IMAGE:figures/full_fig_p005…

**Figure 8.** Figure 8: Question Quality. Agents from left to right on the Xaxis according to their ASR from low to high. 4.6. More Challenging Tasks Decrease Performance While Better Manifest Collaboration Benefits We observe a large negative influence of increased task complexity on agents…

**Figure 7.** Figure 7: Time Allocation. Agents’ performance are ranked from low to high according to their independent SR scores, based on which they are positioned on the X-axis from left to right. The Y-axis depicts each agent’s time allocation. with ∆collaborator = +3.82%). This holds in …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 21 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[3]

Claude , 2023

Anthropic. Claude , 2023. URL https://www.anthropic.com/claude

work page 2023
[4]

Claude-3.5 Sonnet , 2024

Anthropic. Claude-3.5 Sonnet , 2024. URL https://www.anthropic.com/claude/sonnet

work page 2024
[5]

Are they the same picture? adapting concept bottleneck models for human-ai collaboration in image retrieval

Balloli, V., Beery, S., and Bondi-Kelly, E. Are they the same picture? adapting concept bottleneck models for human-ai collaboration in image retrieval. 2024

work page 2024
[6]

Coreqa: Uncovering potentials of language models in code repository question answering, 2025

Chen, J., Zhao, K., Liu, J., Peng, C., Liu, J., Zhu, H., Gao, P., Yang, P., and Deng, S. Coreqa: Uncovering potentials of language models in code repository question answering, 2025. URL https://arxiv.org/abs/2501.03447

arXiv 2025
[7]

Through the theory of mind's eye: Reading minds with multimodal video large language models

Chen, Z., Wang, T., Wang, Y., Kosinski, M., Zhang, X., Fu, Y., and Li, S. Through the theory of mind's eye: Reading minds with multimodal video large language models. ArXiv, abs/2406.13763, 2024 a

arXiv 2024
[8]

T o MB ench: Benchmarking theory of mind in large language models

Chen, Z., Wu, J., Zhou, J., Wen, B., Bi, G., Jiang, G., Cao, Y., Hu, M., Lai, Y., Xiong, Z., and Huang, M. T o MB ench: Benchmarking theory of mind in large language models. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 15959--15983,...

doi:10.18653/v1/2024.acl-long.847 2024
[9]

Devin , 2024

Cognition AI . Devin , 2024. URL https://devin.ai/

work page 2024

Show all 38 references

[10]

DeepSeek V2.5 , 2024

DeepSeek. DeepSeek V2.5 , 2024. URL https://api-docs.deepseek.com/news/news0905

2024
[11]

Docker , 2013

Founadi, K., Hykes, S., and Pahl, S. Docker , 2013. URL https://www.docker.com/

2013
[12]

M., Luck, M., Bu, Q., Qing, Y., and Cui, H

Huang, D., Zhang, J. M., Luck, M., Bu, Q., Qing, Y., and Cui, H. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv preprint arXiv:2312.13010, 2024. doi:10.48550/arXiv.2312.13010

2024 doi arXiv
[13]

R2e: Turning any github repository into a programming agent environment

Jain, N., Shetty, M., Zhang, T., Han, K., Sen, K., and Stoica, I. R2e: Turning any github repository into a programming agent environment. In ICML, 2024

2024
[14]

E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K

Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

2023 arXiv
[15]

E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K

Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2024. doi:10.48550/arXiv.2310.06770

2024 doi arXiv
[16]

FANT o M : A benchmark for stress-testing machine theory of mind in interactions

Kim, H., Sclar, M., Zhou, X., Bras, R., Kim, G., Choi, Y., and Sap, M. FANT o M : A benchmark for stress-testing machine theory of mind in interactions. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language P...

2023 doi
[17]

Theory of mind may have spontaneously emerged in large language models

Kosinski, M. Theory of mind may have spontaneously emerged in large language models. arXiv preprint, abs/2302.02083, 2023

2023 arXiv
[18]

Large language model-based agents for software engineering: A survey

Liu, J., Wang, K., Chen, Y., Peng, X., Chen, Z., Zhang, L., and Lou, Y. Large language model-based agents for software engineering: A survey. ArXiv, abs/2409.02977, 2024. URL https://api.semanticscholar.org/CorpusID:272423732

2024 arXiv
[19]

robots of robots

Machowski, L. and Marwala, T. Nano version control and "robots of robots" – data driven, regenerative production code. 2021 International Conference on Computational Science and Computational Intelligence (CSCI), pp.\ 1938--1943, 2021. URL https://api.semanticscholar.org/Corpu...

2021
[20]

Meta-LLaMA-3.1-70B , 2024 a

Meta AI . Meta-LLaMA-3.1-70B , 2024 a . URL https://huggingface.co/meta-llama/Llama-3.1-70B

2024
[21]

Meta-LLaMA-3.1-8B , 2024 b

Meta AI . Meta-LLaMA-3.1-8B , 2024 b . URL https://huggingface.co/meta-llama/Llama-3.1-8B

2024
[22]

Meta-LLaMA-3.3-70B , 2024 c

Meta AI . Meta-LLaMA-3.3-70B , 2024 c . URL https://huggingface.co/meta-llama/Llama-3.3-70B

2024
[23]

ChatGPT , 2022

OpenAI. ChatGPT , 2022. URL https://openai.com/chatgpt/overview/

2022
[24]

GPT-4o , 2024 a

OpenAI. GPT-4o , 2024 a . URL https://platform.openai.com/docs/models\#gpt-4o

2024
[25]

GPT-4o mini , 2024 b

OpenAI. GPT-4o mini , 2024 b . URL https://platform.openai.com/docs/models\#gpt-4o-mini

2024
[26]

Predicting merge conflicts in collaborative software development

Owhadi-Kareshk, M., Nadi, S., and Rubin, J. Predicting merge conflicts in collaborative software development. 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pp.\ 1--11, 2019. URL https://api.semanticscholar.org/CorpusID:196621587

2019
[27]

Strachan, J. W. A., Albergo, D., Borghini, G., Pansardi, O., Scaliti, E., Gupta, S., Saxena, K., Rufo, A., Panzeri, S., Manzi, G., Graziano, M. S. A., and Becchio, C. Testing theory of mind in large language models and humans. Nature Human Behaviour, 8: 0 1285--1295, 2024

2024
[28]

Git , 2005

Torvalds, L. Git , 2005. URL https://git-scm.com/

2005
[29]

Verma, M., Bhambri, S., and Kambhampati, S. Theory of mind abilities of large language models in human-robot interaction: An illusion? In Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction, HRI '24, pp.\ 36–45, New York, NY, USA, 2024. Associati...

2024
[30]

F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh, J., Tran, H

Wang, X., Li, B., Song, Y., Xu, F. F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh, J., Tran, H. H., Li, F., Ma, R., Zheng, M., Qian, B., Shao, Y., Muennighoff, N., Zhang, Y., Hui, B., Lin, J., Brennan, R., Peng, H., Ji, H., and Neubig, G. O pen H ands: An open platf...

2024 doi arXiv
[31]

Mint: Evaluating llms in multi-turn interaction with tools and language feedback

Wang, X., Wang, Z., Liu, J., Chen, Y., Yuan, L., Peng, H., and Ji, H. Mint: Evaluating llms in multi-turn interaction with tools and language feedback. 2024 b

2024
[32]

Executable code actions elicit better llm agents

Wang, X., Chen, Y., Yuan, L., Zhang, Y., Li, Y., Peng, H., and Ji, H. Executable code actions elicit better llm agents. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2025

2025
[33]

Unleashing cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration

Wang, Z., Mao, S., Wu, W., Ge, T., Wei, F., and Ji, H. Unleashing cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration. In Proc. 2024 Annual Conference of the North American Chapter of the Association for Computational Lingu...

2024
[34]

P., and Morency, L.-P

Wilf, A., Lee, S., Liang, P. P., and Morency, L.-P. Think twice: Perspective-taking improves large language models' theory-of-mind capabilities. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Ling...

2024 doi
[35]

Hi- T o M : A benchmark for evaluating higher-order theory of mind reasoning in large language models

Wu, Y., He, Y., Jia, Y., Mihalcea, R., Chen, Y., and Deng, N. Hi- T o M : A benchmark for evaluating higher-order theory of mind reasoning in large language models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computational Linguistics: EMNLP ...

2023 doi
[36]

O pen T o M : A comprehensive benchmark for evaluating theory-of-mind reasoning capabilities of large language models

Xu, H., Zhao, R., Zhu, L., Du, J., and He, Y. O pen T o M : A comprehensive benchmark for evaluating theory-of-mind reasoning capabilities of large language models. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association fo...

2024 doi
[37]

E., Zhang, A

Yang, J., Jimenez, C. E., Zhang, A. L., Lieret, K., Yang, J., Wu, X., Press, O., Muennighoff, N., Synnaeve, G., Narasimhan, K. R., et al. Swe-bench multimodal: Do ai systems generalize to visual software domains? arXiv preprint arXiv:2410.03859, 2024 a

arXiv
[38]

Towards unified alignment between agents, humans, and environment

Yang, Z., Liu, A., Liu, Z., Liu, K., Xiong, F., Wang, Y., Yang, Z., Hu, Q., Chen, X., Zhang, Z., Luo, F., Guo, Z., Li, P., and Liu, Y. Towards unified alignment between agents, humans, and environment. arXiv preprint, abs/2402.07744, 2024 b

2024 arXiv
[39]

C ode A gent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges

Zhang, K., Li, J., Li, G., Shi, X., and Jin, Z. C ode A gent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa...

2024 doi

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [3]

Claude , 2023

Anthropic. Claude , 2023. URL https://www.anthropic.com/claude

work page 2023

[3] [4]

Claude-3.5 Sonnet , 2024

Anthropic. Claude-3.5 Sonnet , 2024. URL https://www.anthropic.com/claude/sonnet

work page 2024

[4] [5]

Are they the same picture? adapting concept bottleneck models for human-ai collaboration in image retrieval

Balloli, V., Beery, S., and Bondi-Kelly, E. Are they the same picture? adapting concept bottleneck models for human-ai collaboration in image retrieval. 2024

work page 2024

[5] [6]

Coreqa: Uncovering potentials of language models in code repository question answering, 2025

Chen, J., Zhao, K., Liu, J., Peng, C., Liu, J., Zhu, H., Gao, P., Yang, P., and Deng, S. Coreqa: Uncovering potentials of language models in code repository question answering, 2025. URL https://arxiv.org/abs/2501.03447

arXiv 2025

[6] [7]

Through the theory of mind's eye: Reading minds with multimodal video large language models

Chen, Z., Wang, T., Wang, Y., Kosinski, M., Zhang, X., Fu, Y., and Li, S. Through the theory of mind's eye: Reading minds with multimodal video large language models. ArXiv, abs/2406.13763, 2024 a

arXiv 2024

[7] [8]

T o MB ench: Benchmarking theory of mind in large language models

Chen, Z., Wu, J., Zhou, J., Wen, B., Bi, G., Jiang, G., Cao, Y., Hu, M., Lai, Y., Xiong, Z., and Huang, M. T o MB ench: Benchmarking theory of mind in large language models. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 15959--15983,...

doi:10.18653/v1/2024.acl-long.847 2024

[8] [9]

Devin , 2024

Cognition AI . Devin , 2024. URL https://devin.ai/

work page 2024