Pith. sign in

REVIEW 3 major objections 3 minor 2 cited by

RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task Solving

T0 review · 3 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Code agents, this paper argues, should reuse GitHub repositories as modular components, and RepoMaster—mapping structure, feeding only core parts—raises task-pass rates from 40.7% to 62.9% with 95% fewer tokens.

desk verdict A useful repository-aware agent with real reported gains, but the MLE-R protocol and headline numbers need cleaning up before the specific claims are cited. read the letter →

arxiv 2505.21577 v3 pith:SZQJENZG submitted 2025-05-27 cs.SE cs.AI

Huacan Wang , Ziyi Ni , Shuo Zhang , Shuo Lu , Sen Hu , Ziyang He , Chen Hu , Jiaye Lin
show 6 more authors
Yifu Guo Ronghao Chen Xin Li Daxin Jiang Yuntao Du Pin Lyu
This is my paper
classification cs.SEcs.AI
keywords codeagentsGitHubrepositoriesrepositoryexplorationfunctioncallgraphmoduledependencycorecomponentidentificationtokenefficiencyTaskBench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RepoMaster is an agent framework with a central thesis: the bottleneck in complex real-world coding tasks is not a model's ability to write code but its ability to find and understand the right existing code, so an agent should reuse GitHub repositories as modular components rather than generate codebases from scratch. To enable that, the framework first builds a structural map of a repository—a hierarchical code tree, a function call graph, and a module dependency graph—scores each module and class to identify core components, and then runs an interleaved exploration-and-execution loop that prunes what the LLM sees at every step. On MLE-R (22 Kaggle-derived ML tasks), RepoMaster with Claude 3.5 reports a 95.45% valid submission rate and a 27.27% medal rate, a more than five-fold improvement over the best open-source baseline; on the new GitTaskBench it reports a 62.96% task pass rate versus 40.74% for OpenHands while using about 95% fewer tokens. If these results hold, code agents should be designed as repository navigators with aggressive context budgeting, not just better code generators.

What carries the argument

The load-bearing machinery is a three-stage pipeline. A repository-search stage extracts key entities from the user's natural-language request and selects candidate GitHub repositories by README content and star count. A hierarchical-analysis stage performs one Abstract Syntax Tree walk per Python file to harvest modules, classes, and functions, then builds the three artefacts described above; module importance is scored as $I(m) = \min(\sum_{i=1}^6 w_i s_i(m), 10)$ with six features (Dependency via personalized PageRank, Complexity, Usage, Semantic, Doc, Git), and class importance is refined as $J(c) = I(\mu(c)) + |F_c|/\max_{c'} |F_{c'}| + \mathrm{Calls}(F_c)/\max_{c'} \mathrm{Calls}(F_{c'})$, selecting the top-$k$ classes as core components. An autonomous exploration-and-execution stage gives the agent Granular Code View, Dependency Analysis, and Search tools, while information selection keeps each interaction under a per-interaction token limit $L = 8000$ (documents chunked at $L_c = 1000$ tokens with the top $n_c = 4$ segments retained). The central idea is that a loss-minimal structural synopsis plus selective viewing lets the LLM behave like a human programmer who maps the project first, jumps along error traces, and filters out irrelevant detail.

What would settle it

Re-run the MLE-R comparison with the target repository for each task chosen independently of RepoMaster's search (human expert curation, or each baseline's own retrieval) and check whether the medal-rate advantage over OpenHands persists; if it collapses, the claimed five-fold improvement is largely an artifact of benchmark construction. A complementary check: on GitTaskBench, replace the six-feature importance scorer with random selection of the $k=10$ core classes and measure the drop in task pass rate to see whether core-component identification itself carries the performance.

Watch

Extended reading notes

Core claim

The paper's central claim is that goal-oriented repository understanding—built from static structure and maintained by active pruning—is what separates capable code agents from struggling ones. RepoMaster treats a target repository as a set of code atoms (modules, classes, functions, imports) and compresses it into three artefacts: a Hierarchical Code Tree (a nested package-to-module-to-class-to-function containment map), a Function Call Graph (edges for invocations, weights for call frequency), and a Module Dependency Graph (edges for imports, weights for coupling). It then scores modules by six features—dependency centrality via personalized PageRank, complexity, usage frequency, semantic naming, docstring quality, and git activity—and refines to the top-$k$ classes as "core components," which become the agent's initial context together with the README and module summaries. From there the agent alternates between writing and running scripts and calling exploration tools (granular code view, dependency analysis, search), with a multi-level information-selection strategy (key-subtree extraction for code, chunk-and-retrieve for documents, head-and-tail retention for logs) holding each interaction to an 8000-token budget. The paper argues that this design produces both the five-fold medal improvement on MLE-R and the 62.96% task pass rate on GitTaskBench, and that the roughly 95% token saving shows the framework gives the LLM the right information rather than more information.

Load-bearing premise

The load-bearing premise is that the MLE-R comparison is fair—that RepoMaster's own repository search, which picked the repositories all agents were told to use, does not systematically favor its own static-analysis design.

Editorial extensions

If this is right

  • Generalist code agents should add a static structure-mapping stage before execution; the ablation on GitTaskBench shows that removing hierarchical analysis, exploration tools, or information selection each lowers the task pass rate, and removing all three drops it from 40.74% to 24.07%.
  • Repository reuse becomes a practical alternative to generating code from scratch: RepoMaster's gains come from adapting existing repositories, so tasks that map onto open-source code need not be regenerated line by line.
  • Context budgeting is a first-class design axis: 154k tokens per task versus 2883k for OpenHands on GitTaskBench shows that deciding what the LLM sees can matter as much as the reasoning loop itself.
  • The benefit transfers across model families: RepoMaster beats both baselines with GPT-4o, Claude 3.5, and DeepSeek V3, indicating the structural map supplies information the underlying models do not extract on their own.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the MLE-R comparison likely overstates the capability gap, because RepoMaster's own repository search chose the repositories that every agent was then told to use; if that selection suits static analysis, part of the five-fold medal improvement is benchmark construction rather than general capability.
  • Editorial inference: the information-selection techniques (AST-subtree extraction, chunk-and-retrieve, head/tail log retention) look domain-general and should transfer to other long-horizon agent settings such as scientific literature navigation, hardware design files, or large documentation corpora.
  • Editorial inference: a natural next test is whether the six-feature module scorer generalizes to the analogous domains the paper itself names—hardware description languages, robotic middleware, notebook collections—since no evidence here shows that the graphs or scoring transfer.
  • Editorial inference: because RepoMaster uses far fewer tokens while succeeding more often, the binding constraint on the baselines appears to be information allocation rather than reasoning; a controlled experiment giving OpenHands the same core-component list would test whether its pass rate closes the gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes RepoMaster, an end-to-end agent framework for solving complex tasks by reusing GitHub repositories. The framework has three stages: a repository search that selects candidate repositories from a task description, a hierarchical static analysis that builds a code tree, function-call graph, module-dependency graph, and core-component ranking, and an autonomous exploration/execution loop with context-window pruning. The authors evaluate RepoMaster on two benchmarks: MLE-R, a modified subset of MLE-Bench with repositories selected by RepoMaster's own search, and GitTaskBench, a new benchmark of 18 repositories and 54 natural-language tasks. Against OpenHands and SWE-Agent across GPT-4o, Claude 3.5, and DeepSeek V3, they report higher valid-submission and medal rates on MLE-R, higher execution-completion and task-pass rates on GitTaskBench, and substantially lower token consumption. The paper includes ablations, a case study, and full raw-result tables in the appendix.

Significance. If the empirical results hold, this is a useful contribution to the growing line of work on repository-level code agents. The core idea—combining static structural analysis with progressive, context-aware exploration—is sensible and is supported by the internal ablations, which show that each component contributes to the reported gains. The release of GitTaskBench is also a valuable resource, and the large token-efficiency gains are a concrete and practically important strength. However, the paper's headline MLE-R claim is currently misreported, the MLE-R evaluation protocol has an unresolved fairness asymmetry, and the GitTaskBench numbers are internally inconsistent between the main text and the appendix. These issues are fixable within the manuscript's scope, but they must be addressed before the empirical claims can be accepted.

major comments (3)
  1. [Abstract and Table 1] The headline claim of a "110% relative boost in valid submissions" is not reproducible from Table 1. Comparing RepoMaster with Claude 3.5 (95.45% valid submission) against the strongest baseline OpenHands with Claude 3.5 (50.00%) gives a 90.9% relative increase, not 110%. The 110% figure matches the "Above Median" column (95.45 vs 45.45). The abstract should either quote the correct metric or explicitly state the comparison basis. As written, the central quantitative claim is misreported.
  2. [Section 4.1 and Appendix B] The fairness of the MLE-R comparison is not established. The benchmark is constructed by applying RepoMaster's own repository search (Section 3.1) to select a repository for each task, and all agents are then given that repository. The search output is a structured JSON containing fields such as "solution_approach", "performance", and "can_solve_task" (Appendix B). The paper does not state whether this JSON is provided to RepoMaster's context while baselines receive only a repository path. If the search hints are available to RepoMaster, the MLE-R results conflate the value of the search/hints with the value of the exploration framework. Even if the hints are withheld, the comparison denies baselines any retrieval stage, so the reported advantage on MLE-R is not a clean test of the exploration loop. The authors should disclose exactly what information each agent receives, and either (a) run baselines with an equivalent retrieval stage or (b) report an ablation of RepoMaster without the search-selected hints.
  3. [Tables 2 and 5] The GitTaskBench numbers are internally inconsistent. Table 2 reports for OpenHands/Claude 3.5 an execution completion rate of 53.70%, task pass rate of 40.74%, and 2883k tokens, while Table 5 in Appendix G.2 reports 48.15%, 24.07%, and 3,094,878 tokens for the same configuration. Similarly, SWE-Agent/Claude 3.5 appears as 41.67%/22.23%/456k in Table 2 but 44.44%/14.81%/330,676 in Table 5. The 95% token-reduction claim and the pass-rate comparison depend on which set of numbers is correct. The paper should reconcile these tables before the GitTaskBench results can be trusted.
minor comments (3)
  1. [Appendix G.1, Tables 9–10] Some competition names in Tables 9 and 10 differ from those in Tables 6–8 (e.g., "text-normalization-challenge-english-language" in Tables 9–10 versus "text-normalization-challenge-russian-language" in Tables 6–8). Please verify that the task names are consistent across the raw-result tables.
  2. [Figure 1 caption] The caption contains a stray word "preparing" in "Hierarchical Repository Analysis and Autonomous Exploration & Execution"; this appears to be a formatting artifact and should be cleaned up.
  3. [Section 3.2.2, Eq. (2)] Equation (2) sets w_i ≡ 1, so the score is a plain sum of six features; the text calls the features "orthogonal" but does not define or test this property. A brief clarification that equal weighting is a design choice, not a tuned parameter, would help readers interpret the ablation results.

Circularity Check

1 steps flagged · score 2.0 of 10

No equation-level circularity; the only self-referential element is that MLE-R tasks are built from repositories selected by RepoMaster's own search, a benchmark-construction confound rather than a fitted-parameter reduction.

  1. other [Section 4.1 (MLE-R construction, footnote 1), Section 3.1, Appendix B]
    "To construct MLE-R, we select 22 MLE-Bench tasks (covering nearly all MLE-Bench-lite cases) and apply the search procedure described in Section 3.1 to retrieve suitable GitHub repositories for each task, ensuring a fair comparison 1; the tasks’ requirements are set to be completed based on their chosen repository rather than generating code from scratch. [Footnote 1] Ensure a fair comparison, as other general agent frameworks do not support automatic repository retrieval."

    MLE-R is not fully independent of the method under test: its repositories are produced by applying RepoMaster's own Section 3.1 search procedure, and Appendix B shows the selection is based on candidates' 'solution approach' and reported 'performance' (e.g., 'Ensemble of deep learning classifiers (ResNet34)', 'Kappa metric of 0.904'). Because the chosen repository is an output of RepoMaster's pipeline, evaluating RepoMaster on MLE-R measures the method against tasks it helped construct, while footnote 1 states that general agent frameworks do not get an automatic repository-retrieval stage. The Section 4.3 'more than five-fold' medal improvement is therefore partly an artifact of benchmark construction rather than an independent test of the exploration/execution loop alone.

full rationale

No definitional or fitted-parameter circularity is present: RepoMaster's core components (HCT, FCG, MDG, and the importance scoring in Eqs. 1-3) are defined by explicit formulas and tested by ablation, and the headline claims are empirical scores measured on MLE-Bench-derived tasks and GitTaskBench, not quantities derived from the method's own fitted parameters. The single self-referential element is the construction of MLE-R: Section 4.1 applies RepoMaster's own Section 3.1 repository-search procedure to choose each task's repository, so the benchmark's inputs are partly produced by the method under test, and baselines are denied an equivalent retrieval stage (footnote 1). This is a benchmark-construction and external-validity concern rather than a mathematical equivalence, so the circularity score is low. GitTaskBench is introduced by the same team but is an external measurement artifact with its own task-specific success criteria, not a citation used to justify a derivation. Overall, the paper's central results are measured, not derived, and the only circularity-adjacent issue is the MLE-R selection protocol.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central method depends on several hand-chosen hyperparameters and domain assumptions about what makes code important and which parts of logs matter. The new benchmark, GitTaskBench, is an author-created artifact without external validation, and the MLE-R modification uses RepoMaster's own repository search, which introduces assumptions about fair comparison.

free parameters (7)
  • feature weights w_i = w_i = 1 for all six features
    Module importance score in Eq. (2) uses equal weights; no learned or data-driven weighting is reported.
  • top-k core classes = k = 10
    Section 4.2 sets k=10 for the number of core classes included in initial context.
  • top module summaries = 20
    Section 4.2: concise summaries are generated for the top 20 modules by importance score.
  • per-interaction token limit L = 8000
    Section 4.2 sets the maximum token length per interaction to 8000 tokens.
  • document chunk size Lc = 1000
    Section 4.2: unstructured files are split into 1000-token chunks.
  • retained chunks nc = 4
    Section 4.2: the top 4 most relevant segments are retained during viewing.
  • PageRank damping factor alpha = 0.85
    Appendix F Algorithm 1 uses alpha=0.85 for personalized PageRank.
assumptions (5)
  • domain assumption AST parsing captures all essential code structure
    Section 3.2.1 relies on a single AST walk to harvest modules, classes, and functions; this misses dynamic or non-Python behavior.
  • domain assumption PageRank centrality and the six hand-chosen features identify truly important modules
    Section 3.2.2's importance scoring assumes these proxies select components that matter for task solving.
  • domain assumption Log openings and closings contain the diagnostic information
    Section 3.3.3 prunes logs to first and last segments, assuming command invocations and exception traces cluster there.
  • ad hoc to paper GitTaskBench tasks are feasible with the provided repositories
    Appendix A states students validated repos and supplemented resources, but the benchmark is new and unvalidated externally.
  • domain assumption The 22 MLE-R tasks and chosen repositories are representative of repository-reuse workloads
    Section 4.1 modifies MLE-Bench and selects repos via RepoMaster's own search; representativeness is not established.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task Solving." pith.science (2026). https://pith.science/paper/SZQJENZG

@misc{pith2026250521577,
  author       = {Pith},
  title        = {Pith review of: RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task Solving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SZQJENZG}},
  note         = {Machine review of arXiv:2505.21577}
}
read the original abstract

The ultimate goal of code agents is to solve complex tasks autonomously. Although large language models (LLMs) have made substantial progress in code generation, real-world tasks typically demand full-fledged code repositories rather than simple scripts. Building such repositories from scratch remains a major challenge. Fortunately, GitHub hosts a vast, evolving collection of open-source repositories, which developers frequently reuse as modular components for complex tasks. Yet, existing frameworks like OpenHands and SWE-Agent still struggle to effectively leverage these valuable resources. Relying solely on README files provides insufficient guidance, and deeper exploration reveals two core obstacles: overwhelming information and tangled dependencies of repositories, both constrained by the limited context windows of current LLMs. To tackle these issues, we propose RepoMaster, an autonomous agent framework designed to explore and reuse GitHub repositories for solving complex tasks. For efficient understanding, RepoMaster constructs function-call graphs, module-dependency graphs, and hierarchical code trees to identify essential components, providing only identified core elements to the LLMs rather than the entire repository. During autonomous execution, it progressively explores related components using our exploration tools and prunes information to optimize context usage. Evaluated on the adjusted MLE-bench, RepoMaster achieves a 110% relative boost in valid submissions over the strongest baseline OpenHands. On our newly released GitTaskBench, RepoMaster lifts the task-pass rate from 40.7% to 62.9% while reducing token usage by 95%. Our code and demonstration materials are publicly available at https://github.com/QuantaAlpha/RepoMaster.

Figures

Figures reproduced from arXiv: 2505.21577 by the authors.

Figure 1
Figure 1. Overview of RepoMaster, consisting of Repository Search, Hierarchical Repository Analysis [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of RepoMaster’s autonomous exploration–execution loop and an example [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. RepoMaster vs. OpenHands and SWE-Agent: A Comparative Case Study. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Performance comparison of different ablation configurations in RepoMaster. [PITH_FULL_IMAGE:figures/full_fig_p030_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AgentRadio: Passive Awareness for Long-Horizon Multi-Agent Collaboration

    cs.MA 2026-07 accept novelty 6.0 of 10

    Background mention-waiting lets four Claude Code agents share discoveries mid-execution and reach 62.1% on SWE-Atlas QnA versus 32.3% for one agent.

  2. UserTrace: User-Level Requirements Generation and Traceability Recovery from Software Project Repositories

    cs.SE 2025-09 conditional novelty 6.0 of 10

    UserTrace generates user-level requirements from code repositories and recovers live trace links from requirements to implementation, with evaluations suggesting gains over summarization and traceability baselines.

Reference graph

Works this paper leans on

70 extracted references · 28 canonical work pages · cited by 2 Pith papers

  1. [1]

    Tool learning with large language models: A survey

    Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. Tool learning with large language models: A survey. Frontiers of Computer Science, 19(8):198343, 2025

    2025

  2. [2]

    What are tools anyway? a survey from the language model perspective

    Zhiruo Wang, Zhoujun Cheng, Hao Zhu, Daniel Fried, and Graham Neubig. What are tools anyway? a survey from the language model perspective. arXiv preprint arXiv:2403.15452, 2024

    arXiv 2024

  3. [3]

    Trove: Inducing verifiable and efficient toolboxes for solving programmatic tasks

    Zhiruo Wang, Daniel Fried, and Graham Neubig. Trove: Inducing verifiable and efficient toolboxes for solving programmatic tasks. arXiv preprint arXiv:2401.12869, 2024

    arXiv 2024

  4. [4]

    Large language models as tool makers

    Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models as tool makers. arXiv preprint arXiv:2305.17126, 2023

    arXiv 2023

  5. [5]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023

    2023

  6. [6]

    Tree-of-code: A tree- structured exploring framework for end-to-end code generation and execution in complex task handling

    Ziyi Ni, Yifan Li, Ning Yang, Dou Shen, Pin Lv, and Daxiang Dong. Tree-of-code: A tree- structured exploring framework for end-to-end code generation and execution in complex task handling. arXiv preprint arXiv:2412.15305, 2024

    arXiv 2024

  7. [7]

    Llms with chain-of-thought are non-causal reasoners

    Guangsheng Bao, Hongbo Zhang, Linyi Yang, Cunxiang Wang, and Yue Zhang. Llms with chain-of-thought are non-causal reasoners. arXiv preprint arXiv:2402.16048, 2024

    arXiv 2024

  8. [8]

    Self-reflection in llm agents: Effects on problem-solving performance

    Matthew Renze and Erhan Guven. Self-reflection in llm agents: Effects on problem-solving performance. arXiv preprint arXiv:2405.06682, 2024

    arXiv 2024

Show all 70 references
  1. [9]

    Executable code actions elicit better llm agents

    Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better llm agents. In Forty-first International Conference on Machine Learning, 2024

    2024

  2. [10]

    Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges

    Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. arXiv preprint arXiv:2401.07339, 2024

    2024 arXiv

  3. [11]

    Paperbench: Evaluating ai’s ability to replicate ai research

    Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. Paperbench: Evaluating ai’s ability to replicate ai research. arXiv preprint arXiv:2504.01848, 2025

    2025 arXiv

  4. [12]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  5. [13]

    Competition-level code generation with alphacode

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624):1092–1097, 2022

    2022

  6. [14]

    Swe-agent: Agent-computer interfaces enable automated software engineering

    John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528–50652, 2024

    2024

  7. [15]

    Mlagentbench: Evaluating language agents on machine learning experimentation

    Qian Huang, Jian V ora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation. arXiv preprint arXiv:2310.03302, 2023

    2023 arXiv

  8. [16]

    Mle-bench: Evaluating machine learning agents on machine learning engineering

    Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learning agents on machine learning engineering. arXiv preprint arXiv:2410.07095, 2024

    arXiv

  9. [17]

    Gitagent: facilitating autonomous agent with github by tool extension

    Bohan Lyu, Xin Cong, Heyang Yu, Pan Yang, Yujia Qin, Yining Ye, Yaxi Lu, Zhong Zhang, Yukun Yan, Yankai Lin, et al. Gitagent: facilitating autonomous agent with github by tool extension. arXiv preprint arXiv:2312.17294, 2023. 11

    2023 arXiv

  10. [18]

    Ml-bench: Evaluating large language models and agents for machine learning tasks on repository-level code

    Xiangru Tang, Yuliang Liu, Zefan Cai, Yanjun Shao, Junjie Lu, Yichi Zhang, Zexuan Deng, Helan Hu, Kaikai An, Ruijun Huang, et al. Ml-bench: Evaluating large language models and agents for machine learning tasks on repository-level code. arXiv preprint arXiv:2311.09835, 2023

    2023 arXiv

  11. [19]

    Devbench: A comprehensive benchmark for software development

    Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, et al. Devbench: A comprehensive benchmark for software development. CoRR, 2024

    2024

  12. [20]

    Metagpt: Meta programming for multi-agent collaborative framework

    Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352, 3(4):6, 2023

    2023 arXiv

  13. [21]

    Autogen: Enabling next-gen llm applications via multi-agent conversation

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen llm applications via multi-agent conversation. arXiv preprint arXiv:2308.08155, 2023

    2023 arXiv

  14. [23]

    Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    2023 arXiv

  15. [24]

    Openhands: An open platform for ai software developers as generalist agents

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. In The Thirteenth International Conference on Learning Representa...

    2024

  16. [25]

    Aide: Ai-driven exploration in the space of code

    Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Ja- cenko, and Yuxiang Wu. Aide: Ai-driven exploration in the space of code. arXiv preprint arXiv:2502.13138, 2025

    2025 arXiv

  17. [26]

    https://anonymous.4open.science/r/ GitTaskBench-EE47/

    GitTaskBench: Anonymous github repository. https://anonymous.4open.science/r/ GitTaskBench-EE47/. Accessed: May 2025

    2025

  18. [27]

    Code llama: Open foundation models for code

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  19. [28]

    Starcoder: may the source be with you! CoRR, abs/2305.06161, 2023

    Raymond Li, Yushi Yamaguchi, Sinong Liu, Amir Gal, Matthew Ju, Farhad Mireshghallah, Andrew Wang, Jiayi Gao, Wanjun Wang, Chen Lei, and et al. Starcoder: may the source be with you! CoRR, abs/2305.06161, 2023

    2023 arXiv

  20. [29]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  21. [30]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  22. [31]

    Introducing claude

    Anthropic. Introducing claude. https://www.anthropic.com/index/ introducing-claude, 2023. Accessed: 2023-10-20

    2023

  23. [32]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  24. [33]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    2023 arXiv

  25. [34]

    Mixtral of experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. 12

    2024 arXiv

  26. [35]

    Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence

    Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931, 2024

    2024 arXiv

  27. [36]

    Repocoder: Repository-level code completion through iterative retrieval and generation

    Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. Repocoder: Repository-level code completion through iterative retrieval and generation. arXiv preprint arXiv:2303.12570, 2023

    2023 arXiv

  28. [37]

    Can it edit? evaluating the ability of large language models to follow code editing instructions

    Federico Cassano, Luisa Li, Akul Sethi, Noah Shinn, Abby Brennan-Jones, Jacob Ginesin, Edward Berman, George Chakhnashvili, Anton Lozhkov, Carolyn Jane Anderson, et al. Can it edit? evaluating the ability of large language models to follow code editing instructions. arXiv prep...

    2023 arXiv

  29. [38]

    Debugbench: Evaluating debugging capability of large language models

    Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Yinxu Pan, Yesai Wu, Haotian Hui, Weichuan Liu, Zhiyuan Liu, et al. Debugbench: Evaluating debugging capability of large language models. arXiv preprint arXiv:2401.04621, 2024

    2024 arXiv

  30. [39]

    Tool learning with foundation models

    Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Xuanhe Zhou, Yufei Huang, Chaojun Xiao, et al. Tool learning with foundation models. ACM Computing Surveys, 57(4):1–40, 2024

    2024

  31. [40]

    Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings

    Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings. Advances in neural information processing systems, 36:45870–45894, 2023

    2023

  32. [41]

    Where are large language models for code generation on github? arXiv preprint arXiv:2406.19544, 2024

    Xiao Yu, Lei Liu, Xing Hu, Jacky Wai Keung, Jin Liu, and Xin Xia. Where are large language models for code generation on github? arXiv preprint arXiv:2406.19544, 2024

    2024

  33. [42]

    Api-bank: A comprehensive benchmark for tool-augmented llms

    Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tool-augmented llms. arXiv preprint arXiv:2304.08244, 2023

    2023 arXiv

  34. [43]

    Tooleyes: fine-grained evaluation for tool learning capabilities of large language models in real-world scenarios

    Junjie Ye, Guanyu Li, Songyang Gao, Caishuang Huang, Yilong Wu, Sixian Li, Xiaoran Fan, Shihan Dou, Qi Zhang, Tao Gui, et al. Tooleyes: fine-grained evaluation for tool learning capabilities of large language models in real-world scenarios. arXiv preprint arXiv:2401.00741, 2024

    2024 arXiv

  35. [44]

    Toolllm: Facilitating large language models to master 16000+ real-world apis

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789, 2023

    2023 arXiv

  36. [45]

    Microservice APIs: Using Python, Flask, FastAPI, OpenAPI and More

    José Haro Peralta. Microservice APIs: Using Python, Flask, FastAPI, OpenAPI and More . Simon and Schuster, 2023

    2023

  37. [46]

    https://modelcontextprotocol.io/ introduction

    Model context protocol: Introduction. https://modelcontextprotocol.io/ introduction. Accessed: May 2025

    2025

  38. [47]

    Repoagent: An llm-powered open-source framework for repository-level code documentation generation

    Qinyu Luo, Yining Ye, Shihao Liang, Zhong Zhang, Yujia Qin, Yaxi Lu, Yesai Wu, Xin Cong, Yankai Lin, Yingli Zhang, et al. Repoagent: An llm-powered open-source framework for repository-level code documentation generation. arXiv preprint arXiv:2402.16667, 2024

    2024 arXiv

  39. [48]

    https://aider.chat/

    Aider: Ai pair programming in your terminal. https://aider.chat/. Accessed: May 2025

    2025

  40. [49]

    https://github.com/features/copilot

    Github copilot: Your ai pair programmer. https://github.com/features/copilot. Ac- cessed: May 2025

    2025

  41. [50]

    https://www.cursor.com/en

    Cursor: The ai code editor. https://www.cursor.com/en. Accessed: May 2025

    2025

  42. [51]

    A novel neural source code representation based on abstract syntax tree

    Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, Kaixuan Wang, and Xudong Liu. A novel neural source code representation based on abstract syntax tree. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pages 783–794, 2019

    2019

  43. [52]

    The pagerank citation ranking: Bringing order to the web

    Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford infolab, 1999. 13

    1999

  44. [53]

    Hello gpt-4o

    OpenAI. Hello gpt-4o. https://openai.com/index/hello-gpt-4o/ , 2024. Accessed: 2024-05-18

    2024

  45. [54]

    The claude 3 model family: Opus, sonnet, haiku

    AI Anthropic. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 1, 2024

    2024

  46. [55]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. 14 Appendix A. Description of GitTaskBench Although existing benchmarks focu...

    2024 arXiv

  47. [56]

    Code execution completes without errors

  48. [57]

    Output file(s) exist in the specified directory

  49. [58]

    Output file(s) follow the required naming convention

  50. [59]

    Output file(s) are readable image files # Success Criteria

  51. [60]

    Scratches are effectively removed from the input image

  52. [61]

    Image quality is maintained or improved

  53. [62]

    aptos2019-blindness-detection

    Original image content is preserved 15 B. Repository Search Pipeline This section presents the condensed logs generated during the repository retrieval process. Listing 2 document search queries, repository rankings, and selection rationales, thereby ensuring transparency and ...

    2019

  54. [63]

    [Performance]: No specific performance metrics mentioned

    nitesh585/APTOS-2019-Blindness-Detection [Solution Approach]: Utilizes machine learning models for disease detection using retina images. [Performance]: No specific performance metrics mentioned. [Reason] : Focuses on the competition task with a machine learning approach, but ...

    2019

  55. [64]

    [Performance]: No specific performance metrics mentioned

    PrasunDatta/APTOS-2019-Blindness-Detection [Solution Approach]: Aims to detect diabetic retinopathy using retina images, with a focus on automatic screening. [Performance]: No specific performance metrics mentioned. [Reason]: Provides a comprehensive approach but lacks detaile...

    2019

  56. [65]

    [Performance]: No specific performance metrics mentioned

    gdineshk6174/APTOS-2019-Blindness-Detection [Solution Approach]: Machine learning model to identify diabetic retinopathy automatically. [Performance]: No specific performance metrics mentioned. [Reason]: Focuses on automatic identification but lacks detailed performance data

    2019

  57. [66]

    16 [Performance]: Achieved a kappa metric of 0.904 on a test set

    filipmu/Kaggle-APTOS-2019-Blindness [Solution Approach]: Ensemble of deep learning classifiers (ResNet34, EfficientNetB6) trained on original and preprocessed images. 16 [Performance]: Achieved a kappa metric of 0.904 on a test set. [Reason]: Provides a detailed solution appro...

    2019

  58. [67]

    repositories

    tahsin314/40th_place_solution_aptos2019-blindness-detection [Solution Approach]: Information unavailable due to connection error. [Performance]: Information unavailable. [Reason]: Unable to retrieve information. /* Results */ { "repositories": [ { "rank": 1, "name": "filipmu/K...

    2019

  59. [68]

    Check input image existence

  60. [69]

    Set up environment variables

  61. [70]

    Run main script to process image

  62. [71]

    /DeScratch/Global/detection.py

    Verify output. --- System: Input file exists but is an image (cannot display directly). --- Assistant: Set PYTHONPATH and prepare environment. --- System: Environment setup completed. --- Assistant: Run main DeScratch processing script via subprocess. --- System: 23 exitcode: ...

    2013

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.