REVIEW 2 major objections 5 minor 1 cited by
Beyond pip install: Evaluating LLM Agents for the Automated Installation of Python Projects
T0 review · 2 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read An LLM agent can install 55% of studied Python repositories at least once in ten tries.
desk verdict A genuinely new task and a reusable 40-repo benchmark, honestly reported, but the 55% headline is only as strong as the self-admittedly loose 'at least one test passes' oracle. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is INSTALLAMATIC, a two-stage LLM agent. Stage one is an LLM-guided repository search that uses tool calls to list directories, read files, inspect section headers, and record any document the model judges install-relevant, mimicking a human looking for setup instructions. Stage two is a Dockerfile generation and repair loop: the agent summarizes the gathered documentation, writes a Dockerfile that installs dependencies and runs the target project's test suite, builds it inside a virtual machine, and if the build fails, diagnoses the error log and submits a repaired Dockerfile for up to two repair attempts. The success oracle is the test suite itself: an installation counts as successful if at least one test passes, which the paper acknowledges is an imprecise but automatable proxy. The benchmark dataset carries the evaluation: 40 Python repositories sampled from four popularity tiers by number of stars, each tagged with 17 installation-method tags and a ground-truth Dockerfile.
What would settle it
Take the same 40 repositories and re-run the agent using a stricter oracle—for instance, the full test suite passing, or the project's own documented verification command—and compare the per-repository success rates. If repositories that previously succeeded under the 'at least one test passes' rule largely fail under the stricter rule, the 55% headline is an artifact of the chosen oracle rather than a measure of real installation capability.
Extended reading notes
Core claim
The central discovery is that a current LLM-based agent can close the full loop of environment management—finding installation instructions, translating them into a working Dockerfile, and validating through test execution—on a substantial minority of real-world Python repositories. On the new benchmark, 21 of 40 repositories (55%) are installed at least once in ten runs, with the average installation rate across all attempts at 28.8%. A variant that is given perfect recall of the install-relevant documents installs 18 of 34 repositories (34.7% average), isolating the agent's Dockerfile-writing ability from its documentation search ability. The paper also finds that installations requiring extra complexity, such as Poetry with all extras or pytest with special arguments, are almost never successful when those extra steps are undocumented, and that a repair loop adds substantial value: 73.9% of successful installations occur after at least one repair attempt. These results support the paper's position that automatic installation is a viable repository-level task for LLM agents, with documentation quality and repair capability as the dominant success factors.
Load-bearing premise
The whole evaluation rests on the assumption that a Dockerfile build that runs the project's test suite and passes at least one test is a valid sign of successful installation; the paper itself notes that partial test suites can pass without all dependencies being present, and full-suite failures may be unrelated to installation, such as missing API keys.
Editorial extensions
If this is right
- If the 55% per-repository success rate reproduces, LLM agents can already act as practical installation assistants for a substantial minority of actively maintained Python projects, reducing human setup time.
- The observed positive correlations between documentation visibility, informativity, and installation success identify clear targets for maintainers; testing whether adding code examples to install docs improves automated installation rates is a direct next step.
- The finding that most successful installations require at least one repair attempt implies that future environment-management agents should treat repair as a mandatory phase, not an optional enhancement.
- The benchmark of 40 repositories with ground-truth Dockerfiles provides a reusable yardstick for comparing future installation agents and for studying the interaction between documentation quality and agent performance.
Reading between the lines
- A natural extension is to replace the weak oracle ('at least one test passes') with a layered verification, such as importing the package in a fresh interpreter and running a smoke test, which would separate true installation success from partial test-suite luck.
- The token-cost analysis implies that a hybrid search, using a lightweight retriever to shortlist candidate files and then letting the LLM read only those, could cut inference cost by an order of magnitude while preserving most of the recall, making the agent viable outside a research budget.
- The 30-minute build timeout likely suppresses installation rates for repositories with large test suites; an adaptive timeout or test-selection strategy would likely raise measured success on those projects.
- The authors' recommendation to write code examples in installation documentation is testable as an intervention: alter a repository's README to include a worked install-and-test snippet and measure whether the agent's success rate jumps relative to the original version.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces a new benchmark and an LLM-based agent for the task of automatically installing a Python repository from its own documentation. The benchmark consists of 40 open-source Python projects with manually constructed ground-truth Dockerfiles and coarse-grained installation/test tags. The agent, Installamatic, performs an LLM-guided search to collect install-relevant documentation, then writes a Dockerfile that installs dependencies and runs the test suite; failures trigger up to two repair attempts using Docker build logs. The evaluation runs the agent 10 times per repository and uses 'at least one test passes' as the success oracle. The main reported result is that 21 of 40 repositories (55%) are installed at least once, with a 28.8% average installation rate; a 'perfect recall' variant that is given all relevant documents from the start achieves a 34.7% average installation rate over 34 repositories. The paper further analyzes correlations between documentation visibility/informativity and installation success, and discusses failure causes and practical challenges.
Significance. The paper addresses a novel and practically relevant task—automated environment setup—that is underexplored relative to LLM-based code generation and repair. The benchmark with per-repository ground-truth Dockerfiles and tags is a potentially reusable resource, and the public artifact includes prompts, logs, and Docker builds, which is a genuine reproducibility strength. The agent architecture (LLM-guided search plus a Docker build/repair loop in a sandboxed VM) is a sensible design that others can build on. The paper is also candid in enumerating limitations, especially the oracle problem. However, the headline capability claim and the correlational analyses need additional validation before the quantitative results can be accepted; the current evidence is suggestive rather than conclusive.
major comments (2)
- [Section III-C and Section V-A1] The success oracle defined in Section III-C counts a run as successful if the test suite executes and at least one test passes; the paper itself calls this 'imprecise' and identifies a concrete false-positive mechanism in Section V-C4 (modules with different requirements can pass a partial test suite while dependencies are missing). Because the repair loop (Section III-C) is allowed to modify the Dockerfile until the oracle is satisfied, the agent is effectively optimized against this weak oracle: it can keep changing install/test commands until some minimal test succeeds, without ever installing the full dependency set. The 'at least one out of ten runs' criterion amplifies the problem: for a truly uninstallable repository, a per-run false-positive probability of 0.1 gives a 65% chance of being counted as installed at least once (1−0.9^10), and a probability of 0.2 gives 89%. The paper does not bound the false-positive rate, so the headline result of 21/40 (55%) repositories installed at least once is not yet established. Please validate the successful runs by comparing the generated Dockerfiles and installed environments against the ground-truth Dockerfiles (or by running a stricter oracle, such as requiring the full test suite to pass or all ground-truth installation steps to be present), and re-report the installation rates under that validation. The public artifact makes this analysis feasible.
- [Section V-B, Figs. 5a-5c and 6b] The correlations between documentation quality metrics and installation rate are reported as single coefficients without confidence intervals, p-values, or a statement of the correlation type. For n=40, the values r=0.294 (visibility-installation, Fig. 5c) and r=0.309 (recall-installation, Fig. 5a) are not statistically significant at the 0.05 level (two-tailed approximate p≈0.065 and p≈0.052, respectively). The text nonetheless concludes that 'the structure of a repository's documentation does have an effect on the ability of our agent to install the repository' (Section V-B). Please report significance tests or bootstrap confidence intervals for all correlations, state whether Pearson or Spearman coefficients are used, and soften or qualify the causal language accordingly.
minor comments (5)
- [Introduction] The citation 'Dagenais et al. [ ?]' is a dangling placeholder; the reference list does not contain a Dagenais et al. entry. Either add the full reference or remove the sentence.
- [Figure 4] The purple bars labeled 'overlap' are not defined in the caption or the text; please explain what 'overlap' represents in the comparison between the standard and perfect-recall configurations.
- [Section V-A1] The sentence about 'the seven rightmost repositories' is unclear because Figure 4 does not label the horizontal axis with 'rightmost' in the text; please name the seven repositories explicitly.
- [Section IV-B] The definitions of visibility and informativity use the notation 'dockerfile T documentation' without explanation; define the intended operation (likely set intersection of lines) in prose.
- [Throughout] There are several typographical issues, e.g., 'deleloper' in the Introduction, 'artiact' in Section IV-C, and 'beahaviour' in Section V-A1; a thorough proofread is needed.
Circularity Check
No significant circularity: the central installation-rate result is an empirical evaluation against an externally constructed ground-truth benchmark, not a quantity that is fit or defined into existence.
full rationale
The paper's central claim, a 55% at-least-once installation rate over 40 repositories, is produced by running the Installamatic agent and checking its generated Dockerfiles against a success oracle (tests run and at least one test passes, Section III-C). This oracle is external to the agent and manually grounded: each repository has a human-authored exemplar Dockerfile and a manually curated list of install-relevant documents (Section II). No parameter is fitted to the benchmark to produce the headline, and no equation is defined in terms of the result it is said to predict. The informativity metric uses lines from the ground-truth Dockerfile, but only as a descriptive correlation with installation rate (Section IV-B and Figure 6b), not as a fitted input to the agent. The perfect-recall configuration supplies manually confirmed documents, which is an experimental control rather than a circular reuse of the output. The paper does cite prior work by overlapping authors ([13], [21]) for the design choice of LLM-guided search, but that design choice is not the load-bearing derivation of the installation-rate claim, and the claim is independently evaluated against external Dockerfile builds. The paper's own caveat that the at-least-one-test-pass oracle is imprecise (Section III-C and Section V-C4) is a validity concern about false positives, not evidence of circularity, because the oracle is not constructed from the agent's outputs. The results are therefore self-contained relative to the manually built benchmark, and no circular step is present.
Assumptions & free parameters
free parameters (4)
- max_repair_attempts =
2
- build_timeout =
30 minutes
- success_threshold =
at least 1 test passes
- number_of_runs =
10
assumptions (5)
- domain assumption A test suite that runs and has at least one passing test is a valid oracle for successful installation.
- domain assumption Docker containers running in a virtual machine provide a faithful and safe environment for installing arbitrary Python repositories.
- domain assumption Manually constructed ground-truth Dockerfiles and install-relevant document labels are correct.
- domain assumption GPT-4o-mini, with 10 runs per repository, is representative of LLM agents for this task.
- domain assumption The 40 sampled repositories are representative of open-source Python projects at large.
Cite this review
Pith. "Pith review of Beyond pip install: Evaluating LLM Agents for the Automated Installation of Python Projects." pith.science (2026). https://pith.science/paper/V6VULW5P
@misc{pith2026241206294,
author = {Pith},
title = {Pith review of: Beyond pip install: Evaluating LLM Agents for the Automated Installation of Python Projects},
year = {2026},
howpublished = {\url{https://pith.science/paper/V6VULW5P}},
note = {Machine review of arXiv:2412.06294}
}
read the original abstract
Many works have recently proposed the use of Large Language Model (LLM) based agents for performing `repository level' tasks, loosely defined as a set of tasks whose scopes are greater than a single file. This has led to speculation that the orchestration of these repository-level tasks could lead to software engineering agents capable of performing almost independently of human intervention. However, of the suite of tasks that would need to be performed by this autonomous software engineering agent, we argue that one important task is missing, which is to fulfil project level dependency by installing other repositories. To investigate the feasibility of this repository level installation task, we introduce a benchmark of of repository installation tasks curated from 40 open source Python projects, which includes a ground truth installation process for each target repository. Further, we propose Installamatic, an agent which aims to perform and verify the installation of a given repository by searching for relevant instructions from documentation in the repository. Empirical experiments reveal that that 55% of the studied repositories can be automatically installed by our agent at least one out of ten times. Through further analysis, we identify the common causes for our agent's inability to install a repository, discuss the challenges faced in the design and implementation of such an agent and consider the implications that such an agent could have for developers.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 1 Pith paper
-
Beyond Accuracy: Behavioral Dynamics of Agentic Multi-Hunk Repair
On 372 multi-hunk bugs, coding agents repair between 26% and 93% of defects, accuracy falls as edits become more divergent and dispersed, and failed repairs consume up to 343% more tokens.
Reference graph
Works this paper leans on
-
[1]
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones et al. , “At- tention is all you need,” in Advances in Neural Information Processing Systems, I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus et al., Eds., vol. 30. Curran Associates, Inc., 2017
work page 2017
-
[2]
Emergent abilities of large language models,
J. Wei, Y . Tay, R. Bommasani, C. Raffel, B. Zoph et al. , “Emergent abilities of large language models,” Transactions on Machine Learning Research , 2022, survey Certification. [Online]. Available: https://openreview.net/forum?id=yzkSU5zdwD
work page 2022
-
[3]
Lan- guage models are few-shot learners,
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan et al., “Lan- guage models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020
work page 1901
-
[4]
Evaluating large language models trained on code,
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto et al. , “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021
arXiv 2021
-
[5]
Large language models for software engineering: Survey and open problems,
A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta et al. , “Large language models for software engineering: Survey and open problems,” in Proceedings of the 45th IEEE/ACM International Con- ference on Software Engineering: Future of Software Engineering , ser. ICSE-FoSE, May 2023, pp. 31–53
work page 2023
-
[6]
Fuzz4all: Universal fuzzing with large language models,
C. S. Xia, M. Paltenghi, J. Le Tian, M. Pradel, and L. Zhang, “Fuzz4all: Universal fuzzing with large language models,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ser. ICSE ’24. New York, NY , USA: Association for Computing Machinery,
-
[7]
Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models,
C. Lemieux, J. P. Inala, S. K. Lahiri, and S. Sen, “Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), 2023, pp. 919–931
work page 2023
-
[8]
Less training, more repairing please: revisiting automated program repair via zero-shot learning,
C. S. Xia and L. Zhang, “Less training, more repairing please: revisiting automated program repair via zero-shot learning,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022, pp. 959– 971
2022
Show all 34 references
-
[9]
A comprehensive overview of large language models,
H. Naveed, A. U. Khan, S. Qiu, M. Saqib, S. Anwar et al. , “A comprehensive overview of large language models,” arXiv preprint arXiv:2307.06435, 2023
2023 arXiv
-
[10]
Chain-of- thought prompting elicits reasoning in large language models,
J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia et al., “Chain-of- thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24 824–24 837, 2022
2022
-
[11]
React: Synergizing reasoning and acting in language models,
S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran et al. , “React: Synergizing reasoning and acting in language models,” in Proceedings of the Inter- national Conference on Learning Representation, ser. ICLR 2022, 2022
2022
-
[12]
Self-consistency improves chain of thought reasoning in language models,
X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi et al., “Self-consistency improves chain of thought reasoning in language models,” CoRR, vol. abs/2203.11171, 2023
2023 arXiv
-
[13]
A quantitative and qualitative evaluation of llm-based explainable fault localization,
S. Kang, G. An, and S. Yoo, “A quantitative and qualitative evaluation of llm-based explainable fault localization,” Proceedings of the ACM on Software Engineering, vol. 1, no. FSE, pp. 1424–1446, 2024
2024
-
[14]
Intent-driven mobile gui testing with autonomous large language model agents,
J. Yoon, R. Feldt, and S. Yoo, “Intent-driven mobile gui testing with autonomous large language model agents,” in Proceedings of the 16th IEEE International Conference on Software Testing, Verification and Validation, ser. ICST 2024, 2024
2024
-
[15]
Magis: Llm-based multi-agent framework for github issue resolution,
W. Tao, Y . Zhou, W. Zhang, and Y . Cheng, “Magis: Llm-based multi-agent framework for github issue resolution,” arXiv preprint arXiv:2403.17927, 2024
2024 arXiv
-
[16]
Code- plan: Repository-level coding using llms and planning,
R. Bairi, A. Sonwane, A. Kanade, A. Iyer, S. Parthasarathy et al., “Code- plan: Repository-level coding using llms and planning,” Proceedings of the ACM on Software Engineering , vol. 1, no. FSE, pp. 675–698, 2024
2024
-
[17]
Teaching code llms to use autocompletion tools in repository-level code generation,
C. Wang, J. Zhang, Y . Feng, T. Li, W. Sun et al., “Teaching code llms to use autocompletion tools in repository-level code generation,” arXiv preprint arXiv:2401.06391, 2024
2024 arXiv
-
[18]
Software documentation: the practitioners’ perspective,
E. Aghajani, C. Nagy, M. Linares-V ´asquez, L. Moreno, G. Bavota et al., “Software documentation: the practitioners’ perspective,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineer- ing, 2020, pp. 590–601
2020
-
[19]
Language models are few-shot learners,
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan et al. , “Language models are few-shot learners,” 2020. [Online]. Available: https://arxiv.org/abs/2005.14165
2020 arXiv
-
[20]
Reflexion: Language agents with verbal reinforcement learning,
N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan et al. , “Reflexion: Language agents with verbal reinforcement learning,” 2023. [Online]. Available: https://arxiv.org/abs/2303.11366
2023 arXiv
-
[21]
Explainable automated debugging via large language model-driven scientific debugging,
S. Kang, B. Chen, S. Yoo, and J.-G. Lou, “Explainable automated debugging via large language model-driven scientific debugging,” 2023. [Online]. Available: https://arxiv.org/abs/2304.02195
2023 arXiv
-
[22]
Amati, BM25
G. Amati, BM25. Boston, MA: Springer US, 2009, pp. 257–260. [Online]. Available: https://doi.org/10.1007/978-0-387-39940-9 921
2009 doi
-
[23]
Neural models for information retrieval,
B. Mitra and N. Craswell, “Neural models for information retrieval,” arXiv preprint arXiv:1705.01509 , 2017
2017 arXiv
-
[24]
Repairagent: An autonomous, llm-based agent for program repair,
I. Bouzenia, P. Devanbu, and M. Pradel, “Repairagent: An autonomous, llm-based agent for program repair,” 2024. [Online]. Available: https://arxiv.org/abs/2403.17134
2024 arXiv
-
[25]
Lost in the middle: How language models use long contexts,
N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua et al., “Lost in the middle: How language models use long contexts,” Transactions of the Association for Computational Linguistics , vol. 12, pp. 157–173, 2024
2024
-
[26]
Can gpt-4 replicate empirical software engineering research?
J. T. Liang, C. Badea, C. Bird, R. DeLine, D. Ford et al. , “Can gpt-4 replicate empirical software engineering research?” 2024. [Online]. Available: https://arxiv.org/abs/2310.01727
2024 arXiv
-
[27]
Long- context llms struggle with long in-context learning,
T. Li, G. Zhang, Q. D. Do, X. Yue, and W. Chen, “Long- context llms struggle with long in-context learning,” arXiv preprint arXiv:2404.02060, 2024
2024 arXiv
-
[28]
Thread of thought unraveling chaotic contexts,
Y . Zhou, X. Geng, T. Shen, C. Tao, G. Long et al., “Thread of thought unraveling chaotic contexts,” arXiv preprint arXiv:2311.08734 , 2023
2023 arXiv
-
[29]
Opendevin: An open platform for ai software developers as generalist agents,
X. Wang, B. Li, Y . Song, F. F. Xu, X. Tang et al., “Opendevin: An open platform for ai software developers as generalist agents,” arXiv preprint arXiv:2407.16741, 2024
2024 arXiv
-
[30]
Swe-bench: Can language models resolve real-world github issues?
C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei et al. , “Swe-bench: Can language models resolve real-world github issues?” arXiv preprint arXiv:2310.06770, 2023
2023 arXiv
-
[31]
Automated extraction of research software installation instructions from readme files: An initial analysis,
C. Utrilla Guerrero, O. Corcho, and D. Garijo, “Automated extraction of research software installation instructions from readme files: An initial analysis,” in International Workshop on Natural Scientific Language Processing and Research Knowledge Graphs , 2024, pp. 114–133
2024
-
[32]
Large language models for software engineering: A systematic literature review,
X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang et al. , “Large language models for software engineering: A systematic literature review,” arXiv preprint arXiv:2308.10620, 2023
2023 arXiv
-
[33]
Automatic detection of five api documentation smells: Practitioners’ perspectives,
J. Y . Khan, M. T. I. Khondaker, G. Uddin, and A. Iqbal, “Automatic detection of five api documentation smells: Practitioners’ perspectives,” in 2021 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) . IEEE, 2021, pp. 318–329
2021
-
[2024]
Available: https://doi.org/10.1145/3597503.3639121
[Online]. Available: https://doi.org/10.1145/3597503.3639121
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.