{"id":"67bf2183-b397-4502-8e44-616cc2398390","arxiv_id":"2412.06294","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An LLM agent called Installamatic installs and verifies 21 of 40 Python repositories at least once, with a 28.8% average per-attempt success rate.","lead":"This paper tests whether an AI agent can install open-source Python projects by reading each project's own documentation and writing a Dockerfile, succeeding at least once in 21 of 40 projects. It matters because environment setup is a common developer chore that LLM agents have mostly ignored, and the public benchmark gives future agents a concrete task to measure.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 55% headline rests on an 'at least one test passes' oracle that the paper itself calls imprecise; without a stricter validation, false-positive Dockerfiles could inflate the count of installable repos.","rationale":"Reader's weakest assumption is exactly the oracle. I agree. The paper's own Section III-C and V-C4 concede the oracle is imprecise and can produce false positives, and the 55% headline is a count over 10 stochastic runs, so even a modest false-positive rate can flip several repos across the 'at least once' threshold. This is a construct-validity threat to the central capability claim, not an internal inconsistency. The artifact and ground-truth Dockerfiles make the proposed check feasible. I do not think this changes the verdict: the benchmark, the agent design, and the qualitative failure analysis remain valuable, and the paper already flags the oracle as a limitation. The correct disposition is still conditional acceptance pending a stricter validation of the success oracle. I therefore leave the reader's verdict unchanged.","tokens_in":15224,"tokens_out":5852,"duration_ms":63434,"concrete_test":"Using the public artifact, collect every Dockerfile from the 21 repositories that had at least one successful run. Rebuild each in a clean container and apply a stricter oracle: (1) import the repository's top-level package(s) and run a canonical smoke command (e.g., CLI --help) to verify the package itself is present; (2) run the ground-truth exemplar Dockerfile's test command (with the same 30-minute cap) and record whether the agent's passing tests are a strict subset that excludes tests requiring missing optional dependencies. Count how many of the 21 repos fail the stricter oracle. If the number is >2, the 55% figure overstates installability; if it is 0–1, the oracle concern is bounded and the headline stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim—55% of repositories installed at least once—is defined by an oracle that counts a run as successful if the Dockerfile build runs some tests and at least one test passes (Section III-C). The paper explicitly calls this 'imprecise' and gives a concrete false-positive mechanism in Section V-C4: when different modules of a project have different requirements, a partial test suite can pass while dependencies are missing. Since the repair loop is allowed to modify the Dockerfile until one test passes, the agent is effectively optimized against this weak oracle: it can keep changing install/test commands until some minimal test succeeds, without ever installing the full dependency set. Ten runs per repository make the 'at least one out of ten' count sensitive: a per-run false-positive probability of even 0.1–0.2 would move several of the 40 repositories across the threshold. The paper provides ground-truth exemplar Dockerfiles and an artifact, so the direction and size of this bias are checkable; without that check the central capability claim is not yet established as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces a new benchmark and an LLM-based agent for the task of automatically installing a Python repository from its own documentation. The benchmark consists of 40 open-source Python projects with manually constructed ground-truth Dockerfiles and coarse-grained installation/test tags. The agent, Installamatic, performs an LLM-guided search to collect install-relevant documentation, then writes a Dockerfile that installs dependencies and runs the test suite; failures trigger up to two repair attempts using Docker build logs. The evaluation runs the agent 10 times per repository and uses 'at least one test passes' as the success oracle. The main reported result is that 21 of 40 repositories (55%) are installed at least once, with a 28.8% average installation rate; a 'perfect recall' variant that is given all relevant documents from the start achieves a 34.7% average installation rate over 34 repositories. The paper further analyzes correlations between documentation visibility/informativity and installation success, and discusses failure causes and practical challenges.","tokens_in":15416,"tokens_out":8372,"duration_ms":81274,"significance":"The paper addresses a novel and practically relevant task—automated environment setup—that is underexplored relative to LLM-based code generation and repair. The benchmark with per-repository ground-truth Dockerfiles and tags is a potentially reusable resource, and the public artifact includes prompts, logs, and Docker builds, which is a genuine reproducibility strength. The agent architecture (LLM-guided search plus a Docker build/repair loop in a sandboxed VM) is a sensible design that others can build on. The paper is also candid in enumerating limitations, especially the oracle problem. However, the headline capability claim and the correlational analyses need additional validation before the quantitative results can be accepted; the current evidence is suggestive rather than conclusive.","major_comments":[{"comment":"The success oracle defined in Section III-C counts a run as successful if the test suite executes and at least one test passes; the paper itself calls this 'imprecise' and identifies a concrete false-positive mechanism in Section V-C4 (modules with different requirements can pass a partial test suite while dependencies are missing). Because the repair loop (Section III-C) is allowed to modify the Dockerfile until the oracle is satisfied, the agent is effectively optimized against this weak oracle: it can keep changing install/test commands until some minimal test succeeds, without ever installing the full dependency set. The 'at least one out of ten runs' criterion amplifies the problem: for a truly uninstallable repository, a per-run false-positive probability of 0.1 gives a 65% chance of being counted as installed at least once (1−0.9^10), and a probability of 0.2 gives 89%. The paper does not bound the false-positive rate, so the headline result of 21/40 (55%) repositories installed at least once is not yet established. Please validate the successful runs by comparing the generated Dockerfiles and installed environments against the ground-truth Dockerfiles (or by running a stricter oracle, such as requiring the full test suite to pass or all ground-truth installation steps to be present), and re-report the installation rates under that validation. The public artifact makes this analysis feasible.","section":"Section III-C and Section V-A1"},{"comment":"The correlations between documentation quality metrics and installation rate are reported as single coefficients without confidence intervals, p-values, or a statement of the correlation type. For n=40, the values r=0.294 (visibility-installation, Fig. 5c) and r=0.309 (recall-installation, Fig. 5a) are not statistically significant at the 0.05 level (two-tailed approximate p≈0.065 and p≈0.052, respectively). The text nonetheless concludes that 'the structure of a repository's documentation does have an effect on the ability of our agent to install the repository' (Section V-B). Please report significance tests or bootstrap confidence intervals for all correlations, state whether Pearson or Spearman coefficients are used, and soften or qualify the causal language accordingly.","section":"Section V-B, Figs. 5a-5c and 6b"}],"minor_comments":[{"comment":"The citation 'Dagenais et al. [ ?]' is a dangling placeholder; the reference list does not contain a Dagenais et al. entry. Either add the full reference or remove the sentence.","section":"Introduction"},{"comment":"The purple bars labeled 'overlap' are not defined in the caption or the text; please explain what 'overlap' represents in the comparison between the standard and perfect-recall configurations.","section":"Figure 4"},{"comment":"The sentence about 'the seven rightmost repositories' is unclear because Figure 4 does not label the horizontal axis with 'rightmost' in the text; please name the seven repositories explicitly.","section":"Section V-A1"},{"comment":"The definitions of visibility and informativity use the notation 'dockerfile T documentation' without explanation; define the intended operation (likely set intersection of lines) in prose.","section":"Section IV-B"},{"comment":"There are several typographical issues, e.g., 'deleloper' in the Introduction, 'artiact' in Section IV-C, and 'beahaviour' in Section V-A1; a thorough proofread is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope and the topic is timely. The main risk is that the success oracle is too weak, which directly affects the headline 55% claim. Because the artifact is public and includes ground-truth Dockerfiles, it should be feasible for the authors to validate successful runs and re-report the rates; I would encourage the editor to request that as part of the revision. I do not see signs of misconduct; the paper openly acknowledges the oracle limitation. The missing Dagenais et al. reference is a simple fix."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this paper gives the LLM4SE community a genuinely new task and a reusable 40-repo benchmark, and it reports its results honestly. But the headline '55% of repositories installed at least once' is only as strong as their oracle, which they themselves call imprecise, so treat the exact number as provisional.\n\nWhat's actually new: prior work like PlanStep extracts install instructions from READMEs but doesn't execute them; SWE-bench and friends target code edits. This paper defines the installation task, curates 40 popular Python repos with manually verified ground-truth Dockerfiles and a tag taxonomy, and ships a public artifact. That is real, reproducible work. The agent itself is a fairly standard composition of LLM search, Dockerfile generation, and a repair loop, but it does run actual builds in VMs ten times per repo. The failure analysis (e.g., missing '--all-extras', pytest-extra flags, 30-minute timeout) is useful and credible.\n\nThe soft spots are real but mostly acknowledged. The per-attempt success rate uses 'at least one test passes' as the success oracle. The paper says in Section III-C this can yield false positives when modules have different requirements. The repair loop is allowed to tweak the Dockerfile until one test passes, so it can overfit to a minimal test rather than installing the full dependency set. With only ten runs per repo, even a 0.1–0.2 per-run false-positive rate could push several repos over the 'at least once' threshold. The authors also flag the 30-minute timeout as a source of false negatives, so the net bias isn't obvious, but the 55% number should be read as 'passes our weak oracle,' not 'fully installs.' The correlations with visibility/informativity are reported without confidence intervals or significance tests; they're suggestive, not confirmatory. The sample is small and restricted to repos with test suites, which they acknowledge. None of these sink the paper; the benchmark is still useful and the oracle issue can be checked against their ground-truth Dockerfiles.\n\nBottom line: this is a solid, honest empirical study for the agent/SE community. The benchmark and the task definition are the main contributions. It deserves serious peer review; the main request should be a stricter validation of the oracle and error bars on the correlations.","headline":"A genuinely new task and a reusable 40-repo benchmark, honestly reported, but the 55% headline is only as strong as the self-admittedly loose 'at least one test passes' oracle.","tokens_in":15954,"tokens_out":2739,"would_cite":true,"duration_ms":26943,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM agent can install 55% of studied Python repositories at least once in ten tries.","keywords":["LLM agents","automated installation","Python packaging","Dockerfile generation","repository-level tasks","software documentation","test-suite oracle","benchmark"],"falsifier":"Take the same 40 repositories and re-run the agent using a stricter oracle—for instance, the full test suite passing, or the project's own documented verification command—and compare the per-repository success rates. If repositories that previously succeeded under the 'at least one test passes' rule largely fail under the stricter rule, the 55% headline is an artifact of the chosen oracle rather than a measure of real installation capability.","tokens_in":15004,"feed_emoji":"🛠️","tokens_out":12290,"duration_ms":99248,"temperature":0.7,"pith_summary":"This paper introduces a new task for LLM-based software engineering agents: installing an arbitrary open-source Python repository from scratch, rather than editing its code. It contributes a benchmark of 40 actively maintained Python projects, each with a manually verified ground-truth Dockerfile and installation-relevant documentation, and an agent, INSTALLAMATIC, that searches the repository for install instructions, writes a Dockerfile, and validates the installation by running the project's test suite in a virtual machine. In ten repeated attempts per repository, the agent installs 21 of the 40 projects at least once, a 55% per-repository success rate, and 28.8% of all attempts succeed. The paper argues that this demonstrates a meaningful first step toward autonomous dependency management and identifies documentation structure and the need for a repair step as the main levers on success.","feed_headline":"LLM agent installs 21 of 40 Python repos in tests","feed_subtitle":"It reads install docs, writes a Dockerfile, and passes tests on 21 of 40 projects.","key_machinery":"The central object is INSTALLAMATIC, a two-stage LLM agent. Stage one is an LLM-guided repository search that uses tool calls to list directories, read files, inspect section headers, and record any document the model judges install-relevant, mimicking a human looking for setup instructions. Stage two is a Dockerfile generation and repair loop: the agent summarizes the gathered documentation, writes a Dockerfile that installs dependencies and runs the target project's test suite, builds it inside a virtual machine, and if the build fails, diagnoses the error log and submits a repaired Dockerfile for up to two repair attempts. The success oracle is the test suite itself: an installation counts as successful if at least one test passes, which the paper acknowledges is an imprecise but automatable proxy. The benchmark dataset carries the evaluation: 40 Python repositories sampled from four popularity tiers by number of stars, each tagged with 17 installation-method tags and a ground-truth Dockerfile.","core_discovery":"The central discovery is that a current LLM-based agent can close the full loop of environment management—finding installation instructions, translating them into a working Dockerfile, and validating through test execution—on a substantial minority of real-world Python repositories. On the new benchmark, 21 of 40 repositories (55%) are installed at least once in ten runs, with the average installation rate across all attempts at 28.8%. A variant that is given perfect recall of the install-relevant documents installs 18 of 34 repositories (34.7% average), isolating the agent's Dockerfile-writing ability from its documentation search ability. The paper also finds that installations requiring extra complexity, such as Poetry with all extras or pytest with special arguments, are almost never successful when those extra steps are undocumented, and that a repair loop adds substantial value: 73.9% of successful installations occur after at least one repair attempt. These results support the paper's position that automatic installation is a viable repository-level task for LLM agents, with documentation quality and repair capability as the dominant success factors.","pith_inferences":["A natural extension is to replace the weak oracle ('at least one test passes') with a layered verification, such as importing the package in a fresh interpreter and running a smoke test, which would separate true installation success from partial test-suite luck.","The token-cost analysis implies that a hybrid search, using a lightweight retriever to shortlist candidate files and then letting the LLM read only those, could cut inference cost by an order of magnitude while preserving most of the recall, making the agent viable outside a research budget.","The 30-minute build timeout likely suppresses installation rates for repositories with large test suites; an adaptive timeout or test-selection strategy would likely raise measured success on those projects.","The authors' recommendation to write code examples in installation documentation is testable as an intervention: alter a repository's README to include a worked install-and-test snippet and measure whether the agent's success rate jumps relative to the original version."],"forward_implications":["If the 55% per-repository success rate reproduces, LLM agents can already act as practical installation assistants for a substantial minority of actively maintained Python projects, reducing human setup time.","The observed positive correlations between documentation visibility, informativity, and installation success identify clear targets for maintainers; testing whether adding code examples to install docs improves automated installation rates is a direct next step.","The finding that most successful installations require at least one repair attempt implies that future environment-management agents should treat repair as a mandatory phase, not an optional enhancement.","The benchmark of 40 repositories with ground-truth Dockerfiles provides a reusable yardstick for comparing future installation agents and for studying the interaction between documentation quality and agent performance."],"supporting_citations":[{"why":"Provides the tool-use agent pattern that the repository search loop implements.","marker":"[11]"},{"why":"Supplies the chain-of-thought style two-step prompting used to make the agent plan then act during search.","marker":"[10]"},{"why":"Defines repository-level coding as edits satisfying an oracle, the framing the paper extends to environment management.","marker":"[16]"},{"why":"Establishes the dominant repository-level benchmark against which this paper positions its new installation task.","marker":"[30]"},{"why":"Closest prior work, extracting installation instructions from READMEs without executing them, marking the gap this paper fills.","marker":"[31]"}],"fun_headline_variants":["LLM agent installs 55% of Python repos at least once","Installamatic agent installs Python repos by reading docs","Repair loop helps LLM agent install Python repos","Agent reads docs, writes Dockerfile, installs Python repos"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole evaluation rests on the assumption that a Dockerfile build that runs the project's test suite and passes at least one test is a valid sign of successful installation; the paper itself notes that partial test suites can pass without all dependencies being present, and full-suite failures may be unrelated to installation, such as missing API keys.","fun_headline_variants_meta":{"raw":{"variants":["LLM agent installs 55% of Python repos at least once","Installamatic agent installs Python repos by reading docs","Repair loop helps LLM agent install Python repos","Agent reads docs, writes Dockerfile, installs Python repos"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000694,"raw_usage":{"total_tokens":3163,"prompt_tokens":990,"completion_tokens":2173,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":2103}},"tokens_in":606,"tokens_out":2173,"duration_ms":15859,"temperature":1.0,"reasoning_tokens":2103,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:48:25.842819+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same 40 repositories and re-run the agent using a stricter oracle—for instance, the full test suite passing, or the project's own documented verification command—and compare the per-repository success rates. If repositories that previously succeeded under the 'at least one test passes' rule largely fail under the stricter rule, the 55% headline is an artifact of the chosen oracle rather than a measure of real installation capability.","supporting_citations":[{"cited_title":"React: Synergizing reasoning and acting in language models,","cited_arxiv_id":null,"evidence_quote":"Provides the tool-use agent pattern that the repository search loop implements."},{"cited_title":"Chain-of- thought prompting elicits reasoning in large language models,","cited_arxiv_id":null,"evidence_quote":"Supplies the chain-of-thought style two-step prompting used to make the agent plan then act during search."},{"cited_title":"Code- plan: Repository-level coding using llms and planning,","cited_arxiv_id":null,"evidence_quote":"Defines repository-level coding as edits satisfying an oracle, the framing the paper extends to environment management."},{"cited_title":"Automated extraction of research software installation instructions from readme files: An initial analysis,","cited_arxiv_id":null,"evidence_quote":"Closest prior work, extracting installation instructions from READMEs without executing them, marking the gap this paper fills."}],"review_version":1}