{"id":"41d6e086-1d52-471f-9b83-442065f954f1","arxiv_id":"2502.00902","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A web crawl of repositories from NeurIPS, ICML, ICLR, AISTATS, TMLR, and MLOSS quantifies low adoption of software engineering practices like licensing, packaging, and automated testing in ML research.","lead":"This paper measures how often machine learning research code already follows basic software engineering practices by crawling thousands of GitHub repositories linked from major ML conferences and journals. It finds that licenses, packaging, and automated tests are common gaps, and it argues that fixing these would improve reproducibility.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The crawler appears to count every GitHub URL found in a paper PDF as a 'repository associated with the publication,' so the Section 5 adoption rates may mix the paper's own code with third-party libraries cited in references.","rationale":"Good faith reading: the paper is a useful, reproducible survey with a public pipeline and sensible recommendations. The central claim, however, rests on the validity of the crawled repository set. The reader's concern focused on file presence as a proxy for practice adoption; I agree that is an issue, but there is a more upstream threat: the mapping from papers to repositories is not described or validated. The Methods section says 'we extracted links to source code repositories hosted on GitHub' and then treats each as a repository associated with the publication. No filtering by relevance is mentioned, and no manual validation is reported. Since PDFs include many third-party GitHub links, the sample may be contaminated. If the extraction precision is high, the concern does not land and the paper's numbers stand; the proposed annotation check settles it. I would keep the verdict conditional because the pipeline is public and the fix is a re-analysis, not a change in research question. The qualitative recommendation for more rigorous software engineering is plausible independent of the exact percentages, but the paper's specific quantified claims need the validation before they can be relied on.","tokens_in":11634,"tokens_out":8206,"duration_ms":89170,"concrete_test":"Run the crawled pipeline on a stratified random sample of 50 papers (mix of venues/years). For each paper, have two independent annotators identify the code repository(s) that actually implement the paper (e.g., the repo linked in the abstract or code-availability statement). Compare this ground-truth set to the set of GitHub URLs the pipeline extracted from the same PDF. Compute precision (extracted repos that are ground-truth) and recall. If precision is below 0.9 or recall is below 0.8, re-estimate all Section 5 adoption rates using only ground-truth repos; if any headline percentage (e.g., requirements.txt+environment.yml share) shifts by more than 5 percentage points, the paper's empirical claims are not yet supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 4 states that the pipeline extracts 'links to source code repositories hosted on GitHub' from paper PDFs via pdfx and custom NLP, then checks each repository for files. The manuscript does not describe any step that keeps only the repository implementing the paper, nor does it report validation of that mapping. ML paper PDFs routinely contain hyperlinks to PyTorch, JAX, benchmark suites, and other third-party projects in their bibliographies. If those URLs enter the crawl, the denominator for Figures 2-8 is not 'repositories associated with publications' but 'all GitHub URLs appearing in papers.' The paper's central quantified evidence (e.g., 'At ICML in 2024, the combined share ... less than 50%' and 'roughly three-quarters ... without automated testing') would then conflate the authors' research code with unrelated software. The repository is public, which is a strength, but the published text provides no precision/recall check for the extraction step. This is a correctness risk upstream of the file-presence proxy: even a perfect proxy cannot repair a contaminated sample.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that reproducibility in ML research is impeded by underuse of software engineering best practices. It reviews best practices (licensing, README files, formatting/linting/type checking, dependency enumeration, packaging, testing, CI, seed recording, and documentation), then describes an automated pipeline that crawls PDFs from NeurIPS, ICML, ICLR, AISTATS, TMLR, and MLOSS, extracts GitHub URLs, and checks for the presence of a fixed list of files and folders. It reports adoption rates over 2018–2025 for indicators such as README, LICENSE, requirements.txt, environment.yml, pyproject.toml, tests folders, .github/workflows, and lock files, and concludes with a reviewer checklist and community recommendations.","tokens_in":11816,"tokens_out":6210,"duration_ms":60788,"significance":"If the measurement pipeline is valid, this is a valuable large-scale empirical snapshot of a debate that is often conducted anecdotally. The paper's public code repository and the reproducible crawler are genuine strengths, and the qualitative conclusion that best practices are underused is consistent with prior literature. The main risk is that the quantitative estimates rest on an unvalidated mapping from PDF citations to the paper's own code repository and on file presence as a proxy for practice; until those issues are addressed, the exact percentages should be treated as provisional.","major_comments":[{"comment":"The crawler extracts \"links to source code repositories hosted on GitHub\" from PDFs using pdfx and custom NLP, but the method section never describes a step that keeps only the repository implementing the paper, nor any validation of that mapping. ML paper PDFs routinely contain GitHub URLs for frameworks such as PyTorch and JAX, benchmark suites, and other third-party tools in their references. As written, the denominator of Figures 2–8 is therefore \"GitHub URLs appearing in the PDF\" rather than \"repositories associated with the publication.\" This can inflate adoption rates because third-party infrastructure repositories tend to have stronger engineering practices. Please add a precision/recall validation on a random sample, filter out reference-section URLs, or map repositories to papers by title or author metadata, and report how the headline numbers change after filtering.","section":"§4, §5"},{"comment":"The presence of a file or folder is used as a proxy for adoption of the corresponding practice, but the paper does not verify that the license is appropriate, that dependencies are complete or pinned, that tests run, or that CI is active. Some caveats are acknowledged in the text (e.g., the .flake8 discussion), but the abstract and headline statements such as \"the combined share ... less than 50%\" and \"roughly three-quarters ... without automated testing\" are reported as measurements of practice. I recommend adding an explicit limitation paragraph and, if feasible, validating a sample of repositories (e.g., running the tests or checking whether versions are pinned) and reporting confidence intervals around the point estimates.","section":"§5, Figures 2–8"},{"comment":"The claim that roughly three-quarters of the community work without automated testing and workflow automation is based on the absence of .github/workflows and .pre-commit-config.yaml. This undercounts projects that use other CI providers (e.g., Travis, GitLab CI, Azure Pipelines) and equates pre-commit configuration with continuous integration. The statement should be reworded as a lower-bound or upper-bound estimate, or the measure should be validated against a sample of repositories known to use CI.","section":"§5, Figure 8"},{"comment":"The paper states that \"At ICML in 2024, the combined share of projects with requirements.txt and environment.yml files was still less than 50%,\" but the union of these two indicators is not shown in any figure; the figure shows the two shares separately. Please report the union directly or provide the underlying counts so this specific quantitative claim can be checked.","section":"§5 (Requirements documentation)"}],"minor_comments":[{"comment":"The sentence \"We look at the repository root for test or test as well as within asrc or a package folder\" appears to contain typos; it should probably read \"tests\" and \"a src\".","section":"§5, Figure 8 text"},{"comment":"The phrase \"full adoption\" overstates the plotted README data, which range from roughly 92% to 100%; \"near-universal\" would be more accurate.","section":"§5, README paragraph"},{"comment":"The recommendation \"Pip users can run pip freeze > requirements.txt\" should be revised, because pip freeze captures the entire current environment rather than the project's direct dependencies; pip-compile or dependency declarations in pyproject.toml are more appropriate for reproducible research.","section":"§3.4–3.5, §5"},{"comment":"The TMLR versus MLOSS baseline comparison is useful, but the figure and text should make explicit that MLOSS is a software-focused venue whose submissions are by construction reusable software, so it is not a random sample of ML research.","section":"§4, Figure 1"},{"comment":"The reference entry for Stojnic et al. contains a spacing typo (\"Kerkez andShagun Sodhani\").","section":"References"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the manuscript is within scope for a software-engineering/reproducibility venue. The public pipeline and the breadth of venues are genuine strengths. The main risk is the unvalidated URL-to-repository mapping; if the authors can supply precision/recall numbers or a filtered denominator, the paper could become a solid empirical contribution. I do not see a circularity problem; the self-citation to Hoyt et al. is used as a source for a checklist, which is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper before you cite its numbers: it's a useful survey of software engineering practice in ML repos, but its central quantified claims rest on an unvalidated URL-to-repository mapping. The crawler appears to count every GitHub URL found in a paper PDF as a 'repository associated with the publication'; the text does not describe any step that keeps only the repo implementing the paper, and there is no precision/recall check. ML papers routinely link PyTorch, JAX, benchmark suites, and other third-party code in their references. If those URLs enter the denominator, the adoption rates for things like requirements.txt and CI will be inflated, because foundational libraries are far better engineered than typical research code. That does not necessarily overturn the qualitative conclusion that best practices are underused, but it makes the specific percentages untrustworthy.\n\nWhat's genuinely new and good: the multi-venue, multi-year crawl (2018–2025, NeurIPS, ICML, ICLR, AISTATS, TMLR, MLOSS) with a consistent set of file-presence indicators. That's not in earlier work like Raff 2019 or Pineau et al. 2021. The trend data on pyproject.toml uptake and lockfile rarity are plausibly correct in relative terms, and the recommendations are pragmatic and mindful of incentives. The authors released the analysis code, which is good practice. The self-citation to Hoyt et al. is only as inspiration for the checklist, so that's not a problem.\n\nThe file-presence proxy is a second, softer weakness: presence of tests/CI files does not mean they run or are meaningful. The paper admits this only in passing; it should be stated as a limitation and ideally backed with a small validation sample.\n\nNet: the paper deserves a serious referee, but it needs major revision that validates the repository-extraction step, releases the dataset, and adds uncertainty bounds. As it stands, I would not quote the absolute numbers; I'd cite it only for the qualitative trend.","headline":"Useful survey with a load-bearing flaw in its URL-to-repo mapping; don't quote the absolute numbers until the crawler is validated.","tokens_in":12364,"tokens_out":2464,"would_cite":false,"duration_ms":24061,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Software engineering gaps in ML code repositories undermine reproducibility, a survey of major venues finds.","keywords":["machine learning reproducibility","software engineering best practices","dependency management","packaging","continuous integration","testing","research software","conference code survey"],"falsifier":"Take a random sample of repositories the crawler counted as adopting a practice—say, those with requirements.txt, a tests folder, or .github/workflows—attempt a clean install and run of the project in a fresh environment, and compare the success rate with the file-presence rate; if many counted repositories fail to install or run, the paper's adoption estimates overstate true reproducibility.","tokens_in":11436,"feed_emoji":"🔬","tokens_out":4234,"duration_ms":36617,"temperature":0.7,"pith_summary":"This paper argues that the reproducibility crisis in machine learning is partly a software engineering problem: the best practices that let others install, run, and verify code are often skipped. To back this up, it reports a survey of repositories linked from papers at NeurIPS, ICML, ICLR, AISTATS, TMLR, and MLOSS between 2018 and 2025, checking for files that signal licensing, documentation, dependency pinning, packaging, testing, and automation. The headline numbers are low: READMEs are nearly universal, but licenses stagnate below 80 percent, fewer than half of ICML 2024 projects document dependencies, about a quarter have a test folder, and roughly three-quarters lack automated testing or CI. If the picture is right, a large share of ML results cannot be reproduced straightforwardly, and the paper offers a concrete checklist to close the gap.","feed_headline":"Most ML code repos skip tests, pins, and CI","feed_subtitle":"A crawl of six top venues finds dependency files in fewer than half of projects and automated testing in about a quarter.","key_machinery":"The measuring instrument is an automated web-crawl pipeline that downloads papers from the six venues, extracts GitHub repository links, and checks each repository for a fixed list of files and folders—LICENSE, README, requirements.txt, environment.yml, pyproject.toml, setup.py, tests/, docs/, .github/workflows, and others—treating their presence as evidence of the corresponding best practice. This file-presence proxy is what carries the survey; every adoption percentage in the paper is computed from it.","core_discovery":"The central claim is that ML research software is systematically under-engineered relative to what reproducibility requires, and that this is measurable and fixable. The paper's survey finds that while README adoption is near universal and Python dominates, the practices that make code re-runnable elsewhere—explicit dependency lists, lock files, packaging, tests, and CI—are adopted by a minority. The authors interpret the upward trend around 2020 as a response to the NeurIPS code guide, but conclude that progress is too slow and uneven, and that more rigorous, but appropriately scoped, software engineering is needed.","pith_inferences":["The file-presence proxy likely overstates true reproducibility, because a repo can contain requirements.txt with unpinned or incomplete entries, tests that never run, or CI configured but failing; measuring actual install-and-run success would probably show even lower rates.","The survey's Python focus means the adoption numbers are upper bounds for the whole field; repositories in other languages are excluded from the Python-specific metrics.","The same crawl could be re-run periodically to track whether the proposed checklist changes behavior, and could be extended to artifact evaluation tracks or additional venues.","The stated connection between reproducible code and higher citations implies that adopting these practices may also serve authors' self-interest, strengthening the incentive argument beyond community norms."],"forward_implications":["At current adoption rates, a large fraction of ML papers do not enable straightforward re-running of their experiments, so reported results are harder to verify than they should be.","Simple automated commands (pip freeze > requirements.txt, conda env export > environment.yml, pip lock -e .) cover most dependency documentation gaps, so closing the gap is inexpensive.","Encouraging packaging and pyproject.toml would let other groups install and reuse code with standard import statements, reducing code duplication across the field.","The post-2020 improvement aligned with the NeurIPS code guide suggests that venue-level guidance and checklists shift behavior, which supports the paper's recommendation to ask these questions during review.","If MLOSS is a fair baseline, journal and conference code can reach much higher adoption rates, since MLOSS submissions show what is feasible for research software."],"supporting_citations":[{"why":"Defines the Python-specific best practices the survey checks and serves as the reference point for the post-2020 adoption trend.","marker":"(Stojnic et al., 2020)"},{"why":"Provides the NeurIPS reproducibility checklist context that motivated the paper's review-oriented recommendations.","marker":"(Pineau et al., 2021)"},{"why":"Supplies the three-tiered reproducibility scale that motivates dependency documentation, command ordering, and seed fixing.","marker":"(Heil et al., 2021)"},{"why":"Establishes that reproducible articles receive more citations, the incentive argument for adopting best practices.","marker":"(Raff, 2023)"},{"why":"Documents low software engineering adoption in computational science, the baseline expectation the ML survey extends.","marker":"(Johanson & Hasselbring, 2018)"},{"why":"Provides the cheminformatics checklist template that the paper adapts into its proposed submitter and reviewer checklist.","marker":"(Hoyt et al., 2023)"},{"why":"Defines the packaging layout the survey tracks, including src and tests folders and pyproject.toml metadata.","marker":"(Python-Packaging-Authority, 2025)"},{"why":"Recommends pyproject.toml over setup.py via PEP-518, grounding the packaging adoption trend interpretation.","marker":"(Brett Cannon, 2016)"}],"fun_headline_variants":["ML research code skips tests, pinning, and CI","Poor software practices undermine ML reproducibility","Survey finds ML repos lacking reproducibility essentials","Rigorous software engineering key to ML reproducibility"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The survey treats the presence of a file or folder in a GitHub repository as proof that the corresponding practice is actually followed, without checking whether the license is appropriate, dependencies are complete or pinned, tests run, or CI is active.","fun_headline_variants_meta":{"raw":{"variants":["ML research code skips tests, pinning, and CI","Poor software practices undermine ML reproducibility","Survey finds ML repos lacking reproducibility essentials","Rigorous software engineering key to ML reproducibility"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000567,"raw_usage":{"total_tokens":2605,"prompt_tokens":784,"completion_tokens":1821,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":400,"completion_tokens_details":{"reasoning_tokens":1773}},"tokens_in":400,"tokens_out":1821,"duration_ms":13037,"temperature":1.0,"reasoning_tokens":1773,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T17:16:54.973433+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of repositories the crawler counted as adopting a practice—say, those with requirements.txt, a tests folder, or .github/workflows—attempt a clean install and run of the project in a fresh environment, and compare the success rate with the file-presence rate; if many counted repositories fail to install or run, the paper's adoption estimates overstate true reproducibility.","supporting_citations":[{"cited_title":"Tips for releasing research code in machine learning (with official neurips 2020 recommendations), 2020","cited_arxiv_id":null,"evidence_quote":"Defines the Python-specific best practices the survey checks and serves as the reference point for the post-2020 adoption trend."},{"cited_title":"Reproducibility standards for machine learning in the life sciences","cited_arxiv_id":null,"evidence_quote":"Supplies the three-tiered reproducibility scale that motivates dependency documentation, command ordering, and seed fixing."},{"cited_title":"Software engineering for computational science: Past, present, future","cited_arxiv_id":null,"evidence_quote":"Documents low software engineering adoption in computational science, the baseline expectation the ML survey extends."},{"cited_title":"Improving reproducibility and reusability in the journal of cheminformatics","cited_arxiv_id":null,"evidence_quote":"Provides the cheminformatics checklist template that the paper adapts into its proposed submitter and reviewer checklist."},{"cited_title":"Packaging python-projects, 2025","cited_arxiv_id":null,"evidence_quote":"Defines the packaging layout the survey tracks, including src and tests folders and pyproject.toml metadata."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Recommends pyproject.toml over setup.py via PEP-518, grounding the packaging adoption trend interpretation."}],"review_version":1}