Pith. sign in

REVIEW 4 major objections 5 minor 45 references

More Rigorous Software Engineering Would Improve Reproducibility in Machine Learning Research

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Software engineering gaps in ML code repositories undermine reproducibility, a survey of major venues finds.

desk verdict Useful survey with a load-bearing flaw in its URL-to-repo mapping; don't quote the absolute numbers until the crawler is validated. read the letter →

arxiv 2502.00902 v2 pith:7FNV5UPX submitted 2025-02-02 cs.SE cs.LG

Moritz Wolter , Lokesh Veeramacheneni , Charles Tapley Hoyt This is my paper · ORCID
classification cs.SEcs.LG
keywords machinelearningreproducibilitysoftwareengineeringbestpracticesdependencymanagementpackagingcontinuousintegrationtestingresearchconferencecodesurvey
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the reproducibility crisis in machine learning is partly a software engineering problem: the best practices that let others install, run, and verify code are often skipped. To back this up, it reports a survey of repositories linked from papers at NeurIPS, ICML, ICLR, AISTATS, TMLR, and MLOSS between 2018 and 2025, checking for files that signal licensing, documentation, dependency pinning, packaging, testing, and automation. The headline numbers are low: READMEs are nearly universal, but licenses stagnate below 80 percent, fewer than half of ICML 2024 projects document dependencies, about a quarter have a test folder, and roughly three-quarters lack automated testing or CI. If the picture is right, a large share of ML results cannot be reproduced straightforwardly, and the paper offers a concrete checklist to close the gap.

What carries the argument

The measuring instrument is an automated web-crawl pipeline that downloads papers from the six venues, extracts GitHub repository links, and checks each repository for a fixed list of files and folders—LICENSE, README, requirements.txt, environment.yml, pyproject.toml, setup.py, tests/, docs/, .github/workflows, and others—treating their presence as evidence of the corresponding best practice. This file-presence proxy is what carries the survey; every adoption percentage in the paper is computed from it.

What would settle it

Take a random sample of repositories the crawler counted as adopting a practice—say, those with requirements.txt, a tests folder, or .github/workflows—attempt a clean install and run of the project in a fresh environment, and compare the success rate with the file-presence rate; if many counted repositories fail to install or run, the paper's adoption estimates overstate true reproducibility.

Watch

Extended reading notes

Core claim

The central claim is that ML research software is systematically under-engineered relative to what reproducibility requires, and that this is measurable and fixable. The paper's survey finds that while README adoption is near universal and Python dominates, the practices that make code re-runnable elsewhere—explicit dependency lists, lock files, packaging, tests, and CI—are adopted by a minority. The authors interpret the upward trend around 2020 as a response to the NeurIPS code guide, but conclude that progress is too slow and uneven, and that more rigorous, but appropriately scoped, software engineering is needed.

Load-bearing premise

The survey treats the presence of a file or folder in a GitHub repository as proof that the corresponding practice is actually followed, without checking whether the license is appropriate, dependencies are complete or pinned, tests run, or CI is active.

Editorial extensions

If this is right

  • At current adoption rates, a large fraction of ML papers do not enable straightforward re-running of their experiments, so reported results are harder to verify than they should be.
  • Simple automated commands (pip freeze > requirements.txt, conda env export > environment.yml, pip lock -e .) cover most dependency documentation gaps, so closing the gap is inexpensive.
  • Encouraging packaging and pyproject.toml would let other groups install and reuse code with standard import statements, reducing code duplication across the field.
  • The post-2020 improvement aligned with the NeurIPS code guide suggests that venue-level guidance and checklists shift behavior, which supports the paper's recommendation to ask these questions during review.
  • If MLOSS is a fair baseline, journal and conference code can reach much higher adoption rates, since MLOSS submissions show what is feasible for research software.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The file-presence proxy likely overstates true reproducibility, because a repo can contain requirements.txt with unpinned or incomplete entries, tests that never run, or CI configured but failing; measuring actual install-and-run success would probably show even lower rates.
  • The survey's Python focus means the adoption numbers are upper bounds for the whole field; repositories in other languages are excluded from the Python-specific metrics.
  • The same crawl could be re-run periodically to track whether the proposed checklist changes behavior, and could be extended to artifact evaluation tracks or additional venues.
  • The stated connection between reproducible code and higher citations implies that adopting these practices may also serve authors' self-interest, strengthening the incentive argument beyond community norms.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that reproducibility in ML research is impeded by underuse of software engineering best practices. It reviews best practices (licensing, README files, formatting/linting/type checking, dependency enumeration, packaging, testing, CI, seed recording, and documentation), then describes an automated pipeline that crawls PDFs from NeurIPS, ICML, ICLR, AISTATS, TMLR, and MLOSS, extracts GitHub URLs, and checks for the presence of a fixed list of files and folders. It reports adoption rates over 2018–2025 for indicators such as README, LICENSE, requirements.txt, environment.yml, pyproject.toml, tests folders, .github/workflows, and lock files, and concludes with a reviewer checklist and community recommendations.

Significance. If the measurement pipeline is valid, this is a valuable large-scale empirical snapshot of a debate that is often conducted anecdotally. The paper's public code repository and the reproducible crawler are genuine strengths, and the qualitative conclusion that best practices are underused is consistent with prior literature. The main risk is that the quantitative estimates rest on an unvalidated mapping from PDF citations to the paper's own code repository and on file presence as a proxy for practice; until those issues are addressed, the exact percentages should be treated as provisional.

major comments (4)
  1. [§4, §5] The crawler extracts "links to source code repositories hosted on GitHub" from PDFs using pdfx and custom NLP, but the method section never describes a step that keeps only the repository implementing the paper, nor any validation of that mapping. ML paper PDFs routinely contain GitHub URLs for frameworks such as PyTorch and JAX, benchmark suites, and other third-party tools in their references. As written, the denominator of Figures 2–8 is therefore "GitHub URLs appearing in the PDF" rather than "repositories associated with the publication." This can inflate adoption rates because third-party infrastructure repositories tend to have stronger engineering practices. Please add a precision/recall validation on a random sample, filter out reference-section URLs, or map repositories to papers by title or author metadata, and report how the headline numbers change after filtering.
  2. [§5, Figures 2–8] The presence of a file or folder is used as a proxy for adoption of the corresponding practice, but the paper does not verify that the license is appropriate, that dependencies are complete or pinned, that tests run, or that CI is active. Some caveats are acknowledged in the text (e.g., the .flake8 discussion), but the abstract and headline statements such as "the combined share ... less than 50%" and "roughly three-quarters ... without automated testing" are reported as measurements of practice. I recommend adding an explicit limitation paragraph and, if feasible, validating a sample of repositories (e.g., running the tests or checking whether versions are pinned) and reporting confidence intervals around the point estimates.
  3. [§5, Figure 8] The claim that roughly three-quarters of the community work without automated testing and workflow automation is based on the absence of .github/workflows and .pre-commit-config.yaml. This undercounts projects that use other CI providers (e.g., Travis, GitLab CI, Azure Pipelines) and equates pre-commit configuration with continuous integration. The statement should be reworded as a lower-bound or upper-bound estimate, or the measure should be validated against a sample of repositories known to use CI.
  4. [§5 (Requirements documentation)] The paper states that "At ICML in 2024, the combined share of projects with requirements.txt and environment.yml files was still less than 50%," but the union of these two indicators is not shown in any figure; the figure shows the two shares separately. Please report the union directly or provide the underlying counts so this specific quantitative claim can be checked.
minor comments (5)
  1. [§5, Figure 8 text] The sentence "We look at the repository root for test or test as well as within asrc or a package folder" appears to contain typos; it should probably read "tests" and "a src".
  2. [§5, README paragraph] The phrase "full adoption" overstates the plotted README data, which range from roughly 92% to 100%; "near-universal" would be more accurate.
  3. [§3.4–3.5, §5] The recommendation "Pip users can run pip freeze > requirements.txt" should be revised, because pip freeze captures the entire current environment rather than the project's direct dependencies; pip-compile or dependency declarations in pyproject.toml are more appropriate for reproducible research.
  4. [§4, Figure 1] The TMLR versus MLOSS baseline comparison is useful, but the figure and text should make explicit that MLOSS is a software-focused venue whose submissions are by construction reusable software, so it is not a random sample of ML research.
  5. [References] The reference entry for Stojnic et al. contains a spacing typo ("Kerkez andShagun Sodhani").

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the survey's adoption rates are direct external measurements, and the only self-citation is not load-bearing.

full rationale

The paper's central quantitative claims are empirical measurements obtained by crawling conference PDFs, extracting GitHub repository links, and checking for the presence of specific files and folders. Each reported adoption rate in Figures 2-8 is a direct count of observable repository contents, not a prediction derived from a fitted parameter or from the paper's own conclusions. No equation is defined in terms of the quantity it claims to estimate, and no fitted input is renamed as a prediction. The recommended checklist in Section 6 is inspired by Hoyt et al. (2023), a prior work by one co-author, but that citation is used only as a template for recommendations and does not provide the evidence for the survey's adoption rates; the measurements stand independently on the crawl data. The file-presence proxy (e.g., assuming a tests folder implies testing practice) is a measurement validity limitation, not a circular derivation, because the paper's conclusion is that adoption of these file-level indicators is low, and that claim follows directly from the observed file counts. No uniqueness theorem, ansatz, or self-referential result is invoked. Therefore the derivation chain is self-contained with respect to its evidence, and no circular step can be exhibited from the paper's text.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No parameters are fitted. The paper measures file-presence counts, so its epistemic load rests on the proxy assumptions above and on the coverage of the crawl. No new entities or fitted constants are introduced.

assumptions (3)
  • domain assumption GitHub repository links extracted from paper PDFs are representative of the source code supporting ML publications.
    Section 4 describes extraction of GitHub links via pdfx and custom NLP; non-GitHub or unlinked code is not sampled.
  • domain assumption Presence of specific files (LICENSE, requirements.txt, tests folder, .github/workflows, etc.) is a valid proxy for adoption of the corresponding software engineering practice.
    The whole survey in Section 5 counts file and folder existence, not content quality or actual practice usage.
  • domain assumption Python-specific analysis is sufficient because Python dominates ML research code.
    Section 4 states the analysis is Python-specific and that Python adoption is an upper bound; Section 5 reports Python use above 80% since 2021.

how reviews work

0 comments
Cite this review

Pith. "Pith review of More Rigorous Software Engineering Would Improve Reproducibility in Machine Learning Research." pith.science (2026). https://pith.science/paper/7FNV5UPX

@misc{pith2026250200902,
  author       = {Pith},
  title        = {Pith review of: More Rigorous Software Engineering Would Improve Reproducibility in Machine Learning Research},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7FNV5UPX}},
  note         = {Machine review of arXiv:2502.00902}
}
read the original abstract

While experimental reproduction remains a pillar of the scientific method, we observe that the software best practices supporting the reproduction of machine learning ( ML ) research are often undervalued or overlooked, leading both to poor reproducibility and damage to trust in the ML community. We quantify these concerns by surveying the usage of software best practices in software repositories associated with publications at major ML conferences and journals such as NeurIPS, ICML, ICLR, TMLR, and MLOSS within the last decade. We report the results of this survey that identify areas where software best practices are lacking and areas with potential for growth in the ML community. Finally, we discuss the implications and present concrete recommendations on how we, as a community, can improve reproducibility in ML research.

Figures

Figures reproduced from arXiv: 2502.00902 by the authors.

Figure 1
Figure 1. Estimated state of software engineering best practices at [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Estimated state of README and LICENSE file adoption in major [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Repository link crawl results for Python adoption (left) and standalone documentation (right). We [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Adoption of .flake8 configuration files in repositories over time. Python adoption over time Most of the engineering best practices we described previously in Section 3 are Python specific, before moving on, we must therefore check if Python is indeed the most common l…
Figure 5
Figure 5. Figure 5: Requirements documentation over time. The figure illustrates the share of repositories with [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Overview of the lock files the crawler discovered. The environment plot adds the numbers for [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Files and folders which indicate Python packaging adoption over time. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Systematic test adoption over time. The plot for tox combines counts of [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Rare requirements documentation lock-files A Appendix Impact Statement This work exhibits a critical gap in the adoption of software practices to improve reproducibility in ML research and proposes actionable recommendations. Our findings aim to call for action in the …
Figure 10
Figure 10. Figure 10: Adoption of potentially emerging packaging tools. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 39 canonical work pages

  1. [1]

    Mart\' i n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Man\' e , Rajat Monga, Sherry Moore, Der...

    2015

  2. [2]

    JAX : composable transformations of P ython+ N um P y programs, 2025

    James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake Vander P las, Skye Wanderman- M ilne, and Qiao Zhang. JAX : composable transformations of P ython+ N um P y programs, 2025. URL http://github.com/jax-ml/jax

    work page 2025

  3. [3]

    The ml test score: A rubric for ml production readiness and technical debt reduction

    Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, and D Sculley. The ml test score: A rubric for ml production readiness and technical debt reduction. In 2017 IEEE international conference on big data (big data), pp.\ 1123--1132. IEEE, 2017

    work page 2017

  4. [4]

    Donald Stufft Brett Cannon, Nathaniel J. Smith. Pep 518 – specifying minimum build system requirements for python projects, 2016. URL https://peps.python.org/pep-0518/

    work page 2016

  5. [5]

    Pep 751 – a file format to record python dependencies for installation reproducibility, 2024

    Brett Cannon. Pep 751 – a file format to record python dependencies for installation reproducibility, 2024. URL https://peps.python.org/pep-0751/

    work page 2024

  6. [6]

    Pep 621 – storing project metadata in pyproject.toml, 2020

    Brett Cannon, Dustin Ingram, Paul Ganssle, Pradyun Gedam, Sébastien Eustace, Thomas Kluyver, and Tzu-ping Chung. Pep 621 – storing project metadata in pyproject.toml, 2020. URL https://peps.python.org/pep-0621/

    work page 2020

  7. [7]

    Conda documentation, 2025

    Conda-developers. Conda documentation, 2025. URL https://github.com/conda/conda

    work page 2025

  8. [8]

    Chex documentation, 2025

    Jax ecosystem authors. Chex documentation, 2025. URL https://chex.readthedocs.io/en/latest/

    work page 2025

Show all 45 references
  1. [9]

    Manual for package pgfplots

    Christian Feuers \"a nger. Manual for package pgfplots. CTAN, 17, 2011

    2011

  2. [10]

    Pdfx, 2021

    Chris Hager. Pdfx, 2021. URL https://github.com/metachris/pdfx

    2021

  3. [11]

    38 Mason Christopher E

    Benjamin Haibe-Kains, George Alexandru Adam, Ahmed Hosny, Farnoosh Khodakarami, Massive Analysis Quality Control (MAQC) Society Board of Directors Shraddha Thakkar 35 Kusko Rebecca 36 Sansone Susanna-Assunta 37 Tong Weida 35 Wolfinger Russ D. 38 Mason Christopher E. 39 Jones W...

    2020

  4. [12]

    Hatch documentation, 2025

    Hatch-developers. Hatch documentation, 2025. URL https://hatch.pypa.io/latest/

    2025

  5. [13]

    Reproducibility standards for machine learning in the life sciences

    Benjamin J Heil, Michael M Hoffman, Florian Markowetz, Su-In Lee, Casey S Greene, and Stephanie C Hicks. Reproducibility standards for machine learning in the life sciences. Nature Methods, 18 0 (10): 0 1132--1135, 2021

    2021

  6. [14]

    Improving reproducibility and reusability in the journal of cheminformatics

    Charles Tapley Hoyt, Barbara Zdrazil, Rajarshi Guha, Nina Jeliazkova, Karina Martinez-Mayorga, and Eva Nittinger. Improving reproducibility and reusability in the journal of cheminformatics. Journal of Cheminformatics, 15 0 (1): 0 62, 2023

    2023

  7. [15]

    Artificial intelligence faces reproducibility crisis

    Matthew Hutson. Artificial intelligence faces reproducibility crisis. Science, 359, 2018

    2018

  8. [16]

    Software engineering for computational science: Past, present, future

    Arne Johanson and Wilhelm Hasselbring. Software engineering for computational science: Past, present, future. Computing in Science & Engineering, 20 0 (2): 0 90--109, 2018

    2018

  9. [17]

    Ten simple rules for developing usable software in computational biology

    Markus List, Peter Ebert, and Felipe Albrecht. Ten simple rules for developing usable software in computational biology. PLoS Comput. Biol., 13 0 (1), 2017. URL https://doi.org/10.1371/journal.pcbi.1005265

    2017 doi

  10. [18]

    mypy, 2025

    mypy developers. mypy, 2025. URL https://mypy-lang.org/index.html. Accessed: 2025-07-30

    2025

  11. [19]

    Automatic differentiation in pytorch

    Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS 2017 Autodiff Workshop, 2017

    2017

  12. [20]

    Eglen, Daniel S

    Yasset P \' e rez - Riverol, Laurent Gatto, Rui Wang, Timo Sachsenberg, Julian Uszkoreit, Felipe da Veiga Leprevost, Christian Fufezan, Tobias Ternent, Stephen J. Eglen, Daniel S. Katz, Tom J. Pollard, Alexander Konovalov, Robert M. Flight, Kai Blin, and Juan Antonio Vizca \' ...

    2016 doi

  13. [21]

    Improving reproducibility in machine learning research (a report from the neurips 2019 reproducibility program)

    Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivi \`e re, Alina Beygelzimer, Florence d'Alch \'e Buc, Emily Fox, and Hugo Larochelle. Improving reproducibility in machine learning research (a report from the neurips 2019 reproducibility program). Journal o...

    2019

  14. [22]

    Requirements file format - pip documentation v25.0, 2025

    Pip-developers. Requirements file format - pip documentation v25.0, 2025. URL https://pip.pypa.io/en/stable/reference/requirements-file-format/

    2025

  15. [23]

    The logic of scientific discovery

    Karl Popper. The logic of scientific discovery. Routledge, 2005

    2005

  16. [24]

    Ml reproducibility challenge, 2025

    Princeton-AI-Lab. Ml reproducibility challenge, 2025. URL https://reproml.org/. Accessed: 2025-06-30

    2025

  17. [25]

    Andreas Prlic and James B. Procter. Ten simple rules for the open development of scientific software. PLoS Comput. Biol., 8 0 (12), 2012. URL https://doi.org/10.1371/journal.pcbi.1002802

    2012 doi

  18. [26]

    Pytest documentation, 2025

    Pytest-developers. Pytest documentation, 2025. URL https://docs.pytest.org/en/stable/

    2025

  19. [27]

    unittest - unit testing framework, 2025

    Python-developers. unittest - unit testing framework, 2025. URL https://docs.python.org/3/library/unittest.html

    2025

  20. [28]

    Packaging python-projects, 2025

    Python-Packaging-Authority. Packaging python-projects, 2025. URL https://packaging.python.org/tutorials/packaging-projects/

    2025

  21. [29]

    Reproducibility, 2024

    PyTorch-Contributors. Reproducibility, 2024. URL https://pytorch.org/docs/stable/notes/randomness.html

    2024

  22. [30]

    A step toward quantifying independently reproducible machine learning research

    Edward Raff. A step toward quantifying independently reproducible machine learning research. Advances in Neural Information Processing Systems, 32, 2019

    2019

  23. [31]

    Research reproducibility as a survival analysis

    Edward Raff. Research reproducibility as a survival analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.\ 469--478, 2021. URL https://doi.org/10.1609/aaai.v35i1.16124

    2021 doi

  24. [32]

    Does the market of citations reward reproducible work? In Proceedings of the 2023 ACM Conference on Reproducibility and Replicability, pp.\ 89--96, 2023

    Edward Raff. Does the market of citations reward reproducible work? In Proceedings of the 2023 ACM Conference on Reproducibility and Replicability, pp.\ 89--96, 2023

    2023

  25. [33]

    A siren song of open source reproducibility, examples from machine learning

    Edward Raff and Andrew L Farris. A siren song of open source reproducibility, examples from machine learning. In Proceedings of the 2023 ACM Conference on Reproducibility and Replicability, pp.\ 115--120, 2023

    2023

  26. [34]

    Beautiful soup documentation, 2023

    Leonard Richardson. Beautiful soup documentation, 2023. URL https://www.crummy.com/software/BeautifulSoup/bs4/doc/

    2023

  27. [35]

    Pep 484 – type hints, 2014

    Guido van Rossum, Jukka Lehtosalo, and Łukasz Langa. Pep 484 – type hints, 2014. URL https://peps.python.org/pep-0484/

    2014

  28. [36]

    Ten simple rules for reproducible computational research

    Geir Kjetil Sandve, Anton Nekrutenko, James Taylor, and Eivind Hovig. Ten simple rules for reproducible computational research. PLoS Comput. Biol., 9 0 (10), 2013. URL https://doi.org/10.1371/journal.pcbi.1003285

    2013 doi

  29. [37]

    Setuptools documentation, 2025

    Setuptools-Team. Setuptools documentation, 2025. URL https://setuptools.pypa.io/en/stable/

    2025

  30. [38]

    Smith, Thomas Kluyver, and Alyssa Coghlan

    Nathaniel J. Smith, Thomas Kluyver, and Alyssa Coghlan. Pep 517 – a build-system independent format for source trees, 2015. URL https://peps.python.org/pep-0517/

    2015

  31. [39]

    The need for open source software in machine learning

    Sören Sonnenburg, Mikio L Braun, Cheng Soon Ong, Samy Bengio, Leon Bottou, Geoffrey Holmes, Yann LeCun, Klaus-Robert Müller, Fernando Pereira, Carl Edward Rasmussen, et al. The need for open source software in machine learning. Journal of Machine Learning Research, 8 0 (Oct): ...

    2007

  32. [40]

    Sphinx documentation, 2025

    Sphinx-developers. Sphinx documentation, 2025. URL https://www.sphinx-doc.org

    2025

  33. [41]

    Tips for releasing research code in machine learning (with official neurips 2020 recommendations), 2020

    Robert Stojnic, Ross Taylor, Sarthak Pati, Fabian-Robert Stöter, Viktor Kerkez andShagun Sodhani, Hamel Husain, Amit Chaudhary, and Rishabh Jain. Tips for releasing research code in machine learning (with official neurips 2020 recommendations), 2020. URL https://github.com/pap...

    2020

  34. [42]

    A practical taxonomy of reproducibility for machine learning research

    Rachael Tatman, Jake VanderPlas, and Sohier Dane. A practical taxonomy of reproducibility for machine learning research. In 2nd Reproducibility in Machine Learning Workshop at ICML 2018, Stockholm, Sweden., 2018

    2018

  35. [43]

    Pep 8 -- style guide for python code, 2001

    Guido van Rossum, Barry Warsaw, and Nick Coghlan. Pep 8 -- style guide for python code, 2001. URL https://www.python.org/dev/peps/pep-0008/

    2001

  36. [44]

    The five pillars of computational reproducibility: bioinformatics and beyond

    Mark Ziemann, Pierre Poulain, and Anusuiya Bora. The five pillars of computational reproducibility: bioinformatics and beyond. Briefings in Bioinformatics, 24 0 (6), 2023

    2023

  37. [45]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.