REVIEW 4 major objections 6 minor 13 references
What Do Machine Learning Researchers Mean by "Reproducible"?
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Eight distinct rigor problems hide under the word 'reproducibility'.
desk verdict A useful eight-way taxonomy of rigor in ML, but the prevalence percentages in Table 1 are not robust enough to be read as field measurements. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the eight-category taxonomy defined in Table 1, built by a manual review of 101 papers that self-identify as being about repeatability, reproducibility, or replicability. Each category is one named rigor aspect with a one-sentence concern, and the paper uses the taxonomy to do two things: count how often each aspect is the primary focus of current literature, and lay out dependency relations among aspects (solid arrows for hard dependencies such as repeatability before reproducibility, dashed arrows for influences such as model selection and data quality flowing into the other aspects). The taxonomy is what converts an ambiguous slogan into measurable, comparable research topics.
What would settle it
Have several independent coders assign every relevant rigor aspect to each of the same 101 papers instead of one primary category; if many papers touch three or more aspects, or if a broader sample of recent papers falls outside the eight categories, the taxonomy's boundaries and the Table 1 percentages are not stable.
Extended reading notes
Core claim
The paper's central claim is that the community's overloaded use of 'reproducibility' conflates eight distinct aspects of scientific rigor, each with its own question: Can the original authors repeat their own results with their own code and data (repeatability)? Can a different team get the same results using the provided code and data (reproducibility)? Can a different team write new code or use different data and still reach congruent results (replicability)? Can the original method handle new data (adaptability)? How should one reliably choose among competing models (model selection)? How can labeling processes yield stable, low-error labels (label/data quality)? What incentives drive or block rigor (meta & incentive)? And what keeps a solution working as people, code, and data change over time (maintainability)? The paper further claims these aspects interlock: repeatability is a precondition for reproducibility, reproducibility for replicability, and maintainability is essentially iterated replicability over time plus instantaneous repeatability at each point.
Load-bearing premise
The quantitative map stands on the assumption that the 101 manually collected, self-identifying papers fairly represent the field and that each paper can be assigned one primary rigor category.
Editorial extensions
If this is right
- Researchers and reviewers can say which of the eight aspects a given 'reproducibility' study actually addresses, removing a recurring source of confusion.
- Because the aspects form a dependency chain, a failure at the repeatability level undermines any downstream reproducibility or replicability claim.
- Maintainability can be understood as repeatability plus replicability stretched over time, so aging software, drifting labels, and hardware changes are not separate concerns but the same rigor problem.
- The measured proportions identify model selection as the most-studied aspect and expose adaptability and label/data quality as the most neglected, giving a concrete agenda for where new rigor research would matter.
- The paper's recommendation to create a dedicated scientific-rigor track at major AI/ML conferences is a direct corollary of having a named set of topics to consolidate.
Reading between the lines
- An implication the authors leave implicit is that funding agencies and journals could use the eight categories as a checklist when asking what rigor aspect a proposal addresses, which would make the neglected categories harder to overlook.
- This reader infers that the eight aspects are not cleanly disjoint: most papers touch several, so a multi-label coding of the same corpus would likely shift the percentages and may reveal stable pairs such as reproducibility plus model selection.
- A testable extension would be to run the same taxonomy on a broader, automatically sampled corpus across conferences and years to see whether the 4.0% figures for adaptability and label/data quality are a stable property of the field or an artifact of this review.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that the AI/ML community uses "reproducibility" as an umbrella term covering at least eight distinct rigor concerns. It proposes a taxonomy of repeatability, reproducibility, replicability, adaptability, model selection, label/data quality, meta & incentive, and maintainability, defined in Table 1, with the percentage of papers primarily devoted to each aspect reported from the authors' manual categorization of a corpus of 101 papers listed in Appendix A that "self-identify" with reproducibility terminology. The paper also surveys the historical literature for each aspect, proposes dependency relations among the rigor types (Section 3, Figure 1), and recommends that major AI/ML conferences create dedicated rigor tracks.
Significance. If the taxonomy is adopted by the community, it would supply a much-needed shared vocabulary and a map of where rigor effort is concentrated, and the paper's historical scholarship — especially the tracing of model-selection and benchmarking concerns back decades in Section 2.4 — is a genuine service to the field. The paper is also commendably transparent: the entire 101-paper corpus is listed in Appendix A, and the authors explicitly disclose that the category assignments are subjective. These strengths make the qualitative contribution real and citable. However, the quantitative claim that the paper has "quantified the proportion of each type as studied today" is not supported by the evidence as presented, because the corpus is non-systematic and the inclusion filter is applied inconsistently; the percentages in Table 1 should be treated as an illustrative breakdown of a convenience sample, not as measurements of the field, unless the corpus is rebuilt with a documented protocol or the claim is explicitly demoted.
major comments (4)
- [§2 / Appendix A / Table 1] Section 2 states that a paper qualifies for Table 1 only if it "must self-identify itself as being about 'repeatability, reproducibility, or replicability'," but the corpus in Appendix A does not follow that rule: Bates, Hastie, and Tibshirani (2021), Bergmeir, Hyndman, and Koo (2018), Varoquaux (2018), Dacrema, Cremonesi, and Jannach (2019), and Kim et al. (2022) are all categorized as Model Selection papers, yet none of these works frames itself with reproducibility terminology. Appendix A's looser operative criterion ("self-identify as being about 'reproducibility' in some sense") is not consistently defined, so the denominator of Table 1 is not the population described in Section 2, and the percentages cannot be taken at face value.
- [§2 / Table 1 / §4] The corpus is described in Section 2 as "all literature we are aware of," with no search protocol, no statement of venue or database coverage, and no formal inclusion/exclusion criteria beyond self-identification; the Conclusion (§4) nevertheless claims to have "quantified the proportion of each type as studied today." Because the Table 1 percentages (e.g., Model Selection 19.8%, Maintainability 12.9%) are conditional on an unstated and unrepeatable sampling procedure, the quantitative claim is not reproducible by other researchers. The paper should either supply a documented search and screening protocol (keywords, databases, date range, screening steps) or explicitly reframe Table 1 as a descriptive breakdown of the reviewed 101 papers and remove the "quantified" claim from the Conclusion.
- [Appendix A / §2.2] Appendix A discloses that the primary-category assignment is "our subjective call," but no coding rubric is given and no inter-rater reliability is reported; since many of the 101 papers touch multiple aspects, the single-rater assignments determine both the category boundaries and every percentage in Table 1. A concrete example of boundary instability is Musgrave, Belongie, and Lim (2020), coded "Reproducibility" in Appendix A but described in §2.2 as a problem of simultaneous, confounding changes to baselines — a concern that fits the paper's own "Model Selection" definition at least as well. A dual-coding exercise with reported agreement statistics, or at least a published coding rubric with example papers per category, is needed before the proportions can be treated as measurements.
- [Introduction / §2.5 / §2.6 / §4] Even if the inclusion filter were applied consistently, the sampling frame would still not support the Conclusion's framing: the stated construct is the scope of "scientific rigor" (Introduction), yet the corpus is restricted to papers that announce themselves with reproducibility terminology, and the paper itself notes that major bodies of relevant work (label-fusion methods in §2.6, decision-tree robustness studies in §2.5) never use such terms. Table 1 therefore systematically undercounts Adaptability and Label/Data Quality, the very categories whose literatures are least self-labeled, so the reported proportions are biased relative to the stated construct; this limitation should be acknowledged explicitly and the claim softened.
minor comments (6)
- [§3.1] The sentence "the most obvious, and intuitive connections are from repeatability to reproducibility to repeatability" should read "...to replicability"; as written it contradicts the paragraph's own escalation argument (repeatability → reproducibility → replicability).
- [§2.8 / Figure 1] Section 2.8 contains "maintinable" (should be "maintainable"), and Figure 1's legend contains "Eachother"; more substantively, the dashed line labeled "Interact With Each Other" is not explained in the text, so readers cannot tell what distinguishes Maintainability, Data Quality, and Model Selection as an interaction class.
- [Introduction / Appendix A] The Introduction says the 101 papers were "published since 2017," but Appendix A includes Sculley et al. (2015), which violates the stated date window; either update the text or exclude that paper from the corpus.
- [§2.2 / §2.7] The discussion of conference questionnaires and guidelines in §2.2 mentions ACM terminology but does not cite the primary ML reproducibility-policy references (e.g., the NeurIPS reproducibility checklist literature), which would strengthen the incentives discussion in §2.7.
- [§1 / Footnotes 1–2] The Introduction states that the ACM terminology is "still insufficient," and prior taxonomies are cited (Tatman et al. 2018; Gundersen et al. 2018), but the paper never systematically contrasts its eight categories with these frameworks; a brief comparison would help readers identify the novel contribution and would also address the category instability that Footnote 1 concedes.
- [§3 / Figure 1] Figure 1 distinguishes "hard dependencies" (solid lines) from "influencing effects" (dashed lines), but the criteria for this distinction are never defined, so the graph reads as a set of intuitions rather than a checkable model; a sentence explaining how a claimed dependency could in principle be tested would help.
Circularity Check
No significant circularity: the eight-way taxonomy is an inductive literature summary, and the Table 1 percentages are descriptive counts, not fitted predictions.
full rationale
This paper makes no formal derivation, fits no parameters, and predicts no quantity from fitted values. Its contribution is an inductive taxonomy: eight rigor aspects are proposed after a manual review of 101 self-identifying papers, and Table 1 reports the percentage of papers assigned to each aspect. Those percentages are arithmetic summaries of the authors' categorical assignments ('our subjective call'), not out-of-sample predictions, so the 'quantified proportion' claim is a descriptive count of the corpus rather than a result derived from an input in a way that closes a loop. The inclusion rule (self-identification with repeatability/reproducibility/replicability) is a sampling criterion, and any mismatch with Appendix A's looser 'in some sense' language is a potential validity threat to the denominator, but it is not a circular reduction. Several corpus entries are authored by Raff and colleagues, yet they are used as illustrations of existing work, not as load-bearing justification for the taxonomy; the taxonomy is anchored in a broad external bibliography. No uniqueness theorem, ansatz, or known result is renamed as a derivation. Accordingly, there are no circular steps; the weaknesses here are representativeness and inter-rater reliability, not circularity.
Assumptions & free parameters
assumptions (4)
- domain assumption The 101 papers that self-identify as reproducibility-related are representative of the community's usage of the term.
- domain assumption Each paper can be assigned a single primary category despite often touching multiple aspects.
- domain assumption The eight proposed categories are a natural and sufficiently complete partition of scientific rigor research in AI/ML.
- domain assumption ACM's Repeatability/Reproducibility/Replicability terminology is a valid starting point for the first three categories.
Cite this review
Pith. "Pith review of What Do Machine Learning Researchers Mean by "Reproducible"?." pith.science (2026). https://pith.science/paper/JSS4Q4CO
@misc{pith2026241203854,
author = {Pith},
title = {Pith review of: What Do Machine Learning Researchers Mean by "Reproducible"?},
year = {2026},
howpublished = {\url{https://pith.science/paper/JSS4Q4CO}},
note = {Machine review of arXiv:2412.03854}
}
read the original abstract
The concern that Artificial Intelligence (AI) and Machine Learning (ML) are entering a "reproducibility crisis" has spurred significant research in the past few years. Yet with each paper, it is often unclear what someone means by "reproducibility". Our work attempts to clarify the scope of "reproducibility" as displayed by the community at large. In doing so, we propose to refine the research to eight general topic areas. In this light, we see that each of these areas contains many works that do not advertise themselves as being about "reproducibility", in part because they go back decades before the matter came to broader attention.
Figures
Reference graph
Works this paper leans on
-
[7]
MORF: A Framework for Predictive Modeling and Replication At Scale With Privacy-Restricted MOOC Data. In 2018 IEEE International Conference on Big Data (Big Data), 3235–3244. Gardner, J.; Brooks, C.; and Baker, R. S. 2018. Enabling End-To-End Machine Learning Replicability : A Case Study in Educational Data Mining. In Reproducibility in ML W ork- shop, IC...
arXiv 2018
-
[8]
Reproducible Survival Prediction with SEER Cancer Data. In Doshi-V elez, F.; Fackler, J.; Jung, K.; Kale, D.; Ranganath, R.; Wallace, B.; and Wiens, J., eds., Proceed- ings of the 3rd Machine Learning for Healthcare Confer- ence, volume 85, 49–66. Palo Alto, California: PMLR. Se- ries Title: Proceedings of Machine Learning Research. Henderson, P .; Islam,...
work page 2018
-
[9]
Statistical Indistinguishability of Learning Algo- rithms. In Krause, A.; Brunskill, E.; Cho, K.; Engelhardt, B.; Sabato, S.; and Scarlett, J., eds., Proceedings of the 40th International Conference on Machine Learning , vol- ume 202 of Proceedings of Machine Learning Research , 15586–15622. PMLR. Kapoor, S.; and Narayanan, A. 2023. Leakage and the re- pr...
arXiv 2023
-
[13]
Understanding deep learning requires rethinking gen - eralization. In International Conference on Learning Repre- sentations. ArXiv: 1611.03530v2. Zhang, L.; Y ANG, J.; Karbasi, A.; and He, N. 2023. Opti- mal Guarantees for Algorithmic Reproducibility and Gradi- ent Complexity in Convex Optimization. In Thirty-seventh Conference on Neural Information Proc...
arXiv 2023
-
[2008]
Global Ecology and Biogeography , 17(2): 145–151
AUC: a misleading measure of the perfor- mance of predictive distribution models. Global Ecology and Biogeography , 17(2): 145–151. eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1466- 8238.2007.00358.x. Lopresti, D.; and Nagy, G. 2021. Reproducibility: Eval- uating the Evaluations. In Kerautret, B.; Colom, M.; Kr¨ ahenb¨ uhl, A.; Lopresti, D.;...
arXiv 1996
-
[2017]
Transac- tions of the Association for Computational Linguistics , 5: 471–486
Replicability Analysis for Natural Language Process - ing: Testing Significance with Multiple Datasets. Transac- tions of the Association for Computational Linguistics , 5: 471–486. Dror, R.; Baumer, G.; Shlomov, S.; and Reichart, R. 2018. The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing. In Proceedings of the 56th A...
arXiv 2018
-
[2018]
In 2018 44th Euromicro Conference on Software Engineer- ing and Advanced Applications (SEAA) , 50–59
Software Engineering Challenges of Deep Learning. In 2018 44th Euromicro Conference on Software Engineer- ing and Advanced Applications (SEAA) , 50–59. Arteaga, A.; Fuhrer, O.; and Hoefler, T. 2014. Designing Bit-Reproducible Portable High-Performance Application s. In 2014 IEEE 28th International Parallel and Distributed Processing Symposium, 1235–1244. A...
arXiv 2018
-
[2020]
On the Replicability and Reproducibility of Deep Learning in Software Engineering
On the Replicability and Reproducibility of Deep Learning in Software Engineering. 1(1): 1–34. ArXiv: 2006.14244. Liu, C.; Saul, R.; Sun, Y .; Raff, E.; Fuchs, M.; Pantano, T. S.; Holt, J.; and Micinski, K. 2024. Assemblage: Automatic Binary Dataset Construction for Machine Learning. In The Thirty-eight Conference on Neural Information Processing Systems ...
work page Pith review arXiv 2006
Show all 13 references
-
[2022]
In Proceedings of the 45th In- ternational ACM SIGIR Conference on Research and De- velopment in Information Retrieval , SIGIR ’22, 2790–2800
An Inspection of the Reproducibility and Replica- bility of TCT-ColBERT. In Proceedings of the 45th In- ternational ACM SIGIR Conference on Research and De- velopment in Information Retrieval , SIGIR ’22, 2790–2800. New Y ork, NY , USA: Association for Computing Machin- ery. I...
2009
-
[2023]
ACM Trans
A Systematic Study on Reproducibility of Reinforce- ment Learning in Recommendation Systems. ACM Trans. Recomm. Syst., 1(3). Chen, B.; Wen, M.; Shi, Y .; Lin, D.; Rajbahadur, G. K.; and Jiang, Z. M. J. 2022a. Towards training reproducible deep learning models. In Proceedings o...
2018 arXiv
-
[2024]
In Proceedings of the 2nd ACM Conference on Reproducibility and Replicability, ACM REP ’24, 36–45
Source Code Archiving to the Rescue of Reproducible Deployment. In Proceedings of the 2nd ACM Conference on Reproducibility and Replicability, ACM REP ’24, 36–45. New Y ork, NY , USA: Association for Computing Machin- ery. ISBN 9798400705304. Dacrema, M. F.; Cremonesi, P .; an...
2019
-
[2194]
Dror, R.; Baumer, G.; Bogomolov, M.; and Reichart, R
ArXiv: 1909.03004 Issue: 2. Dror, R.; Baumer, G.; Bogomolov, M.; and Reichart, R
1909 arXiv
-
[2932]
Esfandiari, H.; Kalavasis, A.; Karbasi, A.; Krause, A.; Mir - rokni, V .; and V elegkas, G
Series Title: Proceedings of Machine Learning Re- search. Esfandiari, H.; Kalavasis, A.; Karbasi, A.; Krause, A.; Mir - rokni, V .; and V elegkas, G. 2023a. Replicable Bandits. In The Eleventh International Conference on Learning Repre- sentations. Esfandiari, H.; Karbasi, A.;...
2018 arXiv
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.