Pith. sign in

REVIEW 6 minor 45 references

Reproducibility and Open Science in Lattice Quantum Field Theory

T0 review · 0 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The lattice quantum field theory community should routinely share field configurations, downstream data, and analysis workflows to make its published results independently reproducible.

desk verdict A useful, well-rooted community report on reproducibility in lattice QFT; the single cross-check is a robustness demonstration rather than exact reproduction, but that does not sink the recommendations. read the letter →

arxiv 2502.03593 v1 pith:B6WTHO2O submitted 2025-02-05 hep-lat

Ed Bennett , Andreas Athenodorou , Louise Chisholm , Anna Hasenfratz , Carsten Urbach This is my paper · ORCID
classification hep-lat
keywords reproducibilityopensciencelatticequantumfieldtheoryQCDdatasharingFAIRresearchsoftwareengineeringscientificworkflows
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper summarizes a panel discussion on reproducibility and open science in lattice quantum field theory at the field's 2024 conference. It argues that the community was an early adopter of open preprint sharing and a federated lattice data grid, but most published work still does not share the data or software workflows needed to rerun analyses. The central recommendation is to publish field configurations and downstream data products such as correlation functions together with the exact analysis code and workflow, so that independent groups can verify results. The paper reports a concrete success: a shared dataset was re-analyzed with a different workflow and produced consistent answers. It also gives practical advice on embargoes, storage, funding, and making openness count in hiring and promotion.

What carries the argument

The central mechanism is the reproducible workflow cycle: releasing the exact data with persistent identifiers, the exact analysis code, and the workflow that connects them, then letting independent researchers rerun or rebuild the analysis. The paper describes a federated data-grid infrastructure that supplies metadata schemas, binary formats, and searchable catalogs for field configurations, alongside persistent-identifier repositories that can hold smaller downstream products such as correlation functions. The working example is one shared dataset analyzed by two independent workflows that together produced a consistent result.

What would settle it

A systematic attempt to independently reproduce a random sample of lattice publications that have shared their data and workflows, counting how many reruns reproduce the published numbers without consulting the original authors; if most fail despite complete sharing, the paper's core premise is weakened.

Watch

Extended reading notes

Core claim

The paper's claim is that the lattice community's early infrastructure achievements have not been matched by routine sharing of the data and code that underlie published results. It asserts that reproducibility in the sense of 'same data, same analysis steps, same result' is almost never achieved when data and workflows are withheld, because a paper narrative cannot fully specify every step. The demonstrated proof of concept is the public release of gradient-flow data for a twelve-flavor SU(3) beta function and the subsequent independent re-analysis with a different workflow, which qualitatively reproduced the published result. From this, the paper concludes that consistent sharing of configurations, downstream data products, and analysis workflows would allow controversies between groups to be settled by running all groups' data through the same analysis, and would reduce duplicated computation.

Load-bearing premise

That sharing data and analysis code is sufficient for outsiders to independently verify published results; the one successful cross-check reported in the paper has not been shown to generalize across the field.

Editorial extensions

If this is right

  • If adopted, published lattice results become independently checkable by any group with the data and code, not only the originating collaboration.
  • Disputes between groups over contradictory results could be resolved by applying a common analysis to all groups' shared data rather than exchanging papers.
  • Sharing correlation functions alongside papers would let others reuse expensive downstream products without regenerating ensembles.
  • Open analysis workflows would expose numerical errors before publication, since the author reports finding inconsistencies in every retroactive automation exercise they ran.
  • Funder data policies already point toward this kind of sharing, so early adoption helps the community stay ahead of requirements.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If sharing becomes routine, the field could develop community benchmark datasets that test analysis codes against one another, in the spirit of standardized software test suites.
  • Measuring embargoes from first publication rather than generation date would let collaborations finish their analyses while still opening data early, a policy option the paper leaves open.
  • Sustained adoption would likely require hiring and promotion committees to count data and software contributions as research output, since the paper identifies cultural reluctance as a main barrier.
  • The single successful cross-check suggests a low-cost audit strategy for other contested results: release the data first and invite alternative workflows before the dispute intensifies.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

0 major / 6 minor

Summary. This arXiv:2502.03593 manuscript is a Lattice 2024 conference proceedings contribution that summarizes a panel discussion on reproducibility and open science in lattice field theory. It defines reproducibility, replicability, robustness, and FAIR principles; motivates open practices by appeals to scientific trust, public funding, energy use, and funder policies; provides context on the authors' data-sharing experiences, the International Lattice Data Grid, EU Open Science policies and e-infrastructures, SKA data practices, and cultural barriers; and then presents nine audience Q&A items with practical responses. The conclusions recommend more consistent sharing of gauge configurations, downstream data products such as correlation functions, and analysis workflows/software, and invite community involvement via ILDG mailing lists.

Significance. As a proceedings/position paper, the contribution is community-facing rather than technical: it consolidates current policy, infrastructure, and practical advice in one citable place. Strengths include accurate and well-referenced descriptions of EU Horizon Europe Open Science requirements, ILDG modernization, and concrete resources such as Zenodo limits, LaVA training, and the FAIR Data Accelerator. The authors are careful to define terms from The Turing Way and to label the Figure 1 comparison as qualitative verification; they also include a data-access statement. The single cross-check is illustrative, not proof, and the recommendations are plausible and aligned with funder mandates rather than derived from a formal analysis. The paper should be useful to lattice practitioners seeking entry points into open-science practices. The stress-test concern that Figure 1 overstates the evidence does not fully land, because the text explicitly says 'qualitatively verified' and does not claim exact reproduction under the paper's own definition.

minor comments (6)
  1. [Section 3.1 and Conclusions] The Fig. 1 example is explicitly 'qualitatively verified' and therefore demonstrates robustness under the Section 2 definitions (same data, different workflow), not exact reproducibility (same data, same analysis steps). The Conclusions phrase 'publishing data analysis workflows and software ... to enable reproducibility' should be tightened to 'to help enable reproducibility' or 'to make exact reproducibility checkable,' so that the recommendation is not read as being established by the Fig. 1 example.
  2. [Figure 1 caption] The caption says the right panel is 'presented in Ref. [10]', but the text says the workflow was used in Ref. [10] for a different theory; please clarify whether the right panel actually appears in Ref. [10] or is newly generated for this paper.
  3. [Section 2, Section 3.2, Section 3.3.3, Section 4.5] There are several typographical errors: 'whcih' should be 'which', 'authetication' should be 'authentication', 'Europen' should be 'European', and 'Peréz' should be 'Pérez'.
  4. [Section 4.1] The sentence 'in every case has identified and rectified inconsistencies either between the data and the text, or between different presentations of the same data in the same work' is a self-reported anecdote; adding 'in the author's experience' or similar would make the scope of the claim clear.
  5. [Section 4.4] 'There are no organisations that can realistically commit to maintaining storage indefinitely' is a strong universal; 'We know of no organisation that can realistically commit...' would be more precise.
  6. [References] The reference list uses inconsistent access-date formats (e.g., Refs. [25], [30], [45]); unify the date style.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is a panel report and policy discussion with no derivation chain, fitted parameters, or load-bearing self-citation.

full rationale

This paper does not perform a derivation, fit, or prediction; it summarizes a Lattice 2024 panel discussion and makes recommendations about reproducibility and open science. Its central claims are supported by surveys of community practice, funder policies, and concrete examples of data and workflow sharing. The self-citations that appear (Refs. [3,4,7,8,9,10,12,13,14,15,16]) serve as illustrative examples of open workflows or as sources of survey data, not as premises that force the conclusions. Section 3.1 explicitly describes the Fig. 1 comparison as an independent analysis workflow that 'qualitatively verified' previously published results, which is an external cross-check rather than a fitted input renamed as a prediction. Section 4.1 reports the author's experience that retrospective automation exposes errors, but this anecdote is not used as a mathematical premise and does not equate its input with its output. There are no equations whose outputs are identical by construction, no uniqueness theorem imported from the authors' prior work, and no ansatz smuggled in via citation. Any weakness in the evidence base, such as generalizing from a single qualitative cross-check to the recommendation that sharing artifacts 'enables reproducibility', is a matter of supporting evidence rather than circular reasoning. The report is therefore free of circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

This is a position paper rather than a derivation, so there are no free parameters or invented entities. The recommendations rest on normative and definitional premises about the value of open science and the definitions of reproducibility and FAIR, which are listed as domain assumptions.

assumptions (3)
  • domain assumption Reproducibility and openness are desirable goals for computational science.
    Section 2 opens with the Royal Society motto and defines reproducibility as a low bar that results should meet; this normative premise underlies every recommendation in the paper.
  • domain assumption Publicly funded research outputs should be shared with the public.
    Section 2 argues that because most lattice research is government funded, the public should benefit from the results; this is a value premise rather than an established fact.
  • domain assumption The Turing Way definitions of reproducibility, replicability, and robustness are accepted as authoritative.
    Section 2 imports these definitions without critical discussion; all later arguments use them as the framework.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reproducibility and Open Science in Lattice Quantum Field Theory." pith.science (2026). https://pith.science/paper/B6WTHO2O

@misc{pith2026250203593,
  author       = {Pith},
  title        = {Pith review of: Reproducibility and Open Science in Lattice Quantum Field Theory},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B6WTHO2O}},
  note         = {Machine review of arXiv:2502.03593}
}
read the original abstract

Reproducibility and Open Science are increasingly discussed as essential aspects of the research process. While there are areas where the Lattice community has been ahead of the curve with respect to the broader research world in this space, including early adoption of open publications via the arXiv, and the introduction of the International Lattice Data Grid in the 2000s, there are other areas where lattice practitioners could benefit from practices already adopted in other disciplines. In this Contribution, we report the outcomes of a panel discussion on this topic at the Lattice 2024 conference; after a discussion on motivations for work in this space, and introductory discussions of the relevant experiences of the panelists, we provide summaries of answers to the questions posed by the audience in the panel.

Figures

Figures reproduced from arXiv: 2502.03593 by the authors.

Figure 1
Figure 1. Left: The 𝛽 function of the SU(3) theory with twelve fermion flavours transforming in the fun￾damental representation, using ensembles generated with additional smeared Pauli–Villars fields, computed from data for the gradient flow of the Wilson action [7], as presented in Ref. [8]. Right: The same data [7], analysed using an alternative workflow [9], and presented in Ref. [10]. 3. Context In this section, we share … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 38 canonical work pages

  1. [8]

    Hasenfratz and C

    A. Hasenfratz and C. T. Peterson, Phys. Rev. D109, 114507 (2024),2402.18038

    arXiv 2024

  2. [1]

    The Turing Way Community,The Turing Way: A handbook for reproducible, ethical and collaborative research(Zenodo, 2023), URLhttps://the-turing-way.netlify.app/ reproducible-research/reproducible-research

    work page 2023

  3. [2]

    M. D. Wilkinson, M. Dumontier, I. J. Aalbersberg, G. Appleton, M. Axton, A. Baak, N. Blomberg, J.-W. Boiten, L. B. da Silva Santos, P. E. Bourne, et al., Scientific data3, 1 (2016)

    work page 2016

  4. [3]

    Status of reproducibility and open science in hep-lat in 2021

    E. Bennett, PoSLATTICE2022, 337 (2023),2211.15547

    work page Pith review arXiv 2023

  5. [4]

    A.Athenodorou,E.Bennett,J.Lenz,andE.Papadopoullou,PoS LATTICE2022,341(2023), 2212.04853

    work page Pith review arXiv 2023

  6. [5]

    arXiv: open access repository for e-prints, https://arxiv.org/(1991)

    work page 1991

  7. [6]

    Science and Technology Facilities Council (2016), URL https://www.ukri.org/ publications/stfc-scientific-data-policy/

    work page 2016

  8. [7]

    Peterson and A

    C. Peterson and A. Hasenfratz,Twelve flavor SU(3) gradient flow data for the continuous beta-function(2024), URLhttps://doi.org/10.5281/zenodo.10719052

    work page doi:10.5281/zenodo.10719052 2024

Show all 45 references
  1. [9]

    Twelve flavor SU(3) gradient flow data for the continuous beta- function

    E. Bennett,Alternative analysis workflow for "Twelve flavor SU(3) gradient flow data for the continuous beta- function"(2024), URLhttps://doi.org/10.5281/zenodo.13362605

    2024 doi

  2. [10]

    Bennett, A

    E. Bennett, A. Athenodorou, G. Bergner, P. Butti, and B. Lucini, in41st International Sym- posium on Lattice Field Theory(2024), 2410.19484

    2024 arXiv

  3. [11]

    TELOS Collaboration, https://telos-collaboration.github.io

  4. [12]

    Ostmeyer, A

    J. Ostmeyer, A. Sen, and C. Urbach (2024),2411.14981

    2024 arXiv

  5. [13]

    A.Athenodorou,E.Bennett,G.Bergner,P.Butti,J.Lenz,andB.Lucini(2024), 2408.00171

    2024 arXiv

  6. [14]

    E.Bennett,D.K.Hong,H.Hsiao,J.-W.Lee,C.J.D.Lin,B.Lucini,M.Piai,andD.Vadacchino (2024), 2412.01170

    2024 arXiv

  7. [15]

    Bennett, N

    E. Bennett, N. Forzano, D. K. Hong, H. Hsiao, J.-W. Lee, C. J. D. Lin, B. Lucini, M. Piai, D. Vadacchino, and F. Zierler, Phys. Rev. D110, 074504 (2024),2405.05765

    2024 arXiv

  8. [16]

    Bennett et al., Phys

    E. Bennett et al., Phys. Rev. D110, 074509 (2024),2405.01388

    2024 arXiv

  9. [17]

    C. T. H. Davies, A. C. Irving, R. D. Kenway, and C. M. Maynard (UKQCD), Nucl. Phys. B Proc. Suppl.119, 225 (2003),hep-lat/0209121

    2003 arXiv

  10. [18]

    140,213(2005), hep-lat/0409055

    C.M.MaynardandD.Pleiter,Nucl.Phys.BProc.Suppl. 140,213(2005), hep-lat/0409055

    2005 arXiv

  11. [19]

    Di Renzo, in40th International Symposium on Lattice Field Theory(2024), 2401.14752

    F. Di Renzo, in40th International Symposium on Lattice Field Theory(2024), 2401.14752. 16 Reproducibility and Open Science in Lattice Quantum Field Theory Ed Bennett et al

    2024 arXiv

  12. [20]

    Matsufuru, H

    H. Matsufuru, H. Simma, and C. Urbach, in41st International Symposium on Lattice Field Theory (2025)

    2025

  13. [21]

    PUNCH4NFDI,Particles,Universe,NuCleiandHadronsfortheNFDI:Aconsortiumforthe NFDI, URLhttps://www.punch4nfdi.de

  14. [22]

    OpenScienceClusters’ActionforResearch&Society, OSCARS:fosteringtheuptakeofOpen Science in Europe, URLhttps://www.oscars-project.eu

  15. [23]

    GÉANT, https://geant.org/

  16. [24]

    OpenAIRE, https://www.openaire.eu/

  17. [25]

    EUDAT, https://eudat.eu, accessed 2024-08-06

    2024

  18. [26]

    EGI, https://www.egi.eu/

  19. [27]

    PRACE, https://prace-ri.eu/

  20. [28]

    FAIR data accelerator pilot, http://www.uksrc.org/fair-data-pilot/

  21. [29]

    D. del Fierro et al.,Identity negotiation and professional development in the context of the digital transformation of research communities(2023), URLhttps://doi.org/10.5281/ zenodo.12806507

    2023

  22. [30]

    Zenodo, https://www.zenodo.org, accessed 2024-08-06

    2024

  23. [31]

    Zenodo, General policies v1.0, https://about.zenodo.org/policies/, accessed 2024- 08-06

    2024

  24. [32]

    Pérez and B

    F. Pérez and B. E. Granger, Computing in Science and Engineering9, 21 (2007), ISSN 1521-9615, URLhttps://ipython.org

    2007

  25. [33]

    Puebla, G

    I. Puebla, G. A. Ascoli, J. Blume, J. Chodacki, J. Finnell, D. N. Kennedy, B. Mair, M. E. Martone, J. Wittenberg, and J.-B. Poline, PLoS computational biology20, e1012296 (2024)

    2024

  26. [34]

    UK Reproducibility Network, Recognising and rewarding open research: imple- mentation guide (2024), URL https://recognition.ukrn-openresearch.ac.uk/ guide-contents.html

    2024

  27. [35]

    The Lattice Virtual Academy, https://www.ectstar.eu/virtual-platforms/

  28. [36]

    National Aeronautics and Space Administration,Transform to Open Science, URL https: //science.nasa.gov/open-science/tops/

  29. [37]

    net/wiki/spaces/ESDKB/pages/1920827407/2024+Call+for+Proposals

    S.GoedhartandF.Camilo, 2024callforproposals ,URL https://skaafrica.atlassian. net/wiki/spaces/ESDKB/pages/1920827407/2024+Call+for+Proposals. 17 Reproducibility and Open Science in Lattice Quantum Field Theory Ed Bennett et al

    2024

  30. [38]

    Space Telescope Science Institute (2023), URL https:// jwst-docs.stsci.edu/jwst-opportunities-and-policies/ past-jwst-proposal-opportunities/jwst-call-for-proposals-for-cycle-3/ jwst-data-rights-and-duplications-cycle-3#gsc.tab=0

    2023

  31. [39]

    Clery, Science (2022), URLhttps://doi.org/10.1126/science.adf6204

    D. Clery, Science (2022), URLhttps://doi.org/10.1126/science.adf6204

    2022 doi

  32. [40]

    Rantaharju, E

    J. Rantaharju, E. Bennett, M. Dawson, and M. Mesiti, PoSLATTICE2018, 039 (2018), 1806.06043

    2018 arXiv

  33. [41]

    Hettrick, R

    S. Hettrick, R. Bast, S. Crouch, C. Wyatt, O. Philippe, A. Botzki, J. Carver, I. Cosden, F.D’Andrea,A.Dasgupta,etal., InternationalRSESurvey2022 , https://softwaresaved. github.io/international-survey-2022/

    2022

  34. [42]

    Journal of Open Source Software, https://joss.theoj.org

  35. [43]

    Fanelli, Scientometrics90, 891 (2012)

    D. Fanelli, Scientometrics90, 891 (2012)

    2012

  36. [44]

    Mlinarić, M

    A. Mlinarić, M. Horvat, and V. Šupak Smolčić, Biochemia medica27, 447 (2017)

    2017

  37. [45]

    International Lattice Data Grid,Organization of ILDG activities, https://hpc.desy.de/ ildg/organization/, accessed 2024-08-06. 18

    2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.