yProv4ML: Effortless Provenance Tracking for Machine Learning Systems

Gabriele Padovani; Sandro Fiore; Valentine Anantharaj

REVIEW 4 major objections 5 minor 1 cited by

yProv4ML: Effortless Provenance Tracking for Machine Learning Systems

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read yProv4ML records datasets, hyperparameters, metrics, and emissions from ML runs as a standard PROV-JSON provenance graph.

desk verdict A modest but real software contribution—an MLFlow-style provenance logger that outputs W3C PROV-JSON—whose main gap is the absence of any validation that the emitted files are actually conformant. read the letter →

arxiv 2507.01078 v1 pith:MAIW76IX submitted 2025-07-01 cs.LG cs.DC

Gabriele Padovani , Valentine Anantharaj , Sandro Fiore This is my paper · ORCID

classification cs.LGcs.DC

keywords machinelearningprovenanceyProv4MLPROV-JSONgraphexperimenttrackingreproducibilitycarbonmetrics

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces yProv4ML, a Python library that captures what happens during a machine-learning run—datasets, hyperparameters, metrics, system usage, carbon emissions, and model versions—and writes it into a PROV-JSON provenance graph following the W3C PROV standard. The main claim is that this can be done with minimal changes to existing training code, because the logging calls are shaped like those of familiar experiment-tracking tools. The motivation is reproducibility: current trackers store experiment metadata in proprietary formats and pay little attention to lineage, so users cannot easily trace how a model was produced or reuse knowledge from earlier runs. If the library works as described, ML teams can obtain an interoperable, human-readable lineage record that also supports energy-consumption monitoring and cross-run comparison, without being locked into a closed format.

What carries the argument

The load-bearing machinery is the set of yProv4ML directives and the in-memory data model they feed. The directives deliberately resemble the interfaces of existing experiment-tracking systems (start_run, log_param, log_metric, and so on) so that a user can adopt the library without restructuring a training script. The data model accumulates logged parameters (one-time values), metrics (time-series values with context and step), artifacts (files), system metrics (memory, disk, and GPU usage), carbon metrics (emissions and power), model configurations, and checkpoint versions. At end_run this data is serialized into a PROV-JSON graph, and optionally into an SVG or DOT rendering, making the provenance navigable and interoperable.

What would settle it

Run the paper's MNIST example, then apply an extra data transformation on the fly during training without logging it; if the generated PROV-JSON graph does not show that transformation as an activity connected to the dataset, the claim of capturing the data manipulation process is refuted.

Watch

Extended reading notes

Core claim

In the paper's own terms, the central discovery is that a small set of logging directives—start_run, log_param, log_metric, log_system_metrics, log_carbon_metrics, log_artifact, save_model_version, log_model, and end_run—is sufficient to produce a standard W3C PROV-JSON graph capturing the lineage of a training process. The graph links user-logged items with automatically recorded information such as environment variables and required libraries, and each process's graph can be merged into a collection for distributed runs. The authors demonstrate the output on an MNIST classification example and report using the library in large-scale benchmarks and a tropical-cyclone prediction project, arguing that the approach provides a transparent, interoperable alternative to proprietary experiment metadata.

Load-bearing premise

The framework can only record what the user's logging calls explicitly capture; if a meaningful step such as a data transformation inside a custom module is never logged, the provenance graph will be missing that step, and the library does not automatically instrument the code to fill such gaps.

Editorial extensions

If this is right

Users can obtain a standards-compliant provenance record of an ML run by inserting a handful of logging calls, without changing the training logic or switching to a proprietary tracking service.
Because the output is PROV-JSON, provenance can be processed, visualized, and exchanged with standard W3C PROV tooling, removing the lock-in of closed experiment-metadata formats.
When combined with workflow-level provenance libraries from the same yProv family, the ML-level graph nests inside a higher-level workflow graph, giving end-to-end lineage from raw data through model training.
Logging system and carbon metrics per run makes it possible to compare energy use and emissions across experiments, supporting sustainable model-development choices.
Storing hyperparameters and model versions per run lets researchers look up prior experiments and reuse good parameter choices instead of re-running costly searches.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The goal of provenance capture with minimal user effort depends on users remembering to call a logging directive for every piece of information they care about; an obvious testable extension is to combine yProv4ML with automatic instrumentation of the deep-learning framework's autograd graph so that implicit transformations inside custom modules are captured without manual calls.
The paper does not quantify the overhead of the logging calls; a natural benchmark would measure the runtime and memory cost of a run with yProv4ML versus an identical run without it, and versus a run using an existing tracker, to test whether minimal code changes also mean minimal performance loss.
Since PROV-JSON is a W3C standard, a further consequence is that provenance records produced by different tools could be merged across a project, enabling cross-framework lineage queries; yProv4ML would then serve as a bridge rather than an endpoint.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short SoftwareX-style software description. The genuinely new bit is combining the MLFlow-style logging API with PROV-JSON serialization. That fills a real gap: MLFlow stores in its own format, and PROV-JSON is a W3C standard, so a bridge between the two is useful for the provenance and reproducibility crowd. The API is clearly described, the code example is coherent, and the repo is public with a GPLv3 license. Credit where due: the modular design (energy, system, time modules) is sensible, and the authors clearly understand the provenance modeling space.

The soft spots are proportionate. The biggest one, and the stress-test note gets this right, is that the paper never shows a single emitted PROV-JSON file or validates one against the schema. The central claim is that the serialization is "in accordance with the W3C PROV standard," and that assertion is exactly what needs evidence. The dependency on the Prov Python package mitigates the risk—that library is mature and likely produces valid output—but the manuscript doesn't demonstrate it. A screenshot of an SVG graph is not the same as a schema-valid JSON document, and the claimed interoperability with other PROV tooling rests on it.

The completeness concern the reader raised is real but less load-bearing. The paper explicitly says it records what the user elects to track, so the gap of implicit dependencies is acknowledged by scope. Still, "full lineage" in the introduction is oversold; it's a minor overclaim.

What's missing is any systematic evaluation: no comparison against MLFlow, no overhead measurements, no memory footprint numbers, no round-trip parse. For a software paper, that's more than cosmetic; the claim of "minimal code modifications" is asserted, not shown. These are all addressable in revision.

Bottom line: this is a modest but honest contribution. The authors ship code and describe it accurately, modulo the unvalidated PROV-JSON claim. It deserves a serious referee—someone should check the repo, run a small test, and ask for a sample output file and a benchmark. I'd send it to peer review rather than desk reject, with the expectation of a revision adding validation and evaluation.

Referee Report

4 major / 5 minor

Summary. The paper presents yProv4ML, a Python library for provenance tracking in machine learning workflows. It offers an MLFlow-like API (start_run, log_param, log_metric, log_system_metrics, log_carbon_metrics, log_artifact, save_model_version, end_run) that collects datasets, hyperparameters, metrics, system statistics, carbon emissions, and model checkpoints, and claims to serialize the collected information into PROV-JSON, the W3C standard provenance format. The manuscript describes the library's architecture, shows a small PyTorch code snippet for MNIST training, displays two plots of logged metrics, and includes a screenshot of a provenance graph. It also mentions deployments at ORNL on over 500 GPUs and for tropical cyclone prediction, and states that the output is interoperable with other yProv libraries. The paper does not contain a systematic evaluation: there is no comparison with MLFlow or other tools, no measurement of runtime or memory overhead, no validation of the PROV-JSON output against a schema, and no quantitative evidence from the claimed large-scale deployments.

Significance. If the central claim holds, yProv4ML would be a useful contribution: it would let ML practitioners obtain structured, standards-based provenance records with an API similar to MLFlow, thereby improving reproducibility and enabling lineage-aware comparison of experiments. The use of the established PROV-JSON format is a genuine strength over proprietary logging systems, and the claimed interoperability with the yProv framework could be valuable in scientific workflows. The paper's main weakness is the absence of evidence: the PROV-JSON compliance is not demonstrated, the completeness of the captured provenance is undefined, and the performance impact is unmeasured. These gaps are fixable, since the implementation is open-source and could be supplemented with output examples, schema-validation tests, and overhead benchmarks. As presented, however, the paper reads more as a feature announcement than as a verified software contribution.

major comments (4)

[Section 5 and abstract] The central claim that yProv4ML serializes 'in accordance with the W3C PROV standard' and produces PROV-JSON output is not demonstrated anywhere in the manuscript. No actual PROV-JSON file is shown, no validation against the PROV-JSON schema is reported, and no test checks that the emitted graph contains the expected PROV entities, activities, and relations (used, wasGeneratedBy, wasAttributedTo). Since this is the paper's core value proposition, the authors should provide at least one self-contained output example and a validation/round-trip test, and report the result.
[Section 3 and Section 5] The evaluation section is essentially absent: Figure 2 shows two metric plots and Figure 3 shows a graph screenshot, but there is no quantitative assessment of the library. The authors should report the runtime and memory overhead relative to a plain PyTorch run, compare the API and output interoperability with MLFlow, and, if feasible, provide concrete data (e.g., graph size, serialization time) from the claimed 500-GPU deployment. Without such measurements, the 'effortless' and 'lightweight' claims are unsupported.
[Section 2.2 and 2.3] The completeness of the provenance graph is entirely determined by user-inserted directives; there is no automatic instrumentation of implicit dependencies. The code snippet also calls log_dataset, but this directive is not described in Section 2.2, which lists only log_model, log_param, log_metric, log_system_metrics, log_carbon_metrics, log_artifact, save_model_version, and log_current_execution_time. The authors should define log_dataset (and any other omitted directives) and explicitly state that data transformations applied inside custom PyTorch modules are outside the library's scope, so that readers can judge what 'effortless tracking' actually covers.
[Section 4] The impact section states that yProv4ML has been used on more than 500 GPUs at ORNL and for tropical cyclone prediction, but provides no evidence—no logs, no configurations, no reproducible artifacts, and no link to the resulting provenance data. If these are anecdotal application notes, that should be stated; if they are meant to demonstrate scalability, quantitative results and a clear description of the deployment setup are required.

minor comments (5)

[Section 2.2] The description of log_metric says it stores 'a sequence of parameters', but it actually stores a sequence of metric values; this is a wording inconsistency that could confuse readers.
[Section 2.3] The code snippet uses prov4ml.log_dataset(train_loader, 'train_dataset'), but this function is not listed among the directives in Section 2.2; please add it or adjust the snippet.
[Section 1] The phrase 'works conducted with less rigor and in a superficial way' and the argument that hyperparameter uncertainty 'presents challenges in identifying the best model' are imprecise; the motivation should focus on the lack of provenance and lineage information rather than on hyperparameter tuning difficulty.
[General] There are several typographical and stylistic inconsistencies: 'superceded' should be 'superseded' (Section 1), 'on over than 500 GPUs' should be 'on over 500 GPUs' (Section 4), and the formatting of 'prov4ml. start run' in Section 2.3 introduces spurious spaces from the line break.
[Table 1] The dependencies listed in C7 ('Codecarbon, Prov, Pytorch') lack version numbers; please specify the minimum supported versions to aid reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: yProv4ML is a software description whose claims are self-contained and not defined in terms of its own output.

full rationale

This manuscript is a software description of yProv4ML, not a derivation or prediction pipeline. There are no fitted parameters, no equations whose outputs are fed back into their own inputs, and no statistical claim that reduces to a fit by construction. The central claim is that the library provides an MLFlow-like API for logging parameters, metrics, artifacts, system metrics, and carbon metrics, and that the collected information is serialized to PROV-JSON. None of this is defined in terms of the library's own outputs: the logging directives are user-invoked functions, and the PROV-JSON serialization is the stated output format, not a premise used to justify the format. References to prior work, including the yProv framework [10], are contextual and do not carry a load-bearing argument; the paper does not invoke a uniqueness theorem or an author-derived ansatz to force its design choices. The strongest substantive concern raised by the skeptic is that the manuscript does not demonstrate through schema validation or round-trip parsing that the emitted files are conformant PROV-JSON. That is a validation and evidence gap, not a circularity: it concerns whether an assertion is empirically supported, not whether the assertion reduces to its own inputs. The paper also explicitly scopes provenance capture to what the user explicitly logs, which may limit completeness, but that is an acknowledged limitation of coverage rather than a circular step. Therefore no specific circular step can be identified, and the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The paper introduces no free parameters or invented entities. It relies on the adequacy of the PROV standard to represent ML provenance (a domain assumption) and on user logging calls being sufficient to record lineage (also a domain assumption).

assumptions (2)

domain assumption User-initiated logging directives capture the full provenance of an ML process.
The tool only records what the user explicitly logs; the paper presents this as a complete provenance record, but there is no automatic instrumentation to capture implicit dependencies.
domain assumption The PROV data model is adequate for representing ML training provenance.
The paper maps ML concepts onto PROV entities and activities without demonstrating that all ML lineage can be expressed in PROV.

how reviews work

0 comments

Cite this review

Pith. "Pith review of yProv4ML: Effortless Provenance Tracking for Machine Learning Systems." pith.science (2026). https://pith.science/paper/MAIW76IX

@misc{pith2026250701078,
  author       = {Pith},
  title        = {Pith review of: yProv4ML: Effortless Provenance Tracking for Machine Learning Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MAIW76IX}},
  note         = {Machine review of arXiv:2507.01078}
}

read the original abstract

The rapid growth of interest in large language models (LLMs) reflects their potential for flexibility and generalization, and attracted the attention of a diverse range of researchers. However, the advent of these techniques has also brought to light the lack of transparency and rigor with which development is pursued. In particular, the inability to determine the number of epochs and other hyperparameters in advance presents challenges in identifying the best model. To address this challenge, machine learning frameworks such as MLFlow can automate the collection of this type of information. However, these tools capture data using proprietary formats and pose little attention to lineage. This paper proposes yProv4ML, a framework to capture provenance information generated during machine learning processes in PROV-JSON format, with minimal code modifications.

Figures

Figures reproduced from arXiv: 2507.01078 by the authors.

**Figure 3.** Fig.3. In this use case the machine learning application is a simple MNIST [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

**Figure 2.** Left: the training loss is recorded for every epoch using yProv4ML directives and [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

Figures from the paper (1 more)

**Figure 3.** Figure 3: Provenance graph of the ML training process specified in Section 3 (Illustrative 12 [PITH_FULL_IMAGE:figures/full_fig_p012_3.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

OriginBlame: Record- and Token-Level Data Provenance for AI Training Datasets
cs.AI 2026-05 conditional novelty 6.0 of 10

A record- and token-level provenance system that turns author revocation requests into precise forget sets, cutting dataset-level over-deletion from 101x to 1.3x on wiki data.

Reference graph

Works this paper leans on

13 extracted references · 11 canonical work pages · cited by 1 Pith paper

[1]

Reproducibility in machine learning-driven re- search,

H. Semmelrock et al. , “Reproducibility in machine learning-driven re- search,” arXiv preprint arXiv:2307.10320 , 2023

arXiv 2023
[2]

Challenges for the repeatability of deep learning models,

S. S. Alahmari et al., “Challenges for the repeatability of deep learning models,” IEEE Access, vol. 8, pp. 211 860–211 868, 2020

work page 2020
[3]

Out-of-the-box reproducibility: A sur- vey of machine learning platforms,

R. Isdahl and O. E. Gundersen, “Out-of-the-box reproducibility: A sur- vey of machine learning platforms,” in 2019 15th international confer- ence on eScience (eScience) . IEEE, 2019

work page 2019
[4]

Deephyper: Asynchronous hyperparameter search for deep neural networks,

P. Balaprakash et al. , “Deephyper: Asynchronous hyperparameter search for deep neural networks,” in 2018 IEEE 25th international con- ference on high performance computing (HiPC) . IEEE, 2018. 10

work page 2018
[5]

Provenance: a future history,

J. Cheney et al. , “Provenance: a future history,” in Proceedings of the 24th ACM SIGPLAN conference companion on Object oriented program- ming systems languages and applications , 2009

work page 2009
[6]

Towards lightweight data integration using multi-workflow provenance and data observability,

R. Souza, T. J. Skluzacek, S. R. Wilkinson, M. Ziatdinov, and R. F. da Silva, “Towards lightweight data integration using multi-workflow provenance and data observability,” in 2023 IEEE 19th International Conference on e-Science (e-Science) . Limassol, Cyprus: IEEE, 2023, pp. 1–10

work page 2023
[7]

Workflow provenance in the computing continuum for responsible, trustworthy, and energy-efficient ai,

R. Souza et al., “Workflow provenance in the computing continuum for responsible, trustworthy, and energy-efficient ai,” in 2024 IEEE 20th In- ternational Conference on e-Science (e-Science). Osaka, Japan: IEEE, 2024, pp. 1–7

work page 2024
[8]

Accelerating the machine learning lifecycle with mlflow,

M. Zaharia et al. , “Accelerating the machine learning lifecycle with mlflow,” IEEE Data Eng. Bull. , vol. 41, no. 4, pp. 39–45, 2018

work page 2018

Show all 13 references

[9]

Interoperability for provenance-aware databases using PROV and JSON,

X. Niu et al. , “Interoperability for provenance-aware databases using PROV and JSON,” in 7th USENIX Workshop on the Theory and Prac- tice of Provenance (TaPP 15) , 2015

2015
[10]

A graph data model-based micro-provenance approach for multi-level provenance exploration in end-to-end climate workflows,

S. Fiore et al. , “A graph data model-based micro-provenance approach for multi-level provenance exploration in end-to-end climate workflows,” in 2023 IEEE International Conference on Big Data (BigData) . IEEE, 2023

2023
[11]

The mnist database of handwritten digit images for machine learning research,

L. Deng, “The mnist database of handwritten digit images for machine learning research,” IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 141–142, 2012

2012
[12]

Exploring vision transformers on the frontier supercomputer for remote sensing and geoscientific applications,

V. Anantharaj et al. , “Exploring vision transformers on the frontier supercomputer for remote sensing and geoscientific applications,” in IGARSS 2024-2024 IEEE International Geoscience and Remote Sensing Symposium. IEEE, 2024

2024
[13]

A software ecosystem for multi-level provenance man- agement in large-scale scientific workflows for ai applications,

P. Gabriele et al., “A software ecosystem for multi-level provenance man- agement in large-scale scientific workflows for ai applications,” in SC24- W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis . IEEE, 2024. 11 F...

2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Reproducibility in machine learning-driven re- search,

H. Semmelrock et al. , “Reproducibility in machine learning-driven re- search,” arXiv preprint arXiv:2307.10320 , 2023

arXiv 2023

[2] [2]

Challenges for the repeatability of deep learning models,

S. S. Alahmari et al., “Challenges for the repeatability of deep learning models,” IEEE Access, vol. 8, pp. 211 860–211 868, 2020

work page 2020

[3] [3]

Out-of-the-box reproducibility: A sur- vey of machine learning platforms,

R. Isdahl and O. E. Gundersen, “Out-of-the-box reproducibility: A sur- vey of machine learning platforms,” in 2019 15th international confer- ence on eScience (eScience) . IEEE, 2019

work page 2019

[4] [4]

Deephyper: Asynchronous hyperparameter search for deep neural networks,

P. Balaprakash et al. , “Deephyper: Asynchronous hyperparameter search for deep neural networks,” in 2018 IEEE 25th international con- ference on high performance computing (HiPC) . IEEE, 2018. 10

work page 2018

[5] [5]

Provenance: a future history,

J. Cheney et al. , “Provenance: a future history,” in Proceedings of the 24th ACM SIGPLAN conference companion on Object oriented program- ming systems languages and applications , 2009

work page 2009

[6] [6]

Towards lightweight data integration using multi-workflow provenance and data observability,

R. Souza, T. J. Skluzacek, S. R. Wilkinson, M. Ziatdinov, and R. F. da Silva, “Towards lightweight data integration using multi-workflow provenance and data observability,” in 2023 IEEE 19th International Conference on e-Science (e-Science) . Limassol, Cyprus: IEEE, 2023, pp. 1–10

work page 2023

[7] [7]

Workflow provenance in the computing continuum for responsible, trustworthy, and energy-efficient ai,

R. Souza et al., “Workflow provenance in the computing continuum for responsible, trustworthy, and energy-efficient ai,” in 2024 IEEE 20th In- ternational Conference on e-Science (e-Science). Osaka, Japan: IEEE, 2024, pp. 1–7

work page 2024

[8] [8]

Accelerating the machine learning lifecycle with mlflow,

M. Zaharia et al. , “Accelerating the machine learning lifecycle with mlflow,” IEEE Data Eng. Bull. , vol. 41, no. 4, pp. 39–45, 2018

work page 2018