{"id":"690be79b-3408-49d0-b0b4-0f3b3cda93ee","arxiv_id":"2507.01075","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"yProv4ML is a new provenance-tracking library for ML workflows that logs experiments as W3C PROV-compliant provenance graphs and demonstrates its use in large-scale distributed training studies.","lead":"This paper presents yProv4ML, a library that records machine learning experiment provenance in W3C PROV-compliant JSON, and demonstrates it on scaling runs of a vision foundation model on the Frontier supercomputer. The tool aims to give finer-grained lineage than common trackers like MLFlow, with optional Zarr/NetCDF storage that the authors report reduces file sizes by over 90 percent.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The >90% file-size-reduction claim (Section 4, Table 1) rests on a single file, yet the text says 'on average'; if that example is unrepresentative, a central advertised benefit of yProv4ML is unestablished.","rationale":"I read the paper as a software/tool contribution: the deliverable is a working provenance library, evidenced by the GitHub repository and a real Frontier use case. The strongest claim the paper can defend is that the library collects structured, PROV-compliant lineage with a modular plugin interface. However, the headline quantitative claim used to motivate the design—the >90% size reduction—is not yet supported. The text itself calls the work 'Preliminary' and Table 1 contains a single row with no distribution, so the reader's concern is precisely on target. The MODIS-FM use case is a demonstration, not a controlled study; its empty cells and lack of error bars are secondary. I agree with the reader's weakest assumption. Because the central claim is a performance claim about the library, and because the library appears otherwise functional, the appropriate verdict remains CONDITIONAL rather than ACCEPT or REJECT. No change to the reader's verdict is needed.","tokens_in":11108,"tokens_out":4558,"duration_ms":49934,"concrete_test":"Run the size comparison on at least 20 diverse training runs (vary model size, logging frequency, and metric types: scalars, histograms, time series), storing the same metrics as JSON, Zarr, and NetCDF with the same numeric precision (test both float32 and float64). Report the median and interquartile range of total provenance size, both uncompressed and gzip-compressed. If the median reduction is below 90% or the range spans more than 20 percentage points, weaken the claim to a range with the protocol described. As a secondary check, validate the generated PROV-JSON against the W3C PROV-JSON schema to confirm the standard-compliance claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's most concrete quantitative claim is that storing bulky metrics in Zarr/NetCDF instead of JSON yields 'gains of more than 90% on average' (Section 4). That claim is supported only by Table 1, a single file (39.82 MB JSON vs 2.74 MB Zarr, 2.35 MB NetCDF). No measurement protocol is given: we do not know the metric types, numeric precision, logging frequency, or how 'bulky' was defined. The caption says the comparison includes both PROV-JSON and metric files, but the original is one JSON while the converted version splits a small PROV-JSON from the binary metric store, so the comparison depends on how much of the 39.82 MB is textual JSON overhead versus actual data. If the JSON payload uses high-precision decimal strings (e.g., 17-digit floats) and the binary formats store float32, part of the reduction is an artifact of representation, not of the binary container. Moreover, the 'compressed size' column shows a smaller benefit (8.65 MB vs 2.14 MB, ~75%), so the >90% figure applies only to uncompressed storage and would be further reduced in settings where files are gzip-compressed anyway. A single example cannot support an average; the use case on Frontier (Section 5) does not report file sizes across runs, and the finer-grained-lineage claim is not benchmarked against MLflow/W&B. If the reduction is smaller or more variable in typical training workloads, the main advertised benefit of the library is weakened, though the library itself may still be functional.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents yProv4ML, an open-source provenance collection library for machine learning workflows, integrated with the yProv framework and compliant with W3C PROV and ProvML standards. The library logs parameters, metrics, and artifacts, supports user-defined contexts, and can store bulky time-series metrics in Zarr or NetCDF to keep the main PROV-JSON file small. The authors report a file-size reduction of more than 90% for a single representative file (Table 1) and illustrate the tool on a MODIS-FM scaling study on Frontier, where GPU energy consumption and loss are used to identify trade-offs between model size and compute configuration (Figure 3). The paper also describes planned reproducibility features, including git tracking and RO-Crate packaging.","tokens_in":11423,"tokens_out":3336,"duration_ms":35732,"significance":"If the reported benefits are substantiated, yProv4ML would be a useful contribution to ML provenance practice: it provides a standards-based, extensible logging interface with finer-grained lineage than typical experiment trackers, and the Frontier use case demonstrates real-world deployment on a leadership-class supercomputer. The paper's strengths include the public availability of the library, its integration with the yProv ecosystem and workflow tools, and its concrete energy-versus-performance illustration. However, the current evaluation is preliminary: the headline size-reduction figure rests on one measurement, and the scaling-study conclusions are drawn from an incomplete run grid without statistical support.","major_comments":[{"comment":"The claim that storing bulky metrics in Zarr or NetCDF yields 'gains of more than 90% on average' is supported by a single file comparison (39.82 MB JSON vs 2.74 MB Zarr and 2.35 MB NetCDF). The paper does not report the metric types, numeric precision, logging frequency, or the fraction of the original JSON that is textual overhead, so the reader cannot judge whether the reduction is representative or partly an artifact of switching from high-precision decimal strings to binary float32. The compressed-size column shows a much smaller benefit (8.65 MB vs 2.14-2.30 MB, about 75%), so the >90% figure applies only to uncompressed storage. Because file-size reduction is one of the two central advertised benefits, this claim needs to be supported by multiple runs, with protocol details and an analysis of variability.","section":"Section 4, Table 1"},{"comment":"The conclusion that 'the best trade-off between energy consumption and performance is obtained' and that MAE 'presents a steeper trade-off curve' is drawn from Figure 3, in which several configurations are empty because they exceeded the 2-hour walltime. With no statistical analysis, no error bars, and no statement of how the missing cells were treated, the qualitative ranking of architectures and configurations is not yet established. The authors should either restrict the claims to the completed cells, provide a sensitivity analysis, or add error estimates from repeated runs.","section":"Section 5, Figure 3"},{"comment":"The paper states that the core idea of the library is to log more lineage information compared to MLFlow or Weights and Biases, but it does not provide a concrete comparison of the lineage captured by these tools. A feature-by-feature table or a small benchmark showing the additional lineage elements (e.g., contexts, epochs, input/output relationships) would substantiate this central design goal and make the contribution easier to evaluate.","section":"Section 4"}],"minor_comments":[{"comment":"There are several typographical errors, including 'shear amount' (should be 'sheer amount'), 'conjuction' (should be 'conjunction'), and 'are have already been shown' (grammar).","section":"Abstract and Introduction"},{"comment":"In the Related Work section, 'moltitude' should be 'multitude'.","section":"Section 2"},{"comment":"The color scale units ('1e6') and the empty cells should be explained in the caption; also consider labeling axes more clearly and using a perceptually uniform colormap.","section":"Figure 3"},{"comment":"The caption says the measurements include both the PROV-JSON and the additional metric files, but the row names 'Original_file.json', 'Converted_to.zarr', and 'Converted_to.nc' could be clearer about what is being compared, namely a single JSON file versus a split PROV-JSON plus binary metric store.","section":"Table 1"},{"comment":"In the conclusions, 'tracking all experiment runs in a single provenance files' has a plural/singular mismatch; 'files' should be 'file'.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads more like a software demonstration or short workshop paper than a full research article; the core tool appears to exist and work, but the evaluation is preliminary. The heavy reliance on the authors' own prior work is understandable for an ecosystem paper, though a reviewer might ask for an explicit statement of what is new in this version beyond [12]. I recommend major revision rather than rejection because the gaps are empirical and can be addressed within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The gist: this is a solid systems paper about a real software artifact, not a scientific breakthrough. yProv4ML is a provenance collection library that plugs into the existing yProv framework, produces W3C PROV-compliant JSON, and now can push bulky metrics into Zarr or NetCDF while wrapping artifacts in RO-Crates. The MODIS-FM scaling study on Frontier is a genuine demonstration, and the library is open source. Those are real contributions, and the paper is honest about being a work-in-progress.\n\nThe weak spot is the one quantitative headline: “gains of more than 90% on average.” Table 1 shows exactly one file (39.82 MB JSON vs. 2.74 MB Zarr, 2.35 MB NetCDF). The compressed-size column tells a different story: 8.65 MB vs. 2.14 MB and 2.30 MB, which is ~75% reduction, not >90%. There is no described measurement protocol, no statement about metric type or numeric precision, and no definition of “bulky.” So the “average” is not established, and even the example’s benefit depends heavily on whether you compare compressed or uncompressed. This is a research-quality issue, not a fatal one: the library works, and the size reduction is plausible. But the authors need to either drop the word “average” or provide a real set of runs with precision and format details.\n\nThe scaling study is also thin. Figure 3 has empty cells for runs that exceeded the 2-hour walltime, and the conclusions about “best trade-off” and “steeper curve” are drawn from an incomplete grid without error bars or statistical support. Fine for a use case, but it should be labeled as anecdotal.\n\nThe finer-grained-lineage claim versus MLflow/W&B is not benchmarked, just asserted. That is a minor issue for a tool paper, but a comparison would strengthen it.\n\nOverall, this deserves peer review, not desk rejection. Send it to a referee who knows the provenance/MLOps space, and the main request should be: more data on the compression claim, a clean measurement protocol, and a clearer separation between the library’s functionality and the case-study observations. If the compression claim holds up across a few typical workloads, this becomes a useful community resource. If not, it is still a functional library, just with a less impressive advantage.","headline":"A genuinely useful provenance library, but the headline compression claim rests on a single example and needs a real measurement protocol before it can be trusted.","tokens_in":11942,"tokens_out":2250,"would_cite":false,"duration_ms":29000,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces yProv4ML, a provenance collection library for machine learning that records experiment lineage at a finer granularity than existing trackers, in W3C PROV-compliant JSON, and reports more than 90 percent smaller…","keywords":["provenance","machine learning","PROV-JSON","W3C PROV","experiment tracking","energy efficiency","Zarr","NetCDF"],"falsifier":"Run one standard training workload (for example, a vision model on ImageNet) for a fixed number of steps and log identical metrics three ways — JSON-only, Zarr, and NetCDF — across at least fifty runs with varying numbers of logged epochs and metric frequencies. If the median total file-size reduction is far below 90 percent, or if the main PROV-JSON file remains large because textual parameters and artifact links dominate, the paper's headline efficiency claim fails.","tokens_in":10919,"feed_emoji":"📜","tokens_out":7872,"duration_ms":84409,"temperature":0.7,"pith_summary":"yProv4ML is a provenance library for machine learning that tries to make detailed lineage tracking cheap enough to use in large-scale training. The paper argues that existing tools such as MLflow and Weights and Biases log too coarse a record, and that yProv4ML captures per-epoch, per-context information — artifacts, parameters, metrics, and input/output relationships — in a W3C PROV-compliant form without becoming a performance bottleneck. The central efficiency claim is that moving bulky time-series metrics out of the main PROV-JSON file into Zarr or NetCDF companion files cuts the total provenance footprint by more than 90 percent, based on a measured comparison reported in Table 1. A scaling study of the MODIS-FM foundation model on a leadership-class HPC system is offered as evidence that this provenance can then be used to weigh energy consumption against model loss when choosing model size and GPU count. If the library works as described, reproducible and energy-aware ML development would not require a separate heavyweight provenance infrastructure.","feed_headline":"Provenance tool cuts bulky ML logs by 90 percent","feed_subtitle":"yProv4ML records per-epoch lineage in standard PROV-JSON, storing bulky metrics in Zarr or NetCDF.","key_machinery":"The central object is the yProv4ML data model, built around an Experiment that contains multiple Run Executions and subdivides each run into contexts and epochs. Its defining mechanism is the separation of concerns between a small, process-describing PROV-JSON file and companion Zarr or NetCDF files for bulky metrics. The library's logging API mirrors MLflow's, so the additional lineage comes from tracking each logged value as an artifact, parameter, or metric and marking it as an input or output, then recording the W3C PROV relationships 'used' and 'wasGeneratedBy'.","core_discovery":"The paper presents yProv4ML as a provenance-producing library for machine learning. Its central claim is that the library captures lineage at a finer grain than MLflow and Weights and Biases by structuring each experiment into run executions, contexts (training, validation, testing, or user-defined), and epochs, and by logging artifacts, parameters, and metrics with explicit input/output relationships. The output is a W3C PROV-compliant PROV-JSON file, optionally paired with RO-Crate packaging, and the latest version adds the ability to store time-series and bulky metrics in separate Zarr or NetCDF files so the top-level provenance file stays small. On that point the paper reports a measured total provenance size of 39.82 MB in JSON versus 2.74 MB in Zarr and 2.35 MB in NetCDF for one comparison, a reduction above 90 percent. The paper then uses a MODIS-FM foundation-model scaling study run on a leadership-class supercomputer to argue that this granular provenance makes it possible to compare model configurations by GPU-energy consumption times loss, finding the SwinT-V2 architecture preferable at scale.","pith_inferences":["The reported file-size comparison is one example; a broader benchmark across model families and metric cadences would tell whether the 90 percent figure is typical or an outlier.","The same companion-file mechanism could be extended to store weight snapshots or optimizer states in chunked formats, not just metrics, making the provenance record itself a distributed checkpoint catalog.","The empty cells in the scaling grid suggest that experiments exceeding a walltime limit are silently lost; an online provenance system that records partial runs could make such studies more complete without additional compute.","If provenance standards were unified across ML frameworks, the historical runs needed for scaling-law forecasting would be much easier to assemble, but that depends on ecosystem adoption, which the paper does not itself address."],"forward_implications":["If the 90 percent size reduction holds across typical runs, provenance capture ceases to be a storage bottleneck and can be left on by default in large-scale training.","With per-epoch, per-context lineage in a standard format, experiments can be compared run-to-run and across teams without converting between proprietary tracking formats.","The MODIS-FM scaling study indicates that fine-grained provenance can reveal energy-versus-loss trade-offs per model size and GPU count, which is exactly the information needed to budget node-hours.","Because the main PROV-JSON file stays small and self-describing, sharing a single file plus its artifact wrapper (RO-Crate) becomes a plausible path to out-of-the-box reproducibility of a training run.","If provenance histories accumulate, forecasting the effect of a hyperparameter or model-size change could be done from past runs rather than from fresh training."],"supporting_citations":[{"why":"MLflow is the existing experiment-tracking tool whose logging interface yProv4ML mimics and whose granularity it claims to exceed.","marker":"[9]"},{"why":"Defines the PROV-JSON serialization used as the library's default output format.","marker":"[10]"},{"why":"Introduces the PROV-ML taxonomy for the machine learning lifecycle that the library is designed to comply with.","marker":"[16]"},{"why":"Weights and Biases is the cloud-streaming experiment tracker used as a baseline for comparison of provenance granularity and overhead.","marker":"[31]"},{"why":"Describes the yProv framework and its REST API, the surrounding ecosystem into which yProv4ML plugs as a provenance producer.","marker":"[32]"},{"why":"NetCDF is one of the two optimized file formats used to store bulky metrics in companion provenance files.","marker":"[37]"},{"why":"Zarr is the chunked array format that, with NetCDF, underpins the reported file-size reductions.","marker":"[38]"},{"why":"Swin Transformer V2 is one of the two model baselines whose scaling study provides the energy-loss trade-off evidence.","marker":"[42]"},{"why":"The masked autoencoder ViT baseline provides the second model configuration in the MODIS-FM scaling study.","marker":"[43]"}],"fun_headline_variants":["yProv4ML shrinks ML provenance by 90% with Zarr","Fine-grained ML lineage, 90% smaller logs","New tool logs ML provenance at epoch level","yProv4ML: W3C PROV lineage without the bulk","Provenance for ML: 90% size cut, full trace"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The assumption that a typical training run's provenance record is dominated by bulky numeric time series is the load-bearing premise; if textual metadata dominates, the 90 percent reduction will not materialize.","fun_headline_variants_meta":{"raw":{"variants":["yProv4ML shrinks ML provenance by 90% with Zarr","Fine-grained ML lineage, 90% smaller logs","New tool logs ML provenance at epoch level","yProv4ML: W3C PROV lineage without the bulk","Provenance for ML: 90% size cut, full trace"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000602,"raw_usage":{"total_tokens":2831,"prompt_tokens":986,"completion_tokens":1845,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":1758}},"tokens_in":602,"tokens_out":1845,"duration_ms":14020,"temperature":1.0,"reasoning_tokens":1758,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:06:38.216020+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run one standard training workload (for example, a vision model on ImageNet) for a fixed number of steps and log identical metrics three ways — JSON-only, Zarr, and NetCDF — across at least fifty runs with varying numbers of logged epochs and metric frequencies. If the median total file-size reduction is far below 90 percent, or if the main PROV-JSON file remains large because textual parameters and artifact links dominate, the paper's headline efficiency claim fails.","supporting_citations":[{"cited_title":"Accelerating the machine learning lifecycle with mlflow","cited_arxiv_id":null,"evidence_quote":"MLflow is the existing experiment-tracking tool whose logging interface yProv4ML mimics and whose granularity it claims to exceed."},{"cited_title":"The prov-json serialization","cited_arxiv_id":null,"evidence_quote":"Defines the PROV-JSON serialization used as the library's default output format."},{"cited_title":"Provenance data in the machine learning lifecycle in computational science and engineering","cited_arxiv_id":null,"evidence_quote":"Introduces the PROV-ML taxonomy for the machine learning lifecycle that the library is designed to comply with."},{"cited_title":"A graph data model-based micro-provenance approach for multi-level provenance exploration in end-to-end climate workflows","cited_arxiv_id":null,"evidence_quote":"Describes the yProv framework and its REST API, the surrounding ecosystem into which yProv4ML plugs as a provenance producer."},{"cited_title":"Netcdf user’s guide, 1993","cited_arxiv_id":null,"evidence_quote":"NetCDF is one of the two optimized file formats used to store bulky metrics in companion provenance files."},{"cited_title":"https://zarr.dev/","cited_arxiv_id":null,"evidence_quote":"Zarr is the chunked array format that, with NetCDF, underpins the reported file-size reductions."}],"review_version":1}