{"id":"fb112118-c6fb-4072-9730-6901c7f4a665","arxiv_id":"2507.01078","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"yProv4ML is a new Python library that captures ML training provenance, such as parameters, metrics, and system information, in standard PROV-JSON format with an MLFlow-like API.","lead":"This paper presents yProv4ML, a Python library that records machine learning experiment details, including hyperparameters and metrics, and saves them in the standard PROV-JSON format. It aims to let researchers track data lineage and energy usage with minimal code changes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never validates that the emitted files are actually conformant PROV-JSON, so the central 'standard PROV-JSON' claim rests on an untested assumption.","rationale":"I read the paper as a software description, not as a proof of automatic provenance completeness. Its central claim is limited: capture the information the user elects to track in PROV-JSON. The reader's weakest assumption, that explicit logging calls capture complete provenance, is not a flaw relative to that claim, because the authors' own language scopes the tool to user-elected content and 'minimal code modifications' does not imply a fully automatic system. The more fragile premise is format conformance. No output sample is shown, no validation step is described, and the only visual evidence is a graph rendering. Since PROV-JSON has precise schema and semantics, a tool can easily write JSON that is not valid provenance, so the W3C-standard claim is the one that most directly supports the paper's contribution. A parse-and-validate test of the released repository would settle this question quickly, and the paper's condition should be to run it before claiming standard-compliance. My read reinforces the CONDITIONAL verdict rather than changing it, so I leave the verdict unchanged while partially agreeing with the reader's diagnosis.","tokens_in":5716,"tokens_out":4067,"duration_ms":50241,"concrete_test":"Generate the PROV-JSON output by running the Section 2.3 MNIST example for one epoch using the released v1.0 code, then load the emitted file with the W3C prov Python package (ProvDocument.deserialize(filename, format='json')). Assert that it parses without error, that round-tripping to JSON is lossless, and that the graph contains at least one prov:Activity for the training run, a prov:Entity for the logged model, and prov:used or prov:wasGeneratedBy edges connecting the logged dataset, hyperparameters, and metrics to that activity. If the PROV-JSON schema is available, validate the file against it as well. If any of these checks fail—or if the example cannot be run as printed because log_dataset is undocumented in Section 2.2—the central interoperability claim needs to be softened or the library fixed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that yProv4ML captures provenance in W3C PROV-JSON format. For that claim to hold, the files emitted by end_run must be syntactically valid PROV-JSON and, more importantly, must express a semantically meaningful PROV graph: activities, entities, agents, and edges such as used, wasGeneratedBy, and wasAttributedTo that connect the logged dataset, hyperparameters, metrics, and model artifacts. The manuscript provides no evidence of this. Sections 2.2 and 2.3 describe the directives and show a code snippet, and Section 3 shows a screenshot of a graph, but there is no output file, no schema validation, no round-trip parse, and no check that the graph's relationships correspond to PROV semantics. Section 5 asserts that the library serializes 'in accordance with the W3C PROV standard,' but that assertion is exactly what needs support. The dependency on the Prov Python package (Table 1, C7) suggests serialization is delegated to a mature library, which mitigates the risk, but the manuscript never demonstrates this, and the claimed interoperability with yProv4WF and with standard PROV tooling depends on it. This is load-bearing: if the generated JSON is syntactically or semantically invalid, the 'standard format' and 'not restricted to proprietary options' value proposition fails even when every user directive is inserted correctly. The reader's completeness concern is real but is not the most fragile premise, because the paper explicitly scopes itself to what 'the user elects to track'; the format-conformance claim is not scoped away.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents yProv4ML, a Python library for provenance tracking in machine learning workflows. It offers an MLFlow-like API (start_run, log_param, log_metric, log_system_metrics, log_carbon_metrics, log_artifact, save_model_version, end_run) that collects datasets, hyperparameters, metrics, system statistics, carbon emissions, and model checkpoints, and claims to serialize the collected information into PROV-JSON, the W3C standard provenance format. The manuscript describes the library's architecture, shows a small PyTorch code snippet for MNIST training, displays two plots of logged metrics, and includes a screenshot of a provenance graph. It also mentions deployments at ORNL on over 500 GPUs and for tropical cyclone prediction, and states that the output is interoperable with other yProv libraries. The paper does not contain a systematic evaluation: there is no comparison with MLFlow or other tools, no measurement of runtime or memory overhead, no validation of the PROV-JSON output against a schema, and no quantitative evidence from the claimed large-scale deployments.","tokens_in":5915,"tokens_out":3741,"duration_ms":42050,"significance":"If the central claim holds, yProv4ML would be a useful contribution: it would let ML practitioners obtain structured, standards-based provenance records with an API similar to MLFlow, thereby improving reproducibility and enabling lineage-aware comparison of experiments. The use of the established PROV-JSON format is a genuine strength over proprietary logging systems, and the claimed interoperability with the yProv framework could be valuable in scientific workflows. The paper's main weakness is the absence of evidence: the PROV-JSON compliance is not demonstrated, the completeness of the captured provenance is undefined, and the performance impact is unmeasured. These gaps are fixable, since the implementation is open-source and could be supplemented with output examples, schema-validation tests, and overhead benchmarks. As presented, however, the paper reads more as a feature announcement than as a verified software contribution.","major_comments":[{"comment":"The central claim that yProv4ML serializes 'in accordance with the W3C PROV standard' and produces PROV-JSON output is not demonstrated anywhere in the manuscript. No actual PROV-JSON file is shown, no validation against the PROV-JSON schema is reported, and no test checks that the emitted graph contains the expected PROV entities, activities, and relations (used, wasGeneratedBy, wasAttributedTo). Since this is the paper's core value proposition, the authors should provide at least one self-contained output example and a validation/round-trip test, and report the result.","section":"Section 5 and abstract"},{"comment":"The evaluation section is essentially absent: Figure 2 shows two metric plots and Figure 3 shows a graph screenshot, but there is no quantitative assessment of the library. The authors should report the runtime and memory overhead relative to a plain PyTorch run, compare the API and output interoperability with MLFlow, and, if feasible, provide concrete data (e.g., graph size, serialization time) from the claimed 500-GPU deployment. Without such measurements, the 'effortless' and 'lightweight' claims are unsupported.","section":"Section 3 and Section 5"},{"comment":"The completeness of the provenance graph is entirely determined by user-inserted directives; there is no automatic instrumentation of implicit dependencies. The code snippet also calls log_dataset, but this directive is not described in Section 2.2, which lists only log_model, log_param, log_metric, log_system_metrics, log_carbon_metrics, log_artifact, save_model_version, and log_current_execution_time. The authors should define log_dataset (and any other omitted directives) and explicitly state that data transformations applied inside custom PyTorch modules are outside the library's scope, so that readers can judge what 'effortless tracking' actually covers.","section":"Section 2.2 and 2.3"},{"comment":"The impact section states that yProv4ML has been used on more than 500 GPUs at ORNL and for tropical cyclone prediction, but provides no evidence—no logs, no configurations, no reproducible artifacts, and no link to the resulting provenance data. If these are anecdotal application notes, that should be stated; if they are meant to demonstrate scalability, quantitative results and a clear description of the deployment setup are required.","section":"Section 4"}],"minor_comments":[{"comment":"The description of log_metric says it stores 'a sequence of parameters', but it actually stores a sequence of metric values; this is a wording inconsistency that could confuse readers.","section":"Section 2.2"},{"comment":"The code snippet uses prov4ml.log_dataset(train_loader, 'train_dataset'), but this function is not listed among the directives in Section 2.2; please add it or adjust the snippet.","section":"Section 2.3"},{"comment":"The phrase 'works conducted with less rigor and in a superficial way' and the argument that hyperparameter uncertainty 'presents challenges in identifying the best model' are imprecise; the motivation should focus on the lack of provenance and lineage information rather than on hyperparameter tuning difficulty.","section":"Section 1"},{"comment":"There are several typographical and stylistic inconsistencies: 'superceded' should be 'superseded' (Section 1), 'on over than 500 GPUs' should be 'on over 500 GPUs' (Section 4), and the formatting of 'prov4ml. start run' in Section 2.3 introduces spurious spaces from the line break.","section":"General"},{"comment":"The dependencies listed in C7 ('Codecarbon, Prov, Pytorch') lack version numbers; please specify the minimum supported versions to aid reproducibility.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"This is a software-description paper that, for a journal like SoftwareX, might be acceptable if it were accompanied by the minimal evaluation the journal expects. However, the absence of any validation of the PROV-JSON output is a load-bearing gap: the entire value proposition rests on the claim of format compliance, and that claim is currently untested. The authors should be encouraged to add a brief but concrete evaluation section—an example output, a schema-validation test, and an overhead measurement—rather than merely descriptive text. I also note that the paper's length is unusually short for a journal submission, even for SoftwareX, and that the 'Impact' section reads more like a list of anecdotes than a systematic demonstration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short SoftwareX-style software description. The genuinely new bit is combining the MLFlow-style logging API with PROV-JSON serialization. That fills a real gap: MLFlow stores in its own format, and PROV-JSON is a W3C standard, so a bridge between the two is useful for the provenance and reproducibility crowd. The API is clearly described, the code example is coherent, and the repo is public with a GPLv3 license. Credit where due: the modular design (energy, system, time modules) is sensible, and the authors clearly understand the provenance modeling space.\n\nThe soft spots are proportionate. The biggest one, and the stress-test note gets this right, is that the paper never shows a single emitted PROV-JSON file or validates one against the schema. The central claim is that the serialization is \"in accordance with the W3C PROV standard,\" and that assertion is exactly what needs evidence. The dependency on the Prov Python package mitigates the risk—that library is mature and likely produces valid output—but the manuscript doesn't demonstrate it. A screenshot of an SVG graph is not the same as a schema-valid JSON document, and the claimed interoperability with other PROV tooling rests on it.\n\nThe completeness concern the reader raised is real but less load-bearing. The paper explicitly says it records what the user elects to track, so the gap of implicit dependencies is acknowledged by scope. Still, \"full lineage\" in the introduction is oversold; it's a minor overclaim.\n\nWhat's missing is any systematic evaluation: no comparison against MLFlow, no overhead measurements, no memory footprint numbers, no round-trip parse. For a software paper, that's more than cosmetic; the claim of \"minimal code modifications\" is asserted, not shown. These are all addressable in revision.\n\nBottom line: this is a modest but honest contribution. The authors ship code and describe it accurately, modulo the unvalidated PROV-JSON claim. It deserves a serious referee—someone should check the repo, run a small test, and ask for a sample output file and a benchmark. I'd send it to peer review rather than desk reject, with the expectation of a revision adding validation and evaluation.","headline":"A modest but real software contribution—an MLFlow-style provenance logger that outputs W3C PROV-JSON—whose main gap is the absence of any validation that the emitted files are actually conformant.","tokens_in":6500,"tokens_out":2232,"would_cite":false,"duration_ms":23969,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"yProv4ML records datasets, hyperparameters, metrics, and emissions from ML runs as a standard PROV-JSON provenance graph.","keywords":["machine learning","provenance","yProv4ML","PROV-JSON","provenance graph","experiment tracking","reproducibility","carbon metrics"],"falsifier":"Run the paper's MNIST example, then apply an extra data transformation on the fly during training without logging it; if the generated PROV-JSON graph does not show that transformation as an activity connected to the dataset, the claim of capturing the data manipulation process is refuted.","tokens_in":5468,"feed_emoji":"📜","tokens_out":12516,"duration_ms":118601,"temperature":0.7,"pith_summary":"This paper introduces yProv4ML, a Python library that captures what happens during a machine-learning run—datasets, hyperparameters, metrics, system usage, carbon emissions, and model versions—and writes it into a PROV-JSON provenance graph following the W3C PROV standard. The main claim is that this can be done with minimal changes to existing training code, because the logging calls are shaped like those of familiar experiment-tracking tools. The motivation is reproducibility: current trackers store experiment metadata in proprietary formats and pay little attention to lineage, so users cannot easily trace how a model was produced or reuse knowledge from earlier runs. If the library works as described, ML teams can obtain an interoperable, human-readable lineage record that also supports energy-consumption monitoring and cross-run comparison, without being locked into a closed format.","feed_headline":"ML experiment lineage, captured in the standard PROV-JSON format","feed_subtitle":"yProv4ML logs datasets, hyperparameters, metrics, and energy use while producing an interoperable PROV-JSON graph.","key_machinery":"The load-bearing machinery is the set of yProv4ML directives and the in-memory data model they feed. The directives deliberately resemble the interfaces of existing experiment-tracking systems (start_run, log_param, log_metric, and so on) so that a user can adopt the library without restructuring a training script. The data model accumulates logged parameters (one-time values), metrics (time-series values with context and step), artifacts (files), system metrics (memory, disk, and GPU usage), carbon metrics (emissions and power), model configurations, and checkpoint versions. At end_run this data is serialized into a PROV-JSON graph, and optionally into an SVG or DOT rendering, making the provenance navigable and interoperable.","core_discovery":"In the paper's own terms, the central discovery is that a small set of logging directives—start_run, log_param, log_metric, log_system_metrics, log_carbon_metrics, log_artifact, save_model_version, log_model, and end_run—is sufficient to produce a standard W3C PROV-JSON graph capturing the lineage of a training process. The graph links user-logged items with automatically recorded information such as environment variables and required libraries, and each process's graph can be merged into a collection for distributed runs. The authors demonstrate the output on an MNIST classification example and report using the library in large-scale benchmarks and a tropical-cyclone prediction project, arguing that the approach provides a transparent, interoperable alternative to proprietary experiment metadata.","pith_inferences":["The goal of provenance capture with minimal user effort depends on users remembering to call a logging directive for every piece of information they care about; an obvious testable extension is to combine yProv4ML with automatic instrumentation of the deep-learning framework's autograd graph so that implicit transformations inside custom modules are captured without manual calls.","The paper does not quantify the overhead of the logging calls; a natural benchmark would measure the runtime and memory cost of a run with yProv4ML versus an identical run without it, and versus a run using an existing tracker, to test whether minimal code changes also mean minimal performance loss.","Since PROV-JSON is a W3C standard, a further consequence is that provenance records produced by different tools could be merged across a project, enabling cross-framework lineage queries; yProv4ML would then serve as a bridge rather than an endpoint."],"forward_implications":["Users can obtain a standards-compliant provenance record of an ML run by inserting a handful of logging calls, without changing the training logic or switching to a proprietary tracking service.","Because the output is PROV-JSON, provenance can be processed, visualized, and exchanged with standard W3C PROV tooling, removing the lock-in of closed experiment-metadata formats.","When combined with workflow-level provenance libraries from the same yProv family, the ML-level graph nests inside a higher-level workflow graph, giving end-to-end lineage from raw data through model training.","Logging system and carbon metrics per run makes it possible to compare energy use and emissions across experiments, supporting sustainable model-development choices.","Storing hyperparameters and model versions per run lets researchers look up prior experiments and reuse good parameter choices instead of re-running costly searches."],"supporting_citations":[{"why":"It defines the experiment-tracking interface that yProv4ML deliberately imitates, establishing the baseline for the claim of a seamless transition.","marker":"[8]"},{"why":"It specifies PROV-JSON, the serialization standard that yProv4ML outputs as its provenance format.","marker":"[9]"},{"why":"It supplies the definition of provenance as the record of origins, history, and transformations that motivates the whole framework.","marker":"[5]"},{"why":"It describes the wider yProv family and supports the claim that yProv4ML graphs interoperate with workflow-level provenance graphs.","marker":"[10]"},{"why":"It reports the large-scale benchmarks in which yProv4ML was used, providing evidence that the library scales to many-GPU environments.","marker":"[12]"},{"why":"It describes the tropical-cyclone prediction application where yProv4ML collected provenance for a graph neural network, demonstrating use beyond the illustrative example.","marker":"[13]"}],"fun_headline_variants":["ML provenance in standard PROV-JSON with minimal code","Effortless lineage tracking for ML: yProv4ML","Standardize ML experiment lineage with PROV-JSON","Track ML lineage with PROV-JSON, not proprietary formats","Minimal code, maximal transparency: yProv4ML logs ML provenance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework can only record what the user's logging calls explicitly capture; if a meaningful step such as a data transformation inside a custom module is never logged, the provenance graph will be missing that step, and the library does not automatically instrument the code to fill such gaps.","fun_headline_variants_meta":{"raw":{"variants":["ML provenance in standard PROV-JSON with minimal code","Effortless lineage tracking for ML: yProv4ML","Standardize ML experiment lineage with PROV-JSON","Track ML lineage with PROV-JSON, not proprietary formats","Minimal code, maximal transparency: yProv4ML logs ML provenance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000402,"raw_usage":{"total_tokens":2036,"prompt_tokens":825,"completion_tokens":1211,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":441,"completion_tokens_details":{"reasoning_tokens":1126}},"tokens_in":441,"tokens_out":1211,"duration_ms":9914,"temperature":1.0,"reasoning_tokens":1126,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:05:08.238341+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's MNIST example, then apply an extra data transformation on the fly during training without logging it; if the generated PROV-JSON graph does not show that transformation as an activity connected to the dataset, the claim of capturing the data manipulation process is refuted.","supporting_citations":[{"cited_title":"Accelerating the machine learning lifecycle with mlflow,","cited_arxiv_id":null,"evidence_quote":"It defines the experiment-tracking interface that yProv4ML deliberately imitates, establishing the baseline for the claim of a seamless transition."},{"cited_title":"Interoperability for provenance-aware databases using PROV and JSON,","cited_arxiv_id":null,"evidence_quote":"It specifies PROV-JSON, the serialization standard that yProv4ML outputs as its provenance format."},{"cited_title":"Provenance: a future history,","cited_arxiv_id":null,"evidence_quote":"It supplies the definition of provenance as the record of origins, history, and transformations that motivates the whole framework."},{"cited_title":"A graph data model-based micro-provenance approach for multi-level provenance exploration in end-to-end climate workflows,","cited_arxiv_id":null,"evidence_quote":"It describes the wider yProv family and supports the claim that yProv4ML graphs interoperate with workflow-level provenance graphs."},{"cited_title":"Exploring vision transformers on the frontier supercomputer for remote sensing and geoscientific applications,","cited_arxiv_id":null,"evidence_quote":"It reports the large-scale benchmarks in which yProv4ML was used, providing evidence that the library scales to many-GPU environments."},{"cited_title":"A software ecosystem for multi-level provenance man- agement in large-scale scientific workflows for ai applications,","cited_arxiv_id":null,"evidence_quote":"It describes the tropical-cyclone prediction application where yProv4ML collected provenance for a graph neural network, demonstrating use beyond the illustrative example."}],"review_version":1}