{"id":"c3bc5eb8-79ee-4830-a946-e2553a1074f2","arxiv_id":"2412.05882","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"The paper repackages signal-to-noise ratio as a data-quality metric, DDR, and uses controlled synthetic noise to draw accuracy-DDR curves and define a trustworthiness portfolio.","lead":"This paper proposes a metric called DDR, which measures how much of a dataset is signal versus noise. It plots accuracy against DDR for ten machine learning models on synthetic data and suggests using the area under those curves as a trust score.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DDR is not identifiable from data because the required D/E decomposition is assumed without an estimator; Section 4 admits this, so accuracy-DDR curves cannot be computed for real data.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing premise I find: the decomposition Y=D+E must be available or recoverable, and the paper explicitly assumes rather than establishes it. Because DDR is defined only relative to that decomposition, and the paper provides no identification criterion, the entire accuracy-DDR framework cannot be applied to real data; the central claim therefore fails. I also note the standardization theorem is not a proof that DDR is preserved across valid decompositions, but a construction that assumes the target r, and the reported experimental trend is self-contradictory. These reinforce, rather than replace, the primary concern. The reader's verdict of REJECT remains appropriate: the paper's contribution as stated - a data-quality metric computable on observed data and usable to determine model performance - is not supported. I therefore recommend no change to the reader's verdict.","tokens_in":11198,"tokens_out":4550,"duration_ms":44984,"concrete_test":"Analytically construct a finite series Y and two distinct decompositions Y = D1+E1 = D2+E2, each satisfying the paper's independence and zero-mean/homoscedastic assumptions, but with different DDR values r1 != r2 (e.g., D1 a low-frequency trend with white noise E1; D2 a higher-frequency deterministic component with compensating white noise E2). Compute DDR via Eq. 4 for both. If the two values differ, DDR is not identifiable from Y alone and the accuracy-DDR curve is undefined for real data, falsifying the central claim at the point where the paper admits it has no estimator.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim - that DDR-accuracy curves determine model performance - requires DDR to be a well-defined, computable property of a dataset. The paper assumes (Eq. 1) Y(t)=D(t)+E(t) with D,E independent, E zero-mean and homoscedastic, and states in Section 4: 'we assume that a data can be split into the deterministic and non-deterministic components.' No estimator for D or E is provided, and the paper acknowledges that in real data the split 'is very complicated and the solution may be unknown' with 'infinite possible combinations.' This is not a minor technical gap: for a fixed Y, distinct valid decompositions can yield different DDR values, so the accuracy-DDR curve is not a function of the data alone. The DDR-invariant standardization (Eq. 18, Appendix B) does not rescue this: it constructs Dstand as an affine transform of the unknown D and sets Estand variance to 1-r, presupposing the very r it is supposed to compute. Thus the intended application to real-world data is undefined. Additionally, Section 6's own result summary is self-contradictory ('As DDR increases, accuracy also decreases' vs. 'more deterministic component ... higher is the accuracy'), which undercuts the empirical support for the claimed monotone trend.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a new data-quality metric, the deterministic-non-deterministic ratio (DDR), defined as the ratio of deterministic-signal power to total signal power under an assumed additive decomposition Y(t)=D(t)+E(t). It introduces DDR-invariant standardization to produce accuracy-versus-DDR curves for several regression and classification models on synthetic data, and defines a trustworthiness portfolio p_M as the area under the accuracy-DDR curve. The authors claim that DDR quantifies data quality and that accuracy-DDR curves can determine model performance under uncertainty.","tokens_in":11455,"tokens_out":2000,"duration_ms":20168,"significance":"If the DDR were well-defined, computable from data, and shown to be a genuine data-quality measure, the proposed accuracy-DDR curves and trustworthiness portfolio could provide a useful tool for comparing model robustness to noise. The paper's ambition to relate data quality to model performance in a quantitative way is relevant to the ML community. However, the significance is currently undermined by foundational issues: the DDR is not identifiable from real data, the standardization procedure forces the DDR values used in the plots, and the empirical trend is stated inconsistently. The manuscript does not yet deliver a usable metric or a credible validation of its central claim.","major_comments":[{"comment":"The paper assumes that any observed data can be decomposed as Y(t)=D(t)+E(t) with D and E independent and E zero-mean homoscedastic, but it explicitly states that in real data this split 'is very complicated and the solution may be unknown' and that there may be 'infinite possible combinations.' No estimator for D or E is provided (the mention of autoencoders is not developed). Consequently, for a fixed real dataset, distinct valid decompositions can yield different DDR values, so the accuracy-DDR curve is not a function of the data alone. This is a load-bearing gap: the central claim that DDR-accuracy curves determine model performance requires DDR to be a computable property of a dataset.","section":"Section 4, Eq. (1)"},{"comment":"The DDR-invariant standardization is circular. The method solves for alpha, beta, and sigma^2_{E_stand} using equations that contain the target DDR r, including sigma^2_{E_stand} = 1-r and D_stand = alpha*D + beta with alpha = sqrt(r)/SD(t). This construction explicitly imposes that P(D_stand)=r and P(E_stand)=1-r, so the resulting standardized datasets have DDR equal to the chosen r by construction. The theorem stating that the standardized data preserves DDR is therefore not a substantive invariance result; it is a restatement of the definition of the algorithm. The paper does not provide a proof that the algorithm achieves the stated criteria for arbitrary D and E, and the 'approximations' in Eqs. (7)-(9) are not derived in a way that establishes the claimed properties.","section":"Algorithm 1 and Appendix B, Eqs. (7)-(9), (18)"},{"comment":"The theorem is stated without proof. The three criteria (zero mean, unit variance, DDR preserved equal to r) are asserted, but Appendix B only performs a heuristic derivation under the additional assumption D_stand(t)=alpha*D(t)+beta, without justifying why this affine form is sufficient or necessary. Since the theorem is used to justify the entire experimental pipeline, the missing proof is a load-bearing omission. Either a rigorous proof or a clear statement of additional assumptions is required.","section":"Section 4, Theorem (DDR-Invariant Standardization)"},{"comment":"The summary of the empirical results is self-contradictory. The text states: 'As DDR increases, accuracy also decreases which translates to as there is more deterministic component in the data, higher is the accuracy.' The first clause says accuracy decreases with DDR, while the second clause says accuracy increases with the deterministic component (i.e., with higher DDR). These appear to assert opposite monotonic relationships. If the intended claim is that accuracy increases with DDR, then the rest of the discussion and the trustworthiness portfolio interpretation depend on that direction; the contradiction undercuts the empirical support for the claimed trend and must be resolved.","section":"Section 6, Experiment Results paragraph"}],"minor_comments":[{"comment":"There are frequent typos and grammatical errors (e.g., 'Decsision Tree' in Figure 6, 'Classsification' in Figure 8, 'effect' for 'affect' in Section 3, 'uncertainity' in Section 6, 'data quality by as a function' in Section 1). A thorough proofreading pass is needed.","section":"Throughout"},{"comment":"The related-work survey is broad but somewhat unfocused, and several citations are used loosely (e.g., Vapnik and Izmailov are cited as associated with data quality without explanation of the connection). The paper would benefit from a clearer positioning against the aleatoric/epistemic uncertainty literature, which it cites but does not systematically relate to DDR.","section":"Section 2"},{"comment":"The trustworthiness portfolio p_M is defined as the integral of accuracy(DDR) over DDR from 0 to 1. The notation in the text is garbled ('Z 0 1'), and the definitional relationship between Eq. (10), which multiplies accuracy by DDR, and Eq. (11), which integrates accuracy alone, is never explained.","section":"Equation (11)"},{"comment":"The figures are not described in the text in terms of the specific data-generation settings (e.g., number of features, number of samples, noise distributions). Without this information, the experimental setup is not reproducible even though code is referenced.","section":"Figures 1-9"},{"comment":"The paper states that 'For the accuracy-DDR plot for the regression algorithm, the accuracy-DDR relationship is almost linear' and later says the relationship is linear for classification except at low DDR, but also claims MLP is the most reliable because 'accuracy does not change significantly with DDR.' These observations are not quantified, and there are no confidence intervals or error bars in the figures.","section":"Section 6"},{"comment":"The citation to Font-Clos [2021] for hit-and-run sampling is a software repository rather than the original algorithm (Smith 1984), which is cited separately; the relationship between the two should be clarified.","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper's central claim is not yet defensible: the DDR is not identifiable from data, the standardization procedure imposes the DDR by construction, and the reported trend is self-contradictory. These are foundational issues that cannot be addressed by local revisions; they require a fundamentally different formulation or a much more careful identification argument. The paper's current framing as a general model of data quality and model performance is not supported by the evidence presented. The editor may also wish to note that the paper is very preliminary in its presentation and does not yet meet the standard of a rigorous ML methods paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"DDR is a rescaled signal-to-noise ratio, and the paper's validation loop is mostly self-referential: the standardization step solves for parameters that force the desired DDR, so the accuracy–DDR curves are constructed rather than observed. The metric is also not identifiable for real data, as the authors themselves admit. I don't think this is publishable in its current form, but it is a useful illustration of the pitfalls in designing data-quality metrics.\n\nThe paper does some things well. The literature review on noise, data quality, and trust is reasonably broad and correctly points to Zhu and Wu's earlier noise-sensitivity results. The idea of summarizing a model's performance across noise levels as an area under a curve is sensible, and running five regression and five classification models is more than many similar studies bother with.\n\nThe soft spots are substantial. First, under the paper's independence and zero-mean assumptions, DDR is exactly SNR/(1+SNR), so it is a reparameterization of a classical quantity rather than a new measure. Second, the central assumption that Y = D + E with an accessible split is not backed by any estimator. Section 4 acknowledges that in real data the split 'may be unknown' with 'infinite possible combinations,' which means the accuracy–DDR relationship is not a function of the data alone. Third, the DDR-invariant standardization in Appendix B is circular: equations 7–9 set alpha, beta, and sigma-squared to hit a target r, and the theorem claiming invariance is not proved. Fourth, Section 6 contains a direct contradiction: it says 'As DDR increases, accuracy also decreases' and then says more deterministic component means higher accuracy. That undermines the reported trend. Finally, the code is mentioned but not linked, and there are no error bars, so the plots are hard to evaluate.\n\nI don't think the authors are being deliberately misleading; they flag the decomposition problem honestly. But the load-bearing part of the framework—being able to compute DDR from real data—is missing, and the experiments are designed so that the target variable is imposed.\n\nWho would get value from this? Someone studying how to evaluate data-centric AI might read it as a cautionary example of circular evaluation. I would not cite it, and I would not send it to review as is. If it were submitted to me, I would desk reject but invite a resubmission after the authors provide an estimator for the decomposition, test on real data, and fix the contradiction in Section 6.","headline":"A reparameterized SNR dressed as a new metric, with a circular validation loop and an unproven identifiability assumption; useful as a cautionary example, not as a contribution.","tokens_in":11992,"tokens_out":3428,"would_cite":false,"duration_ms":32285,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces a deterministic-non-deterministic ratio (DDR) that quantifies data quality and uses accuracy-DDR curves to score model performance under uncertainty.","keywords":["data quality metric","deterministic-non-deterministic ratio","signal-to-noise ratio","accuracy-DDR curves","model performance evaluation","uncertainty quantification","trustworthiness portfolio","synthetic data generation"],"falsifier":"Generate two datasets with identical DDR and the same deterministic component but different noise distributions, for example Gaussian versus heavy-tailed or homoscedastic versus heteroscedastic, train the same model on each, and compare accuracies; if accuracy differs systematically at the same DDR, then DDR alone does not determine model performance.","tokens_in":10973,"feed_emoji":"📊","tokens_out":7228,"duration_ms":66132,"temperature":0.7,"pith_summary":"This paper tries to establish that the quality of a dataset can be measured by a single number, the deterministic-non-deterministic ratio (DDR), and that this number predicts how well machine learning models will perform on the data. The authors decompose observed data into a deterministic signal plus independent, zero-mean noise, define DDR as the power of the deterministic part divided by the total power, and argue that accuracy rises as DDR rises. They build accuracy-DDR curves from synthetic datasets with controlled noise levels, and define the trustworthiness portfolio as the area under those curves. If the claim holds, model evaluation gains a data-quality axis: instead of a single accuracy number, a model is characterized by how its accuracy degrades as data becomes less deterministic.","feed_headline":"New data-quality ratio predicts accuracy under noisy conditions","feed_subtitle":"The deterministic-non-deterministic ratio turns noise levels into accuracy-DDR curves that grade model trustworthiness.","key_machinery":"The central object is the deterministic-non-deterministic ratio, DDR, the ratio of the power of the deterministic component to the power of the whole signal, $DDR(Y)=P(D)/P(Y)$, where power is $P(X)=\\frac{1}{|T|}\\sum_t X(t)^2$. The argument is carried by the additive decomposition $Y(t)=D(t)+E(t)$ with $E$ zero-mean and constant variance, which lets DDR reduce to $P(D)/(P(D)+P(E))$. To make accuracy comparable across datasets, the paper introduces DDR-invariant standardization, an affine rescaling of $D$ that keeps the dataset mean zero, variance one, and DDR unchanged, and it uses hit-and-run sampling, a procedure that samples points uniformly inside a bounded region, to generate sets of datasets whose squared DDRs are uniformly distributed. The accuracy-DDR curve, and its integral $p_M$, are the output object that converts the ratio into a performance statement.","core_discovery":"The core claim is that a dataset's noise content can be quantified by DDR, defined as $DDR(Y) \\approx \\frac{\\sum_t D(t)^2}{\\sum_t D(t)^2 + \\sum_t E(t)^2}$ for $Y(t)=D(t)+E(t)$ with $E$ independent, zero-mean, homoscedastic noise. The paper's experiments on five regression and five classification models generate synthetic data bottom-up with controlled noise, standardize datasets in a DDR-preserving way, and plot accuracy against DDR; the plots show accuracy increasing with DDR. From these curves the paper defines the trustworthiness portfolio $p_M = \\int_0^1 \\mathrm{accuracy}(r)\\,dr$, interpreted as the area under the accuracy-DDR curve, and uses it to rank models: decision-tree regression and multi-layer perceptron classification come out highest, K-nearest-neighbor models lowest. The authors read these results as evidence that DDR is a valid data-quality metric and that the area under the accuracy-DDR curve measures a model's reliability under uncertainty.","pith_inferences":["If real-data decompositions can be estimated, for example by denoising, DDR could serve as a pre-training diagnostic that flags features whose noise budget will cap achievable accuracy.","The homoscedastic zero-mean assumption could be relaxed; a definition based on expectations rather than sample powers would extend DDR to heteroscedastic or correlated noise, and that extension is testable with the same curves.","The accuracy-DDR curve suggests a model-selection rule: choose the model whose curve is flattest in the DDR range of the deployment environment, rather than the model with the highest accuracy at a single noise level."],"forward_implications":["A model's accuracy-DDR curve gives a per-noise-level performance profile, so two models that tie on raw accuracy can be separated by which one degrades less as DDR falls.","The trustworthiness portfolio, as the area under the accuracy-DDR curve, is a single number that ranks models by performance across uncertainty levels.","DDR-invariant standardization makes it possible to compare accuracy across datasets with different DDR values without standardization artifacts.","The reported experimental rankings (decision-tree regressor and multi-layer perceptron classifier highest, KNN models lowest) are the expected ordering if the accuracy-DDR curves are valid."],"supporting_citations":[{"why":"Establishes that classifier accuracy drops as noise increases, the empirical baseline the paper extends to a quantitative metric.","marker":"Zhu and Wu [2004]"},{"why":"Shows benchmark datasets with little uncertainty give misleadingly high accuracy, motivating a data-quality-aware performance measure.","marker":"Wu and Keogh [2021]"},{"why":"Provides the hit-and-run sampling implementation used to generate DDR tuples with uniformly distributed squared DDRs.","marker":"Font-Clos [2021]"},{"why":"Defines the hit-and-run algorithm for uniform sampling over bounded regions, the basis of the DDR sampling step.","marker":"Smith [1984]"},{"why":"PyTorch is used for data generation and model implementations in the experiments.","marker":"Paszke et al. [2019]"},{"why":"Scikit-learn is used for the baseline regression and classification models.","marker":"Pedregosa et al. [2011]"}],"fun_headline_variants":["New metric DDR links data noise to model accuracy","Data noise ratio predicts machine learning accuracy","DDR curves grade model trustworthiness under noise","New ratio DDR maps noise to trustworthy ML performance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every dataset really is a predictable signal plus independent random noise that averages to zero and has constant spread, and that the split is known or recoverable; without such a split, DDR cannot be computed and the accuracy-DDR curve is undefined.","fun_headline_variants_meta":{"raw":{"variants":["New metric DDR links data noise to model accuracy","Data noise ratio predicts machine learning accuracy","DDR curves grade model trustworthiness under noise","New ratio DDR maps noise to trustworthy ML performance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000452,"raw_usage":{"total_tokens":2221,"prompt_tokens":838,"completion_tokens":1383,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":1326}},"tokens_in":454,"tokens_out":1383,"duration_ms":10803,"temperature":1.0,"reasoning_tokens":1326,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:14:34.567268+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate two datasets with identical DDR and the same deterministic component but different noise distributions, for example Gaussian versus heavy-tailed or homoscedastic versus heteroscedastic, train the same model on each, and compare accuracies; if accuracy differs systematically at the same DDR, then DDR alone does not determine model performance.","supporting_citations":[{"cited_title":"Class noise vs","cited_arxiv_id":null,"evidence_quote":"Establishes that classifier accuracy drops as noise increases, the empirical baseline the paper extends to a quantitative metric."},{"cited_title":"Current time series anomaly detection benchmarks are flawed and are creating the illusion of progress","cited_arxiv_id":null,"evidence_quote":"Shows benchmark datasets with little uncertainty give misleadingly high accuracy, motivating a data-quality-aware performance measure."},{"cited_title":"fontclos/hitandrun: Initial release, Jun 2021","cited_arxiv_id":null,"evidence_quote":"Provides the hit-and-run sampling implementation used to generate DDR tuples with uniformly distributed squared DDRs."},{"cited_title":"Efficient monte carlo procedures for generating points uniformly distributed over bounded regions","cited_arxiv_id":null,"evidence_quote":"Defines the hit-and-run algorithm for uniform sampling over bounded regions, the basis of the DDR sampling step."},{"cited_title":"Pytorch: An imperative style, high-performance deep learning library","cited_arxiv_id":null,"evidence_quote":"PyTorch is used for data generation and model implementations in the experiments."}],"review_version":1}