{"id":"e4032089-c221-466b-bef2-cfc39c264b30","arxiv_id":"2506.09010","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Data pruning importance scores can be extrapolated from a 10-25% random subset to the full dataset with kNN or GNN, saving time with a small accuracy loss.","lead":"This paper proposes computing expensive sample-importance scores for data pruning on a small random subset, then extrapolating those scores to the rest of the dataset using k-nearest-neighbor averaging or a graph neural network. The approach cuts the time of pruning by roughly half on large image benchmarks while keeping accuracy close to full-score pruning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own Table 1 contradicts the claim that extrapolated scores consistently outperform random whenever ground truth does: at 10% subset on ImageNet 50% DU, both GNN (58.30) and KNN (58.51) fall below random (58.56).","rationale":"The central claim has two prongs: time advantage and better-than-random pruning. The time advantage is well supported by Table 1. The better-than-random prong is the one that must hold for the method to be useful, because otherwise one could simply train on a random subset. The paper asserts it 'consistently' holds whenever ground truth helps, but its own Table 1 contains a counterexample at the smallest subset size on ImageNet. This is not a disagreement with an external baseline or a missing comparison; it is an internal inconsistency with the reported numbers. The failure also spotlights the weak assumption: extrapolation relies on score smoothness in the embedding space of a model trained on a small random subset. At 10% on a large, diverse dataset, that smoothness is apparently insufficient for DU scores. The k-selection protocol (B.4) is a separate methodological concern, but it would bias results in favor of the method, so it cannot rescue the counterexample. A conditional acceptance that requires the authors to qualify the claim or show the ImageNet 10% result was a statistical fluke is the appropriate outcome.","tokens_in":18566,"tokens_out":9946,"duration_ms":102669,"concrete_test":"Re-run the ImageNet DU 50% pruning experiment with a 10% subset and 5 seeds, with k fixed to a value chosen on Ds validation (no use of Dr ground truth); compute a paired difference against random pruning. If the mean difference remains ≤0, the 'consistently outperform' claim must be removed or explicitly restricted to medium subset sizes. Additionally, scan every cell of Table 1 and Figure 6 for any negative (extrapolated − random) margin in regimes where ground truth beats random, and report these as exceptions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4 states: 'extrapolated scores consistently outperform random pruning whenever the ground truth score does.' Table 1 reports a direct counterexample. For ImageNet at 50% pruning with DU, the ground-truth score beats random (59.08±0.07 vs 58.56±0.06), so the condition is satisfied. Yet the 10%-subset extrapolations both fall below random: GNN small 58.30±0.17, KNN small 58.51±0.14. This is not a regime where ground truth fails; it is exactly the regime the claim covers. The abstract promises pruning from 'only a small subset of data,' and the same section only claims the small subset 'generally outperforms random pruning,' which is weaker than 'consistently.' The failure is not explained by the k-selection protocol in B.4 (which tunes k on ground-truth scores of Dr, if anything inflating correlations): even with that selection advantage, the small-subset extrapolations underperform random on ImageNet. Thus the central better-than-random claim is overgeneralized and depends on subset size and dataset. The load-bearing assumption is that importance scores are sufficiently smooth in the embedding space of the subset-trained model; Table 1 shows this assumption fails at 10% on ImageNet for DU, exactly where the practical benefit is supposed to be largest.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a score extrapolation framework for data pruning. Instead of training on the full dataset to compute per-sample importance scores for DU and TDDS, the authors train a model on a small random subset D_s, compute ground-truth scores S_s on that subset, and extrapolate scores to the residual set D_r using either a weighted k-nearest-neighbor average (Eq. 1) or a three-layer GCN on a k-NN graph (Sec. 3.2). The resulting scores are used to prune the full dataset. The authors evaluate the method on CIFAR-10, synthetic CIFAR-100, Places-365, and ImageNet, across supervised, unsupervised, and adversarial training, and report accuracy, score correlations, and end-to-end training time. The central claim is that a 10--25% subset suffices to extrapolate importance scores and that extrapolated scores outperform random pruning whenever the ground-truth scores do, with a time advantage already on the first training run.","tokens_in":18765,"tokens_out":5673,"duration_ms":56885,"significance":"If substantiated, the contribution is practically valuable: it would make DU and TDDS pruning applicable to single large-scale training runs without a full-data scoring pass, and the released code plus the breadth of datasets, pruning methods, and training paradigms are clear strengths. The time-accuracy Pareto analysis in Figure 3 is a useful way to frame the practical trade-off. However, the paper's central better-than-random guarantee is stated too strongly and is contradicted by one of its own reported configurations, and the hyperparameter selection protocol in Appendix B.4 uses the extrapolation target (ground-truth scores on D_r) to choose k, which inflates the reported correlations and downstream results. These issues are load-bearing for the paper's headline claim, so the current evidence supports a more modest version of the claim rather than the version stated in Section 4.","major_comments":[{"comment":"The sentence 'across all datasets, extrapolated scores consistently outperform random pruning whenever the ground truth score does' is directly contradicted by Table 1. For ImageNet at 50% DU pruning, the ground-truth score (59.08 +/- 0.07) beats random pruning (58.56 +/- 0.06), so the stated condition is satisfied; yet the 10%-subset extrapolations fall below random: GNN small gives 58.30 +/- 0.17 and KNN small gives 58.51 +/- 0.14. The same overgeneralized wording appears in Section C.2. Because this better-than-random property is the paper's central practical claim, the claim must be qualified by subset size and configuration, or the affected experiments must be revisited and the text revised to report which configurations actually beat random.","section":"Section 4, Table 1"},{"comment":"The model-selection protocol leaks information from the extrapolation target. For KNN, k is chosen by varying k over 10, 20, 50, 100 and reporting the value with the highest Pearson correlation between extrapolated and ground-truth scores evaluated on D_r; for the GNN, the graph neighborhood size k is selected in the same way, and the reported correlations are also computed on D_r. Since D_r is the set whose scores the method is supposed to predict without access to ground truth, selecting k on D_r inflates the correlations in Table 2 and gives the downstream results in Table 1 the benefit of test-set selection. A real user would not have the target scores S on D_r. The authors should either select hyperparameters using only a held-out part of D_s, or show that the downstream pruning accuracy is insensitive to k across the full ranges reported in Tables 10 and 11.","section":"Appendix B.4"},{"comment":"The paper's own qualitative analysis shows that extrapolation is least reliable for atypical or out-of-distribution samples: extrapolated DU scores fail to reproduce the bimodal structure of the ground-truth scores, and the largest rank errors concentrate on samples with unusual backgrounds, multiple objects, or low visual quality. This is exactly the regime where importance scores matter most for pruning. The paper should quantify this limitation rather than only illustrate it: for example, report how many of the top-ranked residual samples are misranked by a large margin, or show how pruning accuracy changes when the residual samples are restricted to neighborhoods with high extrapolation confidence. Without such an analysis, the claim that the method 'accurately predict[s] sample importance' overstates the reliability of score extrapolation on the samples that determine pruning quality.","section":"Section 4, Figure 5 and Limitations"}],"minor_comments":[{"comment":"The sentence 'the GNN's correlation is always higher than the KNN's' is contradicted by Table 3 in the unsupervised setting, where KNN correlations are higher than GNN correlations. The statement should be scoped to the supervised results in Table 2.","section":"Section 4, Table 3"},{"comment":"The legend in panels (a) and (c) lists 'KNN-DU' twice, although the figure appears to include KNN-TDDS curves; the caption also contains the typo 'Synthic CIFAR-100'. Please correct these for readability.","section":"Figure 2"},{"comment":"There are several typos and awkward phrasings, including 'compromising n samples' (should be 'comprising'), 'onlyon on a minor subset', and 'Arvix' in the references. A careful copyedit would improve the presentation.","section":"Abstract and Section 3"},{"comment":"The caption says the table reports results 'at the highest pruning percentages, where the pruning algorithms still outperform random pruning', but the ImageNet 50% DU row with the 10% subset does not outperform random. Please adjust the caption so it does not pre-judge the comparison being reported.","section":"Table 1 caption"},{"comment":"Table 6 lists subset sizes of 40% and 20% for CIFAR-10, while Section 4 says 'if not stated otherwise, we use 20% of the full dataset for D_s'. Please clarify which results in Figure 2 and Table 1 use the 40% subset and which use the 20% subset.","section":"Table 6"}],"recommendation":"major_revision","confidential_remarks":"The central idea is worth pursuing and the empirical scope is broad, but the overclaimed 'consistently outperform random' guarantee and the D_r-based hyperparameter selection need to be fixed before the paper can be accepted. I would be willing to review a revised version. The paper's self-citation to the prior workshop paper [21] is appropriately disclosed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is sensible and mostly well executed: train on a 10–25% subset, compute DU/TDDS scores there, and extrapolate to the full dataset with KNN or GNN. The experiments are broad — four datasets, three training paradigms, both score methods — and the time-accuracy Pareto plots are genuinely informative. The authors also disclose their limitations honestly, including the oversmoothing and outlier failures in Figure 5. If I worked on data pruning I would want this paper on hand.\n\nBut the paper overclaims. The sentence in Section 4 that extrapolated scores \"consistently outperform random pruning whenever the ground truth score does\" is directly contradicted by Table 1: for ImageNet at 50% pruning with DU, ground truth beats random (59.08 vs 58.56), yet both 10%-subset extrapolations fall below random (GNN 58.30, KNN 58.51). The weaker \"generally\" phrasing earlier in the same section is accurate; \"consistently\" is not. The abstract's \"requires training on only a small subset\" also inherits this problem, since the small-subset case is exactly where the benefit is most needed and where the method sometimes fails.\n\nThe second real issue is Appendix B.4: the neighborhood size k is selected by maximizing Pearson correlation against ground-truth scores on Dr, the very residual set whose scores are the extrapolation target. The correlations in Table 2 are therefore inflated, and the downstream pruning results inherit a mild form of test-set peeking — the k choice is tuned on the ground-truth scores it is trying to predict. This is addressable with a validation split or fixed k, but it should be fixed before publication.\n\nAlso missing: no comparison to existing cheap-pruning baselines like RS2 or Selection via Proxy. The time-advantage claim is made against full-training pruning, but the practical question for a single run is whether extrapolation beats these cheaper alternatives. Without that comparison, the contribution is solid but the framing is incomplete.\n\nThese are not fatal flaws. The medium-subset results are consistently better than random, the time savings are real (~1.5–2x), and the methodology is transparent enough to correct. A careful revision that fixes the overclaim, de-biases the k-selection, and adds a cheap-baseline comparison would make this a solid paper. As it stands, it deserves a serious referee but not unconditional acceptance.\n\nMy verdict: send to peer review, with a request to address the k-selection leak and the ImageNet counterexample explicitly.","headline":"Useful empirical extension of score extrapolation to DU/TDDS pruning, but the \"consistently beats random\" claim is contradicted by their own ImageNet table and the k-selection protocol leaks ground-truth information.","tokens_in":19375,"tokens_out":2334,"would_cite":true,"duration_ms":25518,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that per-sample importance scores computed on a small random subset can be extrapolated across a dataset in embedding space, so that data pruning no longer requires a full initial training pass.","keywords":["data pruning","importance score extrapolation","k-nearest neighbors","graph neural networks","dynamic uncertainty","TDDS","adversarial training","unsupervised data pruning"],"falsifier":"Construct a dataset in which the most important samples for pruning are outliers, such as a class-balanced set where a small fraction of images have unusual backgrounds or multiple objects, then compute ground-truth DU scores, extrapolate from a 20% subset, and check whether the extrapolated rankings recover those high-importance outliers; if rank correlation collapses exactly on those points while staying high on prototypical ones, the local-smoothness premise fails in the regime that matters most for pruning.","tokens_in":1459,"feed_emoji":"✂️","tokens_out":2086,"duration_ms":66654,"temperature":0.7,"pith_summary":"The paper tries to make expensive data-pruning scores usable on the first training run by never computing them for the whole dataset. It claims that importance scores computed on a random 10-25% subset can be extrapolated to every other sample in the embedding space of a model trained on that subset, using either a weighted k-nearest-neighbor average or a graph neural network. Across CIFAR-10, synthetic CIFAR-100, Places-365, and ImageNet, and for supervised, unsupervised, and adversarial training, the extrapolated scores track the full-training scores well enough that pruning with them beats random pruning whenever the true score does, with a time advantage already at the first run. The point matters because standard pruning methods like Dynamic Uncertainty and TDDS require a full training pass before pruning pays off, which makes them impractical for single large-scale runs.","feed_headline":"Score extrapolation makes data pruning practical on the first run","feed_subtitle":"Training on 10-25% of the data is enough to propagate importance scores to the whole set and prune faster than random.","key_machinery":"The carrying object is the embedding space of the subset-trained model $\\mathcal{F}_s$: all samples in the full dataset are mapped through this encoder, and the extrapolators transfer scores along local neighborhoods in that space. KNN uses a distance-weighted average of the $k$ nearest seed scores, while the GNN builds a $k$-NN graph over all samples with edge weights $\\exp(-d(\\cdot,\\cdot))$, and message passing lets scores propagate beyond direct neighbors. The load-bearing assumption is that samples that look similar to the subset-trained model have similar importance, so local averaging in this space can recover unseen scores.","core_discovery":"The central claim is that per-sample importance scores are smooth enough, as a function of a model's embedding, that a score computed on a small random subset can be propagated to the rest of the dataset with near-negligible cost. The paper demonstrates this with two propagators: a KNN weighted average and a three-layer graph convolutional network trained with neighbor sampling, both operating in the embedding space of a model trained only on the subset. On four datasets and for the DU and TDDS pruning scores, the extrapolated rankings correlate with ground-truth rankings and yield final accuracies close to full-training pruning while taking less total time. The paper also reports that GNN extrapolation generally correlates better than KNN, that KNN sits on the Pareto frontier of time versus accuracy, and that extrapolation fails to reproduce the bimodal score distribution, oversmoothing it and erring most on outlier images.","pith_inferences":["A likely boundary condition that the paper observes but does not turn into a method is that extrapolation oversmooths the score distribution; any practical extension should model bimodality or explicitly detect outliers rather than averaging over them.","The framework reads as a general cost-transfer trick: any per-sample quantity that is expensive to compute on the full dataset, such as influence functions or data attribution, could be computed on a seed subset and propagated in embedding space, though the paper only tests pruning scores.","A natural stress test would be to extrapolate scores on datasets where importance is concentrated in a thin outlier population, such as mislabeled or long-tail data, since the paper's own qualitative analysis identifies atypical images as the largest source of rank error.","The unsupervised experiment hints that the choice of embedding space matters independently of the propagator, because KNN beat GNN there; comparing extrapolation across different frozen and trained encoders would test this directly."],"forward_implications":["Pruning methods that normally need a full training pass become usable on a single large-scale run, because the measured time includes subset training, extrapolation, and final training and still shows savings already at the first run.","If the ground-truth pruning score beats random pruning, the extrapolated version of that score also beats random pruning in the reported regimes; if the ground-truth score is no better than random, extrapolation does not rescue it.","KNN extrapolation is Pareto-optimal on time versus accuracy for Places-365 and ImageNet, meaning no other tested method gives both better accuracy and less time.","The same extrapolation recipe transfers to unsupervised settings using a foundation model's embeddings and pseudo-labels, and to adversarial training, where it improves robustness over random pruning with negligible extra cost.","Correlation between extrapolated and ground-truth scores tracks downstream accuracy, so improving extrapolation quality is a direct proxy for improving pruning quality."],"supporting_citations":[{"why":"Provides the Dynamic Uncertainty pruning score that the framework extrapolates.","marker":"[7]"},{"why":"Provides the Temporal Dual-Depth Scoring (TDDS) pruning score that the framework extrapolates.","marker":"[8]"},{"why":"Supplies the synthetic CIFAR-100 and synthetic CIFAR-10 datasets and the adversarial training setup used for large-scale evaluation.","marker":"[41]"},{"why":"Supplies the graph convolutional layer used in the GNN-based extrapolator.","marker":"[68]"},{"why":"Supplies the neighbor-sampling scheme that makes GNN training scalable to large graphs.","marker":"[69]"},{"why":"Supplies the unsupervised pseudo-labeling method used to compute unsupervised DU scores.","marker":"[72]"},{"why":"Supplies the fixed foundation-model embeddings used for the unsupervised extrapolation setting.","marker":"[76]"}],"fun_headline_variants":["Prune first-run data with score extrapolation from tiny subsets","Train on 10% data, prune the rest via score extrapolation","Score extrapolation: prune without a full training pass","KNN and GNN extrapolate scores to prune without full retraining"],"cache_read_input_tokens":21504,"weakest_assumption_plain":"Importance scores are locally smooth in the embedding space of a model trained on a small random subset, so nearby points in that space carry similar scores, and the paper's own outlier analysis shows this smoothness breaks down for atypical images.","fun_headline_variants_meta":{"raw":{"variants":["Prune first-run data with score extrapolation from tiny subsets","Train on 10% data, prune the rest via score extrapolation","Score extrapolation: prune without a full training pass","KNN and GNN extrapolate scores to prune without full retraining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00075,"raw_usage":{"total_tokens":3318,"prompt_tokens":904,"completion_tokens":2414,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":2341}},"tokens_in":520,"tokens_out":2414,"duration_ms":16304,"temperature":1.0,"reasoning_tokens":2341,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:57:08.619437+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a dataset in which the most important samples for pruning are outliers, such as a class-balanced set where a small fraction of images have unusual backgrounds or multiple objects, then compute ground-truth DU scores, extrapolate from a 20% subset, and check whether the extrapolated rankings recover those high-importance outliers; if rank correlation collapses exactly on those points while staying high on prototypical ones, the local-smoothness premise fails in the regime that matters most for pruning.","supporting_citations":[{"cited_title":"Large-scale dataset pruning with dynamic uncertainty","cited_arxiv_id":null,"evidence_quote":"Provides the Dynamic Uncertainty pruning score that the framework extrapolates."},{"cited_title":"Semi-supervised classification with graph convolutional networks","cited_arxiv_id":null,"evidence_quote":"Supplies the graph convolutional layer used in the GNN-based extrapolator."},{"cited_title":"Inductive representation learning on large graphs.Advances in Neural Information Processing Systems (NeurIPS), 30, 2017","cited_arxiv_id":null,"evidence_quote":"Supplies the neighbor-sampling scheme that makes GNN training scalable to large graphs."},{"cited_title":"Let go of your labels with unsupervised transfer","cited_arxiv_id":null,"evidence_quote":"Supplies the unsupervised pseudo-labeling method used to compute unsupervised DU scores."}],"review_version":1}