Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty

Baekrok Shin; Changmin Kang; Chulhee Yun; Yeseul Cho

REVIEW 3 major objections 5 minor 2 cited by

Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that dataset pruning can be made lightweight by computing a DUAL score—difficulty times uncertainty—in the first 60 epochs, plus a Beta-sampling selection step that keeps test accuracy at 60% on ImageNet-1k even after…

desk verdict Useful, honest empirical paper whose headline ImageNet gain is not yet attributable to the DUAL score rather than the Beta sampler; worth refereeing. read the letter →

arxiv 2502.06905 v3 pith:PFK5HBEV submitted 2025-02-10 cs.LG cs.AI

Yeseul Cho , Baekrok Shin , Changmin Kang , Chulhee Yun This is my paper · ORCID

classification cs.LGcs.AI

keywords datasetpruningcoresetselectiontrainingdynamicspredictionuncertaintyexampledifficultyBetasamplingearlystoppingimageclassification

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to show that dataset pruning does not need to be more expensive than training itself, a paradox that plagues many existing methods. The authors propose a new scoring metric—DUAL—that combines example difficulty and prediction uncertainty, computed during only the early phase of training, so that the pruning decision is made before a full training run finishes. They also add a pruning-ratio-aware sampling step using a Beta distribution to avoid collapsing test accuracy when most of the dataset is discarded. The result, if correct, would make selective dataset reduction a practical, cheap preprocessing step for large-scale learning, which matters because training-cost savings are exactly the point of pruning.

What carries the argument

The core object is the DUAL score, defined as $$\mathrm{DUAL}_k(x,y) := \left(1-\bar{P}_k\right)\, \sqrt{\frac{\sum_{j=0}^{J-1}\left(P_{k+j}(y\mid x)-\bar{P}_k\right)^2}{J-1}}$$ where $\bar{P}_k$ is the average prediction probability in a window of length $J$, so the score is difficulty ($1-\bar{P}_k$) times uncertainty (the standard deviation). This single number is averaged over all windows up to the computation epoch $T$, and it is used to give each sample a sampling probability via a Beta distribution whose parameters are linear functions of the pruning ratio and of the dataset complexity hyperparameter $c_D$. The Beta sampling step is what turns a naive hard-sample selection into a hybrid difficulty-diversity method at high pruning ratios, and the toy-theorem in Appendix D shows that the DUAL score flips the ordering of two separable points strictly earlier than plain variance does.

What would settle it

Run the same DUAL+$\beta$ procedure on ImageNet-1k with a different architecture family (e.g., ViT or a ConvNeXt) while keeping the score-computation epoch at 60: if the accuracy at 90% pruning drops below the random baseline (52.3% on ResNet-34), the early-to-late ranking transfer breaks and the central claim fails to generalize. A second falsifier would be to compute the Spearman rank correlation between DUAL scores at epoch 60 and Dyn-Unc scores at epoch 90 on ImageNet-1k; if that correlation is close to zero, the 'moon-shaped' argument loses its empirical footing.

Watch

Extended reading notes

Core claim

The central claim is that a single score, the Difficulty and Uncertainty-Aware Lightweight (DUAL) score, can rank examples by their eventual usefulness during the full training run while being computed only 30–60 epochs into a 90–200 epoch schedule. The score is the product of (1 − mean prediction probability) times the standard deviation of predictions within a sliding window; the paper argues that this product identifies uncertain, hard-but-learnable samples earlier than Dyn-Unc's plain variance, because the difficulty factor corrects for the fact that early variance is dominated by easy samples that are still being learned. The paper also claims that threshold pruning at aggressive ratios is fundamentally limited, and that a Beta-distribution sampling step, whose shape depends on the pruning ratio and on the estimated complexity of the dataset, selects a better-balanced subset at extreme ratios. The strongest evidence is the ImageNet-1k result at 90% pruning: 60.0% test accuracy with ResNet-34 using only 66% of the score-computation time of the previous state-of-the-art methods, and the observation that the method removes mislabeled data almost perfectly at low pruning ratios on noisy CIFAR-100.

Load-bearing premise

The load-bearing premise is that the ranking of samples by DUAL score computed in the first 30–60 epochs reliably matches the ranking that a full-training Dyn-Unc score would produce, so that samples flagged as uncertain early are the ones that would prove most informative at the end of training.

Editorial extensions

If this is right

If DUAL's ranking is reliable at epoch 30–60, then pruning can be performed as a cheap preprocessing step: the total cost stays below a single full training run on the original dataset, which is what the current greedy baselines cannot achieve.
The noise experiments suggest that early DUAL scores separate mislabeled samples from genuinely hard samples, because noise is under-learned at that stage; this could make pruning a practical way to clean noisy datasets without explicit noise-detection training.
The cross-architecture results imply that a small model's early scores can be reused to prune for a larger model, further reducing the cost of pruning for architectures that are expensive to train from scratch.
The Beta-sampling step improves even random selection at 90% pruning on CIFAR-100, which, if correct, indicates that the biggest accuracy loss at high pruning ratios comes from representation bias—not from which score is used—and that the sampling rule itself is a reusable component for other scores.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's moon-shaped trajectory argument suggests a testable prediction: for a fixed network family and dataset, the epoch at which DUAL scores stabilize (i.e., the ranking by DUAL at epoch $T_1$ and $T_2$ becomes highly correlated) should predict the best early-stopping point. No such 'stabilization curve' is reported, but the Spearman correlations in Appendix B already give the raw material.
The fact that Beta sampling rescues even random selection at 90% pruning suggests a more general principle: at extreme pruning ratios, the value of any score may be secondary to the sampling distribution, which would motivate evaluating DUAL's Beta shape against other parametric families (e.g., truncated Gaussian) as a direct ablation; the paper does not do this.
The toy theorem is stated for two points with exponential loss and gradient descent; extending the 'earlier flip' argument to stochastic gradients, finite windows $J>1$, and the noisy-label setting the paper emphasizes would be a natural next step, and if it fails, the empirical claims would need to be attributed to network specifics rather than to the score's geometry.
If DUAL+$\beta$ is robust to label noise, then the same score, computed on a small warmup run, could be used to flag subsets for targeted human review before full training; this is a downstream application the paper does not discuss.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The practical contribution is real: computing a pruning score after 30-60 epochs instead of full training, and using a prediction-mean Beta sampler to avoid the accuracy cliff at high pruning ratios. The DUAL score itself—(1 - mean prediction) × prediction std—is a small but sensible twist on Dyn-Unc, and the efficiency claims (15% of CIFAR cost, 66% of ImageNet cost) are attractive if they replicate. The paper is also unusually honest in its ablations: Table 4 shows how much of the gain comes from the sampling alone, and the long-tailed, noise, corruption, cross-architecture, and dynamic-pruning comparisons give the empirical core real breadth. The toy theorem in Appendix D is fine as intuition, and clearly labeled as such.

The soft spots are proportionate but real. The main one is exactly what the stress-test flags: at the headline 90% ImageNet operating point, there is no random+β or EL2N+β control. On CIFAR-100 at 90%, random+β gets 51.76 and DUAL+β gets 54.54, so DUAL adds only 2.78 points; EL2N+β on CIFAR-10 nearly matches DUAL+β. Without the control on ImageNet, the 60.0% result cannot be assigned to the DUAL score rather than to the Beta sampling mechanism. That gap matters because the paper frames DUAL as the novel component, and Beta sampling is already close to BOSS. Also, the reproduced Dyn-Unc baseline underperforms random on ImageNet, which is suspicious and suggests the baseline may have been run with poor settings. ImageNet numbers are single-run, with no error bars. The "first metric to combine difficulty and uncertainty" phrasing is overstated given SIMS and BOSS, though the specific combination is new.

None of this is fatal. The central claim—that lightweight early-training pruning can be state-of-the-art at extreme ratios—is broadly supported on CIFAR and likely true on ImageNet, but the attribution of the gain needs one more experiment. The paper deserves a serious referee; I would send it out with a request for the missing ImageNet control and multi-seed runs. My own verdict would be conditional rather than clear accept.

Referee Report

3 major / 5 minor

Summary. This paper proposes DUAL (Difficulty and Uncertainty-Aware Lightweight), a per-sample score for dataset pruning defined as the product of (1 - mean prediction probability) and the standard deviation of prediction probabilities over a sliding window, averaged over an early training phase (30 epochs on CIFAR, 60 epochs on ImageNet-1k). To mitigate the accuracy collapse at high pruning ratios, the paper adds a Beta-distribution-based sampling scheme whose parameters depend on the pruning ratio so as to progressively include easier samples. Experiments on CIFAR-10/100, ImageNet-1k, label-noisy, image-corrupted, long-tailed, and cross-architecture settings report that DUAL plus Beta sampling achieves state-of-the-art accuracy at lower score-computation cost than full-training baselines, with the headline result being 60.0% top-1 accuracy on ImageNet-1k at 90% pruning using ResNet-34 after 60 epochs of score computation. A toy theoretical analysis (Theorem 3.1) claims that the DUAL score's ordering of two examples flips earlier in training than the Dyn-Unc variance score's ordering.

Significance. If the headline results hold, this is a genuinely useful contribution: it makes dataset pruning cheaper than full training and attains state-of-the-art accuracy at extreme pruning ratios on ImageNet-1k. The empirical package is broad and mostly careful: CIFAR experiments are multi-seed with error bars; the hyperparameters T, J, and c_D are ablated (Figure 6, Figure 9); and the evaluation spans noise, corruption, long-tailed distributions, cross-architecture transfer, and a comparison with dynamic pruning methods (Appendix B.5). The release of code is a further strength. The main weakness is attribution: the paper's own ablations (Table 4) show that the Beta sampler, not the DUAL score, carries most of the benefit at 90% pruning, and the corresponding control on ImageNet is missing. In addition, the closest baseline for the sampling component (BOSS) is never benchmarked, and the theoretical proof in Appendix D contains an informal small-learning-rate step. These gaps need to be closed before the central claim is fully established.

major comments (3)

[§4.2 (Table 2), §4.4 (Table 4)] The headline ImageNet result does not isolate the contribution of the DUAL score. Table 4 shows that at 90% pruning the Beta sampler, not the score, is the dominant factor: on CIFAR-100, random thresholding gives 45.09 while random+β gives 51.76 (+6.67), and the DUAL score adds only 2.78 points over random+β (54.54 vs 51.76); on CIFAR-10, EL2N+β (87.00) is within the reported error bars of DUAL+β (87.09). Table 2 reports no random+β control on ImageNet, so the 7.7-point margin of DUAL+β over Random at 90% pruning (60.0 vs 52.3) cannot be attributed to the DUAL score; since random+β gains 6.67 points over Random on CIFAR-100, random+β alone could plausibly reproduce most of the ImageNet gain. Furthermore, the DUAL score used as a plain thresholding rule at 90% on ImageNet (53.1) is barely above Random (52.3) and well below CCS (57.3), so the state-of-the-art result is created by the sampler rather than by the score. The authors should report random+β, and ideally EL2N+β and Dyn-Unc+β, on ImageNet at 80% and 90% pruning before claiming that the DUAL score is what makes the method state-of-the-art.
[§D.1 (proof of Theorem 3.1)] The strict inequality T_vm < T_v is not proved as stated. The decisive step is the final, unquantified condition: "If the learning rate is sufficiently small, then γ_V(t) cannot significantly increase in one step, allowing γ_V(t) to fall between C^{-1} and 1." This condition is exactly what converts the weak consequence T_vm ≤ T_v into the strict T_vm < T_v, and no bound on η or on the per-step increment of γ_V is provided. The supporting material is also informal: Lemma D.7 is invoked as "Theorem D.7", and Equation (18) is introduced with "assume for now" and justified by "this generally holds for sufficiently small η". Since the main text labels Theorem 3.1 as "Informal", one option is to state the strict-flip claim as a conjecture supported by the empirical figures; if it is to be called a theorem, the small-learning-rate condition needs a quantitative statement.
[§3.3 Remark; §4.1 Baselines; §4.2-4.4] The closest existing method to the Beta-sampling half of the contribution, BOSS (Acharya et al., 2024), is discussed in a Remark but is never included in any comparison table. The Remark claims that the non-linear parameterization (r^{c_D}) and the use of the prediction mean rather than a difficulty score as the PDF argument give DUAL's sampler an advantage over BOSS's mode-linear scheme, but no experiment supports this claim. Given that the paper's own ablations indicate that the sampler carries most of the benefit at high pruning ratios, BOSS is a necessary baseline; without it, the novelty and superiority of the sampling component are unverified.

minor comments (5)

[Table 2] Table 2 reports single-run ImageNet results without error bars; the headline margin over D2 (60.0 vs 55.6 at 90% pruning) should be accompanied by at least two or three runs or an explicit statement that these are single-run results, since the CIFAR tables report five-run averages.
[§4.2, Table 1; Eq. (2)] The reproduced Dyn-Unc baseline is far below Random at high pruning ratios (Table 1: CIFAR-10 at 90% = 59.67 vs Random 83.74), which is at odds with the strong results reported in the original Dyn-Unc paper; please verify that the all-window averaging in Eq. (2) and the hyperparameters match the original Dyn-Unc implementation, since a mismatch would inflate the apparent advantage of DUAL over Dyn-Unc in Tables 1 and 2.
[Appendix A.1 (TDDS)] Appendix A.1 states that TDDS hyperparameters for ImageNet 80% and 90% pruning were "set arbitrarily" because the original paper provides no settings; those rows in Table 2 should be marked as such, or the arbitrary choices documented in the table caption, to avoid giving the impression that these are the original method's reported values.
[Eq. (3); Table 2 caption; Figure 5 caption; Appendix D] Notation and typos: in Eq. (3), k ∈ [T−J+1] should read k ∈ {1, …, T−J+1}; Table 2's caption contains "raio"; Figure 5's caption contains "mislabeld"; and Appendix D refers to "Theorem D.7" where the statement is labeled "Lemma D.7".
[Appendix A.2, §B.1-B.2] The values of c_D are set per dataset and per noise level (Appendix A.2: c_D = 6 for 20%/30% noise, 8 for 40% noise, and 2 for Tiny-ImageNet corruption); please state how these were selected (for example, on a held-out validation set) so that the favorable noise-robustness results cannot be read as per-condition oracle tuning, even though Figure 6 shows robustness across c_D ∈ {3, …, 6} in the clean setting.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DUAL and Beta sampling are defined from early-training predictions and evaluated on held-out test accuracy, with no self-citation chain or definitional reduction.

full rationale

The paper's central claim is that the DUAL score, computed from early-training prediction means and standard deviations, plus pruning-ratio-adaptive Beta sampling, yields accurate coresets at low computational cost. The DUAL score (Eqs. 3-4) is a deterministic composite of the windowed mean and standard deviation of the model's predicted target probability; it is not defined in terms of test accuracy, final Dyn-Unc scores, or any other target outcome. The Beta sampler (Eq. 5) assigns selection probabilities from the prediction mean and the DUAL score, again without using test labels. All headline results (Tables 1, 2 and the CIFAR/ImageNet figures) are test-set accuracies obtained after training fresh models on the pruned subsets, so the evaluation is external to the score construction. The theoretical result (Theorem 3.1, Appendix D) proves, for a two-point linearly separable toy model and under Assumption D.1, that the order flip for the variance-times-mean-complement score occurs strictly earlier than for variance alone; it invokes the external implicit-bias result of Soudry et al. and does not assume the conclusion. No load-bearing self-citations appear: the authors do not cite their own prior work, and no uniqueness theorem from the authors is used to force a choice. The hyperparameters c_D, C, T, and J are tuned per dataset, which is ordinary model selection rather than circularity. The nearest concern is that the ImageNet 90% operating point does not isolate the marginal contribution of DUAL from Beta sampling (no random+beta or EL2N+beta control is reported there); Table 4 on CIFAR even shows that replacing DUAL with EL2N under Beta sampling gives similar accuracy. That is an experimental attribution gap, not a definitional reduction: the score and the sampler are separable components, and neither is fitted to the reported test labels. Therefore no circular step can be exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central empirical claim rests mainly on hyperparameters chosen by hand (c_D, C, T, J) and on external theorems about implicit bias; no entities are invented. The hyperparameters are tuned per dataset, which is standard practice, but they are load-bearing for the reported SOTA margins.

free parameters (4)

c_D (Beta distribution exponent) = 5.5 (CIFAR-10), 4 (CIFAR-100), 11 (ImageNet-1k), 6 (label-noise CIFAR-100), 6/2 (Tiny-ImageNet)
Controls how rapidly Beta sampling shifts probability mass toward easy samples as pruning ratio increases; chosen by hand per dataset based on perceived dataset complexity, and affects high-pruning accuracy.
C (Beta distribution scale) = 15
Scale of Beta parameters; larger C reduces variance and concentrates sampling; fixed across all experiments by the authors.
T (score computation epochs) = 30 (CIFAR), 60 (ImageNet-1k), 50/60 (noise/corruption)
Epoch at which scores are computed; a core design choice for 'lightweight' pruning, selected per dataset; ablation shows 20-50 is robust, but T is still a user-set hyperparameter.
J (sliding window length) = 10
Sliding-window length for prediction variance; fixed and shown robust in Fig. 9, but still a free choice.

assumptions (4)

standard math Soudry et al. (2018) directional convergence and logarithmic growth of gradient descent on separable linear classification with exponential loss.
Invoked in Appendix D.1 (Theorem D.2 and Lemma D.3) to analyze the two-point toy example; not proved in this paper.
domain assumption Exponential loss is a valid proxy for logistic/cross-entropy loss in the implicit-bias regime.
Section D.1 uses exponential loss for tractability, citing Soudry et al. and Gunasekar et al. for similar implicit bias; the gap between this loss and the cross-entropy used in experiments is not formally bridged.
domain assumption Samples with mislabeled or corrupted content show low prediction uncertainty in the early training phase and can be identified by low DUAL scores.
Used to justify noise robustness (Section 4.3.1, Fig. 13); empirically validated on CIFAR-100 but not guaranteed for other noise types.
ad hoc to paper The learning rate is sufficiently small so that the variance-ratio sequence gamma_V(t) does not jump over the critical interval, ensuring T_vm < T_v.
Introduced in Appendix D.1 after Eq. (17) to close the proof gap; no specific bound is given, only a qualitative statement.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty." pith.science (2026). https://pith.science/paper/PFK5HBEV

@misc{pith2026250206905,
  author       = {Pith},
  title        = {Pith review of: Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PFK5HBEV}},
  note         = {Machine review of arXiv:2502.06905}
}

read the original abstract

Recent advances in deep learning rely heavily on massive datasets, leading to substantial storage and training costs. Dataset pruning aims to alleviate this demand by discarding redundant examples. However, many existing methods require training a model with a full dataset over a large number of epochs before being able to prune the dataset, which ironically makes the pruning process more expensive than just training the model on the entire dataset. To overcome this limitation, we introduce a Difficulty and Uncertainty-Aware Lightweight (DUAL) score, which aims to identify important samples from the early training stage by considering both example difficulty and prediction uncertainty. To address a catastrophic accuracy drop at an extreme pruning, we further propose a ratio-adaptive sampling using Beta distribution. Experiments on various datasets and learning scenarios such as image classification with label noise and image corruption, and model architecture generalization demonstrate the superiority of our method over previous state-of-the-art (SOTA) approaches. Specifically, on ImageNet-1k, our method reduces the time cost for pruning to 66% compared to previous methods while achieving a SOTA, specifically 60% test accuracy at a 90% pruning ratio. On CIFAR datasets, the time cost is reduced to just 15% while maintaining SOTA performance.

Figures

Figures reproduced from arXiv: 2502.06905 by the authors.

**Figure 1.** Test accuracy comparison on CIFAR datasets (Left: Results for CIFAR-10, Right: Results for CIFAR-100). The color represents the total computation time, including the time spent training the original dataset for score calculation, for each pruning method. Blue indicates lower computation time, while red indicates higher computation time. Our method demonstrates its ability to minimize computation time while maintaini… view at source ↗

**Figure 2.** The left column (“Epoch 60”) shows the prediction mean and standard deviation, computed using the predicted target probabilities up to epoch 60. The right column (“Epoch 90”) shows corresponding values up to epoch 90. In each row, samples are colored by the normalized Dyn-Unc score computed at epoch 60 for Figure 2a and at epoch 90 for Figure 2b. The epoch at which the score was computed is indicated by a bold outl… view at source ↗

**Figure 3.** Our DUAL score targets similar uncertain samples in the early epoch of 60 (highlighted in bold). Selected samples are finally located in the most uncertain region when the whole training processes are considered. However, score-based approaches including our method, suffer from accuracy drop at the high pruning ratio due to biased representations. To address this, we propose an additional sampling strategy that adap… view at source ↗

Figures from the paper (22 more)

**Figure 4.** Figure 4: Comparison in total time spent (full dataset training, score estimation, and subset training) on CIFAR datasets. While other methods remain ineffective as they require more than full training, our method achieves a 15.5% time reduction with only 30% pruning, approachin…

**Figure 5.** Figure 5: The left figure shows the ratio of pruned mislabeled data under 20% label noise on CIFAR-100 trained with ResNet-18. When label noise is 20%, the optimal value (black dashed line) corresponds to pruning 100% of mislabeled data at a 20% pruning ratio. The middle and rig…

**Figure 6.** Figure 6: Left: T varying while J = 10 and cD = 4. Right: cD varying while T = 30 and J = 10. Three runs are averaged. Beta sampling with existing scores Next, we study the impact of our proposed pruning-ratio-adaptive Beta sampling on existing score metrics. We apply our Beta …

**Figure 7.** Figure 7: Average of Spearman rank correlation among independent runs and overall average of five runs. DUAL score is calculated at 30th epoch. Next, we compute the Dyn-Unc, TDDS, and AUM scores at the 30th epoch, as we do for our method, and then compare the test accuracy on th…

**Figure 8.** Figure 8: Test accuracy comparison under limited computation budget (epoch 30) 14 [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

**Figure 9.** Figure 9: J varies from 5 to 15, showing minimal differences, which demonstrates its robustness. We fix T = 30, CD = 4. Runs are averaged over three runs [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

**Figure 10.** Figure 10: is a visualization of samples kept and pruned by our method. Samples kept by our method are more recognizable. The black swan on the grass and the sun-shaped balloon are rare cases, while the others are more easily recognizable. Samples pruned by our method are either…

**Figure 11.** Figure 11: Comparison of selected subset over different methods at high pruning rates. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]

**Figure 12.** Figure 12: Ratio of pruned mislabeled data under 30% and 40% label noise on CIFAR-100 [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]

**Figure 13.** Figure 13: shows a scatter plot of the CIFAR-100 dataset under 20% label noise. The model is trained for 30 epochs, and we compute the prediction mean (y-axis) and standard deviation (x-axis) for each data point. Red dots represent the 20% mislabeled data. These points remain cl…

**Figure 14.** Figure 14: Examples of the different types of noise used for image corruption. Here we consider motion blur, fog, resolution, rectangle, and Gaussian noise. 10 20 30 40 50 60 70 80 90 Pruning Ratio (%) 0 20 40 60 80 100 Ratio of Pruned Corrupted Data (%) Optimal Random EL2N Dyn-…

**Figure 15.** Figure 15: Ratio of pruned corrupted samples with corruption rate of 20%, 30% and 40% on CIFAR-100. 20 40 60 80 Pruning Ratio (%) 0 20 40 60 80 100 Ratio of Pruned Corrupted Data Motion Random EL2N Dyn-Unc Forgetting DUAL DUAL+ 20 40 60 80 Pruning Ratio (%) Fog 20 40 60 80 Pruni…

**Figure 16.** Figure 16: Illustration of the different types of noise used for image corruption. DUAL pruning prioritizes removing the most challenging corrupted images, such as fog and Gaussian noise. We evaluated the performance of our proposed method across a wide range of pruning levels, …

**Figure 17.** Figure 17: This figure shows test accuracy (y-axis) versus the total number of iterations needed to fully train each subset (x-axis). The top figure corresponds to CIFAR-10, and the bottom to CIFAR-100. Results are averaged over five random seeds. For the static pruning methods,…

**Figure 18.** Figure 18: This figure shows the selection frequency of each sample at a high pruning ratio (90%). The x-axis represents data indices sorted in descending order by selection frequency, and the y-axis indicates the normalized selection ratio—i.e., how often each data point was se…

**Figure 19.** Figure 19: Visualization of Beta distribution for varying C. Large C enables more concentrated targeting. Now we justify our choice of parameters αr and βr in the Beta distribution. When the pruning ratio is set to zero, αr and βr are configured so that the mean of the Beta dist…

**Figure 20.** Figure 20: Visualization of Beta distribution for varying cD. Left subplot corresponds to the value used in CIFAR-10, and the right subplot corresponds to the value used in CIFAR-100. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_20.png]

**Figure 21.** Figure 21: Pruning visualization on CIFAR-100. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_21.png]

**Figure 22.** Figure 22: Illustration of the evolution of the weight as the model learns from the two-point dataset. Observe that the weight learns x2 first (closer to the orange dashed line), but gradually moves towards x1 (closer to the brown dashed line). Here T = 10, 000. We also empirica…

**Figure 23.** Figure 23: Empirical validations of the critical statements in Section D.1.1. We ran experiments and plot the results that both γV (t) (left—in log scale) and ∆ζt (right) are an increasing sequence in terms of t. Here, we set η = 0.0005. The reason is that if the learning rate i…

**Figure 24.** Figure 24: Empirical validations of the critical statements in Section D.1.1. We ran experiments and plot the results that both ζ (1) t (left) and ζ (2) t (right) are extremely close to the midpoint (y (1) t + y (1) t+1)/2 and (y (2) t + y (2) t+1)/2, compared to the interval le…

**Figure 25.** Figure 25: Evolution of x1, x2 by their mean and standard deviation in prediction probabilities at different epochs. The marker ‘o’ and ‘x’ stands for x1 and x2, respectively. The red color indicates the sample to be selected, and the blue color indicates the sample to be pruned…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Model Parallelism With Subnetwork Data Parallelism
cs.LG 2025-07 conditional novelty 6.0 of 10

Training each GPU on a fixed overlapping subnetwork and averaging shared parameters cuts per-device memory by up to 60 percent without exchanging activations, matching DDP accuracy under FLOP-matched budgets.
Extending Dataset Pruning to Object Detection: A Variance-based Approach
cs.CV 2025-05 conditional novelty 5.0 of 10

A variance-based prediction score using IoU and confidence fluctuations across epochs improves dataset pruning for object detection over several baselines.

Reference graph

Works this paper leans on

33 extracted references · 18 canonical work pages · cited by 2 Pith papers

[1]

Balancing feature similarity and label variability for optimal size-aware one-shot subset selection

Abhinab Acharya, Dayou Yu, Qi Yu, and Xumin Liu. Balancing feature similarity and label variability for optimal size-aware one-shot subset selection. In Forty-first International Conference on Machine Learning, 2024

work page 2024
[2]

A closer look at memorization in deep networks

Devansh Arpit, Stanis aw Jastrz e bski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning, pages 233--242. PMLR, 2017

2017
[3]

Curriculum learning

Yoshua Bengio, J \'e r \^o me Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41--48, 2009

work page 2009
[4]

BWS: Best Window Selection Based on Sample Scores for Data Pruning across Broad Ranges

Hoyong Choi, Nohyun Ki, and Hye Won Chung. Bws: Best window selection based on sample scores for data pruning across broad ranges. arXiv preprint arXiv:2406.03057, 2024

work page Pith review arXiv 2024
[5]

Selection via proxy: Efficient data selection for deep learning, 2020

Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning, 2020. URL https://arxiv.org/abs/1906.11829

arXiv 2020
[6]

Data and parameter scaling laws for neural machine translation

Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural machine translation. In ACL Rolling Review - May 2021, 2021. URL https://openreview.net/forum?id=IKA7MLxsLSu

work page 2021
[7]

Implicit bias of gradient descent on linear convolutional networks

Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. Advances in neural information processing systems, 31, 2018

2018
[8]

Deep residual learning for image recognition, 2015

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. URL https://arxiv.org/abs/1512.03385

arXiv 2015

Show all 33 references

[9]

Large-scale dataset pruning with dynamic uncertainty

Muyang He, Shuo Yang, Tiejun Huang, and Bo Zhao. Large-scale dataset pruning with dynamic uncertainty. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7713--7722, 2024

2024
[10]

Benchmarking neural network robustness to common corruptions and perturbations

Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261, 2019

1903 arXiv
[11]

Deep learning scaling is predictable, empirically

Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mos-tofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017

2017 arXiv
[12]

Characterizing structural regularities of labeled data in overparameterized models

Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Characterizing structural regularities of labeled data in overparameterized models. arXiv preprint arXiv:2002.03206, 2020

2002 arXiv
[13]

Selective-supervised contrastive learning with noisy labels

Shikun Li, Xiaobo Xia, Shiming Ge, and Tongliang Liu. Selective-supervised contrastive learning with noisy labels. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 316--325, 2022

2022
[14]

D2 pruning: Message passing for balancing diversity and difficulty in data pruning

Adyasha Maharana, Prateek Yadav, and Mohit Bansal. D2 pruning: Message passing for balancing diversity and difficulty in data pruning. arXiv preprint arXiv:2310.07931, 2023

2023 arXiv
[15]

Making deep neural networks robust to label noise: A loss correction approach

Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1944--1952, 2017

1944
[16]

Deep learning on a data diet: Finding important examples early in training

Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in neural information processing systems, 34: 0 20596--20607, 2021

2021
[17]

Elenberg, and Kilian Q

Geoff Pleiss, Tianyi Zhang, Ethan R. Elenberg, and Kilian Q. Weinberger. Identifying mislabeled data using the area under the margin ranking, 2020. URL https://arxiv.org/abs/2001.10528

2020 arXiv
[18]

A constructive prediction of the generalization error across scales

Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. arXiv preprint arXiv:1909.12673, 2019

1909 arXiv
[19]

Data augmentation as feature manipulation

Ruoqi Shen, S \'e bastien Bubeck, and Suriya Gunasekar. Data augmentation as feature manipulation. In International conference on machine learning, pages 19773--19808. PMLR, 2022

2022
[20]

Very deep convolutional networks for large-scale image recognition, 2015

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2015. URL https://arxiv.org/abs/1409.1556

2015 arXiv
[21]

Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Sys...

2022
[22]

The implicit bias of gradient descent on separable data

Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19 0 (70): 0 1--57, 2018

2018
[23]

Dataset cartography: Mapping and diagnosing datasets with training dynamics

Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Proce...

2020
[24]

Data pruning by information maximization

Haoru Tan, Sitong Wu, Wei Huang, Shizhen Zhao, and XIAOJUAN QI. Data pruning by information maximization. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=93XT0lKOct

2025
[25]

An empirical study of example forgetting during deep neural network learning

Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:1812.05159, 2018

2018 arXiv
[26]

Iterative learning with open-set noisy labels

Yisen Wang, Weiyang Liu, Xingjun Ma, James Bailey, Hongyuan Zha, Le Song, and Shu-Tao Xia. Iterative learning with open-set noisy labels. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8688--8696, 2018

2018
[27]

Robust early-learning: Hindering the memorization of noisy labels

Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nannan Wang, Zongyuan Ge, and Yi Chang. Robust early-learning: Hindering the memorization of noisy labels. In International conference on learning representations, 2020

2020
[28]

Instance correction for learning with open-set noisy labels

Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Jun Yu, Gang Niu, and Masashi Sugiyama. Instance correction for learning with open-set noisy labels. arXiv preprint arXiv:2106.00455, 2021

2021 arXiv
[29]

Moderate coreset: A universal method of data selection for real-world data-efficient deep learning

Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. In The Eleventh International Conference on Learning Representations, 2022

2022
[30]

Mind the boundary: Coreset selection via reconstructing the decision boundary

Shuo Yang, Zhe Cao, Sheng Guo, Ruiheng Zhang, Ping Luo, Shengping Zhang, and Liqiang Nie. Mind the boundary: Coreset selection via reconstructing the decision boundary. In Forty-first International Conference on Machine Learning, 2024

2024
[31]

Spanning training progress: Temporal dual-depth scoring (tdds) for enhanced dataset pruning

Xin Zhang, Jiawei Du, Yunsong Li, Weiying Xie, and Joey Tianyi Zhou. Spanning training progress: Temporal dual-depth scoring (tdds) for enhanced dataset pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26223--26232, 2024

2024
[32]

Coverage-centric coreset selection for high pruning rates

Haizhong Zheng, Rui Liu, Fan Lai, and Atul Prakash. Coverage-centric coreset selection for high pruning rates. arXiv preprint arXiv:2210.15809, 2022

2022 arXiv
[33]

Probabilistic bilevel coreset selection, 2023

Xiao Zhou, Renjie Pi, Weizhong Zhang, Yong Lin, and Tong Zhang. Probabilistic bilevel coreset selection, 2023. URL https://arxiv.org/abs/2301.09880

2023 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

Balancing feature similarity and label variability for optimal size-aware one-shot subset selection

Abhinab Acharya, Dayou Yu, Qi Yu, and Xumin Liu. Balancing feature similarity and label variability for optimal size-aware one-shot subset selection. In Forty-first International Conference on Machine Learning, 2024

work page 2024

[2] [2]

A closer look at memorization in deep networks

Devansh Arpit, Stanis aw Jastrz e bski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning, pages 233--242. PMLR, 2017

2017

[3] [3]

Curriculum learning

Yoshua Bengio, J \'e r \^o me Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41--48, 2009

work page 2009

[4] [4]

BWS: Best Window Selection Based on Sample Scores for Data Pruning across Broad Ranges

Hoyong Choi, Nohyun Ki, and Hye Won Chung. Bws: Best window selection based on sample scores for data pruning across broad ranges. arXiv preprint arXiv:2406.03057, 2024

work page Pith review arXiv 2024

[5] [5]

Selection via proxy: Efficient data selection for deep learning, 2020

Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning, 2020. URL https://arxiv.org/abs/1906.11829

arXiv 2020

[6] [6]

Data and parameter scaling laws for neural machine translation

Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural machine translation. In ACL Rolling Review - May 2021, 2021. URL https://openreview.net/forum?id=IKA7MLxsLSu

work page 2021

[7] [7]

Implicit bias of gradient descent on linear convolutional networks

Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. Advances in neural information processing systems, 31, 2018

2018

[8] [8]

Deep residual learning for image recognition, 2015

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. URL https://arxiv.org/abs/1512.03385

arXiv 2015