{"id":"547438b5-d61d-456d-880a-6fa2eee9ecb6","arxiv_id":"2506.06866","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SAFE couples sharpness-aware minimization with an augmented Lagrangian sparsity constraint, yielding pruned models with better accuracy and noise robustness than standard baselines.","lead":"A new pruning method, called SAFE, trains networks to land on solutions that are both sparse and flat, which keeps accuracy high after most weights are removed. On image classification it beats earlier pruning baselines, and on large language models its extension SAFE+ improves perplexity over SparseGPT, Wanda, and ALPS.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No ablation removes the sharpness perturbation, so the central claim that flatness drives SAFE's gains over baselines is empirically untested.","rationale":"I read the paper as an empirical claim: SAFE/SAFE+ outperform strong baselines, and the explanation is that the min-max formulation finds flat sparse solutions. The strongest empirical evidence (Tables 1, 2, 4, 7) is consistent and the method is plausible. The load-bearing weak point is causal attribution: the experiments do not isolate the flatness term. Baselines like ADMM also use a penalty/dual update; SAFE+ also uses Wanda projection width; and for LLMs SAFE uses 30 epochs of block-wise REM refinement, so compute differs. Without a rho=0 ablation, a skeptical reader cannot tell whether the SAM perturbation matters or whether the gains come from the ADMM constraint-satisfaction schedule (which itself moves weights gradually toward a sparse set) or from longer iterative refinement. The convergence proof, as written, cannot resolve this because it assumes an inner-loop stationarity and a decaying rho schedule that Algorithm 1 does not use; hence the theory does not currently establish that the implemented update minimizes the flatness objective. A rho=0 ablation is cheap, decisive, and exactly targets the paper's title claim. I did not identify a fatal flaw in the empirical numbers; the reader's conditional verdict is appropriate, so I keep the verdict unchanged but add a requirement: if the rho=0 ablation matches SAFE, the central flatness claim should be removed or substantially weakened.","tokens_in":26599,"tokens_out":12696,"duration_ms":137524,"concrete_test":"Run SAFE on ResNet-20/CIFAR-10 at 90%, 95%, 98%, and 99.5% sparsity with the paper's hyperparameters (Table 5/6) but set the perturbation radius rho=0 (or 1e-6), keeping the lambda cosine schedule, dual-update interval, epochs, and batch-norm tuning identical. Compare mean and standard deviation over three seeds against Table 7. If the rho=0 ablation is within 0.2 percentage points of SAFE at every sparsity, the sharpness perturbation contributes little, and the central flatness claim is not supported. As a secondary check, run the same rho=0 ablation for SAFE+ on LLaMA-2-7B at 50% sparsity and compare Wikitext2/C4 perplexity to Table 1; if the perplexity matches within 0.1, the flatness mechanism is not necessary for the LLM results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that formulating pruning as the sparsity-constrained min-max problem (3) and enforcing flatness via the SAM-style perturbation (Section 3.3) is what yields improved subnetworks. But every experiment varies multiple factors at once: SAFE runs an iterative penalty-based optimization with a dual-update interval and a cosine lambda schedule, SAFE+ adds a Wanda/Hessian-based projection, and the LLM experiments add 30 epochs of block-wise REM refinement. No experiment holds these factors fixed and removes only rho. The flatness evidence in Section 4.1 is a single MLP comparison against ADMM showing a lower Hessian eigenvalue; it does not establish that the perturbation is necessary for the accuracy gains, and it lacks reported error bars. The convergence analysis cannot fill this gap: Lemma 3.5 requires sum eta(t) rho(t) < infinity and an x-subproblem solved to stationarity, but Algorithm 1 uses a constant rho and a single gradient descent step per outer iteration, so the stated guarantee does not apply to the implemented method. If a rho=0 version of SAFE matches SAFE's accuracy, the method's improvement is better explained by the ADMM constraint-satisfaction mechanism or by additional compute than by flatness, and the paper's title and motivation overstate the role of flat minima.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SAFE, a pruning method that formulates sparsity-constrained training as the min-max problem min_{||x||_0<=d} max_{||epsilon||_2<=rho} f(x+epsilon), solved via an augmented Lagrangian (ADMM) scheme with hard-thresholding projection, and SAFE+, which replaces the Euclidean projection with a generalized weighted projection based on saliency matrices such as the Hessian diagonal or Wanda-style activation scores. The authors provide a convergence analysis based on smoothness and weak-convexity assumptions, evaluate SAFE/SAFE+ on CIFAR-10/100 with VGG and ResNet models and on LLaMA-2/3 at several sparsities, and report that the methods match or exceed ADMM, GMP, LTH, PBW, MLPrune, SparseGPT, Wanda, and ALPS. They also report robustness to label noise, common corruptions, and adversarial perturbations. The central conceptual claim is that explicitly seeking flat minima is what drives the pruning gains.","tokens_in":26886,"tokens_out":8503,"duration_ms":74797,"significance":"If the empirical results hold, the paper makes a useful contribution: it connects SAM-style flatness optimization with ADMM-based sparsification, and the evaluation is substantial (three seeds, multiple architectures, several sparsities, two image datasets, and three LLMs). The generalized projection view of saliency-based pruning in SAFE+ is a clean unifying idea, and the authors provide implementation code and a wall-clock comparison against ALPS. However, the theoretical guarantee is not matched to the implemented algorithm, and the absence of a rho=0 ablation means the flatness mechanism is not empirically isolated. The significance of the paper therefore depends on the authors adding the missing controls and correcting the convergence statement.","major_comments":[{"comment":"The convergence guarantee stated in Lemma 3.5 does not apply to Algorithm 1 as implemented. The lemma assumes a sequence with sum eta(t) rho(t) < infinity and limsup rho(t) < 1/beta, but the experiments use a constant rho (Section B.1) and Algorithm 1 performs a single gradient step per outer iteration (lines 12-13). Moreover, the proof of Corollary 3.6 in Appendix A.2 explicitly assumes that each x_{k+1} is found by running Equation (5) until convergence, which is not what the pseudocode does. Please either revise the theorem to match the implemented algorithm (for example, with an inexact or finite-time ADMM analysis) or state clearly that the experiments use a variant for which the stated guarantee is not established.","section":"Sections 3.5-3.6, Algorithm 1, Lemma 3.5"},{"comment":"The proof of Lemma A.4 sets hat beta := beta - mu and invokes 'hat beta-smoothness' of hat L, but Lemma A.2 establishes that hat L is (beta + lambda)-smooth and (lambda - mu)-strongly convex. The subsequent estimates in Equations (7)-(10) therefore use the wrong smoothness constant. The proof is internally inconsistent, and the condition lambda > mu from Lemma A.2 is not connected to the experimental choices of lambda, which are as small as 10^{-4} in Table 5. This needs to be corrected before the convergence claim can be accepted.","section":"Appendix A.1, proof of Lemma A.4"},{"comment":"No ablation removes the perturbation (rho=0). The paper's central claim is that flatness enforcement is what improves pruning, but SAFE differs from the ADMM baseline in several ways: the SAM gradient, the cosine lambda schedule, the dual-update interval, and the projection rule. The ablations in Appendix F cover lambda, K, and the schedule, but not rho. A rho=0 control, identical to SAFE in every other respect, is necessary to attribute the gains to flatness rather than to the ADMM constraint-satisfaction mechanism or to the additional compute. Without it, the title-level conclusion is not empirically supported.","section":"Section 4.2 and Appendix F"},{"comment":"The LLM comparison is not matched in compute or optimization iterations. According to Section B.4, SAFE/SAFE+ run 30 epochs of block-wise reconstruction-error minimization, while SparseGPT and Wanda are one-shot methods; Table 11 reports wall-clock times of 310.68 seconds for SAFE versus 15.82 seconds for SparseGPT and 3.98 seconds for Wanda on the same GPU. The perplexity gains in Table 1 may therefore reflect the additional optimization budget rather than the proposed mechanism. Please add an equal-compute or equal-iteration comparison, or report how much of the gain remains after a comparable number of iterations.","section":"Section 4.3, Table 1, Appendix E"},{"comment":"The claim that 'Wanda corresponds to taking P=diag(A^T A)' is imprecise. Wanda's saliency is |W_{ij}| * ||x_j||, the product of weight magnitude and input column norm, whereas the generalized projection with P=diag(A^T A) keeps coordinates according to |x_i + u_i| * ||x_i|| (up to squaring). The correspondence holds only approximately when x+u is close to the final weight. Please state the precise relationship and avoid overclaiming the equivalence.","section":"Section 3.4"}],"minor_comments":[{"comment":"The set A is used in Assumption 3.1 and Definition 3.4 but is never defined; it should be defined explicitly as the sparsity constraint set {x : ||x||_0 <= d}.","section":"Assumption 3.1 and Definition 3.4"},{"comment":"PBW, MLPrune, and LTH entries in Table 7 report single numbers with no standard deviations, while SAFE and other baselines include three-seed statistics; please clarify whether these are single runs or values taken from the original papers.","section":"Table 7"},{"comment":"The derivation silently drops the gradient of epsilon*(x) when forming the update rule; the text calls this removing higher-order terms, but it should be presented explicitly as an approximation with the neglected term identified.","section":"Section 3.3, Equation (4)"},{"comment":"The appendix numbering is inconsistent: Lemma 3.5 is referred to as 'Theorem 3.5' and 'Theorem A.4', and Appendix A.2 is titled 'Proof of Theorem 3.6' for what is Corollary 3.6; please fix the numbering.","section":"Appendix A"},{"comment":"The proof ends with 'hat L(x^{(t)}) -> 0', but the lemma statement requires 'nabla hat L(x^{(t)}) -> 0'; this is a typographical error that should be corrected.","section":"Appendix A.1, final line of Lemma A.4 proof"},{"comment":"The manuscript says the code is provided but does not give a repository URL; please include a link or footnote with the exact location of the code.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The empirical core is substantial and the paper is likely to be of interest to the pruning community, but the missing rho=0 ablation and the compute mismatch in the LLM experiments are exactly the points that reviewers will press on. The convergence proof is also currently inconsistent with the implemented algorithm. If the authors can supply a rho=0 control, an equal-compute comparison, and a corrected or clearly scoped convergence theorem, I would be willing to support acceptance; otherwise the central flatness claim remains unsupported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Should know: this is the first pruning method I know that puts the sparsity constraint and the SAM-style sharpness objective into a single constrained min-max problem and solves it with an augmented Lagrangian. The empirical core is substantial, and the LLM results are the strongest part: SAFE+ beats SparseGPT, Wanda, and ALPS on perplexity across three LLaMA models at 50%, 60%, 4:8, and 2:4 sparsities. That is a concrete result worth taking seriously.\n\nWhat is new: the formulation (3) and the generalized projection in Section 3.4 with P=I, diag(Hessian), diag(g^2), or diag(A^T A) recovering magnitude pruning, OBD, SNIP, and Wanda, respectively, is a fresh way to unify these scoring rules under one optimization. The ADMM treatment is principled, and the comparison with CrAM and IMP+SAM is fair and informative.\n\nSoft spots, in order of how much they matter. First, the convergence theory does not apply to the implemented algorithm. Lemma 3.5 assumes sum eta(t) rho(t) < inf and that the x-subproblem is solved to stationarity; the algorithm uses a single gradient step and constant rho. Corollary 3.6 just points at Huang et al. (2021) without checking the adaptation. In the appendix, Lemma A.4 defines beta-hat = beta - mu while Lemma A.2 states (beta+lambda)-smoothness, so the constants are inconsistent. This is more a sketch than a proof. It does not affect the empirical claims, but it should be flagged. Second, Algorithm 1's pseudocode references x(t+1) in the z-projection and u-update before those iterates are computed; almost certainly a typo for x(t), but as written it is ambiguous. Third, there is no rho=0 ablation. The paper attributes gains to flatness, and there is one MLP flatness visualization (Figure 1), but no experiment removes the perturbation while holding everything else fixed. Given the penalty schedule and BNT are also changing, a rho=0 condition would have isolated the sharpness component. This is a missed test, not a fatal gap. Minor: the abstract says SAFE improves LLM perplexity, but SAFE+ is the consistent winner; SAFE itself is competitive but sometimes worse than ALPS on LLaMA-3. Also, the paper says code is provided but no URL appears in the text.\n\nThe empirical work is solid: three seeds, multiple architectures, hyperparameters tuned on validation splits, external baselines. The issues are mechanical and addressable. This paper deserves a serious referee, and I would cite it in my own work on pruning.","headline":"Strong empirical pruning paper with a novel unified formulation, but the convergence analysis is a sketch and no rho=0 ablation isolates the flatness claim.","tokens_in":27431,"tokens_out":7371,"would_cite":true,"duration_ms":68875,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pruning that seeks flat minima beats standard baselines","keywords":["neural network pruning","sharpness-aware minimization","flat minima","augmented Lagrangian","ADMM","L0 sparsity","post-training pruning","large language models"],"falsifier":"Measure the largest Hessian eigenvalue of SAFE-pruned and ADMM-pruned networks at matched sparsity and accuracy; if SAFE's subnetworks are not consistently flatter, the flatness mechanism is not what carries the gains. Alternatively, construct a loss landscape where the SAM perturbation direction does not align with the sharp direction and check whether SAFE still beats magnitude pruning, which would show the gains come from something other than flatness.","tokens_in":26353,"feed_emoji":"✂️","tokens_out":4610,"duration_ms":45158,"temperature":0.7,"pith_summary":"The paper argues that the performance drop from pruning is partly a flatness problem: pruned subnetworks tend to sit in sharp minima. It formulates pruning as a sparsity-constrained min-max problem that minimizes the worst-case loss over a small ball around the weights while keeping most weights zero, and solves it with an augmented Lagrangian, ADMM-style procedure called SAFE. A generalized projection step (SAFE+) folds saliency scores such as Hessian, gradient, or activation-based criteria into the same framework. The paper reports that the resulting sparse networks generalize better than established baselines on CIFAR image classification and on LLaMA language-model pruning, and are more robust to label noise and input corruption.","feed_headline":"Flat-minima pruning outperforms established baselines","feed_subtitle":"A dual-projection method keeps sparse networks accurate past 99% sparsity and beats SparseGPT on LLaMA models.","key_machinery":"The central object is the augmented Lagrangian of the sparsity-constrained min-max problem: with variables $x$ (weights), $z$ (sparse proxy), and dual variable $u$, the update alternates SAM-style gradient steps on $x$, hard-thresholding (or generalized saliency) projection onto the $\\ell_0$ constraint for $z$, and dual ascent on $u$. The flatness mechanism is the SAM approximation that the worst-case perturbation points along the gradient, $\\epsilon^\\star = \\rho \\, \\nabla f(x) / \\|\\nabla f(x)\\|_2$, so the $x$-update evaluates the gradient at $x + \\epsilon^\\star$. The generalized projection uses a positive-definite diagonal matrix $P$ to reweight the distance metric, so different saliency scores become special cases of the same projection operation.","core_discovery":"The central claim is that jointly enforcing sparsity and flatness produces subnetworks that are both more accurate and more robust than those found by sparsity-only or sharpness-only methods. Concretely, the paper proposes the problem $\\min_{\\|x\\|_0 \\le d} \\max_{\\|\\epsilon\\|_2 \\le \\rho} f(x+\\epsilon)$ and solves it through variable splitting, an augmented Lagrangian, and a projection onto the $\\ell_0$ ball, with the inner maximization handled by the SAM first-order approximation. The extension SAFE+ replaces the Euclidean projection with a generalized $P$-norm projection, so magnitude-, Hessian-, gradient-, or activation-based saliency can be used as the sparsification criterion. The evidence is reported as consistently higher validation accuracy for SAFE on VGG-19 and ResNet models over CIFAR-10/100 at 90% to 99.5% sparsity, lower perplexity for SAFE+ than SparseGPT, Wanda, and ALPS across LLaMA-2-7B/13B and LLaMA-3-8B, and larger accuracy margins over ADMM under label noise.","pith_inferences":["The paper leaves untested whether composing multiple saliencies inside $P$ (for example Hessian and activation information together) would further improve SAFE+, since the generalized projection allows such combinations naturally.","The gap between the convergence proof, which assumes the $x$-minimization is solved to stationarity, and the implemented single gradient step implies a checkable variant: running the $x$-minimization to convergence should either improve the reported results or reveal how much the theory-practice gap matters.","If flatness is the true driver of the gains, SAFE's benefits should transfer to other settings where sharp minima hurt generalization, such as transfer learning or continual learning; that is an implicit prediction the paper does not test.","The robustness results suggest an extension to adversarial training: the paper measures adversarial robustness but does not train adversarially, and SAM-style flatness is believed to relate to adversarial robustness, so combining SAFE with adversarial training is a natural next test."],"forward_implications":["If correct, pruning during training can be reframed as a constrained robust-optimization problem, giving a principled alternative to heuristic masking.","SAFE+ shows that saliency-based selection such as Wanda, OBD, and SNIP can be subsumed under one projection framework, so new saliency scores could be plugged in without changing the optimization.","The observed robustness to label noise and input corruption suggests flat sparse solutions are preferable in noisy real-world deployments.","The method does not require multiple retraining rounds, unlike lottery-ticket or rewind-based pruning, which lowers the cost of reaching extreme sparsity."],"supporting_citations":[{"why":"Supplies the sharpness-aware minimization formulation and the gradient-direction perturbation approximation that SAFE's flatness mechanism builds on.","marker":"Foret et al. (2021)"},{"why":"Supplies the augmented Lagrangian and ADMM machinery used for variable splitting, dual ascent, and the convergence framework.","marker":"Boyd et al. (2011)"},{"why":"Provides the Wanda activation-based saliency that SAFE+ uses as its projection matrix $P$, and a key baseline in the LLM perplexity experiments.","marker":"Sun et al. (2024)"},{"why":"Provides SparseGPT, a central one-shot LLM pruning baseline and the layer-wise reconstruction-error setup that SAFE adapts for language models.","marker":"Frantar & Alistarh (2023)"},{"why":"Provides ALPS, an ADMM-based LLM pruning baseline that SAFE outperforms and whose runtime SAFE is compared against.","marker":"Meng et al. (2024)"},{"why":"Provides CrAM and CrAM+, the closest SAM-inspired pruning baselines, used to argue that SAFE's gains come from its core optimization rather than auxiliary gradient tricks.","marker":"Peste et al. (2022)"},{"why":"Provides the Optimal Brain Damage diagonal-Hessian saliency used as one example of the generalized projection in SAFE+.","marker":"LeCun et al. (1989)"},{"why":"Provides the SNIP gradient-sensitivity saliency used as another example of the generalized projection in SAFE+.","marker":"Lee et al. (2019)"},{"why":"Provides the ADMM pruning baseline used throughout the image-classification and noise-robustness comparisons.","marker":"Zhang et al. (2018)"},{"why":"Supplies the ADMM convergence result for nonconvex nonsmooth objectives that the paper adapts to prove SAFE reaches a stationary point.","marker":"Huang et al. (2021)"}],"fun_headline_variants":["Flat-minima pruning tops baselines at 99% sparsity","SAFE finds sparse flat minima, beats SparseGPT on LLaMA","Pruning that keeps flatness yields robust sparse networks","Joint sparsity-flatness pruning outperforms noisy baselines","Dual-projection pruning keeps sparse nets accurate past 99%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method bets that stepping toward the gradient by radius $\\rho$ is a faithful proxy for a minimum's sharpness, so minimizing the loss at that perturbed point actually yields flat minima; if the loss landscape curves strongly within that ball, the flatness signal is wrong.","fun_headline_variants_meta":{"raw":{"variants":["Flat-minima pruning tops baselines at 99% sparsity","SAFE finds sparse flat minima, beats SparseGPT on LLaMA","Pruning that keeps flatness yields robust sparse networks","Joint sparsity-flatness pruning outperforms noisy baselines","Dual-projection pruning keeps sparse nets accurate past 99%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000537,"raw_usage":{"total_tokens":2571,"prompt_tokens":930,"completion_tokens":1641,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":1552}},"tokens_in":546,"tokens_out":1641,"duration_ms":13415,"temperature":1.0,"reasoning_tokens":1552,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:48:04.716733+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the largest Hessian eigenvalue of SAFE-pruned and ADMM-pruned networks at matched sparsity and accuracy; if SAFE's subnetworks are not consistently flatter, the flatness mechanism is not what carries the gains. Alternatively, construct a loss landscape where the SAM perturbation direction does not align with the sharp direction and check whether SAFE still beats magnitude pruning, which would show the gains come from something other than flatness.","supporting_citations":[{"cited_title":"Sharpness-aware minimization for efficiently improving generalization","cited_arxiv_id":null,"evidence_quote":"Supplies the sharpness-aware minimization formulation and the gradient-direction perturbation approximation that SAFE's flatness mechanism builds on."},{"cited_title":"Distributed optimization and statistical learning via the alternating direction method of multipliers","cited_arxiv_id":null,"evidence_quote":"Supplies the augmented Lagrangian and ADMM machinery used for variable splitting, dual ascent, and the convergence framework."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Wanda activation-based saliency that SAFE+ uses as its projection matrix $P$, and a key baseline in the LLM perplexity experiments."},{"cited_title":"and Alistarh, D","cited_arxiv_id":null,"evidence_quote":"Provides SparseGPT, a central one-shot LLM pruning baseline and the layer-wise reconstruction-error setup that SAFE adapts for language models."},{"cited_title":"Alps: Improved optimization for highly sparse one-shot pruning for large language models","cited_arxiv_id":null,"evidence_quote":"Provides ALPS, an ADMM-based LLM pruning baseline that SAFE outperforms and whose runtime SAFE is compared against."},{"cited_title":"H., and Alistarh, D","cited_arxiv_id":null,"evidence_quote":"Provides CrAM and CrAM+, the closest SAM-inspired pruning baselines, used to argue that SAFE's gains come from its core optimization rather than auxiliary gradient tricks."},{"cited_title":"Optimal brain damage","cited_arxiv_id":null,"evidence_quote":"Provides the Optimal Brain Damage diagonal-Hessian saliency used as one example of the generalized projection in SAFE+."},{"cited_title":"Snip: Single-shot network pruning based on connection sensitivity","cited_arxiv_id":null,"evidence_quote":"Provides the SNIP gradient-sensitivity saliency used as another example of the generalized projection in SAFE+."},{"cited_title":"A systematic dnn weight pruning framework using alternating direction method of multipliers","cited_arxiv_id":null,"evidence_quote":"Provides the ADMM pruning baseline used throughout the image-classification and noise-robustness comparisons."},{"cited_title":"Alternating direction method of multipliers for quantization","cited_arxiv_id":null,"evidence_quote":"Supplies the ADMM convergence result for nonconvex nonsmooth objectives that the paper adapts to prove SAFE reaches a stationary point."}],"review_version":1}