{"id":"5e492165-e8f7-4e24-b776-6113ba3bb36c","arxiv_id":"2412.15127","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SAAP scores LLM structures with a weighted fusion of two importance measures, prunes the most volatile units, and recovers performance with grouped quantized low-rank fine-tuning.","lead":"A team of researchers proposes SAAP, a structured pruning method that removes the most unstable parts of a large language model and then fine-tunes the remaining weights in small quantized groups. The paper reports accuracy gains over several pruning baselines across LLaMA, Vicuna, and LLaMA2/3 models, plus faster token generation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own ablation contradicts its central pruning criterion: removing the adaptive stability indicator improves WikiText2 perplexity, so the advertised selection mechanism is unsupported.","rationale":"The reader's verdict is REJECT, and I agree that rejection is warranted. My concern is essentially the one the reader placed in weakest_assumption: there is no independent support for the volatility-equals-redundancy assumption, and the paper's own Table XI speaks against it. I prefer to anchor the rejection on this direct internal contradiction rather than on the undefined λ weights in Eq. (9) or the group-wise quantization formula, because those could in principle be repaired with clarifications without changing the empirical comparison. An ablation showing the proposed component hurting performance is harder to repair: it undercuts the causal mechanism that makes SAAP's results interpretable and suggests the headline gains over baselines may come from other design choices, such as preserving early and late layers or the fine-tuning scheme. The concrete check is a faithful rerun of the ablation. If the table is simply mislabeled, that is still a serious reporting error that must be corrected before the paper can be accepted; if the table is accurate as printed, the central claim is directly falsified by the paper's own data. Either way, the manuscript as written is not acceptable.","tokens_in":17668,"tokens_out":4262,"duration_ms":38359,"concrete_test":"Re-run Table XI on LLaMA-7B with the same 50 Bookcorpus calibration samples, 20% and 50% pruning ratios, and the same Alpaca fine-tuning. Compare SAAP against a control that uses the same fused importance scores but skips Eqs. (10)-(11), e.g., prunes the lowest-importance groups instead of the most volatile ones. If the no-ASI control again yields lower WikiText2 perplexity, the central claim fails. If the original table has swapped labels, the corrected ablation must be published and the comparison rerun.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption behind SAAP is that the adaptive stability indicator (ASI, Eqs. (10)-(11)) identifies redundant structures worth pruning. Table XI is the ablation designed to test this. It reports WikiText2 perplexity of 15.82 vs 16.35 at 20% pruning and 31.73 vs 32.41 at 50% pruning for 'Without ASI' vs 'SAAP.' Lower is better, so removing ASI improves the pruned model in both settings. The surrounding text claims ASI 'can greatly improve the accuracy of the model,' directly contradicting the table. Moreover, the 'Without ASI' column in Table XI exactly equals the 'Separate Cal' column of Table X, while the 'SAAP' entries equal Table X's 'Weight Fusion' entries, suggesting a possible label swap or copy error. If the table is read literally, the central novelty of the paper—pruning the most volatile structures—is not just unproven but contradicted by its own data, and any headline gains over baselines would have to be attributed to the fixed early/late layers or the fine-tuning scheme rather than to ASI. This is load-bearing because ASI is the paper's distinguishing contribution and the basis for the 'structural importance awareness' in the title.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SAAP, a structured pruning method for LLMs. It computes coarse- and fine-grained importance scores, fuses them through an uncertainty-weighted sum, defines an adaptive stability indicator based on the relative volatility of importance scores across calibration samples, prunes the most volatile structures while keeping fixed early/late layers, and recovers performance with a grouped quantized low-rank fine-tuning scheme. Experiments on LLaMA, Vicuna, LLaMA2, and LLaMA3 at 20% and 50% pruning ratios report accuracy gains over LLM-pruner, LoraPrune, Wanda, and LoRAShear on zero-shot classification and text generation, together with lower perplexity and higher token throughput.","tokens_in":17960,"tokens_out":4862,"duration_ms":37500,"significance":"If the claimed results hold, SAAP would offer a practical structured-pruning recipe with meaningful speed and memory gains, and the paper's breadth of model and dataset coverage is a strength. The paper also includes explicit ablations of its three components and reports memory/throughput statistics, which are useful. However, the central novelty—the adaptive stability indicator—is contradicted by the paper's own ablation, and key parameters and baseline results are missing; in its current form the paper does not establish the headline claims.","major_comments":[{"comment":"The ablation of the adaptive stability indicator (ASI) contradicts the paper's central claim. Table XI reports WikiText2 perplexity of 15.82 at 20% pruning and 31.73 at 50% pruning 'Without ASI', versus 16.35 and 32.41 for SAAP. Since lower perplexity is better, removing ASI improves the pruned model, yet the text states that the volatility-based metric 'can greatly improve the accuracy of the model'. Moreover, the 'Without ASI' numbers equal the 'Separate Cal' entries and the 'SAAP' numbers equal the 'Weight Fusion' entries of Table X, which suggests a label swap or copying error. Because ASI is the paper's distinguishing contribution and the basis for 'structural importance awareness', this internal contradiction is load-bearing and must be resolved before the empirical claims can be assessed.","section":"Section V-D2, Table XI"},{"comment":"The adaptive importance fusion metric in Eq. (9) is a weighted combination of the coarse- and fine-grained importance scores with coefficients 1/(2λ1^2) and 1/(2λ2^2) plus log λ1λ2. The paper never specifies how λ1 and λ2 are chosen or optimized, nor reports their values in any experiment. If the λ's are fit on the same 50 Bookcorpus calibration samples used for pruning selection, then part of the selection criterion is calibrated on the pruning data; if they are manually chosen, they are unreported hyperparameters. Either way, the evaluation is incomplete.","section":"Section IV-B, Eq. (9)"},{"comment":"Equation (14), W_l = a_g(W_g − b_g), is inconsistent with the definitions in Eq. (13), where a is a scale and b = min(W) is a zero-point offset. With those definitions, dequantization should be W_l = a_g W_g + b_g (or equivalently W_l = a_g(W_g + b_g) under a different convention); subtracting b_g as written would not place the dequantized values in the original range. This suggests a sign error in the core grouped fine-tuning formula and undermines the description of the recovery stage.","section":"Section IV-C, Eq. (14)"},{"comment":"The paper claims in the abstract and text that SAAP outperforms LoRAShear, but no LoRAShear results appear anywhere in the comparison tables: Table I lists only '-' for every LoRAShear row at both pruning ratios, and Tables IV–VI and the generalization tables do not include LoRAShear at all. Without LoRAShear measurements on any model, the claimed advantage over this baseline is unsupported.","section":"Section V-B, Tables I and V"}],"minor_comments":[{"comment":"The symbol D is used both for the number of calibration samples in the averaging in Eq. (10) and for the dataset in Eq. (2), which is confusing; use different symbols.","section":"Section IV-B, Eq. (10)"},{"comment":"Reference [36] is cited for Bessel correction but is a control-theory paper and does not seem to be the intended source; the statistical bias correction should be referenced to an appropriate statistics text or derived.","section":"References"},{"comment":"The zero-point offset is defined as b = min(W), but the scale is defined using the full range; for asymmetric quantization the offset is usually round(min(W)/a) or min(W), and the formula should state the rounding convention.","section":"Section IV-C, Eq. (13)"},{"comment":"The caption 'LLM's answer' is grammatically awkward; consider 'LLM responses at different pruning ratios.'","section":"Fig. 4"},{"comment":"The sentence 'the performance of the proposed method is further validated on the WikiText2 test set' appears in the discussion of Table II, but no test-set split is specified; clarify whether WikiText2 refers to the validation or test split.","section":"Section V-A"}],"recommendation":"reject","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First the punchline: this is a genuinely broad empirical study of a plausible pruning recipe—uncertainty-weighted fusion of vector- and element-wise importance, variance-based stability pruning, and grouped int4 low-rank fine-tuning. The experiments cover LLaMA, Vicuna, LLaMA2, and LLaMA3 at 20% and 50% pruning, with four baselines. Table X supports the adaptive fusion over separate or simple weighted fusion, and Table XII shows real speed and memory gains from the grouped fine-tuning. The paper is clearly written and the contribution is incremental but not trivial.\n\nThe soft spots are load-bearing. First, the fusion weights λ1 and λ2 in Eq. (9) are never specified. The text calls them homoscedastic noise but gives no estimation procedure or reported values, so the importance score as defined is not reproducible and may have been tuned on calibration data. Second, Eq. (14) has a sign error in the quantization reconstruction: with b = min(W), the dequantized weight should be a·q + b, not a·(q − b). As written, the core fine-tuning formula is wrong. Third, Table XI is the most serious issue. Its 'Without ASI' column exactly matches Table X's 'Separate Cal' (15.82/31.73), and its 'SAAP' column exactly matches 'Weight Fusion' (16.35/32.41). The text claims ASI 'greatly improves' accuracy, but the table as printed shows ASI hurting perplexity. Either the table is mislabeled or the method's central novelty is contradicted by its own data. A reviewer cannot tell which. I also note LoRAShear results are missing for LLaMA-7B, and no code or data is released.\n\nThe paper deserves a serious referee because the problem is relevant and the experiments are broad, but as written it should be rejected. The authors need to report λ1 and λ2, fix Eq. (14), correct or explain Table XI, and release code. If those are fixed, the method is a plausible incremental contribution to LLM pruning. I would not cite it in its current form.","headline":"Broad experiments and a plausible fusion idea, but unreported fusion weights, a likely quantization sign error, and a self-contradictory ablation make the method irreproducible as written.","tokens_in":18482,"tokens_out":4754,"would_cite":false,"duration_ms":40254,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that the safest structures to prune from a large language model are the ones whose importance scores vary most across calibration inputs, and that pruning those with grouped int4 fine-tuning beats four existing pruning…","keywords":["large language model","structured pruning","importance fluctuation","adaptive importance fusion","stability indicator","group-wise fine-tuning","low-rank adaptation","quantization"],"falsifier":"A controlled rerun that compares SAAP's volatility-based selection with random selection at the same pruning ratios and with the same grouped fine-tuning would settle whether fluctuation identifies redundancy: if random pruning matches or beats SAAP on the held-out zero-shot tasks, the volatility criterion is not what carries the gain. The paper's own ablation tables (Tables X and XI) report contradictory WikiText2 perplexities for the SAAP and without-stability-indicator conditions (16.35 vs 15.82 at 20% and 32.41 vs 31.73 at 50%), so such a rerun using the released code is the direct way to resolve it.","tokens_in":17501,"feed_emoji":"✂️","tokens_out":13597,"duration_ms":82565,"temperature":0.7,"pith_summary":"The paper proposes that a large language model can be pruned safely by removing structures whose importance scores are most volatile across a small calibration set, rather than structures with the lowest average importance. On this principle it builds SAAP, a structured-pruning pipeline that fuses coarse-grained and fine-grained importance into one adaptive score, standardizes the score's fluctuation across layers, and prunes the most fluctuating coupled structures. A grouped int4 fine-tuning step then recovers accuracy while reducing memory and increasing token-generation speed. The paper reports that SAAP beats four existing pruning methods on zero-shot classification and text generation, with accuracy gains of 2.17% on LLaMA-7B, 2.37% on Vicuna-7B, and 2.39% on LLaMA-13B, plus a 5% token-generation speed improvement. If the volatility-equals-redundancy premise holds, pruning decisions become cheaper because they can be made from only fifty random calibration samples rather than from retraining.","feed_headline":"Prune LLMs by cutting volatile structures","feed_subtitle":"SAAP uses importance-score variance to choose what to prune, then grouped int4 fine-tuning recovers accuracy.","key_machinery":"The load-bearing object is the adaptive stability indicator $\\hat{M}_{l,j}$ of Eq. (11), built from the importance fluctuation $M_{l,j}$ of Eq. (10), which measures how much the adaptively fused importance score $I^{\\mathrm{ada}}_i$ for channel $j$ in layer $l$ varies across the calibration samples relative to the weight norm. The adaptive importance score itself fuses vector-wise and element-wise importance from LLM-pruner using a homoscedastic Gaussian likelihood, so the two importance views are weighted by their observation noise parameters $\\lambda_1$ and $\\lambda_2$ rather than by a fixed linear combination. The third mechanism is grouped int4 fine-tuning, which partitions each column of the weight matrix into $L$ groups, quantizes each group with its own scale $a$ and offset $b$, and adds a low-rank update $s\\cdot AB$, cutting the number of fine-tuning parameters from $D_{\\mathrm{in}}D_{\\mathrm{int}} + D_{\\mathrm{int}}D_{\\mathrm{out}}$ to $LD_{\\mathrm{int}} + D_{\\mathrm{int}}D_{\\mathrm{out}}$.","core_discovery":"The central claim is that in a transformer-based LLM, the coupled structures that are redundant are the ones whose importance scores fluctuate most across different calibration inputs. SAAP therefore computes an adaptive importance score by fusing vector-wise and element-wise importance under a homoscedastic Gaussian likelihood, then builds a stability indicator that standardizes each structure's importance variance across calibration samples, and removes the structures with the highest relative volatility. After pruning, it quantizes each column group of the weight matrix to int4 and fine-tunes low-rank adapters per group, so the precision and memory benefits of quantization combine with LoRA-style recovery. The paper argues that this pipeline outperforms LLM-pruner, LoraPrune, Wanda, and LoRAShear, with average accuracy gains of 2.17% on LLaMA-7B, 2.37% on Vicuna-7B, and 2.39% on LLaMA-13B, and a 5% token-generation speed improvement over the baselines.","pith_inferences":["If volatility of importance scores is a stable proxy for redundancy, then any cheap importance estimator could be made uncertainty-aware; a natural extension is to compute the coefficient of variation of existing scores such as weight-times-activation and prune by that, giving methods without SAAP's fusion the same cross-layer ranking.","The paper normalizes fluctuation by the weight norm and then standardizes across structures; an untested question is whether rank-based or unnormalized variance changes the pruning order, which would separate the role of scaling from the role of fluctuation.","The abstract's 5% speed improvement is smaller than the 32-65% speedups reported in the hardware table, so the 5% figure likely refers to the gain over the best baseline after fine-tuning rather than over the dense model; a direct head-to-head benchmark would disambiguate.","Since the calibration set is only fifty samples, the method's success depends on those samples representing the operating distribution; a testable extension is to draw calibration samples from different domains and measure how the pruning order and downstream accuracy change under distribution shift."],"forward_implications":["SAAP turns layer-wise pruning ratios into an emergent property of the model: one standardized fluctuation score ranks all coupled structures across layers, so the pruning decision no longer needs a uniform ratio chosen by hand.","Because the importance estimate uses only fifty Bookcorpus calibration samples, the estimation stage is cheap enough to run at 33B and 65B scale; the paper reports SAAP beating LLM-pruner on WikiText2 perplexity at both 20% and 50% pruning ratios.","Grouped int4 fine-tuning makes the pruned model deployable on a single consumer GPU, with the paper's measurements showing token-generation speed rising by about 65% at a 50% pruning ratio for LLaMA-7B.","The volatility criterion appears to transfer across model families: the paper reports gains on LLaMA-7B, Vicuna-7B, and LLaMA-13B, and also runs LLaMA2 and LLaMA3 variants, suggesting the method is not tied to one architecture."],"supporting_citations":[{"why":"It supplies the vector-wise and element-wise importance calculations that SAAP adaptively fuses, and it defines the structurally coupled groups.","marker":"[18]"},{"why":"It provides the LLaMA foundation models used for the main accuracy and speed claims.","marker":"[19]"},{"why":"It provides the Vicuna model used for one of the claimed accuracy gains and for the generality tests.","marker":"[20]"},{"why":"It serves as a baseline whose weight-times-activation importance metric is compared against in the main tables.","marker":"[22]"},{"why":"It serves as a baseline that uses LoRA with half-space projected gradient for progressive pruning.","marker":"[23]"},{"why":"It provides the LoRA low-rank update that the grouped fine-tuning stage extends and compares against.","marker":"[32]"},{"why":"It provides the quantization-aware QLoRA baseline against which SAAP's grouped int4 fine-tuning is compared.","marker":"[33]"},{"why":"It serves as the LoraPrune baseline combining low-rank decomposition with pruning in the comparison tables.","marker":"[47]"},{"why":"It supplies the Bookcorpus text from which the fifty calibration samples for importance estimation are drawn.","marker":"[48]"},{"why":"It supplies the Alpaca fine-tuning data used in the recovery stage after pruning.","marker":"[49]"}],"fun_headline_variants":["Prune LLMs where importance scores fluctuate most","Cut volatile LLM structures with adaptive pruning","Adaptive pruning targets LLM structures that vary","LLM pruning by structural importance volatility","Prune unstable LLM layers, fine-tune to recover"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a structure whose importance score swings most across fifty randomly chosen Bookcorpus text samples is redundant and therefore safe to remove; if volatility does not track redundancy, the pruning selection loses its basis.","fun_headline_variants_meta":{"raw":{"variants":["Prune LLMs where importance scores fluctuate most","Cut volatile LLM structures with adaptive pruning","Adaptive pruning targets LLM structures that vary","LLM pruning by structural importance volatility","Prune unstable LLM layers, fine-tune to recover"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000343,"raw_usage":{"total_tokens":1903,"prompt_tokens":981,"completion_tokens":922,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":851}},"tokens_in":597,"tokens_out":922,"duration_ms":8881,"temperature":1.0,"reasoning_tokens":851,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:35:40.025921+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled rerun that compares SAAP's volatility-based selection with random selection at the same pruning ratios and with the same grouped fine-tuning would settle whether fluctuation identifies redundancy: if random pruning matches or beats SAAP on the held-out zero-shot tasks, the volatility criterion is not what carries the gain. The paper's own ablation tables (Tables X and XI) report contradictory WikiText2 perplexities for the SAAP and without-stability-indicator conditions (16.35 vs 15.82 at 20% and 32.41 vs 31.73 at 50%), so such a rerun using the released code is the direct way to resolve it.","supporting_citations":[{"cited_title":"Llm-pruner: On the structural pruning of large language models,","cited_arxiv_id":null,"evidence_quote":"It supplies the vector-wise and element-wise importance calculations that SAAP adaptively fuses, and it defines the structurally coupled groups."},{"cited_title":"Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality","cited_arxiv_id":null,"evidence_quote":"It provides the Vicuna model used for one of the claimed accuracy gains and for the generality tests."},{"cited_title":"Qlora: Efficient finetuning of quantized llms","cited_arxiv_id":null,"evidence_quote":"It provides the quantization-aware QLoRA baseline against which SAAP's grouped int4 fine-tuning is compared."}],"review_version":1}