{"id":"bccf2013-0df6-4681-a25f-7f732eeed5b9","arxiv_id":"2501.09949","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Pruning residual blocks, then MLP channels, then attention heads in sequence improves zero-shot accuracy of compressed LLMs compared with block-only pruning.","lead":"MultiPruner is a training-free pruning method that removes Transformer blocks, MLP channels, and attention heads one after another. It reports higher zero-shot accuracy than BlockPruner and other training-free baselines at similar compression ratios across several Llama, Qwen, and Baichuan models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline Llama2-7B result is partly in-sample: the 44:52:4 ratio split and pruning order were tuned on that model, and no error bars are reported, so the cross-model superiority claim is not yet statistically supported.","rationale":"The paper presents a coherent extension of BlockPruner, with code released, ablations, and results across ten models. The reader's CONDITIONAL verdict already captures the main uncertainties, and I do not see a load-bearing logical error that would warrant rejection. However, the strongest concern is more pointed than the reader states: the Llama2-7B result used to motivate the method is partly in-sample, because the ratio weights, pruning order, and head-pruning allocation were selected using that model and then the same model is used as the flagship comparison. The absence of error bars or repeated calibration samples means the margins over BlockPruner (often 2-4 points of average score) cannot be separated from selection noise. The transfer to other models is suggestive but not statistically confirmed, especially because Table 10 admits the configuration changes for GQA models and the Baichuan2-13B row goes against the method. A cross-validation of the hyperparameters and a seeded reproducibility run would settle whether the claim is robust. Since these are verification gaps rather than demonstrated failures, the appropriate verdict remains CONDITIONAL, i.e., unchanged from the reader's verdict.","tokens_in":16430,"tokens_out":8897,"duration_ms":91832,"concrete_test":"Run a cross-validation of the recipe hyperparameters. Tune the Block:MLP:Attention weights and pruning order on Llama3.1-8B at a 22% target ratio, then apply that configuration to Llama2-7B, and apply the Llama2-tuned configuration to Llama3.1-8B; compare the resulting average scores with the paper's reported configuration. Also rerun the full pipeline on Llama2-7B with five different 256-sample calibration subsets and five seeds, reporting mean plus/minus standard deviation for Wikitext2 PPL and downstream average score. If the transferred configuration is worse than the tuned one by more than the BlockPruner margin, or if the 95% confidence interval overlaps BlockPruner's point estimate, the central cross-model claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is that the flagship comparison is not an out-of-sample test. The Block:MLP:Attention ratio split (44:52:4), the coarse-to-fine pruning order, and the decision to allocate only 4% of the target ratio to attention heads were selected on Llama2-7B at a 22% target ratio (Section 3.5, Section 3.6, Figure 4, Table 10). That same Llama2-7B row is then presented in Table 1 as the primary evidence of superiority. In the absence of any seed or calibration-subset variation, the 2.66-point average-score gap over BlockPruner on that row could be a selection artifact. For other models, the split is transferred without re-validation, and Table 10 explicitly changes it for GQA models, so the paper's 'same configuration works across models' claim is not uniform. The Baichuan2-13B row (BlockPruner 59.47 vs MultiPruner 59.12) further shows the advantage is not universal. If the 44:52:4 weights are not near-optimal for a given model or target ratio, the reported gains are a property of the chosen hyperparameters, not of the method itself.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MultiPruner, a training-free structured pruning method for large language models that sequentially removes residual blocks (depth), MLP channel groups (width), and attention heads (width), with a weight-reordering step before width pruning. The method is evaluated on 10 LLMs (Llama2, Llama3, Qwen, Baichuan) at pruning ratios around 10-25%, comparing against BlockPruner, ShortGPT, SliceGPT, LaCo, and Relative Magnitude on Wikitext2 perplexity and five zero-shot tasks. The authors report that MultiPruner achieves the best perplexity and average score on 9 of 10 models, including a reproduced BlockPruner baseline, and provide ablations on pruning order, dimension existence, ratio sensitivity, weight reordering, inference speedups, and a LoRA-based recovery stage.","tokens_in":16678,"tokens_out":5371,"duration_ms":52943,"significance":"If the results hold, MultiPruner is a useful contribution to training-free structured pruning: it shows that combining depth and width pruning can beat depth-only block pruning at similar parameter ratios, and it provides a practical recipe (sequential coarse-to-fine, weight reordering, ratio weights). The paper's strengths include a reproduced BlockPruner baseline, broad model coverage, and explicit ablations. However, the central claim of cross-model superiority is currently weakened by the fact that the headline hyperparameters were selected on the same Llama2-7B model used as the main evidence, and no uncertainty estimates are provided.","major_comments":[{"comment":"The Block:MLP:Attention ratio weights (44:52:4) and the coarse-to-fine pruning order are selected via ablations on Llama2-7B at a 22% target ratio (Figure 4, Table 4), and the same Llama2-7B row is then presented in Table 1 as the primary evidence of superiority over BlockPruner (average score 62.83 vs 60.17). This is an in-sample selection for the flagship result. To support the cross-model claim, the authors should provide out-of-sample validation, e.g., tune the configuration on a different model (or use nested selection on a holdout model) and then report results on Llama2-7B, or alternatively report results over multiple calibration subsets to show the 2.66-point gap is not a selection artifact.","section":"Section 3.5/3.6, Figure 4, Table 1"},{"comment":"No error bars, standard deviations, or multiple runs are reported for any of the zero-shot accuracy or perplexity numbers. Since evaluation is on finite benchmark sets and the calibration subset is randomly drawn from Alpaca, differences of 1-2 points (e.g., PIQA 74.65 vs 74.21 in Table 1) may be within random variation. The authors should report variance across at least a few calibration subsets or seeds, or explicitly justify why the reported numbers are deterministic.","section":"Tables 1, 2, 3, 9"},{"comment":"For Baichuan2-13B, MultiPruner achieves lower Wikitext2 perplexity (10.99 vs 15.36) but a lower average zero-shot score (59.12 vs 59.47) than BlockPruner. The abstract and Section 3.2 state that MultiPruner 'surpasses' or 'outperforms' other training-free pruning methods without qualification. The authors should explicitly acknowledge this exception and discuss its implications for the generality of the method, rather than relying on the word 'most' in a single sentence.","section":"Table 1, Baichuan2-13B row"},{"comment":"The paper states that for GQA models MultiPruner 'generally avoids pruning attention heads' and that ratio weights 'may vary across different LLMs.' This means the method is not a fixed configuration but requires model-specific hyperparameter choices. The manuscript should provide a principled procedure for selecting these weights on a new model (e.g., based on model architecture or a small validation set), or clearly disclose the per-model tuning cost. Otherwise the claim of a 'training-free' method that works 'across various large pre-trained models' is overstated.","section":"Table 10 and Section 3.1"}],"minor_comments":[{"comment":"The thresholds τ1, τ2, and τ3 are cumulative (as clarified in Table 10), but the pseudocode uses a single τ that is overwritten. Adding a comment that τt are cumulative pruning-ratio thresholds would prevent confusion.","section":"Algorithm 1"},{"comment":"The sentence 'MultiPruner removes this assumption and demonstrate that' contains a subject-verb agreement error; it should be 'demonstrates'.","section":"Section 1"},{"comment":"The 'Performance-Recovery Stage' is only one sentence long; since Table 5 reports a LoRA-based recovery result, this section should be expanded to describe the setup (e.g., which layers are adapted, training budget) or moved to the experimental section.","section":"Section 2.2"},{"comment":"The evolutionary search variant (MultiPruner-Evol) is introduced, but the search space S and the number of evaluations N are not specified in the main text or Appendix D. Adding these details would improve reproducibility.","section":"Section 3.4 and Algorithm 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is generally well-written and the code is promised; the main concern is selection bias in the flagship result. I recommend asking the authors to provide out-of-sample validation and uncertainty estimates. The paper fits the journal scope (cs.LG); no issues with citation or novelty in my assessment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper extends BlockPruner by adding MLP channel and attention-head pruning after block pruning, and it works: across ten LLMs the pruned models consistently beat block-only pruning by 2-4 points average zero-shot accuracy at comparable ratios, with lower Wikitext2 PPL. The evaluation is external to the calibration objective (Alpaca PPL is used to choose what to prune; Wikitext2 and five downstream tasks are reported), so there is no load-bearing circularity. The ablation in Table 4 supports the coarse-to-fine order, and Figure 4 shows the sensitivity to ratio weights; the appendix gives full hyperparameters and a 1%-21% ratio sweep. The authors also reproduced the BlockPruner baseline rather than copying numbers, which is more than many pruning papers do. The limitation statement is honest.\n\nThe soft spots are real but not fatal. The 44:52:4 Block:MLP:Attention split and the pruning order were tuned on Llama2-7B at 22%, and that exact row is then the headline evidence. With no seeds or calibration-subset variation reported, the 2.66-point gap there could be partly selection artifact. The Baichuan2-13B row goes the other way (BlockPruner 59.47 vs MultiPruner 59.12), and the paper's own Table 10 notes that GQA models generally avoid attention-head pruning, so \"same configuration works across models\" is not literally true. The abstract and conclusion also overclaim slightly. None of this sinks the central claim: the gains are consistent in 9 of 10 models, the method is plausible, and the ablations give a coherent story.\n\nIf I were refereeing, I would ask for variance reporting (multiple calibration subsets or seeds), a held-out hyperparameter check (e.g., fix the ratio split on one model and show it transfers to others without re-tuning), and a softened claim that acknowledges the GQA exception. I would not desk-reject this; it is a useful empirical contribution for practitioners who want a training-free structured pruning recipe with reproducible settings. I would cite it if I worked on LLM pruning, and I would bring it to a reading group focused on what kinds of hyperparameter tuning actually transfer across models.","headline":"A solid, incremental pruning recipe that mostly delivers what it claims; the main caveat is that the headline result is partly in-sample, but the paper deserves serious refereeing.","tokens_in":711,"tokens_out":944,"would_cite":true,"duration_ms":29722,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MultiPruner prunes LLMs on three axes and reports better zero-shot accuracy than block-only pruning at comparable compression.","keywords":["structured pruning","training-free compression","large language models","residual block pruning","MLP channel pruning","attention head pruning","perplexity importance","zero-shot evaluation"],"falsifier":"Run MultiPruner at 22% on Llama2-7B but replace the Alpaca calibration set with WikiText-2, a code corpus, or a task-mixed set for the importance scores, keeping everything else fixed; if the chosen structures differ from the paper's reported pruned model and the average zero-shot score drops by more than a few points relative to 62.83, the transferability of the calibration signal is undermined. Alternatively, search the Block:MLP:Attention ratio weights on Llama3.1-8B from scratch and compare the resulting model against the fixed 44:52:4 recipe at the same total ratio; the paper's cross-model generality stands or falls on that comparison.","tokens_in":16187,"feed_emoji":"✂️","tokens_out":8271,"duration_ms":72508,"temperature":0.7,"pith_summary":"MultiPruner is a training-free structured pruning recipe for large language models. It tries to establish that pruning a Transformer along three dimensions—residual blocks, MLP channels, and attention heads—produces smaller models that retain more zero-shot accuracy than pruning any single dimension, and that this multidimensional balance is best achieved by pruning coarse structures first and fine ones later. If the paper is right, a standard 22% compression of Llama2-7B can actually improve average downstream accuracy by 2.66 points over BlockPruner at the same compression, with lower perplexity. The care a reader should take: the recipe's gains rest on using a text-likelihood score on a small calibration set as a universal importance signal.","feed_headline":"Three-way LLM pruning beats block-only at same compression","feed_subtitle":"MultiPruner trims Llama2-7B by 22% and lifts average task score from 60.17 to 62.83.","key_machinery":"The load-bearing object is the minimal residual block—either the self-attention or the MLP sub-block within a Transformer layer—treated as the atomic unit for depth pruning, plus a weight-reordering step that sorts channel groups and heads by L1 norm so the width-pruning stages can remove the least important slices in groups. Importance is measured by the model's perplexity on 128 or 256 Alpaca calibration samples: each candidate removal is scored by the perplexity of the masked model, and the candidate with the smallest rise is removed. A fixed allocation of the total pruning ratio, Block:MLP Channel:Attention Head = 44:52:4, was discovered on Llama2-7B and then applied across models, based on sensitivity studies showing that attention heads degrade quickly beyond a small pruning budget while MLP channels tolerate much heavier pruning.","core_discovery":"The paper claims that block-level pruning leaves redundancy on the table, and that reallocating a fixed pruning budget across depth and width yields a strictly better compressed model. Concretely, MultiPruner removes the least important minimal residual blocks (self-attention or MLP) one at a time until block pruning accounts for 44% of the target ratio, reorders weights by L1 norm so dispensable channels move to the end, then prunes MLP channel groups and attention heads until the remaining 56% of the budget is spent, with attention heads capped at about 4%. On Llama2-7B at 21.96% total pruning, this produces WikiText-2 perplexity 9.33 and an average score of 62.83 across five zero-shot tasks, against BlockPruner's 11.51 and 60.17 at 21.99%. The same pattern is reported for Llama2-13B, Llama3, Qwen, and Baichuan models, and an evolutionary-search variant occasionally trades a little perplexity for a higher average score.","pith_inferences":["Because the 44:52:4 split was tuned on Llama2-7B and the appendix notes that GQA models may need fewer removed attention heads, the recipe likely needs per-architecture recalibration; a cheap test is to grid-search only the MLP versus block split on a GQA model and see whether the Pareto front matches the paper's ratio.","The paper's ablations suggest most of the accuracy gain may come from MLP channel pruning rather than the presence of three dimensions; isolating that by comparing block-plus-MLP against block-plus-head at equal total ratio would sharpen the causal story.","Since perplexity and downstream accuracy do not always move together, calibration perplexity is a proxy rather than the objective; a testable extension is to run the same pipeline with an importance metric tuned to downstream task loss and compare the selected structures.","The sequential coarse-to-fine ordering hints that block importance and channel importance are not independent; a joint search that re-evaluates channel importance after each block removal might dominate the fixed-order recipe at higher compression ratios."],"forward_implications":["At a fixed total pruning ratio, pruning blocks, MLP channel groups, and attention heads in that coarse-to-fine order yields lower perplexity and higher average zero-shot accuracy than block-only pruning on the Llama2, Llama3, Qwen, and Baichuan models tested.","Removing any of the three pruning stages degrades results; attention-head pruning alone is catastrophic, while block-plus-MLP-channel pruning without heads still outperforms block-only pruning on Llama2-7B.","The sensitivity analysis implies an allocation rule: keep attention-head pruning near 4% of the total ratio, give MLP channels roughly half the budget, and let block pruning take the remainder.","Weight reordering by L1 norm before width pruning is a necessary step; skipping it raises perplexity from 9.33 to 10.78 on Llama2-7B at 22%.","A 22%-pruned Llama2-7B can be deployed with roughly 1.3x prefill and decode speedups on a CPU, and two epochs of LoRA recovery tuning lift its average score from 62.83 to 64.18."],"supporting_citations":[{"why":"Introduces BlockPruner, the single-dimension block pruning method MultiPruner extends, and supplies the minimal-residual-block formulation plus the calibration and evaluation setup.","marker":"Zhong et al. (2024)"},{"why":"ShortGPT, the baseline showing residual-block redundancy that motivates MultiPruner and serves as a main comparison.","marker":"Men et al. (2024)"},{"why":"SliceGPT, a training-free structural pruning baseline compared in the main results.","marker":"Ashkboos et al. (2024)"},{"why":"LaCo, a layer-collapse pruning baseline compared in the main results.","marker":"Yang et al. (2024b)"},{"why":"Relative Magnitude, a baseline pruning method whose results are compared at matching ratios.","marker":"Samragh et al. (2023)"},{"why":"Provides the Llama-2 models used for the headline results and for discovery of the ratio weights.","marker":"Touvron et al. (2023)"},{"why":"Provides the Llama 3 and Llama 3.1/3.2 models used to test the method on newer architectures.","marker":"Dubey et al. (2024)"},{"why":"Provides the Qwen1.5 models evaluated in the main comparison.","marker":"Bai et al. (2023)"},{"why":"Provides the Qwen2.5-7B model evaluated in the extended comparison.","marker":"Yang et al. (2024a)"},{"why":"The lm-eval-harness framework used to compute zero-shot accuracy on the five downstream tasks.","marker":"Gao et al. (2023)"}],"fun_headline_variants":["MultiPruner trims blocks, channels, and heads for better LLM compression","MultiPruner lifts Llama2-7B zero-shot score from 60.17 to 62.83","Three-way pruning lifts LLM accuracy at same compression","Balanced block, channel, and head pruning beats block-only","MultiPruner's three-way cut beats block-only pruning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a text-likelihood score measured on 128 to 256 Alpaca samples tells which blocks, MLP channels, and attention heads can be safely removed for unrelated downstream tasks, and that the 44:52:4 split of the pruning budget stays near-optimal across different model families and target compression levels.","fun_headline_variants_meta":{"raw":{"variants":["MultiPruner trims blocks, channels, and heads for better LLM compression","MultiPruner lifts Llama2-7B zero-shot score from 60.17 to 62.83","Three-way pruning lifts LLM accuracy at same compression","Balanced block, channel, and head pruning beats block-only","MultiPruner's three-way cut beats block-only pruning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001353,"raw_usage":{"total_tokens":5501,"prompt_tokens":963,"completion_tokens":4538,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":4440}},"tokens_in":579,"tokens_out":4538,"duration_ms":30729,"temperature":1.0,"reasoning_tokens":4440,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:30:12.514481+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run MultiPruner at 22% on Llama2-7B but replace the Alpaca calibration set with WikiText-2, a code corpus, or a task-mixed set for the importance scores, keeping everything else fixed; if the chosen structures differ from the paper's reported pruned model and the average zero-shot score drops by more than a few points relative to 62.83, the transferability of the calibration signal is undermined. Alternatively, search the Block:MLP:Attention ratio weights on Llama3.1-8B from scratch and compare the resulting model against the fixed 44:52:4 recipe at the same total ratio; the paper's cross-model generality stands or falls on that comparison.","supporting_citations":[],"review_version":1}