{"id":"3f8972cc-ea40-4a93-a115-cf260e208e13","arxiv_id":"2412.06419","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A block-level pruning criterion estimates channel importance from activation magnitudes and weight norms in a single forward pass.","lead":"Researchers propose a new pruning method for large language models that removes whole attention heads and feed-forward channels while trying to preserve each transformer block's output. The method needs only one forward pass and reports better zero-shot accuracy and lower perplexity than two common pruning baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FFN importance score ignores the SwiGLU gate in all evaluated models; the Eq (7) derivation does not match the architecture, so the central pruning metric is not established as accurate.","rationale":"The reader's weakest assumption points at the same architectural mismatch, but overstates it: even for SwiGLU the paper's score still yields an upper bound (up to the SiLU range constant), so it is not literally 'not a valid upper bound.' The real problem is that the score is a loose bound that ignores the gate, so minimizing it is not equivalent to minimizing block-output error, and rankings can be wrong. This is load-bearing because the central claim is that the importance scores accurately measure pruning effect; the theoretical derivation is the only support given, since code is missing and tables cannot be independently checked. The proposed overlap/perplexity experiment would settle whether the gate omission changes practical pruning decisions. The verdict remains CONDITIONAL: the empirical claim may hold, but the theoretical justification needs repair and the test above should be reported.","tokens_in":14000,"tokens_out":12808,"duration_ms":135034,"concrete_test":"On LLaMA-7B, use the paper's calibration setup (128 C4 sequences, length 2048). For each FFN intermediate channel j, compute the paper's score s_paper_j = (mean over tokens of |XU_j|) · ||WD_j||_1 and a gate-aware score s_gate_j = (mean over tokens of |SiLU(X'W_G)_j|·|XU_j|) · ||WD_j||_1. Prune the same number of channels (20% FFN sparsity, no head pruning) with each score and without fine-tuning measure (i) the Jaccard overlap of the two pruned-channel sets and (ii) WikiText2 perplexity of the two pruned models. If the overlap is high and the perplexities differ by less than a standard deviation, the gate omission is not practically load-bearing; if the overlap is low and s_gate produces lower perplexity, the paper's score is miscalibrated and the theoretical story is empirically falsified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The FFN metric is derived from Eq (4), which models the block as σ(X'W_U)W_D, but LLaMA-1/2 and Vicuna use a gated SwiGLU FFN: (SiLU(X'W_G) ⊙ X'W_U)W_D. Eq (7) replaces the actual pruned output with σ(XU)WD and σ(s⊙XU)WD, ignoring X'W_G entirely. The actual block-output difference is (1-s)⊙SiLU(X'W_G)⊙X'W_U WD. One can recover a valid but loose constant-factor bound by bounding each SiLU output by ≈1.1, but the score |XU_j||WD_j| is then only an upper bound, not a measure of the effect of pruning: a channel with large |XU_j| and near-zero gate is ranked important although its contribution is negligible, and with a fixed sparsity budget an actually important channel may be pruned instead. Hence Definition 1's claim that sF_j measures influence on the block output is unsupported for the evaluated models, and the omitted gate also propagates into the attention-head score Eq (8) through the (I+|WU||WD|) term. The central method's theoretical justification is therefore not established; the empirical advantage, if real, would need to be explained by a different mechanism than the stated upper bound.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LLM-BIP, a structured pruning method for LLMs that assigns importance scores to FFN channels and attention heads by measuring their influence on the transformer block output. The scores are derived from an upper bound on the block-level reconstruction error and are computed in a single forward pass without gradients. The method is evaluated on LLaMA-7B, Vicuna-7B, and LLaMA-13B at 20%, 50%, and 70% sparsity, reporting lower perplexity and higher zero-shot accuracy than Wanda, LLM-Pruner, and other baselines, both without fine-tuning and after LoRA fine-tuning.","tokens_in":14340,"tokens_out":4292,"duration_ms":44153,"significance":"If the proposed metric is sound, LLM-BIP would be a practically valuable contribution: it is fast (one forward pass), gradient-free, and shows large gains at high sparsity levels, especially at 50% and 70% where structured pruning of LLMs typically fails. The empirical tables are extensive and the method appears to be self-contained relative to the benchmarks, with no parameter fitted to downstream accuracy. However, the central derivation is built on an un-gated FFN model that does not match the evaluated LLaMA/Vicuna architectures, so the theoretical justification for the pruning score is not currently established. The paper also defers a key derivation to a supplementary file that is not present, making the MSA score in Eq. (8) unverifiable.","major_comments":[{"comment":"The derivation of the FFN importance score models the FFN as sigma(X'W_U)W_D (Eq. 4), but LLaMA-1, LLaMA-2, and Vicuna use the gated SwiGLU FFN (SiLU(X'W_G) ⊙ X'W_U)W_D. Eq. (7) therefore bounds the wrong object: the actual block-output change after pruning channel j is (1-s_j)SiLU(X'W_G)_j(X'W_U)_j W_{D,j}, which depends on the gate activation, and the proposed score |XU_j||WD_j| is not an upper bound for this change. A channel with large |XU_j| and near-zero gate output will be ranked important even though its contribution to the block output is negligible. The central claim that sF_j measures influence on the block output is not established for the evaluated architectures; the derivation must be redone with the gate included, or the score must be presented as a heuristic with supporting analysis.","section":"Method, Eqs. (4)-(7), (9)"},{"comment":"The MSA importance score is stated as |XH_j|·|WO_j|(I+|WU||WD|) with the derivation deferred to 'the supplementary materials', but no supplementary derivation is included in the manuscript, and the formula has dimensional ambiguities (for example, the roles of the per-token activation vector and the output-channel index are not specified). Since the coupling term (I+|WU||WD|) involves FFN weights and the FFN is gated in the evaluated models, the expression cannot be taken as established. The authors must provide the full derivation and state the exact dimensions of each term, or remove the claim that Eq. (8) is derived.","section":"Eq. (8), Definition 1, Remark"},{"comment":"The experimental section states that LLaMA-2-13B is evaluated, but Table 3 labels the unpruned baseline as 'LLaMA-13B (Ma, Fang, and Wang 2023)' and lists a baseline perplexity of 11.58 for the original model. It is unclear whether the pruned model is LLaMA-1-13B or LLaMA-2-13B, and the baseline values appear to be taken from another paper. This matters because LLaMA-1 and LLaMA-2 have different architectures and tokenizers, and the 13B results are part of the central empirical claim. Please clarify the exact checkpoint and report the original model's numbers measured with the same evaluation pipeline.","section":"Tables 1-3 and Section 'Experimental Settings'"},{"comment":"Algorithm 1 describes a sequential per-block pruning loop, but the text does not specify whether the forward pass for block l is computed with the original model or with the already-pruned earlier blocks. This distinction is important for the claim that block-wise pruning reduces error accumulation: if the activations are always taken from the unpruned model, then the measured per-block errors in Figure 3 are not the same as the errors that propagate through the pruned network. Please state the exact forward-propagation protocol used during pruning and clarify what Figure 3 measures.","section":"Ablation Study and Algorithm 1"}],"minor_comments":[{"comment":"The inequality chain in Eq. (7) uses both dot products and element-wise products without defining how the vector |XU| and the matrix |WD| are multiplied in the displayed expressions; the final sum suggests a per-channel product, but the preceding line is dimensionally ambiguous. Please make the notation explicit.","section":"Eq. (7)"},{"comment":"There are typographical errors in the tables: 'Importace Propagation' should be 'Importance Propagation', and the 'w/tune' and 'w/o tune' labels are inconsistent with the formatting elsewhere. These should be corrected for reproducibility.","section":"Tables 1-2"},{"comment":"The reported improvements are described variously as 3.26%, 4.49%, and 2.18% average accuracy gains; the abstract and introduction do not make clear which baselines and sparsity levels each number refers to. Please add a precise sentence tying each reported average to the corresponding table and conditioning regime.","section":"Abstract and Introduction"}],"recommendation":"major_revision","confidential_remarks":"The empirical results are potentially interesting, but the central theoretical derivation is currently not valid for the gated SwiGLU architectures used in the experiments. If the authors can supply a corrected derivation for the gated FFN, or explicitly reframe the score as an empirically validated heuristic, a revised version could be publishable. The missing supplementary derivation for Eq. (8) must also be provided. The 13B checkpoint inconsistency should be resolved before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: LLM-BIP is a one-forward-pass structured pruning method that scores FFN channels by |XU|·|WD| and attention output channels by |XH|·|WO|(I+|WU||WD|), then prunes inside each transformer block. The block-level scope is the genuinely new piece; without it, the FFN score is essentially Wanda's activation-times-weight heuristic, and the attention score is a blockwise extension. The experiments are broad: three models, 20–70% sparsity, seven zero-shot tasks, perplexity, speed, latency, calibration-size robustness, and an error-accumulation comparison. If the numbers are right, the method is useful, especially at high sparsity without fine-tuning.\n\nThe stress-test concern is correct and it is central. Eq (4) defines the FFN as σ(X'W_U)W_D, an un-gated MLP; LLaMA and Vicuna use SwiGLU, (SiLU(X'W_G) ⊙ X'W_U)W_D. Eq (7)'s bound is for the un-gated form, and the gate never enters the score. You can recover a loose bound by bounding SiLU by a constant, but then |XU|·|WD| is an upper bound, not a precise measure of a channel's effect on the block output. A channel with a large up-projection activation and a near-zero gate can be scored important even though its actual contribution is negligible. So Definition 1 overclaims. The metric may still work as a heuristic—the no-fine-tuning results are consistent with that—but the paper should frame it that way and not lean on the bound as exact support for these architectures.\n\nOther soft spots: Eq (8) is said to be derived in the supplementary material, which is not included in this arXiv v1, so the attention score is currently unverified. No code is provided, so I cannot check the tables. Baselines are on the older side—BESA is cited but not compared—so the empirical ranking is against a fairly weak field. And the fine-tuning comparison is not controlled: at 50% sparsity LLM-BIP is worse than LLM-Pruner after fine-tuning, which the authors attribute to using 20k samples and one epoch versus 50k and two. That is honest, but it means the headline \"outperforms\" only clearly holds for zero-shot no-fine-tuning results.\n\nWho should read it: practitioners looking for cheap structured-pruning heuristics will want this as a baseline; theory-minded readers should not take the bound at face value. I would send it to peer review rather than desk-reject: the idea is simple, the experiments are extensive, and the core fix is to reframe the metric as an effective heuristic, include the attention derivation, release code, and add controlled fine-tuning and a modern baseline or two. But a referee should put the SwiGLU mismatch front and center.\n\nAll the best.","headline":"A simple, efficient block-wise pruning heuristic with attractive zero-shot numbers, but the derivation meant to justify it is written for an un-gated FFN and does not apply to the SwiGLU models actually pruned.","tokens_in":14820,"tokens_out":6479,"would_cite":true,"duration_ms":65516,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T50","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-BIP prunes transformer blocks by scoring how much each channel and attention head changes the block's output, computing the scores in a single forward pass, and reports that this outperforms Wanda and LLM-Pruner at high sparsity.","keywords":["structured pruning","large language models","block-wise importance propagation","single forward pass pruning","attention head pruning","FFN channel pruning","LLaMA","zero-shot evaluation"],"falsifier":"On a gated LLaMA-7B block, compute the paper's score $|X^U_j||W^D_j|$ for every FFN channel and also measure the true block-output error from removing each channel alone over a fixed calibration set; if the rank correlation between score and measured error is low for channels with small $|X^U_j|$ but large gate outputs, the score fails to measure what the central claim requires. A simpler check is to test whether the Eq. (7) inequality actually holds on the gated architecture for randomly pruned masks; finding even one mask where the bound is violated would refute the derivation, though not necessarily the empirical heuristic.","tokens_in":13776,"feed_emoji":"✂️","tokens_out":6774,"duration_ms":63217,"temperature":0.7,"pith_summary":"LLM-BIP claims that the right unit for judging what to prune in a transformer is the transformer block, not a single layer: it derives closed-form importance scores for FFN channels and attention-head output channels that upper-bound the change in block output when those structures are removed, and it computes the scores from one forward pass. If correct, this gives a fast, gradient-free structured pruning method that preserves more of a model's reasoning ability at 20–70% sparsity than layer-wise or gradient-based alternatives. The paper reports that pruning LLaMA-7B, Vicuna-7B, and LLaMA-13B with these scores reduces perplexity on WikiText2 and PTB and raises zero-shot accuracy relative to Wanda and LLM-Pruner. The contribution is a block-aware importance metric that avoids both unreliable gradients and layer-wise error accumulation.","feed_headline":"One-pass block scoring prunes LLMs with less accuracy loss","feed_subtitle":"LLM-BIP derives closed-form channel and head importance from block output bounds, cutting perplexity and latency.","key_machinery":"The central objects are the block-wise importance scores in Definition 1: $s^F_j = |X^U_j|\\cdot|W^D_j|$ and $s^H_j = |X^H_j|\\cdot|W^O_j|(I+|W^U||W^D|)$, where $X^U_j$ and $X^H_j$ are the output values of the $j$-th FFN channel and attention-head output channel, and $W^D_j$ and $W^O_j$ are the corresponding output-weight rows. They come from an inequality that bounds the $\\ell^1$ change in the block output by $C_\\sigma \\sum_j (1-s^F_j)|X^U_j||W^D_j|$, so selecting channels to keep is equivalent to keeping those with large scores. The same inequality is the reason the method needs only one forward pass: the scores are read off from activations and weights already computed during a single feed-forward sweep, and no gradient or Hessian is required.","core_discovery":"The central claim is that a channel's contribution to a transformer block can be measured by a product of its activation magnitude and the norm of its output weights: $s^F_j = |X^U_j|\\cdot|W^D_j|$ for FFN channels and $s^H_j = |X^H_j|\\cdot|W^O_j|(I+|W^U||W^D|)$ for attention-head output channels. The paper derives these scores as an upper bound on the $\\ell^1$ distance between the original block output and the pruned block output, using Lipschitz continuity of the activation and treating pruning as zeroing entries of $X^U$ or $X^H$. Minimizing the bound leads to pruning channels with small scores, and the bound is computable from a single forward pass with no gradients or Hessians. The paper consequently claims that LLM-BIP prunes structural groups more accurately than global gradient-based methods and more robustly than layer-wise methods, with the gap widening at high sparsity.","pith_inferences":["Because the derivation ignores the gate projection in gated FFNs such as SwiGLU, a natural extension is to multiply the score by the gate activation magnitude $|X^G_j|$; this variant could be tested directly against the paper's score on LLaMA models to see whether the high-sparsity gap closes further.","The same block-output upper-bound logic could be applied to other block-structured components or combined with quantization, since both are block-local reconstruction objectives.","The paper's upper bound sums independent per-channel contributions, so it does not model interactions between simultaneously pruned channels; at very high sparsity, a correction term for co-pruning effects may be needed.","If the metric is as calibration-robust as reported, it could be used in streaming or on-device settings where a pruning decision must be made from a tiny observation budget."],"forward_implications":["Pruning decisions can be made once, in a single forward pass, without computing or storing gradients; the paper reports pruning runs in about half the time of LLM-Pruner on an A100 GPU.","At 20% and 50% sparsity, the pruned LLaMA-7B, Vicuna-7B, and LLaMA-13B models keep higher zero-shot accuracy and lower perplexity than Wanda, LLM-Pruner, magnitude-based, random, and importance-propagation baselines.","Because importance is scored at block level, errors accumulate more slowly across layers than with layer-wise pruning; the paper shows LLM-BIP has smaller block-output reconstruction error than Wanda as depth increases.","The method remains usable without fine-tuning; fine-tuning with LoRA gives further gains mostly at higher sparsity, and at 20% sparsity the pruned model is already near the fine-tuned accuracy.","Structured pruning decisions translate to hardware speedup: the paper reports 24% and 40% inference latency reductions at 20% and 50% sparsity, respectively."],"supporting_citations":[{"why":"Wanda is the layer-wise pruning baseline the method is compared against and the source of the one-forward-pass pruning setup.","marker":"Sun et al. 2024"},{"why":"LLM-Pruner is the gradient-based structural pruning baseline that LLM-BIP claims to outperform in accuracy and speed.","marker":"Ma, Fang, and Wang 2023"},{"why":"Importance propagation is the global pruning baseline whose Lipschitz-based score propagation LLM-BIP contrasts with block-wise scoring.","marker":"Yu et al. 2018"},{"why":"This result shows the Lipschitz assumption fails for self-attention, which motivates pruning within a block rather than globally.","marker":"Kim, Papamakarios, and Mnih 2021"},{"why":"Supplies the LLaMA-7B and LLaMA-13B models on which the pruning scores are evaluated.","marker":"Touvron et al. 2023"},{"why":"Supplies the Vicuna-7B model used as a third evaluation target.","marker":"Wei-Lin Chiang et al. 2023"},{"why":"Documents the error-accumulation problem in layer-wise LLM pruning and motivates the block-wise scope.","marker":"Xu et al. 2024"}],"fun_headline_variants":["One-pass block scores prune LLMs more accurately than gradients","LLM-BIP: prune by block output impact, no backprop needed","Block-wise importance propagation cuts LLM size in one forward pass","Gradient-free channel scoring trims LLMs with less perplexity","Single-pass block-bound scores outperform layer-wise pruning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The derivation models the FFN as a single up projection followed by activation and down projection, while the evaluated LLaMA and Vicuna models use a gated FFN in which a separate gate output multiplies the up output before the down projection; if the gate activation materially changes which channels matter, the score is not a true upper bound on the block error.","fun_headline_variants_meta":{"raw":{"variants":["One-pass block scores prune LLMs more accurately than gradients","LLM-BIP: prune by block output impact, no backprop needed","Block-wise importance propagation cuts LLM size in one forward pass","Gradient-free channel scoring trims LLMs with less perplexity","Single-pass block-bound scores outperform layer-wise pruning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000391,"raw_usage":{"total_tokens":2097,"prompt_tokens":1026,"completion_tokens":1071,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":642,"completion_tokens_details":{"reasoning_tokens":999}},"tokens_in":642,"tokens_out":1071,"duration_ms":7942,"temperature":1.0,"reasoning_tokens":999,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:41:11.475378+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a gated LLaMA-7B block, compute the paper's score $|X^U_j||W^D_j|$ for every FFN channel and also measure the true block-output error from removing each channel alone over a fixed calibration set; if the rank correlation between score and measured error is low for channels with small $|X^U_j|$ but large gate outputs, the score fails to measure what the central claim requires. A simpler check is to test whether the Eq. (7) inequality actually holds on the gated architecture for randomly pruned masks; finding even one mask where the bound is violated would refute the derivation, though not necessarily the empirical heuristic.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Wanda is the layer-wise pruning baseline the method is compared against and the source of the one-forward-pass pruning setup."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This result shows the Lipschitz assumption fails for self-attention, which motivates pruning within a block rather than globally."},{"cited_title":"Gonzalez ; Ion Stoica ; and Eric P","cited_arxiv_id":null,"evidence_quote":"Supplies the Vicuna-7B model used as a third evaluation target."}],"review_version":1}