{"id":"a43c5f55-4965-48b5-82e0-74accfa36bed","arxiv_id":"2501.15255","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"COMP prunes LLMs post-training by removing redundant layers and then neurons ranked by a matrix condition-number metric, recovering accuracy via mask tuning.","lead":"COMP is a pruning method that removes whole layers and then individual neurons from large language models without fine-tuning, using only a few calibration samples and low memory. It reports keeping about 91% of a 7B model's accuracy after cutting 20% of its parameters, which could make on-device LLM deployment cheaper.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never isolates the condition-number neuron-importance metric from random or uniform neuron pruning, so the central claimed advantage of COMP is empirically unsupported.","rationale":"After reading the paper carefully, the most load-bearing weakness is the neuron-importance metric. The central empirical claim—that COMP outperforms LLM-Pruner, ShortGPT, and SliceGPT—could in principle hold even if the metric were random, because the hybrid layer+neuron strategy and mask tuning might be the real drivers. But the paper's stated novelty and a key contribution is the 'new matrix condition-based metric' for neuron importance (Section III.B). This metric is the only component that distinguishes COMP's fine-grained pruning from uniform or random neuron removal. The paper provides no ablation isolating this metric. Figure 2 compares layer, uniform-neuron, and hybrid strategies, all with equal neuron counts; the ablation studies test layer-order iteration and identical-input masking, not neuron ranking. Consequently, the reader's weakest assumption correctly identifies the unvalidated Taylor/Fisher approximation in Eqs. (7)-(9). The 6.13% headline discrepancy is a reporting error, not a challenge to the method's validity; it should be fixed but does not affect the central claim as much as the missing metric validation. The proposed concrete check—a head-to-head comparison of the metric vs. random/uniform neuron selection—would directly resolve whether the metric contributes anything. Without that comparison, the paper should not be accepted as-is; the CONDITIONAL verdict with a required ablation is appropriate. We therefore keep the verdict unchanged.","tokens_in":14291,"tokens_out":7809,"duration_ms":65793,"concrete_test":"Run COMP on LLaMA-2-7B at 20% and 30% pruning, keeping layer pruning, ratio allocation, and mask tuning identical, but replace the neuron importance ranking in Algorithm 1 with (a) a seeded random ordering, (b) uniform neuron counts per dense, and (c) the proposed condition-number metric. Report WikiText2 perplexity and average zero-shot accuracy over the six tasks in Table I. If the condition-number metric outperforms random or uniform by less than 1% accuracy (or 0.5 perplexity), the claimed advantage of the importance metric is not established and the abstract's 'more accurately evaluate neuron importance' should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The key novelty of COMP is the matrix condition-based neuron importance metric in Eqs. (7)-(9). However, no experiment in the paper compares this metric to random selection, uniform neuron pruning, or magnitude-based baselines. Figure 2 compares layer pruning vs. uniform neuron pruning vs. their combination, but all neuron pruning in that figure removes equal counts per dense. The ablations in Section V.C only test iterative layer ordering and identical layer inputs, not the neuron ranking. Thus, the paper's claim that the metric 'more accurately evaluate[s] neuron importance' is not demonstrated. The Taylor expansion around the all-ones mask, with the Hessian replaced by the outer product of the gradient and then diagonalized, is a heuristic with no supporting evidence that it correctly ranks neurons. If the ranking is no better than random, COMP's advantage over baselines could come entirely from the hybrid layer plus uniform neuron pruning and mask tuning, and the central contribution of the paper—the importance metric—would be unsupported. This directly affects the load-bearing assertion that the hybrid strategy with condition-number-based neuron importance achieves the reported results.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"COMP is a post-training structured pruning method for LLMs. It first removes layers based on cosine-similarity redundancy, then performs input-neuron pruning in each remaining layer using a condition-number-based importance metric derived from the mask-tuning normal equations, and finally reconstructs dense outputs by mask tuning. The pruning ratio per layer is allocated via Eq. (10), and an iterative variance-threshold loop determines how many neurons to remove. Experiments on LLaMA-2, OPT, and ChatGLM3 compare with LLM-Pruner, SliceGPT, and ShortGPT on perplexity and zero-shot tasks; the paper reports lower perplexity than baselines at 20-30% pruning and memory usage of 8GB for LLaMA-2-7B.","tokens_in":14484,"tokens_out":8479,"duration_ms":75085,"significance":"The practical motivation is strong: on-device pruning without fine-tuning is an important problem, and the paper demonstrates consistent perplexity improvements over three baselines on five models with only 10 calibration samples. The layer-wise dynamic loading and the use of mask tuning are sensible engineering choices, and the comparison is fairly broad. However, the central novelty—the condition-number neuron-importance metric—is neither theoretically justified nor empirically isolated. The empirical tables support the claim that the full COMP pipeline works better than the tested baselines, but not the claim that the proposed metric is responsible for that advantage. If the missing ablations confirm the metric's value, the paper would be a useful contribution to post-training structured pruning.","major_comments":[{"comment":"The proposed second-order approximation is not actually a Hessian expansion. The Hessian H of κ(A^T A) is replaced by F = gg^T, which for a vector g is rank-one; after assuming diagonality, the \"second-order\" term reduces to 1/2 g_f^2, a function of first derivatives only, and the off-diagonal curvature that the Taylor expansion would contribute is discarded without justification. Since this metric is the paper's central novelty, the derivation needs either a rigorous justification or a direct validation (e.g., correlation between eI_f and the actual reconstruction-loss increase when pruning f). The sign convention in Eq. (7) versus Eq. (9) also needs clarification: if g = ∂κ/∂mc, pruning neuron f changes mc_f from 1 to 0, giving a first-order term -g_f; the expression for eI_f should be stated in terms of that change, not as a generic importance measure.","section":"Section III.B, Eqs. (7)-(9)"},{"comment":"The paper never tests whether the condition-number-based neuron ranking is better than random or uniform selection. In Figure 2, neuron pruning removes equal counts from every dense; the hybrid strategy is compared with layer-only and uniform neuron-only pruning, but not with selective ranking. The ablations in Section V.C cover only iterative layer ordering and identical layer inputs. Consequently, the claim that COMP's hybrid strategy with condition-number-based importance achieves the reported results is not supported by any experiment that isolates the metric. Please add ablations in which the same layer-pruning and mask-tuning pipeline is run with (i) the proposed eI_f ranking, (ii) random neuron selection, (iii) magnitude-based selection, and (iv) uniform per-dense counts.","section":"Section V.C and Figure 2"},{"comment":"The per-layer pruning-ratio formula is not reproducible as written. w_l is described as \"the harmonic mean of the rest layer's importance,\" but no normalization is specified, so the r_l values are not guaranteed to sum to the target r; the term n \\hat N appears to subtract removed-layer parameters using the current layer's parameter count rather than the removed layers' actual counts; and the variance-threshold loop increments v_T without specifying its step or a termination condition, so the algorithm's behavior at the target ratio is under-specified. Please clarify the formula and the loop, or provide the exact update rules used in the experiments.","section":"Section IV.B, Eq. (10), and Algorithm 1"}],"minor_comments":[{"comment":"ShortGPT is cited as [28] (SliceGPT) in the experimental setup; the correct reference is [18].","section":"Section V.A and References"},{"comment":"The abstract claims a 6.13% improvement, but Table I shows 58.74 vs 55.60 for LLaMA-2-7B at 20% pruning, a relative gain of 5.65%, and no 6.13% appears in the tables; please correct the number or specify the comparison used.","section":"Abstract and Table I"},{"comment":"Equation (3) uses X^{l+1}_t for the output of layer l while X^{l,k} denotes inputs to denses; define the layer-input/output notation consistently.","section":"Section III.A"},{"comment":"There are typos in \"Pruing ratio\" and \"remainig\"; please proofread these passages.","section":"Table III and Section IV.B"},{"comment":"The SliceGPT row for LLaMA-2-13B at 20% shows the same value 929.68 for WikiText2 and PTB; please verify this entry.","section":"Table I"},{"comment":"The Limitation paragraph reports 30 minutes for LLaMA-2-7B at 30% pruning and about 1 hour for LLaMA-2-13B; consider including a runtime comparison with baselines, since on-device deployment claims depend on practical time as well as memory.","section":"Section VI (Limitation)"}],"recommendation":"major_revision","confidential_remarks":"The main risk is novelty attribution: the mask-tuning component is adapted from FPT and the layer-redundancy criterion from ShortGPT, so the only genuinely new component is the neuron-importance metric, which is currently unvalidated in isolation. If the authors add the requested ablations and repair the derivation or soften the claims, the paper is within scope for a journal. No ethical concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: COMP is a practical post-training structured pruning recipe that looks genuinely useful for on-device LLM deployment, but the paper never tests its central novelty—the condition-number neuron importance metric—against random or uniform neuron pruning, and the headline 6.13% number does not match Table I.\n\nWhat is actually new: the combination of layer redundancy pruning, per-dense iterative neuron pruning with a variance-threshold stopping rule, and FPT-style mask tuning, with dynamic layer loading that keeps memory at 8GB for a 7B model. The empirical work is broad: LLaMA-2-7B/13B, OPT-6.7B/13B, and ChatGLM3-6B, and COMP consistently beats LLM-Pruner, ShortGPT, and SliceGPT on most reported perplexity and zero-shot accuracy numbers. The memory figures are plausible given the design. That is a solid engineering contribution.\n\nThe soft spots are in proportion.\n\nFirst, the key metric is never isolated. All neuron pruning in the paper removes equal counts per dense, and the ablations only test iterative layer ordering and identical-layer input. There is no comparison of the condition-number ranking against random or uniform neuron selection. So the claim that the metric 'more accurately evaluates neuron importance' is unsupported. The stress-test is right on this.\n\nSecond, the derivation in Eqs. (7)-(9) is a heuristic: a second-order Taylor expansion, then the Hessian is replaced by the outer product of the gradient, then diagonalized. That is essentially a squared-gradient saliency. It might work, but there is no evidence it beats magnitude-based or even random ordering for the mask-tuning objective. This is load-bearing for the fine-grained pruning stage.\n\nThird, the abstract says a 6.13% improvement over LLM-Pruner on LLaMA-2-7B at 20% pruning. Table I gives COMP 58.74 and LLM-Pruner 55.60 average zero-shot, a 3.14-point absolute gain (about 5.6% relative). The number needs correction or explanation.\n\nFourth, there is no code and no error bars. Single-run results on 10 calibration samples are acceptable for a preprint, but a peer-review version should include code and repeated runs.\n\nNone of these is fatal to the empirical claim that the overall method works well. The hybrid layer-plus-neuron strategy with mask tuning is effective. But the specific contribution of the condition-number metric is unproven. This is fixable: add an ablation for neuron ranking, release code, and fix the headline. If those are addressed, this becomes a solid paper.\n\nVerdict: send to peer review with major revision.","headline":"A useful post-training pruning recipe whose headline number is off and whose key neuron-importance metric is never actually tested.","tokens_in":15023,"tokens_out":3654,"would_cite":true,"duration_ms":28992,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"COMP, a post-training structured pruning method, can prune LLaMA-2-7B with only 8GB of memory at a 20% pruning ratio while retaining about 91.2% of its average zero-shot performance.","keywords":["structured pruning","post-training pruning","large language models","neuron importance","mask tuning","matrix condition number","hybrid granularity pruning","on-device deployment"],"falsifier":"Take a fixed layer-pruning schedule and mask-tuning protocol, then replace COMP's condition-number neuron ordering with random or uniform neuron ordering at the same pruning ratios and compare perplexity on WikiText2. If the randomly pruned model matches COMP's perplexity, the condition-number metric is not doing the work; equivalently, compute the metric's rank correlation with the actual leave-one-out output error for each neuron on the calibration data and check whether low score truly means low damage.","tokens_in":14061,"feed_emoji":"✂️","tokens_out":7968,"duration_ms":63553,"temperature":0.7,"pith_summary":"This paper introduces COMP, a structured pruning method that compresses a large language model after training, with no fine-tuning, using little GPU memory. The aim is to make pruning practical on the devices where the model will actually run. COMP prunes whole layers first, then individual neurons inside the remaining layers, ranking neurons by a new matrix-condition-number score, and repairs the damage with mask tuning. The reported result is that LLaMA-2-7B can be pruned at a 20% ratio in about 8GB of memory while keeping roughly 91.2% of its average zero-shot accuracy, beating LLM-Pruner, ShortGPT, and SliceGPT on the same benchmark.","feed_headline":"A pruning method that compresses LLaMA-2-7B using only 8GB of memory","feed_subtitle":"COMP cuts layers and neurons by a condition-number score, then mask-tunes to keep 91% of accuracy at 20% pruning.","key_machinery":"The load-bearing mechanism is the condition-number-based neuron importance metric. For each dense layer, the mask-tuning reconstruction is a least-squares problem whose coefficient matrix is $\\hat{A}^T\\hat{A}$; the paper argues that pruning neurons that increase the condition number of this matrix most makes the mask solution most sensitive to errors. A second-order Taylor expansion of the condition number, with the Hessian approximated by a diagonal Fisher matrix, yields a per-neuron score $eI_f = -g_f + \\tfrac12 F_{ff}$ that sorts neurons for pruning. This score is embedded in an iterative pruning loop: neurons are pruned one dense at a time until the variance of the tuned nonzero mask entries crosses an adaptively raised threshold, and each layer is pruned using the original model's input to stop errors from accumulating. Layer importance, in contrast, is simply one minus the cosine similarity between a layer's input and output, and layers are removed iteratively so later removals see the updated network.","core_discovery":"The central claim is that post-training structured pruning can be both lightweight and broadly applicable if pruning is done at two granularities and performance is restored by mask tuning rather than fine-tuning. On the paper's own account, COMP first removes low-importance layers, then prunes input neurons inside the denses of surviving layers using an importance score derived from the condition number of the mask-tuning least-squares coefficient matrix. The pruned model's outputs are then reconstructed by solving for a tuned mask, iteratively increasing the number of pruned neurons per dense while keeping the variance of mask entries under a threshold. The paper reports that this pipeline cuts the memory needed to prune a 7B model to 8GB, retains 91.2% of original average zero-shot accuracy at 20% pruning, and outperforms the compared post-training baselines across LLaMA-2, OPT, and ChatGLM3 models.","pith_inferences":["The condition-number criterion is defined on any linear layer's coefficient matrix, so a natural extension is to apply the same score to other removables such as attention heads or to semi-structured 2:4 masks; the paper does not test those cases.","The memory footprint scales with the largest dense input dimension rather than total parameter count, so models with unusually wide linear layers like OPT will not all reach the same 8GB budget; the paper's own OPT results hint at this.","A direct test of the neuron-importance metric is to compare COMP's nonrandom neuron selection against random or uniform selection with mask tuning held fixed; if perplexity does not change, the condition-number score is not the reason the method works.","The calibration budget is only 10 samples, so the variance of the method across random calibration draws is an open question and a cheap falsifiability check."],"forward_implications":["Fine-tuning can be removed from the structured-pruning pipeline for LLMs, which removes the need for large labeled datasets and the privacy risk of shipping user data to a cloud server.","A 7B-parameter model can be pruned on hardware with roughly 8GB of GPU memory at 20% compression, with most zero-shot accuracy intact.","The hybrid layer-plus-neuron strategy dominates either granularity alone, and the gap grows at higher pruning ratios such as 30%.","Because the pipeline is agnostic to layer-internal structure, the same pruning recipe applies to LLaMA-2, OPT, and ChatGLM3 families without model-specific adapters.","Iterative layer removal and identical-layer-input mask tuning are both shown to matter; recomputing importance after each layer removal cuts perplexity nearly in half when six layers are removed."],"supporting_citations":[{"why":"Supplies the mask-tuning idea that COMP adapts from BERT-style models to arbitrary LLM architectures.","marker":"[16]"},{"why":"ShortGPT is the layer-only post-training pruning baseline that COMP outperforms, especially on non-LLaMA models.","marker":"[18]"},{"why":"LLM-Pruner is the main gradient-based structured pruning baseline, needing fine-tuning and over 32GB memory for a 7B model.","marker":"[19]"},{"why":"Provides the LLaMA-2-7B and LLaMA-2-13B models on which the primary zero-shot benchmarks are run.","marker":"[27]"},{"why":"SliceGPT is the PCA-based post-training baseline that COMP compares against across all models.","marker":"[28]"},{"why":"Provides the OPT-6.7B and OPT-13B models used to test generality across a second architecture.","marker":"[34]"},{"why":"Source of the ChatGLM3-6B model family used to show COMP works on a third architecture where baselines fail.","marker":"[35]"},{"why":"Source of the only 10 calibration samples used for importance evaluation and mask tuning.","marker":"[45]"}],"fun_headline_variants":["Prune a 7B LLM with just 8GB of RAM","COMP: two-step pruning recovers accuracy without fine-tuning","COMP: 6.13% better pruning than LLM-Pruner on LLaMA-2-7B","Prune 7B LLM in 8GB, retain 91.2% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole neuron-pruning stage depends on a mathematical approximation: that a simplified sensitivity score estimated from a diagonal Fisher matrix ranks neurons correctly, so pruning the lowest-scoring ones does the least damage; if that ranking is wrong, the method's choice of which neurons to prune is arbitrary.","fun_headline_variants_meta":{"raw":{"variants":["Prune a 7B LLM with just 8GB of RAM","COMP: two-step pruning recovers accuracy without fine-tuning","COMP: 6.13% better pruning than LLM-Pruner on LLaMA-2-7B","Prune 7B LLM in 8GB, retain 91.2% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002052,"raw_usage":{"total_tokens":7993,"prompt_tokens":951,"completion_tokens":7042,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":6950}},"tokens_in":567,"tokens_out":7042,"duration_ms":45313,"temperature":1.0,"reasoning_tokens":6950,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:28:00.827177+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed layer-pruning schedule and mask-tuning protocol, then replace COMP's condition-number neuron ordering with random or uniform neuron ordering at the same pruning ratios and compare perplexity on WikiText2. If the randomly pruned model matches COMP's perplexity, the condition-number metric is not doing the work; equivalently, compute the metric's rank correlation with the actual leave-one-out output error for each neuron on the calibration data and check whether low score truly means low damage.","supporting_citations":[{"cited_title":"A fast post-training pruning framework for transformers,","cited_arxiv_id":null,"evidence_quote":"Supplies the mask-tuning idea that COMP adapts from BERT-style models to arbitrary LLM architectures."},{"cited_title":"Llm-pruner: On the structural pruning of large language models,","cited_arxiv_id":null,"evidence_quote":"LLM-Pruner is the main gradient-based structured pruning baseline, needing fine-tuning and over 32GB memory for a 7B model."}],"review_version":1}