{"id":"fd2b2b93-7e6e-4484-9dad-4db98e50951f","arxiv_id":"2412.07174","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SCAP prunes FC-layer input activations using calibrated L1 thresholds plus a mode-centering shift, achieving higher sparsity and about 1.5x the decoding speedup of CATS at similar task quality.","lead":"This paper introduces SCAP, a post-training method that prunes the inputs to fully-connected layers of large language models and adds a mode-centering step that shifts activation distributions to make more values near zero. It reports higher activation sparsity at the same task quality as the prior CATS method, yielding about 1.5 times the decoding speedup.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 1.5x speedup claim rests on a single iso-quality pair assembled from borrowed CATS accuracy numbers; if CATS shifts under identical evaluation, the headline comparison may not be iso-quality.","rationale":"I read the paper in good faith: SCAP is clearly described, the mode-centering algebra in Eqs. 4-6 is exact before pruning, the code is public, and the kernel latency measurements in Fig. 9 give independent support to the claim that sparse-input GEMVs are implementable. The method is also evaluated on more architectures than CATS, which supports the generality contribution. The central quantitative claim, however, is not just that SCAP can prune activations, but that SCAP achieves 48.5% FFN sparsity versus CATS 33.3% at iso quality, giving 1.5x additional decoding speedup. That pair is the fulcrum, and it depends on two numbers being equal: 74.2 for CATS and 74.2 for SCAP. Section D.1 explicitly says the CATS numbers are directly referenced from the CATS paper, not re-measured in the same harness; the SCAP numbers are from the authors' own pipeline. With seven coarse zero-shot tasks, no error bars, and one-decimal averages, equality at 74.2 is fragile evidence for iso-quality. This is not a claim about intent; it is a claim about evidentiary weight. A single re-evaluation of both models in one harness would settle it. The reader's nominated weakest assumption (L1 magnitude importance in Eq. 2) is a legitimate theoretical gap, but it applies equally to CATS and is only indirectly load-bearing for the 1.5x comparison; the iso-quality pairing is more direct. Since the manuscript already acknowledges the borrowed CATS numbers and the reader already conditioned on reproducibility, I would keep the verdict CONDITIONAL, which corresponds to UNCHANGED relative to the reader's verdict.","tokens_in":16543,"tokens_out":10568,"duration_ms":119386,"concrete_test":"Re-evaluate the exact CATS-50 and SCAP-(40,60) Mistral-7B-v0.1 models from Table 5 using the same evaluation harness, task list, prompts, and at least three independent calibration/evaluation seeds, reporting mean and 95% confidence intervals for the zero-shot average. If the CATS mean is not within 0.2 points of the SCAP mean, re-select the iso-quality pair and recompute Table 2's speedup ratio; also benchmark the unmodified official CATS kernel and SCAP's kernel on the same L40S GPU to confirm the 17.7% CATS baseline is not an implementation artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section D.1 states that for the Pareto comparison in Fig. 6 and Table 5, the authors directly referenced results reported in CATS [12], while SCAP points come from their own runs. The headline pair in Table 2 is CATS-50 (74.2% zero-shot average, 33.3% FFN sparsity) and SCAP-(40,60) (74.2%, 48.5%). The two averages are equal only at one decimal place, with per-task differences up to about 1.7 points, and no variance, seeds, or repeated evaluations are reported. Because the central claim is a 1.5x speedup ratio at iso quality, this pair is load-bearing: if the CATS average is actually, say, 74.7 or 73.7 under the same harness, then the two configurations are not iso-quality, and the sparsity gap (48.5% vs 33.3%) is not measured at equal task performance. The kernel comparison has a similar dependence: CATS speedup (17.7%) is from the authors' reproduction of CATS code, and a slower-than-official CATS kernel would inflate the ratio. The L1 magnitude assumption flagged by the reader is real but shared with CATS and indirectly checked by task accuracy; the comparison fairness is more directly tied to the headline.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SCAP, a post-training activation pruning method that induces sparsity in the input activations of fully-connected (FC) layers of Transformers by thresholding on the L1 magnitude of activations, with a Mode-Centering pre-calibration step for activation distributions whose mode is not near zero. The method is calibration-based, requiring no retraining or predictor networks. The authors claim that SCAP achieves a better Pareto trade-off between FFN sparsity and zero-shot task accuracy than the prior post-training method CATS, and they report a 1.5x relative decoding speedup over CATS at iso model quality on Mistral-7B. The paper additionally demonstrates SCAP on a wide range of models, including non-GLU decoders, MoE, Mamba2, vision encoders, and pre-quantized models.","tokens_in":16800,"tokens_out":8164,"duration_ms":79074,"significance":"If the central claims are substantiated, SCAP is a practically useful contribution: it offers a generic, calibration-only sparsification approach that applies to input activations of any FC layer, and the Mode-Centering technique provides a simple fix for non-zero-centered activation distributions. The paper is commendable for releasing code, for covering a broad set of model families, and for including a kernel-level discussion of decoding acceleration. The main significance is conditional on the reliability of the headline iso-quality comparison, which is currently not fully supported by the evidence presented.","major_comments":[{"comment":"The central claim of a 1.5x decoding speedup at iso quality rests on a single pair of configurations (Table 2) in which the CATS accuracy (74.2%) is taken directly from the CATS paper rather than measured with the same evaluation harness, as stated in Section D.1. Per-task differences between CATS and SCAP in Table 5 reach about 1.7 points (e.g., Arc-C 50.4 vs 48.7), and no variance, seeds, or repeated evaluations are reported, so the equality at one decimal place does not establish that the two points are iso-quality. The paper should rerun CATS under the same harness (or provide paired bootstrap confidence intervals) and also verify that the internally reproduced CATS kernel speedup (17.7%) is consistent with the official implementation, since a slower CATS kernel would inflate the ratio. Without this, the headline 1.5x relative speedup is not supported.","section":"Section 4.1 / D.1 / Table 2"},{"comment":"Algorithm 2 is inconsistent with the method description. In a SwiGLU FFN, the input to the Down projection is the gated activation (product of the Up path and the SiLU of the Gate path), but line 8 of Algorithm 2 passes the original input x to SCAP_FC for the Down projection (y ← SCAP_FC(τgated, x, Wdown)), rather than the computed zgated. As written, the algorithm computes the wrong operation and cannot reproduce the Down sparsities reported in Tables 5 and 6. The pseudocode should be corrected and the variable definitions clarified.","section":"Appendix B, Algorithm 2"},{"comment":"The importance measure in Eq. (2) is the L1 magnitude of the input activation, but the paper does not analyze when this is a valid proxy for the actual contribution of that activation to the output, given that weight column norms vary. This matters for the claimed generality of SCAP to all FC layers (attention QKV/output projections, non-GLU Down projections, and MoE layers), where the assumption has not been previously validated. The paper should provide per-layer evidence, such as output reconstruction error or a comparison with a weight-aware importance criterion, to show that L1 thresholding is safe for the new layer types; the downstream task accuracy alone is an indirect check that mixes in the robustness of the task metric.","section":"Eq. (2)"}],"minor_comments":[{"comment":"Algorithm 2, line 3 uses the symbol 'xηW' which is not defined; either use xW for the non-mode-centered case or explicitly define the mode-centered input xη in the pseudocode, and align the threshold comparison (≥ in Alg. 2 vs > in Eq. 2).","section":"Appendix B, Algorithm 2"},{"comment":"The decoding speedups in Table 2 are reported to one decimal place without error bars or the number of repeated runs; please state the measurement uncertainty or provide the number of independent repetitions.","section":"Section 4.2"},{"comment":"The 'Relative (%)' column should specify that it is the relative change in the task metric (zero-shot average or ImageNet top-1) against the unpruned baseline, and the exact task set used for each model should be listed for the non-LLM entries.","section":"Table 4"},{"comment":"The subcaptions in Figure 3 are too terse: identify the model and exact tensor for each panel (e.g., 'post-LayerNorm input to Up/Gate in Mistral-7B') so that the distributions can be reproduced.","section":"Figure 3"},{"comment":"The statement that SCAP 'outperformed TurboSparse' on the remaining tasks is not supported by a per-task table; adding the per-task scores would make the comparison transparent and would also clarify the large GSM8K gap.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid practical contribution with a clear, calibration-only method and broad empirical coverage. The main technical concern is not the core derivation but the fairness and statistical support of the headline iso-quality comparison; this is fixable with additional experiments and tightening of the manuscript. I recommend major revision. The citation and novelty disclosure appear appropriate for this workshop-style paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: SCAP is a sensible extension of CATS with one genuinely new trick (mode-centering) and a broad empirical sweep; the headline 1.5x speedup over CATS is plausible but the iso-quality basis is shakier than the paper lets on, because the CATS numbers are borrowed rather than rerun.\n\nWhat's actually new: pruning input activations of FC layers (instead of post-SiLU outputs) is a simple generalization that makes the kernel uniform and lets you tune Up/Gate/Down sparsity independently. Mode-centering is a clean, parameter-free pre-processing step that shifts the activation mode to zero and folds the compensation into the bias. It produces large sparsity gains on GELU-based FFNs (Falcon, MPT) and the paper shows it helps ViT too. The model coverage is genuinely useful: MoE, Mamba2, quantized Llama-3.1, and DeiT are not just listed, they get sparsity numbers within 1% of baseline. Appendix C is an honest discussion of why batch-size >1 limits acceleration. Code is released.\n\nSoft spots: The central comparison in Figure 6 / Table 2 mixes SCAP's own runs with CATS numbers quoted from the CATS paper. The two chosen points are iso-quality only to one decimal place; individual task scores differ by up to 1.7 points. With no error bars or multiple seeds, the 1.5x ratio is a point estimate, not a measured fact. The kernel benchmark uses a re-implementation of CATS, so a slower/faster port could move the ratio too. None of this is fatal—the method is well described and reproducible—but a referee should demand a rerun of CATS in the same harness and a report of variance.\n\nOn the L1 magnitude concern in the reader's report: that assumption is shared with CATS and is only indirectly validated by downstream task accuracy. It's a limitation of the whole family, not a new flaw in SCAP. The paper does not analyze error propagation, but for an engineering paper the empirical check is reasonable.\n\nVerdict: This is a workshop-quality paper with a real contribution. The mode-centering idea is worth citing. The headline speedup needs to be re-measured head-to-head, but if it survives that, the practical value is clear. A serious referee should engage and ask for a tight experimental re-run.","headline":"SCAP is a practical extension of CATS with a genuinely useful mode-centering trick; the 1.5x speedup claim is plausible but rests on borrowed CATS numbers and needs a head-to-head rerun.","tokens_in":17363,"tokens_out":2584,"would_cite":true,"duration_ms":25270,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SCAP prunes FC inputs and centers activation modes, hitting 48.5% FFN sparsity on Mistral-7B at iso quality versus CATS' 33.3% and a 1.5x relative decoding speedup.","keywords":["activation sparsity","post-training pruning","feed-forward networks","mode-centering calibration","L1 thresholding","LLM decoding speedup","SiLU and GELU activations","Transformer inference"],"falsifier":"Compare pruned and unpruned outputs layer by layer on held-out text: if many activations just below the calibrated threshold fall in weight columns with large column norms, the reconstruction error of the sparse layer will grow faster than the reported task-accuracy curves; that is directly measurable.","tokens_in":16325,"feed_emoji":"⚡","tokens_out":4963,"duration_ms":51409,"temperature":0.7,"pith_summary":"SCAP is a post-training method for making large language models sparser without retraining or uptraining. The paper argues that pruning the input activations of fully-connected layers, rather than the outputs of activation functions, is a more general and flexible way to induce sparsity, and that a simple Mode-Centering step makes this much more effective on GELU and SiLU models. The headline result is that at equal zero-shot task accuracy on Mistral-7B, SCAP reaches 48.5% feed-forward sparsity versus CATS' 33.3%, translating into a 27.1% decoding speedup versus 17.7% for CATS—a 1.5x relative gain. The paper also shows the method extends to MoE, Mamba2, vision transformers, and pre-quantized models, which matters because it promises practical, low-cost activation sparsity for modern non-ReLU LLMs.","feed_headline":"Activation pruning hits 48.5% sparsity, beats CATS by 1.5x","feed_subtitle":"SCAP centers activation peaks at zero, then prunes FC inputs, buying 27.1% decoding speedup vs CATS's 17.7%.","key_machinery":"The central machinery is threshold-based activation pruning on FC inputs (Eq. 2): each element $X_{ij}$ is kept only if $|X_{ij}| > \\tau$, with $\\tau$ calibrated as a quantile of saved calibration activations. The second ingredient is Mode-Centering calibration (Eq. 6): a static mode $\\eta$ is estimated offline and subtracted from the activation, with the compensation folded into the fused bias, so the distribution's peak moves to zero and many more values fall below the pruning threshold. Together they decouple sparsity across Up, Gate, and Down projections, which the paper identifies as the source of its Pareto advantage over CATS.","core_discovery":"SCAP applies an L1-magnitude pruner to the input activation $X$ of any fully-connected layer, masking elements with $|X_{ij}| < \\tau$, where $\\tau$ is set by a quantile of calibration activations. Because the pruner sits at the FC input, the same kernel and calibration procedure apply to the Up, Gate, and Down projections of GLU FFNs, to attention projections, and to non-GLU GELU networks. For activations whose distribution peaks away from zero, the paper introduces Mode-Centering: it estimates the mode $\\eta$ offline, computes $Y = (X - \\eta)W + b_{\\mathrm{fused}}$, and folds the compensating $\\eta W$ into the bias, shifting the density peak to zero and making L1 thresholding far more effective. The paper's central empirical claim is that at equal zero-shot accuracy on Mistral-7B, SCAP achieves 48.5% FFN sparsity versus CATS' 33.3%, producing a 27.1% decoding speedup versus 17.7%—a 1.5x relative improvement.","pith_inferences":["The same mode-centering logic could be pushed further with per-layer affine calibration beyond a scalar shift—such as a learned shift plus scale—which might concentrate even more activation mass near zero, though the paper only tests a single scalar mode.","SCAP's speedup numbers assume batch-1 decoding; the paper's own appendix shows overlapping sparsity drops sharply as beam width or batch size grows, so the practical 1.5x advantage is likely smaller in high-throughput batched serving.","Pruning at FC inputs changes the activation distribution seen by every downstream layer, and the paper validates this only through aggregated task accuracy; a direct layer-by-layer error-propagation study would be a natural next test.","The paper compares against trained sparsity methods like TurboSparse at lower sparsity but far lower compute cost; a natural extension is combining SCAP with parameter-efficient fine-tuning to close the sparsity gap."],"forward_implications":["Post-training activation sparsity no longer depends on ReLU activations; SCAP works on SiLU/GELU models like Mistral, Llama-2, Falcon, and MPT without any uptraining.","Because sparsity is applied at FC inputs, SCAP can set different sparsity levels for Up, Gate, and Down projections, which is what yields higher FFN sparsity at a given task-accuracy tolerance than CATS.","Mode-Centering raises Down-projection prunability in non-GLU FFNs: Falcon-7B goes from 30.5% to 50.3% sparsity and MPT-7B from 12.7% to 57.4% within about 1% relative task drop.","The method transfers across model families—Mixtral, Mamba2, DeiT vision transformers, and pre-quantized Llama models—all at roughly -1% relative task accuracy.","Because SCAP requires only a single GPU for calibration and no predictor training, it makes activation sparsification feasible on modest hardware in a post-training workflow."],"supporting_citations":[{"why":"CATS is the prior post-training activation-sparsification baseline that SCAP must beat; it supplies the Pareto comparison and the 33.3% sparsity / 17.7% speedup reference points.","marker":"[12]"},{"why":"TurboSparse is the state-of-the-art training-based Relufication method used as the SOTA comparison, providing the 82.2% FFN sparsity figure that SCAP is contrasted against.","marker":"[26]"},{"why":"The Lazy Neuron Phenomenon establishes that activation sparsity is prevalent in pretrained Transformers, which is the underlying motivation for exploiting it.","marker":"[13]"},{"why":"ReLU Strikes Back provides the evidence that SiLU and GELU activations are dense while ReLU is sparse, motivating a post-training method for non-ReLU models.","marker":"[17]"},{"why":"Deja Vu introduced contextual sparsity and predictor-based sparse execution, the alternative approach SCAP avoids by pruning FC inputs directly.","marker":"[15]"},{"why":"This work identifies memory bandwidth as the primary bottleneck in decoding, which is the rationale for why sparse GEMV layers should produce speedup.","marker":"[20]"},{"why":"C4 is the calibration corpus used by SCAP to estimate pruning thresholds and mode values for all experiments.","marker":"[21]"}],"fun_headline_variants":["SCAP: 48.5% sparsity, 1.5x speedup vs CATS","Mode-Centering makes activation pruning 1.5x faster","Post-training calibration boosts LLM sparsity and speed","SCAP achieves 48.5% pruning at same accuracy, 1.5x speedup"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"A small activation magnitude is treated as a reliable sign that the corresponding weight column contributes little to the output, even though the paper does not examine weight magnitudes or error propagation.","fun_headline_variants_meta":{"raw":{"variants":["SCAP: 48.5% sparsity, 1.5x speedup vs CATS","Mode-Centering makes activation pruning 1.5x faster","Post-training calibration boosts LLM sparsity and speed","SCAP achieves 48.5% pruning at same accuracy, 1.5x speedup"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000987,"raw_usage":{"total_tokens":4179,"prompt_tokens":931,"completion_tokens":3248,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":3162}},"tokens_in":547,"tokens_out":3248,"duration_ms":22364,"temperature":1.0,"reasoning_tokens":3162,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:04:29.479425+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare pruned and unpruned outputs layer by layer on held-out text: if many activations just below the calibrated threshold fall in weight columns with large column norms, the reconstruction error of the sparse layer will grow faster than the reported task-accuracy curves; that is directly measurable.","supporting_citations":[{"cited_title":"The Lazy Neuron Phenomenon: On Emergence Of Activation Sparsity In Transformers","cited_arxiv_id":null,"evidence_quote":"The Lazy Neuron Phenomenon establishes that activation sparsity is prevalent in pretrained Transformers, which is the underlying motivation for exploiting it."},{"cited_title":"del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, and Mehrdad Farajtabar","cited_arxiv_id":null,"evidence_quote":"ReLU Strikes Back provides the evidence that SiLU and GELU activations are dense while ReLU is sparse, motivating a post-training method for non-ReLU models."},{"cited_title":"Deja Vu: contextual sparsity for efficient LLMs at inference time","cited_arxiv_id":null,"evidence_quote":"Deja Vu introduced contextual sparsity and predictor-based sparse execution, the alternative approach SCAP avoids by pruning FC inputs directly."},{"cited_title":"Efficiently Scaling Transformer Inference","cited_arxiv_id":null,"evidence_quote":"This work identifies memory bandwidth as the primary bottleneck in decoding, which is the rationale for why sparse GEMV layers should produce speedup."}],"review_version":1}