Pith. sign in

REVIEW 12 cited by

The Super Weight in Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2411.07191 v2 pith:Q6CQSEQJ submitted 2024-11-11 cs.CL cs.AI

Mengxia Yu , De Wang , Qi Shan , Colorado J Reed , Alvin Wan This is my paper · ORCID
classification cs.CLcs.AI
keywords superweightlargemodeloutliersparametersquantizationweights
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Recent works have shown a surprising result: a small fraction of Large Language Model (LLM) parameter outliers are disproportionately important to the quality of the model. LLMs contain billions of parameters, so these small fractions, such as 0.01%, translate to hundreds of thousands of parameters. In this work, we present an even more surprising finding: Pruning as few as a single parameter can destroy an LLM's ability to generate text -- increasing perplexity by 3 orders of magnitude and reducing zero-shot accuracy to guessing. We propose a data-free method for identifying such parameters, termed super weights, using a single forward pass through the model. We additionally find that these super weights induce correspondingly rare and large activation outliers, termed super activations. When preserved with high precision, super activations can improve simple round-to-nearest quantization to become competitive with state-of-the-art methods. For weight quantization, we similarly find that by preserving the super weight and clipping other weight outliers, round-to-nearest quantization can scale to much larger block sizes than previously considered. To facilitate further research into super weights, we provide an index of super weight coordinates for common, openly available LLMs.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Individual Parameters in Weight-Sparse Transformers Appear Interpretable

    cs.LG 2026-07 conditional novelty 6.5 of 10

    An automated LLM pipeline finds that 12–31% of nonzero weights in weight-sparse transformers admit short, held-out-validated descriptions of when they matter, far above dense controls.

  2. Learning in Curved Weight Space:Exponential-Linear Weight Reparameterization for Improved Optimization

    cs.LG 2026-07 unverdicted novelty 6.0 of 10

    SEL weight reparameterization reaches matched OpenWebText validation loss in 1.32–1.49× fewer transformer steps via a sign-aware exponential-linear map and mismatched initialization.

  3. Super Weights in LLMs and the Failure of Selective Training

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Training Super Weights in isolation collapses LLM performance to random guessing, while full-layer low-rank updates succeed, showing parameter importance does not imply trainability.

  4. Float8@2bits: Entropy Coding Enables Data-Free Model Compression

    cs.LG 2026-01 conditional novelty 6.0 of 10

    EntQuant stores LLM weights at ~2 bits per parameter by entropy-coding Float8 weights, matching data-dependent compression quality without needing calibration data.

  5. Artifacts and Attention Sinks: Structured Approximations for Efficient Vision Transformers

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A training-free Nyström attention variant with farthest-point sampling preserves ViT performance at lower cost, and masking attention-sink tokens yields modest downstream gains.

  6. Structured Pruning for Diverse Best-of-N Reasoning Optimization

    cs.CL 2025-06 reject novelty 6.0 of 10

    SPRINT learns to select which attention heads to prune per question, improving Pass@N over random head selection and multinomial sampling on MATH500 and GSM8K.

  7. Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decomposition

    cs.LG 2025-06 conditional novelty 6.0 of 10

    ODLRI initializes the low-rank component using activation-outlier channels, improving low-bit compression of large language models over the CALDERA baseline.

  8. Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inference

    cs.CV 2025-05 conditional novelty 6.0 of 10

    DPQ is a post-training quantization algorithm that stores weights in INT4, computes in FP8, and uses Hessian-based group-aware reordering to keep accuracy near the full-precision model.

  9. Double-Scoring: Reliable Extraction of Strong Lottery Tickets

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Double-scoring replaces layerwise sparsity selection with fixed half-density masking over an augmented score tensor, yielding higher untrained-subnetwork accuracy but at uncontrolled effective sparsity.

  10. Neural FOXP2 -- Language Specific Neuron Steering for Targeted Language Improvement in LLMs

    cs.CL 2026-02 reject novelty 5.0 of 10

    A three-stage SAE-plus-SVD steering recipe claims to make Hindi or Spanish the default language of an LLM at inference time, but the visible manuscript reports only expected, not measured, outcomes.

  11. Sparse Neuron Ablation Triggers Catastrophic Collapse of the Language Core in Large Vision-Language Models

    cs.AI 2025-11 conditional novelty 5.0 of 10

    Ablating just four neurons in LLaVA-1.5-7b's language-model down-projection layer triggers complete output collapse, with critical neurons concentrated in the language backbone.

  12. Tensorization is a powerful but underexplored tool for compression and interpretability of neural networks

    cs.LG 2025-05 conditional novelty 4.0 of 10

    The paper makes the case that tensorized neural networks offer valuable compression, scaling, and interpretability advantages that the deep learning community has not yet fully exploited.

Pith tools