Pith. sign in

REVIEW 19 cited by

The Unreasonable Ineffectiveness of the Deeper Layers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.17887 v2 pith:64RTIIKS submitted 2024-03-26 cs.CL cs.LGstat.ML

Andrey Gromov , Kushal Tirumala , Hassan Shapourian , Paolo Glorioso , Daniel A. Roberts This is my paper · ORCID
classification cs.CLcs.LGstat.ML
keywords layersknowledgelayercommondeeperfindfinetuningmethods
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

How is knowledge stored in an LLM's weights? We study this via layer pruning: if removing a certain layer does not affect model performance in common question-answering benchmarks, then the weights in that layer are not necessary for storing the knowledge needed to answer those questions. To find these unnecessary parameters, we identify the optimal block of layers to prune by considering similarity across layers; then, to "heal" the damage, we perform a small amount of finetuning. Surprisingly, with this method we find minimal degradation of performance until after a large fraction (up to half) of the layers are removed for some common open-weight models. From a scientific perspective, the robustness of these LLMs to the deletion of layers implies either that current pretraining methods are not properly leveraging the parameters in the deeper layers of the network or that the shallow layers play a critical role in storing knowledge. For our study, we use parameter-efficient finetuning (PEFT) methods, specifically quantization and Low Rank Adapters (QLoRA), such that each of our experiments can be performed on a single 40GB A100 GPU.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 19 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Bekko Embedding: Parameter-Efficient Multilingual Retrieval with Ultra-Compact Encoders

    cs.IR 2026-07 conditional novelty 7.0 of 10

    Bekko a8m, with 7.7M active parameters, scores 56.2 on MMTEB Multilingual v2 Retrieval, beating mE5 models and BGE-M3, while a25m reaches 57.5, on par with gte-multilingual-base.

  2. Wrong Before Right: Late Rescue and Interface Failure in Aligned Language Models

    cs.CL 2026-07 conditional novelty 7.0 of 10

    Aligned LMs transiently commit to wrong mid-layer preferences that late layers rescue; this wrong-dip predicts structural compression flips, is recipe-specific and trainable, and is distinct from interface failure.

  3. Rethinking Depth Pruning for Vision Transformers: A Heterogeneity-Aware Perspective

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Heterogeneity-aware depth pruning of attention and activation layers, guided by a polynomial model-accuracy predictor, delivers up to 1.58× speedup on DeiT-B and 5.19× when combined with width pruning.

  4. Latent Communication Between Language Model Agents: Channels, Alignment, and the Limits of Text

    cs.CL 2026-05 conditional novelty 6.0 of 10

    Latent communication between LLM agents preserves far more SAE features than text, but those extra features encode surface form and provide no task-level advantage on text-expressible tasks.

  5. When Does Sparsity Mitigate the Curse of Depth in LLMs

    cs.CL 2026-03 conditional novelty 6.0 of 10

    Implicit and explicit sparsity reduce residual-stream variance and improve layer effectiveness metrics, enabling a depth-scaling recipe with about 4.6 points higher downstream accuracy.

  6. Inverse Depth Scaling From Most Layers Being Similar

    cs.LG 2026-02 conditional novelty 6.0 of 10

    LLM loss decreases roughly inversely with depth because most layers act as a redundant ensemble that averages errors, not as a compositional hierarchy.

  7. Amber Pruner: Leveraging N:M Activation Sparsity for Efficient Prefill in Large Language Models

    cs.LG 2025-08 unverdicted novelty 6.0 of 10

    Amber Pruner proposes training-free N:M activation sparsity for LLM prefill; however, the supplied manuscript body is an unrelated paper.

  8. PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning

    cs.MM 2025-07 conditional novelty 6.0 of 10

    Keeping only the first 12 layers of Qwen2-VL plus self-distillation and a modality-aware contrastive loss yields a 3B unified multimodal retriever within 1.8 points of the 7B model on M-BEIR.

  9. Towards Distributed Neural Architectures

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A general routing framework lets each token take its own path through a network, matching dense baselines while enabling learned compute savings and interpretable path specialization.

  10. GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling

    cs.LG 2025-06 conditional novelty 6.0 of 10

    GPAS scales down intermediate activations while preserving backward gradients, reducing activation variance growth in Pre-LN transformers and improving pretraining convergence and downstream performance.

  11. GPTailor: Large Language Model Pruning Through Layer Cutting and Stitching

    cs.CL 2025-06 conditional novelty 6.0 of 10

    GPTailor searches over layer removal, layer selection, and layer merging across fine-tuned model variants to produce smaller LLMs that retain more benchmark performance than single-model pruning.

  12. MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Dynamic multiway dense connections across Transformer layers let a 2.8B model match a 6.9B model's pretraining quality and few-shot performance.

  13. CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield

    cs.CL 2026-06 unverdicted novelty 5.0 of 10

    Selective-pivot-token training plus layer-averaging-with-recurrence reportedly gives 2.5x parameter compression on a small Korean LLM, but the efficiency claim lacks its decisive controls and the abstract advertises r...

  14. A Hamiltonian-Inspired Local-Operator Ansatz for Slimming Large Language Models

    cs.CL 2026-05 unverdicted novelty 5.0 of 10

    Tensor Mixture (MixT) compresses transformer linear layers into local tensor sums that execute directly, preserving MMLU accuracy up to a sharp per-model boundary and cutting LLaMA2-7B parameters by 47.5%.

  15. Short-LVLM: Compressing and Accelerating Large Vision-Language Models by Pruning Redundant Layers

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A training-free layer pruning framework for large vision-language models, combining token importance scoring with subspace-compensated weight projection, preserves most accuracy while speeding inference.

  16. A Survey on Latent Reasoning

    cs.CL 2025-07 conditional novelty 4.0 of 10

    A survey that organizes latent reasoning methods into vertical recurrence, horizontal recurrence, and infinite-depth diffusion, arguing that silent reasoning can beat explicit chain-of-thought.

  17. SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decoupling

    cs.CL 2025-06 conditional novelty 4.0 of 10

    SkipGPT learns per-token, per-module routers that dynamically skip attention and MLP blocks, then uses LoRA to recover most of the original model's quality.

  18. Leveraging Stochastic Depth Training for Adaptive Inference

    cs.LG 2025-05 conditional novelty 4.0 of 10

    Stochastic-depth-trained ResNets can skip many residual blocks at inference with small accuracy loss, and sensitivity-based selection of which blocks to skip enables zero-overhead adaptive inference.

  19. Void in Language Models

    cs.CL 2025-05 conditional novelty 4.0 of 10

    A non-trainable L2-norm thresholding method detects "void" layers in instruction-tuned LLMs, and skipping these layers sometimes improves accuracy while reducing layer usage.

Pith tools