Pith. sign in

REVIEW 10 cited by

Not All Layers of LLMs Are Necessary During Inference

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.02181 v3 pith:BFJHWNJT submitted 2024-03-04 cs.CL cs.AIcs.LG

Siqi Fan , Xin Jiang , Xiang Li , Xuying Meng , Peng Han , Shuo Shang , Aixin Sun , Yequan Wang
show 1 more author
Zhongyuan Wang
This is my paper
classification cs.CLcs.AIcs.LG
keywords llmsadainferinferencelayersresultstasksachieveduring
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Due to the large number of parameters, the inference phase of Large Language Models (LLMs) is resource-intensive. However, not all requests posed to LLMs are equally difficult to handle. Through analysis, we show that for some tasks, LLMs can achieve results comparable to the final output at some intermediate layers. That is, not all layers of LLMs are necessary during inference. If we can predict at which layer the inferred results match the final results (produced by evaluating all layers), we could significantly reduce the inference cost. To this end, we propose a simple yet effective algorithm named AdaInfer to adaptively terminate the inference process for an input instance. AdaInfer relies on easily obtainable statistical features and classic classifiers like SVM. Experiments on well-known LLMs like the Llama2 series and OPT, show that AdaInfer can achieve an average of 17.8% pruning ratio, and up to 43% on sentiment tasks, with nearly no performance drop (<1%). Because AdaInfer does not alter LLM parameters, the LLMs incorporated with AdaInfer maintain generalizability across tasks.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning

    cs.CV 2026-03 conditional novelty 7.0 of 10

    A small tool-free MLLM plus answer-separability gating bypasses agentic tool loops for many queries, yielding 1.1–3.35× speedup with preserved or higher accuracy.

  2. PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning

    cs.MM 2025-07 conditional novelty 6.0 of 10

    Keeping only the first 12 layers of Qwen2-VL plus self-distillation and a modality-aware contrastive loss yields a 3B unified multimodal retriever within 1.8 points of the 7B model on M-BEIR.

  3. Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exits

    cs.RO 2025-05 conditional novelty 6.0 of 10

    Edge-cloud speculative decoding runs faster when early exits in the server model let the client pre-draft the next candidate tokens before final verification is complete.

  4. BiggerGait: Unlocking Gait Recognition with Layer-wise Representations from Large Vision Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Combining features from intermediate layers of large vision models improves gait recognition accuracy, and the proposed BiggerGait baseline achieves state-of-the-art results on CCPG and cross-domain benchmarks.

  5. TRACE for Tracking the Emergence of Semantic Representations in Transformers

    cs.CL 2025-05 reject novelty 6.0 of 10

    Using Hessian curvature, intrinsic dimensionality, and linguistic probes on a synthetic frame-semantic corpus, the paper claims a coordinated intersection-based phase transition in small transformers, though the marke...

  6. Learning to Skip the Middle Layers of Transformers

    cs.LG 2025-06 conditional novelty 5.0 of 10

    A gated 'skip the middle' Transformer fails to beat fewer-layer dense baselines on the cross-entropy/FLOPs trade-off at 12 layers and 10B tokens.

  7. DASH: Input-Aware Dynamic Layer Skipping for Efficient LLM Inference with Markov Decision Policies

    cs.CL 2025-05 conditional novelty 5.0 of 10

    DASH trains a token-level policy to skip, quantize, or fully run each Transformer layer, reporting 1.33x-2.0x compute reduction with smaller accuracy loss than prior skipping methods.

  8. The Hard Decision Layer: Evidence for Committed Inference in Transformers

    cs.AI 2026-05 conditional novelty 4.0 of 10

    In multiple-choice QA, each tested transformer shows a single layer where the final answer's rank suddenly collapses to near-final; the layer's position is roughly stable across datasets, LoRA fine-tuning, and option counts.

  9. SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decoupling

    cs.CL 2025-06 conditional novelty 4.0 of 10

    SkipGPT learns per-token, per-module routers that dynamically skip attention and MLP blocks, then uses LoRA to recover most of the original model's quality.

  10. DLP: Dynamic Layerwise Pruning in Large Language Models

    cs.CL 2025-05 conditional novelty 4.0 of 10

    DLP assigns each LLM layer a sparsity rate derived from the median of Wanda-style weight-activation scores, improving perplexity and zero-shot accuracy at high sparsity versus uniform and outlier-based layerwise pruning.

Pith tools