Pith. sign in

REVIEW 11 cited by

You Only Cache Once: Decoder-Decoder Architectures for Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.05254 v2 pith:P7XRNOTI submitted 2024-05-08 cs.CL

Yutao Sun , Li Dong , Yi Zhu , Shaohan Huang , Wenhui Wang , Shuming Ma , Quanlu Zhang , Jianyong Wang
show 1 more author
Furu Wei
This is my paper · ORCID
classification cs.CL
keywords yococachesmodelonceonlycontextcross-decoderdecoder-decoder
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We introduce a decoder-decoder architecture, YOCO, for large language models, which only caches key-value pairs once. It consists of two components, i.e., a cross-decoder stacked upon a self-decoder. The self-decoder efficiently encodes global key-value (KV) caches that are reused by the cross-decoder via cross-attention. The overall model behaves like a decoder-only Transformer, although YOCO only caches once. The design substantially reduces GPU memory demands, yet retains global attention capability. Additionally, the computation flow enables prefilling to early exit without changing the final output, thereby significantly speeding up the prefill stage. Experimental results demonstrate that YOCO achieves favorable performance compared to Transformer in various settings of scaling up model size and number of training tokens. We also extend YOCO to 1M context length with near-perfect needle retrieval accuracy. The profiling results show that YOCO improves inference memory, prefill latency, and throughput by orders of magnitude across context lengths and model sizes. Code is available at https://aka.ms/YOCO.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers

    cs.LG 2026-07 conditional novelty 7.0 of 10

    In looped transformers, the recurrence-indexed KV cache is low-rank along the loop axis, and a latent codec fitted to teacher activations compresses it up to 21× with small quality loss.

  2. Understanding Is Done Early: A Depth Division of Labor in Large Language Models and Its Use for Unbounded-Context Memory

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Long-context memory can be organized by caching intermediate residual states and recomputing only query-conditioned upper layers over a bounded retrieved pack.

  3. What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents

    cs.LG 2026-07 conditional novelty 6.0 of 10

    KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.

  4. Do Value Vectors in Deep Layers Need Context from the Residual Stream?

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    Deep transformer layers can replace context-dependent value vectors with per-token lookup tables (Bank of Values), improving validation loss and the 21-benchmark average at 135M–780M while cutting FLOPs and the value cache.

  5. CaliDrop: KV Cache Compression with Calibration

    cs.CL 2025-07 conditional novelty 6.0 of 10

    CaliDrop adds a stale-query calibration term on top of token eviction, improving accuracy at high KV compression ratios with modest throughput overhead.

  6. IAM: Efficient Inference through Attention Mapping between Different-scale LLMs

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Attention matrices of same-series small and large LLMs are similar enough that replacing up to 50% of a large model's attention layers with the small model's matrices preserves most performance while reducing KV cache...

  7. KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embedding

    cs.CL 2025-07 conditional novelty 6.0 of 10

    By downsampling key and value head dimensions and retraining with distillation, KV-Latent cuts KV cache memory by about 50% on 7-8B LLMs while keeping average benchmark scores within about 1 point of the base model.

  8. SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

    cs.LG 2026-06 conditional novelty 5.0 of 10

    A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.

  9. DAC: A Dynamic Attention-aware Approach for Task-Agnostic Prompt Compression

    cs.CL 2025-07 conditional novelty 5.0 of 10

    A dynamic attention-aware metric for task-agnostic prompt compression that combines entropy and attention scores, with iterative entropy recalculation, outperforms prior entropy-based compressors on multiple benchmarks.

  10. Position: Episodic Memory is the Missing Piece for Long-Term LLM Agents

    cs.AI 2025-02 conditional novelty 5.0 of 10

    The authors propose episodic memory, with five defining properties, as the unifying framework needed for LLM agents to learn and remember over long time horizons.

  11. ECHO-LLaMA: Efficient Caching for High-Performance LLaMA Training

    cs.LG 2025-05 reject novelty 4.0 of 10

    Shared KV caching across the last 25% to 50% of layers, adapted incrementally from pretrained LLaMA weights, is claimed to improve training speed and inference throughput with little accuracy loss.

Pith tools