Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Tensor-parallel latent attention lets MLA-pretrained models decode with a sharded key–value cache and an all-reduce, delivering 1.79–1.93x throughput at 32K context without retraining.

desk verdict TPLA is a practical retraining-free recipe for tensor-parallel MLA decode with real speedups, but the softmax-slicing approximation that backs the near-lossless claim is asserted rather than derived, internally inconsistent as stated, and not what the implementation actually computes. read the letter →

arxiv 2508.15881 v2 pith:QLZYHMPR submitted 2025-08-21 cs.LG cs.AI

Xiaojuan Tang , Fanxu Meng , Pingzhi Tang , Yuxuan Wang , Di Yin , Xing Sun , Muhan Zhang This is my paper
classification cs.LGcs.AI
keywords tensorparallelismmulti-headlatentattention(MLA)KVcachecompressionprefill-decodeseparationorthogonalreparameterizationPCAHadamardtransformLLMinferenceacceleration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the key–value cache advantage of Multi-Head Latent Attention (MLA) can be made to survive tensor parallelism, which ordinary MLA fails at because every device must load the full latent vector. The proposed scheme, Tensor-Parallel Latent Attention (TPLA), splits the latent key–value representation and each head's input dimension across devices, computes attention on each shard, and combines the shards with an all-reduce. An orthogonal reparameterization—Hadamard for normalization, PCA for the attention logits—makes the local computations proportional to the global ones, so existing MLA-pretrained checkpoints such as DeepSeek-V3 and Kimi-K2 can be converted with little or no training. If the approximation holds, tensor-parallel decoding of these models runs 1.79x and 1.93x faster at a 32K context while staying within a few points of the original accuracy. Because TPLA is algebraically a grouped latent attention with replicated heads, it can ride on fused-attention kernels such as FlashAttention-3.

What carries the argument

The central object is Tensor-Parallel Latent Attention (TPLA): an orthogonal reparameterization U of the MLA weights (absorbed into the down/up projections) followed by splitting the latent KV vector cKV into shards across devices, each device computing softmax(Q_i (cKV_i U)^T) with all query heads replicated, then all-reducing the output. Two conditions carry the argument: Condition 1 requires the RMS norms of the shards to be proportional to the global norm (easy for Hadamard or PCA), and Condition 2 requires the shard-local query–key products to be proportional to the global logit (asserted via PCA eigenvalue proportions α and β). Prefill–decode separation keeps exact MLA in prefill and u

What would settle it

Measure the per-head approximation error directly: for a trained MLA model, compute exact MLA attention logits and TPLA shard-local logits on the same inputs, and find the head with the largest projection of its query onto the trailing PCA components of cKV. If that head's shard-local softmax differs from the global softmax by more than a small KL divergence (say >0.1) while high-variance-aligned heads stay close, Condition 2 fails. A simpler experiment: run the paper's MLA-to-TPLA conversion with g=3 shards on DeepSeek-V2-Lite and check whether WikiText-2 perplexity jumps far beyond the repor

Watch

Extended reading notes

Core claim

TPLA is an inference-time restructuring of MLA: absorb an orthogonal transform U into the MLA projections, split the latent cache cKV into two shards and each head's input dimension accordingly, let every device see all query heads but only its local shard of the latent, compute softmax locally, and all-reduce the attention output. The authors establish two conditions under which local computation equals global computation—the shard RMS norms must be proportional to the global norm (Condition 1), and the shard query–key products must be proportional to the global logit (Condition 2)—and they show how to satisfy them approximately via PCA (with α and β as variance fractions) or, for normaliza

Load-bearing premise

The load-bearing premise is that after a PCA rotation, each shard's local query–key product is proportional to the full query–key product (with the same scale constants for every token), so the softmax over a shard approximates the softmax over the whole latent vector; that proportionality is asserted from eigenvalue fractions, not proven.

Editorial extensions

If this is right

  • Existing MLA-pretrained models can be served with tensor-parallel decoding without retraining; with prefill–decode separation the conversion is training-free and keeps most of the original accuracy.
  • At 32K context, decode throughput rises 1.79x for DeepSeek-V3 and 1.93x for Kimi-K2 at maximum batch size, with per-device KV reads cut from 576 to 320 dimensions at TP=2.
  • TPLA preserves each head's access to the full latent representation, so it avoids the catastrophic drop seen when MLA is converted to grouped latent attention (e.g., WikiText-2 perplexity 6.31 to 2212 for GLA, versus 7.24 for TPLA).
  • The sharding pattern is a GLA special case with doubled heads, so FlashAttention-3 and similar optimized kernels apply to TPLA with no major framework changes.
  • Keeping exact MLA in the compute-bound prefill phase cuts time-to-first-token by up to 1.45x compared with running TPLA everywhere, while the decode phase still gets the memory savings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's Condition 2 is asserted through PCA eigenvalue fractions rather than derived; if a head's query aligns with the low-variance tail of the latent key–value distribution, the shard-local attention can silently depart from MLA, and this risk grows with more than two shards (the paper itself concedes g>2 as a limitation).
  • A hybrid reparameterization—Hadamard-style balancing for RMSNorm shards and a different orthogonal design that also balances query–key products—could plausibly extend TPLA beyond two groups, which the paper leaves open.
  • TPLA's prefill–decode separation is a general deployment principle: use the exact attention only where it buys accuracy (prompt processing) and a sharded approximation only where memory bandwidth dominates (autoregressive tokens); other low-rank attention variants could adopt the same split.
  • If the approximation proves stable across heads and contexts, tensor-parallel serving of any MLA checkpoint becomes a drop-in configuration change, which would change the practical trade-off between MLA and GQA-style models in distributed inference.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. TPLA modifies MLA-style inference to make it tensor-parallel friendly: the latent KV vector cKV and each attention head's input dimension are split across devices, attention is computed independently per shard, and outputs are combined with an all-reduce. To make the split approximately lossless, the paper proposes orthogonal reparameterizations (Hadamard or PCA) of the latent space before slicing, with scaling coefficients estimated from calibration data, and a prefill/decode separation that keeps standard MLA math during prefill while using TPLA during decode. Experiments on DeepSeek and Kimi-K2 checkpoints report 1.79x/1.93x decode speedups at 32K context and claim near-parity accuracy, mainly for the prefill-decode separated variant.

Significance. If the accuracy-preservation claim held, TPLA would be a practically valuable systems contribution: it would give MLA-style models a tensor-parallel decode path with a per-device KV cache of 320 dimensions versus MLA's 576 for DeepSeek-V3, at equal attention FLOPs and without retraining, while remaining compatible with FlashAttention-3. The paper's strengths include a clean algebraic derivation of matrix absorption, a correct Proposition 1 for RMSNorm under orthogonal transforms, an explicit complexity analysis, public code, and speed benchmarks on large-scale checkpoints. However, the near-losslessness claim is not fully established: plain training-free TPLA loses 13.7 points on LongBench for DeepSeek-V3 and 17.9 points for DeepSeek-V2-Lite, and the key softmax-slicing condition is asserted rather than derived. The central idea is promising, but the accuracy claim needs direct validation and reframing.

major comments (4)
  1. [§4.2, Eq. (22)] Condition 2 is the linchpin of the training-free accuracy claim, but it is asserted rather than derived. PCA on cKV controls the variance of the latent features, not the query-dependent bilinear forms Q'_i(cKVU)_i^T. A head whose query aligns with the low-variance tail can place significant logit mass in either shard, so the shard-local softmax can depart arbitrarily from the global softmax. The paper's own Hadamard counterexample in §4.3.1 demonstrates this failure for a balanced orthogonal transform. Please add a direct empirical test: for the deployed DeepSeek checkpoints, report per-head/per-layer cosine similarity between global MLA logits and shard-local logits, and the KL divergence between the global softmax and TPLA's scaled local softmax, on held-out contexts. Without such evidence, Table 2's larger drops under plain TPLA remain unexplained.
  2. [§4.2, Eq. (22) vs. §4.3.2] As written, Eq. (22) is internally overconstrained. Writing A=Q'(cKVU)^T and A_i=Q'_i(cKVU)_i^T, Condition 2 states A≈μA0≈νA1. Since A=A0+A1, this forces A1≈(μ−1)A0 and A0≈(ν−1)A1, hence (μ−1)(ν−1)≈1. With μ=α, ν=β and α+β≈1 from Eq. (25), this gives αβ≈1, impossible for nondegenerate α,β. If the intended relation is instead A0≈μA and A1≈νA, please state that explicitly and revise Eq. (22). Moreover, the actual implementation in Eq. (6) uses unscaled local softmax; no condition is stated or checked for that form. Clarify how α,β,μ,ν enter the deployed computation.
  3. [Table 2 / Abstract] The abstract's 'minimal accuracy degradation' is not supported by plain training-free TPLA: LongBench average drops from 58.19 to 44.52 for DeepSeek-V3 and from 28.90 to 10.98 for DeepSeek-V2-Lite. Near parity is achieved only by TPLA (PD-sep.), which uses unmodified MLA during prefill and TPLA only during decode. The paper should make PD-sep. the central accuracy claim, report plain TPLA's limitation explicitly, and soften the unqualified 'minimal' wording. The results in Table 2 are also single-run; error bars or multiple-seed evaluation would strengthen the comparison.
  4. [Eq. (25), §5.1] The scaling coefficients α,β,μ,ν are estimated from WikiText-2 latent caches, so the near-losslessness claim is data-dependent. No sensitivity analysis is provided for the calibration corpus, for layer-wise variation, or for the exact form of the latent features used. The limitation section concedes that the PCA-based partition fails for g>2, which further limits the generality of the claim. Please add a robustness study across calibration sets and report the eigenvalue-sum ranges; also fix the overlapping index ranges in Eq. (25) (the sums both include λ_{d/2}, so α+β>1 as written).
minor comments (5)
  1. [Throughout] Typos and formatting: 'archieves' (§2), 'comparsion' (§4.5), 'Throughout' in the Figure 3 caption, and a duplicated enumeration item '4)' in the Introduction. These should be corrected.
  2. [Figure 2] The color-and-texture legend is hard to read in print. Consider separating the 'norm only', 'softmax only', and 'both' results into three panels, and add error bars or at least state the number of evaluation runs.
  3. [§4.3.2, Eq. (25)] Please specify exactly how the PCA calibration is performed: per layer or shared, on cKV before or after RMSNorm, and how the 'excluding position features' exclusion is implemented in the collected tensor F. This matters for reproducing α and β.
  4. [§5.4.1] The throughput measurements use MoE-removed models and maximum batch sizes. State whether the reported speedups carry over to the full MoE models, and report GPU type, batch sizes, and whether the all-reduce cost is included. The current caption and text leave this underspecified.
  5. [References] FlashAttention-3 is mentioned as an implementation backend but is not cited. Add the appropriate reference, and consider citing the GLA paper [20] with its full title in the bibliography.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TPLA's approximation is empirically evaluated rather than derived from fitted inputs.

full rationale

The derivation chain is not circular. The core reparameterization (Proposition 1 and Eq. 21) is an exact algebraic identity under orthogonal transformations; Conditions 1 and 2 are stated as approximation conditions, not derived from the conclusion of near-losslessness. The scaling constants α, β, μ, ν are estimated from WikiText-2 latent caches (Eq. 25), but the paper's fidelity claims are evaluated on held-out commonsense and LongBench benchmarks, so these are calibration inputs rather than fitted predictions. The paper's own Hadamard counterexample (§4.3.1) and the g>2 limitation are explicit admissions of approximation error, not disguised inputs. The only self-citation, TransMLA [16], is used to extend TPLA to non-MLA checkpoints (LLaMA-2-7B) and is not load-bearing for the main DeepSeek/Kimi speedup claims. No step reduces by construction to its own inputs; the asserted Condition 2 is an unverified assumption, which is a correctness risk rather than a circular step.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new physical or mathematical entities are introduced; the replication factor g is a configuration choice over an existing attention structure. The paper's contribution is a rearrangement of existing matrices, projections, and calibration statistics.

free parameters (3)
  • alpha, beta (mu = nu = alpha, beta), per-layer RMSNorm and softmax shard scale factors = not reported numerically
    Computed from the ratio of PCA eigenvalues of WikiText-2 calibration caches (Eq. 25). They directly scale the shard-local RMSNorm (Eq. 19) and, by Condition 2, the softmax scores. The paper gives no numeric values, so they function as data-fitted constants rather than derived quantities.
  • per-layer orthogonal rotation U (PCA eigenvectors) = d x d matrix per layer, not shipped
    Eigenvectors of the covariance of calibration-set latent features (Section 4.3.2). This matrix is the mechanism that makes slicing approximately lossless; it is entirely data-derived and not provided with the paper.
  • Hadamard random sign diagonal D = random +/-1 entries
    Random diagonal to break deterministic structure of the Hadamard transform (Section 4.3.1). A stochastic choice, not fitted to data; this variant is not the one used for the headline results.
assumptions (5)
  • ad hoc to paper Softmax score factorization (Condition 2): after PCA, shard-local logits Q'0(cKV U0)^T and Q'1(cKV U1)^T are each proportional to the global Q'(cKV U)^T
    Stated in Section 4.2 and implemented with mu = nu = alpha, beta in Section 4.3.2. No proof from softmax structure; the paper's own example (Section 4.3.1) shows the analogous condition fails for Hadamard, and the limitation section concedes it fails for g > 2. The accuracy-preservation claim depends on this.
  • domain assumption Query-key inner-product variance is captured by the top PCA components of the latent features alone
    PCA is computed on the covariance of cKV (Section 4.3.2), while Condition 2 concerns Q cKV^T, where Q is head-specific and absorbs W_UK. The paper assumes head queries align with the dominant latent principal components; this is not stated or argued.
  • domain assumption alpha and beta are approximately fixed per-sample constants (Condition 1)
    Eqs. 18 and 19 treat alpha and beta as invariant to the input data distribution. PCA guarantees signal variance ratios only in expectation over the calibration distribution; per-sample equality is an approximation the experiments do not quantify.
  • domain assumption WikiText-2 is a representative calibration distribution for the deployed models
    Section 4.3.2 calibrates on WikiText-2; the rotation U and scale constants transfer to DeepSeek-V3 and Kimi-K2, which trained on different corpora. The paper shows no sensitivity analysis to the calibration set.
  • domain assumption MoE-removed models are adequate for measuring relative decode throughput
    Section 5.4.1 removes MoE layers 'for timing' and measures maximum-batch throughput on the attention-only core. The stated 1.79x and 1.93x speedups apply to this stripped setting, not to the full deployed models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference." pith.science (2026). https://pith.science/paper/QLZYHMPR

@misc{pith2026250815881,
  author       = {Pith},
  title        = {Pith review of: TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QLZYHMPR}},
  note         = {Machine review of arXiv:2508.15881}
}
read the original abstract

Multi-Head Latent Attention (MLA), introduced in DeepSeek-V2, compresses key-value states into a low-rank latent vector, caching only this vector to reduce memory. In tensor parallelism (TP), however, attention heads are computed across multiple devices, and each device must load the full cache, eroding the advantage of MLA over Grouped Query Attention (GQA). We propose Tensor-Parallel Latent Attention (TPLA): a scheme that partitions both the latent representation and each head's input dimension across devices, performs attention independently per shard, and then combines results with an all-reduce. TPLA preserves the benefits of a compressed KV cache while unlocking TP efficiency. Unlike Grouped Latent Attention (GLA), every head in TPLA still leverages the full latent representation, maintaining stronger representational capacity. TPLA is drop-in compatible with models pre-trained using MLA: it supports MLA-style prefilling and enables efficient tensor-parallel decoding without retraining. Applying simple orthogonal transforms -- e.g., the Hadamard transform or PCA -- before TP slicing further mitigates cross-shard interference, yielding minimal accuracy degradation. By reducing the per-device KV cache for DeepSeek-V3 and Kimi-K2, we achieve 1.79x and 1.93x speedups, respectively, at a 32K-token context length while maintaining performance on commonsense and LongBench benchmarks. TPLA can be implemented with FlashAttention-3, enabling practical end-to-end acceleration.

Figures

Figures reproduced from arXiv: 2508.15881 by the authors.

Figure 1
Figure 1. Comparison of MLA, GLA, and TPLA. In MLA, each device must load the entire KV cache. In GLA, each [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Accuracy across multiple benchmarks under different tensor-parallelism methods (indicated by colors) and [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Throughout (Decoding) comparing MLA and TPLA. [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Latency (TTFT) comparing TPLA and TPLA (pd sep.). [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Training-Memory Regression in MLA Sequence Parallelism: Why Megatron-Core Forbids Absorption, and LAGA -- a Communication-Efficient Fix

    cs.DC 2026-07 conditional novelty 6.0 of 10

    LAGA replaces MLA training's per-head K/V all-to-all with a latent all-gather and local up-projection, matching explicit-form memory while cutting collective communication ~1.98x.

  2. Think Before You Grid-Search: Floor-First Triage for LLM Serving

    cs.PF 2026-07 conditional novelty 6.0 of 10

    LLM serving should triage by five-resource analytical floors and wall ordering, not grid search; on 16×H20, TP16 is capacity-capped at ~70 while EP+DP attention reaches ~644 concurrent 8K requests.

Reference graph

Works this paper leans on

63 extracted references · 17 canonical work pages · cited by 2 Pith papers

  1. [1]

    Hello GPT-4o, 2024

    OpenAI. Hello GPT-4o, 2024

    2024

  2. [2]

    Claude 3.5 sonnet, 2024

    Anthropic. Claude 3.5 sonnet, 2024

    2024

  3. [3]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    arXiv 2024

  4. [4]

    Improving language understanding by generative pre-training

    Alec Radford. Improving language understanding by generative pre-training. 2018

    2018

  5. [5]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  6. [6]

    Palu: Compressing kv-cache with low-rank projection

    Chang Chi-Chih, Lin Wei-Cheng, Lin Chien-Yu, Chen Chong-Yan, Hu Yu-Fang, Wang Pei-Shuo, Huang Ning-Chi, Ceze Luis, Abdelfattah Mohamed, S., and Wu and, Kai-Chiang. Palu: Compressing kv-cache with low-rank projection. arXiv preprint arXiv:2407.21118, 2024

    arXiv 2024

  7. [7]

    xkv: Cross-layer svd for kv-cache compression

    Chang Chi-Chih, Lin Chien-Yu, Akhauri Yash, Lin Wei-Cheng, Wu Kai-Chiang, Ceze Luis, and Abdelfattah Mohamed, S. xkv: Cross-layer svd for kv-cache compression. arXiv preprint arXiv:2503.18893, 2025

    arXiv 2025

  8. [8]

    Transformers are multi-state rnns

    Oren Matanel, Hassid Michael, Yarden Nir, Adi Yossi, and Schwartz Roy. Transformers are multi-state rnns. arXiv preprint arXiv:2401.06104, 2024

    arXiv 2024

Show all 63 references
  1. [9]

    Kivi: A tuning-free asymmetric 2bit quantization for kv cache

    Liu Zirui, Yuan Jiayi, Jin Hongye, Zhong Shaochen, Xu Zhaozhuo, Braverman Vladimir, Chen Beidi, and Hu Xia. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750, 2024

    2024 arXiv

  2. [10]

    Ladder-residual: parallelism-aware architecture for accelerating large model inference with communication overlapping

    Zhang Muru, Mishra Mayank, Zhou Zhongzhu, Brandon William, Wang Jue, Kim Yoon, Ragan-Kelley Jonathan, Song Shuaiwen, Leon, Athiwaratkun Ben, and Dao Tri. Ladder-residual: parallelism-aware architecture for accelerating large model inference with communication overlapping. arXi...

    2025 arXiv

  3. [11]

    Spd: Sync-point drop for efficient tensor parallelism of large language models

    Kim Han-Byul, Hoang Duc, Kundu Arnav, Samragh Mohammad, and Cho Minsik. Spd: Sync-point drop for efficient tensor parallelism of large language models. arXiv preprint arXiv:2502.20727, 2025

    2025 arXiv

  4. [12]

    Tensor-parallelism with partially synchronized activations

    Lamprecht Itay, Karnieli Asaf, Hanani Yair, Giladi Niv, and Soudry Daniel. Tensor-parallelism with partially synchronized activations. arXiv preprint arXiv:2506.19645v1, 2025

    2025

  5. [13]

    Flash com- munication: Reducing tensor parallelization bottleneck for fast large language model inference

    Li Qingyuan, Zhang Bo, Ye Liang, Zhang Yifan, Wu Wei, Sun Yerui, Ma Lin, and Xie Yuchen. Flash com- munication: Reducing tensor parallelization bottleneck for fast large language model inference. arXiv preprint arXiv:2412.04964, 2024

    2024 arXiv

  6. [14]

    Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model

    Smith Shaden, Patwary Mostofa, Norick Brandon, LeGresley Patrick, Rajbhandari Samyam, Casper Jared, Liu Zhun, Prabhumoye Shrimai, Zerveas George, Korthikanti Vijay, Zhang Elton, Child Rewon, Aminabadi Reza, Yazdani, Bernauer Julie, Song Xia, Shoeybi Mohammad, He Yuxiong, Houst...

    2022 arXiv

  7. [15]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sang- hai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023. 14 Tensor Parallel Latent Attention

    2023 arXiv

  8. [16]

    Transmla: Multi-head latent attention is all you need

    Fanxu Meng, Pingzhi Tang, Zengwei Yao, and Muhan Zhang. Transmla: Multi-head latent attention is all you need. arXiv preprint arXiv:2502.07864, 2025

    2025 arXiv

  9. [17]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

    DeepSeek-AI. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. CoRR, abs/2405.04434, 2024

    2024 arXiv

  10. [18]

    Llama 3 model card, 2024

    AI@Meta. Llama 3 model card, 2024

    2024

  11. [19]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  12. [20]

    Hardware-efficient attention for fast decoding

    Zadouri Ted, Strauss Hubert, and Dao Tri. Hardware-efficient attention for fast decoding. arXiv preprint arXiv:2505.21487v1, 2025

    2025 arXiv

  13. [21]

    Model tells you what to discard: Adaptive kv cache compression for llms

    Ge Suyu, Zhang Yunan, Liu Liyuan, Zhang Minjia, Han Jiawei, and Gao Jianfeng. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801, 2023

    2023 arXiv

  14. [22]

    Dynamickv: Task-aware adaptive kv cache compression for long context llms

    Zhou Xiabin, Wang Wenbin, Zeng Minyan, Guo Jiaxian, Liu Xuebo, Shen Li, Zhang Min, and Ding Liang. Dynamickv: Task-aware adaptive kv cache compression for long context llms. arXiv preprint arXiv:2412.14838, 2024

    2024 arXiv

  15. [23]

    Compresskv: Semantic retrieval heads know what tokens are not important before generation

    Lin Xiaolin, Wang Jingcun, Kondrateva Olga, Shi Yiyu, Li Bing, and Zhang Grace, Li. Compresskv: Semantic retrieval heads know what tokens are not important before generation. arXiv preprint arXiv:2508.02401v1, 2025

    2025 arXiv

  16. [24]

    Snapkv: Llm knows what you are looking for before generation

    Li Yuhong, Huang Yingbing, Yang Bowen, Venkitesh Bharat, Locatelli Acyr, Ye Hanchen, Cai Tianle, Lewis Patrick, and Chen Deming. Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404.14469, 2024

    2024 arXiv

  17. [25]

    Lazyllm: Dynamic token pruning for efficient long context llm inference

    Fu Qichen, Cho Minsik, Merth Thomas, Mehta Sachin, Rastegari Mohammad, and Najibi Mahyar. Lazyllm: Dynamic token pruning for efficient long context llm inference. arXiv preprint arXiv:2407.14057, 2024

    2024 arXiv

  18. [26]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models

    Zhang Zhenyu, Sheng Ying, Zhou Tianyi, Chen Tianlong, Zheng Lianmin, Cai Ruisi, Song Zhao, Tian Yuandong, Ré Christopher, Barrett Clark, Wang Zhangyang, and Chen Beidi. H2o: Heavy-hitter oracle for efficient generative inference of large language models. arXiv preprint arXiv:2...

    2023 arXiv

  19. [27]

    Model tells you where to merge: Adaptive kv cache merging for llms on long-context tasks

    Wang Zheng, Jin Boxiao, Yu Zhongzhi, and Zhang Minjia. Model tells you where to merge: Adaptive kv cache merging for llms on long-context tasks. arXiv preprint arXiv:2407.08454, 2024

    2024 arXiv

  20. [28]

    Zsmerge: Zero-shot kv cache compression for memory-efficient long-context llms

    Liu Xin, Liu Pei, and Tang Guoming. Zsmerge: Zero-shot kv cache compression for memory-efficient long-context llms. arXiv preprint arXiv:2503.10714, 2025

    2025

  21. [29]

    Efficient long-context llm inference via kv cache clustering

    Hu Jie, Wang Shengnan, He Yutong, Gong Ping, Yi Jiawei, Zhang Juncheng, Bai Youhui, Chen Renhai, Zhang Gong, Li Cheng, and Yuan Kun. Efficient long-context llm inference via kv cache clustering. arXiv preprint arXiv:2506.11418, 2025

    2025

  22. [30]

    Kvsharer: Efficient inference via layer-wise dissimilar kv cache sharing

    Yang Yifei, Cao Zouying, Chen Qiguang, Qin Libo, Yang Dongjie, Zhao Hai, and Chen Zhi. Kvsharer: Efficient inference via layer-wise dissimilar kv cache sharing. arXiv preprint arXiv:2410.18517, 2024

    2024 arXiv

  23. [31]

    Inference-friendly models with mixattention.arXiv preprint arXiv:2409.15012, 2024

    Rajput Shashank, Sheng Ying, Owen Sean, and Chiley Vitaliy. Inference-friendly models with mixattention.arXiv preprint arXiv:2409.15012, 2024

    2024 arXiv

  24. [32]

    Layer-condensed kv cache for efficient inference of large language models

    Wu Haoyi and Tu Kewei. Layer-condensed kv cache for efficient inference of large language models. arXiv preprint arXiv:2405.10637, 2024

    2024 arXiv

  25. [33]

    A systematic study of cross-layer kv sharing for efficient llm inference

    Wu You, Wu Haoyi, and Tu Kewei. A systematic study of cross-layer kv sharing for efficient llm inference. arXiv preprint arXiv:2410.14442, 2024

    2024 arXiv

  26. [34]

    Reducing transformer key-value cache size with cross-layer attention

    Brandon William, Mishra Mayank, Nrusimha Aniruddha, Panda Rameswar, and Kelly Jonathan, Ragan. Reducing transformer key-value cache size with cross-layer attention. arXiv preprint arXiv:2405.12981, 2024

    2024 arXiv

  27. [35]

    Lorc: Low-rank compression for llms kv cache with a progressive compression strategy

    Zhang Rongzhi, Wang Kuang, Liu Liyuan, Wang Shuohang, Cheng Hao, Zhang Chao, and Shen Yelong. Lorc: Low-rank compression for llms kv cache with a progressive compression strategy. arXiv preprint arXiv:2410.03111, 2024

    2024 arXiv

  28. [36]

    Matryoshkakv: Adaptive kv compression via trainable orthogonal projection

    Lin Bokai, Zeng Zihao, Xiao Zipeng, Kou Siqi, Hou Tianqi, Gao Xiaofeng, Zhang Hao, and Deng Zhijie. Matryoshkakv: Adaptive kv compression via trainable orthogonal projection. arXiv preprint arXiv:2410.14731, 2024

    2024 arXiv

  29. [37]

    Effectively compress kv heads for llm.arXiv preprint arXiv:2406.07056, 2024

    Yu Hao, Yang Zelan, Li Shen, Li Yong, and Wu Jianxin. Effectively compress kv heads for llm.arXiv preprint arXiv:2406.07056, 2024

    2024 arXiv

  30. [38]

    Kvquant: Towards 10 million context length llm inference with kv cache quantization

    Hooper Coleman, Kim Sehoon, Mohammadzadeh Hiva, Mahoney Michael, W., Shao Yakun, Sophia, Keutzer Kurt, and Gholami Amir. Kvquant: Towards 10 million context length llm inference with kv cache quantization. arXiv preprint arXiv:2401.18079, 2024. 15 Tensor Parallel Latent Attention

    2024 arXiv

  31. [39]

    Million: Mastering long-context llm inference via outlier-immunized kv product quantization

    Wang Zongwu, Xu Peng, Liu Fangxin, Hu Yiwei, Sun Qingxiao, Li Gezi, Li Cheng, Wang Xuan, Jiang Li, and Guan Haibing. Million: Mastering long-context llm inference via outlier-immunized kv product quantization. arXiv preprint arXiv:2504.03661, 2025

    2025 arXiv

  32. [40]

    Qaq: Quality adaptive quantization for llm kv cache

    Dong Shichen, Cheng Wen, Qin Jiayu, and Wang Wei. Qaq: Quality adaptive quantization for llm kv cache. arXiv preprint arXiv:2403.04643, 2024

    2024 arXiv

  33. [41]

    Tailorkv: A hybrid framework for long-context inference via tailored kv cache optimization

    Yao Dingyu, Shen Bowen, Lin Zheng, Liu Wei, Luan Jian, Wang Bin, and Wang Weiping. Tailorkv: A hybrid framework for long-context inference via tailored kv cache optimization. arXiv preprint arXiv:2505.19586, 2025

    2025 arXiv

  34. [42]

    Large scale distributed deep networks

    Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc’aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, et al. Large scale distributed deep networks. Advances in neural information processing systems, 25, 2012

    2012

  35. [43]

    Horovod: fast and easy distributed deep learning in tensorflow

    Alexander Sergeev and Mike Del Balso. Horovod: fast and easy distributed deep learning in tensorflow. arXiv preprint arXiv:1802.05799, 2018

    2018 arXiv

  36. [44]

    Gpipe: Efficient training of giant neural networks using pipeline parallelism

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019

    2019

  37. [45]

    Pipedream: Generalized pipeline parallelism for dnn training

    Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. In Proceedings of the 27th ACM symposium on operating systems principles, p...

    2019

  38. [46]

    Megatron-lm: Training multi-billion parameter language models using model parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  39. [47]

    An efficient 2d method for training super-large deep learning models

    Xu Qifan, Li Shenggui, Gong Chaoyu, and You Yang. An efficient 2d method for training super-large deep learning models. arXiv preprint arXiv:2104.05343, 2021

    2021 arXiv

  40. [48]

    Maximizing parallelism in distributed training for huge neural networks

    Bian Zhengda, Xu Qifan, Wang Boxiang, and You Yang. Maximizing parallelism in distributed training for huge neural networks. arXiv preprint arXiv:2105.14450, 2021

    2021 arXiv

  41. [49]

    Sequence parallelism: Long sequence training from system perspective

    Li Shenggui, Xue Fuzhao, Baranwal Chaitanya, Li Yongbin, and You Yang. Sequence parallelism: Long sequence training from system perspective. arXiv preprint arXiv:2105.13120, 2021

    2021 arXiv

  42. [50]

    NVIDIA dynamo, a low-latency distributed inference framework for scaling reasoning ai models

    Amr Elmeleegy, Harry Kim, David Zier, Kyle Kranen, Neelay Shah, Ryan Olson, and Omri Kahalon. NVIDIA dynamo, a low-latency distributed inference framework for scaling reasoning ai models. NVIDIA Developer Blog, March 2025. Published March 18, 2025

    2025

  43. [51]

    Sandwich: Separating prefill-decode compilation for efficient cpu llm serving

    Juntao Zhao, Jiuru Li, and Chuan Wu. Sandwich: Separating prefill-decode compilation for efficient cpu llm serving. arXiv preprint arXiv:2507.18454, 2025

    2025 arXiv

  44. [52]

    {DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. {DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2...

    2024

  45. [53]

    Deepseek LLM: scaling open-source language models with longtermism

    DeepSeek-AI. Deepseek LLM: scaling open-source language models with longtermism. CoRR, abs/2401.02954, 2024

    2024 arXiv

  46. [54]

    Kimi k2: Open agentic intelligence

    Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025

    2025 arXiv

  47. [55]

    Mea- suring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mea- suring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRevie...

    2021

  48. [56]

    Think you have solved question answering? try arc, the AI2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457, 2018

    2018 arXiv

  49. [57]

    PIQA: reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelli...

    2020

  50. [58]

    Hellaswag: Can a machine really finish your sentence? In Anna Korhonen, David R

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Anna Korhonen, David R. Traum, and Lluís Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL ...

    2019

  51. [59]

    Can a suit of armor conduct electricity? A new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, Proceedings of the 2018 Conference on Empiri...

    2018

  52. [60]

    Winogrande: an adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99–106, 2021

    2021

  53. [61]

    Pointer sentinel mixture models, 2016

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016

    2016

  54. [62]

    Smollm-corpus

    Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. Smollm-corpus. 2024

    2024

  55. [63]

    Longbench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508, 2023. 17

    2023 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.