Pith. sign in

REVIEW 5 major objections 5 minor 3 cited by

Hardware-Efficient Attention for Fast Decoding

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Two attention designs cut KV cache memory in half with no quality loss and decode up to 2× faster.

desk verdict Solid systems paper with clean small-scale quality experiments; the serving-benchmark numbers come from a different, untrained configuration, so the 'practical replacement' claim outruns the evidence. read the letter →

arxiv 2505.21487 v1 pith:VOOFIOVD submitted 2025-05-27 cs.LG cs.CL

Ted Zadouri , Hubert Strauss , Tri Dao This is my paper · ORCID
classification cs.LGcs.CL
keywords attentionmechanismsKVcachecompressioninferenceefficiencytensorparallelismarithmeticintensitygroupedlatentgrouped-tiedspeculativedecoding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Decoding a large language model is bottlenecked by loading the cached key-value states of previous tokens from memory, not by computation. This paper argues attention should be redesigned so each byte loaded from the KV cache does more arithmetic, and proposes two variants that do so without hurting model quality. Grouped-Tied Attention (GTA) reuses a single tied state as both key and value within each group, cutting the KV cache roughly in half relative to Grouped-Query Attention (GQA) at the same quality. Grouped Latent Attention (GLA) splits the compressed latent KV representation into separate heads that shard across GPUs, matching Multi-head Latent Attention (MLA) quality while fetching a smaller per-device cache and running up to 2× faster in decoding and serving. If these results hold at scale, attention would no longer force a trade-off between quality and memory efficiency during generation.

What carries the argument

The load-bearing object is the arithmetic intensity of decoding attention, $I \approx 2 g_q / m_{kv}$ FLOPs per byte loaded, with group size $g_q = h_q/h_{kv}$ (queries per KV head) and $m_{kv}$ the multiplicity of key/value states ($m_{kv}=1$ tied, $m_{kv}=2$ separate). GTA raises $I$ by tying K and V into one shared state and rotating only a partial RoPE slice; GLA raises $I$ while preserving parallel sharding by using $h_c$ latent heads of dimension $2d_h$ that split across tensor-parallel ranks. The zero-redundancy condition $g_q \le h_q/N$ on $N$ shards is the constraint that keeps high intensity from collapsing into cache duplication.

What would settle it

Train a several-hundred-billion-parameter model with GLA and compare it against a matched MLA model on standard downstream benchmarks; a meaningful quality gap would falsify the parity claim, as would serving benchmarks where the GLA kernel is not faster than FlashMLA under the reported query lengths.

Watch

Extended reading notes

Core claim

The paper's central claim is that the decoding bottleneck is not computation but memory traffic: each token's attention loads a KV cache that grows with batch and context, and standard attention does only about one FLOP per byte fetched. By measuring arithmetic intensity, the paper shows that increasing the group size $g_q$ (query heads per KV head) raises intensity to roughly $2g_q/m_{kv}$ FLOPs per byte, where $m_{kv}=1$ when keys and values are tied and 2 when separate, and that this gain eventually collides with distributed parallelism: with $N$ tensor-parallel shards, the duplication factor is $D=\lceil N g_q / h_q\rceil$, and zero duplication requires $g_q \le h_q/N$. GTA is the tied-state version of GQA—one state serves as both key and value, with only a small separate RoPE head for position—roughly halving the KV cache at equal quality. GLA is a grouped version of MLA: latent heads of dimension $2d_h$ are sharded across devices so each GPU fetches a smaller cache, matching MLA quality at 1.47B parameters while decoding and serving up to 2× faster.

Load-bearing premise

The load-bearing premise is that the quality parity measured at up to 1.47B parameters on FineWeb-Edu continues to hold for 100B-plus models, the scale where MLA is actually deployed.

Editorial extensions

If this is right

  • GTA matches GQA quality with roughly half the KV cache, so switching from GQA to GTA buys longer contexts or larger batches at the same memory cost.
  • GLA matches MLA quality up to 1.47B parameters while halving the per-device KV cache under tensor parallelism of degree 2 or more, implying longer generations or larger batches before hitting memory limits.
  • In online serving with 64 concurrent requests on eight H100s, GLA-8 cut median end-to-end latency from 381 to 179 seconds versus MLA and raised throughput by about 70%.
  • In speculative decoding (query length 2), the GLA kernel is over 2× faster than FlashMLA, and at query length 1 it is about 20% faster.
  • The distributed offset calculation makes page-size-1 paged KV as fast as page-size-64, enabling prefix caching without a slowdown.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: If the quality parity holds at 100B+ scale, GLA could replace MLA in deployed long-context models, where the per-device cache duplication of MLA is currently mitigated by hybrid data parallelism that adds synchronization stalls.
  • Inference: The paper's framing implies that the next bottleneck after KV traffic will be compute at query length 2 or more; combining GLA with speculative decoding or multi-token prediction could compound the speedup.
  • Inference: A directly testable extension of the paper's logic is to swap GLA-8 into a Llama-4-style GQA-8 architecture at hundreds of billions of parameters; the paper states this comparison is open.
  • Inference: The partial-RoPE and tied-KV insight suggests a family of hybrids—tying only a fraction of layers, or combining GTA-style tying with post-hoc KV quantization—that the paper does not evaluate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper proposes two attention mechanisms for efficient LLM decoding: Grouped-Tied Attention (GTA), which ties key and value into a single state shared by a group of query heads with partial RoPE, and Grouped Latent Attention (GLA), which splits MLA's latent into h_c grouped latent heads of dimension 2d_h. The paper derives arithmetic-intensity estimates for these architectures, describes CUDA kernel optimizations (software pipelining, warp specialization, distributed offset calculation for paged KV), and reports: (i) quality comparisons on FineWeb-Edu at 183M–1.47B parameters showing GTA-4 roughly matching or improving on GQA-4 and GLA-2 roughly matching MLA; (ii) kernel benchmarks versus FlashMLA with up to 2x speedup at query length 2 and h_q=128; (iii) online serving benchmarks on a restructured DeepSeek-Coder-V2 model with randomly initialized GLA weights showing up to ~2x throughput/latency gains over MLA in some configurations. The central claim is that GTA is an efficient replacement for GQA and GLA is a practical replacement for MLA.

Significance. If the claims are substantiated, GLA would be a practically important extension of MLA: by grouping latent heads it avoids MLA's per-device duplication under tensor parallelism while preserving the low-KV-cache benefit, and GTA offers a simple tied-KV alternative to GQA. The paper's strengths include a clean arithmetic-intensity framework, parameter-matched quality experiments at several scales, and open-source kernel engineering with measurable optimizations (e.g., page-size-1 paged KV matching page-size-64 speed, Figure 6). The main risk is that the headline 'practical replacement' claim rests on splicing together quality results for GLA-2 (<=1.47B) and system results for an untrained GLA-8 configuration on a different model, with no configuration where both quality and speed are demonstrated.

major comments (5)
  1. [§5.1, §5.2–5.3, §7] Quality and system evaluations use disjoint configurations. The quality evidence for GLA is limited to GLA-2 (h_c=2, d_c=2d_h) on models up to 1.471B (Tables 2–5), while the kernel and serving benchmarks use GLA-8 (h_c=8, d_c=256; total latent 2048 bytes/token versus MLA's 512) on a restructured DeepSeek-Coder-V2 model with randomly initialized weights (Appendix B.6, Figure 4). No experiment shows that GLA-8 retains MLA-comparable quality at any scale. The conclusion in Section 7 that 'GLA is a practical replacement for MLA' therefore extrapolates across both configuration and scale. Please either train and evaluate GLA-8 at least at the 1.47B scale, or explicitly restrict the quality-parity claim to GLA-2 and present the system results as workload-level illustrations of an untrained configuration.
  2. [§5.3, Figure 4] Kernel benchmarks use a query-head count not covered by the quality experiments. Section 5.3 and Figure 4 report decoding speed for query heads h_q=128 and query lengths 1 and 2, while all quality-validated models use h_q=12–16 (Tables 6–10). Because the arithmetic intensity grows with h_q (Table 1), the claimed 1.2–2x kernel speedup is only established for h_q=128. The paper should report kernel latencies at the h_q values of the trained models, or provide a sweep over h_q showing the speedup persists at lower h_q.
  3. [Appendix B.6] Online-serving results are for randomly initialized GLA weights. Appendix B.6 states 'To simulate GLA, we restructure the MLA latent dimension to GLA with randomly initializing weights since we benchmark performance, not accuracy, in this phase.' While this is legitimate for a system-level bandwidth study, the title and abstract claim that 'GLA reduces end-to-end latency and increases token throughput by up to 2x' without flagging that the benchmarked model has never been trained. Please state this limitation in the main text alongside the speed claims and avoid using the serving numbers as direct evidence for the 'practical replacement' conclusion.
  4. [§5.2, Tables 27–32] The GLA-8 versus MLA serving comparison confounds grouping with total latent size. GLA-8 stores 8x256 = 2048 bytes/token of latent across the system, four times MLA's 512 bytes/token; per-device savings come from spreading this larger cache over TP ranks. Consequently Tables 27–32 do not isolate the effect of GLA's grouped-latent sharding from the effect of a larger aggregate cache. A comparison with a GLA variant of the same total latent budget (e.g., h_c=8, d_c=64) would separate these factors; please add such an ablation or soften the interpretation.
  5. [§5.1, Tables 2–5] Quality results lack error bars or multi-seed runs. The parity claims in Section 5.1 rely on small differences (e.g., 1.47B: FineWeb-Edu PPL 10.218 vs 10.256 and downstream 60.0% vs 59.1% for GLA-2 vs MLA, Table 5). Single training runs cannot distinguish these from noise, so statements such as 'GLA consistently matches or exceeds MLA accuracy on all model scales' (Section 7) are stronger than the data support. At minimum, acknowledge this in the limitations and use confidence-aware language.
minor comments (5)
  1. [Table 1] The column headers and arithmetic-intensity formulas are garbled in the formatted text (e.g., the GLA-2 and GLA columns are not clearly separated from the General Variant row); please reformat for readability.
  2. [§5.2, Figure 5, Table 35] Section 5.2 says GLA-8 achieves 'roughly 2.5x the MLA throughput', but Table 35 reports values implying 2.7x (100.68 vs 37.20 tokens/s); please align the numbers.
  3. [Table 31] The caption says both models run with 'eight-way tensor parallelism and four-way data parallel attention', but the configuration is TP=4, DP=2; the caption should match the actual parallelism scheme.
  4. [§3.3.2] The sentence 'we set the h_c = 2 for GLA' appears in the general methodology, but later experiments use GLA-8; clarify that this sentence refers to the quality experiments only.
  5. [Appendix B.6.2, Tables 29 and 31] Several parallelism descriptors are inconsistent: Table 29's caption says 'eight-way tensor parallelism and four-way data parallel attention' while the configuration is TP=2, DP=4; please audit all parallelism descriptors in the appendix.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: GTA and GLA are validated empirically against external baselines, and the paper's kernel/serving claims are benchmark measurements rather than derivations from the claim itself.

full rationale

The paper's central claims are empirical rather than definitional. Quality parity for GTA versus GQA and GLA versus MLA is established by training models from scratch on FineWeb-Edu and comparing against external baselines (Tables 2-5, 11-17), with parameter-count matching performed by widening MLPs as disclosed in Appendix B.1; no fitted parameter is renamed as a prediction, and no quality result is used as an input to the architecture definitions. The kernel and serving benchmarks are direct measurements: GLA kernels are compared against DeepSeek's FlashMLA, and the serving experiments use SGLang live-server mode with DeepSeek-Coder-V2 Base. The paper explicitly states that the GLA serving simulation uses randomly initialized weights 'since we benchmark performance, not accuracy, in this phase' (Appendix B.6), which is appropriate for throughput/latency comparisons because memory traffic is determined by the KV-cache layout rather than by the trained values. Self-citations to FlashAttention-3, the Mamba training recipe, and RedPajama are tooling or dataset choices and are not load-bearing for the architecture's validity; no uniqueness theorem, ansatz, or derived result is imported from the authors' prior work to force the conclusion. The main weakness is a scaling/validity gap, not circularity: quality is demonstrated only for GLA-2 up to 1.471B parameters, while the serving benchmarks use GLA-8 with random weights, and Section 6 explicitly defers 'evaluating larger-scale models with more latent heads.' This is an extrapolation concern and should be weighed as correctness risk, but it does not make the derivation circular.

Assumptions & free parameters 4 free parameters · 3 assumptions · 2 invented entities

The core architecture claims rest on a small number of empirical and scaling assumptions: the memory-bound decoding model, the low-rank and partial-RoPE priors for GTA, and the transfer of moderate-scale quality to production-scale models. Architecture hyperparameters are hand-picked to match MLA's cache size rather than fitted to data.

free parameters (4)
  • GLA number of latent heads h_c = 2
    Chosen so GLA's unsharded KV cache matches MLA's 4d_h and to demonstrate sharding with TP>=2; not fitted to data but a hand-picked design point (Section 3.3.2).
  • GLA latent head dimension d_c = 2d_h (e.g., 256 for d_h=128)
    Set to half of MLA's 4d_h so two heads match MLA's total latent dimension; hand-picked to equalize cache size.
  • GTA RoPE dimension (partial) = d_h/2 (32 or 64)
    Only half the key head is rotated, based on prior partial-RoPE findings; the unrotated half is tied to value. The split is a design choice, and its quality impact is ablated only partially (Table 19).
  • Serving benchmark GLA latent dimension for DeepSeek-Coder-V2 = 256 per head, 8 heads
    Used in throughput benchmarks; the GLA weights are randomly initialized to measure speed only, which assumes speed is weight-independent.
assumptions (3)
  • domain assumption Decoding latency is dominated by loading the KV cache from HBM, so arithmetic intensity is the right design lens.
    Used throughout Section 3.1 to motivate the architecture; supported by prior work (Recasens et al., 2025) and the paper's own roofline analysis.
  • domain assumption Key-value states are low-rank and partial RoPE preserves quality, so tying K and V is safe.
    Invoked in Section 3.3.1 to justify GTA, citing Saxena et al. 2024, Yu et al. 2024a, and Black et al. 2022; these are external empirical findings, not proven in this paper.
  • domain assumption Quality results at 1.47B parameters on FineWeb-Edu generalize to larger models and other data distributions.
    The paper concludes GLA is a practical replacement for MLA while only evaluating up to 1.47B; the scaling assumption is explicitly flagged as future work in Section 6.
invented entities (2)
  • GTA tied KV state
    purpose: A single cached state serves as both key and value for a group of query heads, cutting KV cache size.
    No independent evidence outside this paper; the architecture is evaluated only in the paper's own training runs.
  • GLA grouped latent heads
    purpose: Multiple low-rank latent heads (instead of MLA's single latent) allow TP sharding without duplicating the latent per device.
    Same as above; only in-paper evidence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hardware-Efficient Attention for Fast Decoding." pith.science (2026). https://pith.science/paper/VOOFIOVD

@misc{pith2026250521487,
  author       = {Pith},
  title        = {Pith review of: Hardware-Efficient Attention for Fast Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VOOFIOVD}},
  note         = {Machine review of arXiv:2505.21487}
}
abstract

LLM decoding is bottlenecked for large batches and long contexts by loading the key-value (KV) cache from high-bandwidth memory, which inflates per-token latency, while the sequential nature of decoding limits parallelism. We analyze the interplay among arithmetic intensity, parallelization, and model quality and question whether current architectures fully exploit modern hardware. This work redesigns attention to perform more computation per byte loaded from memory to maximize hardware efficiency without trading off parallel scalability. We first propose Grouped-Tied Attention (GTA), a simple variant that combines and reuses key and value states, reducing memory transfers without compromising model quality. We then introduce Grouped Latent Attention (GLA), a parallel-friendly latent attention paired with low-level optimizations for fast decoding while maintaining high model quality. Experiments show that GTA matches Grouped-Query Attention (GQA) quality while using roughly half the KV cache and that GLA matches Multi-head Latent Attention (MLA) and is easier to shard. Our optimized GLA kernel is up to 2$\times$ faster than FlashMLA, for example, in a speculative decoding setting when the query length exceeds one. Furthermore, by fetching a smaller KV cache per device, GLA reduces end-to-end latency and increases throughput in online serving benchmarks by up to 2$\times$.

Figures

Figures reproduced from arXiv: 2505.21487 by the authors.

Figure 1
Figure 1. Memory-loading schematics during decoding of MLA [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of Grouped-Tied Attention (GTA). A single projection produces a tied KV state that serves as both key and value. The full tied KV dimension is used as the value. For the keys, half of the key dimension comes from the tied KV vector (no positional encoding applied), and the other half comes from a separate single-head projection (where RoPE is applied); this separate half is broadcast to all heads in the gro… view at source ↗
Figure 3
Figure 3. Roofline analysis of BF16 decoding on a single H100 80GB SXM5. In this figure only, the numeric suffix (e.g., GQA-128) indicates the number of query heads ℎ𝑞; elsewhere in the paper, it denotes ℎ𝑘𝑣 . Left, 𝐿𝑞=1: With ℎ𝑞=128, MLA attains an arithmetic intensity of ∼2·ℎ𝑞=256, near the compute roof of ∼295 FLOPs/byte of H100, whereas GLA–128 with two latent heads remains on the I/O roof with arithmetic intensity of∼ℎ𝑞=… view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Left: Decoding speed of MLA and GLA on H100 80GB SMX5 GPU (theoretical max BF16 compute 989 TFLOPS/s and memory 3350 GB/s), for query length 1 where MLA is close to being bottlenecked by compute (reaching 610 TFLOPS/s) while GLA has not yet saturated compute (360 TFLOP…
Figure 5
Figure 5. Figure 5: Output throughput (higher is better) under live server benchmark. Left: For 16 concurrent requests for long-context prefill 32K/64K with 4K decode length, GLA-8 with TP=8 outperforms MLA with a hybrid of TP and DP across eight GPUs. Right: With 16 concurrent requests w…
Figure 6
Figure 6. Figure 6: Decoding speed of GLA on H100 80GB SMX5 GPU (theoretical max BF16 compute 989 TFLOPS/s and memory [PITH_FULL_IMAGE:figures/full_fig_p027_6.png]
Figure 7
Figure 7. Figure 7: Median end-to-end latency (left), lower is better, and output throughput (right), higher is better, of MLA and GLA-8 [PITH_FULL_IMAGE:figures/full_fig_p028_7.png]
Figure 8
Figure 8. Figure 8: Median end-to-end latency (left), lower is better, and output throughput (right), higher is better, of MLA and GLA [PITH_FULL_IMAGE:figures/full_fig_p029_8.png]
Figure 9
Figure 9. Figure 9: Mean service-level metrics for MLA and GLA on x8 GPU TP server; the table reports end-to-end latency, time [PITH_FULL_IMAGE:figures/full_fig_p031_9.png]
Figure 10
Figure 10. Figure 10: Token throughput at 64 concurrent requests (left) and 128 concurrent requests (right), where higher is better. [PITH_FULL_IMAGE:figures/full_fig_p031_10.png]
Figure 11
Figure 11. Figure 11: Median E2E latency at 64 concurrent request (left) and 128 concurrent request (right), where higher is better. [PITH_FULL_IMAGE:figures/full_fig_p032_11.png]
Figure 12
Figure 12. Figure 12: Median end-to-end latency (left), lower is better, and output throughput (right), higher is better, under TP [PITH_FULL_IMAGE:figures/full_fig_p033_12.png]
Figure 13
Figure 13. Figure 13: Median end-to-end latency (left), lower is better, and output throughput (right), higher is better, where the sequence [PITH_FULL_IMAGE:figures/full_fig_p034_13.png]
Figure 14
Figure 14. Figure 14: Demonstration of MLA and GLA for TP for degree of 8 on long decode tasks. With 256 number of prompts and [PITH_FULL_IMAGE:figures/full_fig_p035_14.png]
Figure 15
Figure 15. Figure 15: Left: Decoding speed of MLA and GLA on H100 80GB SMX5 GPU (theoretical max BF16 compute 989 TFLOPS/s and memory 3350 GB/s), for query length 2. At query length 2, GLA saturates compute (700 TFLOPS/s) and memory (3030 GB/s). Right: Peak BF16 theoretical peak FLOPs (TFL…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Think Before You Grid-Search: Floor-First Triage for LLM Serving

    cs.PF 2026-07 conditional novelty 6.0 of 10

    LLM serving should triage by five-resource analytical floors and wall ordering, not grid search; on 16×H20, TP16 is capacity-capped at ~70 while EP+DP attention reaches ~644 concurrent 8K requests.

  2. TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference

    cs.LG 2025-08 conditional novelty 5.0 of 10

    TPLA splits the latent KV cache across tensor-parallel GPUs while keeping every head's full view, yielding 1.79x and 1.93x decode speedups on DeepSeek-V3 and Kimi-K2 at 32K context with modest accuracy loss.

  3. SeerAttention-R: Sparse Attention Adaptation for Long Reasoning

    cs.LG 2025-06 conditional novelty 4.0 of 10

    A learned gate selects the important KV blocks during long decoding, preserving math reasoning accuracy while skipping up to 90% of attention work.

Reference graph

Works this paper leans on

85 extracted references · 17 canonical work pages · cited by 3 Pith papers

  1. [1]

    Gulavani, and Ramachandran Ramjee

    Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, and Ramachandran Ramjee. Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills, 2023. URL https://arxiv.org/abs/2308.16369

    arXiv 2023

  2. [2]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023. URL https://arxiv.org/abs/2305.13245

    arXiv 2023

  3. [3]

    Deepspeed inference: Enabling efficient inference of transformer models at unprecedented scale, 2022

    Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, and Yuxiong He. Deepspeed inference: Enabling efficient inference of transformer models at unprecedented scale, 2022. URL https://arxiv.org/abs/2207.00032

    arXiv 2022

  4. [4]

    How to scale your model

    Jacob Austin, Sholto Douglas, Roy Frostig, Anselm Levskaya, Charlie Chen, Sharad Vikram, Federico Lebron, Peter Choy, Vinay Ramasesh, Albert Webson, and Reiner Pope. How to scale your model. Online, 2025. Retrieved from https://jax-ml.github.io/scaling-book/

    2025

  5. [5]

    Round and round we go! what makes rotary positional encodings useful?, 2025

    Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. Round and round we go! what makes rotary positional encodings useful?, 2025. URL https://arxiv.org/abs/2410.06205

    arXiv 2025

  6. [6]

    Singe: Leveraging warp specialization for high performance on gpus

    Michael Bauer, Sean Treichler, and Alex Aiken. Singe: Leveraging warp specialization for high performance on gpus. In Proceedings of the 19th ACM SIGPLAN symposium on Principles and practice of parallel programming, pages 119--130, 2014

    2014

  7. [7]

    Cosmopedia, February 2024

    Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. Cosmopedia, February 2024. URL https://huggingface.co/datasets/HuggingFaceTB/cosmopedia

    2024

  8. [8]

    PIQA : reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA : reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, New York, NY, USA, February 7--12, 2020, pages 7432--7439. AAAI Press, 2020

    2020

Show all 85 references
  1. [9]

    Gpt-neox-20b: An open-source autoregressive language model, 2022

    Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. Gpt-neox-20b: An ope...

    2022 arXiv

  2. [10]

    Reducing transformer key-value cache size with cross-layer attention, 2024

    William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan Kelly. Reducing transformer key-value cache size with cross-layer attention, 2024. URL https://arxiv.org/abs/2405.12981

    2024 arXiv

  3. [11]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffr...

    2020 arXiv

  4. [12]

    Abdelfattah, and Kai-Chiang Wu

    Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S. Abdelfattah, and Kai-Chiang Wu. Palu: Compressing kv-cache with low-rank projection, 2024. URL https://arxiv.org/abs/2407.21118

    2024 arXiv

  5. [13]

    What rotary position embedding can tell us: Identifying query and key weights corresponding to basic syntactic or high-level semantic information

    Yiting Chen and Junchi Yan. What rotary position embedding can tell us: Identifying query and key weights corresponding to basic syntactic or high-level semantic information. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://ope...

    2024

  6. [14]

    Magicpig: Lsh sampling for efficient llm generation, 2024

    Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, Yuandong Tian, Matthijs Douze, Leon Bottou, Zhihao Jia, and Beidi Chen. Magicpig: Lsh sampling for efficient llm generation, 2024. URL https://arxiv.org/abs/2410.16179

    2024 arXiv

  7. [15]

    Flashattention-2: Faster attention with better parallelism and work partitioning, 2023

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023. URL https://arxiv.org/abs/2307.08691

    2023 arXiv

  8. [16]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, 2024

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, 2024. URL https://arxiv.org/abs/2405.21060

    2024 arXiv

  9. [17]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. URL https://arxiv.org/abs/2205.14135

    2022 arXiv

  10. [18]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024

    DeepSeek-AI. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024. URL https://arxiv.org/abs/2405.04434

    2024 arXiv

  11. [19]

    Deepseek-v3 technical report, 2025

    DeepSeek-AI. Deepseek-v3 technical report, 2025. URL https://arxiv.org/abs/2412.19437

    2025 arXiv

  12. [20]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025 arXiv

  13. [21]

    The pile: An 800gb dataset of diverse text for language modeling, 2020

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020. URL https://arxiv.org/abs/2101.00027

    2020 arXiv

  14. [22]

    Mahoney, and Kurt Keutzer

    Amir Gholami, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W. Mahoney, and Kurt Keutzer. Ai and memory wall, 2024. URL https://arxiv.org/abs/2403.14123

    2024 arXiv

  15. [23]

    Liu, Hasan Hassan, Kevin K

    Saugata Ghose, Abdullah Giray Yağlıkçı, Raghav Gupta, Donghyuk Lee, Kais Kudrolli, William X. Liu, Hasan Hassan, Kevin K. Chang, Niladrish Chatterjee, Aditya Agrawal, Mike O'Connor, and Onur Mutlu. What your dram power models are not telling you: Lessons from a detailed experi...

    2018 arXiv

  16. [24]

    Slim attention: cut your context memory in half without loss of accuracy -- k-cache is all you need for mha, 2025

    Nils Graef and Andrew Wasielewski. Slim attention: cut your context memory in half without loss of accuracy -- k-cache is all you need for mha, 2025. URL https://arxiv.org/abs/2503.05840

    2025 arXiv

  17. [25]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, and et. al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    2024 arXiv

  18. [26]

    Inc. Groq. What is a language processing unit?, 2024. URL https://groq.com/wp-content/uploads/2024/07/GroqThoughts_WhatIsALPU-vF.pdf. Groq white paper

    2024

  19. [27]

    Mamba: Linear-time sequence modeling with selective state spaces, 2024

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2024. URL https://arxiv.org/abs/2312.00752

    2024 arXiv

  20. [28]

    Fastdecode: High-throughput gpu-efficient llm serving using heterogeneous pipelines, 2024

    Jiaao He and Jidong Zhai. Fastdecode: High-throughput gpu-efficient llm serving using heterogeneous pipelines, 2024. URL https://arxiv.org/abs/2403.11421

    2024 arXiv

  21. [29]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3--7, 2021, 2021

    2021

  22. [30]

    Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length llm inference with kv cache quantization, 2024. URL https://arxiv.org/abs/2401.18079

    2024 arXiv

  23. [31]

    Multi-matrix factorization attention, 2025

    Jingcheng Hu, Houyi Li, Yinmin Zhang, Zili Wang, Shuigeng Zhou, Xiangyu Zhang, Heung-Yeung Shum, and Daxin Jiang. Multi-matrix factorization attention, 2025. URL https://arxiv.org/abs/2412.19255

    2025 arXiv

  24. [32]

    Data movement is all you need: A case study on optimizing transformers, 2021

    Andrei Ivanov, Nikoli Dryden, Tal Ben-Nun, Shigang Li, and Torsten Hoefler. Data movement is all you need: A case study on optimizing transformers, 2021. URL https://arxiv.org/abs/2007.00072

    2021 arXiv

  25. [33]

    Towards economical inference: Enabling deepseek's multi-head latent attention in any transformer-based llms, 2025

    Tao Ji, Bin Guo, Yuanbin Wu, Qipeng Guo, Lixing Shen, Zhan Chen, Xipeng Qiu, Qi Zhang, and Tao Gui. Towards economical inference: Enabling deepseek's multi-head latent attention in any transformer-based llms, 2025. URL https://arxiv.org/abs/2502.14837

    2025

  26. [34]

    Weight decay induces low-rank attention layers, 2024

    Seijin Kobayashi, Yassir Akram, and Johannes Von Oswald. Weight decay induces low-rank attention layers, 2024. URL https://arxiv.org/abs/2410.23819

    2024 arXiv

  27. [35]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023. URL https://arxiv.org/abs/2309.06180

    2023 arXiv

  28. [36]

    Software pipelining: An effective scheduling technique for vliw machines

    Monica Lam. Software pipelining: An effective scheduling technique for vliw machines. In Proceedings of the ACM SIGPLAN 1988 conference on Programming Language design and Implementation, pages 318--328, 1988

    1988

  29. [37]

    Flashmla: Efficient mla decoding kernels

    Jiashi Li. Flashmla: Efficient mla decoding kernels. https://github.com/deepseek-ai/FlashMLA, 2025

    2025

  30. [38]

    Pytorch distributed: Experiences on accelerating data parallel training, 2020

    Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala. Pytorch distributed: Experiences on accelerating data parallel training, 2020. URL https://arxiv.org/abs/2006.15704

    2020 arXiv

  31. [39]

    Sigma: Differential rescaling of query, key and value for efficient language models, 2025

    Zhenghao Lin, Zihao Tang, Xiao Liu, Yeyun Gong, Yi Cheng, Qi Chen, Hang Li, Ying Xin, Ziyue Yang, Kailai Yang, Yu Yan, Xiao Liang, Shuai Lu, Yiming Huang, Zheheng Luo, Lei Qu, Xuan Feng, Yaoxiang Wang, Yuqing Xia, Feiyang Chen, Yuting Jiang, Yasen Hu, Hao Ni, Binyang Li, Guosh...

    2025 arXiv

  32. [40]

    Decoupled weight decay regularization, 2019

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https://arxiv.org/abs/1711.05101

    2019 arXiv

  33. [41]

    Fineweb-edu: The finest collection of educational content, 2024

    Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: The finest collection of educational content, 2024. URL https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu. Accessed: [date]

    2024

  34. [42]

    Transmla: Multi-head latent attention is all you need, 2025

    Fanxu Meng, Zengwei Yao, and Muhan Zhang. Transmla: Multi-head latent attention is all you need, 2025. URL https://arxiv.org/abs/2502.07864

    2025 arXiv

  35. [43]

    The Llama 4 Herd: The Beginning of a New Era of Natively Multimodal AI Innovation , 2025

    Meta AI . The Llama 4 Herd: The Beginning of a New Era of Natively Multimodal AI Innovation , 2025. URL https://ai.meta.com/blog/llama-4-multimodal-intelligence/. Accessed 29 Apr 2025

    2025

  36. [44]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018

    2018

  37. [45]

    Orca: Progressive learning from complex explanation traces of gpt-4, 2023

    Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4, 2023. URL https://arxiv.org/abs/2306.02707

    2023 arXiv

  38. [46]

    Efficient large-scale language model training on gpu clusters using megatron-lm, 2021

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Anand Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model training on gpu clus...

    2021 arXiv

  39. [47]

    NVIDIA H100 tensor core gpu architecture

    NVIDIA. NVIDIA H100 tensor core gpu architecture. https://resources.nvidia.com/en-us-tensor-core/gtc22-whitepaper-hopper, 2022

    2022

  40. [48]

    NVIDIA Blackwell architecture technical brief

    NVIDIA. NVIDIA Blackwell architecture technical brief. https://resources.nvidia.com/en-us-blackwell-architecture, 2024

    2024

  41. [49]

    Nvlink, 2024

    NVIDIA Corporation . Nvlink, 2024. URL https://www.nvidia.com/en-us/data-center/nvlink/

    2024

  42. [50]

    Reducing shared memory footprint to leverage high throughput on tensor cores and its flexible api extension library, 2023

    Hiroyuki Ootomo and Rio Yokota. Reducing shared memory footprint to leverage high throughput on tensor cores and its flexible api extension library, 2023. URL https://arxiv.org/abs/2308.15152

    2023 arXiv

  43. [51]

    Openai o1 system card, 2024

    OpenAI. Openai o1 system card, 2024. URL https://arxiv.org/abs/2412.16720

    2024 arXiv

  44. [52]

    Efficiently scaling transformer inference, 2022

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference, 2022. URL https://arxiv.org/abs/2211.05102

    2022 arXiv

  45. [53]

    Recasens, Ferran Agullo, Yue Zhu, Chen Wang, Eun Kyung Lee, Olivier Tardieu, Jordi Torres, and Josep Ll

    Pol G. Recasens, Ferran Agullo, Yue Zhu, Chen Wang, Eun Kyung Lee, Olivier Tardieu, Jordi Torres, and Josep Ll. Berral. Mind the memory gap: Unveiling gpu bottlenecks in large-batch llm inference, 2025. URL https://arxiv.org/abs/2503.08311

    2025 arXiv

  46. [54]

    Winogrande : An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande : An adversarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligen...

    2020

  47. [55]

    Eigen attention: Attention in low-rank space for kv cache compression, 2024

    Utkarsh Saxena, Gobinda Saha, Sakshi Choudhary, and Kaushik Roy. Eigen attention: Attention in low-rank space for kv cache compression, 2024. URL https://arxiv.org/abs/2408.05646

    2024 arXiv

  48. [56]

    Flashattention-3: Fast and accurate attention with asynchrony and low-precision

    Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. Advances in Neural Information Processing Systems, 37: 0 68658--68685, 2024 a

    2024

  49. [57]

    Flashattention-3: Fast and accurate attention with asynchrony and low-precision, 2024 b

    Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision, 2024 b . URL https://arxiv.org/abs/2407.08608

    2024 arXiv

  50. [58]

    Fast transformer decoding: One write-head is all you need, 2019

    Noam Shazeer. Fast transformer decoding: One write-head is all you need, 2019. URL https://arxiv.org/abs/1911.02150

    2019 arXiv

  51. [59]

    Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with a si...

    2023 arXiv

  52. [60]

    Loki: Low-rank keys for efficient sparse attention, 2024

    Prajwal Singhania, Siddharth Singh, Shwai He, Soheil Feizi, and Abhinav Bhatele. Loki: Low-rank keys for efficient sparse attention, 2024. URL https://arxiv.org/abs/2406.02542

    2024 arXiv

  53. [61]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024. URL https://arxiv.org/abs/2408.03314

    2024 arXiv

  54. [62]

    Roformer: Enhanced transformer with rotary position embedding, 2023

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023. URL https://arxiv.org/abs/2104.09864

    2023 arXiv

  55. [63]

    Seesaw: High-throughput llm inference via model re-sharding, 2025

    Qidong Su, Wei Zhao, Xin Li, Muralidhar Andoorveedu, Chenhao Jiang, Zhanda Zhu, Kevin Song, Christina Giannoula, and Gennady Pekhimenko. Seesaw: High-throughput llm inference via model re-sharding, 2025. URL https://arxiv.org/abs/2503.06433

    2025 arXiv

  56. [64]

    Shadowkv: Kv cache in shadows for high-throughput long-context llm inference, 2024

    Hanshi Sun, Li-Wen Chang, Wenlei Bao, Size Zheng, Ningxin Zheng, Xin Liu, Harry Dong, Yuejie Chi, and Beidi Chen. Shadowkv: Kv cache in shadows for high-throughput long-context llm inference, 2024. URL https://arxiv.org/abs/2410.21465

    2024 arXiv

  57. [65]

    CUTLASS , January 2023

    Vijay Thakkar, Pradeep Ramani, Cris Cecka, Aniket Shivam, Honghao Lu, Ethan Yan, Jack Kosaian, Mark Hoemmen, Haicheng Wu, Andrew Kerr, Matt Nicely, Duane Merrill, Dustyn Blasig, Fengqi Qiao, Piotr Majcher, Paul Springer, Markus Hohnerbach, Jin Wang, and Manish Gupta. CUTLASS ,...

    2023

  58. [66]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  59. [67]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurip...

    2017

  60. [68]

    Gpt-j-6b: a 6 billion parameter autoregressive language model, 2021

    Ben Wang and Aran Komatsuzaki. Gpt-j-6b: a 6 billion parameter autoregressive language model, 2021

    2021

  61. [69]

    Redpajama: an open dataset for training large language models, 2024

    Maurice Weber, Daniel Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. Redpaja...

    2024 arXiv

  62. [70]

    Liu, and Matt Gardner

    Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions, 2017. URL https://arxiv.org/abs/1707.06209

    2017 arXiv

  63. [71]

    Roofline: an insightful visual performance model for multicore architectures

    Samuel Williams, Andrew Waterman, and David Patterson. Roofline: an insightful visual performance model for multicore architectures. Commun. ACM, 52 0 (4): 0 65–76, April 2009. ISSN 0001-0782. doi:10.1145/1498765.1498785. URL https://doi.org/10.1145/1498765.1498785

    2009

  64. [72]

    Longvideobench: A benchmark for long-context interleaved video-language understanding, 2024

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding, 2024. URL https://arxiv.org/abs/2407.15754

    2024 arXiv

  65. [73]

    Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation, 2023

    Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation, 2023. URL https://arxiv.org/abs/2203.16487

    2023 arXiv

  66. [74]

    Efficient streaming language models with attention sinks, 2024

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2024. URL https://arxiv.org/abs/2309.17453

    2024 arXiv

  67. [75]

    Quick and (not so) dirty: Unsupervised selection of justification sentences for multi-hop question answering

    Vikas Yadav, Steven Bethard, and Mihai Surdeanu. Quick and (not so) dirty: Unsupervised selection of justification sentences for multi-hop question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Met...

    2019

  68. [76]

    Rope to nope and back again: A new hybrid attention strategy, 2025

    Bowen Yang, Bharat Venkitesh, Dwarak Talupuru, Hangyu Lin, David Cairuz, Phil Blunsom, and Acyr Locatelli. Rope to nope and back again: A new hybrid attention strategy, 2025. URL https://arxiv.org/abs/2501.18795

    2025

  69. [77]

    Gated linear attention transformers with hardware-efficient training, 2024

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training, 2024. URL https://arxiv.org/abs/2312.06635

    2024 arXiv

  70. [78]

    Effectively compress kv heads for llm, 2024 a

    Hao Yu, Zelan Yang, Shen Li, Yong Li, and Jianxin Wu. Effectively compress kv heads for llm, 2024 a . URL https://arxiv.org/abs/2406.07056

    2024 arXiv

  71. [79]

    Affordable generative agents, 2024 b

    Yangbin Yu, Qin Zhang, Junyou Li, Qiang Fu, and Deheng Ye. Affordable generative agents, 2024 b . URL https://arxiv.org/abs/2402.02053

    2024 arXiv

  72. [80]

    Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. Native sparse attention: Hardware-aligned and natively trainable sparse attention...

    2025 arXiv

  73. [81]

    HellaSwag : Can a machine really finish your sentence? In Anna Korhonen, David R

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag : Can a machine really finish your sentence? In Anna Korhonen, David R. Traum, and Llu \' s M \` a rquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguist...

    2019 doi

  74. [82]

    Tensor product attention is all you need, 2025

    Yifan Zhang, Yifeng Liu, Huizhuo Yuan, Zhen Qin, Yang Yuan, Quanquan Gu, and Andrew Chi-Chih Yao. Tensor product attention is all you need, 2025. URL https://arxiv.org/abs/2501.06425

    2025

  75. [83]

    H _2 o: Heavy-hitter oracle for efficient generative inference of large language models, 2023

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H _2 o: Heavy-hitter oracle for efficient generative inference of large language models, 2023. URL https://a...

    2023 arXiv

  76. [84]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs, 2024 a . URL https://arxiv.org/abs...

    2024 arXiv

  77. [85]

    Sglang: Efficient execution of structured language model programs

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs. Advances in Neural Information Processing Systems, 37...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.