Pith. sign in

REVIEW 2 major objections 4 minor 89 references

By compressing chunks of past tokens, a single transformer can change its compute-memory budget at test time and still match or beat a field of efficient architectures on long-context recall.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 23:28 UTC pith:6ZM7GQB5

load-bearing objection CAT is a genuinely interesting architecture with broad experiments, but the training-mask positional mismatch and the parameter mismatch need addressing before the results can be taken at face value. the 2 major comments →

arxiv 2511.05313 v2 pith:6ZM7GQB5 submitted 2025-11-07 cs.LG

Controllably Efficient Language Models

Jatin Prakash , Aahlad Puli , Rajesh Ranganath This is my paper
classification cs.LG
keywords efficient language modelschunked compressionlong-context recalltest-time efficiency controlKV cache reductiondense attentiontransformer inferenceadaptive architectures
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

CAT (Compress & Attend Transformer) splits a sequence into chunks of size C, compresses each chunk into a single vector with a small bidirectional transformer, and has a causal decoder predict the current chunk by attending only to the compressed vectors of earlier chunks—never to raw past tokens. Because chunk size sets the length of the sequence the decoder attends to, it sets the compute and memory cost, and because the paper trains one model with several chunk sizes at once, that cost can be chosen at inference time with no retraining. The paper's central empirical claim is that this single adaptive model, evaluated at chunk sizes 4, 8, 16 and 32, matches or outperforms ten popular efficient architectures—linear, hybrid, and sparse—on real-world long-context recall and long-context understanding at comparable cost, while generating 1.4–3.7x faster than a dense transformer. If true, this means practitioners would not need to choose and train a separate architecture for each latency/quality requirement; one model interpolates between a cheap fast mode and an expensive accurate mode.

Core claim

CAT stores the past as a sequence of compressed chunk vectors instead of a fixed-size state or sparse token subset. A bidirectional transformer compresses each C-token chunk into a vector; a causal transformer decodes each chunk from the tokens inside it plus all earlier chunk vectors (Eq. 1). Because raw past tokens are dropped, attention memory shrinks by factor C while still growing with context length, so recall degrades slowly. Trained once at chunk sizes 4, 8, 16, 32, the same model can be served at any budget. CAT-4 beats the dense transformer on language modeling and in-context recall while being ~1.5x faster and 2x lower memory; other settings match or beat ten efficient baselines a

What carries the argument

The load-bearing object is the chunked conditional factorization of Eq. (1): each chunk is predicted from the tokens already inside it plus the compressed vectors of all earlier chunks. The enabling mechanism is a custom attention mask: training interleaves each chunk with its compressed vector so tokens in chunk i attend only to earlier compressed vectors and within-chunk tokens. That reuses the same key-value vectors for all later chunks, cutting training attention cost from O(N^2) to O(N^2/C), and lets generation discard raw past tokens, shrinking the KV cache (stored attention memory) by a factor C. The mask must implement Eq. (1) exactly.

Load-bearing premise

The results stand or fall on whether the custom attention mask used in training implements exactly the chunked conditional distribution the generation loop serves (Eq. 1); if the mask leaks information from raw past chunks or blocks information the served model would see, the reported model is not the one being benchmarked.

What would settle it

Take a trained CAT and a long prefix. Compute next-token log-probabilities for every chunk boundary two ways: (1) the masked parallel training forward pass, and (2) the chunk-by-chunk generation loop with KV-cache prefill. If the two distributions differ beyond floating-point precision, the training objective is not the served model; inspecting the attention mask for any nonzero weight from a token in chunk i to a raw token outside chunk i would show the leak directly.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Chunk size becomes a test-time control: one CAT model can serve low-latency, low-memory requests at large chunk sizes and high-recall requests at small chunk sizes, with no retraining and no extra model copies.
  • The KV-cache reduction is roughly linear in chunk size, so even C=4 cuts stored attention memory by 4x; at C=32 the paper measures up to 9.5x lower total memory and 3.2x faster generation than a dense transformer across sequence lengths.
  • On real-world in-context recall, CAT at C=4 and C=8 beats the dense transformer and all ten efficient baselines, indicating that chunked compression preserves more retrievable information per byte of memory than fixed recurrent states or heuristic sparsity.
  • The adaptive model is somewhat weaker than separately trained fixed-chunk CATs (Appendix A.3), but still outperforms the baselines, so the multi-budget convenience costs only a small quality margin.
  • Because CAT is built from dense attention and compression only, existing GPU-friendly attention kernels and KV-compression techniques can be applied on top of it; the paper notes its current training kernel is not yet optimized (Appendix B.5).

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the CAT results hold, the right comparison for efficient architectures may not be 'one big recurrent state vs. many tokens' but 'how many compressed states fit in a given memory budget'; storing many small chunk vectors appears to preserve more retrievable information than compressing all history into one state.
  • The mask-equivalence assumption can be tested directly: comparing training-forward vs. generation-forward next-token probabilities on a held-out long sequence would settle whether the reported gains transfer to the deployed chunk-by-chunk loop.
  • The paper leaves open whether chunk size can be chosen adaptively per input; its future-work section suggests RL post-training for exactly this, which would make the dial automatic rather than user-set.
  • Scaling beyond the 1-parameter range would show whether optimal chunk sizes shift: at larger scales and longer training, the compressor may learn to surface more information per chunk, moving the quality-cost frontier toward larger C.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes the Compress & Attend Transformer (CAT), which compresses each chunk of tokens into a vector via a dense transformer compressor, and decodes each chunk by attending to past compressed representations. Chunk size controls the trade-off between quality and computational/memory cost, and training with several chunk sizes yields a single model whose operating point can be chosen at test time. The authors report extensive experiments comparing CAT to dense, sparse, linear, and hybrid baselines on language modeling, common-sense reasoning, in-context recall, and long-context benchmarks. They also report generation speed and memory advantages over a dense transformer. A training implementation uses a custom attention mask over an interleaved sequence of chunks and compressed representations; generation uses a chunk-level KV cache. The central claim is that a single adaptive CAT model matches or outperforms ten baselines at comparable inference costs.

Significance. The proposed architecture is a simple, elegant way to obtain flexible memory and compute efficiency while retaining unrestricted access to the past via learned compression. The test-time controllability of chunk size in a single model is a distinctive and practically useful feature. The paper provides a large set of benchmarks, ablations, and public code, which support reproducibility. However, the current empirical evidence does not yet establish the central claim because of the training-inference positional mismatch and the parameter-scale confound.

major comments (2)
  1. [App. B.1/B.3 vs Eq. (1)] training uses an interleaved sequence {c_i, f(c_i)} with a custom mask, while inference serves chunks from a KV cache containing only consecutive f(c_i). With rotary positions (App. D.1), the relative distance from a token in c_i to f(c_{i-2}) is about C+j+2 in training but 2 at inference. No positional re-indexing is described, so the model is trained under a different positional distribution than the one used in generation. This undermines the claim that Eq. (1) is the trained distribution, and the reported recall (Tables 3, 5) may not be for the stated architecture. The authors should either adapt training positions to match inference or provide a rigorous argument for invariance.
  2. [Table 15 vs Tables 2–5] CAT uses ~1B total parameters (820M decoder + compressor/embedding) while baselines use 260–310M. The 'comparable inference costs' claim does not control for capacity; the gains over baselines could reflect extra parameters rather than the compression mechanism. Figure 3 reports throughput for CAT, but this does not isolate the effect of chunked compression from the wider decoder. Provide a parameter-matched comparison (e.g., a CAT with decoder width D) or baseline with similar parameters to support the central comparison.
minor comments (4)
  1. [Abstract and Section 4] Inconsistent speedup numbers: abstract says 1.4–3.7x, Section 4 reports 1.4–3.2x, and Section 5 says 1.5x. Please harmonize.
  2. [App. A.7] Contains 'refer to the comparative figure in ??'—a missing reference.
  3. [App. A.5] Typo: 'seperate' should be 'separate'.
  4. [Listing 1 (App. B.1)] The comment 'note that this loop is done in parallel with the custom attention mask' is confusing because the code shows a sequential loop; clarify the intended parallelism.

Circularity Check

0 steps flagged

No significant circularity: CAT's results are empirical comparisons against external benchmarks and baselines, with no fitted prediction renamed as a result and no load-bearing self-citation chain.

full rationale

The paper's central claims are empirical and are tested against external data and baselines rather than derived from the model's own fitted values. Equation (1) defines the CAT predictive distribution, and the training procedure in Section 2.1 / Appendix B.1 is an implementation designed to compute that distribution in parallel; the chunk size is a controllable architectural parameter, not a quantity fitted to the benchmarks. Tables 2-5 evaluate against independent benchmarks (FineWeb-Edu, LAMBADA, WikiText, LongBench, RULER, EVAPORATE, BabiLong) and external baselines (Dense, Sparse, Mamba2, GDN, GDN-Hybrid) run with their official implementations. The 'single adaptive model' property is trained by explicitly sampling chunk sizes C={4,8,16,32} during training, so the later evaluations at those chunk sizes are honest measurements of the trained behavior, not predictions forced by construction. The paper even reports the less favorable comparison in Appendix A.3 (fixed-chunk CATs outperform adaptive CATs on recall), which is the opposite of hiding a forced result. The only self-citation, Zhang & Ranganath (2025), appears in the discussion of RL post-training and is not load-bearing for any architectural claim. There is no imported uniqueness theorem, no ansatz justified solely by a self-citation, and no renaming of a known empirical pattern as a new result. The reviewer-identified concern about a possible positional mismatch between the masked training sequence and chunk-by-chunk generation (Appendix B.1 vs. B.3) is a potential train/inference correctness threat, not a circularity: it does not make any claimed output equivalent to the model's inputs by definition. Therefore the appropriate circularity score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claims rest on standard transformer components plus a small set of architectural choices (decoder width, compressor depth, chunk sizes, adaptive projection). No new physical entities or exotic postulates are introduced. The most consequential assumption is the training mask equivalence, which is an implementation-level premise rather than a fitted parameter.

free parameters (4)
  • Decoder hidden size scale factor = 2 (D_g = 2D)
    Chosen based on WikiText-103 ablations (Appendix C.2, Table 12) showing D_g = 2D gives better perplexity and recall; a key architectural choice for the main results.
  • Compressor depth scale factor = L/4 (3 layers for L=12)
    Selected after ablation (Appendix C.3, Table 14) showed no perplexity benefit from deeper compressors; reduces training cost.
  • Chunk size set = {4, 8, 16, 32}
    Hand-selected to span the quality-efficiency tradeoff; not fitted to data but a design choice that defines the test-time control range.
  • Projection matrix interpolation for adaptive chunk sizes = linear interpolation (Beyer et al., 2023)
    A design choice to handle variable projection shapes across chunk sizes; reported to work better than separate projections (Appendix B.4).
axioms (4)
  • domain assumption The chunked conditional distribution pθ defined in Eq. (1) is the correct training objective for the CAT architecture.
    The paper assumes that modeling each chunk conditioned on all previous chunk representations and within-chunk previous tokens is a valid autoregressive factorization, and that optimizing this objective yields the desired controllable efficiency behavior.
  • ad hoc to paper The custom attention mask (Figure 8) exactly implements the conditional distribution of Eq. (1) during training, so training and inference are consistent.
    The parallel training trick relies on this equivalence; it is stated but not formally proven. Any mismatch would invalidate the experiments.
  • domain assumption The compressor can be trained end-to-end with only the next-token prediction loss, without a separate compression loss.
    The paper assumes this joint training is sufficient for the compressor to learn chunk representations that preserve task-relevant information; no auxiliary losses or guarantees are provided.
  • standard math Standard transformer components (attention, MLP, positional embeddings, softmax) behave as expected in this architecture.
    The paper builds directly on dense transformer building blocks and uses standard kernels and implementations.

pith-pipeline@v1.3.0-alltime-deepseek · 27206 in / 10155 out tokens · 88424 ms · 2026-08-03T23:28:41.090486+00:00 · methodology

0 comments
read the original abstract

The substantial inference costs of attention in transformers motivated the development of efficient sequence mixers: namely sparse and sliding window attention, convolutions and linear attention. Although these approaches result in impressive reductions in inference costs, they often trade-off with quality, specifically in-context recall. Apriori fixing this quality-cost tradeoff at training time means being suboptimal from the get-go: some downstream applications might fundamentally require more memory for in-context recall, while other tasks may require lower latency and memory. We propose a conceptually simple meta-sequence mixer with inference-cost controllability: the Compress & Attend Transformer (CAT). CAT decodes chunks of tokens by attending to compressed chunks of the sequence so far. Both compression and decoding can use any existing sequence mixer. Decoding from the compressed sequence yields compute and memory savings, with chunk size setting the operating point on the quality-cost trade-off. Importantly, training CAT across multiple chunk sizes at once unlocks test-time control of this trade-off without any retraining, all in a single model. Instantiated with the most basic choice, dense attention as the mixer, CAT surprisingly suffices to match 10 popular and diverse efficient models (linear, hybrids, sparse) on real-world long-context recall at comparable inference costs, all from a single trained model. CAT further performs competitively on long-context understanding benchmarks while providing 1.4-3.7x higher generation throughput than a dense transformer. Code is at: https://github.com/rajesh-lab/cat-transformer

Figures

Figures reproduced from arXiv: 2511.05313 by Aahlad Puli, Jatin Prakash, Rajesh Ranganath.

Figure 1
Figure 1. Figure 1: CAT unlocks test-time control of quality-efficiency trade-offs, where a single adaptive CAT model (all red dots come from a single model) outperforms nearly every popular efficient architecture on real-world in-context recall tasks across varying compute-memory budgets. 1 arXiv:2511.05313v1 [cs.LG] 7 Nov 2025 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The Compress and Attend Transformer (CAT) architecture. CAT chunks up a sequence of length N into N/C chunks of C tokens (illustrated for C = 3). Each chunk is parallelly compressed into a chunk representation. CAT then decodes each chunk by attending to past chunk representations. Compression results in a reduced sequence length enabling compute and memory savings during decoding. Chunk size in CAT acts a… view at source ↗
Figure 3
Figure 3. Figure 3: A single CAT model generates 1.4 − 3.2× faster than the dense transformer while showcasing upto 2.2−9.5× lower memory usage. Per [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: CATs scale like their dense transformer coun￾terparts while being up to 3× faster and 9× more memory-efficient. All CAT points come from a sin￾gle model at a particular scale, evaluated at different chunk sizes. Ablations: We investigate how different choices affect performance of CATs in App. C. We further provide performance of CATs when trained independently for a single chunk size in App. A.3. Instanti… view at source ↗
Figure 6
Figure 6. Figure 6: Loss curves when finetuning different CATs on samples from S-NIAH-U task. Model Before After CAT-4 46.3 97.1 CAT-8 47.3 97.0 CAT-16 3.8 94.2 CAT-32 0.0 64.3 [PITH_FULL_IMAGE:figures/full_fig_p020_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Block Transformer (Ho et al., 2024; Yu et al., 2023) (across different configurations and hyperparam￾eters) fails to solve a simple MQAR task with only 4 key-value pairs, tested on modest sequence length of 256 tokens, possibly due to fixed memory. CAT solves the task with ease due to flexible yet efficient memory. Note that training of CAT stops when it solves the task perfectly. components (embedder, glo… view at source ↗
Figure 8
Figure 8. Figure 8: Sequence length is 128, and the chunk size that we use in this particular attention mask is [PITH_FULL_IMAGE:figures/full_fig_p023_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

89 extracted references · 52 linked inside Pith

  1. [1]

    gpt-oss-120b & gpt-oss-20b model card

    Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    Zoology: Measuring and improving recall in efficient language models

    Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher R \'e . Zoology: Measuring and improving recall in efficient language models. arXiv preprint arXiv:2312.04927, 2023 a

    Pith/arXiv arXiv 2023

  3. [3]

    Language models enable simple systems for generating structured views of heterogeneous data lakes

    Simran Arora, Brandon Yang, Sabri Eyuboglu, Avanika Narayan, Andrew Hojel, Immanuel Trummer, and Christopher R \'e . Language models enable simple systems for generating structured views of heterogeneous data lakes. arXiv preprint arXiv:2304.09433, 2023 b

    Pith/arXiv arXiv 2023

  4. [4]

    Simple linear attention language models balance the recall-throughput tradeoff

    Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher R \'e . Simple linear attention language models balance the recall-throughput tradeoff. arXiv preprint arXiv:2402.18668, 2024 a

    Pith/arXiv arXiv 2024

  5. [5]

    Just read twice: closing the recall gap for recurrent language models

    Simran Arora, Aman Timalsina, Aaryan Singhal, Benjamin Spector, Sabri Eyuboglu, Xinyi Zhao, Ashish Rao, Atri Rudra, and Christopher R \'e . Just read twice: closing the recall gap for recurrent language models. arXiv preprint arXiv:2407.05483, 2024 b

    Pith/arXiv arXiv 2024

  6. [6]

    How to scale your model

    Jacob Austin, Sholto Douglas, Roy Frostig, Anselm Levskaya, Charlie Chen, Sharad Vikram, Federico Lebron, Peter Choy, Vinay Ramasesh, Albert Webson, and Reiner Pope. How to scale your model. 2025. Retrieved from https://jax-ml.github.io/scaling-book/

    2025

  7. [7]

    Neural machine translation by jointly learning to align and translate

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014

    Pith/arXiv arXiv 2014

  8. [8]

    Longbench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508, 2023

    Pith/arXiv arXiv 2023

  9. [9]

    Transformers need glasses! information over-squashing in language tasks

    Federico Barbero, Andrea Banino, Steven Kapturowski, Dharshan Kumaran, Jo \ a o Madeira Ara \'u jo, Oleksandr Vitvitskyi, Razvan Pascanu, and Petar Veli c kovi \'c . Transformers need glasses! information over-squashing in language tasks. Advances in Neural Information Processing Systems, 37: 0 98111--98142, 2024

    2024

  10. [10]

    Large concept models: Language modeling in a sentence representation space

    Lo \" c Barrault, Paul-Ambroise Duquenne, Maha Elbayad, Artyom Kozhevnikov, Belen Alastruey, Pierre Andrews, Mariano Coria, Guillaume Couairon, Marta R Costa-juss \`a , David Dale, et al. Large concept models: Language modeling in a sentence representation space. arXiv preprint arXiv:2412.08821, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    Titans: Learning to memorize at test time

    Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to memorize at test time. arXiv preprint arXiv:2501.00663, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    Longformer: The long-document transformer

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020

    Pith/arXiv arXiv 2004

  13. [13]

    Flexivit: One model for all patch sizes

    Lucas Beyer, Pavel Izmailov, Alexander Kolesnikov, Mathilde Caron, Simon Kornblith, Xiaohua Zhai, Matthias Minderer, Michael Tschannen, Ibrahim Alabdulmohsin, and Filip Pavetic. Flexivit: One model for all patch sizes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 14496--14506, 2023

    2023

  14. [14]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.\ 7432--7439, 2020

    2020

  15. [15]

    Adapting language models to compress contexts

    Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https://openreview.net/forum?id=kp1U6wBPXq

    2023

  16. [16]

    Overcoming a theoretical limitation of self-attention

    David Chiang and Peter Cholak. Overcoming a theoretical limitation of self-attention. arXiv preprint arXiv:2202.12172, 2022

    Pith/arXiv arXiv 2022

  17. [17]

    Generating long sequences with sparse transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019

    Pith/arXiv arXiv 1904

  18. [18]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    Pith/arXiv arXiv 2018

  19. [19]

    Funnel-transformer: Filtering out sequential redundancy for efficient language processing

    Zihang Dai, Guokun Lai, Yiming Yang, and Quoc Le. Funnel-transformer: Filtering out sequential redundancy for efficient language processing. Advances in neural information processing systems, 33: 0 4271--4282, 2020

    2020

  20. [20]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    Flashattention: Fast and memory-efficient exact attention with io-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e . Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35: 0 16344--16359, 2022

    2022

  22. [22]

    Matformer: Nested transformer for elastic inference

    Fnu Devvrit, Sneha Kudugunta, Aditya Kusupati, Tim Dettmers, Kaifeng Chen, Inderjit Dhillon, Yulia Tsvetkov, Hannaneh Hajishirzi, Sham Kakade, Ali Farhadi, and Prateek Jain. Matformer: Nested transformer for elastic inference. In Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@NeurIPS 2...

    2023

  23. [23]

    Flex attention: A programming model for generating optimized attention kernels

    Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, and Horace He. Flex attention: A programming model for generating optimized attention kernels. arXiv preprint arXiv:2412.05496, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint arXiv:1903.00161, 2019

    Pith/arXiv arXiv 1903

  25. [25]

    Hungry hungry hippos: Towards language modeling with state space models

    Daniel Y Fu, Tri Dao, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher R \'e . Hungry hungry hippos: Towards language modeling with state space models. arXiv preprint arXiv:2212.14052, 2022

    Pith/arXiv arXiv 2022

  26. [26]

    Scaling up test-time compute with latent reasoning: A recurrent depth approach

    Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint arXiv:2502.05171, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    Ai and memory wall

    Amir Gholami, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W Mahoney, and Kurt Keutzer. Ai and memory wall. IEEE Micro, 44 0 (3): 0 33--39, 2024

    2024

  28. [28]

    Multi-token attention

    Olga Golovneva, Tianlu Wang, Jason Weston, and Sainbayar Sukhbaatar. Multi-token attention. arXiv preprint arXiv:2504.00927, 2025

    Pith/arXiv arXiv 2025

  29. [29]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    Pith/arXiv arXiv 2023

  30. [30]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher R \'e . Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021

    Pith/arXiv arXiv 2021

  31. [31]

    Transformer in transformer

    Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. Advances in neural information processing systems, 34: 0 15908--15919, 2021

    2021

  32. [32]

    Block transformer: Global-to-local language modeling for fast inference

    Namgyu Ho, Sangmin Bae, Taehyeon Kim, Hyunjik Jo, Yireun Kim, Tal Schuster, Adam Fisch, James Thorne, and Se-Young Yun. Block transformer: Global-to-local language modeling for fast inference. Advances in Neural Information Processing Systems, 37: 0 48740--48783, 2024

    2024

  33. [33]

    functorch: Jax-like composable function transforms for pytorch

    Richard Zou Horace He. functorch: Jax-like composable function transforms for pytorch. https://github.com/pytorch/functorch, 2021

    2021

  34. [34]

    Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Dynamic chunking for end-to-end hierarchical sequence modeling

    Sukjun Hwang, Brandon Wang, and Albert Gu. Dynamic chunking for end-to-end hierarchical sequence modeling. arXiv preprint arXiv:2507.07955, 2025

    Pith/arXiv arXiv 2025

  36. [36]

    Repeat after me: Transformers are better than state space models at copying

    Samy Jelassi, David Brandfonbrener, Sham M Kakade, and Eran Malach. Repeat after me: Transformers are better than state space models at copying. arXiv preprint arXiv:2402.01032, 2024

    Pith/arXiv arXiv 2024

  37. [37]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https://arxi...

    Pith/arXiv arXiv 2023

  38. [38]

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017

    Pith/arXiv arXiv 2017

  39. [39]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Fran c ois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pp.\ 5156--5165. PMLR, 2020

    2020

  40. [40]

    Babilong: Testing the limits of llms with long context reasoning-in-a-haystack, 2024

    Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: Testing the limits of llms with long context reasoning-in-a-haystack, 2024

    2024

  41. [41]

    Matryoshka representation learning

    Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representation learning. Advances in Neural Information Processing Systems, 35: 0 30233--30249, 2022

    2022

  42. [42]

    Inference-time hyper-scaling with kv cache compression

    Adrian a \'n cucki, Konrad Staniszewski, Piotr Nawrot, and Edoardo M Ponti. Inference-time hyper-scaling with kv cache compression. arXiv preprint arXiv:2506.05345, 2025

    arXiv 2025

  43. [43]

    A training-free sub-quadratic cost transformer model serving framework with hierarchically pruned attention

    Heejun Lee, Geon Park, Youngwan Lee, Jaduk Suh, Jina Kim, Wonyoung Jeong, Bumsik Kim, Hyemin Lee, Myeongjae Jeon, and Sung Ju Hwang. A training-free sub-quadratic cost transformer model serving framework with hierarchically pruned attention. arXiv preprint arXiv:2406.09827, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    Infinitehip: Extending language model context up to 3 million tokens on a single gpu, 2025

    Heejun Lee, Geon Park, Jaduk Suh, and Sung Ju Hwang. Infinitehip: Extending language model context up to 3 million tokens on a single gpu, 2025. URL https://arxiv.org/abs/2502.08910

    Pith/arXiv arXiv 2025

  45. [45]

    Snapkv: Llm knows what you are looking for before generation

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems, 37: 0 22947--22970, 2024

    2024

  46. [46]

    Openceres: When open information extraction meets the semi-structured web

    Colin Lockard, Prashant Shiralkar, and Xin Luna Dong. Openceres: When open information extraction meets the semi-structured web. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp.\ 3047--3056, 2019

    2019

  47. [47]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    Pith/arXiv arXiv 2017

  48. [48]

    Beyondweb: Lessons from scaling synthetic data for trillion-scale pretraining

    Pratyush Maini, Vineeth Dorna, Parth Doshi, Aldo Carranza, Fan Pan, Jack Urbanek, Paul Burstein, Alex Fang, Alvin Deng, Amro Abbas, et al. Beyondweb: Lessons from scaling synthetic data for trillion-scale pretraining. arXiv preprint arXiv:2508.10975, 2025

    Pith/arXiv arXiv 2025

  49. [49]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016

    Pith/arXiv arXiv 2016

  50. [50]

    Online normalizer calculation for softmax

    Maxim Milakov and Natalia Gimelshein. Online normalizer calculation for softmax. arXiv preprint arXiv:1805.02867, 2018

    Pith/arXiv arXiv 2018

  51. [51]

    Olmoe: Open mixture-of-experts language models

    Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, et al. Olmoe: Open mixture-of-experts language models. arXiv preprint arXiv:2409.02060, 2024

    Pith/arXiv arXiv 2024

  52. [52]

    Hierarchical transformers are more efficient language models

    Piotr Nawrot, Szymon Tworkowski, Micha Tyrolski, ukasz Kaiser, Yuhuai Wu, Christian Szegedy, and Henryk Michalewski. Hierarchical transformers are more efficient language models. arXiv preprint arXiv:2110.13711, 2021

    Pith/arXiv arXiv 2021

  53. [53]

    Efficient transformers with dynamic token pooling

    Piotr Nawrot, Jan Chorowski, Adrian a \'n cucki, and Edoardo M Ponti. Efficient transformers with dynamic token pooling. arXiv preprint arXiv:2211.09761, 2022

    Pith/arXiv arXiv 2022

  54. [54]

    Dynamic memory compression: Retrofitting llms for accelerated inference

    Piotr Nawrot, Adrian a \'n cucki, Marcin Chochowski, David Tarjan, and Edoardo M Ponti. Dynamic memory compression: Retrofitting llms for accelerated inference. arXiv preprint arXiv:2403.09636, 2024

    Pith/arXiv arXiv 2024

  55. [55]

    The sparse frontier: Sparse attention trade-offs in transformer llms

    Piotr Nawrot, Robert Li, Renjie Huang, Sebastian Ruder, Kelly Marchisio, and Edoardo M Ponti. The sparse frontier: Sparse attention trade-offs in transformer llms. arXiv preprint arXiv:2504.17768, 2025

    Pith/arXiv arXiv 2025

  56. [56]

    Asynchronous rlhf: Faster and more efficient off-policy rl for language models

    Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, and Aaron Courville. Asynchronous rlhf: Faster and more efficient off-policy rl for language models. arXiv preprint arXiv:2410.18252, 2024

    Pith/arXiv arXiv 2024

  57. [57]

    The lambada dataset: Word prediction requiring a broad discourse context

    Denis Paperno, Germ \'a n Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fern \'a ndez. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031, 2016

    Pith/arXiv arXiv 2016

  58. [58]

    Hierarchical transformers for long document classification

    Raghavendra Pappagari, Piotr Zelasko, Jes \'u s Villalba, Yishay Carmiel, and Najim Dehak. Hierarchical transformers for long document classification. In 2019 IEEE automatic speech recognition and understanding workshop (ASRU), pp.\ 838--844. ieee, 2019

    2019

  59. [59]

    Image transformer

    Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International conference on machine learning, pp.\ 4055--4064. PMLR, 2018

    2018

  60. [60]

    The fineweb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydl \' c ek, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37: 0 30811--30849, 2024

    2024

  61. [61]

    Rwkv: Reinventing rnns for the transformer era

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048, 2023

    Pith/arXiv arXiv 2023

  62. [62]

    Hyena hierarchy: Towards larger convolutional language models

    Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher R \'e . Hyena hierarchy: Towards larger convolutional language models. In International Conference on Machine Learning, pp.\ 28043--28078. PMLR, 2023

    2023

  63. [63]

    Qwen3-next: Towards ultimate training & inference efficiency, 2025

    Qwen. Qwen3-next: Towards ultimate training & inference efficiency, 2025. URL https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd&from=research.latest-advancements-list. Accessed: 2025-09-18

    2025

  64. [64]

    Rae, Anna Potapenko, Siddhant M

    Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SylKikSYDH

    2020

  65. [65]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  66. [66]

    Know what you don't know: Unanswerable questions for squad

    Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822, 2018

    Pith/arXiv arXiv 2018

  67. [67]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64 0 (9): 0 99--106, 2021

    2021

  68. [68]

    Fast transformer decoding: One write-head is all you need

    Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019

    Pith/arXiv arXiv 1911

  69. [69]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017

    Pith/arXiv arXiv 2017

  70. [70]

    Spacebyte: Towards deleting tokenization from large language modeling

    Kevin Slagle. Spacebyte: Towards deleting tokenization from large language modeling. Advances in Neural Information Processing Systems, 37: 0 124925--124950, 2024

    2024

  71. [71]

    Retentive network: A successor to transformer for large language models

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023

    Pith/arXiv arXiv 2023

  72. [72]

    Quest: Query-aware sparsity for efficient long-context llm inference

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context llm inference. arXiv preprint arXiv:2406.10774, 2024

    Pith/arXiv arXiv 2024

  73. [73]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    Pith/arXiv arXiv 2023

  74. [74]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  75. [75]

    An empirical study of mamba-based language models

    Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, et al. An empirical study of mamba-based language models. arXiv preprint arXiv:2406.07887, 2024

    Pith/arXiv arXiv 2024

  76. [76]

    A systematic analysis of hybrid linear attention

    Dustin Wang, Rui-Jie Zhu, Steven Abreu, Yong Shan, Taylor Kergan, Yuqi Pan, Yuhong Chou, Zheng Li, Ge Zhang, Wenhao Huang, et al. A systematic analysis of hybrid linear attention. arXiv preprint arXiv:2507.06457, 2025

    Pith/arXiv arXiv 2025

  77. [77]

    Rnns are not transformers (yet): The key bottleneck on in-context retrieval, 2024

    Kaiyue Wen, Xingyu Dang, and Kaifeng Lyu. Rnns are not transformers (yet): The key bottleneck on in-context retrieval, 2024. URL https://arxiv.org/abs/2402.18510

    Pith/arXiv arXiv 2024

  78. [78]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025 a

    Pith/arXiv arXiv 2025

  79. [79]

    Gated delta networks: Improving mamba2 with delta rule

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. In The Thirteenth International Conference on Learning Representations, 2025 b . URL https://openreview.net/forum?id=r8H7xhYPwz

    2025

  80. [80]

    Long-context language modeling with parallel context encoding

    Howard Yen. Long-context language modeling with parallel context encoding. Master's thesis, Princeton University, 2024

    2024

Showing first 80 references.