Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Efficient Pretraining Length Scaling

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Repeating input tokens during pretraining is a viable length-scaling axis when the repeated tokens' KV cache is discarded after decoding, yielding accuracy gains without extra inference memory.

desk verdict Useful KV-cache-friendly token-repetition architecture, but the central 'pretraining length scaling' claim is unproven because the comparisons bundle repetition with more FLOPs, fewer unique tokens, and more test-time compute. read the letter →

arxiv 2504.14992 v2 pith:4ZVIP2LD submitted 2025-04-21 cs.CL

Bohong Wu , Shen Yan , Sijun Zhang , Jianqiao Lu , Yutao Zeng , Ya Wang , Xun Zhou This is my paper · ORCID
classification cs.CL
keywords pretraininglengthscalingKVcachemanagementtokenrepetitionhiddendecodingtokensparallelslidingwindowattentionchunk-wiselargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that length scaling, already useful in post-training, can also be harnessed during pretraining by simply repeating each input token K times. It introduces PHD-Transformer, a training scheme that keeps the KV cache of only the first copies of tokens and discards the cache of repeated copies after use, so inference memory and latency stay close to a vanilla transformer. Two variants, PHD-SWA and PHD-CSWA, add a small local window over the repeated tokens, with chunking to prevent prefill time from growing linearly. On 1.2B and 550M models, the scheme lowers training loss and raises average downstream accuracy by 1.5 to 2.0 percentage points over a vanilla baseline. A sympathetic reader would take the paper's contribution to be an inference-friendly way to buy pretraining quality by spending more compute on repeated tokens.

What carries the argument

The central mechanism is an attention-mask and KV-cache split between original tokens and hidden decoding tokens. Original tokens, the first copies, are globally attended to and their key/value vectors are retained, while hidden decoding tokens, the repeated copies, are decoded and then their key/value vectors are dropped, so only the original sequence length is cached and the final copy of each position carries the next-token loss. PHD-SWA adds a sliding window over recent hidden decoding tokens to keep some local cache with O(1) extra memory, and PHD-CSWA restricts the window to fixed chunks so that prefill time does not grow linearly with K.

What would settle it

Train a vanilla transformer with FLOPs matched to PHD-CSWA-2 and PHD-CSWA-3 on the same unique 500B tokens, for example by running two or three epochs over the same data or by increasing depth to match compute, and compare average accuracy on the six benchmarks. If the compute-matched vanilla model reaches or exceeds the 1.5 and 2.0 point gains, the central claim that the KV-cache design enables length scaling rather than extra compute would collapse.

Watch

Extended reading notes

Core claim

The paper claims that a transformer can be pretrained with its input sequence repeated K times, using the first copy of each token to generate the persistent KV cache while immediately discarding the KV cache of the hidden decoding tokens after they contribute to next-token prediction. This keeps the KV cache size identical to a vanilla transformer while giving the model extra computational depth per token. Empirically, PHD-CSWA-2-16-32 improves average accuracy on six benchmarks by 1.5 points over a 1.2B vanilla baseline, PHD-CSWA-3-16-32 improves it by 2.0 points, and PHD-SWA with K=5 improves a 550M baseline by 1.8 points; decoding latency stays within about 20 percent of vanilla even at K=256.

Load-bearing premise

The paper's main experiments compare PHD models against a single vanilla baseline trained on the same 500B consumed tokens but with fewer unique tokens and much less compute, so the reported gains are only attributable to length scaling if extra compute and repeated exposure to the same data are not responsible.

Editorial extensions

If this is right

  • Pretraining length scaling via token repetition is a workable axis: larger K gives lower loss and higher downstream accuracy, at least up to K=5 in the tested settings.
  • Inference cost stays near vanilla: KV cache size is unchanged, decoding latency grows by at most about 20 percent even at K=256, and CSWA keeps prefill time close to vanilla.
  • A small sliding window over hidden decoding tokens captures most of the benefit, so the extra KV memory is only O(1).
  • Chunk-wise sliding window attention removes the linear prefill growth of the sliding-window variant with negligible performance loss.
  • The attention-rearrangement and KV-sharing design is compatible with other kernel-level and cache-management optimizations, so it can be layered onto standard efficient transformers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The decisive untested comparison is a compute-matched vanilla baseline: if a vanilla model trained with the same FLOPs on the same unique tokens matches the gains, then the contribution is extra compute rather than length scaling specifically.
  • The same discard-after-use KV trick could be applied to inference-time chain-of-thought or thinking tokens, giving models a bounded-memory way to spend more forward passes before answering.
  • Because only the final copy receives the next-token loss, the repeated copies act as an implicit depth increase; a natural extension is to vary K adaptively by token or training stage rather than fixing it for the whole corpus.
  • The reported gains are on commonsense and language-understanding benchmarks at 1.2B and below, so whether the pattern transfers to frontier-scale models or to math and reasoning tasks is still open.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript proposes the Parallel Hidden Decoding Transformer (PHD) family for what it calls pre-training length scaling: input sequences are repeated K times, the first ('original') copies retain global KV caches while the repeated ('hidden decoding') copies are discarded after use, and the next-token prediction loss is taken from the final copy. PHD-SWA adds a sliding window over hidden decoding tokens and PHD-CSWA chunks that window to avoid pre-fill growth. Experiments on 1.2B and 550M models trained on OLMo2-style data report lower training loss, roughly 1.5-2.0% average downstream accuracy gains over vanilla baselines (Tables 1-2), and near-vanilla pre-fill and decoding latency (Figure 9).

Significance. The architecture idea is clear and the efficiency measurements are genuinely useful: keeping the KV cache at the vanilla size while adding K parallel decoding tokens addresses a real inference bottleneck, and the chunked sliding-window variant is a reasonable engineering solution. The paper also includes ablations (Figures 6-7) that support the chosen window and chunk sizes. However, the empirical evidence for the central 'pre-training length scaling' claim is currently confounded in two ways: the PHD runs are not matched to the vanilla baseline in compute or unique data, and the evaluation protocol appears to increase K at test time for PHD models only. Until controlled comparisons isolate the pretraining effect, the observed gains could be explained by extra training FLOPs, repeated-data epochs, or test-time repetition. If the authors supply the missing controls and clarify the evaluation protocol, this could become a solid contribution to efficient pretraining and inference.

major comments (3)
  1. [Section 3.2, Table 1, Figure 5] The main accuracy comparison is not matched in compute or data. If the shared 500B value on the Figure 5 x-axis is consumed tokens, then PHD-CSWA-2 and PHD-CSWA-3 process 500B tokens with K=2 and K=3 repetitions, i.e., only 250B and 167B unique tokens, while the vanilla baseline sees 500B unique tokens; the PHD runs also incur roughly K times more FLOPs. The reported loss drop and average accuracy gain are therefore compatible with a compute-scaling or repeated-epochs explanation rather than 'length scaling' as a distinct pretraining phenomenon. The paper should add a vanilla baseline with matched FLOPs (e.g., 1500B consumed tokens for the K=3 condition, either from additional data or from repeating the same corpus) and state the unique-token count and FLOP count for every run.
  2. [Section 3.2 and Section 3.4, Tables 1-2, Equation (2)] The evaluation protocol confounds the effect with test-time computation. The accuracy tables and Figure 8 compare PHD-K models, which appear to be evaluated with K repeated tokens, against a vanilla baseline evaluated with K=1. Since Equation (2) defines a valid attention mask for any transformer weights, a vanilla model can be run in the same repeated-token mode without any PHD-style pretraining; this is precisely what Section 3.4 calls 'decoding computation scaling.' Without cross-evaluations - vanilla evaluated at K>1 and, where possible, PHD-trained models evaluated at K=1 - the reported gains cannot be attributed to the pretraining objective. This is a load-bearing control and should be reported at least for the 550M/Table 2 setting.
  3. [Section 3.4 and Abstract] The paper's own framing is ambiguous about what is being scaled. The abstract claims pre-training length scaling, while Section 3.4 is titled 'Decoding Token Scaling' and Table 2's caption states that 'downstream performance scales w.r.t. the increase of decoding computation.' The manuscript should state explicitly what changes during evaluation when K is increased, and then design the experiments so that the pretraining contribution is separately identifiable from the decoding-time contribution.
minor comments (5)
  1. [Figures 1, 5, 6, 7, 8] The x-axis label 'Cosumed T okens' contains typos and should read 'Consumed Tokens'.
  2. [Section 2.2] The phrase 'origin tokens' is used inconsistently; the rest of the paper uses 'original tokens.'
  3. [Section 2.3] The claim that rearranging token positions 'reducing the complexity of attention computation to O(K)' is underspecified, since the original sequence length t also enters the complexity; please state the complexity in terms of both t and K.
  4. [Section 3.3.1] The window-size and chunk-size ablations report only training and validation loss; adding a downstream benchmark to these ablations would make the chosen hyperparameters more convincing.
  5. [Section 3.5] Figure 9 would benefit from a statement of the exact measurement protocol, including sequence length, batch size, and number of repeated runs, since the reported 20% decoding-latency increase is central to the efficiency claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's claims are empirical and its hyperparameters are selected by ablation, not derived from the target results.

full rationale

After walking the derivation chain, no load-bearing step reduces to its own inputs. The paper is an empirical method paper: it defines PHD's attention mask in Eq. (2), fixes K, W, C as hyperparameters, and reports benchmark numbers (Tables 1-2, Figures 5-8). The window/chunk sizes are chosen by explicit ablations (Figures 6-7) and then reused, which is standard hyperparameter selection rather than circular reasoning. There are no self-citation chains, no imported uniqueness theorem, and no fitted parameter renamed as a prediction. The reader-flagged concern that PHD models are trained and evaluated with K repetitions while the vanilla baseline uses K=1 is a real experimental-confound/correctness issue (test-time compute is not matched), but it is not a circularity: the improvement is not equivalent to an input of the method by construction. Per the review rules, that concern belongs in correctness risk, not in the circularity score.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on empirical assumptions about the benefit of token repetition and on the architectural choice of discarding hidden decoding KV caches. No formal derivation is given, and the main free parameters are the repetition count K, window size W, and chunk size C, all chosen by hand or ablation.

free parameters (3)
  • window_size_W = 16
    Selected by ablation on validation loss (Figure 6); larger windows help, but gains saturate quickly, so W=16 was used for all main experiments.
  • chunk_size_C = 32
    Selected by ablation on validation loss (Figure 7); C=32 balances prefill overhead and performance for the main runs.
  • scaling_factor_K = 2 or 3 in main experiments; 5 in scaling study
    Chosen by hand as the token repetition count; not tuned, but the central 'length scaling' variable.
assumptions (4)
  • domain assumption Repeating input tokens during pretraining provides a useful training signal (i.e., length scaling is beneficial).
    Central premise; tested only in the confounded comparison of Section 3.2/Figure 5, not against a compute-matched baseline.
  • domain assumption The attention mask in Eq. 2 preserves causal semantics and allows standard backprop; original tokens provide global key-values while hidden decoding tokens are consumed locally.
    Architectural design assumption; the notation of Eq. 2 is ambiguous, and correctness is only indirectly validated by downstream accuracy.
  • domain assumption Discarding hidden decoding KV caches after use (or keeping only a local window) does not harm long-range dependency modeling.
    Inference-efficiency claim relies on this; the paper provides no direct measurement of long-range dependency quality.
  • domain assumption OLMo2 training setup and the chosen benchmarks are representative enough for the reported improvements to generalize.
    All experiments use one codebase (OLMo2) and one data mixture; generalizability beyond this setup is untested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Pretraining Length Scaling." pith.science (2026). https://pith.science/paper/4ZVIP2LD

@misc{pith2026250414992,
  author       = {Pith},
  title        = {Pith review of: Efficient Pretraining Length Scaling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4ZVIP2LD}},
  note         = {Machine review of arXiv:2504.14992}
}
read the original abstract

Recent advances in large language models have demonstrated the effectiveness of length scaling during post-training, yet its potential in pre-training remains underexplored. We present the Parallel Hidden Decoding Transformer (\textit{PHD}-Transformer), a novel framework that enables efficient length scaling during pre-training while maintaining inference efficiency. \textit{PHD}-Transformer achieves this through an innovative KV cache management strategy that distinguishes between original tokens and hidden decoding tokens. By retaining only the KV cache of original tokens for long-range dependencies while immediately discarding hidden decoding tokens after use, our approach maintains the same KV cache size as the vanilla transformer while enabling effective length scaling. To further enhance performance, we introduce two optimized variants: \textit{PHD-SWA} employs sliding window attention to preserve local dependencies, while \textit{PHD-CSWA} implements chunk-wise sliding window attention to eliminate linear growth in pre-filling time. Extensive experiments demonstrate consistent improvements across multiple benchmarks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Scaling Latent Reasoning via Looped Language Models

    cs.CL 2025-10 unverdicted novelty 7.0 of 10

    Looped language models with latent iterative computation and entropy-regularized depth allocation achieve performance matching up to 12B standard LLMs through superior knowledge manipulation.

  2. The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook

    cs.AI 2026-04 accept novelty 5.0 of 10

    A large survey organizes latent-space work in language-based models by foundation, evolution, four mechanisms, seven abilities, and open challenges.

Reference graph

Works this paper leans on

63 extracted references · 12 canonical work pages · cited by 2 Pith papers

  1. [1]

    Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat S. Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen ...

    arXiv doi:10.48550/arxiv.2404.14219 2024

  2. [2]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895–4901, 2023

    2023

  3. [3]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. CoRR, abs/2004.05150, 2020. URL https://arxiv.org/abs/2004.05150

    arXiv 2004

  4. [4]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. InProceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020

    2020

  5. [5]

    Striped attention: Faster ring attention for causal transformers.CoRR, abs/2311.09431, 2023

    William Brandon, Aniruddha Nrusimha, Kevin Qian, Zachary Ankner, Tian Jin, Zhiye Song, and Jonathan Ragan-Kelley. Striped attention: Faster ring attention for causal transformers.CoRR, abs/2311.09431, 2023. doi: 10.48550/ARXIV.2311.09431. URL https://doi.org/10.48550/arXiv.2311.09431

    arXiv doi:10.48550/arxiv.2311.09431 2023

  6. [6]

    Step-level value preference optimization for mathematical reasoning

    Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Step-level value preference optimization for mathematical reasoning. arXiv preprint arXiv:2406.10858, 2024

    arXiv 2024

  7. [7]

    Inner thinking transformer: Leveraging dynamic depth scaling to foster adaptive internal thinking

    Yilong Chen, Junyuan Shang, Zhenyu Zhang, Yanxi Xie, Jiawei Sheng, Tingwen Liu, Shuohuan Wang, Yu Sun, Hua Wu, and Haifeng Wang. Inner thinking transformer: Leveraging dynamic depth scaling to foster adaptive internal thinking. arXiv preprint arXiv:2502.13842, 2025

    arXiv 2025

  8. [8]

    Generating long sequences with sparse transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. CoRR, abs/1904.10509, 2019. URLhttp://arxiv.org/abs/1904.10509

    arXiv 1904

Show all 63 references
  1. [9]

    Unified scaling laws for routed language models

    Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International conference on machine learning, pages 4057–408...

    2022

  2. [10]

    Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  3. [11]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  4. [12]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRe- view.net, 2024. URL https://openreview.net/forum?id=mZn2Xyh9Ec

    2024

  5. [13]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing ...

    2022

  6. [14]

    Flash-decoding for long-context inference, October

    Tri Dao, Daniel Haziza, Francisco Massa, and Grigory Sizov. Flash-decoding for long-context inference, October

  7. [15]

    Longnet: Scaling transformers to 1, 000, 000, 000 tokens

    Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Nanning Zheng, and Furu Wei. Longnet: Scaling transformers to 1, 000, 000, 000 tokens. CoRR, abs/2307.02486, 2023. doi: 10.48550/ARXIV.2307.02486. URL https://doi.org/10.48550/arXiv.2307.02486

    2023 doi arXiv

  8. [16]

    Scaling up test-time compute with latent reasoning: A recurrent depth approach

    Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint arXiv:2502.05171, 2025

    2025 arXiv

  9. [17]

    Think before you speak: Training language models with pause tokens

    Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. InThe TwelfthInternational Conference on Learning Representations

  10. [18]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  11. [19]

    Psydial: personality-based synthetic dialogue generation using large language models.arXiv preprint arXiv:2404.00930, 2024

    Ji-Eun Han, Jun-Seok Koh, Hyeon-Tae Seo, Du-Seong Chang, and Kyung-Ah Sohn. Psydial: personality-based synthetic dialogue generation using large language models.arXiv preprint arXiv:2404.00930, 2024

    2024 arXiv

  12. [20]

    Training large language models to reason in a continuous latent space.arXiv preprint arXiv:2412.06769, 2024

    Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space.arXiv preprint arXiv:2412.06769, 2024

    2024 arXiv

  13. [21]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  14. [22]

    Scaling laws for transfer.arXiv preprint arXiv:2102.01293, 2021

    Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer.arXiv preprint arXiv:2102.01293, 2021

    2021 arXiv

  15. [23]

    Flashdecoding++: Faster large language model inference on gpus.CoRR, abs/2311.01282, 2023

    Ke Hong, Guohao Dai, Jiaming Xu, Qiuli Mao, Xiuhong Li, Jun Liu, Kangdi Chen, Yuhan Dong, and Yu Wang. Flashdecoding++: Faster large language model inference on gpus.CoRR, abs/2311.01282, 2023. doi: 10.48550/ ARXIV.2311.01282. URL https://doi.org/10.48550/arXiv.2311.01282

    2023 doi arXiv

  16. [24]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    doi arXiv

  17. [25]

    Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention.Advances in Neural Information Processing Systems, 37:52481–52515, 2024

    Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir Abdi, Dongsheng Li, Chin-Yew Lin, et al. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention.Advances in Neural Information Processing S...

    2024

  18. [26]

    Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression

    Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Ann...

    2024 doi

  19. [27]

    Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770, 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770, 2023

    2023 arXiv

  20. [28]

    Scaling laws for neural language models.arXiv preprintarXiv:2001.08361, 2020

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprintarXiv:2001.08361, 2020

    2001 arXiv

  21. [29]

    Natural questions: a benchmark for question answering research

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computa...

    2019

  22. [30]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Jason Flinn, Margo I. Seltzer, Peter Druschel, Antoine Kaufmann, and...

    2023

  23. [31]

    Minimax-01: Scaling foundation models with lightning attention.arXiv preprint arXiv:2501.08313, 2025

    Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, et al. Minimax-01: Scaling foundation models with lightning attention.arXiv preprint arXiv:2501.08313, 2025

    2025 arXiv

  24. [32]

    Snapkv: LLM knows what you are looking for before generation.CoRR, abs/2404.14469,

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: LLM knows what you are looking for before generation.CoRR, abs/2404.14469,

    arXiv

  25. [33]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024

    2024 arXiv

  26. [34]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  27. [35]

    Ring attention with blockwise transformers for near-infinite context

    Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. CoRR, abs/2310.01889, 2023. doi: 10.48550/ARXIV.2310.01889. URLhttps://doi.org/10.48550/arXiv.2310. 01889

    2023 doi arXiv

  28. [36]

    Cotformer: More tokens with attention make up for less depth

    Amirkeivan Mohtashami, Matteo Pagliardini, and Martin Jaggi. Cotformer: More tokens with attention make up for less depth. InWorkshop on Advancing Neural Network Training: Computational Efficiency, Scalability,and Resource Optimization (WANT@NeurIPS 2023), 2023

    2023

  29. [37]

    2 olmo 2 furious.arXiv preprint arXiv:2501.00656, 2024

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious.arXiv preprint arXiv:2501.00656, 2024

    2024 arXiv

  30. [38]

    Learning to reason with llms, 2024

    OpenAI. Learning to reason with llms, 2024. URL https://openai.com/index/ learning-to-reason-with-llms/

    2024

  31. [39]

    Learning to reason with llms, 2025

    OpenAI. Learning to reason with llms, 2025. URLhttps://openai.com/index/openai-o3-mini/

    2025

  32. [40]

    Vicky Zhao, Lili Qiu, and Dongmei Zhang

    Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. Llmlingua-2: Data distillation for efficient and faithful task-agnostic prompt compression. In Lun-Wei...

    2024

  33. [41]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling, 2024

    2024

  34. [42]

    Sparq attention: Bandwidth-efficient llm inference.arXiv preprint arXiv:2312.04985, 2023

    Luka Ribar, Ivan Chelombiev, Luke Hudlass-Galley, Charlie Blake, Carlo Luschi, and Douglas Orr. Sparq attention: Bandwidth-efficient llm inference.arXiv preprint arXiv:2312.04985, 2023

    2023 arXiv

  35. [43]

    Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9):99–106, 2021

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9):99–106, 2021

    2021

  36. [44]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  37. [45]

    Flashattention-3: Fast and accurate attention with asynchrony and low-precision.CoRR, abs/2407.08608, 2024

    Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision.CoRR, abs/2407.08608, 2024. doi: 10.48550/ARXIV. 2407.08608. URL https://doi.org/10.48550/arXiv.2407.08608

    2024 doi arXiv

  38. [46]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  39. [47]

    Sparsebert: Rethinking the importance analysis in self-attention

    Han Shi, Jiahui Gao, Xiaozhe Ren, Hang Xu, Xiaodan Liang, Zhenguo Li, and James Tin-Yau Kwok. Sparsebert: Rethinking the importance analysis in self-attention. In Marina Meila and Tong Zhang, editors,Proceedings of the 38th International Conference on Machine Learning, ICML 20...

    2021

  40. [48]

    Llm pretraining with continuous concepts.arXiv preprint arXiv:2502.08524, 2025

    Jihoon Tack, Jack Lanchantin, Jane Yu, Andrew Cohen, Ilia Kulikov, Janice Lan, Shibo Hao, Yuandong Tian, Jason Weston, and Xian Li. Llm pretraining with continuous concepts.arXiv preprint arXiv:2502.08524, 2025

    2025 arXiv

  41. [49]

    Commonsenseqa: A question answering challenge targeting commonsense knowledge.arXiv preprint arXiv:1811.00937, 2018

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge.arXiv preprint arXiv:1811.00937, 2018

    2018 arXiv

  42. [50]

    Quest: Query-aware sparsity for efficient long-context llm inference

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context llm inference. InInternational Conference on Machine Learning, pages 47901–47911. PMLR, 2024

    2024

  43. [51]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  44. [52]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  45. [53]

    Spatten: Efficient sparse attention architecture with cascade token and head pruning

    Hanrui Wang, Zhekai Zhang, and Song Han. Spatten: Efficient sparse attention architecture with cascade token and head pruning. InIEEE International Symposium on High-Performance Computer Architecture, HPCA 2021, Seoul, South Korea,February27 - March3, 2021, pages 97–110. IEEE,...

    2021

  46. [54]

    Openhands: An open platform for ai software developers as generalist agents

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. In The Thirteenth International Conference on Learning Representa...

    2024

  47. [55]

    Chain-of-thought prompting elicits reasoning in large language models.Advancesin neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advancesin neural information processing systems, 35:24824–24837, 2022

    2022

  48. [56]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. InThe TwelfthInternational Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URLhttps://openre...

    2024

  49. [57]

    Native sparse attention: Hardware-aligned and natively trainable sparse attention

    Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv preprint arXiv:2502.11089, 2025

    2025 arXiv

  50. [58]

    Big bird: Transformers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Flo...

    2020

  51. [59]

    Quiet-star: Language models can teach themselves to think before speaking

    Eric Zelikman, Georges Raif Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah Goodman. Quiet-star: Language models can teach themselves to think before speaking. InFirst Conference on Language Modeling, 2024

    2024

  52. [60]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  53. [61]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Syste...

    2023

  54. [2023]

    Accessed: 2024-9-29

    URL https://crfm.stanford.edu/2023/10/12/flashdecoding.html. Accessed: 2024-9-29

    2023

  55. [2024]

    URL https://doi.org/10.48550/arXiv.2404.14469

    doi: 10.48550/ARXIV.2404.14469. URL https://doi.org/10.48550/arXiv.2404.14469

    doi arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.