Pith. sign in

REVIEW 3 major objections 5 minor 44 references

StagFormer: Time Staggering Transformer Decoding for RunningLayers In Parallel

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read StagFormer matches a 36-layer model while running two 18-layer stacks in parallel.

desk verdict A genuinely new way to trade one token of context for depth-parallel decoding, with surprisingly clean quality results at 300B tokens, but the latency headline is a simulation that drops inter-chip communication. read the letter →

arxiv 2501.15665 v2 pith:NLJYUII4 submitted 2025-01-26 cs.LG cs.AI

Dylan Cutler , Arun Kandoor , Nishanth Dikkala , Nikunj Saunshi , Xin Wang , Rina Panigrahy This is my paper · ORCID
classification cs.LGcs.AI
keywords StagFormerstaggeredtransformerparalleldecodingcross-attentioninferencelatencydepthparallelismlanguagemodelingweightsharing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

StagFormer is a decoder-only Transformer variant that breaks the strict sequential dependency of layers during decoding. At time step i, the upper stack is forbidden to use the lower stack's representation of token i and instead cross-attends only to the lower stack's representations of tokens 1 through i-1, while receiving the raw embedding of token i directly. That one-step lag lets the two stacks execute in parallel on separate accelerators, so the effective latency of a deep model approaches that of a model with half the layers. The paper reports that a separate-weights StagFormer with two 18-layer stacks matches or beats a 36-layer baseline on Pile perplexity and downstream average, and that variants with shared weights, local cross-attention, more stacks, and recurrent decoding each trade quality, memory, and speed differently.

What carries the argument

The central mechanism is the staggered dependency between stacks. With two stacks, the first stack processes the current token normally, but the second stack computes its representation from the token embedding plus cross-attention to the first stack's representations of tokens up to i-1, never to the current token's first-stack representation. This removes the data dependency that forces layers to run sequentially, because while the second stack finishes predicting the next token, the first stack is already computing representations for that token. The price is extra cross-attention parameters and a 50 percent larger KV cache for separate weights, and a tripled KV cache for shared weights.

What would settle it

Run the two stacks on two physical accelerators and measure end-to-end tokens per second including cross-stack activation transfer and duplicated embedding and softmax tables; if the observed per-token latency is not close to roughly half the baseline's, the paper's central latency claim fails. Separately, train a variant that removes the one-step lag by letting the upper stack attend to the current token's first-stack representation; if that variant is markedly better, the claimed quality parity is partly a product of the added cross-attention parameters rather than the stagger itself.

Watch

Extended reading notes

Core claim

The central claim is that the usual all-layers-before-next-token dependency is not necessary for quality: a Transformer can be partitioned into stacks that cross-attend to the previous stack's activations with a one-time-step lag, and this staggering makes decoding depth-parallel while staying quality neutral. In the paper's main comparison, a separate-weights StagFormer with two 18-layer stacks reaches 3.756 Pile perplexity versus 3.780 for the 36-layer baseline and a 47.0 average downstream score versus 45.0, with strong gains on SQuADv2, Lambada, and HellaSwag and neutral results on SuperGLUE. The same idea extends to shared weights, where two passes through one 18-layer network close much of the gap between the 18-layer and 36-layer baselines at roughly the 18-layer parameter count, and to local cross-attention, where a window of 512 keeps quality while a window of 1 collapses it.

Load-bearing premise

The headline latency gain assumes the two stacks can run in parallel with negligible inter-chip communication; if copying activations and duplicated tables between chips is slow, the speedup shrinks even though quality parity may survive.

Editorial extensions

If this is right

  • A StagFormer with two stacks can be decoded with wall-clock latency closer to a model with half the layers when the stacks run on separate chips, because at each time step the two halves work simultaneously.
  • The one-step cross-stack delay is not fatal for quality: the separate-weights model surpasses the same-depth baseline on most measured tasks, so the added cross-attention can substitute for the missing same-token representation.
  • With shared weights, StagFormer becomes a parameter-efficient way to add effective depth, closing much of the quality gap an 18-layer Transformer has against a 36-layer one while adding only about 0.2 billion parameters.
  • Bounded cross-attention windows are a workable memory-latency knob: a window of 512 preserves quality, but a window of 1 degrades it sharply, so the upper stack needs multi-position context.
  • Scaling to more than two stacks lowers quality, though learning a linear combination of each stack's output lets p=3 and p=4 models stay competitive on some tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One implication the authors leave implicit is that the same staggering recipe could be applied to unequally sized stacks or to individual layer groups, enabling finer-grained pipeline parallelism with adjustable delay budgets.
  • The quality parity suggests a testable hypothesis about Transformer internals: the lower layers' representation of the current token may be largely redundant for next-token prediction once prior-token hidden states are available.
  • On real distributed hardware the speedup is bounded by inter-chip transfer and single-program-multiple-data overhead; a fair comparison should measure end-to-end tokens per second with the model sharded across two devices, including activation copies and duplicated embedding and softmax tables.
  • The recurrent shared-weights variant points toward a middle path between Transformers and recurrent networks; a natural extension would train with a fixed-size compressed hidden state instead of a growing cross-attention KV cache.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes StagFormer, a decoder-only Transformer whose layers are partitioned into p stacks; the upper stacks receive token embeddings for the current position and cross-attend to the previous stack's representations only up to position i-1, thereby breaking the per-token sequential dependency along depth. At decode time the stacks can execute in parallel at the cost of a one-token delay in cross-stack information. The authors pretrain separate-weights (2.9B) and shared-weights (1.8B) variants on The Pile for roughly 300B tokens and report that the p=2 separate-weights model matches or slightly outperforms a 2.8B 36-layer baseline on Pile perplexity and downstream tasks, while a simulated latency benchmark shows decoding gains. Extensions include local cross-attention, p>2 staggering with learned stack mixing, and a recurrent inference approximation for the shared-weights variant.

Significance. If the quality-neutrality result holds, StagFormer offers a genuinely different way to parallelize decoding along the depth axis, complementary to speculative decoding and attention sparsification. The paper's quality evidence is substantial: large-scale pretraining against external baselines, with algorithms specified and no circular fitting. The central speedup claim, however, rests entirely on a simulation that explicitly ignores inter-chip communication and SPMD overhead; because the architecture's raison d'être is wall-clock latency, the contribution is not fully established until this is measured or modeled. The quality result is interesting enough to warrant a revision rather than rejection.

major comments (3)
  1. [Section 2, 'Quantifying the Latency Benefits'; Section 5] The speedup claim is based on a simulation that 'ignore[s] the inter-chip communication cost between the first and second stacks,' and Section 5 admits that this communication 'prevents one from realizing the full theoretical latency benefit' and that SPMD execution requires duplicating token embeddings and softmax tables. Since the central claimed benefit is wall-clock decoding speedup, a real two-device measurement, or at least a quantitative model of communication and SPMD overhead, is needed to establish the result. As written, Figure 3 reports an upper-bound scenario, not an achieved speedup. The comparison should also include a tensor-parallel baseline that uses the same two chips without any architectural change, to isolate the benefit of StagFormer.
  2. [Algorithm 1, Step 2; Algorithm 4, Step 2] The assignment 'u = 0 when j = h+1 and u = j otherwise' is not executable: for j = h+2, the representation t^{1,...,i}_j has not yet been computed when layer L'_j is applied. The correct recurrence is u = j-1, as written in Algorithm 2. The same typo appears in Algorithm 4 for the p>2 variant. As printed, the core pseudocode cannot be run, so the experiments as described do not correspond to an executable algorithm; this must be fixed for reproducibility.
  3. [Section 3.2 and Figure 3] The text says 'The analysis is presented in Table 3,' but Table 3 reports shared-weights recurrent inference quality results, not latency; Figure 3 is the latency plot. This cross-reference error obscures the only latency evidence in the paper. Please correct the reference and describe the simulation setup (chip model, arithmetic intensity, batch size, communication assumptions) in the main text rather than referring to a table that does not contain the latency analysis.
minor comments (5)
  1. [Figure 3 caption] The caption reads 'StagFormer is able to decode 2,048 tokens than baseline'; a word such as 'faster' is missing.
  2. [Section 4.3] 'imploring a simple change' should be 'employing a simple change'.
  3. [Tables 1, 2, 4-7] All quality numbers appear to come from single training runs with no error bars or seeds. Reporting variance or at least stating that these are single runs would help the reader assess the 'match or outperform' claim.
  4. [Section 3.2] The phrase 'latency benchmarking results on accelerator hardware' overstates what is presented; the results are from a simulation that ignores inter-chip communication. Please rephrase to 'simulated latency benchmarking'.
  5. [Section 5] The limitation discussion is candid and welcome, but it should be moved earlier or at least echoed where the latency speedup is first claimed, so that the reader does not encounter the unqualified speedup claim in the Introduction and Section 2 without the caveat.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: quality is externally benchmarked and the latency savings are an analytical FLOP bound with clearly disclosed communication-cost caveats.

full rationale

The paper's central claims are supported by external empirical benchmarks and explicit arithmetic, not by definitions or fitted inputs. Quality neutrality rests on Table 1, where StagFormer p=2 is compared against independently trained 18-layer and 36-layer baselines on Pile perplexity and standard downstream tasks (HellaSwag, ARC, WinoGrande, SuperGLUE, SQuADv2, GEM-XSum); these are measured outcomes, not parameters fitted to guarantee the conclusion. The latency analysis in Section 2 is an analytical FLOP count: the baseline forward pass is 2e + l(m+a) FLOPs and the StagFormer total is 2e + 3l(m+a)/2, with parallel execution reducing the wall-clock equivalent to 2e + l(m+a)/2 in an ideal scenario. This is a mathematical upper-bound calculation, not a prediction manufactured from StagFormer's own outputs. The paper explicitly discloses that the latency simulation 'ignore[s] the inter-chip communication cost between the first and second stacks' and Section 5 concedes that communication 'prevents one from realizing the full theoretical latency benefit'; this is a stated limitation of an empirical/simulated speedup claim, not a circular step. The learnable coefficients alpha and the extra cross-attention weights are trained model parameters, and their existence does not by construction produce the reported quality or latency numbers. Self-citations (Baykal et al. 2024, Panigrahi et al. 2024, Gatmiry et al. 2024, Saunshi et al. 2025) appear only in related-work or comparison contexts and are not load-bearing for the architecture's claimed validity. No step in the paper reduces, by definition or by self-citation, to its own inputs.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on standard transformer machinery, on the empirical premise that delayed cross-attention does not degrade quality, and on the hardware assumption that parallel stacks can run with negligible communication overhead. Quality results are single-run empirical observations without error bars, and no code or checkpoints are released.

free parameters (3)
  • Stack-output mixing coefficients alpha_k (p > 2 variant) = learned during pretraining
    Algorithm 4 combines the outputs of the p stacks with learnable scalars; these are fitted to training data and affect p=3 and p=4 results.
  • Cross-attention projection weights in upper stacks = learned during pretraining
    The quality-neutrality result depends on additional cross-attention parameters in the second stack offsetting the one-step information delay; these are trained model parameters.
  • Local cross-attention window size = chosen by authors (512, 128, 1)
    Window sizes are handpicked in Section 4.2; Tables 4 and 5 show quality depends strongly on this choice.
assumptions (5)
  • domain assumption Standard Transformer layer behavior (self-attention, feed-forward, residual connections, layer norm, causal masking) is as described.
    Invoked throughout Section 2 and Algorithm 1 as the building block for both stacks.
  • ad hoc to paper Two stacks can be executed in parallel on separate accelerators with negligible inter-chip communication cost.
    Used in the simulated latency analysis in Section 2, which explicitly ignores inter-chip communication, and acknowledged as unrealized in Section 5.
  • domain assumption A one-step delay in cross-stack attention does not degrade quality beyond what extra cross-attention parameters can offset.
    Central design premise; supported only by single-run empirical tables, not by a derivation.
  • domain assumption Sequential training with causal masking on the cross-attention path transfers to parallel decoding at inference.
    Algorithm 1 trains with two sequential passes; decode runs stacks in parallel; the paper assumes the two behaviors match.
  • domain assumption The Pile dataset and the chosen downstream benchmarks are sufficient measures of language modeling quality.
    Used in Section 3 to support the quality-neutrality claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of StagFormer: Time Staggering Transformer Decoding for RunningLayers In Parallel." pith.science (2026). https://pith.science/paper/NLJYUII4

@misc{pith2026250115665,
  author       = {Pith},
  title        = {Pith review of: StagFormer: Time Staggering Transformer Decoding for RunningLayers In Parallel},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NLJYUII4}},
  note         = {Machine review of arXiv:2501.15665}
}
abstract

Decoding in a Transformer based language model is inherently sequential as a token's embedding needs to pass through all the layers in the network before the generation of the next token can begin. In this work, we propose a new architecture StagFormer (Staggered Transformer), which staggers execution along the sequence axis and thereby enables parallelizing the decoding process along the depth of the model. We achieve this by breaking the dependency of the token representation at time step $i$ in layer $l$ upon the representations of tokens until time step $i$ from layer $l-1$. Instead, we stagger the execution and only allow a dependency on token representations until time step $i-1$. The later sections of the Transformer still get access to the "rich" representations from the prior section but only from those token positions which are one time step behind. StagFormer allows for different sections of the model to be executed in parallel yielding a potential speedup in decoding while being quality neutral in our simulations. We also explore many natural extensions of this idea. We present how weight-sharing across the different sections being staggered can be more practical in settings with limited memory. We explore the efficacy of using a bounded window attention to pass information from one section to another which helps drive further latency gains for some applications. We also explore the scalability of the staggering idea over more than 2 sections of the Transformer. Finally, we show how one can approximate a recurrent model during inference using weight-sharing. This variant can lead to substantial gains in quality for short generations while being neutral in its latency impact.

Figures

Figures reproduced from arXiv: 2501.15665 by the authors.

Figure 1
Figure 1. Depiction of forward pass in a standard Transformer compared with that of [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Depiction of the parallel execution of stacks T1 and T2 in a 2-stack StagFormer. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Simulated Latency Benchmarking for a baseline Transformer (dotted) vs a compa [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Plot of the training perplexity loss for the 18 layer baseline (black), 18 layer shared [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Timing Diagram of Prefill vs Decode steps for Recurrent Inference with Shared [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 7 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Alternating updates for efficient transformers

    Cenk Baykal, Dylan Cutler, Nishanth Dikkala, Nikhil Ghosh, Rina Panigrahy, and Xin Wang. Alternating updates for efficient transformers. Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  3. [3]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer, 2020. URL https://arxiv.org/abs/2004.05150

    arXiv 2020

  4. [4]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    arXiv 2020

  5. [5]

    Medusa: Simple llm inference acceleration framework with multiple decoding heads

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774, 2024

    arXiv 2024

  6. [6]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022

    arXiv 2022

  7. [7]

    Le, and Ruslan Salakhutdinov

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context, 2019. URL https://arxiv.org/abs/1901.02860

    arXiv 2019

  8. [8]

    Universal transformers

    Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and ukasz Kaiser. Universal transformers. arXiv preprint arXiv:1807.03819, 2018

    arXiv 2018

Show all 44 references
  1. [9]

    Bert: Pre-training of deep bidirectional transformers for language understanding, 2019

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL https://arxiv.org/abs/1810.04805

    2019 arXiv

  2. [10]

    The pile: An 800gb dataset of diverse text for language modeling, 2020

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020. URL https://arxiv.org/abs/2101.00027

    2020 arXiv

  3. [11]

    Reddi, Stefanie Jegelka, and Sanjiv Kumar

    Khashayar Gatmiry, Nikunj Saunshi, Sashank J. Reddi, Stefanie Jegelka, and Sanjiv Kumar. Can looped transformers learn to implement multi-step gradient descent for in-context learning? In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonatha...

    2024

  4. [12]

    Lee, and Dimitris Papailiopoulos

    Angeliki Giannou, Shashank Rajput, Jy yong Sohn, Kangwook Lee, Jason D. Lee, and Dimitris Papailiopoulos. Looped transformers as programmable computers, 2023. URL https://arxiv.org/abs/2301.13196

    2023 arXiv

  5. [13]

    Mamba: Linear-time sequence modeling with selective state spaces, 2024

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2024. URL https://arxiv.org/abs/2312.00752

    2024 arXiv

  6. [14]

    Efficiently modeling long sequences with structured state spaces, 2022

    Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces, 2022. URL https://arxiv.org/abs/2111.00396

    2022 arXiv

  7. [15]

    Training compute-optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  8. [16]

    Block-recurrent transformers, 2022

    DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur. Block-recurrent transformers, 2022. URL https://arxiv.org/abs/2203.07852

    2022 arXiv

  9. [17]

    Staircase attention for recurrent processing of sequences

    Da Ju, Stephen Roller, Sainbayar Sukhbaatar, and Jason E Weston. Staircase attention for recurrent processing of sequences. Advances in Neural Information Processing Systems, 35: 0 13203--13213, 2022

    2022

  10. [18]

    Matryoshka representation learning

    Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representation learning. Advances in Neural Information Processing Systems, 35: 0 30233--30249, 2022

    2022

  11. [19]

    Albert: A lite bert for self-supervised learning of language representations

    Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942, 2019

    1909 arXiv

  12. [20]

    Fast inference from transformers via speculative decoding, 2023

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding, 2023. URL https://arxiv.org/abs/2211.17192

    2023 arXiv

  13. [21]

    Sci-cot: Leveraging large language models for enhanced knowledge distillation in small models for scientific qa, 2023

    Yuhan Ma, Haiqi Jiang, and Chenyou Fan. Sci-cot: Leveraging large language models for enhanced knowledge distillation in small models for scientific qa, 2023. URL https://arxiv.org/abs/2308.04679

    2023 arXiv

  14. [22]

    Efficient stagewise pretraining via progressive subnetworks

    Abhishek Panigrahi, Nikunj Saunshi, Kaifeng Lyu, Sobhan Miryoosefi, Sashank Reddi, Satyen Kale, and Sanjiv Kumar. Efficient stagewise pretraining via progressive subnetworks. arXiv preprint arXiv:2402.05913, 2024

    2024 arXiv

  15. [23]

    Efficiently scaling transformer inference, 2022

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference, 2022. URL https://arxiv.org/abs/2211.05102

    2022 arXiv

  16. [24]

    Improving language understanding by generative pre-training

    Alec Radford. Improving language understanding by generative pre-training. 2018

    2018

  17. [25]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. URL https://arxiv.org/abs/1910.10683

    2023 arXiv

  18. [26]

    Know what you don't know: Unanswerable questions for squad, 2018

    Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad, 2018. URL https://arxiv.org/abs/1806.03822

    2018 arXiv

  19. [27]

    Winogrande: An adversarial winograd schema challenge at scale, 2019

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019. URL https://arxiv.org/abs/1907.10641

    2019 arXiv

  20. [28]

    Accelerating transformer inference for translation via parallel decoding

    Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Riccardo Marin, and Emanuele Rodol \`a . Accelerating transformer inference for translation via parallel decoding. arXiv preprint arXiv:2305.10427, 2023

    2023 arXiv

  21. [29]

    Reasoning with latent thoughts: On the power of looped transformers

    Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J Reddi. Reasoning with latent thoughts: On the power of looped transformers. arXiv preprint arXiv:2502.17416, 2025

    2025 arXiv

  22. [30]

    Blockwise parallel decoding for deep autoregressive models, 2018

    Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autoregressive models, 2018. URL https://arxiv.org/abs/1811.03115

    2018 arXiv

  23. [31]

    Roformer: Enhanced transformer with rotary position embedding, 2023

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023. URL https://arxiv.org/abs/2104.09864

    2023 arXiv

  24. [32]

    Spectr: Fast speculative decoding via optimal transport

    Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. Spectr: Fast speculative decoding via optimal transport. Advances in Neural Information Processing Systems, 36, 2024

    2024

  25. [33]

    Efficient transformers: A survey, 2022

    Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey, 2022. URL https://arxiv.org/abs/2009.06732

    2022 arXiv

  26. [34]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017. URL https://proceedings.neurips.cc/paper/2017/file/3f5e...

    2017

  27. [35]

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems, 2020. URL https://arxiv.org/abs/1905.00537

    2020 arXiv

  28. [36]

    Smoothquant: Accurate and efficient post-training quantization for large language models

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp.\ 38087--38099. PMLR, 2023

    2023

  29. [37]

    A survey on knowledge distillation of large language models

    Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116, 2024

    2024 arXiv

  30. [38]

    Gspmd: General and scalable parallelization for ml computation graphs, 2021

    Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, and Zhifeng Chen. Gspmd: General and scalable parallelization for ml c...

    2021 arXiv

  31. [39]

    Physics of language models: Part 2.1, grade-school math and the hidden reasoning process

    Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of language models: Part 2.1, grade-school math and the hidden reasoning process. arXiv preprint arXiv:2407.20311, 2024

    2024 arXiv

  32. [40]

    Hellaswag: Can a machine really finish your sentence?, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?, 2019. URL https://arxiv.org/abs/1905.07830

    2019 arXiv

  33. [41]

    Are more layers beneficial to graph transformers? In The Eleventh International Conference on Learning Representations, 2023

    Haiteng Zhao, Shuming Ma, Dongdong Zhang, Zhi-Hong Deng, and Furu Wei. Are more layers beneficial to graph transformers? In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=uagC-X9XMi8

    2023

  34. [42]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  35. [43]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  36. [44]

    hidden state

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.