Pith. sign in

REVIEW 3 major objections 5 minor 31 references

Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that depth-adaptive looped language models — which spend fewer loops on easy tokens — can be served at up to 99% of the theoretical speed-up, once inference schedules each loop iteration instead of the whole forward pass.

desk verdict First real serving stack for depth-adaptive looped LMs, but the Ouro speedup claim rests on a cache layout that destroys the model's accuracy; Huginn results are the solid half. read the letter →

arxiv 2608.09444 v1 pith:7M63IHHA submitted 2026-08-10 cs.LG cs.CLcs.DC

Kristian Schwethelm , Daniel Rueckert , Georgios Kaissis This is my paper · ORCID
classification cs.LGcs.CLcs.DC
keywords loopedlanguagemodelsdepth-adaptiveinferencecontinuousdepthbatchingearly-exitschedulingKVcachedesignLLMservingsystemsrooflinemodeladaptivecompute
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Looped language models promise depth-adaptive inference: by iterating a shared block of layers a variable number of times per token, easy tokens consume less compute than hard ones. The paper's claim is that this promise has been blocked by the serving stack, not by the models, because variable-depth tokens break the uniform forward pass that standard batching requires. It introduces continuous depth batching (CDB), which schedules each loop iteration as its own unit, routes prelude and coda stages through separate queues, and hides scheduling behind GPU execution with a lookahead gate. On Ouro 1.4B and Huginn 3.5B, the paper reports that CDB reaches up to 99% of the theoretical maximum speed-up from adaptive depth, delivering 1.5–1.9× higher offline throughput and 45–90% lower normalized latency under dynamic load. If the paper is right, depth-adaptive looped LMs become practical to serve, and the queue-based recipe transfers to any model with variable per-token depth.

What carries the argument

The load-bearing mechanism is a queue-based decode scheduler that treats the prelude, recurrent core, and coda as independent stages and batches tokens per loop iteration rather than per token generation. Three pieces carry the argument: the refill mode, which exploits the recurrent core's shared weights to admit new tokens into slots freed by early exits; the lookahead gate, which reads the exit signal one step early so batch preparation overlaps GPU execution, cutting measured per-step device idle time to 0.67%; and depth-aware KV caches — a last-exited layout that copies a token's final state into the depth slots it skipped, and a shared layout that keeps one KV slot per layer and overwrites it at every loop, trading attention semantics for an $r_{\max}$-fold memory saving. The analytical machinery is the FLOP bound $\frac{F_0 + r_{\max}F_r}{F_0 + \bar d F_r} \le \frac{r_{\max}}{\bar d}$, where $\bar d$ is the mean exit depth, together with the roofline model $t(N_c, N_b) = \delta + \gamma N_b + \alpha N_c$, whose saturation batch size $B^* = \delta/(\gamma + \alpha L)$ marks the boundary between the memory-bound regime where refill pays and the compute-bound regime where early exits save time directly.

What would settle it

Serve Ouro 1.4B under CDB with the first-then-shared or full depth-indexed KV cache — the layouts that keep GSM8K accuracy near 71–78% — and compare throughput and normalized latency against the same engine running full depth: if the reported 1.5–1.9× throughput gain and 45–90% latency reduction do not appear, the central claim fails for the model on which it is strongest. A second check is to measure the recurrent-step latency curve on hardware with much cheaper weight reload and confirm whether the refill advantage disappears as $B^*$ shrinks, as the roofline model predicts.

Watch

Extended reading notes

Core claim

The paper's central claim is that depth-adaptive decoding of looped language models can be served efficiently, and that continuous depth batching is the first end-to-end implementation to do so. CDB decomposes generation into four queues — prefill, prelude, recurrent core, and coda — and schedules at the granularity of individual loop iterations, so tokens at different depths share the same forward pass even though they exit at different times. A lookahead gate makes the exit decision one loop step in advance, giving the CPU a full step to prepare the next batch while the GPU runs the current one, and a refill mode fills slots freed by early exits with fresh tokens to keep the recurrent batch large. Evaluating with exit traces replayed from Alpaca and ShareGPT, the paper finds that CDB realizes up to 99% of the theoretical FLOP-bound speed-up on Ouro 1.4B and Huginn 3.5B, translating to 1.5–1.9× higher offline throughput than continuous batching at full depth and 45–90% lower normalized latency under Poisson arrivals. The paper further derives a FLOP-based upper bound and a roofline latency model that locate the benefit of depth adaptivity in the memory-bound regime, where a step's cost is dominated by reloading the recurrent core's weights rather than by batch size.

Load-bearing premise

The load-bearing premise is that the single-slot shared KV cache, used in every serving experiment, is an acceptable configuration for the benchmarked models — yet the paper's own Appendix C.1 shows this cache collapses Ouro's GSM8K accuracy from 77.86% to 0.23%, so the headline speed-ups may not hold under the cache layout that preserves Ouro's quality.

Editorial extensions

If this is right

  • On the fully looped Ouro architecture, CDB with refill reaches 94–99% of the FLOP bound and stays within 2% of the baseline when no early exit occurs, so the scheduler's own overhead is negligible.
  • On Huginn, with transformer layers in the prelude and coda, CDB still reaches up to 1.9× throughput but only 73–92% of the FLOP bound, so heavy boundary stages are the main structural factor that erodes adaptive-depth gains.
  • Refill's advantage is largest at batch sizes below $B^*$, where decode is memory-bound and shrinking the batch saves no wall-clock time; above $B^*$ the two modes converge, so no-refill is preferable when simplicity matters.
  • The lookahead gate imposes a minimum depth $r_{\min}=2$, which caps the achievable speed-up for models with small $r_{\max}$; the paper shows the one-step-earlier decision matches the original gate's accuracy.
  • CDB executes any given exit rule without changing model predictions, so the scheduler is compatible with learned gates, training-free convergence criteria, and depth-routing methods, not tied to a particular model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • All serving numbers use the single-slot shared KV cache; because the paper's own ablation shows that cache collapses Ouro's GSM8K accuracy from 77.86% to 0.23%, a direct test is CDB with a first-then-shared cache — which keeps accuracy near 71% — to see whether the 1.5–1.9× throughput gain survives an accuracy-preserving layout.
  • The roofline model implies a hardware prediction the paper does not test: on accelerators where weight reload is cheap (small $\delta$), $B^*$ shrinks, the memory-bound regime nearly disappears, and refill should lose most of its throughput advantage.
  • The paper replays fixed exit traces for reproducibility; in a live deployment the gate would react to queue state, and the interaction between exit-depth distribution and serving dynamics is not covered by the reported gains.
  • The stage-queue abstraction invites composition with chunked prefill and speculative decoding, but those combinations are untested; in particular, a speculative head would add a second per-token routing decision inside the loop that the lookahead gate would have to cover.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes continuous depth batching (CDB), a serving technique for depth-adaptive looped language models. CDB decomposes decode into separate prelude, recurrent-core, and coda queues, introduces depth-aware KV-cache layouts, and hides scheduling overhead with asynchronous batch preparation and a lookahead exit gate. The authors derive a FLOP-based upper bound on the speed-up from adaptive depth and a roofline latency model, then evaluate CDB on Ouro 1.4B and Huginn 3.5B, reporting up to 99% of the theoretical maximum speed-up, 1.5-1.9x offline throughput gains, and 45-90% lower normalized latency under dynamic serving load.

Significance. If the experimental claims held, this would be a meaningful systems contribution: it would show that depth-adaptive looped LMs can be served efficiently end to end and would provide a queue-based scheduling recipe applicable beyond the two benchmarked models. The paper has clear strengths: the FLOP bound in Eq. (2) is simple and internally consistent; the roofline model in Section 5 is fitted to careful latency measurements; the scheduler design addresses a real and previously unimplemented problem; and the authors evaluate two architecturally distinct looped LMs. However, the headline serving results are currently undermined by two experimental choices: the shared KV cache used for Ouro is shown in the paper itself to collapse Ouro's task accuracy, and exit decisions are replayed from precomputed traces rather than produced live by the gate. These issues are load-bearing for the speedup claims, so the paper requires major revision.

major comments (3)
  1. [Section 6.1 / Appendix C.1, Table 1] Section 6.1 states that 'Both models use a shared KV cache,' but Appendix C.1, Table 1 shows that the single-slot shared layout collapses Ouro's GSM8K accuracy from 77.86% to 0.23%. The Ouro throughput and latency results in Figures 5 and 6 are therefore measured on a configuration of Ouro that does not perform the benchmarked task, and the abstract's 1.5-1.9x speedup claim cannot be assumed to transfer to the actual Ouro model. The paper should either re-run the serving experiments with a cache layout that preserves accuracy (e.g., full depth-indexed or first-then-shared, which retains 71.34%), or report accuracy for the served configuration and explicitly argue why the speedup transfers despite the accuracy collapse.
  2. [Appendix B.2 / Section 4.3] Appendix B.2 states that 'we do not use the exit gate during our experiments' and that per-token exits are recorded once in advance and then replayed. This means the lookahead gate described in Section 4.3 is not exercised end to end in the serving benchmarks; the reported throughput, latency, and 0.67% idle-time figures characterize trace-replay execution only, not live adaptive inference. Because per-token exit decisions are the defining adaptive component of CDB, the current experiments cannot validate the claim that CDB realizes the adaptive-depth speedup. Please run the serving experiments with live gate decisions under the same KV-cache layout used for serving, or clearly separate trace-replay benchmarks from live-gate validation and temper the claims accordingly.
  3. [Section 6.2 / Eq. (2)] The comparison to the FLOP bound in Section 6.2, where CDB is reported to reach 94-99% of the theoretical maximum for Ouro, uses the upper bound of Eq. (2), which by construction ignores prefill, scheduling, KV-cache overhead, and boundary-stage costs. Combined with trace replay, this metric mostly characterizes queue mechanics rather than adaptive inference. The paper should state explicitly that Eq. (2) is an idealized decode-only reference, and it should report the actual mean exit depth and the accuracy of the served model under the benchmarked configuration, since the bound's value depends directly on \bar{d}.
minor comments (5)
  1. [General] No code or artifact link is provided; for a systems paper whose main contribution is an implementation, releasing the scheduler, kernels, and replay traces would substantially aid reproducibility.
  2. [Figure 3] The definition of 'per-step device idle (%)' and the measurement conditions should be clarified, including whether the reported 0.67% idle time was measured with live gate decisions or with replayed traces.
  3. [Section 6.2] The paper notes that the minimum coda batch size for Huginn 'should thus scale with the decode batch size,' but this parameter is not swept; a sensitivity analysis would clarify how robust the Huginn speedups are to this choice.
  4. [Appendix C.1] The catastrophic Ouro accuracy drop under single-slot sharing (0.23%) is only in the appendix; it should be highlighted in the main text wherever the shared cache is described as the standard serving configuration.
  5. [Section 5] The roofline model fit in Appendix A is reported as essentially perfect (R^2 >= 0.999) for the latency-versus-depth fit; reporting confidence intervals or residual plots for the batch-size fits would strengthen the claim that B* is estimated reliably.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: the headline speedups are benchmarked against an independent FLOP bound, the fitted roofline only explains refill's regime, and the sole self-citation is non-load-bearing.

full rationale

CDB's headline speedup is not circular. The FLOP bound in Eq. (2) is a derived upper bound from the model's layer structure and the replayed per-token exit depths; the reported '94-99% of the bound' is a ratio of independently measured throughput to that bound, not a fit of the bound to the measured speedup. The roofline model of Section 5 is fitted to recurrent-step latencies but is used only to explain when refill helps; the offline and online results are measured against a shared CB baseline with identical kernels and admission policy, so the main claims do not reduce to the roofline fit. The lookahead gate's validation in Appendix C.2 is a distillation check against teacher exit decisions, and the main serving experiments replay recorded exit decisions rather than relying on the lookahead head's accuracy, so no serving result is forced by construction. The only self-citation (Schwethelm et al., 2026, used in Discussion to note that compute-optimal models use fewer loops) is a side remark and does not carry the derivation. The shared-KV-cache accuracy collapse documented in Appendix C.1 is a real validity and generalizability concern for Ouro, but it is an experimental gap, not a circular reduction of the scheduler's claims to their inputs.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on stage-separable decode, depth-mixed batching, the one-step lookahead gate, a fitted roofline latency model, and the use of a shared KV cache. The last premise is the least supported: the paper's own appendix shows it destroys Ouro's accuracy, so the headline speedups are measured on a configuration that is not the accurate Ouro model.

free parameters (4)
  • Roofline intercept δ (weight-reload cost). = ≈5.3 ms (Ouro 1.4B), ≈10.6 ms (Ouro 2.6B), ≈2.3 ms (Huginn 3.5B).
    Fitted from measured recurrent-step latency in Appendix A; determines B* and the memory-bound regime where refill is claimed to help.
  • Roofline per-token compute slope γ. = Not reported numerically.
    Fitted slope of t(B) in Appendix A; used with δ and α to compute B*.
  • Roofline KV streaming slope α. = Not reported numerically.
    Fitted slope versus context length; drives the decrease of B* with L.
  • Minimum coda batch size for Huginn. = 32.
    Hand-chosen in Section 6.1 to limit boundary-stage overhead; affects refill behavior and the reported speedups.
assumptions (5)
  • domain assumption Looped LM decode can be decomposed into prelude, recurrent core, and coda stages that run as separate GPU operations without changing the model's predictions.
    Used throughout Sections 2 and 4.1; the scheduler's correctness depends on stage separability.
  • domain assumption Tokens at different loop depths can be batched together in one recurrent core step because the core shares weights across depths.
    Section 4.1; refill mode needs this to fill freed slots with new tokens at different loop progress.
  • domain assumption Exit decisions can be made one loop step in advance with a lookahead gate without changing the exit distribution or accuracy.
    Section 4.3 and Appendix C.2; the low-overhead asynchronous scheduler depends on this, and it is only validated empirically on two models.
  • ad hoc to paper The recurrent-step latency follows the additive roofline model t(B)=δ+(γ+αL)B.
    Section 5 and Appendix A; used to define B* and the refill benefit, with δ, γ, α fitted to the same data.
  • ad hoc to paper The single-slot shared KV cache is an acceptable serving configuration for both benchmarked models.
    Section 6.1; Appendix C.1 contradicts this for Ouro (0.23% vs 77.86% GSM8K), so the premise is load-bearing and currently unsupported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching." pith.science (2026). https://pith.science/paper/7M63IHHA

@misc{pith2026260809444,
  author       = {Pith},
  title        = {Pith review of: Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7M63IHHA}},
  note         = {Machine review of arXiv:2608.09444}
}
abstract

A main promise of looped language models (LMs) is depth-adaptive inference. By iterating a block of shared layers a variable number of times, the model can use less compute for "easy" tokens and more for "hard" ones. However, this adaptivity breaks standard batching: tokens in the same batch now require a different number of loops, so there is no unified forward pass, making efficient inference difficult. Standard inference frameworks like vLLM schedule on the token level and cannot handle this because tokens need to be removed from the batch within the forward pass. Loop-level scheduling has been proposed as a solution, but never implemented end to end. The key challenge is that looped architectures also contain non-looped boundary stages (e.g., token embedding and LM head) that must be scheduled at different frequencies than the loop. We introduce continuous depth batching (CDB), which schedules at the granularity of individual loop iterations. CDB handles boundary stages and loop steps in separate priority queues, makes exit decisions one step ahead, and overlaps all scheduling work with GPU computation. On Ouro 1.4B and Huginn 3.5B, CDB can realize up to $99\%$ of the theoretical maximum speed-up from adaptive-depth, translating to $1.5$-$1.9\times$ higher offline throughput and $45$-$90\%$ lower normalized latency under dynamic serving load.

Figures

Figures reproduced from arXiv: 2608.09444 by the authors.

Figure 1
Figure 1. Left: looped LM decoding three tokens xt, each using a different number of loops rt. Every decode step runs the prelude Pθ once, loops the recurrent core up to rmax = 3 times, and finally runs the coda Cθ, which samples the next token. Right: three requests under CDB with two batch slots and depth-adaptive looping. Cell color identifies the sequence and darker tints mark later tokens. Refill fills slots freed by an … view at source ↗
Figure 2
Figure 2. Left: we decompose generation into four queues: prefill for waiting requests, and prelude, recurrent core, and coda for decode. After each loop step, the exit decision routes tokens either back to the recurrent queue or out to the coda. Right: two depth-aware KV-cache designs for three tokens that exit at depths (r1, r2, r3) = (2, 1, 3). Each cell shows the loop step whose state is stored. 4 METHOD: CONTINUOUS DEPTH… view at source ↗
Figure 3
Figure 3. Left: the CPU prepares the next batch (N+1) from the previous step’s exit decision (N−1) while the GPU runs the current loop step in parallel (N). We also overlap the prelude and coda on a second GPU stream. Right: without CUDA graphs, the GPU idles 40% of the time. Static shapes reduce idle time by 28.9 percentage points, and the lookahead gate removes a further 10.4 percentage points, leaving only 0.67% idle time.… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Decode latency of a recurrent step versus batch size at context lengths [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Offline throughput normalized to CB (full depth) for Alpaca (top) and ShareGPT (bottom). [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Normalized latency against request rate at two exit thresholds per model, on Alpaca (top) [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Decode latency of a recurrent step versus batch size at context lengths [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Empirical CDFs of prompt (left) and output (right) token lengths for the two workloads [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Exit-depth distributions for different gate thresholds. Each cell shows the fraction of tokens [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Depth-adaptive decoding against fixed-depth decoding on GSM8K (flexible match, greedy [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 11 canonical work pages

  1. [1]

    The case for co-designing model architectures with hardware

    Quentin Anthony, Jacob Hatef, Deepak Narayanan, Stella Biderman, Stas Bekman, Junqi Yin, Aamir Shafi, Hari Subramoni, and Dhabaleswar Panda. The case for co-designing model architectures with hardware. In Proceedings of the 53rd International Conference on Parallel Processing, ICPP '24, pp.\ 84–96, New York, NY, USA, 2024. Association for Computing Machin...

    arXiv 2024

  2. [2]

    Relaxed recursive transformers: Effective parameter sharing with layer-wise lora, 2025 a

    Sangmin Bae, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Seungyeon Kim, and Tal Schuster. Relaxed recursive transformers: Effective parameter sharing with layer-wise lora, 2025 a . URL https://arxiv.org/abs/2410.20672

    arXiv 2025

  3. [3]

    Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation

    Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Aaron Courville, and Se-Young Yun. Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025 b . URL https://openreview...

    2025

  4. [4]

    Pondernet: Learning to ponder

    Andrea Banino, Jan Balaguer, and Charles Blundell. Pondernet: Learning to ponder. In 8th ICML Workshop on Automated Machine Learning (AutoML), 2021. URL https://openreview.net/forum?id=1EuxRTe0WN

    2021

  5. [5]

    Scaling laws meet model architecture: Toward inference-efficient LLM s

    Song Bian, Tao Yu, Shivaram Venkataraman, and Youngsuk Park. Scaling laws meet model architecture: Toward inference-efficient LLM s. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=0TmVqOpBbK

    work page 2026

  6. [6]

    bViT: Investigating Single-Block Recurrence in Vision Transformers for Image Recognition

    Michal Byra, Pawel Olszowiec, Grzegorz Stefanski, Grzegorz Gruszczynski, and Alberto Presta. bvit: Investigating single-block recurrence in vision transformers for image recognition, 2026. URL https://arxiv.org/abs/2605.10661

    work page Pith review arXiv 2026

  7. [7]

    Training verifiers to solve math word problems, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168

    arXiv 2021

  8. [8]

    Adaptive loops and memory in transformers: Think harder or know more? In Workshop on Latent & Implicit Thinking Going Beyond CoT Reasoning , 2026

    Markus Frey, Behzad Shomali, Ali Hamza Bashir, David Berghaus, Joachim Koehler, and Mehdi Ali. Adaptive loops and memory in transformers: Think harder or know more? In Workshop on Latent & Implicit Thinking Going Beyond CoT Reasoning , 2026. URL https://openreview.net/forum?id=F87X9c107e

    work page 2026

Show all 31 references
  1. [9]

    Think-at-hard: Selective latent iterations to improve reasoning language models, 2025

    Tianyu Fu, Yichen You, Zekai Chen, Guohao Dai, Huazhong Yang, and Yu Wang. Think-at-hard: Selective latent iterations to improve reasoning language models, 2025. URL https://arxiv.org/abs/2511.08577

    2025 arXiv

  2. [10]

    Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein

    Jonas Geiping, Sean Michael McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. In The Thirty-ninth Annual Conference ...

    2025

  3. [11]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda ...

    2025

  4. [12]

    Loop, think, & generalize: Implicit reasoning in recurrent-depth transformers, 2026

    Harsh Kohli, Srinivasan Parthasarathy, Huan Sun, and Yuekun Yao. Loop, think, & generalize: Implicit reasoning in recurrent-depth transformers, 2026. URL https://arxiv.org/abs/2604.07822

    2026 arXiv

  5. [13]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, S...

    2023

  6. [14]

    Hu, and Jonathan May

    Ryan Lee, Jacob Biloki, Edward J. Hu, and Jonathan May. Sparse layers are critical to scaling looped language models, 2026. URL https://arxiv.org/abs/2605.09165

    2026 arXiv

  7. [15]

    Ponderlm-3: Adaptive token-wise pondering with differentiable masking, 2026

    He Li, Feichen Song, Boyi Zeng, Shixiang Song, Zhiqin John Xu, Ziwei He, and Zhouhan Lin. Ponderlm-3: Adaptive token-wise pondering with differentiable masking, 2026. URL https://arxiv.org/abs/2603.02023

    2026

  8. [16]

    Co TF ormer: A chain of thought driven architecture with budget-adaptive computation cost at inference

    Amirkeivan Mohtashami, Matteo Pagliardini, and Martin Jaggi. Co TF ormer: A chain of thought driven architecture with budget-adaptive computation cost at inference. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?i...

    2025

  9. [17]

    Mixture-of-depths: Dynamically allocating compute in transformer-based language models, 2024

    David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically allocating compute in transformer-based language models, 2024. URL https://arxiv.org/abs/2404.02258

    2024 arXiv

  10. [18]

    How much is one recurrence worth? iso-depth scaling laws for looped language models, 2026

    Kristian Schwethelm, Daniel Rueckert, and Georgios Kaissis. How much is one recurrence worth? iso-depth scaling laws for looped language models, 2026. URL https://arxiv.org/abs/2604.21106

    2026 arXiv

  11. [19]

    Flashattention-3: Fast and accurate attention with asynchrony and low-precision

    Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.ne...

    2024

  12. [20]

    ShareGPT , 2023

    ShareGPT Team . ShareGPT , 2023. URL https://sharegpt.com

    2023

  13. [21]

    Loopvit: Scaling visual arc with looped transformers, 2026

    Wen-Jie Shu, Xuerui Qiu, Rui-Jie Zhu, Harold Haodong Chen, Yexin Liu, and Harry Yang. Loopvit: Scaling visual arc with looped transformers, 2026. URL https://arxiv.org/abs/2602.02156

    2026

  14. [22]

    Adaponderlm: Gated pondering language models with token-wise adaptive depth, 2026

    Shixiang Song, He Li, Zitong Wang, Boyi Zeng, Feichen Song, Yixuan Wang, Zhiqin John Xu, Ziwei He, and Zhouhan Lin. Adaponderlm: Gated pondering language models with token-wise adaptive depth, 2026. URL https://arxiv.org/abs/2603.01914

    2026

  15. [23]

    Michaelov, Chris, Chessing234, Hanwool Albert Lee, Janna, Leonid Sinev, Khalid, Kiersten Stokes, and Zdeněk Kasner

    Lintang Sutawika, Hailey Schoelkopf, Leo Gao, Baber Abbasi, Stella Biderman, Jonathan Tow, ben fattori, Charles Lovering, farzanehnakhaee70, Jason Phang, Anish Thite, Fazz, Aflah, Niklas, Thomas Wang, sdtblck, nopperl, gakada, researcher2, tttyuntian, Julen Etxaniz, James A. M...

    2026 doi

  16. [24]

    Sparse universal transformer

    Shawn Tan, Yikang Shen, Zhenfang Chen, Aaron Courville, and Chuang Gan. Sparse universal transformer. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https://openreview.net/forum?id=yXYJPAlLqn

    2023

  17. [25]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following LLaMA model, 2023. URL https://github.com/tatsu-lab/stanford_alpaca

    2023

  18. [26]

    Recurrent-depth VLA : Implicit test-time compute scaling of vision-language-action models via latent iterative reasoning

    Yalcin Tur, Jalal Naghiyev, Haoquan Fang, Wei-Chuan Tsai, Jiafei Duan, Dieter Fox, and Ranjay Krishna. Recurrent-depth VLA : Implicit test-time compute scaling of vision-language-action models via latent iterative reasoning. In Workshop on Latent & Implicit Thinking Going Beyo...

    2026

  19. [27]

    Memory-efficient looped transformer: Decoupling compute from memory in looped language models, 2026

    Victor Conchello Vendrell, Arnau Padres Masdemont, Niccolò Grillo, Jordi Ros-Giralt, Arash Behboodi, and Fabio Valerio Massoli. Memory-efficient looped transformer: Decoupling compute from memory in looped language models, 2026. URL https://arxiv.org/abs/2605.07721

    2026 arXiv

  20. [28]

    Roofline: an insightful visual performance model for multicore architectures

    Samuel Williams, Andrew Waterman, and David Patterson. Roofline: an insightful visual performance model for multicore architectures. Commun. ACM, 52 0 (4): 0 65–76, April 2009. ISSN 0001-0782. doi:10.1145/1498765.1498785. URL https://doi.org/10.1145/1498765.1498785

    2009

  21. [29]

    Transformers: State-of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020

  22. [30]

    Orca: A distributed serving system for Transformer-Based generative models

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for Transformer-Based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp.\ 521--538, Carlsbad, CA, July 2022....

    2022

  23. [31]

    Scaling latent reasoning via looped language models, 2025

    Rui-Jie Zhu, Zixuan Wang, Kai Hua, Tianyu Zhang, Ziniu Li, Haoran Que, Boyi Wei, Zixin Wen, Fan Yin, He Xing, Lu Li, Jiajun Shi, Kaijing Ma, Shanda Li, Taylor Kergan, Andrew Smith, Xingwei Qu, Mude Hui, Bohong Wu, Qiyang Min, Hongzhi Huang, Xun Zhou, Wei Ye, Jiaheng Liu, Jian ...

    2025 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.