RW-TTT: Batched Serving for Request-Owned Test-Time Training State

Han Chen; Hao Zhang; Jian Yang; Sirui Han; Yao Tian; Yike Guo; Zhizhuo Kou

RW-TTT: Batched Serving for Request-Owned Test-Time Training State

T0 review · 2 major / 0 minor · reviewed 2026-06-29 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read RW-TTT batches TTT decode steps across requests by tagging each with owner, version, and read/write effect, then commits updates only to the matching owner.

desk verdict RW-TTT gives a tagging scheme to batch per-request TTT updates safely enough to get 9x throughput on eight streams, but the safety rests on owner/version checks and one benchmark rather than stronger validation. read the letter →

arxiv 2605.28053 v1 pith:NZ264K2F submitted 2026-05-27 cs.LG

Jian Yang , Zhizhuo Kou , Yao Tian , Hao Zhang , Han Chen , Sirui Han , Yike Guo This is my paper · ORCID

classification cs.LG

keywords test-timetrainingbatchedLLMservingrequest-ownedstatefastweightsinferenceoptimization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Test-time training adapts an LLM by updating request-specific state such as fast weights during generation. Standard batching assumes shared static weights and therefore either runs requests serially or risks corrupting per-request state. The paper formulates the resulting constraint as read-write TTT serving and solves it with a tagging scheme that identifies compatible phases for simultaneous execution. On one GPU the method delivers 274.61 aggregate tokens per second across eight InPlace-TTT streams while matching sequential behavior on the RULER benchmark.

What carries the argument

Owner/version/READ-WRITE tagging scheme that restricts batch formation to non-conflicting phases and restricts commits to the matching owner.

What would settle it

An execution trace in which two requests pass the owner/version checks yet one request's final output differs from the output obtained by running the same requests sequentially.

Watch

Extended reading notes

Core claim

RW-TTT tags every decode step with its owner identifier, version counter, and READ or WRITE effect; it forms batches only from mutually compatible phases and commits each update exclusively to the owning request's state, thereby restoring safe batching without altering the underlying TTT algorithm.

Load-bearing premise

The owner and version tags together with selective commit are sufficient to prevent any cross-request state corruption.

Editorial extensions

If this is right

Eight concurrent fast-weight TTT streams run at 9.31 times the throughput of sequential execution under identical memory limits.
Per-stream replica replication is no longer required to achieve isolation, freeing memory that can be used for longer contexts or more streams.
Correctness on long-context tasks is preserved when the tagging rules are followed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same tagging discipline could be applied to other request-owned state updates such as online learning of low-rank adapters.
The approach may reduce the need for separate model replicas in multi-tenant serving environments that support stateful inference.
If the checks scale to larger batch sizes, the method could change the economics of deploying TTT-augmented models on shared hardware.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

read the letter

The main takeaway is that this paper solves a practical barrier in LLM serving: how to batch decode steps when each request maintains its own changing TTT state like fast weights or deltas. They tag every step with its owner, version, and read/write effect, batch only compatible phases, and commit writes only to the matching owner. On one GPU running eight InPlace-TTT streams this yields 274.61 aggregate tokens per second, 9.31 times sequential execution and 3.44 times separate replicas at the same memory budget, while RULER scores stay the same.

What is new is the concrete read-write TTT serving formulation plus the owner/version/READ-WRITE tagging rules that decide batch eligibility. Earlier serving work assumed static shared weights, so this directly targets the corruption risk that appears once state becomes request-owned.

The paper does a solid job turning the problem into an engineering fix and showing measurable speedups without obvious behavior change on their benchmark. The numbers are presented as direct measurements, not derived from fitted parameters.

The soft spot is the completeness of the compatibility rules. The validation consists of owner/version matching plus unchanged RULER results. That leaves open whether the tags catch every interleaving case at the level of batched GEMM kernels or version increments. No formal invariant, no exhaustive small-case enumeration, and no reduction to serializable schedules are mentioned, so the claim that selective commit prevents corruption rests on empirical checks rather than exhaustive argument. Experimental details on setup, variability, and edge cases would also need to be clearer in the full text.

This is for inference-systems people who want to add per-request adaptation to production serving. A reader working on that exact constraint would get concrete value from the tagging approach and the reported gains.

It deserves peer review because it identifies a real deployment obstacle and supplies a working mitigation with throughput evidence, even if the correctness argument could be strengthened.

Referee Report

2 major / 0 minor

Summary. The manuscript presents RW-TTT, a batched serving system for LLMs that perform request-owned test-time training (e.g., fast weights or low-rank deltas). It tags each decode step with owner, version, and READ/WRITE effect, batches only compatible phases, and commits updates only to the owning request. On one GPU with eight InPlace-TTT streams it reports 274.61 aggregate tok/s (9.31x over sequential serving, 3.44x over per-stream replicas under identical memory budget), while preserving RULER benchmark behavior and passing owner/version checks.

Significance. If the batching rules are sound, the work enables substantially higher throughput for serving adaptive per-request TTT models without state corruption, addressing a practical barrier to deploying such methods at scale under fixed memory constraints.

major comments (2)

[Abstract] Abstract: the reported throughput (274.61 tok/s) and speedups (9.31x, 3.44x) are stated without any description of experimental setup, number of trials, error bars, hardware configuration details, or how the compatibility rules were validated beyond the owner/version checks.
[Abstract] Abstract: the central correctness claim (selective batching plus owner-only commit never corrupts request-owned state) rests solely on passing owner/version checks and RULER preservation; no formal invariant, model-checked argument, or exhaustive enumeration of interleaving scenarios at the level of batched GEMM operations is supplied.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the comments. We address each major comment point by point below.

read point-by-point responses

Referee: [Abstract] Abstract: the reported throughput (274.61 tok/s) and speedups (9.31x, 3.44x) are stated without any description of experimental setup, number of trials, error bars, hardware configuration details, or how the compatibility rules were validated beyond the owner/version checks.

Authors: The abstract is written for brevity. Full experimental details—including the single-GPU configuration, eight InPlace-TTT streams, and validation via owner/version checks plus RULER—are provided in the Experiments section. We will revise the abstract to include a concise statement of the hardware setup and a pointer to the evaluation section for trials, error bars, and compatibility validation. revision: yes
Referee: [Abstract] Abstract: the central correctness claim (selective batching plus owner-only commit never corrupts request-owned state) rests solely on passing owner/version checks and RULER preservation; no formal invariant, model-checked argument, or exhaustive enumeration of interleaving scenarios at the level of batched GEMM operations is supplied.

Authors: Correctness follows from the explicit per-step tagging (owner, version, READ/WRITE) and the deterministic selective-batching rules that only combine compatible phases while restricting commits to the owner. These rules are validated by the owner/version checks (which would surface any corruption) and by unchanged RULER behavior. The current manuscript does not supply a formal invariant or model-checked argument; we will add an expanded discussion of the phase-compatibility invariants in the revision. revision: partial

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical system measurements with no self-referential derivations or fitted predictions.

full rationale

The paper describes an engineering system (RW-TTT) for batched TTT serving using owner/version/READ-WRITE tagging, reports direct performance measurements (274.61 tok/s, speedups vs. baselines), and validates behavior via RULER benchmark and owner/version checks. No equations, derivations, or predictions are present that reduce to inputs by construction, self-citations, or fitted parameters. The central claims rest on empirical results under stated assumptions rather than any load-bearing self-referential logic. This is self-contained against external benchmarks with no circular steps.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The approach rests on the domain assumption that GPU kernels can be selectively batched based on per-request labels without hidden side effects, and that the memory budget comparison to per-stream replicas is fair. No free parameters or invented entities are described.

assumptions (1)

domain assumption Compatible phases can be identified solely by owner/version/READ-WRITE tags without additional runtime checks.
Invoked when the system batches only compatible phases.

how reviews work

0 comments

Cite this review

Pith. "Pith review of RW-TTT: Batched Serving for Request-Owned Test-Time Training State." pith.science (2026). https://pith.science/paper/NZ264K2F

@misc{pith2026260528053,
  author       = {Pith},
  title        = {Pith review of: RW-TTT: Batched Serving for Request-Owned Test-Time Training State},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NZ264K2F}},
  note         = {Machine review of arXiv:2605.28053}
}

read the original abstract

Test-time training (TTT) adapts an LLM during generation by reading and updating request-owned state, such as fast weights, low-rank deltas, or streaming learner state. This breaks batched LLM serving, which assumes shared static weights: serial execution is correct but slow, while naive batching can corrupt request state. We formulate this problem as read-write TTT serving and present RW-TTT , which tags each decode step with its owner, version, and READ/WRITE effect, batches only compatible phases, and commits updates only to the owner. On one GPU with eight fast-weight InPlace-TTT streams, RW-TTT reaches 274.61 aggregate tok/s, 9.31x over sequential serving and 3.44x over per-stream replicas under the same memory budget. It preserves behavior on RULER, a long-context benchmark, and passes owner/version checks.

Figures

Figures reproduced from arXiv: 2605.28053 by the authors.

**Figure 1.** Per-request READ/WRITE semantics. Each decode step evaluates the shared base model with the request-owned TTT state and exposes either a versionpreserving READ event or a version-creating WRITE event. request r becomes ready at decode step tready(r, e) and is issued at tissue(r, e), then 0 ≤ tissue(r, e) − tready(r, e) ≤ w. (4) Thus w = 0 is greedy batching, and finite w bounds starvation in decode steps while perm… view at source ↗

**Figure 2.** RW-TTT serving architecture. Requests carry owner-versioned mutable state alongside ordinary KV cache. The planner automatically groups compatible READ/WRITE transitions by backend type, operator shape, placement, and valid owner version; batched operators return outputs and updated state through the owner map. The figure shows serving semantics, not the backend update rule. here because it records request-local evi… view at source ↗

**Figure 3.** Scaling under a 16K prompt. Left: feasible [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: WRITE-side operator speedups from Triton [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

3 extracted references · 2 canonical work pages

[1]

2024.ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving

Peft: State-of-the-art parameter-efficient fine- tuning methods. Yifan Qiao, Shu Anzai, Shan Yu, Haoran Ma, Shuo Yang, Yang Wang, Miryung Kim, Yongji Wu, Yang Zhou, Jiarong Xing, and 1 others. 2024. Conserve: Fine-grained gpu harvesting for llm online and offline co-serving.arXiv preprint arXiv:2410.01228. Chaoyi Ruan, Yinhe Chen, Dongqi Tian, Yandong Shi...

work page arXiv 2024
[2]

Qwen3 Technical Report

Qwen3 technical report.arXiv preprint arXiv:2505.09388. Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soo- jeong Kim, and Byung-Gon Chun. 2022. Orca: A distributed serving system for {Transformer-Based} generative models. In16th USENIX symposium on operating systems design and implementation (OSDI 22), pages 521–538. Lianmin Zheng, Liangsheng Yin, Zhiqiang...

work page Pith review arXiv 2022
[3]

In18th USENIX Symposium on Operat- ing Systems Design and Implementation (OSDI 24), pages 193–210

{DistServe}: Disaggregating prefill and de- coding for goodput-optimized large language model serving. In18th USENIX Symposium on Operat- ing Systems Design and Implementation (OSDI 24), pages 193–210. 10 A Reproducibility Notes We distinguish three throughput scopes.Decode-onlycounts only the decode execution window after prompt state is available.Prefil...

Pith tools

Reviewed June 29, 2026 · model on record in the stance chip above.

[1] [1]

2024.ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving

Peft: State-of-the-art parameter-efficient fine- tuning methods. Yifan Qiao, Shu Anzai, Shan Yu, Haoran Ma, Shuo Yang, Yang Wang, Miryung Kim, Yongji Wu, Yang Zhou, Jiarong Xing, and 1 others. 2024. Conserve: Fine-grained gpu harvesting for llm online and offline co-serving.arXiv preprint arXiv:2410.01228. Chaoyi Ruan, Yinhe Chen, Dongqi Tian, Yandong Shi...

work page arXiv 2024

[2] [2]

Qwen3 Technical Report

Qwen3 technical report.arXiv preprint arXiv:2505.09388. Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soo- jeong Kim, and Byung-Gon Chun. 2022. Orca: A distributed serving system for {Transformer-Based} generative models. In16th USENIX symposium on operating systems design and implementation (OSDI 22), pages 521–538. Lianmin Zheng, Liangsheng Yin, Zhiqiang...

work page Pith review arXiv 2022

[3] [3]

In18th USENIX Symposium on Operat- ing Systems Design and Implementation (OSDI 24), pages 193–210

{DistServe}: Disaggregating prefill and de- coding for goodput-optimized large language model serving. In18th USENIX Symposium on Operat- ing Systems Design and Implementation (OSDI 24), pages 193–210. 10 A Reproducibility Notes We distinguish three throughput scopes.Decode-onlycounts only the decode execution window after prompt state is available.Prefil...