Pith. sign in

REVIEW 2 major objections 5 minor 37 references

This paper claims that LLM tokenization can be made stateful: session continuations need only re-tokenize a small window around the append, and full contexts can be tokenized exactly on a GPU, with all emitted IDs identical to full referenc

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 03:18 UTC pith:BYOOE7TE

load-bearing objection Strong systems paper with unusually honest validation; the universal splice guarantee rests on deferred per-family proofs, but the evidence is good enough to referee and likely publish with artifacts. the 2 major comments →

arxiv 2607.29678 v2 pith:BYOOE7TE submitted 2026-07-31 cs.CL cs.DCcs.PF

TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving

Zhenyu Zhang , Zhichao Cao This is my paper
classification cs.CL cs.DCcs.PF
keywords tokenizationLLM servingBPEprefix cachingstateful tokenizationGPU tokenizationsplice certificateagent workloads
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

TokTier targets the dominant workload in coding agents: many calls that append a small amount of text to a long transcript. The paper's central claim is that the emitted token IDs can always be made identical to full reference tokenization while doing work proportional to the change, not the whole context. For continuations, it re-tokenizes a window around the append and splices cached IDs only when a per-request check finds a stable pre-tokenization boundary; for full contexts, it rewrites the sequential GPT-family regex into run-local rules and runs exact pre-tokenization and BPE on a GPU. A layered validation battery — per-request checks, differential campaigns over 1.5e10 split comparisons, and runtime sampling — reports zero divergence. If correct, this removes tokenization from the critical path of long-context agent serving and cuts time-to-first-token substantially.

Core claim

The paper's discovery is a pair of equivalences. First, a tokenizer whose front end (added tokens, normalization, pre-tokenization) is followed by a stateless per-piece encoder admits a splice certificate: if a matched run of token records contains a character-class transition at which the pre-tokenizer's right-side output is provably independent of left context, then concatenating cached left records and freshly tokenized right records yields exactly the full-reference ID sequence. Second, the regex pre-tokenizers of GPT-family tokenizers can be decomposed into character-class runs and local piece-start predicates, so exact pre-tokenization becomes parallel across characters and BPE can run

What carries the argument

The splice certificate (a synchronizing boundary): a character-class transition such as a letter-to-space switch at which the pre-tokenizer's future output is independent of all left context; combined with an equal-run match on token IDs, it licenses the splice. The proof rests on the factorization of the tokenizer as a front end G producing units followed by a stateless, per-unit encoder E, and on a losslessness theorem for splice certificates. The GPU path's machinery is run decomposition: characters are classified into four classes, maximal runs summarized, and piece starts computed from position within a run plus bounded lookback and per-run aggregates, removing the sequential regex scan

Load-bearing premise

Everything rests on the assumption that the reference tokenizer's encoding stage is deterministic and stateless across pieces (per-piece BPE with no cross-piece history) and that each family's synchronizing-boundary set is complete; if any supported configuration hides cross-piece state or a boundary set is incomplete, a splice could be accepted where no true stable boundary exists.

What would settle it

Encode a request twice in the same process after a 4,096+ character prefix and check whether the second encoding differs; or craft a text whose token IDs under the shipped repair path diverge from the frozen reference's full-text tokenization — the paper's own runtime verifier caught a production tokenizer on exactly the first test, and any single divergence would invalidate the zero-divergence claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • In agent-like workloads where sessions grow monotonically, tokenization cost no longer accumulates quadratically; each continuation costs O(append + window) instead of O(context).
  • Full-context requests (initializations and rebuilds) can be served by a GPU path that matches the reference bit for bit, avoiding the correctness drift of approximate GPU tokenizers.
  • Serving engines that accept precomputed token IDs can reuse the same prefix-cache keys as text paths, so this front-end change requires no engine modification.
  • The validation methodology (per-request checks, differential campaigns, runtime sampling) provides a template for admitting fast paths in other exact-computation settings.
  • Time-to-first-token falls when tokenization is removed from the critical path, with reported median reductions of 16–34% in engine-in-loop measurements.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The splice-certificate technique generalizes beyond BPE: any pipeline that factors as front end plus stateless per-piece encoder (e.g., WordPiece variants, some byte-level encoders) can get the same incremental repair, provided a per-family synchronizing-boundary set is proven complete.
  • The run-decomposition trick suggests that many sequential regex-based data transformations with bounded lookback could be parallelized exactly; the paper's per-alternative derivation is a recipe for doing so.
  • A testable extension is delta-size-aware routing: appends above roughly 30–50K characters could redirect to the GPU full-tokenization path, removing the current tail case and likely extending the measured TTFT gains to heavier appends.
  • The paper's discovery of a history-dependent tokenizer bug suggests that cached-prefix systems should treat 'same text, same IDs' as an empirical invariant, not an assumption; shadow verification could become standard deployment practice.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper argues that LLM front ends should stop re-tokenizing whole agent sessions on every turn. From 153,951 collected agent calls plus three independent trace sources, it shows the median call appends ~1.4K characters to an 86–123K-token context, and only 1.0–3.6% of calls need full-context tokenization. TokTier is a tokenization service with two modes and one contract: emitted token IDs must always equal full reference tokenization. Session continuations use per-session token state, re-tokenize a small window around the append, and splice only when a per-request check finds a stable pre-tokenization boundary whose soundness is stated in the Appendix A splice theorem; failed checks widen the window or fall back to the reference engine. Calls without a reusable prefix use an exact GPU path that reformulates GPT-family regex pre-tokenization as run-local per-character predicates, followed by size-specialized BPE kernels. Differential campaigns report zero divergence over 1.50×10^10 split-level checks, a 12.4 TB corpus sweep, and 93,000+ replayed agent steps across 17 tokenizer families. In vLLM serving, median TTFT falls 16–34% in loaded regimes; four repair cores plus one GPU sustain 1,821 requests/s under a 50 ms P99 objective, where a 16-core stateless front end saturates at 40 requests/s.

Significance. If the correctness contract holds, this is a strong systems contribution. It documents an O(N) per-turn, O(N^2) per-session work amplification that prompt caching does not remove, and it shows a practical way to reduce the common case to O(Δ+w) while keeping reference-equivalent token IDs. The GPU path is a constructive reformulation of the reference regex rather than a relaxation, and the split-level validation idea — comparing intermediate piece boundaries, not just final IDs — is a genuinely good testing method that caught two real bugs (the o200k seam bug and the Unicode-version table skew). The paper is unusually careful with evidence: version-pinned, content-addressed manifests; workload cross-checks against three independent trace sources; a disclosed null result at 28K tokens; a disclosed P90 reversal under recorded arrivals; and a clean separation of the scanned and served throughput accounts. The shadow verifier also exposed a real history-dependent bug in a widely deployed Rust tokenizer, which substantiates the paper's central worry without undermining its own reference anchoring. No fitted constants enter the core derivations. The main gap is reviewability: the univers

major comments (2)
  1. [§3.2/§A.3, stable-boundary condition 3] The central contract — 'the emitted token IDs are always identical to full reference tokenization of the request text' (§3) — is categorical, and the only mechanism that converts finite differential evidence into a universal statement is the splice theorem combined with the per-family synchronizing-boundary sets of §A.3. But the full discharge proofs are explicitly deferred to a companion document, and §A.3 asserts that the sets are 'proven to reset the pre-tokenizer under every left context' without showing any derivation. This is not pedantic: the manuscript itself records that its earlier length-only check was defeated by context-dependent digit grouping (§3.2), and §5.8 finds a real production engine whose IDs depend on encode history. The o200k case-boundary and digit-grouping transitions are precisely the classes in which such counterexamples live, and the paper's own validation-bo
  2. [Appendix A, Assumption 2] Assumption 2 is explicitly labeled load-bearing: the splice theorem requires the front end to factor as F = E*∘G with the model stage E deterministic and stateless across units, and the theorem's conclusion otherwise collapses. The manuscript's support is that 'differential campaigns support it for the reference implementation.' Since §5.8 demonstrates a widely deployed engine violating the analogue of this assumption (same input, different IDs after a 4096-character prefix encode), the assumption is not vacuous, and the unconditional contract requires it to hold for the pinned reference (tokenizers==0.22.2). A short direct verification would close the gap: HF fast BPE encodes each pre-tokenized piece independently with no cross-piece merge state; added-token extraction is a leftmost-longest literal pass; normalization is per-segment before pre-tokenization. Stating these properties expl
minor comments (5)
  1. [Table 3 note] The note says the 4.4 M shape 'exceeds the graph path's largest capture bucket (222 bytes)'. As printed this is incoherent: 222 bytes is far below 4.4 M characters. Presumably 2^22 bytes (~4 MiB) was intended; please correct.
  2. [Figure 13 caption] The caption contains 'text tier text tier' (duplicated axis labels), and Figure 1 similarly shows 'T okTier' mid-word line breaks. Please fix the layout artifacts.
  3. [§5.2] The arithmetic connecting the 1.50×10^10 split checks to per-family document counts is distributed across several paragraphs (four families, 3.7×10^9 documents each). A small table tying each family's document count to its split-check total would make the headline number auditable.
  4. [§3.3] '15 of the 17 families we examined meet these conditions' — the reader must cross-reference Table 4 and §A.3 to identify which families these are. Please name the 15 (or at least list the two excluded families) in one place.
  5. [§2.3 / Fig. 15(b)] The text says the median call has h between 0.98 and 0.99, while Fig. 15(b) reports per-source means of 0.91 and 0.85. The two are not inconsistent, but a sentence clarifying that the means and medians are reported separately would prevent a misreading.

Circularity Check

0 steps flagged

No circular derivation: the splice certificate and GPU reformulation are checked against the external reference and do not reduce to their inputs.

full rationale

TokTier's central claims are not circular. The splice theorem (Appendix A, Theorem A.5) derives exactness from a certificate (Definition A.4) whose conditions (C1/C2) are strictly stronger than local token-ID agreement; Section 3.2's digit-grouping counterexample explicitly shows that a matched ID run alone is insufficient, so the theorem is not a self-definitional restatement of the check. Assumption 2 (F = E* ∘ G, with per-unit stateless E) is a stated precondition, supported by differential testing against the frozen HuggingFace reference and by the Section 5.8 history-dependent-divergence finding, not by a self-citation. The GPU path is a constructive reformulation of GPT-family regex pre-tokenization (Section 4.1), validated at both split and ID levels against the external reference across 1.50×10^10 split checks, a 12.4 TB corpus sweep, and 93,000+ replayed agent steps; it is not a fitted parameter renamed as prediction. The one flagged gap is that the per-family discharge proofs of the synchronizing-boundary sets are deferred: Appendix A.3 says "Full per-family discharge proofs, the v1→v2 repair history (the digit-grouping counterexample of §3.2), and the adversarial discovery battery are in the companion document shipped with the artifact." That is missing support and an omitted proof, not circularity, and the paper states the validation boundary honestly: "Zero divergence in the reported campaigns is evidence for the tested artifacts, not a proof about future tokenizer versions." No load-bearing self-citations or imported uniqueness theorems were found.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

No new physical entities are introduced; the splice certificate and run decomposition are formal constructs and checks, not entities with independent falsifiable handles. The free parameters are engineering choices (window size, retry schedule, sampling rate, routing threshold) that affect performance and coverage but not the correctness theorem; none is fitted to make a prediction match.

free parameters (5)
  • repair window w = 512 chars default
    Hand-chosen window over which the session suffix plus append is re-tokenized. 99.995% of 56,052 real splices accept on the first attempt, but the sensitivity of acceptance rate to w is not systematically swept; it bounds both repair latency and splice coverage.
  • retry schedule = double w, at most 5 retries
    Hand-chosen upper bound on boundary-search cost before falling back to full reference tokenization; affects tail latency on adversarial edits, not correctness.
  • minimum equal-run length = 2 tokens
    Conservative heuristic in the stable boundary check (§3.2), combined with the >max-token-length cover and the synchronizing-boundary condition. Rejecting a valid splice only costs latency; accepting an invalid one is prevented by the other conditions.
  • shadow verifier sampling rate = 5% (serving), 100% (offline)
    Operational parameter trading verifier cost against detection latency; §5.8 measures detection delay as geometric, median 13 faulty requests at 5% sampling.
  • GPU/CPU routing threshold = 2 KB segment size
    Hand-chosen threshold below which kernel-launch overhead dominates and the CPU reference path is used; affects only cost/routing, not the output contract.
axioms (5)
  • domain assumption Front end factors as F = E*∘G with E deterministic and stateless across units (pipeline fidelity).
    Appendix A, Assumption 2 — the paper itself labels this 'the load-bearing assumption'; the §5.8 history-dependent-tokenizer bug is a violation of its analogue in a non-reference engine.
  • domain assumption Chunks are encoded with add_special_tokens=False, no padding/truncation; post-processing applied once after the merge.
    Appendix A, Assumption 1; matches the deployed merge path; failure would break the concatenation homomorphism on which the splice theorem relies.
  • domain assumption The per-family synchronizing-boundary sets discharge conditions (C1)/(C2) of the splice certificate.
    Appendix A.3 — the discharge proofs are deferred to a companion document; if a set is incomplete, a splice could be accepted without a true stable boundary.
  • domain assumption The run-local piece-start predicate exactly reproduces the leftmost-first regex of the cl100k/o200k/DeepSeek families at the pinned Unicode tables.
    §4.1 — asserted constructively but validated by differential testing rather than by a proof in the main text; the Unicode-table skew found in §5.2 shows the sensitivity of this assumption.
  • domain assumption The six-user interactive traces characterize coding-agent tokenization workloads more broadly.
    §2.2 — cross-checked against provider-side aggregates, a public autonomous trace, and TraceLab; the panel is small and concentrated on two coding agents.

pith-pipeline@v1.3.0-alltime-deepseek · 24 in / 18017 out tokens · 230416 ms · 2026-08-04T03:18:57.811573+00:00 · methodology

0 comments
read the original abstract

LLM serving stacks cache prompt KV state, yet the front end still re-tokenizes the full request text on every call. Coding agents pay the most: each call resubmits a long transcript after a small append, and reuse is hard because a short append can move token boundaries near the end of the prior sequence. Across 153,951 agent calls, the median append is 1.4K characters; only 1.0-3.6% of calls start or rebuild a session, but those carry multi-million-character contexts. At the fleet's 94.1% prompt-cache hit rate approaching 0.99, tokenization grows from 10% to 64% of time to first token. TokTier is a stateful CPU+GPU tokenization service for this two-mode workload with one contract: emitted token IDs are always identical to full reference tokenization of the request text. For session continuations it re-tokenizes a small window around the append and splices only when a per-request check finds a stable pre-tokenization boundary, else it widens or falls back. For calls without a reusable prefix it decomposes GPT-family regex pre-tokenization into run-local rules and runs exact pre-tokenization and BPE on a GPU. A sampled shadow verifier re-checks live traffic. Differential campaigns over 17 production tokenizer families ($1.5\times10^{10}$ split checks, a 12.4TB real-text corpus, 93,000+ replayed agent steps) show zero divergence. Incremental repair takes 0.5-1.1ms from 100K to 3M characters, up to $437\times$ faster than HF tokenization and $2.1\times$ faster at 1M characters than the strongest cache-based baseline (Gigatoken) fully prewarmed. GPU full tokenization encodes 1M characters in 0.87ms, $491\times$ below HF and $23.4\times$ below the fastest published CPU method. With vLLM, median time to first token drops 16-34% and P99 23%; under a 50ms P99 objective, four repair cores plus one GPU sustain 1,821 requests/s where a 16-core stateless front end saturates at 40.

Figures

Figures reproduced from arXiv: 2607.29678 by Zhenyu Zhang, Zhichao Cao.

Figure 1
Figure 1. Figure 1: One agent turn through today’s stack and through TokTier. The request appends a median ∼1.4K characters to a transcript that was tokenized on previous turns, and today’s front end re-scans the complete context before the prefix cache can act. TokTier repairs session continuations around the append, routes rare initializations and rebuilds to an exact GPU path or the reference CPU path, and samples all outp… view at source ↗
Figure 2
Figure 2. Figure 2: Joint distribution of context size 𝑁 and new tokens per call Δ over the 153,951 interactive calls, in the API’s token accounting. Session continuations concentrate one to three orders of magnitude below the diagonal, so most calls add little text to a large context. Session initializations and rebuilds sit on the Δ=𝑁 diagonal and carry complete contexts up to 106 tokens. Marginal distributions and per￾sour… view at source ↗
Figure 3
Figure 3. Figure 3: Request lifecycle through TokTier. The router checks for live session token state, then sends session contin￾uations to boundary repair, large state-miss segments to the GPU path, and small segments or any fast-path failure to the reference CPU path. The session store keeps token IDs and byte spans per live session. A shadow verifier re-tokenizes a sampled fraction of emitted IDs against the reference engi… view at source ↗
Figure 4
Figure 4. Figure 4: A boundary inside the word “pipeline” changes the token sequence (Llama-3.1-8B tokenizer, real token IDs below each box). Independent tokenization produces two tokens (␣pipe+line) where full tokenization produces one (␣pipeline). Repair re-tokenizes the affected region and reuses the cached prefix only after finding a stable boundary, reproducing the serial stream bit for bit. lifetimes. Large state-miss s… view at source ↗
Figure 6
Figure 6. Figure 6: Run decomposition. The reference regex exposes a serial dependency between matches. The equivalent formulation computes character classes, maximal runs, and a local piece-start predicate with parallel passes. The resulting pieces feed size-specialized BPE kernels. 4.2 GPU BPE After pre-tokenization, each piece is encoded independently. Merge ranks, vocabulary entries, and byte strings are packed into GPU h… view at source ↗
Figure 6
Figure 6. Figure 6: Run decomposition. The reference regex exposes a serial dependency between matches. The equivalent formulation computes character classes, maximal runs, and a local piece-start predicate with parallel passes. The resulting pieces feed size-specialized BPE kernels. session state for later repair uses the reference tokenizer to obtain spans. Unsupported tokenizers and requests that fail a family check remain… view at source ↗
Figure 7
Figure 7. Figure 7: Performance against every measured baseline. (a) Single-request P50 latency versus complete context on identical real texts. Incremental-repair lines price one append on a live session under the protocol of §5.3, and full-tokenization lines price a full encode of a fresh request under the protocol of §5.4. Gigatoken runs in its most favorable, fully prewarmed mode and wins below the 100 K–500 K crossover. … view at source ↗
Figure 7
Figure 7. Figure 7: Performance against every measured baseline. (a) Single-request P50 latency versus complete context on identical real texts. Incremental-repair lines price one append on a live session under the protocol of §5.3, and full-tokenization lines price a full encode of a fresh request under the protocol of §5.4. Gigatoken runs in its most favorable, fully prewarmed mode and wins below the 100 K–500 K crossover. … view at source ↗
Figure 8
Figure 8. Figure 8: Incremental repair latency versus complete context length on the public-trace replay (log–log, solid P50, dashed P90). The Rust session store keeps repair at 0.5–1.1 ms from 100 K to 3 M characters, while full CPU re-tokenization grows with context size. The dotted line is the prior Python-store implementation on the same protocol [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Boundary-check behavior over 56,052 replayed real splices (SWE-smith streams under three families, plus the public Codex trace). (a) The default 512-character window accepts 56,049 splices on the first attempt at every append size, three splices widen the window once, and none falls back to full retokenization. (b) Repair wall time grows with the append and not with the context, as 𝑂(Δ + 𝑤) predicts. Laten… view at source ↗
Figure 10
Figure 10. Figure 10: Full tokenization single-request encode P50 (Qwen3 family, steady state). (a) With array token deliv￾ery, every dispatch variant stays near or below one millisec￾ond across three decades of request size, and the consumer RTX 5090 leads the server card. (b) The same fused+graph kernels behind two delivery channels. Materializing a Python list[int] adds interpreter interface cost that dominates be￾yond 100 … view at source ↗
Figure 10
Figure 10. Figure 10: Full tokenization single-request encode P50 (Qwen3 family, steady state). (a) With array token deliv￾ery, every dispatch variant stays near or below one millisec￾ond across three decades of request size, and the consumer RTX 5090 leads the server card. (b) The same fused+graph kernels behind two delivery channels. Materializing a Python list[int] adds interpreter interface cost that dominates be￾yond 100 … view at source ↗
Figure 11
Figure 11. Figure 11: Tail behavior in recorded burst scenarios. Adding CPU workers does not remove the session-initialization service￾time floor (a), and at low worker counts initialization work drags continuation P99 up (b). A four-core repair pool plus one GPU keeps continuation and initialization P99 low. The host-contention companion panel is [PITH_FULL_IMAGE:figures/full_fig_p013_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: P99 sojourn time versus offered load (Poisson, measured mixture, 60 s steady state per point, and open mark￾ers are backlogged points reported as-is). The tier reaches 1,821 requests/s under a 50 ms P99 objective, while stateless CPU configurations saturate at 33–40 requests/s and only the GPU-only front end holds the 10 ms objective. board power. These figures support deployment feasibility, while the la… view at source ↗
Figure 14
Figure 14. Figure 14: What session state costs and what its lifetime buys. (a) Store memory for one growing session on the replayed real trace. Accounted state is ∼16 bytes per token, the Rust store’s resident growth stays between 1.2 and 22 MB per session across strains and families, and the prior Python store paid 70–129 MB for the same sessions. (b) Share of the 153,951 trace calls that would find live token state, as a fun… view at source ↗
Figure 14
Figure 14. Figure 14: What session state costs and what its lifetime buys. (a) Store memory for one growing session on the replayed real trace. Accounted state is ∼16 bytes per token, the Rust store’s resident growth stays between 1.2 and 22 MB per session across strains and families, and the prior Python store paid 70–129 MB for the same sessions. (b) Share of the 153,951 trace calls that would find live token state, as a fun… view at source ↗
Figure 15
Figure 15. Figure 15: Anatomy of coding-agent tokenization workloads across nine machines, two ecosystems, and a public autonomous￾agent trace. (a) Per-call increment Δ. (b) Per-call cache hit ratio ℎ. (c) Context size 𝑁. (d) LLM calls per human turn. (e) Turn-gap distribution against the 5-minute and 1-hour cache TTLs. (f) Tokens-to-process per call, session continuations versus session initializations. Continuations process … view at source ↗
Figure 15
Figure 15. Figure 15: Anatomy of coding-agent tokenization workloads across nine machines, two ecosystems, and a public autonomous￾agent trace. (a) Per-call increment Δ. (b) Per-call cache hit ratio ℎ. (c) Context size 𝑁. (d) LLM calls per human turn. (e) Turn-gap distribution against the 5-minute and 1-hour cache TTLs. (f) Tokens-to-process per call, session continuations versus session initializations. Continuations process … view at source ↗
Figure 16
Figure 16. Figure 16: Per-core context throughput on the replay, two accounts kept separate (log–log). Under the served account, repair passes every 𝑂(𝑁) engine as context grows, crossing Gigatoken’s prewarmed ceiling (355 MB/s) at ∼200 K bytes, while its scanned account stays at 1.8–3.6 MB/s. it has not seen before. On this replay corpus even 100 K of new transcript is mostly repeated pretokens, so its P50 at 8 M moves by und… view at source ↗
Figure 16
Figure 16. Figure 16: Per-core context throughput on the replay, two accounts kept separate (log–log). Under the served account, repair passes every 𝑂(𝑁) engine as context grows, crossing Gigatoken’s prewarmed ceiling (355 MB/s) at ∼200 K bytes, while its scanned account stays at 1.8–3.6 MB/s. so an append costs one rescan plus the handful of pretokens it has not seen before. On this replay corpus even 100 K of new transcript … view at source ↗
Figure 17
Figure 17. Figure 17: Host-contention companion panel to [PITH_FULL_IMAGE:figures/full_fig_p021_17.png] view at source ↗
Figure 17
Figure 17. Figure 17: Host-contention companion panel to [PITH_FULL_IMAGE:figures/full_fig_p022_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Measured routing phase diagram on the append￾size × context plane, under exploration-prototype account￾ing rather than shipped defaults. Small green points are incremental repairs from the append-size sweep, and the six large markers are cells where a GPU full-tokenization rebuild and the best CPU repair option were measured head to head. Real appends concentrate far left of the GPU region, and past the G… view at source ↗
Figure 18
Figure 18. Figure 18: Measured routing phase diagram on the append￾size × context plane, under exploration-prototype account￾ing rather than shipped defaults. Small green points are incremental repairs from the append-size sweep, and the six large markers are cells where a GPU full-tokenization rebuild and the best CPU repair option were measured head to head. Real appends concentrate far left of the GPU region, and past the G… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 8 linked inside Pith

  1. [1]

    Anthropic. 2024. Prompt Caching with Claude. Explicit 5-minute and 1-hour cache TTLs

    2024

  2. [2]

    Anthropic. 2026. Models overview.https://platform.claude.com/docs/ en/about-claude/models/overview. Claude Fable 5 and Claude Opus 5: 1M token context window. Accessed July 30, 2026

    2026

  3. [3]

    Euijun Chung, Yuxiao Jia, Aaron Jezghani, and Hyesoon Kim. 2026. Characterizing CPU-Induced Slowdowns in Multi-GPU LLM Inference. arXiv:2603.22774

    Pith/arXiv arXiv 2026

  4. [4]

    DeepSeek-AI. 2024. DeepSeek-V3 Technical Report. arXiv:2412.19437

    Pith/arXiv arXiv 2024

  5. [5]

    DeepSeek-AI. 2024. DeepSeek-V3 tokenizer.json. HuggingFace model artifact, revision e815299b.https://huggingface.co/deepseek- ai/DeepSeek-V3

    2024

  6. [6]

    Xiang Deng, Jeff Da, Edwin Pan, et al . 2025. SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? arXiv:2509.16941. 16 TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving

    Pith/arXiv arXiv 2025

  7. [7]

    fastokens contributors. 2026. fastokens: A Fast BPE Tokenizer with a Rust Backend. Version 0.2.0, Apache-2.0. Supported by vLLM v0.23.0 and later as an opt-in tokenizer backend that must be installed separately.https://github.com/crusoecloud/fastokens

    2026

  8. [8]

    HuggingFace. 2019. HuggingFace Tokenizers. Rust library; version 0.22.2 in the frozen environment of this paper.https://github.com/ huggingface/tokenizers

    2019

  9. [9]

    Inferact. 2026. codex_swebenchpro_traces: Agentic Workload Traces of Codex on SWE-Bench Pro. HuggingFace dataset, MIT license. Redacted spans are replaced by length-preserving filler text. File codex_swebenchpro.json, SHA-256 prefix 670f1ae8325fd70a, retrieved July 13, 2026

    2026

  10. [10]

    Vibhu Jawa. 2021. Run State of the Art NLP Workloads at Scale with RAPIDS, HuggingFace, and Dask. NVIDIA De- veloper Blog; BERT WordPiece batch tokenization. Accessed July 31, 2026.https://developer.nvidia.com/blog/run-state-of-the-art-nlp- workloads-at-scale-with-rapids-huggingface-and-dask/

    2021

  11. [11]

    Shenghu Jiang and Ruihao Gong. 2026. Incremental BPE Tokenization. InProceedings of ICML. arXiv:2605.30813

    Pith/arXiv arXiv 2026

  12. [12]

    Venu Gopal Kadamba and Kanishkha Jaisankar. 2026. GPUTOK: GPU Accelerated Byte Level BPE Tokenization. arXiv:2603.02597

    arXiv 2026

  13. [13]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica

  14. [14]

    Yuhan Liu, Hanchen Li, Yihua Cheng, et al. 2024. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving. InProceedings of SIGCOMM

    2024

  15. [15]

    George C. Necula. 2000. Translation Validation for an Optimizing Com- piler. InProceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI)

    2000

  16. [16]

    NVIDIA. 2024. NVIDIA Blackwell Platform: GB200 NVL72. Vendor- reported claim: up to 30× LLM-inference throughput vs. the same num- ber of H100 GPUs; named mechanisms are FP4 in a second-generation Transformer Engine and fifth-generation NVLink. Accessed July 31, 2026.https://nvidianews.nvidia.com/news/nvidia-blackwell-platform- arrives-to-power-a-new-era-...

    2024

  17. [17]

    NVIDIA. 2025. NVIDIA Dynamo: A Datacenter-Scale Distributed Inference Serving Framework. Router documentation: backend handlers receive pre-tokenized requests

    2025

  18. [18]

    OpenAI. 2023. tiktoken: A Fast BPE Tokeniser for Use with OpenAI’s Models.https://github.com/openai/tiktoken

    2023

  19. [19]

    OpenAI. 2026. GPT-5.6 Sol Model | OpenAI API.https://developers. openai.com/api/docs/models/gpt-5.6-sol. Context window: 1,050,000 tokens. Accessed July 30, 2026

    2026

  20. [20]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, et al. 2024. Splitwise: Ef- ficient Generative LLM Inference Using Phase Splitting. InProceedings of ISCA. Azure LLM inference traces

    2024

  21. [21]

    Amir Pnueli, Michael Siegel, and Eli Singerman. 1998. Translation Validation. InTools and Algorithms for the Construction and Analysis of Systems (TACAS)

    1998

  22. [22]

    Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading More Storage for Less Computation — A KVCache-Centric Architecture for Serving LLM Chatbot. InProceedings of the 23rd USENIX Conference on File and Storage Technologies (FAST). USENIX Association, 155–170

    2025

  23. [23]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. GPT-2; byte-level BPE with regex pre-tokenization.https: //github.com/openai/gpt-2

    2019

  24. [24]

    Marcel Rød. 2026. Gigatoken: SIMD and Cache Hierarchies for 1000x Faster Byte-Pair Encoding Tokenization on Modern CPUs. Audited at version 0.9.0, commit 0d9765fa.https://github.com/marcelroed/ gigatoken

    2026

  25. [25]

    Wei Shao, Lingchao Zheng, Pengyu Wang, Peizhen Zheng, Jun Li, and Yuwei Fan. 2026. LoPT: Lossless Parallel Tokenization Acceleration for Long Context Inference of Large Language Model. InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, San Diego, Cal...

    2026

  26. [26]

    Shubham Tiwari, Tapan Chugh, Nash Rickert, Simon Peter, Ratul Ma- hajan, and Haiying Shen. 2026. CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agents. arXiv:2606.16824

    arXiv 2026

  27. [27]

    vLLM contributors. 2026. [RFC]: Rust front-end. GitHub issue 40846, open; proposed as an experimental preview behind an opt-in flag, with the Python front end kept as the default.https://github.com/vllm- project/vllm/issues/40846; parity tracking #44280

    2026

  28. [28]

    Yuxin Wang, Yuhan Chen, Zeyu Li, Xueze Kang, Yuchu Fang, Yeju Zhou, Yang Zheng, Zhenheng Tang, Xin He, Rui Guo, Xin Wang, Qiang Wang, Amelie Chi Zhou, and Xiaowen Chu. 2025. BurstGPT: A Real-World Workload Dataset to Optimize LLM Serving Systems. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD). 5831–5841.https:...

    arXiv 2025

  29. [29]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent- Computer Interfaces Enable Automated Software Engineering. In Proceedings of NeurIPS. arXiv:2405.15793

    Pith/arXiv arXiv 2024

  30. [30]

    Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Find- ing and Understanding Bugs in C Compilers. InProceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI)

    2011

  31. [31]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. InProceedings of ICLR. arXiv:2210.03629

    Pith/arXiv arXiv 2023

  32. [32]

    Amos You. 2025. BlockBPE: Parallel BPE Tokenization. arXiv:2507.11941; ES-FoMo III workshop at ICML 2025

    Pith/arXiv arXiv 2025

  33. [33]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. InProceedings of the 16th USENIX Symposium on Operating Systems Design and Implementa- tion (OSDI). USENIX Association, 521–538

    2022

  34. [34]

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, et al. 2024. SGLang: Efficient Execution of Structured Language Model Programs. InPro- ceedings of NeurIPS. RadixAttention prefix caching

    2024

  35. [35]

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xu- anzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized Large Language Model Serving. InProceedings of the 18th USENIX Symposium on Operat- ing Systems Design and Implementation (OSDI). USENIX Association, 193–210

    2024

  36. [36]

    HF tokenizers

    Kan Zhu, Mathew Jacob, Chenxi Ma, Yi Pan, Stephanie Wang, Arvind Krishnamurthy, and Baris Kasikci. 2026. TraceLab: Characterizing Coding Agent Workloads for LLM Serving. arXiv:2606.30560; dataset release v0.0.1, CC BY 4.0, syfi_coding_trace SHA-256 prefix 9d265eae69a31cae.https://github.com/uw-syfi/TraceLab. A The Splice Theorem This appendix states and p...

    Pith/arXiv arXiv 2026

  37. [2023]

    InProceedings of SOSP

    Efficient Memory Management for Large Language Model Serving with PagedAttention. InProceedings of SOSP