{"id":"f85f554f-ca76-469a-a506-d84728fab1d5","arxiv_id":"2608.13263","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A translation table plus asynchronous live-token repacking lets token-granular KV eviction reclaim underutilized physical blocks in PagedAttention-style serving runtimes, improving memory efficiency and throughput in paired vLLM experiments.","lead":"vToken inserts a token-level virtualization layer between LLM cache-eviction policies and block-based memory management, so dead tokens can be physically reclaimed instead of leaving partially empty blocks allocated. It matters because serving capacity is often memory-bound, and the paper reports cutting retained KV blocks by 27.2%-72.3% and extending feasible concurrency by up to 2x in vLLM tests.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing state-of-the-art baselines means the reported 27.2%-72.3% block reduction may not demonstrate a real capability gap.","rationale":"The reader's weakest assumption is exactly that the paired comparison to Naive-Evict does not isolate a real capability gap because no state-of-the-art reclamation systems were implemented. This is the most load-bearing concern because the abstract, introduction, and conclusion all cite the quantitative improvements as the primary evidence of vToken's value. If a prior system already recovers most partially live blocks, the reported 27.2%-72.3% block reduction and up to 1.37x throughput gain would shrink substantially, reducing the paper's contribution from a new capability to an incremental redesign of existing compaction. The concrete test of implementing Zipage or PagedEviction directly addresses the gap, and the result would either substantiate vToken's advantage or reveal that Naive-Evict is a deliberately weak baseline. Other issues, such as the discrepancy in headline percentages between the abstract (27.2%-72.3%) and intro (16%-85%) and the lack of error bars, are real but secondary; they affect precision rather than the central mechanism. Given that the reader already issued a CONDITIONAL verdict, this concern reinforces that judgment rather than changing it.","tokens_in":126,"tokens_out":9632,"duration_ms":110843,"concrete_test":"Run the same workloads under a minimal implementation of Zipage or PagedEviction in vLLM, using identical H2O/Scissorhands/Random eviction decisions and memory budgets. For Zipage, implement bounded per-request block caps with greedy relocation of retained tokens; for PagedEviction, restrict eviction to whole blocks. Compare retained blocks per request, SLA-constrained throughput, and feasible concurrency against vToken. If vToken's block reduction over these realistic baselines is under about 10 percentage points or reverses, the claimed capability gap is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"vToken's central empirical claim is that it turns token-level liveness into physical capacity. The paired Naive-Evict baseline is constructed to retain every partially live block, so the 27.2%-72.3% block reduction is expected from any reclamation mechanism. The paper's own Section 7 names three systems that already perform block-level compaction or page-aligned reclamation—PagedEviction, DiffKV, and Zipage—but none is implemented or measured. DiffKV performs parallel KV compaction with a mixed-precision page manager; Zipage enforces a bounded per-request KV-block budget by relocating retained entries; PagedEviction aligns eviction to page boundaries. If any of these already recover most of the intra-block waste, vToken's headline numbers measure only the gap between 'no reclamation' and 'any reclamation,' not a new capability. The claim that vToken is policy-neutral and kernel-preserving is a design argument, but the quantitative advantage over existing state of the art is unverified. Since the abstract and conclusion present the percentages as the primary evidence of practical value, the absence of these baselines is a load-bearing gap, not a stylistic omission.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes vToken, a token-level virtualization layer for block-managed KV caches in LLM serving. It introduces a per-request token table that decouples logical token liveness from physical block placement, and a reclamation backend that asynchronously repacks live tokens from partially live blocks and returns freed blocks to the allocator. The design is implemented on vLLM and evaluated with H2O, Random, and Scissorhands policies on Mistral-7B, Llama-3.1-8B, and Qwen2.5-14B. The headline claims are that vToken reduces retained KV blocks per request by 27.2%-72.3%, improves SLA-constrained throughput by up to 1.37x, extends maximum feasible concurrency by up to 2x, preserves PagedAttention kernels and CUDA Graph compatibility, and reduces per-policy integration footprint from 500+ to under 50 lines.","tokens_in":18811,"tokens_out":4607,"duration_ms":42831,"significance":"If validated, vToken addresses a real and underappreciated gap: token-level eviction policies are semantically finer than the block-level reclamation interface of PagedAttention-style runtimes, so token-level decisions leave physical capacity trapped in partially live blocks. The paired methodology cleanly isolates the effect of the reclamation backend from the eviction policy, the capacity-frontier result is replicated on a third model (Qwen2.5-14B), and the CUDA Graph compatibility argument is concrete. However, the quantitative advantage over existing reclamation-capable systems is not demonstrated, and the paper contains several internal inconsistencies in the headline numbers. The value of the abstraction is plausible, but the evidence as presented does not support all of the claims made in the abstract and conclusion.","major_comments":[{"comment":"The evaluation compares vToken only against Native vLLM and Naive-Evict, both of which retain partially live blocks because they perform no physical reclamation. Section 7 names three systems that already perform block-level compaction or page-aligned reclamation—PagedEviction, DiffKV, and Zipage—but none is implemented or measured. Since Naive-Evict deliberately disables any reclamation, the reported 27.2%-72.3% block reduction and up to 2x concurrency extension show the gap between no reclamation and any reclamation, not a capability advantage over systems that already recover partially live blocks. This is load-bearing because the abstract and conclusion present these numbers as the primary evidence of practical value. The authors should either implement and measure at least one state-of-the-art reclamation baseline, or substantially weaken the claim to an isolation of the reclamation mechanism rather than a comparison against existing systems.","section":"§7 / §5.1"},{"comment":"The headline numbers are internally inconsistent. The abstract and conclusion state retained blocks are reduced by 27.2%-72.3%, but §1 states 16%-85%. The abstract states SLA-constrained throughput improvement of up to 1.37x, while §5.3 reports Scissorhands throughput gains of 33.3%-103.7%, which is up to roughly 2.04x, and §5.4 reports concurrency extension of up to 2x. These ranges cannot all be correct without explicit qualifiers defining the experimental setup for each number. The authors need to reconcile the ranges and make clear which policies, models, and workloads each number corresponds to.","section":"Abstract vs §1 vs §5.3"},{"comment":"Most headline quantitative claims are presented without uncertainty quantification. Figures 7 and 8 and the throughput/latency percentages in §5.3 are point estimates with no error bars or run counts; only the capacity-frontier experiment in §5.4 states that throughput is averaged over three runs. Given that SLA-constrained throughput and p95 latency are noisy metrics, the claims of 'up to 1.37x' throughput and 'up to 2x' concurrency need either confidence intervals or repeated-run data to be properly evaluated.","section":"§5.3, Fig. 8"},{"comment":"The fragmentation threshold θ_F=0.25 is chosen by empirical exploration on the evaluation workloads and then used in all default runs. The sensitivity analysis in Fig. 12 shows throughput is relatively stable across the threshold range, which mitigates the concern, but retained KV capacity does move with the threshold. The paper should state explicitly whether the headline block-reduction and concurrency results persist when θ_F is re-selected per workload, or report the headline metrics across the full θ_F range.","section":"§5.6"}],"minor_comments":[{"comment":"The arithmetic '2×40×5120×2 B≈0.8 MB' gives 819,200 bytes, which is approximately 0.78 MiB; the units should be stated consistently.","section":"§2.1"},{"comment":"The degree-1 row reports ThroughputΔ=-2.9% while the text says the degree-1 case 'shows only the small overhead'; the sign convention for Δ (negative meaning throughput decrease) should be clarified in the caption or text.","section":"§5.7, Table 1"},{"comment":"The paper provides no artifact, repository link, or instructions for reproducing the experiments; a code-release statement would strengthen the reproducibility of the claims.","section":"General"},{"comment":"The preliminary experiment in Fig. 1 reports low-utilization block fractions without error bars or run counts; a sentence describing replication would help the reader assess the 40%-60% waste claim.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The core idea is timely and the paired experimental design is a good start, but the missing state-of-the-art baselines and the numeric inconsistencies in the headline claims are significant enough that I would not accept the paper in its current form. If the authors add even one reclamation-capable baseline and tighten the presentation, the result could be a solid systems contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. First, it identifies a real gap: token-level eviction decisions leave partially live blocks unreclaimable in block-based runtimes, and the token-table plus asynchronous compaction design is a clean, policy-neutral fix that preserves PagedAttention kernels and CUDA Graphs. Second, the reported numbers are honest about what they compare against, but the abstract sells them as if they demonstrate a gap to the state of the art. They don't, because PagedEviction, DiffKV, and Zipage are cited and described but never implemented or measured.\n\nWhat is actually new: the virtualization layer itself, exposing token-level liveness as a first-class runtime concept while leaving the block substrate intact. The paper is clearly written, the four correctness invariants are stated precisely, and the paired Naive-Evict protocol cleanly isolates the effect of the reclamation backend. The capacity-frontier result on Qwen with controlled budgets is a useful replication, and the authors deserve credit for honest scoping in Section 6: they state that the native path remains preferable without memory pressure and that the overlap results are a stress test, not steady-state costs.\n\nSoft spots, in order of severity. First, the missing SOTA baselines are a load-bearing gap. A baseline that retains every partially live block will always lose to any compaction mechanism; the 27.2–72.3% block reduction measures the gap between 'no reclamation' and 'some reclamation,' not a new capability. The paper's own related work describes three systems that already reclaim intra-block waste, and none is compared. Second, the headline numbers are internally inconsistent: the abstract says 27.2–72.3% and up to 1.37× throughput, but Section 1 says 16–85% and Section 5.3 reports Scissorhands throughput gains of 33.3–103.7%, which contradicts the abstract's upper bound. No error bars are given for most headline numbers, and no code or artifacts are released, so replication requires substantial implementation choices. Minor point: the fragmentation threshold is tuned on the same evaluation workloads, but the sensitivity sweep shows it is not a dominant parameter, so this is a small blemish, not a fatal one.\n\nThe central design argument holds up; the empirical support for superiority over existing reclamation systems does not. This paper deserves serious review at a systems venue because the problem is real, the abstraction is plausible, and the invariants are testable. The authors should be required to fix the number inconsistencies, report variance, and compare against at least one of the three cited reclamation systems before acceptance. Send it to referees, but expect significant revision.","headline":"The indirection-layer design is genuinely new and the paired evaluation is clean, but the headline numbers compare against a strawman baseline while the cited state of the art goes unmeasured.","tokens_in":19374,"tokens_out":2327,"would_cite":true,"duration_ms":22643,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"vToken adds a token-level virtualization layer that decouples KV token liveness from physical block placement, reclaiming up to 72.3% of retained KV blocks and doubling feasible concurrency without changing attention kernels.","keywords":["KV cache management","token-level eviction","block fragmentation","memory virtualization","LLM inference","PagedAttention","lazy compaction","CUDA Graph"],"falsifier":"Run an implemented page-aligned or bounded-budget reclamation system under the same eviction policies, models, and KV budgets; if retained blocks per request drop as much without a token-table indirection layer, the claimed capability gap does not exist.","tokens_in":18414,"feed_emoji":"⚡","tokens_out":7275,"duration_ms":64876,"temperature":0.7,"pith_summary":"vToken is a runtime layer that fixes a granularity mismatch: token-level cache eviction decides which tokens are dead, but block-based serving runtimes can only free whole blocks, so blocks that still hold any live token stay allocated even if most of their slots are holes. The paper proposes a per-sequence token table that maps logical token identities to physical slots and a reclamation backend that asynchronously repacks live tokens out of underutilized blocks, turning logical liveness into reusable physical memory. On paired tests where both variants make identical eviction decisions, vToken reduces retained KV blocks per request by 27.2%–72.3%, raises SLA-constrained throughput by up to 1.37x, and extends the maximum feasible concurrency by up to 2x under a fixed KV budget. If true, this means existing block-based serving systems can get the memory benefits of token-level eviction without changing attention kernels or giving up CUDA Graph execution.","feed_headline":"vToken reclaims partially filled KV blocks, boosting concurrency 2x","feed_subtitle":"Token-level eviction now reclaims up to 72.3% of retained KV blocks per request.","key_machinery":"The token table is the central object: a per-sequence array that maps each logical token ID to a physical location (block ID, offset) and a liveness bit. It exposes three operations—mark a token evicted, register newly generated tokens, and apply a list of moves after relocation copies finish—so eviction policies never touch blocks and the runtime never reasons about token importance. Around it sits a lazy-compaction reclamation backend that monitors per-block live-token counts, admits relocation plans only when projected block reduction is positive and destination headroom exists, and copies KV entries asynchronously after the current forward pass, using a CUDA event to make relocated data visible to the next attention kernel only when needed. The combination turns token liveness into reclaimable physical capacity without modifying attention kernels.","core_discovery":"The paper's central claim is that token-level KV eviction delivers its memory savings only when a runtime boundary decouples logical token liveness from physical block placement; without that boundary, the savings stay trapped as intra-block fragmentation. vToken realizes the boundary with a logical address space per request: policies call an evict-token operation that marks a token dead in a token table while the KV entry remains physically in place, and a physical reclamation backend later compacts live tokens from low-utilization blocks into destination blocks, updates the table, and returns emptied blocks to the allocator. Relocation copies run after the current decoding step on a separate stream, with a CUDA-event dependency guarding the next attention launch, which preserves the existing slot-mapping mechanism and CUDA Graph replay. The paper argues this is a missing abstraction layer, not merely an engine-specific optimization, because the same hooks port to any PagedAttention-style runtime.","pith_inferences":["If measured against a competing runtime that already relocates live tokens out of partial blocks, the reported 27.2%–72.3% reduction may shrink; the paper's contribution is the missing boundary, not the relocation trick itself.","The logical-token view suggests the same layer could manage KV tiering or offload: liveness is already separated from placement, so deciding which physical storage tier a token occupies becomes a placement policy.","Composing the layer with representation-changing compression (quantization, mixed precision, depth compression) seems natural because the token table is independent of the KV tensor's physical shape, though the paper evaluates only uniform full-precision KV.","A cheap test of transferability: enable the layer with a very low eviction ratio and dense blocks; near-zero block reduction would confirm the gains scale with fragmentation rather than with indirection overhead."],"forward_implications":["A block-based serving system can host token-level eviction policies without redesigning its allocator or attention kernels, so H2O-, StreamingLLM-, and Scissorhands-style policies become drop-in modules.","Under memory pressure, fewer retained blocks per request means more concurrent requests fit in the same KV block pool, directly raising the feasible concurrency frontier.","Because reclamation is deferred and batched, eviction cost is paid only when fragmentation is actionable, keeping the steady-state decoding path nearly free.","The benefit is workload- and policy-dependent: it is largest when live tokens are scattered (Random, Scissorhands) and smaller when retention is structured (H2O), so the layer matters most where policies already reduce logical KV demand aggressively.","vToken is a pressure-activated extension: the native full-KV path remains the right choice when memory is not the bottleneck."],"supporting_citations":[{"why":"introduces PagedAttention, the block-based KV substrate and slot-mapping interface that vToken layers above.","marker":"[17]"},{"why":"H2O, the representative token-granular eviction policy whose retained-token pattern creates the intra-block holes vToken reclaims.","marker":"[36]"},{"why":"Scissorhands, the policy whose dispersed retention produces the largest reclamation gains in the evaluation.","marker":"[22]"},{"why":"PagedEviction, the closest alternative that constrains eviction to page boundaries; the paper contrasts its policy coupling with vToken's policy-neutral boundary.","marker":"[9]"},{"why":"DiffKV, a mixed-precision compaction runtime the paper positions as orthogonal because it changes the KV representation rather than reclaiming holes in uniform KV.","marker":"[35]"},{"why":"Zipage, a concurrent bounded-cache relocation pipeline whose fixed per-request budget is compared with vToken's policy-neutral virtualization.","marker":"[18]"},{"why":"vAttention, page-granular address virtualization that vToken stacks above for token-level reclamation.","marker":"[28]"}],"fun_headline_variants":["vToken virtualizes token liveness to reclaim KV block fragmentation","Token-level virtualization frees trapped KV memory, 2x concurrency","vToken: a missing abstraction layer for KV eviction, 2x concurrency","Reclaim KV fragmentation with vToken's logical token table"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that both tested variants differ only in the reclamation layer, so the measured block savings come from reclamation rather than from some other change; if a competing system already recovers most partially live blocks, the headline gains would be smaller.","fun_headline_variants_meta":{"raw":{"variants":["vToken virtualizes token liveness to reclaim KV block fragmentation","Token-level virtualization frees trapped KV memory, 2x concurrency","vToken: a missing abstraction layer for KV eviction, 2x concurrency","Reclaim KV fragmentation with vToken's logical token table"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000992,"raw_usage":{"total_tokens":4206,"prompt_tokens":952,"completion_tokens":3254,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":3177}},"tokens_in":568,"tokens_out":3254,"duration_ms":23507,"temperature":1.0,"reasoning_tokens":3177,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:06:15.300107+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an implemented page-aligned or bounded-budget reclamation system under the same eviction policies, models, and KV budgets; if retained blocks per request drop as much without a token-table indirection layer, the claimed capability gap does not exist.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Zipage, a concurrent bounded-cache relocation pipeline whose fixed per-request budget is compared with vToken's policy-neutral virtualization."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"vAttention, page-granular address virtualization that vToken stacks above for token-level reclamation."}],"review_version":1}