{"id":"37a2319c-5060-441c-ace5-aad33cad08ed","arxiv_id":"2505.07203","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"PrefillOnly serves single-token LLM requests with hybrid prefilling and JCT-aware scheduling, achieving up to 4x higher throughput on long-input workloads.","lead":"This paper presents PrefillOnly, a serving engine for LLM workloads that need just one output token, such as recommendation and credit verification. It cuts GPU memory by chunking non-attention layers and uses predictable job-finish times for scheduling, reaching up to 4x higher query rates at similar latency.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central 4x-QPS claim is tested only on long-input traces; the no-batching design (§6.1) is never compared against a batched prefill-only baseline on short prompts, so the broad 'prefill-only workload' claim is not established.","rationale":"The paper is internally coherent for the evaluated long-input regime: hybrid prefilling and suffix KV discarding are concrete, well-motivated mechanisms, and the memory-side benefit for long requests is plausible even without an artifact. The load-bearing issue is the generality of the central claim. Section 2.4 asserts that prefill-only inputs are typically long and compute-bound; Section 6.1 uses that assertion to reject batching; yet the evaluation in Table 1 covers only long-input traces, so the assertion is never tested against the most relevant alternative (a batching prefill-only scheduler). If short-prompt prefill-only workloads are common, the 1.4-4.0x claim would not hold for them, and the paper's framing would overstate the result. This is a scope/correctness risk, not a logical contradiction. The reader's weakest assumption identified exactly this long-input/compute-bound premise, and I agree. Because the reader already returned CONDITIONAL and the concern is addressable through an additional experiment and claim scoping rather than a demonstrated failure, no verdict change is needed.","tokens_in":851,"tokens_out":978,"duration_ms":108586,"concrete_test":"Build a batching prefill-only baseline, for example vLLM's continuous batching or chunked prefill configured to batch multiple single-token requests, and run both engines on the same hardware with a short-prompt trace (input lengths 256-2,048 tokens, Poisson arrivals at increasing rates). Record achieved QPS at a fixed P99 latency SLO, e.g., 100 ms. If the batching baseline sustains more QPS at the same or lower P99 than PrefillOnly, the §6.1 no-batching premise fails for short-prompt workloads and the headline claim must be restricted to long inputs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"PrefillOnly deliberately processes requests one at a time (§6.1), justified by §2.4's premise that prefill-only inputs are 'typically long' and that the workload is GPU-compute-bound. The abstract and conclusion, however, state the improvement for 'prefill-only workload' without that scope, and the evaluation uses only two synthetic traces with mean input lengths of roughly 14k tokens (post recommendation) and 50k tokens (credit verification). No experiment covers short prompts (e.g., 256-2,000 tokens), where a single prefill leaves the GPU underutilized and batching multiple requests can substantially improve throughput. The strongest quantitative claim, '1.4-4.0x larger query-per-second' without inflated latency, is therefore demonstrated only inside the long-input regime, and the central design decision not to batch is untested exactly where it is most questionable. This is not an internal inconsistency: if the intended contribution is explicitly scoped to long-input prefill-only workloads, the argument holds as far as the presented data goes. But as written, the headline claim is broader than the evidence, and a batching prefill-only scheduler could plausibly outperform PrefillOnly on short-prompt discriminative tasks (e.g., data labeling or spam classification). The missing counterfactual is the single most load-bearing gap because §6.1's no-batching decision underpins the whole scheduler design.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PrefillOnly, an LLM inference engine targeting requests that generate a single output token (prefill-only workloads). It has two mechanisms: hybrid prefilling, which chunks linear (non-attention) layers to reduce intermediate-tensor memory while running attention normally, enabling longer inputs and suffix KV-cache discarding without parallelization; and continuous JCT calibration, which re-estimates job completion times from input length and prefix-cache hit length and schedules one request at a time via SRJF-style selection with a fairness offset. The evaluation, implemented on vLLM with torch.compile, compares against PagedAttention, chunked prefill, pipeline parallel, and tensor parallel across L4/A100/H100 setups, Llama-8B/DeepSeek-Qwen-32B/Llama-70B, and two synthetic datasets (post recommendation and credit verification), reporting 1.4-4.0x higher QPS at equal average/P99 latency and up to 5x MIL expansion.","tokens_in":19647,"tokens_out":5935,"duration_ms":58124,"significance":"If the claims hold, PrefillOnly addresses a real and emerging workload class with a clean memory argument: since no decoding follows, per-layer KV caches for the generated suffix are dead on arrival, and linear-layer chunking is mathematically exact because linear operators act tokenwise. The multi-hardware/multi-model evaluation and the use of Poisson arrivals are strengths. The central quantitative claim, however, is currently demonstrated only for long-input synthetic traces; the design decision to avoid batching and the scheduler benefit itself are not isolated from the memory optimizations, and no artifacts are released. With scope stated precisely, or with the missing short-input/batched experiments, the contribution would be solid; as written, the headline 'prefill-only workload' claim overreaches the evidence.","major_comments":[{"comment":"The central claim in the Abstract and §7 — 'can process up to 4x larger queries per second without inflating average and P99 latency' — is made for prefill-only workloads in general, but the evaluation uses only long-input traces: post-recommendation profiles of about 11-17k tokens and credit-verification profiles of 40-60k tokens. The premise behind the one-at-a-time scheduler in §6.1 is that prefill-only input is 'typically long' and compute-bound, which is exactly the regime in which not batching is plausible. No experiment uses short prompts (e.g., 256-2,000 tokens) and no batched prefill-only scheduler appears among the baselines, so the key design trade-off is untested where it is most questionable. Either the claims must be explicitly scoped to long-input prefill-only workloads, or the evaluation needs a short-input workload and a batching-enabled baseline.","section":"§6.1, §7.1 (Table 1)"},{"comment":"The evaluation does not isolate the scheduling contribution. The post-recommendation discussion credits continuous JCT calibration with avoiding prefix-cache throttling, but Figure 9 compares full PrefillOnly only against the four baselines; there is no ablation that keeps hybrid prefilling and suffix discarding fixed and compares SRJF with continuous calibration against, say, FIFO or one-shot JCT-based SRJF. Without such an ablation, the magnitude of the 1.4-4.0x QPS gain attributable to the scheduler, as opposed to the memory/MIL improvements, cannot be assessed.","section":"§7.2, Figures 6-9"},{"comment":"The JCT model is calibrated by regression on the same hardware and is described as enabling 'precisely determine the JCT' (§1, §2.6). The only evidence is a Pearson correlation of 0.987 on one model (Qwen 32B FP8), and the proxy is the number of cache-miss tokens. Since the scheduling policy's cache-hit benefit depends on JCT ordering rather than absolute accuracy, report prediction error or rank-correlation on the actual models/hardware used in §7.1, and state whether the same fitted model transfers across the Llama-8B and Llama-70B setups.","section":"§6.3"},{"comment":"The workload traces are synthetic and no artifacts or generation scripts are released. The only descriptions are distribution parameters (Table 1), so the central quantitative results cannot be reproduced or audited. At minimum, release the trace generators and seeds, and state whether result variability across seeds is small.","section":"§7.1, datasets"}],"minor_comments":[{"comment":"'upto' should be 'up to' in both the Abstract and the Conclusion.","section":"Abstract and §10"},{"comment":"The update 'score_min ← jct' should be 'score_min ← score'; as written, the algorithm retains the raw JCT value rather than the fairness-adjusted score used for selection.","section":"Algorithm 1, line 11"},{"comment":"The text says the evaluation was run on 'three different hardware setups', but Table 3 and Figure 6 show four setups (L4, A100, H100 w/o NVLink, H100 w/ NVLink); unify the counts.","section":"§7.2 and Figure 7 caption"},{"comment":"Reference [2] is cited for vLLM but its title is 'Extensions in arc: How to import, add, & open' and appears to be a different document; this citation should be corrected.","section":"Reference [2]"},{"comment":"The spellings 'p99' and 'P99' are used inconsistently; choose one convention.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid systems contribution whose main vulnerability is scope: the abstract and conclusion generalize beyond the long-input regime evaluated, and the scheduling benefit is not isolated. I do not see grounds for rejection; the issues are fixable either by adding a short-input/batched experiment or by precisely scoping the claims, and by adding the missing ablation. The lack of released artifacts is also worth pressing given the synthetic traces."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: PrefillOnly is a genuine systems contribution, and the core techniques are sound, but the headline 4x QPS claim is broader than what the evaluation actually shows.\n\nThe genuinely new piece is hybrid prefilling. Since only attention layers need full-sequence state, chunking the linear layers while leaving attention unchunked cuts the dominant temporary-tensor memory without changing the numerics. The MIL numbers in Table 2 and Figure 10 support that. The JCT-calibration scheduler also makes sense: with fixed output length, JCT is knowable up to prefix-cache state, and re-estimating before every scheduling decision is a clean way to keep cache-hit requests from being evicted. Related work is cited fairly; the comparison to vLLM, Sarathi-Serve, and DistServe does not overstate the novelty.\n\nNow the soft spots, in proportion. The stress-test note lands. Section 6.1 argues against batching because prefill-only inputs are \"typically long\" and compute-bound, but the evaluation never varies input length. Both traces are long: 11k-17k tokens for post recommendation, 40k-60k for credit verification. On short prompts, say 256 to 2k tokens, a single prefill leaves the GPU underutilized, and batching several requests will likely win. The abstract and conclusion don't caveat this. If the paper were scoped to long-context discriminative workloads, the data would support the claim; as written, the \"prefill-only workload\" claim overgeneralizes.\n\nSmaller issues: the evaluation uses synthetic traces with no released artifact; there is no ablation that isolates continuous JCT calibration against naive SRJF (the example in Figure 5 is illustrative, not measured); and the abstract's \"KV cache of only the last computed layer\" phrasing is looser than the implementation, which preserves prefix caches and discards suffix KV caches. These are addressable and not fatal.\n\nWho benefits: people working on LLM serving for classification, recommendation, and data labeling with long prompts, plus anyone designing scheduling for fixed-output-length workloads. It deserves a serious referee -- I would send it forward despite the scoping problem, because the core idea is sound and the open issue is empirical, not conceptual.","headline":"Good systems idea -- hybrid prefilling is clean and the scheduler makes sense -- but the 4x QPS headline outruns the evidence because the no-batching design is only tested on long prompts.","tokens_in":20207,"tokens_out":2365,"would_cite":true,"duration_ms":25571,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PrefillOnly claims to serve single-token LLM workloads at up to 4x higher query rates by keeping only one layer of KV cache and scheduling with exact completion-time estimates.","keywords":["LLM inference","prefill-only workload","KV cache","hybrid prefilling","continuous JCT calibration","shortest remaining job first","prefix caching","GPU memory"],"falsifier":"Run PrefillOnly and a batching engine on a prefill-only workload whose prompts average a few hundred tokens, such as short data-labeling queries, and compare throughput and P99 latency at equal query rates; if the batching engine matches or beats PrefillOnly, the compute-bound long-input premise fails.","tokens_in":19134,"feed_emoji":"⚡","tokens_out":6680,"duration_ms":56085,"temperature":0.7,"pith_summary":"PrefillOnly is an inference engine for a workload the paper identifies: LLM requests that generate exactly one output token, as in recommendation, credit checking, and data labeling. The paper argues that existing engines waste GPU memory by storing full multi-layer KV caches and waste scheduling opportunities by treating completion time as uncertain, and that both wastes can be eliminated when the output length is fixed at one. Its central claim, supported by evaluation across four hardware setups, is that PrefillOnly serves 1.4-4.0x higher query rates than four baselines without raising average or P99 latency, and extends maximum input length by up to 5x without parallelizing inference. If true, this would let a fixed GPU fleet serve far more single-token LLM decisions, or serve longer inputs without paying the throughput cost of chunking or cross-GPU communication.","feed_headline":"One-token LLM jobs run up to 4x denser with no latency hit","feed_subtitle":"Discarding unneeded caches and scheduling by finish time lets an engine handle 4x more one-token queries per second.","key_machinery":"The two load-bearing mechanisms are hybrid prefilling and continuous JCT calibration. Hybrid prefilling forwards a request's non-attention (linear) layers chunk-by-chunk while running attention layers normally, shrinking the peak memory of intermediate tensors so the engine can keep only one layer's KV cache and discard suffix caches; it is implemented through a graph-compilation pass rather than kernel rewrites. Continuous JCT calibration re-estimates every waiting request's completion time at each scheduling step, using profiled JCT as a function of input length and prefix-cache-hit tokens, and schedules the shortest remaining job first, with a queue-time offset to prevent starvation. Together they let PrefillOnly process requests one at a time without batching, which the paper argues is the right choice for compute-bound prefill workloads.","core_discovery":"The paper's discovery is that a prefill-only workload-one token out, no decoding-inverts the memory and scheduling assumptions of LLM serving. Because the KV cache will never be reused for long decoding, PrefillOnly keeps only the cache of one layer and discards suffix KV caches, and because the output length is fixed, it can compute each request's completion time before scheduling. The paper shows that the main obstacle to exploiting these properties is not the KV caches themselves but the large intermediate tensors of non-attention layers, and that chunking those linear layers while leaving attention intact reduces peak GPU memory enough to make the cache savings real. On that basis PrefillOnly claims to handle 1.4-4.0x the query-per-second of page-based attention, chunked prefill, pipeline-parallel, and tensor-parallel baselines at equal or better average and P99 latency.","pith_inferences":["If the long-input premise fails, for example a labeling workload where prompts average a few hundred tokens, batching would likely beat one-at-a-time scheduling, so the reported speedup should not be expected there.","The JCT proxy of cache-miss tokens achieves a Pearson correlation of 0.987 on one A100 setup; the paper leaves open whether that proxy holds across other models, quantization levels, and interconnects, so re-profiling per deployment is a natural extension.","Because hybrid prefilling is implemented via graph compilation over linear layers, the same memory reduction could apply to any transformer-style model with MLP blocks, and possibly to encoder-only classification models.","The fairness parameter trades worst-case latency against average latency, and could be exposed as a per-application SLO knob in production."],"forward_implications":["A GPU that stores only one layer of KV cache can serve inputs several times longer than an engine that stores all layers, without chunking or parallelizing attention.","Continuous JCT calibration raises prefix-cache hit rates by prioritizing requests whose prefix is currently cached, lowering both average and P99 latency.","PrefillOnly's throughput advantage grows in compute-bound, long-input workloads and shrinks at low QPS, where tensor-parallel baselines can offer lower latency.","The same design can apply to prefill nodes in disaggregated prefill-decode serving, since those nodes also see one-token outputs.","Suffix KV cache discarding trades future prefix reuse for memory; the paper notes that offloading to CPU rather than discarding is a compatible extension."],"supporting_citations":[{"why":"This reference documents a single LLM serving many discriminative tasks, motivating the existence of the prefill-only workload.","marker":"[12]"},{"why":"This reference supplies the serving engine PrefillOnly builds on and the page-based attention management baseline.","marker":"[22]"},{"why":"This reference supplies the chunked-prefill baseline and the measured throughput cost of chunking attention.","marker":"[3]"},{"why":"This reference supplies the prefix-caching scheme that continuous JCT calibration extends.","marker":"[56]"},{"why":"This reference provides the continuous-batching baseline that motivates why batching is not assumed beneficial for compute-bound prefill workloads.","marker":"[50]"},{"why":"This reference provides the JCT-aware shortest-job-first scheduling precedent that PrefillOnly adapts.","marker":"[15]"},{"why":"This reference supplies the graph-compilation mechanism used to implement hybrid prefilling without rewriting low-level kernels.","marker":"[4]"}],"fun_headline_variants":["4x more one-token LLM queries without latency penalty","Prefill-only engine packs 4x the requests, no slowdown","One-token LLM serving: 4x throughput with same latency","Single-token LLM inference engine boosts throughput 4x","LLM prefill-only workloads get 4x QPS, zero latency hit"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The design assumes prefill-only workloads have long inputs and are GPU-compute-bound, so that discarding suffix KV caches, processing requests one at a time, and predicting JCT are the right optimizations; if many requests are short, batching would likely outperform the one-by-one scheduler.","fun_headline_variants_meta":{"raw":{"variants":["4x more one-token LLM queries without latency penalty","Prefill-only engine packs 4x the requests, no slowdown","One-token LLM serving: 4x throughput with same latency","Single-token LLM inference engine boosts throughput 4x","LLM prefill-only workloads get 4x QPS, zero latency hit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000209,"raw_usage":{"total_tokens":1431,"prompt_tokens":991,"completion_tokens":440,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":347}},"tokens_in":607,"tokens_out":440,"duration_ms":3646,"temperature":1.0,"reasoning_tokens":347,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:23:03.811446+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run PrefillOnly and a batching engine on a prefill-only workload whose prompts average a few hundred tokens, such as short data-labeling queries, and compare throughput and P99 latency at equal query rates; if the batching engine matches or beats PrefillOnly, the compute-bound long-input premise fails.","supporting_citations":[{"cited_title":"Efficient memory management for large language model serving with 13 pagedattention","cited_arxiv_id":null,"evidence_quote":"This reference supplies the serving engine PrefillOnly builds on and the page-based attention management baseline."},{"cited_title":"Taming{Throughput-Latency} tradeoff in{LLM} inference with {Sarathi-Serve}","cited_arxiv_id":null,"evidence_quote":"This reference supplies the chunked-prefill baseline and the measured throughput cost of chunking attention."},{"cited_title":"Orca: A distributed serving system for {Transformer-Based} generative models","cited_arxiv_id":null,"evidence_quote":"This reference provides the continuous-batching baseline that motivates why batching is not assumed beneficial for compute-bound prefill workloads."},{"cited_title":"Tiresias: A {GPU} cluster manager for distributed deep learning","cited_arxiv_id":null,"evidence_quote":"This reference provides the JCT-aware shortest-job-first scheduling precedent that PrefillOnly adapts."},{"cited_title":"Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation","cited_arxiv_id":null,"evidence_quote":"This reference supplies the graph-compilation mechanism used to implement hybrid prefilling without rewriting low-level kernels."}],"review_version":1}