{"id":"4e53563a-0c42-4273-a310-015aec1329de","arxiv_id":"2501.14784","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"DeServe combines KV cache offloading and microbatch scheduling to maintain high LLM offline inference throughput in high-latency decentralized environments.","lead":"DeServe is a system that runs large language model inference on idle GPUs connected over slow, decentralized networks. It reports 6.7x to 12.6x higher throughput than existing serving systems by offloading KV cache to CPU memory and scheduling microbatches to hide network latency.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline fairness: DeServe's own unoptimized pipeline already beats vLLM pp by 2.2-3.7x, so the advertised 6.7-12.6x speedup is not yet attributable to the proposed KV-offload and microbatch-scheduling optimizations.","rationale":"The reader's weakest assumption is baseline fairness, and the strongest evidence for that concern is internal: DeServe (pp) and DeServe (opt) share kernels and model, so their ratio isolates the paper's mechanisms. That ratio is about 2.3-3.4x, not 6.7-12.6x. The additional factor in the headline comes from DeServe (pp) outperforming vLLM (pp) even before the proposed optimizations are applied. The paper gives no indication that vLLM (pp) was tuned to implementation parity, and the vLLM (tp) row failed in the cross-region test, making vLLM (pp) the only external baseline. This is not a dispute about external consensus; it is an internal attribution problem in the headline claim. The paper deserves credit for a reproducible codebase and an honest ablation, and those support the reader's CONDITIONAL verdict. My concern reinforces the condition rather than moving the verdict, so the verdict should remain UNCHANGED.","tokens_in":12500,"tokens_out":5391,"duration_ms":51780,"concrete_test":"Under the same Section 5 workload and GCP setup, run vLLM (pp) with the FlashInfer attention/KV kernels and the same number of pipeline stages and microbatches as DeServe (pp), then repeat the 16/32/58.4/64 ms latency sweep, and report DeServe (opt)/DeServe (pp) as an internal-control ratio. If vLLM (pp) moves to within ~10% of DeServe (pp) while the opt/pp ratio stays 2.5-3.4x, the headline should be re-scaled to the internal gain; if vLLM (pp) stays near 89 tokens/s centrally and the 58.4 ms gap remains above 6x, the baseline concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 4's headline comparison targets vLLM (pp), but DeServe (pp) — the same DeServe codebase without either optimization — already achieves 194.6 tokens/s centrally vs 89.1 for vLLM (pp), and 133.7 vs 36.1 at 64 ms, i.e. 2.18x to 3.70x across the latency columns. The proposed KV-offload and microbatch-scheduling optimizations add only 2.29x centrally, 3.14x at 58.4 ms, and 2.51-3.42x at 16-64 ms over DeServe (pp). Therefore roughly half of the advertised 6.7-12.6x appears to be an implementation-level baseline gap (kernels, microbatch configuration, pipeline tuning), not the paper's algorithmic contribution. Since the abstract and contribution claim attribute the full ratio to DeServe's design, a controlled baseline is needed before the central claim can be accepted as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents DeServe, a decentralized offline LLM serving system that combines pipeline parallelism with KV-cache offloading to CPU memory and microbatch scheduling to hide network latency. Using a Llama 3 70B workload on eight GPUs, Table 4 reports output throughput for vLLM baselines from 89.1 tokens/s centrally to 36.1 tokens/s at 64 ms latency, while DeServe with optimizations reaches 434-458 tokens/s across latency settings; the abstract summarizes this as a 6.7x-12.6x improvement in high-latency environments. The paper also develops a cost model based on cloud, decentralized platform, and mining-style GPU prices, and sketches a blockchain-based framework for payment and inference-correctness arbitration.","tokens_in":12710,"tokens_out":5098,"duration_ms":46573,"significance":"If the optimization-specific gains are reproducible, the paper makes a useful contribution: KV-cache offloading with overlapped PCIe transfer and microbatch filling of latency bubbles is a sound and practical idea for decentralized serving, and the cross-region real-world experiment with simulated-latency validation is a strength. The code is released, which aids reproducibility. However, the headline claim is currently overstated because it is measured against a vLLM pipeline-parallelism baseline that DeServe's own unoptimized pipeline already outperforms by a factor of 2.2-3.7x. The incremental contribution of the proposed optimizations over DeServe (pp) is 2.5-3.4x in high-latency settings, which is still meaningful but not the advertised 6.7-12.6x. The paper needs a controlled baseline comparison and a clearer separation of end-to-end gains from optimization-specific gains.","major_comments":[{"comment":"The headline 6.7x-12.6x improvement is confounded by baseline implementation differences. In Table 4, DeServe (pp) — the authors' own pipeline without KV offloading or microbatch scheduling — already achieves 194.6 tokens/s centrally versus 89.1 for vLLM (pp), and 133.7 versus 36.1 at 64 ms, a 2.18-3.70x gap. The optimizations described in Section 4 add only 2.29x centrally and 2.51-3.42x at 16-64 ms over DeServe (pp). Therefore roughly half of the advertised ratio is an implementation-level difference (kernel choice, microbatch configuration, engine overhead), not the algorithmic contribution of KV-cache offloading and microbatch scheduling. The abstract and contribution list should either compare against a vLLM baseline configured with comparable kernels and microbatch settings, or the claim should be revised to report the optimization-specific gain separately from the end-to-end system gain.","section":"Table 4 and Abstract"},{"comment":"The microbatch scheduling algorithm is under-specified. The text gives one illustrative example with network latency equal to half the stage time and says that adding two extra microbatches fills the bubbles, but there is no general rule, pseudocode, or formula for choosing the number of microbatches as a function of network latency, stage time, and available memory. Since microbatch scheduling is one of the two main contributions, the paper needs to state the scheduling policy precisely and explain how it is configured in the experiments. Without this, the results in Table 4 cannot be reproduced or unambiguously attributed to the proposed algorithm.","section":"Section 4.3 and Figure 2"},{"comment":"Table 4 reports a single throughput number per configuration with no standard deviation, number of runs, or vLLM version and configuration details. The east-west (58.4 ms) and simulated 64 ms columns are close but not identical (434.1 vs 456.8), and without repeated trials it is unclear whether the 6.7-12.6x ratios are stable or whether the simulated-latency validation is within noise. Please report variance and the exact hardware, software, and vLLM configuration used for each baseline.","section":"Section 5"}],"minor_comments":[{"comment":"The caption for panel (c) reads 'Pipeline Parallelism In Centralized Environment' but the panel illustrates the decentralized setting with added microbatches; this appears to be a typo and should be corrected.","section":"Figure 2"},{"comment":"The sentence 'The vLLM threw out an error during the benchmark' is informal; please describe the error and the vLLM version/config that produced it.","section":"Section 5"},{"comment":"Equation (2) defines MG as the product of PCIe bandwidth W and stage time TS, but no values of W or TS are given for the experimental setup, and no validation is shown that this sizing is sufficient to hide offload latency.","section":"Section 4.2"},{"comment":"The profit model assumes unified input/output pricing P = PI = PO; this simplification should be stated prominently in the text near Table 2, since different platforms charge different rates for input and output tokens.","section":"Section 3.3"},{"comment":"The correctness-protection framework is described at a high level with no implementation or evaluation; if it is intended as a contribution, at least a discussion of overhead or a reference implementation is needed, and otherwise it should be clearly marked as a system sketch.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a systems/ML-systems venue, but the current abstract significantly overstates the result relative to what Table 4 supports. The authors should be encouraged to re-run the comparison with a more carefully controlled vLLM baseline and to report optimization-specific gains separately. The on-chain framework section is speculative and could be trimmed without affecting the core serving-system contribution. No concerns about citation integrity or novelty disclosure; the related work is adequate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is worth your time: combining KV-cache offloading with microbatch scheduling to keep GPUs busy in high-latency, decentralized offline LLM serving. The internal comparison holds up — DeServe with both optimizations beats its own unoptimized pipeline by a consistent 2.5-3.4x across latencies, which is a solid signal that the mechanisms (two global page pools, overlap of swap with compute, extra microbatches to fill bubbles) actually do something. The paper also ships code, validates its latency simulation against a real east-west run (64 ms simulated vs 58.4 ms real), and writes the cost model with external prices. Those are real strengths.\n\nBut the headline claim is currently not supported. The advertised 6.7-12.6x is measured against vLLM pipeline parallelism, and DeServe's own unoptimized pipeline already beats vLLM pp by 2.2-3.7x in the same table. So roughly half of the advertised speedup is implementation-level baseline gap — different kernels, microbatch configuration, or pipeline tuning — not the paper's algorithmic contribution. That is a load-bearing confound. The abstract and contributions attribute the full ratio to DeServe's design, which is misleading. The fix is straightforward: benchmark against vLLM with the same kernels and a controlled microbatch configuration, or honestly decompose the speedup into baseline gap plus optimization gain, and claim only the latter.\n\nMinor issues: no error bars or multiple runs for the key numbers (single 20-minute run), and the 'first work' claim is asserted without comparing to Helix or Petals, which are cited but not benchmarked. The correctness-protection discussion is high-level and doesn't affect the serving claims. None of these are fatal; the central optimization idea is plausible and the internal evidence supports it.\n\nBottom line: this paper deserves a serious referee, but the current headline number needs to be re-scoped. I would send it to a systems venue with a request for a controlled baseline and variance reporting. The optimization technique itself is worth citing and building on, even if the current empirical claim is too strong.","headline":"A real optimization idea with a consistent internal gain, but the headline 6.7-12.6x speedup is over-claimed because the vLLM baseline is not controlled.","tokens_in":13224,"tokens_out":2046,"would_cite":true,"duration_ms":21160,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DeServe claims that KV cache offloading plus microbatch scheduling lets decentralized, high-latency GPU fleets serve offline LLM workloads at 6.7x–12.6x the throughput of existing serving baselines.","keywords":["decentralized LLM serving","offline batch inference","KV cache offloading","microbatch scheduling","pipeline parallelism","high-latency networks","throughput optimization","consumer GPUs"],"falsifier":"Run DeServe (opt) and the vLLM pipeline-parallel baseline on identical hardware with the same model and workload at controlled latencies of 16, 32, 64, and 256 ms, using the same latency-simulation mechanism for both systems; if the throughput gap narrows to well below 6.7x when baseline engineering is equalized, the claimed advantage is an artifact of implementation rather than of KV cache offloading and microbatch scheduling.","tokens_in":12272,"feed_emoji":"⚡","tokens_out":4822,"duration_ms":43560,"temperature":0.7,"pith_summary":"DeServe sets out to make offline LLM inference affordable by pooling idle, geographically scattered GPUs instead of renting expensive cloud instances. The paper's central claim is that two coordinated techniques—offloading KV cache to CPU memory and inserting extra microbatches to fill pipeline bubbles—let a decentralized pipeline sustain near-centralized throughput even when network latency reaches hundreds of milliseconds. A sympathetic reader would take the contribution to be the first serving design aimed specifically at the high-latency setting, with experiments reporting 6.7x–12.6x throughput over existing pipeline-parallel baselines. If true, this would make LLM serving viable on cheap, intermittent compute such as GPU mining networks, where the break-even throughput is far lower than on clouds.","feed_headline":"Decentralized GPUs run LLM pipelines 6.7-12.6x faster","feed_subtitle":"KV cache offload and extra microbatches erase network-latency bubbles, keeping throughput flat from 1 to 256 ms.","key_machinery":"The load-bearing mechanism is the pairing of a KV-cache offloading scheme with microbatch scheduling. DeServe splits GPU memory into one local page pool per microbatch plus two global page pools ($G_0$ and $G_1$) of size $M_G = W \\times T_S$, sized by PCIe bandwidth $W$ and pipeline stage time $T_S$, and overlaps swapping out one microbatch's cache with prefetching another's while computation runs. The available GPU memory per microbatch becomes $M'_B = \\frac{M_{KV} - 2M_G}{N_B} + M_G$, so even as the number of microbatches $N_B$ grows, each microbatch keeps a guaranteed memory floor. Microbatch scheduling then injects enough extra microbatches to fill the network-latency bubbles, converting idle time into useful computation.","core_discovery":"The paper argues that in a pipeline-parallel LLM served across machines linked by high-latency networks, throughput collapses for two reasons: each microbatch gets a small slice of GPU memory, capping batch size, and network latency leaves GPUs idle in bubbles between microbatch executions. DeServe's optimization is to swap KV cache between GPU and CPU through two global page pools while a microbatch runs, enlarging each microbatch's effective KV memory, and to add extra microbatches whose execution fills the latency bubbles. In real experiments across us-east-1 and us-west-4 at 58.4 ms latency, the optimized system sustains about 434 output tokens/s, versus 37.3 tokens/s for the vLLM pipeline-parallel baseline, an 11.6x gain; across simulated latencies from 16 to 256 ms, the gains over that baseline range from roughly 6.7x to 12.6x.","pith_inferences":["The bubble-filling scheduling idea is not transformer-specific: any staged computation whose inter-stage transfer is slow relative to stage execution could add extra in-flight units to keep workers busy, so the technique may generalize to geo-distributed batch pipelines.","The global page pool size is tied to PCIe bandwidth, meaning the effective cache hierarchy becomes GPU memory plus CPU memory; faster interconnects or disaggregated memory would shift the optimal trade-off between offload size and microbatch count.","A direct comparison against a stronger optimized baseline, such as the same engine implementing chunked prefill or disaggregated serving, would separate the gains attributable to DeServe's two techniques from gains due to overall system engineering.","The correctness-protection discussion stops short of a working implementation, so the economic story depends on whether optimistic or zero-knowledge arbitration can be made cheap enough to preserve the off-chain throughput advantage."],"forward_implications":["At 58.4 ms real inter-region latency, DeServe's optimized pipeline keeps output throughput near 434 tokens/s while the unoptimized pipeline falls to 138 tokens/s and the vLLM baseline to 37 tokens/s.","Raising simulated latency from under 1 ms to 256 ms costs DeServe almost nothing (445.2 down to 442.9 tokens/s), so in the tested regime throughput stops being network-limited.","Because KV offloading guarantees a per-microbatch memory floor, operators can increase pipeline depth to accommodate longer network delays without halving the batch size of every microbatch.","At decentralized mining compute prices, a throughput of roughly 108 tokens/s already breaks even against typical Llama-70B inference pricing, and DeServe's optimized numbers exceed that threshold in every latency setting tested."],"supporting_citations":[{"why":"Supplies the KV cache concept that DeServe offloads and uses to bound per-microbatch memory.","marker":"Pope et al., 2023"},{"why":"Introduces the pipeline-parallel microbatch execution model that DeServe's scheduling extends.","marker":"Huang et al., 2019"},{"why":"Provides the pipeline schedule idea that DeServe adapts to fill network-latency bubbles.","marker":"Li et al., 2021"},{"why":"Shows KV cache streaming and offload, the basis for DeServe's global and local page pools.","marker":"Strati et al., 2024"},{"why":"Defines the Llama 3 architecture and layer structure used in the workload and memory estimates.","marker":"Dubey et al., 2024"}],"fun_headline_variants":["Idle GPUs across networks serve LLMs 12x faster","DeServe beats vLLM up to 12.6x on slow networks","KV cache offload delivers 6.7-12.6x throughput jump","Offline LLM inference gets decentralized speedup"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline gain assumes that DeServe's measured throughput advantage comes from its two optimizations and not from the fact that its own codebase is simply faster than the baseline implementation it is compared against.","fun_headline_variants_meta":{"raw":{"variants":["Idle GPUs across networks serve LLMs 12x faster","DeServe beats vLLM up to 12.6x on slow networks","KV cache offload delivers 6.7-12.6x throughput jump","Offline LLM inference gets decentralized speedup"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1353,"prompt_tokens":877,"completion_tokens":476,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":399}},"tokens_in":493,"tokens_out":476,"duration_ms":4919,"temperature":1.0,"reasoning_tokens":399,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:13:52.437487+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DeServe (opt) and the vLLM pipeline-parallel baseline on identical hardware with the same model and workload at controlled latencies of 16, 32, 64, and 256 ms, using the same latency-simulation mechanism for both systems; if the throughput gap narrows to well below 6.7x when baseline engineering is equalized, the claimed advantage is an artifact of implementation rather than of KV cache offloading and microbatch scheduling.","supporting_citations":[],"review_version":1}