Pith. sign in

REVIEW 4 major objections 7 minor 26 references

Structured knowledge priors can drive physical KV access so long-context serving no longer scales with full prompt length.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 19:46 UTC pith:NY66C4X4

load-bearing objection Clean systems framing and a real ablation, but the headline “decoupling” result is shown mainly where most of the long context is irrelevant by construction, and the speedups are modest. the 4 major comments →

arxiv 2607.24260 v1 pith:NY66C4X4 submitted 2026-07-27 cs.LG

KAP: Bridging the Knowledge Selection-Runtime Consumption Gap in LLM Systems

Shuo Wang , Fang Xi , Wenyuan Huang , Qing Wang , Junming Su This is my paper
classification cs.LG
keywords knowledge access planningruntime access planKV cachelong-context servingGraph-RAGspeculative decodingphase boundaryLLM systems
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

LLM systems already spend heavy work selecting what knowledge matters—ranked evidence, graphs, confidence—but once that work is flattened into a prompt, the serving backend treats every token the same and must keep reading the full key-value cache while decoding. The paper names this mismatch the Knowledge Selection–Runtime Consumption gap: richer upstream selection makes prompts longer and serving costlier even when only a small slice of context drives the answer. Knowledge Access Planning (KAP) treats those structured priors as first-class runtime artifacts. A compiler turns them into a runtime access plan—a universal intermediate representation that says which knowledge units to expose, how they map to cache objects, how to keep logical positions intact, and how to verify—while an executor uses the plan to guide physical KV access without changing the logical prompt, weights, or training. GraphSpec, a Graph-RAG plus serving-backend realization, keeps answer quality on par with full-context decoding on 4K–128K QA while cutting proposal-time KV access to 5.5% of the source state at 128K and raising decode throughput up to about 1.19×. A phase-boundary cost model explains when plan-guided execution wins: acceptance must stay high relative to retained KV so memory traffic savings beat verification overhead. If that holds, long-context cost can scale with selected knowledge rather than with serialized length.

Core claim

The authors claim that compiling structured knowledge-selection priors into a runtime access plan lets the serving backend consume only a selected KV view during proposal while preserving full logical prompt semantics and full-context verification, thereby decoupling physical decode-time KV traffic from prompt length and entering a positive-speedup regime when proposal acceptance stays high relative to retained KV.

What carries the argument

The runtime access plan—the universal IR A = (U, M, pos, ρ_access, ρ_verify)—compiles selected knowledge units, runtime object mappings, logical positions, and access/verification policies so the executor can run plan-guided selected-KV proposal with full-context verification without rewriting model conditioning.

Load-bearing premise

Proposal acceptance must stay high enough versus how little KV is kept that memory savings still beat verification and runtime overhead once real serving costs are included, and the upstream knowledge priors must be good enough to sustain that acceptance.

What would settle it

On the same long-context QA setup, if measured proposal acceptance falls to or below the retained-KV ratio (or phase slack goes non-positive) while answer quality holds, or if end-to-end decode throughput fails to rise as retained KV drops from ~40% toward ~5% across 4K–128K, the central speedup claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Physical decode cost can scale with selected evidence rather than with serialized context length when priors compile into high-acceptance, low-retention plans.
  • Knowledge-selection frontends become part of serving co-design: prior quality is judged by induced acceptance–retention margin, not only by retrieval metrics.
  • Prompt compression alone is not a substitute: materializing the same selected spans as a short prompt can lose quality that plan-guided access with full logical context preserves.
  • Other frontends (multimodal alignment, agent memory, confidence) can target the same IR if backend executors map plans onto their cache and attention objects.
  • Million-token windows need not imply proportionally larger physical KV traffic if plan-guided consumption stays inside the positive phase region.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Serving stacks that expose stable selected-KV workspaces and position-preserving metadata will be easier to plug into this style of plan execution than stacks that rebuild shapes every step.
  • Weak or poorly calibrated retrieval may force broader retention or frequent full-context fallback, so systems gains could vanish even when the IR is correctly implemented.
  • The same plan IR could later drive tiered or disaggregated KV placement, not only self-speculative proposal, if verification policies remain the quality gate.
  • Operators may start budgeting context by expected phase slack (acceptance minus retained ratio) rather than by raw token limits.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper identifies the "Knowledge Selection–Runtime Consumption" (KSRC) gap: knowledge-selection frontends (e.g., Graph-RAG) produce structured relevance priors, but prompt serialization discards them, so the serving backend consumes the full KV state uniformly at decode time. The proposed remedy, Knowledge Access Planning (KAP), is a compiler–executor architecture organized around a runtime access plan IR, A = (U, M, pos, ρ_access, ρ_verify), that carries frontend priors into the backend as physical KV-access policies while preserving logical prompt semantics. The authors instantiate KAP as GraphSpec (Graph-RAG frontend, vLLM backend): the base model proposes tokens attending only to a logical-position-preserving selected-KV view, and a full-context verification pass (Top-3 tolerant acceptance, k=6) commits tokens. A memory-bound cost model yields a phase boundary (h−s)KV > (1−h)W_main (Eqs. 9–13). Experiments on SPIQA with a nested fixed-evidence/hard-negative sweep from 4K to 128K report proposal-visible KV falling to 5.5% at 128K with h≈0.95, pass-rate parity with full-context decoding (Table 3), and decode-throughput speedup rising from 1.08× to 1.19× (Table 2), directionally consistent with the phase model (Table 5).

Significance. If the results hold, the paper makes a useful contribution: the KSRC gap is a real and articulately framed architectural mismatch, and elevating frontend priors to a first-class execution IR with explicit invariants (prior preservation, logical-position preservation, verification-aware execution) is a clean abstraction that could organize future serving-system design. Specific strengths: (i) the phase-boundary model is derived from a stated cost model with honest assumptions, not fitted to the speedup claim, and RQ4 checks only directional consistency without refitting; (ii) the RQ3 ablations (Physical/Random/Local concat) are a genuinely informative attempt to separate the plan mechanism from prompt shortening; (iii) the quality gate is supported by a blinded human audit of the LLM-judge threshold (95% agreement, threshold fixed ex ante, Table 1); (iv) GraphSpec requires no weight or training changes and is built on a widely used serving stack, which aids adoptability. That said, the empirical demonstration is confined to one model, one dataset, and one (deliberately easy) context-growth regime, and the measured end-to-end gains (≤1.19×) are more modest than the framing language;

major comments (4)
  1. [§5.1, Tables 2–3] §5.1 (context-length control) + Tables 2–3: the nested construction holds the core evidence fixed and appends query-specific hard negatives, so the ~94.5% of the 128K context excluded from the proposal view is non-answer-bearing by construction. h≈0.95 at s=0.055 is therefore close to guaranteed rather than evidence that plan-guided decoupling survives when relevant information is distributed across the context. The paper's own phase model (Eq. 12, h>s) concedes the coupling, but the abstract-level claims ('decouples physical KV consumption from prompt length', 'shifting the scaling trajectory') are validated only in the concentrated-relevance regime. A concrete, cheap test would address most of this: (a) randomize the position of the fixed evidence core within the 128K context (needle-position variation) and report h, s, pass rate per position bucket; (b) ideally, add one distributed-re
  2. [§5.4, Table 4] Table 4 (RQ3): the Physical-concat ablation is described as materializing 'the same evidence selected by GraphSpec', yet its served prompt is 0.95K tokens while GraphSpec's proposal-visible KV at 64K is 10.8% of ~64.9K ≈ 7K tokens. If the spans are identical, the budget mismatch needs explanation; if they are not, the comparison conflates evidence budget with execution strategy, and the 73.9% vs 87.9% pass-rate gap cannot be attributed to the runtime access plan alone. The 'decisive ablation' claim requires a budget-matched Physical-concat row (same selected spans at ~10.8% of context, served as a shortened prompt with original positional metadata either preserved or removed) so that the plan mechanism is isolated from the budget.
  3. [Abstract, §5.2, §8] The central efficiency claim is overstated relative to the measurements. Table 2 shows decode-throughput speedups of 1.08×–1.19× and TTFT essentially unchanged (38.31s vs 37.90s at 128K — prefill is untouched). The abstract's 'fundamentally shifting the scaling trajectory of long-context generation' is not supported by a sub-1.2× decode gain; the honest claim is a widening decode-throughput gap as context grows, consistent with Eq. 13. Please recalibrate the abstract/§1/§8 language to what Tables 2 and 5 actually show, and state explicitly that prefill cost and TTFT are out of scope of the current executor.
  4. [§5.1, §5.2, §6] Empirical scope: all results use a single model (Qwen3-VL-32B-Instruct) and a single benchmark (SPIQA), with no comparison against existing sparse-KV speculative systems (TriForce, RAPID, LongSpec — all cited in §6) or KV-selection methods (Quest, SnapKV). The isolation argument in §6 is reasonable for the KAP-vs-prompt-materialization question, but for the throughput claim the reader cannot tell whether 1.19× is good, typical, or weak relative to prior self-speculative long-context systems. At minimum, add one baseline comparison or a quantitative discussion of why the mechanisms are orthogonal and composable rather than competing. Relatedly, §5.1 never explains how SPIQA's multimodal evidence (figures, tables) enters the KV state or the plan; please clarify.
minor comments (7)
  1. [§4.1, Eq. (6)] Eq. (6): Cost_base = (1+kh)(W_main+KV) treats expected accepted tokens as kh. Under i.i.d. per-token acceptance h, E[accepted] = h(1−h^k)/(1−h) ≈ 5.3 for k=6, h=0.95 (vs kh=5.7). The quantitative impact on the boundary is small, but the assumption should be stated explicitly.
  2. [Table 2] Table 2: TTFT is non-monotone in context length (1.51s at 4K vs 0.65s at 8K) for both methods; presumably warmup or scheduling artifacts, but this should be explained or smoothed, since TTFT is a headline metric.
  3. [Table 5] Table 5: the column headers appear to be misaligned — the first numeric column (0.401…0.055) is s but the printed headers read 'Lengths h h−s'; the s header is missing.
  4. [§5.1] §5.1: the LLM-as-judge model identity and prompt are not specified; only the rubric weights and threshold are given. Given that pass rate is the primary quality metric, please name the judge model and report its context access (does the judge see the reference answer only, or also the source passages?).
  5. [Table 1] Table 1: gold-answer evidence recall of the frozen retrieval frontend is 68.8%; it would help to note how this bounds the achievable pass rate and whether the quality parity in Table 3 is conditional on retrieval coverage.
  6. [Abstract, §8] Tone: 'paradigm-shifting' (Abstract), 'fundamentally shifting', and 'universal IR' are stronger than a single-instantiation evaluation supports; consider tempering. Also check PDF artifacts such as 'GRAPHSPECmaintains' in the abstract.
  7. [§3.2] §3.2: it would help to state explicitly how RoPE/positional indices are assigned to proposed tokens generated over the compacted view (contiguous compact positions vs original logical positions), since this determines proposal–verifier logit comparability; the text asserts position preservation but the mechanism for the newly generated tokens is only implicit.

Circularity Check

0 steps flagged

No significant circularity: phase boundary is a stated cost-model inequality, and speedup/quality are measured against an independent full-context baseline.

full rationale

The paper’s load-bearing claims are systems/empirical, not a closed-form prediction forced by its inputs. Section 4 derives the positive-speedup condition from an explicit memory-bound cost model (Cost_plan vs Cost_base → Eqs. 9–13), with GraphSpec specialization W_prop = W_main yielding the necessary margin h > s and break-even KV_crit. That derivation does not define h or s in terms of measured speedup, nor fit them to force Δ_phase > 0; RQ4 only checks directional consistency of observed (s, h, h−s, speedup) “without refitting the cost model.” End-to-end TPS/TTFT and answer quality are measured against the same complete prompt and full-context vLLM baseline; the LLM-as-judge pass threshold is fixed a priori and audited against human majority vote (Table 1). Physical-concat / random / local-window ablations (Table 4) further separate plan-guided execution from mere prompt shortening. Related work cites external systems; there is no load-bearing self-citation uniqueness theorem or ansatz smuggled in as external fact. Experimental scope limits (nested hard-negative length sweep making relevant fraction shrink) affect generalization, not circularity of the derivation chain. No equation equates claimed speedup to a quantity defined as that speedup.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

Load-bearing content is mostly standard serving/RAG engineering plus modeling choices for the break-even analysis and evaluation protocol. No new physical entities. Free knobs are the usual systems hyperparameters (proposal length, Top-3 rule, KV budget, nested distractor construction) that shape reported h, s, and speedup.

free parameters (4)
  • proposal length k = 6
    Fixed to k=6 speculative tokens in experiments; cancels from ideal break-even but affects finite-cycle amortization and overheads.
  • Top-3 tolerant verification rule = accept if proposed token in full-context top-3
    Defines measured acceptance h_top3 used in the specialized phase boundary (Eq. 16) and executor; chosen for QA-oriented serving rather than derived.
  • retained-KV / evidence budget B = varies by length; 5.5% proposal KV at 128K
    Compiler selects evidence under budget B; directly sets s and couples to h. Values appear as operating points (e.g. 5.5% at 128K) rather than a learned law.
  • LLM-as-judge pass threshold 0.5 = 0.5
    Binary quality pass rate uses pre-fixed 0.5 on weighted similarity; audited but still a chosen cutoff on a proprietary-style judge rubric.
axioms (5)
  • domain assumption Autoregressive decode is well approximated as memory-bound with cost ≈ W_main + KV per full-context step; scheduler, metadata, sync, and launch overheads can be ignored for the idealized break-even.
    Stated in §4.1; underpins Eqs. 3–13 and the claim that h>s predicts positive speedup regime.
  • domain assumption Preserving logical positions while compacting physical KV pages leaves model conditioning and proposal/verifier logit alignment intact.
    Core IR invariant in §2.1 and GraphSpec materialization §3.1; required for verification-aware selective access without retraining.
  • domain assumption Graph-RAG hybrid retrieval priors (vector + rerank + node scores + PPR) identify spans sufficient for high proposal acceptance on the evaluation distribution.
    Frontend is treated as given (§3.1); empirical h≈0.95 depends on prior quality called out in §4.2 and Limitations.
  • ad hoc to paper Nested hard-negative lengthening with fixed core evidence isolates context-length effects without changing the answer-relevant set.
    Evaluation protocol in §5.1; supports RQ1–2 scaling claims but is a constructed regime, not organic corpus length growth.
  • domain assumption Same base model for proposal and verification implies W_prop = W_main.
    §4.2 Eq. 10 specialization to GraphSpec self-speculation.
invented entities (2)
  • Runtime access plan IR A = (U, M, pos, ρ_access, ρ_verify) no independent evidence
    purpose: Universal intermediate representation carrying frontend knowledge selection into backend physical KV access policy.
    Central abstraction of KAP (§2.1); engineering contract rather than a physical entity. Independent evidence is the GraphSpec realization and ablations, not an external measurement.
  • Knowledge Selection–Runtime Consumption (KSRC) gap no independent evidence
    purpose: Names the architectural mismatch the paradigm claims to bridge.
    Problem framing (§1); useful taxonomy, not an empirically distinct natural kind beyond known prompt-vs-runtime cost tension.

pith-pipeline@v1.2.0-grok45-kimik3 · 24273 in / 3906 out tokens · 79508 ms · 2026-07-31T19:46:40.049387+00:00 · methodology

0 comments
read the original abstract

Modern LLM systems increasingly rely on knowledge-selection processes that produce high-value structured priors, such as ranked evidence, graph topology, multimodal alignment, and confidence signals. Yet LLM serving remains fundamentally oblivious to this rich structure: once such signals are serialized into a prompt, the backend observes only a flat token sequence, forcing dense and uniform consumption of the full key-value (KV) state during decoding. We term this architectural mismatch the Knowledge Selection-Runtime Consumption (KSRC) gap: richer contexts enlarge the full-prompt KV footprint and decode-time memory traffic, increasing latency and degrading throughput even when reasoning depends on only a small fraction of the context. To bridge the gap, we propose Knowledge Access Planning (KAP), a paradigm-shifting execution abstraction that elevates structured knowledge priors from passive prompt-construction hints into first-class physical execution artifacts. KAP establishes a universal intermediate representation (IR)-the runtime access plan-which compiles structured knowledge signals to govern physical KV access without altering logical prompt semantics, model weights, or training procedures. Through this IR, KAP shifts LLM serving from token-aware context consumption to plan-driven, knowledge-aware runtime consumption. We instantiate KAP with GraphSpec, a compiler-executor realization connecting structured knowledge selection to an LLM serving backend. We derive a phase-boundary model for the positive-speedup regime of plan-guided execution. Across 4K-128K long-context QA workloads, GraphSpec maintains answer quality comparable to full-context decoding while decoupling physical KV consumption from prompt length, reducing proposal-time KV access to 5.5% of source KV state at 128K, and fundamentally shifting the scaling trajectory of long-context generation.

Figures

Figures reproduced from arXiv: 2607.24260 by Fang Xi, Junming Su, Qing Wang, Shuo Wang, Wenyuan Huang.

Figure 1
Figure 1. Figure 1: The KSRC gap and KAP-enabled LLM serving. A knowledge-selection frontend produces structured knowledge priors. Conventional serving exposes only a serialized prompt to the LLM serving backend and uniformly consumes full-context KV state, creating the KSRC gap. KAP instead carries these priors across the boundary through a runtime access plan—a universal IR that determines a selected-KV view and enables pla… view at source ↗
Figure 2
Figure 2. Figure 2: KAP compiler–executor architecture. The KAP compiler takes structured knowledge priors together with the complete logical prompt and emits a runtime access plan, the universal IR for LLM serving. The plan encodes selected knowledge units, runtime object mappings, logical positions, runtime access policy, and verification policy. The KAP executor interprets the plan to perform plan-guided selected-KV access… view at source ↗
Figure 3
Figure 3. Figure 3: GRAPHSPEC realization of the KAP compiler–executor architecture. Compiler: the GraphSpec KAP compiler grounds Graph-RAG priors in evidence and token spans, maps selected spans to paged-KV runtime objects, materializes a logical-position-preserving selected-KV view, and encodes the corresponding mappings and policies in a runtime access plan. Executor: the GraphSpec KAP executor interprets the plan through … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 18 linked inside Pith

  1. [6]

    URLhttps://arxiv.org/abs/2404.16130. Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed Aly, Beidi Chen, and Carole-Jean Wu. Layerskip: Enabling early exit inference and self-speculative decoding. In Proceedings of the 62nd Annual Meeting of the Asso...

    Pith/arXiv arXiv

  2. [7]

    Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang

    URLhttps://aclanthology.org/2024.acl-long.681/. Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of llm infer- ence using lookahead decoding. InProceedings of the 41st International Conference on Machine Learning,

    2024

  3. [8]

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang

    URLhttps://arxiv.org/abs/2402.02057. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. REALM: Retrieval- augmented language model pre-training. InProceedings of the 37th International Conference on Machine Learning,

    Pith/arXiv arXiv

  4. [10]

    Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H

    URLhttps: //aclanthology.org/2023.emnlp-main.825/. Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. InAdvances in Neural Information Processing Syst...

    Pith/arXiv arXiv 2023

  5. [11]

    Bernal Jimenez Gutierrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su

    URLhttps://arxiv.org/abs/2503.14649. Bernal Jimenez Gutierrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. Hipporag: Neu- robiologically inspired long-term memory for large language models. InAdvances in Neural Information Processing Systems,

    Pith/arXiv arXiv

  6. [12]

    Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, and Xin Jin

    URLhttps://arxiv.org/abs/2405.14831. Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, and Xin Jin. Ragcache: Efficient knowledge caching for retrieval-augmented generation.arXiv preprint arXiv:2404.12457,

    Pith/arXiv arXiv

  7. [13]

    Vladimir Karpukhin, Barlas O ˘guz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih

    URLhttps://arxiv.org/abs/2404.12457. Vladimir Karpukhin, Barlas O ˘guz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. InPro- ceedings of the 2020 Conference on Empirical Methods in Natural Language Processing,

    Pith/arXiv arXiv 2020

  8. [15]

    Yaniv Leviathan, Matan Kalman, and Yossi Matias

    URLhttps://doi.org/10.1145/3600006.3613165. Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via spec- ulative decoding. InInternational Conference on Machine Learning,

    arXiv

  9. [16]

    Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin

    URLhttps://arxiv.org/abs/2005.11401. Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. Compressing context to enhance inference efficiency of large language models. InProceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing,

    Pith/arXiv arXiv 2005

  10. [17]

    emnlp-main.391/

    URLhttps://aclanthology.org/2023. emnlp-main.391/. Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation. InAdvances in Neural Information Processing Systems, 2024a. URLhttps:// arxiv.org/abs/2404.14469. Yuhui Li, Fangyun...

    Pith/arXiv arXiv 2023

  11. [18]

    Songshuo Lu, Hua Wang, Yutian Rong, Zhi Chen, and Yaohua Tang

    URLhttps://aclanthology.org/ 2024.tacl-1.9/. Songshuo Lu, Hua Wang, Yutian Rong, Zhi Chen, and Yaohua Tang. Turborag: Accelerating retrieval-augmented generation with precomputed kv caches for chunked text. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,

    2024

  12. [19]

    URL https://aclanthology.org/2025.emnlp-main.334/. Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Ar- feen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating generative large language model serving with tree-based specula...

    2025

  13. [20]

    18 Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean

    URLhttps://dl.acm.org/doi/10.1145/3620666.3651335. 18 Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scal- ing transformer inference. InProceedings of Machine Learning and Systems,

    arXiv

  14. [21]

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Re, Ion Stoica, and Ce Zhang

    URL https://arxiv.org/abs/2211.05102. Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Re, Ion Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with a single gpu. InProceedings of the 40th International Con- ference on Machine Learning,

    Pith/arXiv arXiv

  15. [22]

    Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen

    URLhttps: //doi.org/10.1016/j.neucom.2023.127063. Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding.arXiv preprint arXiv:2404.11912,

    arXiv 2023

  16. [23]

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han

    URLhttps://arxiv.org/abs/2404.11912. Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context llm inference. InInternational Conference on Machine Learning,

    Pith/arXiv arXiv

  17. [24]

    Rishabh Tiwari, Haocheng Xi, Aditya Tomar, Coleman Richard Charles Hooper, Sehoon Kim, Maxwell Horton, Mahyar Najibi, Michael W

    URLhttps://arxiv.org/abs/2406.10774. Rishabh Tiwari, Haocheng Xi, Aditya Tomar, Coleman Richard Charles Hooper, Sehoon Kim, Maxwell Horton, Mahyar Najibi, Michael W. Mahoney, Kurt Keutzer, and Amir Gholami. QuantSpec: Self-speculative decoding with hierarchical quantized KV cache. InProceedings of the 42nd International Conference on Machine Learning, vol...

    Pith/arXiv arXiv

  18. [25]

    Wenhan Xiong, Xiang Li, Srini Iyer, Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mehdad, Scott Yih, Sebastian Riedel, Douwe Kiela, and Barlas Oguz

    URLhttps://arxiv.org/abs/2309.17453. Wenhan Xiong, Xiang Li, Srini Iyer, Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mehdad, Scott Yih, Sebastian Riedel, Douwe Kiela, and Barlas Oguz. Answering complex open- domain questions with multi-hop dense retrieval. InInternational Conference on Learning Rep- resentations,

    Pith/arXiv arXiv

  19. [27]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun

    URLhttps://arxiv.org/ abs/2405.16444. Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for transformer-based generative models. In16th USENIX Symposium on Operating Systems Design and Implementation,

    Pith/arXiv arXiv

  20. [28]

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E

    URLhttps://arxiv.org/abs/2306.14048. Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang: Efficient execution of structured language model programs. InAdvances in Neural Information Processing Systems,

    Pith/arXiv arXiv

  21. [29]

    URLhttps://arxiv.org/abs/2312.07104. 20

    Pith/arXiv arXiv

  22. [2020]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica

    URLhttps://aclanthology.org/2020.emnlp-main.550/. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th Symposium on Operating Systems Prin- ciples,

    2020

  23. [2021]

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu

    URLhttps: //aclanthology.org/2021.eacl-main.74/. Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Llmlingua: Com- pressing prompts for accelerated inference of large language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,

    2021

  24. [2022]

    URLhttps://doi.org/10.1109/SC41404.2022.00051. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Mil- lican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer...

    Pith/arXiv arXiv 2022

  25. [2023]

    Guanzheng Chen, Qilong Feng, Jinjie Ni, Xin Li, and Michael Qizhe Shieh

    URLhttps://arxiv.org/abs/2302.01318. Guanzheng Chen, Qilong Feng, Jinjie Ni, Xin Li, and Michael Qizhe Shieh. RAPID: Long-context inference with retrieval-augmented speculative decoding. InProceedings of the 42nd Interna- tional Conference on Machine Learning, volume 267 ofProceedings of Machine Learning Re- search, pp. 8093–8107. PMLR,

    Pith/arXiv arXiv

  26. [2024]

    16 Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, and Wen Xiao

    URLhttps://arxiv.org/abs/ 2401.10774. 16 Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, and Wen Xiao. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. InConference on Language Modeling,

    Pith/arXiv arXiv