Pith. sign in

REVIEW 2 major objections 6 minor 23 references

HBF Sucks! A Full-Stack Characterization of High-Bandwidth Flash for KV-Centric LLM Serving

T0 review · 2 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Swapping an SSD tier for High-Bandwidth Flash slows LLM serving by 2–5.5x, a full-stack simulation study finds.

desk verdict Important, well-executed negative result on drop-in HBF for transient KV, with trace representativeness as the main boundary. read the letter →

arxiv 2608.11668 v2 pith:T2Y5ZY3M submitted 2026-08-12 cs.AR

Zhuoran Li , Zhuohang Bian , Xin Huang , Yibo Zhao , Guangyu Sun , Youwei Zhuo This is my paper · ORCID
classification cs.AR
keywords largelanguagemodelsKVcachehigh-bandwidthflashmemorysystemsstorageLLMservingSSDoffloadingthermalthrottling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the obvious upgrade path for LLM serving—keep the Mooncake-style KV-offload stack and swap its SSD backing tier for High-Bandwidth Flash (HBF)—backfires. In a simulated H100/B200 serving campaign covering five models and four production traces, complete HBF-1 and HBF-2 organizations raise average end-to-end latency by 2–5.5x and cut maximum SLO goodput by 1.1–2.7x relative to capacity-matched SSD baselines. The explanation is that HBF's media speed barely touches the critical path: a 3.75x media-latency improvement moves end-to-end latency less than 1%, while the package trade halves GPU near-tier capacity and the hierarchy hands HBF a write-heavy, low-reuse stream that overheats it and wears it out. The paper's cost-benefit model says a faster far tier helps only when read I/O is the exposed bottleneck, reads outweigh writes, and delivered bandwidth is sustainable. If the paper is right, HBF belongs in LLM serving only via selective, reuse-aware placement with write budgeting and thermal coordination, not as a drop-in SSD replacement.

What carries the argument

The load-bearing mechanism is a single cost–benefit identity for the net latency change of swapping in an HBF tier: $\Delta_{\mathrm{HBF}} = f\,\Delta t_{\mathrm{media}} - C_{\mathrm{move}} - C_{\mathrm{pkg}} - C_{\mathrm{var}} - C_{\mathrm{life}}$. Faster flash improves only the first term, and only in proportion to $f$, the fraction of end-to-end time spent in exposed secondary-tier I/O; every other change the swap makes—near-tier capacity and bandwidth lost, movement and control overhead, throttling and variability, endurance—enters as a subtracted cost. From this identity the paper derives three falsifiable necessary conditions: C1 (read I/O is the serving bottleneck, so $f$ is large), C2 (useful reads per write exceed break-even, $\rho > \rho^*$), and C3 (sustained bandwidth stays near peak). Six findings falsify each condition for transient KV behind an SSD-style connector. The machinery also includes the extended TokenSim serving simulator, per-tier byte counters, and the 3D-ICE thermal model that turn these conditions into measurements.

What would settle it

Run the same capacity-matched HBF-2 versus SSD comparison on a production trace whose flash-tier reads-per-write exceeds the break-even threshold $\rho^*$ and whose HBF-resident KV fraction exceeds about 15%; if end-to-end latency then falls below the SSD baseline, the paper's three conditions fail to predict the outcome.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that replacing an SSD KV tier with High-Bandwidth Flash under an unchanged Mooncake-style runtime makes LLM serving slower on every request metric: average end-to-end latency rises 2–5.5x, delivered throughput falls 4–34%, and maximum SLO goodput falls 1.1–2.7x across H100 and B200. A media-latency sweep recovers the exposed secondary-tier fraction $f \approx 1\%$, meaning even an infinitely fast medium could remove at most about one percent of critical-path time; the dominant loss is the near-tier capacity and bandwidth the package trade gives up. Per-tier byte counters over full two-hour replays show writes outnumber reads on every trace (write/read ratio 1.14x–4.90x), SSD write batching transfers almost no benefit to HBF, a 3D-ICE thermal model caps sustained single-stack bandwidth at 202 GB/s, and a TLC HBF tier wears out at 0.56x the life of the capacity-matched SSD pool. The paper concludes that HBF is not the problem; using it as a faster SSD for transient KV is.

Load-bearing premise

The four Qwen-Bailian production traces are taken to represent the general class of transient KV workloads; if real deployments produce substantially more read-heavy flash traffic, such as long-lived shared prefixes or agent sessions, the write-heavy Finding 3 and the failure of condition C2 would not generalize.

Editorial extensions

If this is right

  • HBF should not be deployed as a drop-in replacement for an SSD KV-offload tier in Mooncake-style serving: doing so raises end-to-end latency and lowers SLO goodput at matched capacity.
  • The three conditions (critical-path read exposure, reads per write above break-even, sustainable bandwidth) are device-agnostic and can be used to evaluate other package-local far tiers such as CXL-attached or near-data flash.
  • HBF earns its place in LLM serving only for read-mostly, high-reuse objects such as model weights and shared prefixes, delivered with reuse-aware admission, write budgeting, and thermal coordination.
  • SSD-style write batching does not transfer to HBF because the medium lacks the fixed per-operation latency and random-access penalties that batching amortizes; at 512-token blocks the benefit turns negative.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if a future HBF-3 package restores near-tier HBM capacity (six HBM plus six HBF stacks), the dominant $C_{\mathrm{pkg}}$ penalty shrinks, and the three-condition model predicts HBF could help for KV if the runtime also raises the HBF-resident fraction beyond the roughly 15% measured here; this is directly testable with the paper's simulator.
  • Editorial inference: the paper's traces represent today's conversational and code workloads; agent sessions with long-lived tool-use histories or shared document corpora could produce a substantially higher flash-tier $\rho$, shifting the break-even boundary and potentially making some transient-KV placements viable.
  • Editorial inference: because batching gains vanish on HBF, device-specific optimizations should target write reduction—delaying writes, coarser write granularity, or in-place update schemes—rather than amortizing per-operation latency.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper asks whether replacing an SSD backing tier with High-Bandwidth Flash (HBF) in a Mooncake-style KV offloading stack improves LLM serving. It formalizes a cost-benefit model with three necessary conditions for a faster far tier to help: C1 (read I/O must be a large exposed fraction of the critical path), C2 (reads per write at the flash tier must exceed a break-even value), and C3 (sustained bandwidth must be near the rated peak). Using an extended TokenSim, four two-hour Qwen-Bailian production traces, five dense and MoE models, and H100/B200 profiles, the paper compares complete HBF-1 and HBF-2 architectures against capacity-matched SSD baselines. It reports that HBF-1 and HBF-2 are slower on every request metric (mean end-to-end latency 2--5.5x higher, maximum SLO goodput 1.1--2.7x lower), that the media-latency sweep recovers an exposed-fraction f of only about 1%, that the delivered reads-per-write rho is 0.20--0.88 on all four traces, that a 3D-ICE thermal model throttles the stack at about 202 GB/s, and that a TLC HBF tier wears out at 0.56x the lifetime of the SSD pool it replaces. The paper concludes that HBF as a drop-in SSD replacement for transient KV is net-negative, while acknowledging that HBF can help for read-mostly objects such as shared prefixes and weights.

Significance. If the results hold, the paper is a valuable negative result for an active research direction: it shows that simply swapping an SSD for package-local HBF under an SSD-oriented runtime can backfire, and it supplies a falsifiable three-condition framework (Eq. 6) that separates media speed from complete-architecture effects. The paper is unusually honest about its inputs: HBF service values are sourced or swept rather than presented as measured silicon, the media-latency sweep recovers f without fitting, the rho values come from full-replay per-tier byte counters, and the thermal and endurance estimates are based on explicit models with labeled assumptions. The architecture comparison is clearly scoped: the paper states that it evaluates complete HBF organizations under an SSD-style runtime, not the isolated medium. The main risk to significance is external validity, since the empirical core rests on four traces from one provider and one time window.

major comments (2)
  1. [§4.2, §7.1, §9.3] The generalization from four Qwen-Bailian traces to the broad class 'transient KV' is a load-bearing step for the C2 failure. The delivered rho in Table 7 is not an intrinsic property of transient KV; it emerges from the interaction between the Mooncake-style runtime's reuse policy, the near-tier capacity, and each trace's reuse distribution. The paper's own Figure 12 places shared prefixes in the 'conditional strong HBF regime,' so the boundary is acknowledged. If a deployment has shared-prefix or agent-session working sets that exceed the 48--96 GB near tier, the flash tier could receive a read-repaying stream and rho could exceed the break-even value. I request either a more careful scope (e.g., 'on these traces and this runtime') or an added sensitivity analysis, such as a synthetic read-heavy trace or a sweep over near-tier capacity, that quantifies when C2 would fail.
  2. [§8.1, Table 2] Finding 5 states that the stack 'hits the 80 C limit at only 202.27 GB/s ... far below interface peak,' but the per-stack bandwidth implied by Table 2 is about 200 GB/s: HBF-2/H100 has 0.6 TB/s across three stacks and HBF-1/H100 has 1.2 TB/s across six stacks. If the configured per-stack interface peak is 200 GB/s, then the thermal limit is not 'far below' peak but essentially at the configured peak, which changes the interpretation of the throttling result. Please clarify what 'interface peak' means at the single-stack level, state the per-stack versus aggregate bandwidth explicitly, and quantify how much the thermal controller reduces delivered bandwidth relative to the unconstrained run in the replay of Figure 10. This does not necessarily overturn Finding 5, but the current wording overstates the thermal margin and is not internally consistent with Table 2.
minor comments (6)
  1. [§5, Eq. (6)] Equation (6) is presented as a cost-benefit relation, but the cost terms C_move, C_pkg, C_var, and C_life are not individually measured or bounded; the paper measures f and rho directly and discusses the other terms qualitatively. Please state explicitly that Eq. (6) is a conceptual decomposition used to organize the necessary conditions, not a calibrated predictive model.
  2. [§6.1, Fig. 4] The text reports '4,741 matched pairs' but does not define what constitutes a matched pair or how many total comparisons were made across models, traces, systems, and load points. Please specify the counting rule so the statistic is reproducible.
  3. [§4.1] The extension validation is described as 'regression checks cover block/byte conservation, save/load ordering, deterministic replay, and per-tier counters,' but no quantitative results are given. A short table or a reference to the repository with these checks would strengthen confidence in the simulator extensions.
  4. [§8.2] Finding 6 extrapolates from two-hour replays to 'TB/day' without discussing diurnal traffic variation or the representativeness of the chosen two-hour window. Please add a caveat about daily extrapolation or report the sensitivity to the time window.
  5. [§7.2, Table 8] The HBF-2 controls rows ('batched, all blocks' and '1-block control') are presented without an SSD analogue; consider moving them to a separate table or clarifying why SSD analogues are not applicable so the reader does not misread them as paired comparisons.
  6. [Title / Abstract] The title 'HBF Sucks!' is informal for a journal venue and, taken alone, overstates the scope. Please consider a more neutral title that includes the qualifier 'as an SSD replacement for transient KV,' which is the actual claim of the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the negative result is measured, not derived from fitted inputs.

full rationale

The paper's derivation chain is a measurement campaign organized by a cost-benefit decomposition, not a circular prediction loop. Equation (6) is an identity separating the media-speed term from movement, package, variability, and lifetime costs; each term is then quantified independently. The exposed fraction f is recovered from a controlled media-latency sweep via the invertible relation in Eq. (7), where the measured end-to-end shift is a direct simulator output rather than a fitted parameter. The read/write ratio rho in Finding 3 comes from per-tier byte counters over full two-hour replays of four production traces. The thermal limit in Finding 5 is produced by an independent 3D-ICE model driven by sourced NAND read/write energies and a standard 80C junction limit, not by tuning the model to force throttling. The endurance comparison in Finding 6 uses a labeled, deliberately favorable TLC assumption with unit write amplification. The headline latency and goodput degradations (2-5.5x and 1.1-2.7x) are direct comparisons of complete HBF-1/HBF-2 organizations against capacity-matched SSD baselines in the simulator. The only self-citation is TokenSim, which is an open-source simulator with published real-machine validation; it is used as a tool, and the paper's regression checks cover the extensions. The paper also explicitly labels HBF device parameters as projected or swept and states that the architecture comparison does not isolate the medium, which is an honest boundary rather than a hidden assumption. The representativeness of the four Qwen-Bailian traces is a generalization risk, not a circularity, because the traces are external data and are not fitted to produce the conclusion. No load-bearing step reduces, by the paper's own equations or by self-citation, to its own inputs.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No numeric free parameters are fitted to produce the paper's conclusions; values are either sourced from cited hardware roadmaps, measured from traces, or explicitly swept. The central model (Eq. 6-8) has no fitted constants. The main load-bearing inputs are external domain assumptions about roadmap hardware, simulator fidelity, trace representativeness, thermal geometry, and endurance envelope.

assumptions (5)
  • domain assumption HBF roadmap parameters for HBF-1/HBF-2 (capacities, bandwidth, package power) are accurate projections of future hardware.
    Used throughout Sections 4 and 6; sourced from roadmap [9] and HAVEN [8], not measured silicon.
  • domain assumption The extended TokenSim faithfully models queueing, admission, and KV transfer behavior of a Mooncake-style serving system.
    Central results in Sections 6-8 are simulator outputs; only regression checks and prior TokenSim validation are offered in Section 4.1.
  • domain assumption The four Qwen-Bailian traces are representative of production transient KV workloads.
    Finding 3 and the write-heavy characterization rest entirely on these traces (Section 4.2, Section 7.1).
  • domain assumption A 16-Hi TLC NAND stack modeled in 3D-ICE with an 80 C junction limit approximates HBF thermal behavior.
    Finding 5 (Section 8.1) uses this model to derive the 202.27 GB/s sustainable bandwidth.
  • domain assumption TLC endurance with unit write amplification is a favorable bound for HBF wear.
    Finding 6 (Section 8.2) compares write rates against this budget; the authors state any real NAND management would lower it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HBF Sucks! A Full-Stack Characterization of High-Bandwidth Flash for KV-Centric LLM Serving." pith.science (2026). https://pith.science/paper/T2Y5ZY3M

@misc{pith2026260811668,
  author       = {Pith},
  title        = {Pith review of: HBF Sucks! A Full-Stack Characterization of High-Bandwidth Flash for KV-Centric LLM Serving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T2Y5ZY3M}},
  note         = {Machine review of arXiv:2608.11668}
}
abstract

A faster storage device should make serving faster. We find the opposite. High-Bandwidth Flash (HBF) stacks NAND behind a wide, package-local interface, promising flash-scale capacity with far lower read latency and higher bandwidth than an SSD. The obvious move is to keep an SSD-style Mooncake KV-offloading stack and swap in HBF underneath. We built that system and measured it: an extended TokenSim, four complete two-hour Qwen-Bailian production traces, five dense and mixture-of-experts models, and H100/B200 profiles. The upgrade backfires. Average end-to-end latency rises 2--5.5$\times$ and maximum SLO goodput falls 1.1--2.7$\times$ across H100 and B200, so the faster device yields a slower system. A cost-benefit model explains the paradox: a faster far tier pays off only when read I/O is the bottleneck, reads outweigh writes, and delivered bandwidth is sustainable. Transient KV violates all three at once. Buying flash through the package costs GPU near-tier capacity and bandwidth, while HBF's own read/write latency barely matters: scaling it 3.75$\times$ moves latency less than 1\%. Worse, the two-tier hierarchy keeps reuse in the near tier and hands HBF a relentless write-heavy stream. Writes outnumber reads on every trace, so a 3D-ICE model shows the stack hits its thermal limit well below peak bandwidth, and a TLC tier wears out sooner than the SSD pool it replaced. The device is fine; the drop-in deployment is not. HBF sucks as an SSD replacement for transient KV, but earns its place in LLM serving when used selectively with reuse-aware placement, write budgeting, and thermal coordination.

Figures

Figures reproduced from arXiv: 2608.11668 by the authors.

Figure 1
Figure 1. SSD offloading of KV: host-staged path (a) versus [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Roadmap HBF system layouts. halving near-tier capacity versus an all-HBM package—is an ac￾ceptable trade rather than the wholesale demotion HBF-1 makes. HBM holds weights and active, write-heavy state while HBF holds long-context KV, and one scheduler sees compute, HBM, and HBF over the local memory path; the cost moves from bandwidth loss to package area, cooling, and the near-tier capacity given up. 3.2.3 HBF-3: i… view at source ↗
Figure 4
Figure 4. A faster connector, a slower system: HBF cuts trans [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: End-to-end latency versus load on B200, normalized within each panel to the low-load SSD12 value ( [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Delivered token throughput normalized to SSD12 ( [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Time-to-first-token (top) and time-between-tokens (bottom) versus load, normalized within each panel to the low [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Base-die NMP operator leverage, HBF coverage, [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 10
Figure 10. Figure 10: (a) Dynamic power and peak temperature vs. [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 11
Figure 11. Figure 11: KV write volume and endurance budget. A–T de [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 12 canonical work pages

  1. [1]

    Haratsch, Ken Mai, and Onur Mutlu

    Yu Cai, Yixin Luo, Erich F. Haratsch, Ken Mai, and Onur Mutlu. 2015. Data Retention in MLC NAND Flash Memory: Characterization, Optimization, and Recovery. InProceedings of the 21st IEEE International Symposium on High Per- formance Computer Architecture. 551–563. doi:10.1109/HPCA.2015.7056062

    arXiv 2015

  2. [2]

    DeepSeek. 2026. Context Caching on Disk. DeepSeek API Documentation. Ac- cessed: 2026-07-15. https://api-docs.deepseek.com/guides/kv_cache/

    work page 2026

  3. [3]

    Eran Gal and Sivan Toledo. 2005. Algorithms and Data Structures for Flash Memories.Comput. Surveys37, 2 (2005), 138–163. doi:10.1145/1089733.1089735

    work page arXiv 2005

  4. [4]

    Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. 2024. Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention. In2024 USENIX Annual Technical Conference (USENIX ATC 24). USENIX Asso- ciation, Santa Clara, CA, 111–126. https://www.usenix.org/conferen...

    2024

  5. [5]

    In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khan- delwal, and Lin Zhong. 2024. Prompt Cache: Modular Attention Reuse for Low-Latency Inference. InProceedings of Machine Learning and Systems, P. Gibbons, G. Pekhimenko, and C. De Sa (Eds.), Vol. 6. 325–338. https://proceedings.mlsys.org/paper_files/paper/2024/file/ a66caa1703fe34705a4368c3014c...

    2024

  6. [6]

    Minho Ha, Euiseok Kim, and Hoshik Kim. 2026. H3: Hybrid Architecture Using High Bandwidth Memory and High Bandwidth Flash for Cost-Efficient LLM Inference.IEEE Computer Architecture Letters25, 1 (2026), 49–52. doi:10.1109/ LCA.2026.3660969

    work page arXiv 2026

  7. [7]

    Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. 2024. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization. InAd- vances in Neural Information Processing Systems (NeurIPS)

    2024

  8. [8]

    Po-Kai Hsu, Weihong Xu, Qunyou Liu, Tajana Rosing, and Shimeng Yu. 2026. HAVEN: High-Bandwidth Flash Augmented Vector Engine for Large-Scale Ap- proximate Nearest-Neighbor Search Acceleration. arXiv:2603.01175 [cs.AR] https://arxiv.org/abs/2603.01175

    work page arXiv 2026

Show all 23 references
  1. [9]

    Joungho Kim. 2026. HBF Technology, Workload Analysis and Roadmap. Pre- sentation, TeraByte Interconnection and Package Laboratory, KAIST, February 10, 2026

    2026

  2. [10]

    KIOXIA America, Inc. 2024. KIOXIA SSDs for Supermicro Servers: Data Sheet. Version 1.0. https://americas.kioxia.com/content/dam/kioxia/en-us/business/ ssd/oem/supermicro/asset/KIOXIA_Supermicro_SSD_Data_Sheet_Global.pdf

    2024

  3. [11]

    KVCache.AI Team. 2026. Mooncake SSD Offloading: Breaking Through Mem- ory Limits to Expand KV Cache Capacity. Mooncake Design Documenta- tion. Accessed: 2026-07-19. https://kvcache-ai.github.io/Mooncake/design/ssd- offload.html

    2026

  4. [12]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Mem- ory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Princ...

    2023

  5. [13]

    Linux Kernel Documentation. 2026. io_uring Subsystem Documentation. Ac- cessed: 2026-07-15. https://docs.kernel.org/io_uring/

    2026

  6. [14]

    Yuhan Liu, Yihua Cheng, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Rui Zhang, Kuntai Du, and Junchen Jiang. 2025. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference. arXiv:2510.09665 [cs.DC] https://arxiv.org/abs/2510.09665

    2025

  7. [15]

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. InProceedings of the 41st International Confer- ence on Machine Learning (ICML)

    2024

  8. [16]

    Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yong- wei Wu, Weimin Zheng, and Xinran Xu. 2025. MOONCAKE: trading more stor- age for less computation — a KVCache-centric architecture for serving LLM chatbot. InProceedings of the 23rd USENIX Conference ...

    2025

  9. [17]

    Shi Qiu, Yifan Hu, Xintao Wang, Wenhao Zhu, Jianqin Yan, Hao Chen, Kaiqiang Xu, Kai Chen, and Yiming Zhang. 2026. Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serving. arXiv:2605.03375 [cs.DC] https://arxiv. org/abs/2605.03375

    2026 arXiv

  10. [18]

    Arvind Sridhar, Alessandro Vincenzi, Martino Ruggiero, Thomas Brunschwiler, and David Atienza. 2010. 3D-ICE: Fast Compact Transient Thermal Modeling for 3D ICs with Inter-tier Liquid Cooling.Proceedings of the IEEE/ACM International Conference on Computer-Aided Design (ICCAD)(...

    2010

  11. [19]

    Jiahao Wang, Jinbo Han, Xingda Wei, Sijie Shen, Dingyan Zhang, Chenguang Fang, Rong Chen, Wenyuan Yu, and Haibo Chen. 2025. KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider. InProceedings of the 2025 USENIX Annual Technical Confe...

    2025

  12. [20]

    Xinyu Wang, Yalong Xue, Xiaotian Sun, Xiaoyu Zhang, Chunmeng Dou, Xueqi Li, and Xiaoming Chen. 2026. FlashAccel: Leveraging High-Bandwidth Flash for High-Throughput LLM Inference. arXiv:2607.10186 [cs.AR] https://arxiv.org/ abs/2607.10186

    2026 arXiv

  13. [21]

    Feiyang Wu, Zhuohang Bian, Guoyang Duan, Tianle Xu, Junchi Wu, Teng Ma, Yongqiang Yao, Ruihao Gong, and Youwei Zhuo. 2025. TokenSim: Enabling Hardware and Software Exploration for Large Language Model Inference Sys- tems. arXiv:2503.08415 [cs.DC] https://arxiv.org/abs/2503.08415

    2025 arXiv

  14. [22]

    Zhiqiang Xie, Ziyi Xu, Mark Zhao, Yuwei An, Vikram Sharma Mailthody, Scott Mahlke, Michael Garland, and Christos Kozyrakis. 2025. Strata: Hierarchical Context Caching for Long Context Language Model Serving. arXiv:2508.18572 [cs.DC] https://arxiv.org/abs/2508.18572

    2025 arXiv

  15. [23]

    Lui, and Haibo Chen

    Yanqi Zhang, Yuwei Hu, Runyuan Zhao, John C.S. Lui, and Haibo Chen. 2025. DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction. InProceedings of the 19th USENIX Symposium on Op- erating Systems Design and Implementation (OSDI). USENIX ...

    2025

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.