Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attention

Gwangsun Kim; Hyungkyu Ham; Junhyeong Bae; Myeongjae Jeon; Seungheon Lee

REVIEW 2 major objections 6 minor 99 references

Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attention

T0 review · 2 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that retrieval-based sparse attention moves the binding constraint to the KV cache and index keys, and that a general-purpose processing-near-memory node holding that state—with GPUs running projections and MoE…

desk verdict A serious PNM design for retrieval-based sparse attention whose architectural logic is sound, but the headline numbers rest on an unvalidated HBM-to-LPDDR latency transfer in the simulator. read the letter →

arxiv 2608.03555 v1 pith:MTIBYTUE submitted 2026-08-04 cs.AR

Hyungkyu Ham , Junhyeong Bae , Seungheon Lee , Myeongjae Jeon , Gwangsun Kim This is my paper · ORCID

classification cs.AR

keywords retrieval-basedsparseattentionprocessing-near-memoryKVcacheoffloadingLLMservingmixture-of-expertsmicro-batchschedulingoperationalintensityagenticworkloads

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Retrieval-based sparse attention lets frontier LLMs serve million-token contexts by reading only a small top-k subset of the KV cache each decode step, but it must still store the entire cache plus per-token index keys, and it must scan all of those keys at every step. The paper's claim is that this scan changes where attention should be computed: not on the GPU, whose HBM is needed for model weights, and not in fixed-function memory-side units, which cannot track the score functions and selection granularities that change across models and generations. Working from that claim, the paper proposes KARAT, a node of general-purpose processing-near-memory devices that hold the KV cache and index keys, execute the scan, selection, gather, and attention, and leave projections and MoE layers to the GPU. If the paper is right, this split raises the sustainable batch size enough to improve SLO-constrained decode throughput per watt by 2.09–6.13x over a GPU-only baseline on three state-of-the-art models, and it runs training-free sparse attention methods without hardware change at 1.36–3.21x.

What carries the argument

KARAT (KV-cache-resident Accelerator for Retrieval-based ATtention): a device pairing 512 GB of LPDDR5X at 1.2 TB/s with an 18-core general-purpose compute engine rated 250 TFLOPS FP8 and 125 TFLOPS BF16, giving a ridge point around 208 FLOPs/byte, above the 4–128 FLOPs/byte operational-intensity range of the indexer scan. Operational intensity is the ratio of arithmetic to bytes moved, and the argument turns on the ridge point staying above the scan's intensity so the near-memory compute never becomes the bottleneck. Thirty-two such devices form a KARAT node with 16 TB capacity at the same 5.6 kW power and 38.4 TB/s bandwidth as an eight-GPU node. Two scheduling mechanisms complete the design:OFMS (opportunistic, fine-grained micro-batch scheduling) reorders GPU kernels across two micro-batches to hide expert all-to-all communication, and CMR (context-length-aware micro-batch rebalancing) repartitions requests by aggregate context length so the two micro-batches carry similar token counts despite the variance in long-context traces.

What would settle it

Run identical memory-bound attention and indexer kernels on a full H200 and on an H200 MIG1g.35gb instance, measure sustained DRAM bandwidth and per-kernel latency, and if the MIG slice delivers less bandwidth than the full H200, update the KARAT lookup tables and re-run the simulator to see whether the 2.09–6.13x margin survives.

Watch

Extended reading notes

Core claim

The central discovery is that the indexer scan—not attention itself—is the operation that dictates the design of a KV-cache-offload system for retrieval-based sparse attention. The scan reads every index key in the context on every decode step, so its memory traffic and FLOPs scale together, and its operational intensity of 4–128 FLOPs/byte does not improve with batch size, unlike the weight reads of dense and MoE layers. The paper therefore derives four requirements—capacity per watt an order of magnitude above HBM, bandwidth per watt comparable to HBM, a compute ridge point above the scan's intensity, and programmability across the five stages—and shows that LPDDR5X plus 18 general-purpose cores, as in a KARAT device, is the design point satisfying all four. With the KV cache on KARAT nodes, GPU HBM is dedicated to weights, the global batch grows until MoE layers become compute-bound, and two scheduling mechanisms hide the pipeline bubbles that arise when GPU and KARAT micro-batches alternate. The paper reports 2.09–6.13x throughput per TDP under a 100 ms P99 time-between-tokens SLO for model-native sparse attention, with the largest gains on the longest contexts.

Load-bearing premise

Every quantitative result rests on profiling KARAT kernels on an H200 MIG1g.35gb instance, assuming that slice provides the same memory bandwidth as a full H200 and that scaling the SM count linearly is valid; if MIG actually partitions or reduces memory bandwidth, the KARAT latency table understates memory-bound time and the reported speedups shrink.

Editorial extensions

If this is right

Freed from the KV cache, GPU HBM goes to model weights, so the decode batch can grow until the MoE layers become compute-bound; on the longest trace the paper reports sustained concurrency rising from roughly 250–340 sessions on the GPU-only system to about 4,000–5,000 with KARAT.
Adding memory capacity alone does not unlock the gain: the LPDDR-based SPNM baseline with the same capacity achieves only a 1.19x geometric-mean speedup versus KARAT's 2.23x, because its compute cannot sustain the indexer intensities of 64–128 FLOPs/byte.
Because the KARAT cores are programmable, training-free sparse attention methods from all three selection families—page-based, clustering-based, and hash-based—run without hardware change, improving throughput per watt by 1.36–3.21x over GPU-only.
The best system configuration shifts once KARAT holds the KV cache: the optimal expert-parallelism degree drops from 64 on the all-GPU system to 16–32, since the expert all-to-all then carries less data, and the grid search finds 1.7–7.8x throughput per TDP over the all-GPU column.
If successor memory technologies keep the same per-watt capacity and bandwidth ratios, the paper's design point carries over to future generations of LPDDR, GDDR, and HBM without re-derivation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The quantitative range is a simulator prediction built from H200 MIG profiling, not a measurement of a working KARAT system; a bandwidth microbenchmark on the MIG slice versus a full H200 is the cheapest way to test whether those latencies are optimistic.
The same partition rule could generalize beyond KV caches to any memory-resident inference state whose reads dominate and whose access algorithms drift across model generations, such as embedding tables, long-term memory, or tool-call histories.
The two schedulers are designed to be decentralized and latency-adaptive, so the micro-batch rebalancing idea is portable to any heterogeneous serving pipeline with stage-duration variance, not only the GPU-plus-PNM split.
A useful stress test is to rerun the evaluation under a stricter SLO (for example, a 50 ms P99 time-between-tokens target) or with longer traces, since the speedup stems from batch-size gains that a tighter latency target could partially cap.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my read. The thing to know: this is a serious systems proposal, and the central architectural move is right—put general-purpose compute next to large LPDDR KV capacity, sized to the retrieval indexer's operational intensity rather than low-OI dense attention. But every quantitative headline (2.09–6.13x, 1.36–3.21x) comes out of an in-house simulator whose KARAT latency model is borrowed from H200 MIG hardware without validating the HBM3e-to-LPDDR5X memory behavior. That is the soft spot that matters.

What's genuinely new: the R1–R4 requirements analysis is a useful reframing of prior PIM/PNM work. I buy that prior designs assume low OI and fixed-function attention, and that retrieval-based sparse attention breaks both assumptions. The OI numbers (4–128) ground the design in a reproducible way. OFMS and CMR are smaller ideas, but they are clean, and the ablation is honest. The evaluation is also more careful than most in this space: real agentic traces, a P99 TBT SLO, iso-power comparisons, sensitivity to link bandwidth/latency and core count, and they let SPNM beat KARAT on Quest/STARC rather than hiding it. The citation pattern is thorough and fair.

Soft spots, in order of weight. First, the calibration: the MIG slice matches KARAT's bandwidth, so the reader's worry about MIG partitioning is a misreading. The real issue is that a 1g.35gb H200 slice is HBM3e, not LPDDR5X; the paper adjusts energy only and leaves latency and controller behavior unchanged. For SCORE that may be fine—it is a streaming scan—but GATHER and ATTEND are random-access and latency-sensitive, and at the 4k–5k session operating points in Fig. 10 the KARAT iteration time sits close to the 100 ms SLO. If LPDDR5X access latency is worse than modeled, sustainable concurrency drops and the headline shrinks. No code, data, or prototype is released to check this. Second, the GPNM core count is tuned to the evaluated models; removing 8 cores costs 27% throughput, so the design point is sensitive to the benchmark set. That is not fatal—the sensitivity analysis exposes it—but it makes the absolute numbers conditional. Third, the simulator itself is in-house, and the quantitative results have no error bars. None of this undermines the qualitative argument, which rests on capacity and bandwidth reasoning rather than fitted constants.

Who should read it: architects and systems folks working on long-context serving, KV-cache offload, or PNM/PIM designs. It deserves a serious referee, not a desk reject. I'd send it to review and ask for code/data plus a calibration section that either measures LPDDR5X behavior or argues from JEDEC specs why latency invariance is safe.

Referee Report

2 major / 6 minor

Summary. The paper proposes KARAT, a heterogeneous LLM decode-serving system that moves the KV cache and index keys out of GPU HBM into processing-near-memory (PNM) devices built with LPDDR5X, while GPUs retain the model weights and execute projections and MoE layers. The authors derive four design requirements from an analysis of retrieval-based sparse attention, propose a general-purpose PNM compute engine sized to the operational intensity of the indexer scan, and add two scheduler mechanisms (OFMS and CMR) to reduce pipeline bubbles. The evaluation uses an in-house event-driven simulator calibrated with H200 profiling, three frontier MoE models, three real agentic traces, and several baselines (HiSparse, Duplex, AttAcc, SPNM), reporting 2.09-6.13x throughput-per-TDP over a GPU-only baseline under a 100 ms P99 TBT SLO for model-native sparse attention and 1.36-3.21x for training-free methods.

Significance. If the latency model is trustworthy, this is a meaningful architectural contribution. The paper correctly identifies that retrieval-based sparse attention has higher operational intensity than dense GQA/MHA, so prior PIM/PNM compute provisioning is insufficient; it provides a general-purpose design that covers multiple indexer families; and it shows capacity-driven batch-size gains under an iso-power constraint. The evaluation is careful in several respects: multiple independent baselines, iso-power system comparisons, sensitivity sweeps over link bandwidth, link latency, core count, and expert skew, a workload-aware configuration search, and conservative area/power estimates. However, all quantitative results rest on a simulator whose KARAT kernel latencies are calibrated from an H200 MIG slice and then assumed to transfer to LPDDR5X, and no artifact or prototype is available to check that calibration.

major comments (2)

[§IV-A and §IV-F] The KARAT kernel latency LUT is profiled on an H200 MIG1g.35gb slice, and Section IV-F states that only energy, not latency, is corrected for the LPDDR5X memory technology. I do not think the MIG partition itself is the problem, since the partitioned HBM bandwidth plausibly matches KARAT's 1.2 TB/s; the load-bearing unvalidated transfer is from HBM3e to LPDDR5X. GATHER and ATTEND are random-access and latency-sensitive, and Figure 10 shows KARAT iteration times at the 4k-5k session operating points sitting close to the 100 ms P99 TBT SLO. An optimistic LPDDR5X latency would therefore directly inflate the sustainable concurrency and every speedup in Figure 9. Please add a sensitivity sweep that scales KARAT memory access latency (e.g., 1.2x, 1.5x, 2.0x) and report whether the 2.09-6.13x headline range survives, or validate the conversion with a detailed DRAM controller/PNM model.
[§IV-A and §IV-B] The central quantitative claim is produced entirely by an in-house event-driven simulator with lookup-table models, and no artifact is released: there is no simulator binary, no LUT data, no trace preprocessing code, and no KARAT prototype. Because the headline throughput gains are simulator outputs rather than analytical bounds, an independent check is impossible. Please describe how the simulator was validated against an end-to-end GPU serving system beyond single-kernel profiling, and release the simulator, LUTs, and trace preprocessing so that the 2.09-6.13x result can be reproduced or falsified.

minor comments (6)

[Abstract and §III-E] There are missing spaces in 'proposeKARAT' and 'proposeopportunistic' in the abstract and main text; please fix these typographical issues.
[§IV-A] The phrase 'which provides the same memory bandwidth' is ambiguous: it should explicitly state the MIG slice's memory bandwidth (presumably 1.2 TB/s) and compare it with the KARAT device's LPDDR5X bandwidth, so the reader does not have to infer the intended referent.
[Figure 2] The stored/activated bar chart is hard to read: the caption, axis values, and the '32x/6.6x/9.8x' labels do not clearly identify which quantities are being compared, making the '6.6-32x less KV-cache traffic' claim difficult to verify. Please annotate the segments and state the formula used for the reduction ratio.
[Table II] The MiniMax-M3 top-k entry '128x16 (+1 local)' should be resolved to a single token count and block count so that the activated-traffic calculations in Section II-B are reproducible.
[Throughout] The phrase 'throughput per TDP' should be 'TDP-normalized throughput' or 'throughput per watt under TDP,' since TDP is a design limit rather than a measured power value; this would also make the normalization of the headline numbers clearer.
[Figure 14] The axis labels for the GPU/KARAT counts do not make the iso-power condition obvious; adding the aggregate TDP for each column would help the reader verify that all configurations are compared under the same 44.8 kW budget.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the throughput results are computed from independently profiled latency inputs and a design-point sweep, not reduced to their own inputs.

full rationale

The paper's headline speedups are produced by an event-driven simulator that consumes operation-latency lookup tables populated from offline profiling on H200 hardware, a MIG slice for KARAT kernels, ASTRA-sim communication models, and vendor memory specifications. The KARAT device parameters (18 GPNM cores, 175 W TDP, 512 GB LPDDR5X) come from a design-space sweep at fixed 1.2 TB/s bandwidth and from area/power estimation; they are inputs to the evaluation rather than values that re-encode the reported 2.09-6.13x outcome. Requirements R1-R4 are derived from measured observations (O1-O5) and motivate the architecture, but the architecture is then evaluated against GPU-only, HiSparse, Duplex, AttAcc, and SPNM baselines under a 100 ms P99 TBT SLO. None of the claimed predictions is defined in terms of its own output: no fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and self-citations such as [28] and [38] are contextual related-work comparisons rather than load-bearing premises. Sensitivity sweeps over link bandwidth, link latency, core count, and expert skew further show that the conclusions are not forced by construction. The MIG-vs-LPDDR5X kernel-latency calibration is a modeling-validity risk that belongs to correctness assessment, not circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The headline quantitative claims rest on the MIG bandwidth-parity assumption for KARAT latency, the core-count and TDP provisioning choices, and the LPDDR5X per-watt numbers taken from specifications. No unexplained physical entities are introduced beyond the proposed, unbuilt KARAT device.

free parameters (3)

GPNM core count = 18
Selected as the knee of the latency versus core-count sweep over the three evaluated models (Fig. 7a-b). Sensitivity (Fig. 13c) shows throughput drops 27% with 10 cores and rises only 3.8% with 26 cores, so the headline results depend on this provisioning.
KARAT device TDP = 175 W
Set at a quarter of the H200 TDP after measuring 147.7 W maximum on a MIG slice. The throughput-per-TDP normalization in Fig. 9 uses this value, so all normalized speedups are sensitive to it.
Micro-batch count = 2
OFMS and CMR assume exactly two micro-batches per pipeline; a different split would change bubble behavior and the reported scheduling gains.

assumptions (5)

ad hoc to paper An H200 MIG1g.35gb instance provides the same memory bandwidth as a full H200, and KARAT kernel latency scales linearly with SM count
Used in section IV-A to build the KARAT operation-latency lookup table. MIG partitions memory controllers per instance, so this may overstate KARAT bandwidth and understate kernel times.
domain assumption LPDDR5X per-watt capacity and bandwidth from JEDEC and vendor specifications are achievable without additional overhead beyond the modeled PHY
R1 and R2 rely on LPDDR5X having 19x the capacity/W of HBM3 at 0.8x the bandwidth/W (Table IV, from [70]); real system overheads could weaken R2.
domain assumption The indexer scan's operational intensity of 4-128 FLOPs/byte is representative for retrieval-based sparse attention
Measured on three frontier models (Table II, Fig. 4). R3 and the GPNM core sizing depend on this range; future indexers outside it would make the device compute-bound.
domain assumption MoE expert routing is uniform in the main evaluation
Section IV-A assigns tokens to experts by uniform sampling for the headline results; the skew analysis in section IV-D partially de-risks this assumption.
domain assumption Prefill-decode disaggregation with independent pools is the deployment context
The decode pool is evaluated in isolation (sections II-A and IV-A) with the prefill pool held constant; end-to-end gains including prefill are not quantified.

invented entities (1)

KARAT device (18 GPNM cores, LPDDR5X-9600, PCIe x4, 175 W)
purpose: Holds the KV cache and index keys outside GPU memory and executes SCORE, SELECT, GATHER, and ATTEND for retrieval-based sparse attention
The device is simulated, not fabricated; performance comes from H200 MIG profiling plus die-shot and JEDEC energy estimates, so there is no independent hardware measurement.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attention." pith.science (2026). https://pith.science/paper/MTIBYTUE

@misc{pith2026260803555,
  author       = {Pith},
  title        = {Pith review of: Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MTIBYTUE}},
  note         = {Machine review of arXiv:2608.03555}
}

read the original abstract

This paper presents a heterogeneous decode-phase serving system that relocates the KV cache out of GPU memory, motivated by the retrieval-based sparse attention that recent frontier LLMs adopt to serve million-token contexts. It partitions a decode step by operation type: GPU nodes hold the model weights and execute the projections and MoE layers, while processing-near-memory (PNM) nodes hold the KV cache and index keys and execute every operation that reads them. We first show that the assumptions behind prior PIM and PNM designs no longer hold for these operations, and derive four design requirements for such a node. From these requirements, we propose KARAT (KV-cache-resident Accelerator for Retrieval-based ATtention), a general-purpose PNM design that is the design point meeting all four. A KARAT device combines large LPDDR capacity with general-purpose compute sized for the retrieval indexer, serving an operational intensity beyond what PIM/PNM designs built for low-intensity GEMV target while accommodating diverse sparse attention algorithms that fixed-function units cannot support as they evolve. To reduce pipeline bubbles as the two device types alternate between micro-batches, we further propose opportunistic, fine-grained micro-batch scheduling (OFMS), which hides expert all-to-all behind the other micro-batch's GEMMs, and context-length-aware micro-batch rebalancing (CMR), which equalizes their token counts despite the variance in context length. Across three state-of-the-art models and real agentic traces, our proposed system improves throughput per TDP under a service-level objective by 2.09-6.13x over a GPU-only baseline and runs training-free sparse attention methods with 1.36-3.21x improvements.

Figures

Figures reproduced from arXiv: 2608.03555 by the authors.

**Figure 2.** Stored and activated KV cache and index keys at 1M context for the [PITH_FULL_IMAGE:figures/full_fig_p001_2.png] view at source ↗

**Figure 3.** Context length distribution of multi-turn agentic workloads [10], [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (8 more)

**Figure 4.** Figure 4: Decode cost of one layer that executes the retrieval indexer and sparse [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 6.** Figure 6: (a-d) Proposed system organization (e) Execution mapping of one decode layer. Colors denote the executing device: projections and MoE run on [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]

**Figure 7.** Figure 7: Compute provisioning for the KARAT device. Indexer scan (a) latency [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

**Figure 8.** Figure 8: Timeline showing different micro-batch scheduling approaches with [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

**Figure 9.** Figure 9: Throughput per TDP under a 100 ms P99 TBT SLO, normalized to [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]

**Figure 10.** Figure 10: Decode iteration time breakdown on the KVCT trace at the highest [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]

**Figure 13.** Figure 13: Sensitivity of KARAT to (a) GPU–KARAT link bandwidth, (b) [PITH_FULL_IMAGE:figures/full_fig_p010_13.png]

**Figure 14.** Figure 14: Iso-power (44.8 kW) design space exploration over the GPU-to [PITH_FULL_IMAGE:figures/full_fig_p010_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

99 extracted references · 42 canonical work pages

[1]

Nvidia h200 sxm 141 gb,

“Nvidia h200 sxm 141 gb,” NVIDIA. [Online]. Available: https: //www.nvidia.com/en-us/data-center/h200/
[2]

TensorRT-LLM,

“TensorRT-LLM,” NVIDIA. [Online]. Available: https://github.com/ NVIDIA/TensorRT-LLM
[3]

Lmcache: Turboboosting vllm with 7x faster access to 100x more kv caches,

“Lmcache: Turboboosting vllm with 7x faster access to 100x more kv caches,” [Online]. Available: https://blog.lmcache.ai/2024-09-17- release/, September 2024

2024
[4]

Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,

“Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,” Google DeepMind, 2025. [Online]. Available: https://storage. googleapis.com/deepmind-media/gemini/gemini_v2_5_report.pdf

2025
[5]

Openai models,

“Openai models,” 2025, accessed: 2025-07-13. [Online]. Available: https://platform.openai.com/docs/models

2025
[6]

Taming throughput-latency tradeoff in llm inference with sarathi-serve,

A. Agrawalet al., “Taming throughput-latency tradeoff in llm inference with sarathi-serve,” inProceedings of the 18th USENIX Conference on Operating Systems Design and Implementation, ser. OSDI’24. USA: USENIX Association, 2024

2024
[7]

Claude Opus 5,

Anthropic, “Claude Opus 5,” https://www.anthropic.com/news/claude- opus-5, 2026, large language model. Accessed: Jul. 24, 2026

2026
[8]

Introducing claude opus 4.6,

Anthropic, “Introducing claude opus 4.6,” https://www.anthropic.com/ news/claude-opus-4-6, 2026, large language model. Accessed via An- thropic API, Jul. 2026

2026

Show all 99 references

[9]

Indexcache: Accelerating sparse attention via cross-layer index reuse,

Y . Baiet al., “Indexcache: Accelerating sparse attention via cross-layer index reuse,”arXiv preprint arXiv:2603.12201, 2026

2026
[10]

Swe-chat: Coding agent interactions from real users in the wild,

J. Baumannet al., “Swe-chat: Coding agent interactions from real users in the wild,”arXiv preprint arXiv:2604.20779, 2026. [Online]. Available: https://arxiv.org/pdf/2604.20779

2026 arXiv
[11]

Retroinfer: A vector-storage approach for scalable long-context LLM inference,

Y . Chenet al., “Retroinfer: A vector-storage approach for scalable long-context LLM inference,”CoRR, vol. abs/2505.02922, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2505.02922

2025 doi arXiv
[12]

Magicpig: LSH sampling for efficient LLM generation,

Z. Chenet al., “Magicpig: LSH sampling for efficient LLM generation,” inThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. [Online]. Available: https://openreview.net/forum?id=ALzTQUgW8a

2025
[13]

Llmservingsim: A hw/sw co-simulation infrastructure for llm inference serving at scale,

J. Choet al., “Llmservingsim: A hw/sw co-simulation infrastructure for llm inference serving at scale,” in2024 IEEE International Symposium on Workload Characterization (IISWC), 2024, pp. 15–29

2024
[14]

37.3 a 2nm all-digital 14.4gb/s/pin lpddr6 phy with quarter-rate clocking architecture and multi-level fifo-based speculative dfe,

Y . Choiet al., “37.3 a 2nm all-digital 14.4gb/s/pin lpddr6 phy with quarter-rate clocking architecture and multi-level fifo-based speculative dfe,” in2026 IEEE International Solid-State Circuits Conference (ISSCC), vol. 69, 2026, pp. 628–630

2026
[15]

Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

DeepSeek-AI, “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” 2025. [Online]. Available: https: //arxiv.org/abs/2501.12948

2025 arXiv
[16]

Deepseek-v3.2: Pushing the frontier of open large language models,

DeepSeek-AI, “Deepseek-v3.2: Pushing the frontier of open large language models,” 2025

2025
[17]

Deepseek-v3 technical report,

DeepSeek-AIet al., “Deepseek-v3 technical report,”CoRR, vol. abs/2412.19437, 2024. [Online]. Available: https://doi.org/10.48550/ arXiv.2412.19437

2024 doi arXiv
[18]

Swe-bench pro: Can ai agents solve long- horizon software engineering tasks?

X. Denget al., “Swe-bench pro: Can ai agents solve long- horizon software engineering tasks?” 2025. [Online]. Available: https://arxiv.org/abs/2509.16941

2025 arXiv
[19]

com/EPIC-RPI/STARC, 2025, accessed: 2026-08-01

EPIC Lab, Rensselaer Polytechnic Institute, “STARC,” https://github. com/EPIC-RPI/STARC, 2025, accessed: 2026-08-01

2025
[20]

Starc: Selective token access with remapping and clustering for efficient llm decoding on pim systems,

Z. Fanet al., “Starc: Selective token access with remapping and clustering for efficient llm decoding on pim systems,” inProceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ser. ASPLOS ’26, 20...

2026
[21]

Mtia: First generation silicon targeting meta’s recommendation systems,

A. Firoozshahianet al., “Mtia: First generation silicon targeting meta’s recommendation systems,” inProceedings of the 50th Annual International Symposium on Computer Architecture, ser. ISCA ’23,
[22]

kv-cache-tester: Inference server cache performance test- ing suite,

C. Fox, “kv-cache-tester: Inference server cache performance test- ing suite,” https://github.com/callanjfox/kv-cache-tester, 2025, gitHub repository. Accessed: 2026-07-26

2025
[23]

Rdma over ethernet for distributed training at meta scale,

A. Gangidiet al., “Rdma over ethernet for distributed training at meta scale,” inProceedings of the ACM SIGCOMM 2024 Conference, ser. ACM SIGCOMM ’24, 2024, p. 57–70. [Online]. Available: https://doi.org/10.1145/3651890.3672233

2024
[24]

Seerattention: Self-distilled attention gating for efficient long-context prefilling,

Y . Gaoet al., “Seerattention: Self-distilled attention gating for efficient long-context prefilling,” inThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [Online]. Available: https://openreview.net/forum?id=Nf8yfPDFTl

2025
[25]

Glm-5: from vibe coding to agentic engineering,

GLM-5-Teamet al., “Glm-5: from vibe coding to agentic engineering,”
[26]

The llama 3 herd of models,

A. Grattafioriet al., “The llama 3 herd of models,” 2024. [Online]. Available: https://arxiv.org/abs/2407.21783

2024 arXiv
[27]

Pim is all you need: A cxl-enabled gpu-free system for large language model inference,

Y . Guet al., “Pim is all you need: A cxl-enabled gpu-free system for large language model inference,” inProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ser. ASPLOS ’25, 2025, p. 862–881. ...

2025
[29]

Fastermoe: modeling and optimizing training of large-scale dynamic pre-trained models,

J. Heet al., “Fastermoe: modeling and optimizing training of large-scale dynamic pre-trained models,” inProceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel 12 Programming, ser. PPoPP ’22, 2022, p. 120–134. [Online]. Available: https://doi.org/1...

2022
[30]

Shyla: 3d-stacked nvm-dram hybrid llm-inference ar- chitecture exploiting data and memory heterogeneity,

L. Heet al., “Shyla: 3d-stacked nvm-dram hybrid llm-inference ar- chitecture exploiting data and memory heterogeneity,” inProceedings of the 2026 ACM/IEEE 53rd Annual International Symposium on Computer Architecture (ISCA). IEEE, 2026, pp. 654–670

2026
[32]

Transparent offloading and mapping (tom): enabling programmer-transparent near-data processing in gpu systems,

K. Hsiehet al., “Transparent offloading and mapping (tom): enabling programmer-transparent near-data processing in gpu systems,” in Proceedings of the 43rd International Symposium on Computer Architecture, ser. ISCA ’16. IEEE Press, 2016, p. 204–216. [Online]. Available: https...

2016 doi
[33]

Hybridspec: Exploiting hybrid-bonding memory to accelerate llm serving through heterogeneous architecture and specu- lative decoding,

Z. Huanget al., “Hybridspec: Exploiting hybrid-bonding memory to accelerate llm serving through heterogeneous architecture and specu- lative decoding,” inProceedings of the 2026 ACM/IEEE 53rd Annual International Symposium on Computer Architecture (ISCA). IEEE, 2026, pp. 588–603

2026
[34]

codex_swebenchpro_traces,

Inferact Inc., “codex_swebenchpro_traces,” https://huggingface.co/ datasets/Inferact/codex_swebenchpro_traces, 2026, accessed: 2026-08- 01

2026
[35]

A cost-effective near-storage processing solution for offline inference of long-context llms,

H. Janget al., “A cost-effective near-storage processing solution for offline inference of long-context llms,” inProceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ser. ASPLOS ’26, 2026, p. 5...

2026
[36]

2023, standard

JEDEC Solid State Technology Association,Low Power Double Data Rate (LPDDR) 5/5X, JEDEC Solid State Technology Association, Jun. 2023, standard

2023
[37]

Scalable processing-near-memory for 1m-token llm inference: Cxl-enabled kv-cache management beyond gpu limits,

D. Kimet al., “Scalable processing-near-memory for 1m-token llm inference: Cxl-enabled kv-cache management beyond gpu limits,” in Proceedings of the 2025 International Conference on Parallel Archi- tectures and Compilation Techniques, ser. PACT ’25, 2025

2025
[38]

Toward standardized near-data processing with unrestricted data placement for gpus,

G. Kimet al., “Toward standardized near-data processing with unrestricted data placement for gpus,” inProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’17, 2017. [Online]. Available: https://doi.org/10.1145/3...

2017
[40]

Samsung pim/pnm for transfmer based ai : Energy efficiency on pim/pnm cluster,

J. H. Kimet al., “Samsung pim/pnm for transfmer based ai : Energy efficiency on pim/pnm cluster,” in2023 IEEE Hot Chips 35 Symposium (HCS), 2023, pp. 1–31

2023
[41]

A silicon-proven unified low-latency CXL controller and port-based routing switch for memory-centric fabrics,

M. Kwonet al., “A silicon-proven unified low-latency CXL controller and port-based routing switch for memory-centric fabrics,” in2026 ACM/IEEE 53rd Annual International Symposium on Computer Archi- tecture (ISCA). IEEE, 2026, pp. 688–702

2026
[42]

Efficient memory management for large language model serving with pagedattention,

W. Kwonet al., “Efficient memory management for large language model serving with pagedattention,” inProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. [Online]. Available: https://doi.org/10.1145/3600006.3613165

2023
[43]

Minimax sparse attention,

X. Laiet al., “Minimax sparse attention,” 2026. [Online]. Available: https://arxiv.org/abs/2606.13392

2026 arXiv
[44]

Hardware architecture and software stack for pim based on commercial dram technology,

S. Leeet al., “Hardware architecture and software stack for pim based on commercial dram technology,” inProceedings of the 48th Annual International Symposium on Computer Architecture, ser. ISCA ’21. IEEE Press, 2021, p. 43–56. [Online]. Available: https://doi.org/10.1109/ISCA...

2021
[45]

Pond: Cxl-based memory pooling systems for cloud platforms,

H. Liet al., “Pond: Cxl-based memory pooling systems for cloud platforms,” inProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ser. ASPLOS 2023, 2023, p. 574–587. [Online]. Available: https:...

2023
[46]

Snapkv: Llm knows what you are looking for before generation,

Y . Liet al., “Snapkv: Llm knows what you are looking for before generation,” inAdvances in Neural Information Processing Systems, A. Globersonet al., Eds., vol. 37. Curran Associates, Inc., 2024, pp. 22 947–22 970. [Online]. Available: https://proceedings.neurips.cc/paper_fil...

2024
[47]

Meridian: In-memory acceleration for rag with document attention decomposition,

C. Liuet al., “Meridian: In-memory acceleration for rag with document attention decomposition,” inProceedings of the 2026 ACM/IEEE 53rd Annual International Symposium on Computer Architecture (ISCA). IEEE, 2026, pp. 387–401

2026
[48]

Chime: A case for efficient long-context attention- fc disaggregated inference with dimm-pim,

Q. Liuet al., “Chime: A case for efficient long-context attention- fc disaggregated inference with dimm-pim,” inProceedings of the 2026 ACM/IEEE 53rd Annual International Symposium on Computer Architecture (ISCA). IEEE, 2026, pp. 620–636

2026
[49]

Nvidia’s ad102 officially revealed, how close were the previ- ous estimates?

Locuza, “Nvidia’s ad102 officially revealed, how close were the previ- ous estimates?” https://locuza.substack.com/p/nvidias-ad102-officially- revealed, sep 2022, accessed: 2026-07-26

2022
[50]

MoBA: Mixture of block attention for long- context LLMs,

E. Luet al., “MoBA: Mixture of block attention for long- context LLMs,” inThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [Online]. Available: https: //openreview.net/forum?id=RlqYCpTu1P

2025
[51]

Sac: Disaggregated kv cache system for sparse attention llms with cxl,

R. Maet al., “Sac: Disaggregated kv cache system for sparse attention llms with cxl,” 2026. [Online]. Available: https://arxiv.org/abs/2606. 19746

2026
[52]

Tpp: Transparent page placement for cxl-enabled tiered-memory,

H. A. Marufet al., “Tpp: Transparent page placement for cxl-enabled tiered-memory,” inProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ser. ASPLOS 2023, 2023, p. 742–755. [Online]. Availabl...

2023
[53]

HBM3E product brief,

Micron Technology, Inc., “HBM3E product brief,” https: //assets.micron.com/adobe/assets/urn:aaid:aem:b710d8f2-7f66-44c1- a234-456e2b986347/renditions/original/as/hbm3e-product-brief.pdf, 2024, form factor 11 mm×11 mm×0.72 mm; bandwidth>1.2 TB/s. Accessed: 2026-07-31

2024
[54]

g ed., May 2025, production Data Sheet

Micron Technology, Inc.,315b: x32 Automotive LPDDR5X SDRAM, rev. g ed., May 2025, production Data Sheet

2025
[56]

Early silicon of raptor: The first 3d-dram accelerator for generative inference,

P. J. Nairet al., “Early silicon of raptor: The first 3d-dram accelerator for generative inference,” inProceedings of the 2026 ACM/IEEE 53rd Annual International Symposium on Computer Architecture (ISCA). IEEE, 2026, pp. 2632–2647

2026
[57]

NVIDIA DGX B200 datasheet,

NVIDIA, “NVIDIA DGX B200 datasheet,” https://resources.nvidia. com/en-us-dgx-systems/dgx-b200-datasheet, 2024

2024
[58]

NVIDIA DGX H200 datasheet,

NVIDIA, “NVIDIA DGX H200 datasheet,” https://resources.nvidia. com/en-us-dgx-systems/dgx-h200-datasheet, 2024

2024
[59]

SLA-based planner,

NVIDIA, “SLA-based planner,” NVIDIA Dynamo Documenta- tion, https://docs.nvidia.com/dynamo/v-0-8-1/components/planner/sla- based-planner, 2026, accessed: 2026-08-01

2026
[60]

Multi-process service: Appendix: Tools and in- terface reference,

NVIDIA Corporation, “Multi-process service: Appendix: Tools and in- terface reference,” https://docs.nvidia.com/deploy/mps/appendix-tools- and-interface-reference.html, 2026, accessed: 2026-08-01

2026
[61]

(2026) Overall architecture — NVIDIA Dynamo documentation

NVIDIA Corporation. (2026) Overall architecture — NVIDIA Dynamo documentation. [Online]. Available: https://docs.nvidia.com/ dynamo/v1.3.0/design-docs/overall-architecture

2026
[62]

Planner design,

NVIDIA Corporation, “Planner design,” NVIDIA Dynamo Documentation, Design Docs — Component Design. https://docs.nvidia.com/dynamo/v1.3.0/design-docs/component- design/planner-design, 2026, v1.3.0 (stable). Accessed: 2026-08-01

2026
[63]

Supported MIG profiles — NVIDIA multi- instance GPU user guide,

NVIDIA Corporation, “Supported MIG profiles — NVIDIA multi- instance GPU user guide,” https://docs.nvidia.com/datacenter/tesla/ mig-user-guide/supported-mig-profiles.html#id7, 2026, table 11: GPU Instance Profiles on H200, v610. Accessed: 2026-08-01

2026
[64]

Fine-grained dram: energy-efficient dram for extreme bandwidth systems,

M. O’Connoret al., “Fine-grained dram: energy-efficient dram for extreme bandwidth systems,” inProceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecture, ser. MICRO-50 ’17, 2017, p. 41–54. [Online]. Available: https: //doi.org/10.1145/3123939.3124545

2017
[65]

Exegpt: Constraint-aware resource scheduling for llm inference,

H. Ohet al., “Exegpt: Constraint-aware resource scheduling for llm inference,” inProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ser. ASPLOS ’24, 2024, p. 369–384. [Online]. Available: htt...

2024
[66]

ChatGPT (GPT-5),

OpenAI, “ChatGPT (GPT-5),” https://chat.openai.com/, 2026, large language model. Accessed: Jul. 15, 2026. 13

2026
[68]

Attacc! unleashing the power of pim for batched transformer-based generative model inference,

J. Parket al., “Attacc! unleashing the power of pim for batched transformer-based generative model inference,” inProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ser. ASPLOS ’24, 2024, p. 1...

2024
[69]

A 192-gb 12-high 896-gb/s hbm3 dram with a tsv auto-calibration scheme and machine-learning-based layout opti- mization,

M.-J. Parket al., “A 192-gb 12-high 896-gb/s hbm3 dram with a tsv auto-calibration scheme and machine-learning-based layout opti- mization,” in2022 IEEE International Solid-State Circuits Conference (ISSCC), vol. 65, 2022, pp. 444–446

2022
[70]

An lpddr-based cxl-pnm platform for tco- efficient inference of transformer-based large language models,

S.-S. Parket al., “An lpddr-based cxl-pnm platform for tco- efficient inference of transformer-based large language models,” in 2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA), 2024, pp. 970–982. [Online]. Available: https://doi.org/10.1109/H...

2024
[71]

Apple m2 die shot and architecture analysis – big cost increase and a15 based ip,

D. Patel, “Apple m2 die shot and architecture analysis – big cost increase and a15 based ip,” https://semianalysis.com/2022/06/10/apple- m2-die-shot-and-architecture, jun 2022, accessed: 2025-11-04

2022
[72]

Splitwise: Efficient generative llm inference using phase splitting,

P. Patelet al., “Splitwise: Efficient generative llm inference using phase splitting,” in2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), 2024, pp. 118–132. [Online]. Available: https://doi.org/10.1109/ISCA59077.2024.00019

2024
[73]

Mooncake: A kvcache-centric disaggregated architecture for llm serving,

R. Qinet al., “Mooncake: A kvcache-centric disaggregated architecture for llm serving,”ACM Trans. Storage, Nov. 2025, just Accepted. [Online]. Available: https://doi.org/10.1145/3773772

2025 doi
[74]

Efficient interactive LLM serving with proxy model-based sequence length prediction,

H. Qiuet al., “Efficient interactive LLM serving with proxy model-based sequence length prediction,”CoRR, vol. abs/2404.08509,

arXiv
[75]

Longsight: Compute-enabled memory to accelerate large-context llms via sparse attention,

D. Quinnet al., “Longsight: Compute-enabled memory to accelerate large-context llms via sparse attention,” inProceedings of the 58th IEEE/ACM International Symposium on Microarchitecture, ser. MICRO ’25, 2025, p. 34–48. [Online]. Available: https: //doi.org/10.1145/3725843.3756062

2025
[76]

Drex: Accurate and scalable dense retrieval acceleration via algorithmic-hardware codesign,

D. Quinnet al., “Drex: Accurate and scalable dense retrieval acceleration via algorithmic-hardware codesign,” inProceedings of the 52nd Annual International Symposium on Computer Architecture, ser. ISCA ’25, 2025, p. 1108–1124. [Online]. Available: https: //doi.org/10.1145/369...

2025
[77]

Sparq attention: bandwidth-efficient llm inference,

L. Ribaret al., “Sparq attention: bandwidth-efficient llm inference,” inProceedings of the 41st International Conference on Machine Learning, ser. ICML’24. JMLR.org, 2024

2024
[78]

AttAcc simulator,

SCALE Lab, Seoul National University, “AttAcc simulator,” https:// github.com/scale-snu/attacc_simulator, 2024, accessed: 2026-08-01

2024
[79]

LLMSimulator,

SCALE Lab, Seoul National University, “LLMSimulator,” https:// github.com/scale-snu/LLMSimulator, 2024, accessed: 2026-08-01

2024
[80]

Toolformer: Language models can teach themselves to use tools,

T. Schicket al., “Toolformer: Language models can teach themselves to use tools,” inAdvances in Neural Information Processing Systems, A. Ohet al., Eds., vol. 36. Curran Associates, Inc., 2023, pp. 68 539–68 551. [Online]. Available: https://proceedings.neurips.cc/paper_files/...

2023
[81]

Ianus: Integrated accelerator based on npu-pim unified memory system,

M. Seoet al., “Ianus: Integrated accelerator based on npu-pim unified memory system,” inProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ser. ASPLOS ’24, 2024, p. 545–560. [Online]. Availab...

2024
[82]

Dynamollm: Designing llm inference clusters for performance and energy efficiency,

J. Stojkovicet al., “Dynamollm: Designing llm inference clusters for performance and energy efficiency,” in2025 IEEE International Symposium on High Performance Computer Architecture (HPCA), 2025, pp. 1348–1362. [Online]. Available: https://doi.org/10.1109/ HPCA61900.2025.00102

2025
[83]

Quest: query-aware sparsity for efficient long-context llm inference,

J. Tanget al., “Quest: query-aware sparsity for efficient long-context llm inference,” inProceedings of the 41st International Conference on Machine Learning, ser. ICML’24. JMLR.org, 2024

2024
[84]

Astra-sim2.0: Modeling hierarchical networks and disaggregated systems for large-model training at scale,

W. Wonet al., “Astra-sim2.0: Modeling hierarchical networks and disaggregated systems for large-model training at scale,” in2023 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 2023, pp. 283–294

2023
[85]

Memexplorer: Navigating the heterogeneous memory design space for agentic inference npus,

H. Wuet al., “Memexplorer: Navigating the heterogeneous memory design space for agentic inference npus,” 2026. [Online]. Available: https://arxiv.org/abs/2604.16007

2026 arXiv
[86]

Efficient streaming language models with attention sinks,

G. Xiaoet al., “Efficient streaming language models with attention sinks,”arXiv, 2023

2023
[87]

Hisparse: Turbocharging sparse attention with hierarchical memory,

Z. Xie, Z. Huang, and T. Huang, “Hisparse: Turbocharging sparse attention with hierarchical memory,” https://www.lmsys.org/blog/2026- 04-10-sglang-hisparse/, Apr. 2026, lMSYS Org Blog. Accessed: 2026- 07-26

2026
[88]

Strata: Hierarchical context caching for long context language model serving,

Z. Xieet al., “Strata: Hierarchical context caching for long context language model serving,”CoRR, vol. abs/2508.18572, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2508.18572

2025 doi arXiv
[90]

Beluga: A cxl-based memory architecture for scalable and efficient llm kvcache management,

X. Yanget al., “Beluga: A cxl-based memory architecture for scalable and efficient llm kvcache management,”Proc. ACM Manag. Data, vol. 4, no. 1, Apr. 2026. [Online]. Available: https://doi.org/10.1145/3786627

2026 doi
[91]

ReAct: Synergizing reasoning and acting in language models,

S. Yaoet al., “ReAct: Synergizing reasoning and acting in language models,” inInternational Conference on Learning Representations (ICLR), 2023

2023
[92]

Tract: Disaggregated llm serving with cxl shared memory kv cache at rack-scale,

D. Yoonet al., “Tract: Disaggregated llm serving with cxl shared memory kv cache at rack-scale,” 2025. [Online]. Available: https://arxiv.org/abs/2512.18194

2025
[93]

Available: https://arxiv.org/abs/2601.06288

[Online]. Available: https://arxiv.org/abs/2601.06288
[94]

Duplex: A device for large language models with mixture of experts, grouped query attention, and continuous batching,

S. Yunet al., “Duplex: A device for large language models with mixture of experts, grouped query attention, and continuous batching,” in2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO), 2024, pp. 1429–1443. [Online]. Available: https://doi.org/10.1109/MI...

2024
[95]

H2o: heavy-hitter oracle for efficient generative inference of large language models,

Z. Zhanget al., “H2o: heavy-hitter oracle for efficient generative inference of large language models,” inProceedings of the 37th International Conference on Neural Information Processing Systems, ser. NIPS ’23. Red Hook, NY , USA: Curran Associates Inc., 2023

2023
[96]

Deepep: an efficient expert-parallel communication library,

C. Zhaoet al., “Deepep: an efficient expert-parallel communication library,” https://github.com/deepseek-ai/DeepEP, 2025

2025
[97]

Patterns behind chaos: Forecasting data movement for efficient large-scale moe llm inference,

Z. Yuet al., “Patterns behind chaos: Forecasting data movement for efficient large-scale moe llm inference,” inProceedings of the 2026 ACM/IEEE 53rd Annual International Symposium on Computer Architecture (ISCA). IEEE, 2026, pp. 68–84

2026
[98]

Sglang: efficient execution of structured language model programs,

L. Zhenget al., “Sglang: efficient execution of structured language model programs,” inProceedings of the 38th International Conference on Neural Information Processing Systems, ser. NIPS ’24. Red Hook, NY , USA: Curran Associates Inc., 2024

2024
[99]

Distserve: disaggregating prefill and decoding for goodput-optimized large language model serving,

Y . Zhonget al., “Distserve: disaggregating prefill and decoding for goodput-optimized large language model serving,” ser. OSDI’24. USA: USENIX Association, 2024

2024
[100]

Octopus: Enhancing CXL memory pods via sparse topology,

Y . Zhonget al., “Octopus: Enhancing CXL memory pods via sparse topology,” in23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26). Renton, W A: USENIX Association, May 2026, pp. 1303–1322. [Online]. Available: https://www.usenix.org/conference/nsdi26/...

2026
[101]

InfLLM-v2: Dense-sparse switchable attention for seamless short-to-long adaptation,

W. Zhaoet al., “InfLLM-v2: Dense-sparse switchable attention for seamless short-to-long adaptation,” inThe Fourteenth International Conference on Learning Representations, 2026. [Online]. Available: https://openreview.net/forum?id=ZzF9V0H6Vi

2026
[105]

Megascale-infer: Efficient mixture-of-experts model serving with disaggregated expert parallelism,

R. Zhuet al., “Megascale-infer: Efficient mixture-of-experts model serving with disaggregated expert parallelism,” inProceedings of the ACM SIGCOMM 2025 Conference, ser. SIGCOMM ’25, 2025, p. 592–608. [Online]. Available: https://doi.org/10.1145/3718958. 3750506 14

2025 doi
[2023]

Available: https://doi.org/10.1145/3579371.3589348

[Online]. Available: https://doi.org/10.1145/3579371.3589348
[2024]

Available: https://doi.org/10.48550/arXiv.2404.08509

[Online]. Available: https://doi.org/10.48550/arXiv.2404.08509

doi arXiv
[2026]

Available: https://arxiv.org/abs/2602.15763

[Online]. Available: https://arxiv.org/abs/2602.15763

arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Nvidia h200 sxm 141 gb,

“Nvidia h200 sxm 141 gb,” NVIDIA. [Online]. Available: https: //www.nvidia.com/en-us/data-center/h200/

[2] [2]

TensorRT-LLM,

“TensorRT-LLM,” NVIDIA. [Online]. Available: https://github.com/ NVIDIA/TensorRT-LLM

[3] [3]

Lmcache: Turboboosting vllm with 7x faster access to 100x more kv caches,

“Lmcache: Turboboosting vllm with 7x faster access to 100x more kv caches,” [Online]. Available: https://blog.lmcache.ai/2024-09-17- release/, September 2024

2024

[4] [4]

Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,

“Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,” Google DeepMind, 2025. [Online]. Available: https://storage. googleapis.com/deepmind-media/gemini/gemini_v2_5_report.pdf

2025

[5] [5]

Openai models,

“Openai models,” 2025, accessed: 2025-07-13. [Online]. Available: https://platform.openai.com/docs/models

2025

[6] [6]

Taming throughput-latency tradeoff in llm inference with sarathi-serve,

A. Agrawalet al., “Taming throughput-latency tradeoff in llm inference with sarathi-serve,” inProceedings of the 18th USENIX Conference on Operating Systems Design and Implementation, ser. OSDI’24. USA: USENIX Association, 2024

2024

[7] [7]

Claude Opus 5,

Anthropic, “Claude Opus 5,” https://www.anthropic.com/news/claude- opus-5, 2026, large language model. Accessed: Jul. 24, 2026

2026

[8] [8]

Introducing claude opus 4.6,

Anthropic, “Introducing claude opus 4.6,” https://www.anthropic.com/ news/claude-opus-4-6, 2026, large language model. Accessed via An- thropic API, Jul. 2026

2026