{"id":"0c81363d-10bc-476e-b3c8-6568af74fdee","arxiv_id":"2608.03555","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Heterogeneous serving that moves the KV cache and retrieval-based sparse attention to general-purpose processing-near-memory devices improves simulated decode throughput per TDP by 2.09-6.13x over a GPU-only baseline.","lead":"This paper proposes KARAT, a memory-side accelerator that holds an LLM's long-context KV cache and runs the retrieval step of sparse attention while GPUs keep the model weights and MoE layers. In simulation, pairing GPUs with KARAT devices improves SLO-constrained decode throughput per watt by 2.09-6.13x over a GPU-only system.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"KARAT kernel latencies are borrowed from an H200 MIG slice without correcting HBM3e-versus-LPDDR5X memory behavior; because the indexer scan dominates KARAT time and the P99 TBT SLO is tight, an optimistic KARAT LUT can inflate all reported speedups.","rationale":"I agree with the CONDITIONAL verdict. The paper's design chain is coherent: observations O1–O5 lead to four requirements, which motivate LPDDR5X capacity, general-purpose near-memory compute, and the OFMS/CMR scheduling techniques. The evaluation uses real agentic traces, H200 profiling, and the released simulators for the baselines, and the ablation and sensitivity analyses are reasonable. The strongest quantitative claim, however, is entirely mediated by an in-house simulator whose KARAT-side LUT is calibrated on an H200 MIG slice. The reader's stated weakest assumption misidentifies the mechanism: the MIG slice's partitioned bandwidth is not a bug but the intended way to approximate KARAT's 1.2 TB/s LPDDR5X bandwidth. What remains unvalidated is the HBM3e-to-LPDDR5X transfer of latency and controller behavior. Because the indexer scan is the dominant KARAT-side cost and the 100 ms P99 TBT SLO is the binding constraint at the operating points in Fig. 10, even a modest optimistic bias in the KARAT LUT would shift the sustainable concurrency and compress or erase the reported 2.09–6.13x. The paper also gives no error bars and releases no code or data, so the magnitude of this bias cannot be assessed from the manuscript. I would not reject: the architecture is plausible and the sensitivity analysis shows robustness to link parameters. But the central numbers should remain conditional on an LPDDR5X-specific calibration.","tokens_in":26787,"tokens_out":19351,"duration_ms":169064,"concrete_test":"Run a controlled calibration experiment: on the same workload, execute (i) a sequential 1.2 TB/s streaming read and (ii) a 2 KB random gather on the H200 MIG1g.35gb slice and on a real LPDDR5X-9600 CXL-PNM/CXL-memory module. Derive per-operation latency ratios (LPDDR5X/HBM) for streaming and random access. Replace the KARAT LUT entries accordingly (SCORE scaled by the streaming ratio, GATHER/ATTEND by the random-access ratio plus controller overhead), keep all other simulator inputs fixed, and rerun the §IV-B model/trace matrix. If the ratio exceeds roughly 1.2–1.5 for random access, the P99-TBT-limited concurrency for KARAT configurations should be recomputed; the headline 2.09–6.13x range should be reported with the corrected LUT.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In §IV-A the authors derive the KARAT latency LUT by profiling an H200 MIG1g.35gb slice 'which provides the same memory bandwidth' and scaling SM count. The reader's version of this concern—that MIG partitioning understates bandwidth—appears to rest on a misreading: the intended referent is the KARAT device's own 1.2 TB/s LPDDR5X bandwidth, and the MIG slice's partitioned HBM bandwidth is precisely what makes it a bandwidth match. The load-bearing issue is therefore not MIG partitioning per se but the unvalidated transfer from HBM3e to LPDDR5X. The MIG slice's memory system has HBM3e latency, row/bank behavior, and refresh/controller characteristics; §IV-F adjusts only energy and leaves latencies unchanged. SCORE is a streaming scan and may be insensitive, but GATHER and ATTEND are random-access and latency-sensitive, and at the 4k–5k session operating points in Fig. 10 KARAT iteration time sits close to the 100 ms P99 TBT SLO. If LPDDR5X per-access latency or controller overhead makes the real kernels slower than the MIG-derived LUT, the simulator's sustainable concurrency—and hence the 2.09–6.13x headline—falls. No code, data, or a real LPDDR5X PNM prototype is released to check this calibration.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes KARAT, a heterogeneous LLM decode-serving system that moves the KV cache and index keys out of GPU HBM into processing-near-memory (PNM) devices built with LPDDR5X, while GPUs retain the model weights and execute projections and MoE layers. The authors derive four design requirements from an analysis of retrieval-based sparse attention, propose a general-purpose PNM compute engine sized to the operational intensity of the indexer scan, and add two scheduler mechanisms (OFMS and CMR) to reduce pipeline bubbles. The evaluation uses an in-house event-driven simulator calibrated with H200 profiling, three frontier MoE models, three real agentic traces, and several baselines (HiSparse, Duplex, AttAcc, SPNM), reporting 2.09-6.13x throughput-per-TDP over a GPU-only baseline under a 100 ms P99 TBT SLO for model-native sparse attention and 1.36-3.21x for training-free methods.","tokens_in":27172,"tokens_out":13885,"duration_ms":123935,"significance":"If the latency model is trustworthy, this is a meaningful architectural contribution. The paper correctly identifies that retrieval-based sparse attention has higher operational intensity than dense GQA/MHA, so prior PIM/PNM compute provisioning is insufficient; it provides a general-purpose design that covers multiple indexer families; and it shows capacity-driven batch-size gains under an iso-power constraint. The evaluation is careful in several respects: multiple independent baselines, iso-power system comparisons, sensitivity sweeps over link bandwidth, link latency, core count, and expert skew, a workload-aware configuration search, and conservative area/power estimates. However, all quantitative results rest on a simulator whose KARAT kernel latencies are calibrated from an H200 MIG slice and then assumed to transfer to LPDDR5X, and no artifact or prototype is available to check that calibration.","major_comments":[{"comment":"The KARAT kernel latency LUT is profiled on an H200 MIG1g.35gb slice, and Section IV-F states that only energy, not latency, is corrected for the LPDDR5X memory technology. I do not think the MIG partition itself is the problem, since the partitioned HBM bandwidth plausibly matches KARAT's 1.2 TB/s; the load-bearing unvalidated transfer is from HBM3e to LPDDR5X. GATHER and ATTEND are random-access and latency-sensitive, and Figure 10 shows KARAT iteration times at the 4k-5k session operating points sitting close to the 100 ms P99 TBT SLO. An optimistic LPDDR5X latency would therefore directly inflate the sustainable concurrency and every speedup in Figure 9. Please add a sensitivity sweep that scales KARAT memory access latency (e.g., 1.2x, 1.5x, 2.0x) and report whether the 2.09-6.13x headline range survives, or validate the conversion with a detailed DRAM controller/PNM model.","section":"§IV-A and §IV-F"},{"comment":"The central quantitative claim is produced entirely by an in-house event-driven simulator with lookup-table models, and no artifact is released: there is no simulator binary, no LUT data, no trace preprocessing code, and no KARAT prototype. Because the headline throughput gains are simulator outputs rather than analytical bounds, an independent check is impossible. Please describe how the simulator was validated against an end-to-end GPU serving system beyond single-kernel profiling, and release the simulator, LUTs, and trace preprocessing so that the 2.09-6.13x result can be reproduced or falsified.","section":"§IV-A and §IV-B"}],"minor_comments":[{"comment":"There are missing spaces in 'proposeKARAT' and 'proposeopportunistic' in the abstract and main text; please fix these typographical issues.","section":"Abstract and §III-E"},{"comment":"The phrase 'which provides the same memory bandwidth' is ambiguous: it should explicitly state the MIG slice's memory bandwidth (presumably 1.2 TB/s) and compare it with the KARAT device's LPDDR5X bandwidth, so the reader does not have to infer the intended referent.","section":"§IV-A"},{"comment":"The stored/activated bar chart is hard to read: the caption, axis values, and the '32x/6.6x/9.8x' labels do not clearly identify which quantities are being compared, making the '6.6-32x less KV-cache traffic' claim difficult to verify. Please annotate the segments and state the formula used for the reduction ratio.","section":"Figure 2"},{"comment":"The MiniMax-M3 top-k entry '128x16 (+1 local)' should be resolved to a single token count and block count so that the activated-traffic calculations in Section II-B are reproducible.","section":"Table II"},{"comment":"The phrase 'throughput per TDP' should be 'TDP-normalized throughput' or 'throughput per watt under TDP,' since TDP is a design limit rather than a measured power value; this would also make the normalization of the headline numbers clearer.","section":"Throughout"},{"comment":"The axis labels for the GPU/KARAT counts do not make the iso-power condition obvious; adding the aggregate TDP for each column would help the reader verify that all configurations are compared under the same 44.8 kW budget.","section":"Figure 14"}],"recommendation":"major_revision","confidential_remarks":"This is a solid systems proposal and the evaluation is more careful than many in the genre. The main risk is not the architecture idea but the unvalidated HBM3e-to-LPDDR5X latency calibration and the closed simulator. If the authors add the requested latency sensitivity and release artifacts, I would support acceptance; I am not recommending rejection because the capacity/bandwidth argument is physically grounded and the sensitivity to core count and interconnect is already reported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The thing to know: this is a serious systems proposal, and the central architectural move is right—put general-purpose compute next to large LPDDR KV capacity, sized to the retrieval indexer's operational intensity rather than low-OI dense attention. But every quantitative headline (2.09–6.13x, 1.36–3.21x) comes out of an in-house simulator whose KARAT latency model is borrowed from H200 MIG hardware without validating the HBM3e-to-LPDDR5X memory behavior. That is the soft spot that matters.\n\nWhat's genuinely new: the R1–R4 requirements analysis is a useful reframing of prior PIM/PNM work. I buy that prior designs assume low OI and fixed-function attention, and that retrieval-based sparse attention breaks both assumptions. The OI numbers (4–128) ground the design in a reproducible way. OFMS and CMR are smaller ideas, but they are clean, and the ablation is honest. The evaluation is also more careful than most in this space: real agentic traces, a P99 TBT SLO, iso-power comparisons, sensitivity to link bandwidth/latency and core count, and they let SPNM beat KARAT on Quest/STARC rather than hiding it. The citation pattern is thorough and fair.\n\nSoft spots, in order of weight. First, the calibration: the MIG slice matches KARAT's bandwidth, so the reader's worry about MIG partitioning is a misreading. The real issue is that a 1g.35gb H200 slice is HBM3e, not LPDDR5X; the paper adjusts energy only and leaves latency and controller behavior unchanged. For SCORE that may be fine—it is a streaming scan—but GATHER and ATTEND are random-access and latency-sensitive, and at the 4k–5k session operating points in Fig. 10 the KARAT iteration time sits close to the 100 ms SLO. If LPDDR5X access latency is worse than modeled, sustainable concurrency drops and the headline shrinks. No code, data, or prototype is released to check this. Second, the GPNM core count is tuned to the evaluated models; removing 8 cores costs 27% throughput, so the design point is sensitive to the benchmark set. That is not fatal—the sensitivity analysis exposes it—but it makes the absolute numbers conditional. Third, the simulator itself is in-house, and the quantitative results have no error bars. None of this undermines the qualitative argument, which rests on capacity and bandwidth reasoning rather than fitted constants.\n\nWho should read it: architects and systems folks working on long-context serving, KV-cache offload, or PNM/PIM designs. It deserves a serious referee, not a desk reject. I'd send it to review and ask for code/data plus a calibration section that either measures LPDDR5X behavior or argues from JEDEC specs why latency invariance is safe.","headline":"A serious PNM design for retrieval-based sparse attention whose architectural logic is sound, but the headline numbers rest on an unvalidated HBM-to-LPDDR latency transfer in the simulator.","tokens_in":27684,"tokens_out":3815,"would_cite":true,"duration_ms":35176,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that retrieval-based sparse attention moves the binding constraint to the KV cache and index keys, and that a general-purpose processing-near-memory node holding that state—with GPUs running projections and MoE…","keywords":["retrieval-based sparse attention","processing-near-memory","KV cache offloading","LLM serving","mixture-of-experts","micro-batch scheduling","operational intensity","agentic workloads"],"falsifier":"Run identical memory-bound attention and indexer kernels on a full H200 and on an H200 MIG1g.35gb instance, measure sustained DRAM bandwidth and per-kernel latency, and if the MIG slice delivers less bandwidth than the full H200, update the KARAT lookup tables and re-run the simulator to see whether the 2.09–6.13x margin survives.","tokens_in":26605,"feed_emoji":"⚡","tokens_out":12271,"duration_ms":101349,"temperature":0.7,"pith_summary":"Retrieval-based sparse attention lets frontier LLMs serve million-token contexts by reading only a small top-k subset of the KV cache each decode step, but it must still store the entire cache plus per-token index keys, and it must scan all of those keys at every step. The paper's claim is that this scan changes where attention should be computed: not on the GPU, whose HBM is needed for model weights, and not in fixed-function memory-side units, which cannot track the score functions and selection granularities that change across models and generations. Working from that claim, the paper proposes KARAT, a node of general-purpose processing-near-memory devices that hold the KV cache and index keys, execute the scan, selection, gather, and attention, and leave projections and MoE layers to the GPU. If the paper is right, this split raises the sustainable batch size enough to improve SLO-constrained decode throughput per watt by 2.09–6.13x over a GPU-only baseline on three state-of-the-art models, and it runs training-free sparse attention methods without hardware change at 1.36–3.21x.","feed_headline":"Memory-side engine lifts long-context LLM serving 6.13x","feed_subtitle":"Keeping the KV cache near general-purpose cores frees GPU memory for weights, lifting SLO-bound tokens per watt.","key_machinery":"KARAT (KV-cache-resident Accelerator for Retrieval-based ATtention): a device pairing 512 GB of LPDDR5X at 1.2 TB/s with an 18-core general-purpose compute engine rated 250 TFLOPS FP8 and 125 TFLOPS BF16, giving a ridge point around 208 FLOPs/byte, above the 4–128 FLOPs/byte operational-intensity range of the indexer scan. Operational intensity is the ratio of arithmetic to bytes moved, and the argument turns on the ridge point staying above the scan's intensity so the near-memory compute never becomes the bottleneck. Thirty-two such devices form a KARAT node with 16 TB capacity at the same 5.6 kW power and 38.4 TB/s bandwidth as an eight-GPU node. Two scheduling mechanisms complete the design:OFMS (opportunistic, fine-grained micro-batch scheduling) reorders GPU kernels across two micro-batches to hide expert all-to-all communication, and CMR (context-length-aware micro-batch rebalancing) repartitions requests by aggregate context length so the two micro-batches carry similar token counts despite the variance in long-context traces.","core_discovery":"The central discovery is that the indexer scan—not attention itself—is the operation that dictates the design of a KV-cache-offload system for retrieval-based sparse attention. The scan reads every index key in the context on every decode step, so its memory traffic and FLOPs scale together, and its operational intensity of 4–128 FLOPs/byte does not improve with batch size, unlike the weight reads of dense and MoE layers. The paper therefore derives four requirements—capacity per watt an order of magnitude above HBM, bandwidth per watt comparable to HBM, a compute ridge point above the scan's intensity, and programmability across the five stages—and shows that LPDDR5X plus 18 general-purpose cores, as in a KARAT device, is the design point satisfying all four. With the KV cache on KARAT nodes, GPU HBM is dedicated to weights, the global batch grows until MoE layers become compute-bound, and two scheduling mechanisms hide the pipeline bubbles that arise when GPU and KARAT micro-batches alternate. The paper reports 2.09–6.13x throughput per TDP under a 100 ms P99 time-between-tokens SLO for model-native sparse attention, with the largest gains on the longest contexts.","pith_inferences":["The quantitative range is a simulator prediction built from H200 MIG profiling, not a measurement of a working KARAT system; a bandwidth microbenchmark on the MIG slice versus a full H200 is the cheapest way to test whether those latencies are optimistic.","The same partition rule could generalize beyond KV caches to any memory-resident inference state whose reads dominate and whose access algorithms drift across model generations, such as embedding tables, long-term memory, or tool-call histories.","The two schedulers are designed to be decentralized and latency-adaptive, so the micro-batch rebalancing idea is portable to any heterogeneous serving pipeline with stage-duration variance, not only the GPU-plus-PNM split.","A useful stress test is to rerun the evaluation under a stricter SLO (for example, a 50 ms P99 time-between-tokens target) or with longer traces, since the speedup stems from batch-size gains that a tighter latency target could partially cap."],"forward_implications":["Freed from the KV cache, GPU HBM goes to model weights, so the decode batch can grow until the MoE layers become compute-bound; on the longest trace the paper reports sustained concurrency rising from roughly 250–340 sessions on the GPU-only system to about 4,000–5,000 with KARAT.","Adding memory capacity alone does not unlock the gain: the LPDDR-based SPNM baseline with the same capacity achieves only a 1.19x geometric-mean speedup versus KARAT's 2.23x, because its compute cannot sustain the indexer intensities of 64–128 FLOPs/byte.","Because the KARAT cores are programmable, training-free sparse attention methods from all three selection families—page-based, clustering-based, and hash-based—run without hardware change, improving throughput per watt by 1.36–3.21x over GPU-only.","The best system configuration shifts once KARAT holds the KV cache: the optimal expert-parallelism degree drops from 64 on the all-GPU system to 16–32, since the expert all-to-all then carries less data, and the grid search finds 1.7–7.8x throughput per TDP over the all-GPU column.","If successor memory technologies keep the same per-watt capacity and bandwidth ratios, the paper's design point carries over to future generations of LPDDR, GDDR, and HBM without re-derivation."],"supporting_citations":[{"why":"Defines DeepSeek-V3.2's model-native retrieval-based sparse attention with an exact global top-k indexer, one of the three primary evaluation targets.","marker":"[16]"},{"why":"Defines GLM-5.2's MLA plus DSA attention with cross-layer index reuse, one of the three evaluated models and a source of the indexer operational-intensity range.","marker":"[25]"},{"why":"Defines MiniMax-M3's GQA-based block sparse attention, the contrast case with coarse 128-token selection and low indexer intensity.","marker":"[43]"},{"why":"Supplies the page-based Quest training-free sparse attention method, one of the three selection families tested.","marker":"[83]"},{"why":"Supplies MagicPIG's LSH-sampling selection, the training-free method whose operations fixed-function near-memory units cannot run.","marker":"[12]"},{"why":"Supplies STARC's clustering-based selection and the PIM-based STARC baseline used in the comparison.","marker":"[20]"},{"why":"Supplies AttAcc, the HBM-PIM attention baseline whose low-intensity compute and limited capacity the paper argues no longer fit retrieval-based sparse attention.","marker":"[68]"},{"why":"Supplies Duplex, the PIM baseline that adds compute but no memory capacity and is provisioned for an operational intensity of 8.","marker":"[94]"},{"why":"Supplies the LPDDR-based CXL-PNM platform used as the SPNM baseline and as the source for comparing memory capacity and bandwidth per watt.","marker":"[70]"},{"why":"Supplies HiSparse, the host-DRAM KV offload baseline, representing the capacity-without-compute approach.","marker":"[87]"}],"fun_headline_variants":["KV cache offload to PNM lifts SLO-bound serving 6.13x","Sparse attention served from PNM: 2.09-6.13x throughput per TDP","General-purpose memory compute accelerates sparse attention 6x","Heterogeneous decode: weights on GPU, KV on PNM, 6.13x gain","Memory-resident indexer frees GPU HBM, boosting SLO throughput 6x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every quantitative result rests on profiling KARAT kernels on an H200 MIG1g.35gb instance, assuming that slice provides the same memory bandwidth as a full H200 and that scaling the SM count linearly is valid; if MIG actually partitions or reduces memory bandwidth, the KARAT latency table understates memory-bound time and the reported speedups shrink.","fun_headline_variants_meta":{"raw":{"variants":["KV cache offload to PNM lifts SLO-bound serving 6.13x","Sparse attention served from PNM: 2.09-6.13x throughput per TDP","General-purpose memory compute accelerates sparse attention 6x","Heterogeneous decode: weights on GPU, KV on PNM, 6.13x gain","Memory-resident indexer frees GPU HBM, boosting SLO throughput 6x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001062,"raw_usage":{"total_tokens":4551,"prompt_tokens":1141,"completion_tokens":3410,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":757,"completion_tokens_details":{"reasoning_tokens":3298}},"tokens_in":757,"tokens_out":3410,"duration_ms":22076,"temperature":1.0,"reasoning_tokens":3298,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:49:59.442661+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run identical memory-bound attention and indexer kernels on a full H200 and on an H200 MIG1g.35gb instance, measure sustained DRAM bandwidth and per-kernel latency, and if the MIG slice delivers less bandwidth than the full H200, update the KARAT lookup tables and re-run the simulator to see whether the 2.09–6.13x margin survives.","supporting_citations":[{"cited_title":"Quest: query-aware sparsity for efficient long-context llm inference,","cited_arxiv_id":null,"evidence_quote":"Supplies the page-based Quest training-free sparse attention method, one of the three selection families tested."},{"cited_title":"Hisparse: Turbocharging sparse attention with hierarchical memory,","cited_arxiv_id":null,"evidence_quote":"Supplies HiSparse, the host-DRAM KV offload baseline, representing the capacity-without-compute approach."}],"review_version":1}