Cosmos: A CXL-Based Full In-Memory System for Approximate Nearest Neighbor Search

Hyunjeong Shim; Jinin So; Jung Ho Ahn; Minyong Yoon; Sang-Soo Park; Seoyoung Ko; Si-Dong Roh; Sungmin Yun; Taeksang Song; Wanju Doh

REVIEW 4 major objections 5 minor 13 references

Cosmos: A CXL-Based Full In-Memory System for Approximate Nearest Neighbor Search

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A CXL memory device with built-in general-purpose cores can run the entire approximate nearest neighbor search locally, removing the host and PCIe link from the critical path.

desk verdict A coherent CXL-PNM architecture proposal with a genuine contribution in full ANNS offload and rank-level distance computation, but the headline speedups are upper bounds because the GPC and PU compute are unmodeled in the simulator. read the letter →

arxiv 2505.16096 v1 pith:7QOPFRMU submitted 2025-05-22 cs.AR

Seoyoung Ko , Hyunjeong Shim , Wanju Doh , Sungmin Yun , Jinin So , Yongsuk Kwon , Sang-Soo Park , Si-Dong Roh

show 3 more authors

Minyong Yoon Taeksang Song Jung Ho Ahn

This is my paper · ORCID

classification cs.AR

keywords CXLapproximatenearestneighborsearchprocessingnearmemoryrank-levelparallelismbillion-scalevectordatabaseretrieval-augmentedgenerationadjacency-awaredataplacementgraph-basedANNS

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Cosmos claims that the host CPU and PCIe link can be taken out of the critical path of billion-scale approximate nearest neighbor search by placing programmable general-purpose cores inside CXL memory devices and letting those cores execute the entire search loop: graph traversal, distance computation, and candidate list maintenance. If correct, vector databases can live entirely in CXL-attached memory while the host only dispatches queries and aggregates local top-k results, avoiding the capacity limits of DRAM and the latency limits of SSD-based systems. The paper also argues that computing partial distances in parallel across DRAM ranks and placing nearby clusters on different CXL devices each contribute large throughput gains, with reported improvements of up to 6.72x over a host-side CXL baseline and 2.35x over a prior distance-offload design.

What carries the argument

The load-bearing mechanism is the general-purpose core embedded in each CXL controller, which runs the ANNS loop locally: it loads the query, follows graph edges, requests neighbor vectors, updates the candidate list, and writes back only the local top-k results to the host. Around it sit two complementary mechanisms: rank-level processing units that split each vector's dimensions across DRAM ranks to compute partial distances in parallel on 64-byte sub-vectors, and an adjacency-aware placement algorithm that uses centroid distances and cluster sizes to spread nearby clusters across devices without runtime profiling.

What would settle it

Build a cycle-accurate model or a small FPGA/ASIC prototype that includes the GPC pipeline and rank-level PUs, run the same SIFT1B and DEEP1B query traces through it, and compare end-to-end QPS. If the measured throughput improvement over the host-side CXL baseline falls materially below 6.72x (SIFT1B) and 5.35x (DEEP1B), or below 2.35x over the distance-offload baseline, the central claim is falsified; the specific number to watch is whether GPC instruction throughput or PU processing latency becomes the new bottleneck.

Watch

Extended reading notes

Core claim

The central discovery is that a compute-capable CXL memory device can act as the executing engine, not just the storage tier, for graph-based ANNS. By integrating a programmable general-purpose core in the CXL controller, Cosmos moves graph traversal and candidate-list management out of the host; a rank-level processing unit computes partial L2 or inner-product distances concurrently on each DRAM rank; and an adjacency-aware placement algorithm assigns clusters to CXL devices so that neighboring clusters land on different devices. The paper reports that this combination raises query throughput by up to 6.72x over a host-side CXL baseline on SIFT1B traces and 5.35x on DEEP1B, and by 2.35x over a state-of-the-art CXL-based distance-offload approach, while substantially reducing graph-traversal and distance-calculation latency.

Load-bearing premise

The paper assumes that a general-purpose core embedded in a CXL memory controller can execute the graph traversal and candidate-list maintenance quickly enough, and that the rank-level processing units add negligible latency, so that memory bandwidth rather than the core becomes the bottleneck.

Editorial extensions

If this is right

Host-side bandwidth stops being the bottleneck: the only data crossing the PCIe link is the incoming query and the returned local top-k lists, so more CXL devices can be added without saturating the host interface.
Distance computation becomes a parallel, bandwidth-efficient operation: splitting each vector's dimensions across DRAM ranks lets partial distances be computed concurrently in each rank's processing unit.
Query load stays balanced across CXL devices even when many queries hit nearby clusters, because the adjacency-aware placement deliberately separates neighboring clusters.
The same CXL hardware can track changes in ANN algorithms or search parameters through its programmable general-purpose core, avoiding fixed-accelerator redesign.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the reported simulator injects precomputed memory traces and does not model the general-purpose core's instruction pipeline or the rank-level processing units' area and power cost, the 6.72x and 2.35x gains are best read as upper bounds until a cycle-accurate or silicon-level validation appears.
The full-offload pattern should transfer to other graph workloads where traversal, scoring, and top-k maintenance are the hot loop, such as sparse graph analytics or recommendation scoring, as long as the working set fits in CXL-attached memory.
The adjacency-aware placement could be paired with a query-router that deliberately sends each query to the device holding the closest cluster first, with spill-over to neighbors only if the candidate list is not full, which might cut redundant probing.
If the GPC and rank-level processing units prove cheap in area and power, CXL memory could become a general near-memory compute tier for database operators, not just ANNS.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

I read the Cosmos paper straight through, and I mostly agree with the reader's conditional verdict, though I'd put a little more weight on the architecture being genuinely new. The three pieces—GPCs inside CXL controllers for full graph traversal and candidate-list management, rank-level PUs for partial-distance accumulation, and the adjacency-aware placement heuristic—are a real combination that I haven't seen in the cited prior work. The placement algorithm is simple, parameter-light, and tested against round-robin on the same traces, so that part is evaluated honestly and the qualitative finding is credible. No fitted constants, no self-citation padding, no circular derivation. The ablations are clean enough to see which component does what.

The soft spot is exactly where the stress-test note lands. The simulator injects precomputed DiskANN memory traces into Ramulator and measures query latency. The GPC's instruction execution, the candidate-list heap operations, the graph traversal control flow, and the rank-PU's distance-accumulation latency are all treated as free or implicit. That means the 6.72x and 2.35x numbers are not measurements of the proposed system; they are measurements of a memory system that assumes the novel compute elements add no time. That's a load-bearing caveat because the whole point of Cosmos is to move compute into the memory side—if that compute is slow or power/area-constrained, the gains shrink. The comparison to CXL-ANNS also excludes hop-count-based caching, which the paper calls negligible; that might be true, but it's a claim, not a demonstration.

None of this makes the paper wrong. The qualitative direction—offload more, compute partial distances in parallel at the rank level, place clusters by adjacency—is sound and likely in the right ballpark. But the quantitative headline is an upper bound, not a measured result, and the paper doesn't report recall-parity data, which matters for an approximate search system. No artifact is released either, so independent reproduction is not possible.

Who gets value from this? Systems and architecture people working on CXL-PNM, near-memory processing, or billion-scale ANNS. It's a solid position paper with a concrete design and a plausible evaluation, but the claims would need hardware modeling or RTL validation before I'd trust the speedups. It deserves a serious referee—I would send it to review—and I'd push for the authors to model the GPC and PU explicitly, or at minimum bound their overhead, before publication. I wouldn't cite it in the next year as a quantitative result, but I'd cite it as a design proposal.

Referee Report

4 major / 5 minor

Summary. The paper proposes COSMOS, a CXL-attached memory system in which general-purpose cores inside CXL controllers execute the full graph-based ANNS pipeline (graph traversal, distance computation, candidate-list updates, and top-k selection), while rank-level processing units compute partial distances in parallel to exploit DRAM rank parallelism. It also proposes an adjacency-aware cluster placement algorithm that distributes neighboring clusters across CXL devices to balance query load. The evaluation uses Ramulator with DiskANN memory-access traces from SIFT1B and DEEP1B, reporting up to 6.72x higher throughput than a host-side CXL baseline and about 2.35x over the CXL-ANNS baseline.

Significance. If the architectural assumptions hold, the paper addresses a timely and important problem: reducing host-device data movement and PCIe round trips for billion-scale RAG retrieval. The work has several strengths: it uses independent public datasets, compares against a published baseline, provides an explicit placement algorithm, and evaluates the placement component against round-robin on identical traces, so that component is not circular. The central weakness is that the quantitative headline is an upper bound rather than a measured result: the simulator models only DRAM/memory-system behavior, while the GPC instruction stream and rank-level PU execution costs are absent. Because the paper's claimed contribution is precisely that these compute elements can be embedded in CXL devices without becoming bottlenecks, the missing compute model is load-bearing for the speedup claims.

major comments (4)

[Section V-A] The simulator injects only precomputed DiskANN memory traces into Ramulator. Section V-A states that node-visit traces were extracted and that the generated memory requests were injected to measure query latency, which means graph traversal, candidate-list maintenance, and local top-k selection exist in the evaluation only as memory-access patterns. Section IV-A, however, assigns all of these operations to the GPC. The GPC's instruction throughput, pipeline stalls, temporary-buffer accesses, and control-flow overhead are not modeled. The reported 6.72x, 5.35x, and 2.35x improvements therefore assume that the GPC is never the bottleneck, which is exactly the assumption that the paper needs to justify. A GPC timing model, or at least a sensitivity analysis over GPC cycles per node visit and per candidate update, is required before the headline throughput claims can be considered supported.
[Section IV-A / Fig. 3(c)] The rank-level PUs are modeled as free computation. The architecture assigns partial-distance accumulation to PUs inside each DRAM rank, yet the evaluation does not model PU latency, throughput, die area, power, or the overhead of transferring partial results from the PUs to the GPC. The 'w/o rank.' ablation in Fig. 4(a) attributes a large performance gain to the rank-level PUs, but because the only simulated component is the DRAM memory system, this gain is really the effect of assuming distance computation costs nothing beyond the memory requests it generates. Without a quantitative model of PU cost and its interaction with DRAM commands, the rank-level design cannot be distinguished from an idealized zero-overhead accelerator.
[Section V-B] The CXL-ANNS reproduction excludes hop-count-based graph caching on the stated grounds that it is 'beyond the scope of this work' and has 'negligible impact.' This is not supported by the presented data: Fig. 4(b) shows graph traversal as a substantial latency component for CXL-ANNS, and caching directly targets that component. Excluding a baseline optimization can only overstate COSMOS's relative speedup. The authors should either implement the caching component or provide trace-level measurements demonstrating that its latency impact is indeed negligible, and this justification should appear before the 2.35x comparison is reported.
[Section V-B / Fig. 4(a)] All throughput results are reported only as normalized values, with no absolute QPS, absolute latency, or confidence intervals for the 10,000-query traces. Given that the entire evaluation is simulation-based, absolute numbers are important for assessing whether the modeled configuration is realistic and for reproducing the claims. The authors should report raw QPS and mean/p99 latencies for at least SIFT1B and DEEP1B, along with variance across query batches.

minor comments (5)

[Algorithm 1] The penalty computation in lines 5-8 is unclear: 'proximity' is initialized to the number of devices and then decremented after each adjacent cluster found on a candidate device, so later adjacent clusters contribute smaller penalties. The intended priority (e.g., weighting the closest neighbor most heavily) should be stated explicitly, and the decrement order should be defined with respect to the sorted adjacency list.
[Abstract / Section V-B] The abstract reports '2.35x over a state-of-the-art CXL-based solution,' but Section V-B does not state this number explicitly; it appears only in Fig. 4(a). The text should identify the CXL-ANNS comparison value for both SIFT1B and DEEP1B.
[Section IV-B] The statement that mlock() pins HDM regions needs clarification: HDM is device-attached memory mapped into the host physical address space, so the authors should specify whether mlock() pins host pages, device memory, or both, and how this interacts with the static HDM mapping described earlier.
[Fig. 3(c)] The figure shows 64B sub-vector segments per rank, but it is not explained how vector dimensions are partitioned across ranks for different data types (uint8 for SIFT, fp32 for DEEP, int8 for MSSPACEV). A brief description of the dimension-to-rank mapping would help readers evaluate the generality of the rank-level PU design.
[Section V-C] The load-imbalance results in Fig. 5(a) are presented without numerical values in the text; the authors should report the LIR numbers and, ideally, the standard deviation across query sets, to support the claim that COSMOS 'consistently' achieves lower LIR than round-robin.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's architecture and placement algorithm are evaluated against independent baselines and external datasets, and no claim reduces to its own inputs by construction.

full rationale

The paper's performance claims are supported by a Ramulator-based simulation that injects memory requests derived from DiskANN node-visit traces, not by fitting parameters to the target metrics. The baseline comparisons (DRAM-only, CXL-ANNS, round-robin placement) are independent of the proposed mechanism. The adjacency-aware placement algorithm is a heuristic whose evaluation measures load imbalance on the same trace workload, but this is a sanity check of the algorithm's intended behavior, not a circular derivation of a prediction from the algorithm's definition. The paper does not define a key quantity in terms of the result it claims to predict, nor does it fit a parameter and then rename that fit as a prediction. The only self-citation of note is reference [11] (a Samsung CXL-PNM platform paper with overlapping authors), used as background evidence that CXL devices can include near-memory compute; this is not load-bearing for the paper's central claims, which stand on the simulation study. The main limitation is architectural modeling completeness: the GPC's instruction throughput and the rank-level PU's compute latency are not explicitly modeled, so the reported speedups are best interpreted as upper bounds. However, an unmodeled component is a validation gap, not a circularity, and the paper's derivation chain does not reduce to its own inputs.

Assumptions & free parameters 1 free parameters · 4 assumptions · 2 invented entities

Cosmos is an architecture paper, so there are no fitted physical constants. The ledger records the design assumptions the central QPS claims depend on: the in-memory GPC compute model, rank-PU partial-distance behavior, trace representativeness, and the hand-chosen adjacency penalty in Algorithm 1. The GPC and rank-level PU are proposed hardware components with no independent evidence outside the simulator.

free parameters (1)

Adjacency penalty step in Algorithm 1 = 1
In Algorithm 1, loss is incremented by a linear proximity score that starts at num_devices and decrements by 1 for each adjacent cluster already placed on a device. The unit step is a hand-chosen weighting, and no sensitivity analysis is reported; the load-balance results in Fig. 5 could depend on this weighting.

assumptions (4)

standard math L2 distance and inner product over a vector can be computed as the sum of partial distances over dimension partitions.
Used by rank-level PUs in Section IV-A and Fig. 3(c); mathematically valid.
domain assumption An in-memory general-purpose core in a CXL device can run graph traversal, candidate maintenance, and top-k update fast enough not to become the bottleneck.
The central offload claim assumes this; the simulator injects DiskANN memory requests but does not model GPC instruction timing (Sections IV-A and V-A).
domain assumption Ramulator plus the custom CXL-PNM modeling captures the memory access and bandwidth behavior of the proposed CXL devices, including CXL protocol overhead.
Evaluation uses Ramulator with a 1TB CXL configuration (Section V-A); CXL switch and protocol timing are not described as modeled in detail.
domain assumption DiskANN traces from 10,000 queries per dataset are representative of production RAG retrieval workloads.
Section V-A derives memory requests from a clustering-enhanced DiskANN over SIFT1B and DEEP1B; no validation is given that these traces match real RAG query distributions.

invented entities (2)

Rank-level Processing Unit (PU)
purpose: Computes partial L2 and inner-product distances on 64B sub-vector segments within each DRAM rank to exploit rank-level parallelism.
Introduced in Section IV-A and Fig. 3(c); no prototype or independent measurement exists, only simulator modeling.
General-Purpose Core (GPC) in the CXL memory controller
purpose: Executes the full graph traversal, candidate list management, and local top-k selection on the CXL device.
Introduced in Sections I and IV-A; no implementation or cycle-accurate evaluation is provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Cosmos: A CXL-Based Full In-Memory System for Approximate Nearest Neighbor Search." pith.science (2026). https://pith.science/paper/7QOPFRMU

@misc{pith2026250516096,
  author       = {Pith},
  title        = {Pith review of: Cosmos: A CXL-Based Full In-Memory System for Approximate Nearest Neighbor Search},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7QOPFRMU}},
  note         = {Machine review of arXiv:2505.16096}
}

read the original abstract

Retrieval-Augmented Generation (RAG) is crucial for improving the quality of large language models by injecting proper contexts extracted from external sources. RAG requires high-throughput, low-latency Approximate Nearest Neighbor Search (ANNS) over billion-scale vector databases. Conventional DRAM/SSD solutions face capacity/latency limits, whereas specialized hardware or RDMA clusters lack flexibility or incur network overhead. We present Cosmos, integrating general-purpose cores within CXL memory devices for full ANNS offload and introducing rank-level parallel distance computation to maximize memory bandwidth. We also propose an adjacency-aware data placement that balances search loads across CXL devices based on inter-cluster proximity. Evaluations on SIFT1B and DEEP1B traces show that Cosmos achieves up to 6.72x higher throughput than the baseline CXL system and 2.35x over a state-of-the-art CXL-based solution, demonstrating scalability for RAG pipelines.

Figures

Figures reproduced from arXiv: 2505.16096 by the authors.

**Figure 2.** (a) Memory latency hierarchy highlighting the potential of CXL [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** (a) Overview of the system architecture. (b) CXL controller architecture featuring a general-purpose core for executing graph-based ANN search via [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: (a) Relative query throughput (Query Per Second, QPS). (b) Break [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 11 canonical work pages

[1]

Towards Understanding Systems Trade-offs in Retrieval- Augmented Generation Model Inference,

M. Shenet al., “Towards Understanding Systems Trade-offs in Retrieval- Augmented Generation Model Inference,” 2024, arXiv:2412.11854

arXiv 2024
[2]

Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG,

A. Singhet al., “Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG,” 2025, arXiv:2501.09136

arXiv 2025
[3]

Scalable Billion-point Approximate Nearest Neighbor Search Using SmartSSDs,

B. Tianet al., “Scalable Billion-point Approximate Nearest Neighbor Search Using SmartSSDs,” inUSENIX ATC, 2024

work page 2024
[4]

Characterizing the Dilemma of Performance and Index Size in Billion-Scale Vector Search and Breaking It with Second-Tier Memory

R. Chenget al., “Characterizing the Dilemma of Performance and Index Size in Billion-Scale Vector Search and Breaking It with Second-Tier Memory,” 2024, arXiv:2405.03267

work page Pith review arXiv 2024
[5]

NDSEARCH: Accelerating Graph-Traversal-Based Approximate Nearest Neighbor Search through Near Data Processing,

Y . Wanget al., “NDSEARCH: Accelerating Graph-Traversal-Based Approximate Nearest Neighbor Search through Near Data Processing,” inISCA, 2024

work page 2024
[6]

Direct Access, High-Performance Memory Disaggre- gation with DirectCXL,

D. Gouket al., “Direct Access, High-Performance Memory Disaggre- gation with DirectCXL,” inUSENIX ATC, 2022

work page 2022
[7]

DiskANN: fast accurate billion-point nearest neighbor search on a single node,

S. J. Subramanyaet al., “DiskANN: fast accurate billion-point nearest neighbor search on a single node,” inNeurIPS, 2019

work page 2019
[8]

Chameleon: A Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Models,

W. Jianget al., “Chameleon: A Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Models,”Proc. VLDB Endowment, 2024

work page 2024

Show all 13 references

[9]

CXL-ANNS: Software-Hardware Collaborative Memory Disaggregation and Computation for Billion-Scale Approximate Nearest Neighbor Search,

J. Janget al., “CXL-ANNS: Software-Hardware Collaborative Memory Disaggregation and Computation for Billion-Scale Approximate Nearest Neighbor Search,” inUSENIX ATC, 2023

2023
[10]

PIM Is All You Need: A CXL-Enabled GPU-Free System for Large Language Model Inference,

Y . Guet al., “PIM Is All You Need: A CXL-Enabled GPU-Free System for Large Language Model Inference,” inASPLOS, 2025

2025
[11]

An LPDDR-based CXL-PNM Platform for TCO- efficient Inference of Transformer-based Large Language Models,

S.-S. Parket al., “An LPDDR-based CXL-PNM Platform for TCO- efficient Inference of Transformer-based Large Language Models,” in HPCA, 2024

2024
[12]

Big ANN Benchmarks,

“Big ANN Benchmarks,” 2024. [Online]. Available: https: //big-ann-benchmarks.com

2024
[13]

Ramulator: A Fast and Extensible DRAM Simulator,

Y . Kim, W. Yang, and O. Mutlu, “Ramulator: A Fast and Extensible DRAM Simulator,” p. 45–49, 2016

2016

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Towards Understanding Systems Trade-offs in Retrieval- Augmented Generation Model Inference,

M. Shenet al., “Towards Understanding Systems Trade-offs in Retrieval- Augmented Generation Model Inference,” 2024, arXiv:2412.11854

arXiv 2024

[2] [2]

Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG,

A. Singhet al., “Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG,” 2025, arXiv:2501.09136

arXiv 2025

[3] [3]

Scalable Billion-point Approximate Nearest Neighbor Search Using SmartSSDs,

B. Tianet al., “Scalable Billion-point Approximate Nearest Neighbor Search Using SmartSSDs,” inUSENIX ATC, 2024

work page 2024

[4] [4]

Characterizing the Dilemma of Performance and Index Size in Billion-Scale Vector Search and Breaking It with Second-Tier Memory

R. Chenget al., “Characterizing the Dilemma of Performance and Index Size in Billion-Scale Vector Search and Breaking It with Second-Tier Memory,” 2024, arXiv:2405.03267

work page Pith review arXiv 2024

[5] [5]

NDSEARCH: Accelerating Graph-Traversal-Based Approximate Nearest Neighbor Search through Near Data Processing,

Y . Wanget al., “NDSEARCH: Accelerating Graph-Traversal-Based Approximate Nearest Neighbor Search through Near Data Processing,” inISCA, 2024

work page 2024

[6] [6]

Direct Access, High-Performance Memory Disaggre- gation with DirectCXL,

D. Gouket al., “Direct Access, High-Performance Memory Disaggre- gation with DirectCXL,” inUSENIX ATC, 2022

work page 2022

[7] [7]

DiskANN: fast accurate billion-point nearest neighbor search on a single node,

S. J. Subramanyaet al., “DiskANN: fast accurate billion-point nearest neighbor search on a single node,” inNeurIPS, 2019

work page 2019

[8] [8]

Chameleon: A Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Models,

W. Jianget al., “Chameleon: A Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Models,”Proc. VLDB Endowment, 2024

work page 2024