{"id":"ac2e5d52-e20c-4e4a-8660-edbb45a64596","arxiv_id":"2411.18308","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CXL memory traffic, especially non-temporal stores, can degrade concurrent DRAM and SSD bandwidth by up to 93.2%, and CPU quota or MBA regulation can restore about 99% of DRAM bandwidth.","lead":"This paper measures how CXL memory traffic interferes with ordinary DRAM and SSD performance on real CXL hardware, finding up to 93% bandwidth loss under non-temporal stores. It is the first real-hardware study of CXL interference and proposes software and hardware mitigations to recover most of the lost performance.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"External validity is the load-bearing risk: the >90% MMEM suppression and TOR mechanism are demonstrated on only two platforms, and System A/B differ in CXL device type, SSD type, and OS, so the headline magnitude may be platform-specific.","rationale":"The reader and I identify the same load-bearing concern: the external validity of a two-platform characterization. The paper's internal measurements are plausible, and the authors provide a useful consistency check between cxlMemTest and CXL PMU counters. However, the strongest quantitative claim, 93.2% suppression, is a maximum over thread-count configurations on a single system (System B), and the second system differs in CXL device, storage type, memory size, and OS. Therefore the exact magnitude and the proposed TOR mechanism cannot yet be assumed to generalize across CXL implementations and server topologies. This does not contradict the paper's findings; it just means the current evidence supports a CONDITIONAL verdict rather than a fully general conclusion. The reader's conditional verdict already reflects this, so no verdict change is needed. Minor additional concerns (unreleased code, lack of error bars, a small discrepancy between abstract and Section V.A mitigation numbers) are secondary and do not alter the central assessment.","tokens_in":11382,"tokens_out":5911,"duration_ms":56896,"concrete_test":"Run the MMEM-vs-CXL ntst microbenchmark from Section III on a third platform with a different CXL device (e.g., an ASIC expander on an AMD Genoa or a second Intel Xeon system with the same FPGA), using the authors' harness and the same CPU-frequency lock and cache-clearing protocol, and report the full MMEM bandwidth vs CXL-thread-count curve with error bars. If the median/typical suppression is below, say, 50% while only the maximum exceeds 90%, or if the third platform does not reproduce the TOR occupancy increase, the headline magnitude and mechanism are platform-specific.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim (up to 93.2% MMEM suppression under CXL ntst, Insight #1) and the proposed TOR-based mechanism (Insight #2, H1) are inferred from measurements on exactly two systems. Table II shows System A and System B differ simultaneously in CXL device (Montage ASIC vs Intel Agilex FPGA), storage (SAS SSD vs NVMe PCIe 4.0), memory capacity, and OS/kernel, so no controlled comparison isolates the effect of the CXL implementation or uncore topology. The paper reports only the maximum interference across thread counts (Table III caption), with no per-configuration distribution or confidence intervals, so it is impossible to tell whether 93.2% is a robust worst-case or an outlier. The TOR congestion explanation is inferred from PMU counters (Table IV) with large multiplicative changes, but no direct manipulation of TOR or device-side traffic is performed to confirm causation. If on a third platform the suppression is much smaller, or if the TOR counters can be decoupled from ntst traffic, Insight #1/#2 and the H1 recommendation would need qualification. This is a threat to external validity rather than an internal inconsistency; the within-platform measurements are plausible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript presents an empirical study of performance interference between CXL memory traffic and host main memory (MMEM) and SSD storage traffic. The authors built a configurable microbenchmark and ran it on two real CXL systems: System A (Montage ASIC CXL memory, SAS SSD, CentOS 7) and System B (Intel Agilex FPGA CXL memory, NVMe SSD, Ubuntu). They report that CXL non-temporal stores (ntst) can suppress concurrent MMEM bandwidth by up to 93.2% and SSD bandwidth by up to 61.3% in microbenchmarks, with real applications seeing up to 90% degradation. They propose a reverse-reasoning analysis using uncore PMU counters and kernel profiling, attributing the main mechanism to table-of-request (TOR) congestion. They also evaluate CPU-quota cgroup limiting, frequency scaling, and Intel MBA as mitigations, claiming recovery to 92-99% of original MMEM bandwidth. The paper draws six insights and practical hardware/software takeaways, and claims to be the first characterization of CXL interference on real hardware.","tokens_in":11632,"tokens_out":7803,"duration_ms":72361,"significance":"If the central measurements are robust, this is a useful addition to the CXL characterization literature: it goes beyond isolated CXL performance and addresses co-tenant interference, which is relevant for real CXL adoption. The use of two real CXL devices (ASIC and FPGA) and the cross-validation of instruction-level bandwidth measurements with CXL PMU counters are concrete strengths; the counter agreement (e.g., 13169 MB/s versus 13342 MB/s for the same configuration) is a good reliability check. The paper also offers a falsifiable design suggestion (specialized TOR for slow-tier memory) and evaluates practical mitigations. However, the generality of the quantitative claims and the causal mechanism are not fully established by the evidence presented, so the contribution is currently strongest as a platform-specific characterization rather than a general law about CXL interference.","major_comments":[{"comment":"The headline figures, including the 93.2% MMEM suppression under CXL ntst on System B, are reported only as the maximum interference across thread-count configurations. No variance, confidence interval, or per-thread distribution is given for these maxima, so a reviewer cannot determine whether 93.2% is a robust worst case or a single outlier point. The problem is compounded by the footnote to Figure 4, which states that 'in certain scenarios, the results are unstable and exhibit significant fluctuations' and that such cases are represented as 0; this can make severe but noisy interference appear as no influence. To support Insight #1 and the abstract's central claim, the paper must report the full distribution (or at least median, interquartile range, and number of runs) for the configurations that produce the maximum, and it must report unstable scenarios separately instead of encoding them as zero.","section":"§III.B, Table III"},{"comment":"The external-validity claims are broader than the data. Only two platforms are evaluated, and they differ simultaneously in CXL device (Montage ASIC versus Intel Agilex FPGA), SSD interface (SAS versus NVMe), CPU model, memory capacity, and OS/kernel. Since no variable is varied in isolation, the observed up-to-93% suppression and the general statement that CXL traffic significantly impairs concurrent MMEM and SSD traffic cannot be separated from implementation or platform effects. The paper should either add a controlled comparison (for example, the same CXL device on a second host, or a second device on the same host) or explicitly scope all conclusions and takeaways to the two tested configurations. A limitations paragraph naming this threat to external validity is required before the results can be presented as a general characterization of CXL interference.","section":"§II, Table II"},{"comment":"The TOR-congestion mechanism is asserted more strongly than the evidence supports. The supporting data are ratios of PMU events under interference versus no interference (for example, unc_cha_tor_occupancy.ia_miss ratio 4090x and unc_cha_tor_inserts.ia_miss ratio 155x), from which the paper 'deduces' that CXL traffic occupies TOR for a lengthy period. These counters are consistent with the hypothesis, but they do not establish causation or exclusivity: no direct manipulation of TOR capacity, occupancy threshold, or device-side traffic is performed, and the very large occupancy ratio could reflect measurement semantics such as accumulated cycles rather than queue-depth saturation. To support H1, the paper should report TOR occupancy or utilization as a function of injected CXL ntst bandwidth and show that MMEM/SSD interference tracks TOR utilization, or explicitly relabel Insight #2 as a hypothesis consistent with the counters rather than a demonstrated root cause.","section":"§III.C, Table IV, Insight #2 / H1"},{"comment":"The real-application evaluation uses the same maximum-only reporting and additionally truncates unstable results to zero. Figure 4's y-axis is logarithmic but includes values labeled 0, which cannot be represented on a log scale; the footnote indicates that unstable scenarios are set to 0. It is therefore unclear whether the many zero or near-zero bars (for example, several Type B and Type D cases) reflect no interference or dropped measurements. Since the paper uses this figure to claim that 'in most cases there is solid contention and interference,' the zeros need to be disaggregated: how many runs were unstable, what was the variance, and which scenarios were dropped. Without this, the real-application conclusions and the 'up to 20.8x performance decline' claim in Section IV.B are not fully auditable.","section":"§IV, Figure 4"}],"minor_comments":[{"comment":"The abstract refers to 'CXL-Interplay' while the title and the rest of the paper use 'CXL-Interference'; please make the naming consistent.","section":"Abstract"},{"comment":"The colorbar ranges are ambiguous: 'Mild Suppression', -5% to -10%, and 'Moderate Suppression', -40% to -10%, overlap at -10%, and 'No Influence' within -5% leaves the boundary at -5% undefined. Please clarify the exact intervals.","section":"Table III, colorbar"},{"comment":"The caption says that 'the x-axis and y-axis represent the operation of the background and the device under test,' but the table is a static matrix without visible axes. Please add explicit row and column headers, and consider reporting absolute bandwidths as well as percentages.","section":"Table III caption"},{"comment":"The explanation for RocksDB's performance improvement under CXL ld traffic relies on a 24% reduction in instruction count, but the text does not state how instructions were counted or whether this reduction was statistically significant across runs; please add measurement details.","section":"§IV.A, RocksDB promotion"},{"comment":"The sentence 'restricted to 100% (1/16 of the time originally occupied)' is confusing; please clarify whether 100% refers to a full CPU quota of one core and how the factor 1/16 is derived from 16 background threads.","section":"§V.A"}],"recommendation":"major_revision","confidential_remarks":"I agree with the reader's conditional assessment: the within-platform measurements are internally consistent and the PMU cross-validation is a strength, but the paper generalizes from two platforms and from correlational counters. I would not reject the paper; the contribution is valuable, but it needs transparent statistics, a clear separation of measured facts from hypotheses, and an explicit limitations section before it should be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is the first real-hardware characterization of CXL interference I've seen, and it deserves a proper peer review. The headline finding—CXL non-temporal stores can suppress concurrent MMEM bandwidth by up to 93% on a real system—is striking, and the paper backs it up with cross-validated instruction-level and hardware counter measurements. The observation that CXL ld traffic can actually improve RocksDB throughput is unexpected and worth explaining.\n\nThe paper does several things well. Two different CXL devices (Montage ASIC and Intel Agilex FPGA) are measured. The microbenchmarks are systematic, and the reverse-reasoning using TOR occupancy counters and kernel function profiling is a reasonable attempt to move past black-box numbers. The mitigation experiments (CPU quota, frequency scaling, MBA) are practical, and the 99% recovery with MBA is a useful datapoint for system operators.\n\nThe soft spots are mostly about external validity. Only two systems, and they differ in CXL device, SSD type, OS, and kernel, so you can't attribute the interference magnitude to any one factor. The paper reports only the maximum interference across thread counts, with no variance or confidence intervals; footnote 3 admits some unstable scenarios were zeroed. The TOR-congestion mechanism (Insight #2) is inferred from PMU counters with large multiplicative changes, but there's no direct manipulation to confirm causation. These issues don't undermine the internal consistency of the measurements, but they should make readers cautious about generalizing the 93% number to other CXL implementations or server topologies.\n\nThe code isn't released yet (the URL is a placeholder), which limits reproducibility until it appears. Still, the data and method are sound enough that I'd send this to a serious referee. Anyone designing CXL-enabled systems or scheduling workloads on them should read this. I'd cite it if I were writing about CXL interference. Recommended for peer review.","headline":"First real-hardware CXL interference study with a striking 93% suppression finding; external validity is the main risk, but the paper deserves peer review.","tokens_in":12150,"tokens_out":1960,"would_cite":true,"duration_ms":18427,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On two real CXL hardware platforms, background CXL non-temporal store traffic suppresses concurrent main memory bandwidth by up to 93.2%.","keywords":["CXL","CXL memory interference","non-temporal store","Table of Request (TOR)","main memory bandwidth","SSD interference","memory bandwidth regulation","performance characterization"],"falsifier":"Run the same microbenchmark on a third-generation CXL system with a different uncore architecture, such as an AMD or ARM server or a different CXL controller, and check whether 16 CXL non-temporal-store threads still suppress concurrent main-memory bandwidth by more than 50% and whether the TOR occupancy counter still rises by thousands of times. If the suppression is small or the counter does not move, the TOR-congestion explanation and the generality claim fail.","tokens_in":11224,"feed_emoji":"📉","tokens_out":7026,"duration_ms":60800,"temperature":0.7,"pith_summary":"This paper aims to establish that CXL memory devices are not passive, isolated components: when they run alongside ordinary DRAM and SSD workloads, co-tenancy can substantially degrade the other workloads. The authors measure this on two real CXL systems, one with an ASIC-based and one with an FPGA-based CXL memory expander, and report that non-temporal store (ntst) traffic from the CXL device suppresses concurrent main-memory bandwidth by up to 93.2% and also hurts SSD throughput. They trace the cause to long occupancy of the Table of Request (TOR), the queue tracking pending cache and uncore transactions, and show that throttling CXL traffic via CPU quotas or memory-bandwidth caps recovers most of the lost bandwidth. If correct, CXL memory management and co-scheduling policies must account for interference, not just capacity and latency.","feed_headline":"CXL non-temporal stores can cut co-running memory bandwidth by 93%","feed_subtitle":"A real-hardware study shows CXL non-temporal stores clog the uncore; software throttling restores 99% of lost bandwidth.","key_machinery":"The load-bearing mechanism is the Table of Request (TOR), a queue between cores and the last-level cache that tracks pending Cache Home Agent (CHA) transactions, with one TOR per core and per PCIe interface. The paper argues that CXL traffic, particularly non-temporal store (ntst), occupies the TOR for long periods, delaying requests from co-running processes. This explains why CXL ntst is far more disruptive than CXL load/store, why MOVDIR64B (which leaves the write-combining buffer more quickly) causes less suppression, and why cache-partitioning remedies fail against ntst, which bypasses cache allocation. Supporting evidence comes from uncore performance counters for TOR occupancy and inserts, plus kernel hotspot shifts in memcpy and copyout functions.","core_discovery":"The central claim is that CXL memory traffic, especially non-temporal stores, interferes with co-running memory and storage workloads in real servers, and the interference is asymmetric: CXL hurts main memory (MMEM) and SSD much more than they hurt CXL. On System B, 16 CXL ntst threads reduce concurrent MMEM bandwidth by 93.2%, while CXL load/store traffic causes over 40% suppression; on both systems SSD random-write throughput drops by roughly 60%. The authors' reverse-reasoning analysis attributes the mechanism to TOR congestion: CXL miss requests occupy the Table of Request queue for long periods, inflating L2 miss latency by 15x and TOR occupancy counters by about 4090x. They then demonstrate that software regulation, via cgroup CPU quotas, frequency scaling, and memory-bandwidth allocation, can restore MMEM bandwidth to 92%-99% of baseline at a small cost in CXL bandwidth.","pith_inferences":["If the TOR-occupancy mechanism is the true bottleneck, the interference magnitude should scale with the number of in-flight CXL misses and with the number of CXL devices sharing the uncore; a system with multiple CXL expanders on the same socket should show suppression even worse than 93%.","The two-platform measurement leaves open which CXL controller, PCIe topology, or uncore design points are protected; a third platform with a different uncore architecture, such as AMD or ARM, could confirm or bound the generalization.","The bandwidth-restoration data support an adaptive online policy: continuously monitor CXL ntst bandwidth and TOR occupancy, then throttle CXL traffic with CPU quotas or MBA whenever co-running bandwidth-sensitive workloads are detected; the paper does not specify such a controller, but its measurements provide the needed input-output relationship.","Since MOVDIR64B interferes less than ntst, kernel and compiler writers could reduce interference by preferring write-combining direct stores for CXL memory updates, provided write-atomicity guarantees are acceptable."],"forward_implications":["If CXL ntst traffic is left unregulated, co-located main-memory and NVMe/SAS workloads can lose most of their bandwidth, a practical constraint for CXL memory pooling in shared clouds.","Software regulation is a workable first line of defense: cgroup CPU quotas recover about 92% of MMEM bandwidth, and memory-bandwidth allocation recovers 99%, 95%, and 98% across the three tested applications, at a cost of only a few GB/s of CXL bandwidth.","Because ntst bypasses the cache, cache-partitioning hardware mechanisms will not mitigate this interference; device-side or TOR-side regulators are the indicated hardware path.","CXL load traffic can sometimes improve SSD and filesystem workload performance, up to 21.2% for RocksDB on System A, an effect tied to reduced instruction counts once CXL memory bandwidth exceeds about 8 GB/s.","The asymmetry of interference means CXL memory behaves as a noisy neighbor to DRAM, while DRAM and SSD are mild neighbors to CXL, which matters for memory-tiering and co-scheduling decisions.","The paper's own mitigation results imply that identifying and throttling high-bandwidth CXL ntst processes is sufficient to restore most lost performance, without redesigning the memory system."],"supporting_citations":[{"why":"Supplies the real-hardware CXL memory microbenchmark and baseline characterization of genuine CXL-ready systems that this work extends to interference.","marker":"[28]"},{"why":"Defines the uncore performance counters for TOR occupancy and inserts that the reverse-reasoning analysis uses to attribute interference to Table of Request congestion.","marker":"[15]"},{"why":"Provides the memory-bandwidth allocation feature used in the mitigation experiments to restore MMEM bandwidth.","marker":"[10]"},{"why":"Documents MOVDIR64B semantics and write-combining buffer behavior, used to explain why direct stores interfere less than non-temporal stores.","marker":"[12]"},{"why":"Specifies the CXL protocol stack and datapaths that define the shared PCIe and uncore topology analyzed in the paper.","marker":"[8]"},{"why":"Provides the FIO/libaio storage benchmark used to generate SSD randread and randwrite interference workloads.","marker":"[3]"}],"fun_headline_variants":["CXL stores choke co-running memory bandwidth by 93%","Real-hardware CXL study: non-temporal stores cut memory bandwidth 93%","CXL interference is asymmetric: it hurts memory and SSD far more","CXL non-temporal stores cause 93% bandwidth drop, 15x L2 latency","Throttling CXL stores restores 99% of lost co-running bandwidth"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's quantitative claims assume that what happened on these two specific CXL test machines, one ASIC-based and one FPGA-based, also happens on other CXL controllers, PCIe topologies, and server uncore designs.","fun_headline_variants_meta":{"raw":{"variants":["CXL stores choke co-running memory bandwidth by 93%","Real-hardware CXL study: non-temporal stores cut memory bandwidth 93%","CXL interference is asymmetric: it hurts memory and SSD far more","CXL non-temporal stores cause 93% bandwidth drop, 15x L2 latency","Throttling CXL stores restores 99% of lost co-running bandwidth"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001101,"raw_usage":{"total_tokens":4539,"prompt_tokens":837,"completion_tokens":3702,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":453,"completion_tokens_details":{"reasoning_tokens":3601}},"tokens_in":453,"tokens_out":3702,"duration_ms":26685,"temperature":1.0,"reasoning_tokens":3601,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:18:30.485950+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same microbenchmark on a third-generation CXL system with a different uncore architecture, such as an AMD or ARM server or a different CXL controller, and check whether 16 CXL non-temporal-store threads still suppress concurrent main-memory bandwidth by more than 50% and whether the TOR occupancy counter still rises by thousands of times. If the suppression is small or the counter does not move, the TOR-congestion explanation and the generality claim fail.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the uncore performance counters for TOR occupancy and inserts that the reverse-reasoning analysis uses to attribute interference to Table of Request congestion."},{"cited_title":"Intel ® resource director technology (intel ® rdt) framework,","cited_arxiv_id":null,"evidence_quote":"Provides the memory-bandwidth allocation feature used in the mitigation experiments to restore MMEM bandwidth."},{"cited_title":"Corporation, Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 2B: Instruction Set Reference, M-U","cited_arxiv_id":null,"evidence_quote":"Documents MOVDIR64B semantics and write-combining buffer behavior, used to explain why direct stores interfere less than non-temporal stores."},{"cited_title":"Compute express link specification,","cited_arxiv_id":null,"evidence_quote":"Specifies the CXL protocol stack and datapaths that define the shared PCIe and uncore topology analyzed in the paper."},{"cited_title":"Flexible i/o tester,","cited_arxiv_id":null,"evidence_quote":"Provides the FIO/libaio storage benchmark used to generate SSD randread and randwrite interference workloads."}],"review_version":1}