Themis: Software-Defined Hardware Prefetching

Adrian Sampson; Akanksha Jain; Baris Kasikci; Chris Kennelly; David Li; Hanna Alam; Heiner Litz; Kan Zhu; Keisuke Kamahori; Milad Hashemi

Themis: Software-Defined Hardware Prefetching

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Themis claims that a single software-set bit in each page-table entry, telling the hardware prefetcher whether a data page is worth prefetching, removes about 40% of useless prefetch requests and improves performance for every prefetcher te

desk verdict Novel page-granular software-controlled prefetch throttling that deserves a serious look; the evaluation is broad, but λ tuning on test workloads and an internal SPEC mismatch need fixing. read the letter →

arxiv 2608.00259 v2 pith:34JPC5RN submitted 2026-07-31 cs.AR cs.OS

Keisuke Kamahori , Neil Adit , Kan Zhu , Yuqi Mai , Victor Lee , Heiner Litz , Chris Kennelly , Snehasish Kumar

show 8 more authors

Hanna Alam Milad Hashemi David Li Adrian Sampson Baris Kasikci Tipp Moseley Parthasarathy Ranganathan Akanksha Jain

This is my paper · ORCID

classification cs.ARcs.OS

keywords hardwareprefetchingprefetchthrottlingpage-tablehintsprofile-guidedoptimizationhardware-softwareco-designdatacenterworkloadsmemorybandwidthTLB

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that, for datacenter workloads, the useless prefetches emitted by hardware prefetchers can be filtered at page granularity without sacrificing coverage, and that software is the better place to do the filtering. It proposes Themis, which uses a profile-guided per-page prefetch-enable bit stored in page-table entries and replicated in TLB entries. A lightweight kernel module samples cache evictions to learn which pages receive useless prefetches, then sets or clears the bit; the hardware prefetcher reads the bit and stops training and predicting for disabled pages. In simulated datacenter traces, this removes about 40.6% of useless prefetch requests on average across seven state-of-the-art prefetchers, raises useful-prefetch coverage by 5.7%, and improves IPC for all seven (geometric mean +3.5%, with BOP +4.1%, SPP+PPF +3.1%, and Pythia +1.4%). A sympathetic reader would care because it offers a low-cost, post-silicon knob that makes existing hardware prefetchers more accurate in bandwidth-constrained environments without new instructions or binary changes.

What carries the argument

The load-bearing mechanism is a per-page prefetch-enable bit delivered through the page-table/TLB path and read directly by the hardware prefetcher as an enable signal. The software sets the bit according to the criterion λ × #useful < #useless, using eviction records sampled by a small FIFO in the cache PMU; because hints propagate lazily with normal TLB fills, updating the page table costs no shootdowns. The hardware change is minimal: a small buffer to log useless-prefetch evictions for profiling, and the prefetcher gating its training and prediction logic on the hint bit.

What would settle it

Profile the same set of pages in two disjoint execution windows (or on two different cores) and compare the per-page prefetch-usefulness decisions. If a substantial fraction of pages flip from useful to useless between windows—or if re-profiling every 1 billion instructions instead of every 40 billion changes the set of disabled pages and materially improves IPC—then the stability assumption behind Themis's one-shot profiling is wrong, and the headline gains would not carry to production.

Watch

Extended reading notes

Core claim

The central discovery is that, in datacenter workloads, prefetch usefulness is strongly concentrated at the page level: 50–75% of pages have prefetch accuracy below 20%, so a per-page on/off switch can remove most useless prefetches while keeping the useful ones. The paper shows that this switch can be implemented with a single hint bit carried from the OS page table through the TLB to the prefetcher, avoiding the hundreds of kilobytes of on-chip storage a hardware-only per-page tracker would need. The software computes the hint from the ratio of useful to useless prefetch evictions using the criterion λ × #useful < #useless, updates page-table entries lazily with no TLB shootdown, and the p

Load-bearing premise

The load-bearing premise is that a page's prefetch usefulness is stable enough that a profile taken once from a single 100-million-instruction region on one core stays valid for other threads, later execution windows, and all ten evaluation samples; if page usefulness drifts over time or varies across cores, the one-shot hints become stale and the reported speedups lose their basis.

Editorial extensions

If this is right

Any existing hardware prefetcher can be wrapped with Themis; the paper reports IPC gains for all seven evaluated prefetchers on datacenter traces (BOP +4.1%, SPP+PPF +3.1%, Pythia +1.4%) with average useless-prefetch reduction of 40.6%.
Because hints travel through existing page-table/TLB attributes, deployment needs no binary rewriting, no ISA changes, and no TLB shootdowns; profiling overhead is about 0.2% of execution time when re-profiling roughly every 15 seconds.
The benefit grows as DRAM bandwidth shrinks, and the optimal throttling strength (λ) shifts downward, so the same mechanism can tune prefetchers for bandwidth-constrained many-core systems.
Disabling training on marked pages is essential: without it, the average gain drops from 3.5% to 1.9%, and the effect is comparable to or better than enlarging the prefetcher's on-chip metadata tables.
Themis outperforms CLIP, a state-of-the-art throttler, on datacenter traces: CLIP slows the system by 4.9% on average while Themis speeds it up by 3.5%, showing that page-granular software control suits large-footprint workloads better than PC-granular hardware throttling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If page-granular usefulness remains stable across cores and longer production windows, the same PTE hint channel could carry richer directives—multi-bit aggressiveness levels, page-local prefetching, or per-page prefetch-distance hints—rather than a single on/off bit.
Because the mechanism works by improving utilization of small hardware metadata tables, it could plausibly be combined with temporal or machine-learning prefetchers whose accuracy depends heavily on table capacity; the paper evaluates spatial and ML prefetchers, so this is an untested extrapolation.
A testable extension is adaptive re-profiling: instead of a fixed λ and a ~40-billion-instruction cadence, tie re-profiling frequency to page churn or phase changes, since the overhead argument assumes page prefetchability is static after allocation.
One bit per page implies coarse control for huge pages; on 2 MB pages, mixed-content regions may need either finer-grained sub-page hints or acceptance of coarser throttling.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Plain take: Themis is a sensible, well-executed hardware-software co-design for prefetch throttling. The new bit is page-granular control: per-page prefetch-enable flags computed by software profiling and delivered through PTE/TLB attributes, so any hardware prefetcher sees a cheap enable signal. No ISA changes, no recompilation, and profiling can be done in the kernel with a small PMU FIFO. That's a real contribution, and it's orthogonal to existing prefetchers. The evaluation is the widest I've seen in this space: seven prefetchers, Google traces plus SPEC2017 and GAP, sensitivity across DRAM bandwidth, metadata size, and λ. The IPC gains are modest (3.5% mean on datacenter traces) but consistent, and the coverage increase despite throttling is a nice result. The paper deserves review.

Soft spots, in order: λ=4 is chosen from the same workloads used for the headline numbers, and Figure 17 shows the optimum shifts with bandwidth. The flat sensitivity (0–4) limits the damage, but held-out tuning would make the numbers cleaner. Second, the SPEC numbers disagree internally: the abstract/intro say 0.7% IPC improvement, while §4.2 and Figure 10 report 1.3%; the range also doesn't match. That's a fixable error, but it should be caught before acceptance. Third, the stability validation (Fig. 12) is weaker than it looks. The histogram is built from ten traces, each presumably classified with its own profile, while the evaluation uses one profile for all ten. So it shows page usefulness is stable across regions, but not that a single 100M-instruction profile is enough to reproduce that classification. The stress-test note calls this 'self-referential'; I'd say it's an indirect test, not circular, especially since the actual evaluation profiles on a region disjoint from the evaluation samples. Still, reporting variance across the ten samples would address the same concern more directly.

Absent error bars: for 1–4% IPC effects, ten samples per workload should be summarized with confidence intervals. The paper currently averages them silently.

Bottom line: the central mechanism holds up; the load-bearing stability claim is plausible but under-validated. Send it to a serious referee. If the authors fix the SPEC mismatch, release artifacts, and add a held-out stability experiment, it's a solid ISCA/HPCA-quality paper.

Referee Report

4 major / 5 minor

Summary. The paper proposes Themis, a hardware-software co-design mechanism that controls hardware data prefetchers at page granularity. A software profiler computes per-page prefetch-usefulness statistics, stores a prefetch-enable hint in page-table/TLB attributes, and hardware prefetchers use this hint to suppress training/prediction on pages where prefetching is unlikely to be useful. The mechanism is evaluated in ChampSim on Google datacenter traces, SPEC2017, and GAP, across seven hardware prefetchers. The main reported results are a ~40.6% reduction in useless prefetches, a mean 3.5% IPC improvement on Google traces (with all seven prefetchers benefiting), and a small average coverage increase of 5.7%, attributed to reduced cache pollution and better use of prefetcher training resources.

Significance. If correct, the paper makes a timely and useful contribution: it offers a low-cost, orthogonal mechanism for improving prefetcher effectiveness in bandwidth-constrained datacenter workloads, with minimal hardware changes and no binary/ISA modifications. The use of page-table attributes as a communication channel is plausible, and the evaluation is unusually broad: seven prefetchers, public Google traces, SPEC2017, and GAP, plus sensitivity studies on lambda, DRAM bandwidth, training behavior, and prefetcher metadata sizes. A particular strength is that the main evaluation uses a single 100M-instruction profile region that is at least 100M instructions away from the ten evaluation regions, so the headline results are not a same-region self-test. The paper also explicitly compares against CLIP and hardware-only throttling baselines. These strengths make the central idea worth pursuing. The main weaknesses are in the completeness of the profiling/implementation story and in the selection of lambda on the test workloads.

major comments (4)

[§3.3, Eq. (1)] The profiling flow as described cannot compute the criterion in Eq. (1). Step 1 buffers only useless prefetches: "Cache evictions for useless prefetches are buffered" and a useless prefetch is identified by the prefetch tag bit still being set at eviction. Eq. (1), however, requires per-page counts of both useful and useless prefetches (lambda x #useful < #useless). The paper does not explain how per-page useful-prefetch counts are obtained from the proposed PMU FIFO, nor how the kernel derives the ratio shown in the §3.2 figure. If the ChampSim evaluation instead uses ground-truth per-page useful/useless counts from a perfect profile, then the end-to-end mechanism—including FIFO sampling loss and PMU event fidelity—is not actually evaluated. The authors need to specify the full profiling data path or evaluate the profile mechanism's fidelity.
[§3.3 / §4.1] The delivery of page hints to L2 prefetchers is under-specified. Hints are stored in PTEs/TLBs as attributes of virtual pages, but the evaluated prefetchers operate at the L2 and the paper states "we assume prefetchers are working on physical addresses; hence, there is no additional overhead to look up the page attribute values." This assumption skips the central implementation question: how does a physical-address prefetcher know the hint for a predicted address, especially for prefetches that cross page boundaries or target pages other than the triggering demand page? A TLB attribute on a demand load does not automatically provide the attribute for a prefetch target physical page. The paper should explain the hardware lookup (or attribute propagation) for prefetch targets and account for any overhead in the model.
[§3.2 / §4.4, Fig. 15] The usefulness factor lambda=4 is chosen empirically on the same Google workloads used for the headline results (§3.2: "We empirically find lambda=4 to work best"; §4.4 Fig. 15 selects the peak on the evaluation set). This is a form of test-set tuning. The authors mitigate this by showing the sensitivity is modest (less than 1% average IPC difference for lambda in [2.5, 5]), but the reported mean speedup in Fig. 7 may still be slightly optimistic. The paper should either report results for a lambda chosen on a validation split or present the headline numbers across a range of lambda values to make the selection bias transparent.
[§4.1 / §4.3, Fig. 12] The stability argument for the page-usefulness assumption does not fully cover the multi-core usage model. The profile is collected from a single core and then shared across all cores/threads of a multi-threaded Google workload, but Fig. 12 measures consistency across trace segments of the same workload, not across different cores or thread interleavings affecting the same physical page. A page could be prefetch-friendly for one core's access stream and not for another, or could change behavior under different interleavings. Since the correctness of the whole mechanism depends on page-level usefulness being stable enough for a one-profile-fits-all-cores approach, the paper should at least report whether the single-core profile's decisions agree with per-core profiles, or discuss why cross-core variation is not a concern for the evaluated workloads.

minor comments (5)

[§4.2] The first paragraph of §4.2 says "Across seven prefetchers, geometric mean of speedup is 3.5% (0.4% for arizona, 3.1% for bravo.a, ...)." The parenthetical values are per-workload means, not per-prefetcher numbers. This should be reworded to avoid confusing the level of aggregation.
[§4.2 / Fig. 8] The paper alternates between "coverage" as a fraction of LLC misses and "the number of useful prefetches." Since Themis reduces total misses, a normalized fraction can increase even if the absolute number of useful prefetches is flat. Please state explicitly whether the +5.7% coverage increase is an absolute count or a normalized fraction, and define the metric once in §2.
[§3.4] The claim that PTE attribute updates cost "around 1 μs per bit" needs a brief description of how this number was measured or estimated, since it is used in the 0.2% overhead calculation.
[§4.3 / Fig. 12] The histogram would benefit from reporting the number of pages included, the bin width, and whether pages appearing in only one trace were excluded (the text says at least two traces). This helps the reader judge the statistical weight of the bimodal distribution.
[Throughout] There are several spacing/formatting issues in the PDF text (e.g., "Themisutilizes", "Themisis"). A final copyedit pass is needed.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: page-hint control is empirically validated out-of-sample; Equation 1 is a policy, not a tautology.

full rationale

The central derivation is not circular. Equation (1) (λ×#useful < #useless) is a policy for selecting pages on which to disable prefetching, not a mathematical identity that entails the reported reduction. The headline results—40.6% useless-prefetch reduction, 3.5% IPC improvement, and 5.7% coverage increase—are measured on evaluation regions that are disjoint from the profiling region, as stated in §4.1: 'We ensure a gap of at least 100 million instructions between the profiling region and the regions used for performance measurement.' Nothing in Eq. (1) forces the out-of-sample magnitudes or the coverage increases; they are simulator outcomes. The §4.3 stability check is also not self-referential: Figure 12 is constructed by independently recomputing the disable decision from ten traces ('we collect profiling data from ten distinct traces and then analyze how often page addresses were marked to disable prefetching'), so it tests the page-stability premise rather than assuming it. The main tuning concern is that λ=4 is selected from the same Google workload results (Fig. 15); however, the paper reports low sensitivity (<1% IPC difference across λ=2.5–5), so this is a benchmark-selection caveat rather than a structural circularity. Self-citations such as Limoncello [52] are motivational and independently supported by the in-paper data in Figure 2. The explicit limitation in §4.4—'Here, we show the results with one trace per workload due to long simulation time'—affects the bandwidth-sensitivity evidence but not the paper's central claim. No equation reduces to its input by construction, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 1 free parameters · 5 assumptions · 2 invented entities

Central claim rests on one fitted parameter (λ), several domain assumptions about workload stability/representativeness, and two proposed hardware mechanisms with no independent silicon evidence. The paper provides internal consistency data (Fig. 12, training on/off) that partially supports the stability assumption, but the evaluation does not simulate TLB propagation, page walks, or virtual-address prefetching, so those costs are assumed away.

free parameters (1)

λ (usefulness factor) = 4 (at 25.6 GB/s; optimal shifts to 0.5–3 for 1.6–12.8 GB/s)
Introduced in Eq. 1; 'We empirically find λ=4 to work best' (§3.2, §4.4). It sets the disable threshold and is tuned on the same workloads used for headline results.

assumptions (5)

domain assumption A page's prefetch usefulness is stable over long execution windows and across cores; a 100M-instruction single-core profile generalizes to other regions.
Used in profiling methodology §4.1 and overhead analysis §4.3 (Fig. 12). If false, directives would be stale and speedups would not transfer.
domain assumption Disabling prefetch and training on a page based on profile will not systematically miss high-value pages that the profiler did not observe.
The criterion in Eq. 1 only sees sampled evictions; unsampled pages get no directive and remain enabled. Soundness depends on sample representativeness.
domain assumption ChampSim simulations with physical-address prefetchers and no modeled page walk/TLB propagation cost faithfully represent the proposed hardware.
Stated §4.1: 'we assume prefetchers are working on physical addresses; hence, there is no additional overhead to look up the page attribute values.' Real L1 prefetchers may be virtual-address and hint transport has costs.
ad hoc to paper Prefetch-usefulness can be tracked by a cache-tag bit and sampled through a PMU FIFO without perturbing cache behavior.
Proposed microarchitectural mechanism §3.3; no silicon or detailed cycle-level model of the FIFO is provided.
domain assumption Arm PBHA or equivalent PTE attribute bits exist and are propagated with each memory operation.
Cited as existing (§3.1), but the paper's evaluation does not simulate TLB entry propagation or shootdown-free lazy updates.

invented entities (2)

Per-page prefetch-enable attribute bit in PTE/TLB
purpose: Carries software-computed directive to hardware prefetcher to enable/disable training and prediction on a page.
The bit is the core interface but only exists in the design/simulation; no real implementation or independent measurement is provided. It relies on Arm PBHA availability, but the specific prefetch-enable semantics are new.
Cache PMU useless-prefetch eviction FIFO
purpose: Samples physical addresses of prefetched-but-unused cache lines for software profiling.
Proposed hardware addition; no implementation or measured overhead is given. The LBR analogy is not a verification.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Themis: Software-Defined Hardware Prefetching." pith.science (2026). https://pith.science/paper/34JPC5RN

@misc{pith2026260800259,
  author       = {Pith},
  title        = {Pith review of: Themis: Software-Defined Hardware Prefetching},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/34JPC5RN}},
  note         = {Machine review of arXiv:2608.00259}
}

read the original abstract

Data cache misses represent a significant portion of stall cycles in datacenter workloads. Hardware prefetchers that reduce such stalls by fetching data ahead of time have become increasingly sophisticated. However, to achieve high coverage, they have to prefetch aggressively, generating many inaccurate accesses that waste memory bandwidth. This is problematic in datacenter environments where memory bandwidth is a limited resource due to high multi-tenancy. We observe that for datacenter workloads, inaccurate prefetches can be effectively filtered on a data page granularity, without sacrificing prefetch coverage. However, storing per-page metadata about prefetch usefulness in hardware is costly, so we propose a novel hardware-software interface for data prefetching: The software directs the hardware on where to prefetch, and the hardware identifies and issues prefetches in the regions of interest. We propose Themis, a profile-guided hardware prefetching solution that implements this new interface. Themis utilizes page-level hints stored in page-table entries to disable the prefetcher for certain data pages at runtime. Themis requires no binary or ISA changes and can be used to optimize processes without disrupting their execution. Themis is also orthogonal to existing works on prefetching and can be applied to optimize any hardware prefetcher. Our results show that Themis is able to achieve around 40% reduction in useless prefetch requests, resulting in speedup for all the evaluated prefetchers for datacenter workloads, including 4.1% for BOP, 3.1% for SPP+PPF, and 1.4% for Pythia.