{"id":"8c1e04d8-3eed-4062-8ec7-2bbb31518cc2","arxiv_id":"2512.25065","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"Vulcan finds instance-specific cache and memory-tiering heuristics via LLM-driven evolutionary search, but its evaluation overlaps training traces with test traces and the abstract overstates the body's results.","lead":"Vulcan uses LLMs plus evolutionary search to generate specialized cache-eviction and memory-tiering heuristics, splitting learned decision rules from trusted system code. Its reported gains (up to 69% in a few caching clusters; 2.5–7.9% on tiered memory) are weakened by train/test leakage and by an abstract that promises different results than the body.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cache-evaluation train/test leakage is the load-bearing flaw: cluster MRR scores include the search trace, so the reported 1.94–69% gains are not demonstrated; a held-out per-trace split is needed.","rationale":"The reader and I both identify the same load-bearing weakness: the cache eviction experiment has train/test leakage because the cluster-level numbers include the trace used for selection. The text at §4.1.3 is explicit about this ('uses one trace from within the cluster... evaluated on all traces within this cluster'), and no train/test separation is reported. This is a true methodological flaw that directly invalidates the 'up to 69%' cache-eviction claim as stated — the search trace is not disclosed, clustering is unvalidated, and no per-trace breakdown or error bars are provided. The claim that VULCAN 'outperforms all human-designed state-of-the-art algorithms' is also contradicted by the body: the heuristic ranks second or third in most clusters and only beats the best baseline in C1, C2, C3. I agree this is the primary correctness risk. My verdict is CONDITIONAL rather than REJECT because the underlying framework — policy/mechanism separation plus LLM-driven evolutionary search — is plausibly salvageable and the primary flaw is empirical, not fundamental. The fix is straightforward: disclose search traces, report held-out per-trace results, and re-derive cluster averages. The inconsistent abstract numbers (spot-VM appears only in the abstract; cache/tiering percentages differ between abstract and body) are additional correctness concerns, but they are secondary to the leakage. The paper itself does not claim formal verification, so the absence of formal proof is not a defect; the restricted language Anvil is mentioned as guaranteeing properties but is never demonstrated, another gap but one that is secondary to the empirical leakage.","tokens_in":30287,"tokens_out":1793,"duration_ms":16448,"concrete_test":"Re-run the §4.1.4 evaluation with a clean split: (1) identify and disclose the single search trace per cluster; (2) report per-trace MRR for every trace, separating the search trace from held-out traces; (3) recompute cluster-average MRR excluding the search trace (or using leave-one-out). If the VULCAN heuristic fails to beat GDSF on held-out traces in C2/C3, the central cache-eviction claim collapses; if it still outperforms, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim — that VULCAN's LLM search yields instance-specialized heuristics that match or beat strong hand-designed baselines — rests on the cache evaluation in §4.1.3. The paper states that 'the evaluator harness uses one trace from within the cluster to score candidate solutions,' then the final heuristic 'is evaluated on all traces within this cluster.' The cluster-average MRR in §4.1.4 therefore includes the very trace used to select the heuristic. With 10 clusters and no disclosed identity of the search trace, a top-1 or close-to-top-1 performance on that trace is mechanically favored by evolution; even 69% MRR improvement in C2 could be an artifact of overfitting to a single, possibly unrepresentative trace. The abstract's spot-VM claim is absent from the body and the abstract's '2x lower miss ratios / 10% higher performance' figures do not match the body's 1.94–69% (cache) and 2.5–7.9% (tiering) numbers, so the paper's headline claims are internally inconsistent. This is not just a missing error bar: it is a train/test contamination that directly undermines the only quantitative evidence for the main contribution. The body's own Figure 8 shows the synthesized heuristic trailing GDSF in most clusters, contradicting the abstract's 'outperform all' framing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VULCAN, a framework that uses LLM-driven evolutionary search to synthesize instance-specialized systems heuristics. The key idea is to separate policy from mechanism through VALUE and RANK interfaces, so that the LLM is only asked to generate a stateless scoring or ranking function while trusted scaffolding handles the rest. The authors instantiate the framework on cache eviction and memory tiering, and report that synthesized cache policies outperform strong baselines by up to 69% in per-cluster miss-rate reduction, and that tiering policies improve on vanilla ARMS by 2.5-7.9%. The paper also presents an LLM-assisted survey of 660 OSDI/NSDI papers to argue that the VALUE/RANK interfaces are broadly applicable.","tokens_in":30620,"tokens_out":8386,"duration_ms":85635,"significance":"If validated, the paper would make a useful contribution to automated systems heuristic design: the interface abstraction is clean, the generated heuristics are human-readable, and the idea of specializing policies to instances is timely. The paper also provides code pointers ([27], OpenEvolve [90]) and a large-scale LLM-assisted survey of the literature in Appendix A, which is a useful auxiliary contribution. However, the empirical evaluation as presented does not establish the central claims: the cache evaluation has a train/test contamination problem, the abstract advertises contributions that do not appear in the body, and the memory-tiering results rest on a narrow comparison without variance analysis.","major_comments":[{"comment":"The cache-evaluation protocol is contaminated. The text states that \"the evaluator harness uses one trace from within the cluster to score candidate solutions,\" and that the final heuristic \"is then evaluated on all traces within this cluster.\" Thus the cluster-averaged MRR reported in Figure 8 includes the very trace used to select the heuristic. The identity of the search trace is not disclosed, no held-out split is performed, and no per-trace results are reported. The advertised gains (1.94–69%, including the 69% result for C2) are therefore not a valid estimate of generalization; they may be inflated by overfitting to a single trace. A held-out evaluation, with the search trace excluded and per-trace results reported, is required before the main cache claims can be assessed.","section":"§4.1.3–§4.1.4"},{"comment":"The abstract advertises support for spot-VM scheduling (\"up to 4.9x higher savings\") and a restricted language called Anvil that \"guarantees important properties by construction.\" Neither spot-VM nor Anvil is defined or evaluated anywhere in the body. The abstract's headline numbers also do not match the body: the abstract says \"up to 2x lower miss ratios\" and \"up to 10% higher application performance,\" while the body reports 1.94–69% MRR for cache eviction and 2.5–7.9% for memory tiering. This makes it unclear which claims are actually being defended and overstates the evaluated scope of the work.","section":"Abstract vs. body"},{"comment":"The text itself reports that the synthesized heuristic is best in only three of ten clusters (C1, C2, C3), is second to GDSF in four clusters (C5, C6, C8, C9), and is third in the remaining clusters. This contradicts the full-text abstract's statement that the heuristics \"outperform all human-designed state-of-the-art algorithms.\" It also weakens the central claim of instance-optimality: in seven of ten instances the synthesized policy ranks second or third. Figure 8 is presented without per-cluster numeric values or error bars, making the magnitude of the advantage impossible to evaluate.","section":"§4.1.4, Figure 8"},{"comment":"The queue-topology results for C7 and C8 use the same instance-generation protocol as §4.1, so the same train/test contamination concern applies: candidate topologies are scored on a trace from the cluster and then evaluated on the cluster, with no explicit held-out split. In addition, the experiments switch to a size-agnostic setting, making the 1.0% and 3.2% improvements difficult to compare with the size-aware RANK-based evaluation in §4.1.4. Without a clean held-out protocol and variance estimates, the claim that the synthesized queue topologies outperform all seventeen baselines is not supported.","section":"§4.2.4"},{"comment":"The memory-tiering evaluation compares only against \"vanilla ARMS\" on four workloads and reports improvements of 2.5–7.9%. No comparison is made to Memtis or other state-of-the-art tiering policies, no repeated runs or confidence intervals are reported, and there is no ablation separating the effect of the synthesized policy from the effect of the richer 20-window access history added by VULCAN. These small gains need considerably more experimental support before the paper can claim superiority over existing tiering systems.","section":"§5.2"}],"minor_comments":[{"comment":"The runtime instance classifier shown in Figure 5 is described but never evaluated. If the paper claims to support runtime instance detection and policy selection, this component needs at least a proof-of-concept measurement.","section":"§3.2.1"},{"comment":"The choice of K=10 for KMeans and the selection of fifteen trace features are not validated. The paper should justify the cluster count and feature set, and ideally show sensitivity to these choices, especially since clusters define the notion of \"instance.\" The use of only the first 50,000 requests per trace (<1% of the trace) also deserves a representativeness check.","section":"§4.1.2"},{"comment":"Table 4 lists congestion control as a possible instantiation with an eBPF-based policy module and Mahimahi evaluation, but no congestion-control experiment appears in the paper. Either remove the row or add the corresponding evaluation.","section":"Table 4"},{"comment":"The prompt text contains a duplicated line describing the history metadata (\"auto info = history.get_metadata(obj_id)\" appears twice), and the prose in the same appendix has a small typo (\"some some illustrative examples\"). These should be cleaned up.","section":"Appendix B, Listing 3"}],"recommendation":"reject","confidential_remarks":"To the editor: The framework and interface abstractions are potentially interesting, and Appendix A's LLM-assisted survey is a useful methodological contribution. However, the central empirical claims are not supported as submitted: the cache evaluation suffers from train/test contamination, the abstract advertises contributions that are absent from the body (spot-VM, Anvil), and the reported results contradict each other in places. A full re-evaluation with proper held-out splits and aligned claims would be needed before this work could be considered further."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Three things to know. The VALUE/RANK interface is a genuinely useful reframing of policy/mechanism separation, and the OSDI/NSDI survey is a nice piece of evidence that the abstraction covers a lot of ground. But the empirical case, as reported, doesn't support the paper's own headline: the full-text abstract says VULCAN \"outperform[s] all human-designed state-of-the-art algorithms\" by up to 69% and 7.9%, while the body's Figure 8 shows the synthesized cache heuristic ranking second or third in 7 of 10 clusters, trailing GDSF. Worse, the submitted abstract mentions spot-VM scheduling, Anvil, and 4.9x/2x/10% claims that appear nowhere in the full text. That is a serious consistency problem.\n\nThe more technical issue is train/test leakage in the cache study. Section 4.1.3 says the search scores candidates on one trace from a cluster; the final heuristic is then evaluated on all traces in that cluster—including the search trace. The cluster-average MRR (e.g., 69% in C2) therefore includes the very trace used to select the heuristic. Without disclosing which trace was used and reporting per-trace results, the generalization claim is not demonstrated. The same setup appears to apply to the queue-topology results for C7 and C8, where they claim 1.0% and 3.2% gains.\n\nTo be fair, the body is more honest than the abstract: it says \"either nearly match or outperform,\" and the QT section claims wins only on two clusters. The idea is sensible, the scaffolding is real, and the tiered-memory study uses actual hardware (though the 2.5–7.9% gains have no error bars). The KMeans instance definition is unvalidated—15 hand-picked features and K=10 with no stability check—so the \"instance\" boundaries are a weak point too.\n\nBottom line: the paper is worth a serious referee and the interface contribution deserves citation, but the current evidence does not support the claims as written. The fix is straightforward: proper held-out splits, per-trace numbers, reconcile the abstracts, and release the harness. I'd send it out but expect major revision.","headline":"The VALUE/RANK interface is a genuinely useful reframing, but the empirical claims are not supported as stated due to train/test leakage and an abstract that overstates the body's own results; worth reading for the idea, not yet for the numbers.","tokens_in":31156,"tokens_out":4379,"would_cite":true,"duration_ms":42857,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that systems heuristics can be synthesized per deployment instance by having LLMs evolve small stateless scoring functions against trusted mechanisms, and that the resulting policies match or beat hand-designed state-of-th","keywords":["systems heuristics","LLM code synthesis","instance-optimal policies","cache eviction","memory tiering","evolutionary search","policy-mechanism separation","VALUE/RANK interfaces"],"falsifier":"Take one trace per cluster, run Vulcan's search to completion, then measure the discovered policy on held-out traces in the same cluster as well as on traces from other clusters. If the per-cluster advantage over GDSF or S3-FIFO vanishes or reverses, the instance-specialization claim is not supported; if it survives, the claim is robust. A second check: compare the best policy selected on the search trace against a policy found by random search with the same number of evaluations.","tokens_in":30100,"feed_emoji":"⚙️","tokens_out":6931,"duration_ms":74773,"temperature":0.7,"pith_summary":"The paper argues that systems heuristics can be synthesized rather than hand-authored, and that the reason LLM-based synthesis fails in this domain is structural: current heuristics entangle decision logic with low-level mechanisms. Vulcan's answer is to separate the two with narrow interfaces—VALUE, a function from state to a scalar decision, and RANK, a per-object scoring function—and to let evolutionary search over LLM-generated scoring functions find instance-specialized policies. The paper reports that on ten cache-eviction workload clusters the synthesized policies match or beat state-of-the-art baselines, by up to 69% in one cluster, and that queue-topology variants beat seventeen baselines by 1.0–3.2%; in memory tiering, synthesized promotion policies beat the ARMS system by 2.5–7.9% across four application instances. The broader aspiration is to change the cost model of heuristic design so that specialization becomes the default, with humans defining interfaces and evaluation harnesses instead of tuning constants.","feed_headline":"LLM-designed cache policies beat hand-tuned ones by up to 69%","feed_subtitle":"Vulcan separates policy from mechanism so a small model can evolve instance-specialized heuristics safely.","key_machinery":"The VALUE and RANK interfaces. VALUE reduces a policy to a function value(X) computing a scalar from system features (e.g., cwnd); RANK reduces it to a per-object score(X, o_i) whose top-K selection is performed by a reusable mechanism (full sort, sample sort, or priority queue). The evolution loop pairs an LLM generator with an evaluator harness; the template constrains the function signature and features, while the harness returns a single optimization metric. A second, 'queue topology' form asks the LLM to co-design initial-placement and transition functions among a small set of FIFO/LRU queues—a value-style coding of routing decisions—which yields constant-time eviction policies. The cen","core_discovery":"The central claim is that constraining an LLM to write a single stateless scoring function—rather than an entire mechanism-entangled heuristic—turns heuristic synthesis into a tractable search problem, and that the resulting policies can beat hand-designed ones for a specific deployment instance. The interfaces are the load-bearing invention: every task is recast as either value(X), computing a scalar, or score(X, o_i), ranking objects, and the mechanism (priority queue, full sort, queue topology) is provided by trusted scaffolding. The paper claims this makes validation nearly trivial: any real-valued function is a well-formed policy, so 'it may be a poor policy, but it cannot be an invalid","pith_inferences":["Beyond the paper: the learned scoring functions double as a readable explanation of what matters for an instance (e.g., NVM bandwidth saturation for GUPS, burst-phase detection for Silo), so the same pipeline could be used as an automated workload-characterization tool.","Beyond the paper: because the search uses one trace per cluster during selection and reports cluster averages that include that trace, a held-out evaluation would be needed to confirm that the discovered heuristics generalize rather than overfit; this is an open question the paper leaves implicit.","Beyond the paper: the abstract advertises spot-VM scheduling savings, but the body's evaluation covers cache eviction and memory tiering only; transferring the interface to admission control would require building the mechanism and harness for that domain.","Beyond the paper: the low search cost suggests continuous re-specialization is feasible—an instance classifier could trigger a fresh search whenever the workload drifts, making the heuristic itself a managed resource."],"forward_implications":["If the claim holds, heuristic redesign stops being a human bottleneck: the same template can be pointed at a new workload cluster and, in hours, produce a specialized policy whose cost is measured in API calls rather than engineer months.","Interface safety means synthesized policies can be put on the hot path without a separate verification layer; a function that returns a number cannot break the system even if it is stupid.","The interface taxonomy predicts which tasks can be automated: the paper's survey of 660 recent systems papers finds 71 VALUE tasks and 158 RANK tasks among 234 identified resource-management tasks, so the method should transfer to scheduling, prefetching, congestion control, and admission control.","Instance-specialization can become continuous: an automated instance classifier can notice a workload shift, trigger a new search, and deploy a new heuristic, making one-size-fits-all a design choice rather than a necessity."],"fun_headline_variants":["LLM heuristics: up to 4.9x VM savings, 2x lower cache misses","Vulcan lets LLMs write safe, specialized systems heuristics","LLM-found heuristics beat hand-tuned ones for cache, VM, memory","Safe LLM-generated heuristics, specialized per deployment"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that one trace drawn from a cluster is representative enough of that cluster to serve as the search objective, while the final reported cluster average includes that same trace; if that trace is not representative, the reported gains may be partly an artifact of selection rather than specialization.","fun_headline_variants_meta":{"raw":{"variants":["LLM heuristics: up to 4.9x VM savings, 2x lower cache misses","Vulcan lets LLMs write safe, specialized systems heuristics","LLM-found heuristics beat hand-tuned ones for cache, VM, memory","Safe LLM-generated heuristics, specialized per deployment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001521,"raw_usage":{"total_tokens":5925,"prompt_tokens":733,"completion_tokens":5192,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":477,"completion_tokens_details":{"reasoning_tokens":5109}},"tokens_in":477,"tokens_out":5192,"duration_ms":40413,"temperature":1.0,"reasoning_tokens":5109,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T13:09:40.971629+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one trace per cluster, run Vulcan's search to completion, then measure the discovered policy on held-out traces in the same cluster as well as on traces from other clusters. If the per-cluster advantage over GDSF or S3-FIFO vanishes or reverses, the instance-specialization claim is not supported; if it survives, the claim is robust. A second check: compare the best policy selected on the search trace against a policy found by random search with the same number of evaluations.","supporting_citations":[],"review_version":1}