{"id":"ba968ccd-119d-4b8e-b314-ac6522306f44","arxiv_id":"2608.08344","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A two-statistic pre-flight predicts when permutation search beats random sampling, and exhaustive ordering tests show instruction order alone shifts LLM accuracy by 90.6 points.","lead":"PRISM is a pre-flight protocol that measures a fitness landscape with two cheap statistics before choosing how to search a permutation space. In exhaustive tests, reordering six fixed reasoning instructions changed a language model's accuracy on math problems from 6.3% to 96.9%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PRISM's regime classifier is supported, but the operational pre-flight link from 'guiding' to 'search beats sampling' fails on the exact six-instruction landscape and the decision bands are not out-of-sample validated.","rationale":"I read the paper as making two distinct contributions: an exact empirical map of instruction ordering (C3) and a predictive pre-flight protocol (C1). The evidence for C3 is strong and honestly reported: all 720 orderings evaluated, a 6.3-96.9 percentage-point range, coherent position effects, capped budgets, and the explicit withdrawal of the earlier speedup claim all support the ordering-effect finding. The stress-test centers on whether the pre-flight is a prediction rather than a description. The registered NAS forecasts and the parity falsification are real prospective or falsifying evidence, so the paper is not circular in every case. However, the decision rule is left qualitative ('empirical operating bands'), and the one exact 'guiding' landscape fails the operational test: the forecasted structured advantage is absent in the 40-run audit, with overlapping intervals. The paper's own language ('Guidance does not guarantee a large optimizer advantage'; ledger verdict 'scoped') concedes this, which means the abstract overstates C1. The condition for acceptance is clear: freeze quantitative thresholds on a training set, validate on held-out exact landscapes, and revise C1 to claim regime classification rather than guaranteed search advantage. That is a revisable conditional, not a rejection. My concern differs from the reader's emphasis in that the most load-bearing point is not just threshold calibration but the missing operational link between a correct 'guiding' classification and the promised search advantage, which the six-instruction landscape directly tests and fails.","tokens_in":11908,"tokens_out":6166,"duration_ms":62189,"concrete_test":"Freeze the protocol into a quantitative decision rule before any new landscape is measured: specify numeric cutoffs for rho1 and FDC (or a fitted logistic rule) using only the six synthetic/pipeline landscapes and the parity landscape as a training set. Then register forecasts on at least four held-out exact permutation landscapes (e.g., new six-instruction sets, new pipeline systems) and compare predicted 'search beats sampling' against the 40-run outcome. Also re-run the six-instruction case as a calibration check: if the frozen rule does not classify it as 'guiding' with a projected search advantage, or if held-out forecast accuracy is no better than chance, the operating bands are in-sample and C1 should be tempered.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central practical claim is that the pre-flight can forecast when structured search will beat uniform sampling. That forecasting link is the least secure part of the argument. First, the decision rule is only described through Figure 4's shaded regions and Table 1's 'empirical operating bands, not universal constants'; no quantitative thresholds, calibration holdout, or cross-validation are reported, so it is unclear whether the reported selector accuracy (0.925 at B=100 in Section B.2) is a fit or a prediction. Second, the one exact landscape where the pre-flight correctly classifies the regime as 'guiding' is the six-instruction surface (Section 7): rho1=0.547, exact FDC=-0.346, insertion selected. Yet the 40-run audit gives evolution 9.9 [7,13] vs uniform sampling 10.8 [8,14] evaluations to optimum, with overlapping intervals. Table 5 records the verdict as 'scoped', not correct. Thus regime classification works, but the operational inference that a guiding regime means structured search will outperform sampling fails on the paper's cleanest example. The abstract's promise to 'identify when structured search is likely to outperform random sampling' is not supported; what is supported is identifying operator alignment and near-random/deceptive regimes. The NAS forecasts (13/18 correct, with all LOW misses conservative) are a genuine prospective step but do not close this gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"PRISM proposes a pre-flight protocol for permutation optimization: before spending the main search budget, it measures one-move autocorrelation rho1 and fitness-distance correlation FDC on a small probe set, classifies the landscape into guiding, near-random, or deceptive regimes, and selects an executor (exact enumeration, uniform sampling, evolutionary search, surrogate, or transfer-guided list). The paper claims four results: (C1) the two-statistic pre-flight separates operator-aligned, near-random, and deceptive regimes; (C2) exact enumeration exposes a case where structured search is worse than uniform sampling (the parity landscape); (C3) exhaustive evaluation of all 720 orderings of six fixed reasoning instructions shows a 6.3%–96.9% accuracy range with transferable position effects; and (C4) cross-domain studies cover scientific pipelines, tabular NAS forecasts, and RL pilots. The manuscript emphasizes retained forecasts, misses, and reproducibility, including a forecast ledger, bootstrap intervals, and a reported external inference cost below US$20.","tokens_in":12082,"tokens_out":3991,"duration_ms":35749,"significance":"If the forecasting claim held, PRISM would turn optimizer selection in permutation spaces from universal superiority arguments into measured, budgeted decisions. The paper's strengths are substantial: exact enumeration of all 720 instruction orderings on a fixed 32-question GSM8K subset; a 40-run parity audit with Wilson intervals showing uniform sampling outperforming elitist search; bootstrap intervals for transfer and baseline comparisons; 18 registered NAS forecasts with retained misses; and a detailed reproducibility and resource-accounting record. These support the negative result that structured search is not universally better than sampling and the empirical claim that instruction order is a real optimization coordinate. The main risk is the forecasting link: the paper does not yet demonstrate out-of-sample validity of its regime thresholds, and on the paper's cleanest guiding landscape the structured-search-versus-sampling comparison shows overlapping intervals, so the operational promise that PRISM 'identifies when structured search is likely to outperform random sampling' remains unsupported.","major_comments":[{"comment":"The paper's headline operational claim—that PRISM identifies when structured search is likely to outperform random sampling—is contradicted by the paper's own cleanest guiding example. The six-instruction landscape has rho1 = 0.547 and exact FDC = -0.346, is classified as guiding, and insertion is selected, yet the 40-run audit gives evolutionary search 9.9 [7,13] versus uniform sampling 10.8 [8,14] distinct evaluations to optimum, with overlapping bootstrap intervals. Table 5 records the verdict as 'scoped', not correct. Since this is the one exact landscape with a guiding classification and a higher-powered audit, the manuscript needs either to revise the abstract's promise (e.g., to 'identify operator-aligned and near-random/deceptive regimes') or to add a pre-flight criterion—such as optimum density or a margin threshold—that would have predicted the overlap.","section":"§7, Table 5"},{"comment":"The reported selector accuracy of 0.925 at B=100 is not established as a prediction. Table 1 describes the thresholds as 'empirical operating bands, not universal constants,' and Figure 4's shaded regions appear drawn around the plotted points used to report accuracy. No calibration holdout, cross-validation, or independent threshold-setting procedure is described, so the reader cannot distinguish a fit to these landscapes from a forecast for new landscapes. Please report how the bands were derived (for example, fixed a priori, optimized on a training subset, or fitted post hoc) and evaluate the selector on landscapes excluded from threshold construction.","section":"§5, §B.2, Table 1, Figure 4"},{"comment":"The proxy FDC, defined in Eq. (4) as using the best frozen probe sample as the target when exact optima are unknown, is load-bearing for all non-enumerated forecasts (eight-module, MATH-500, NAS slices, RL pilots). The Limitations section concedes that this proxy 'can miss the relevant basin.' The paper does not quantify how often or by how much the proxy diverges from exact FDC on landscapes where both are computable, such as the six-instruction or parity surfaces. Please report a sensitivity analysis comparing proxy and exact FDC on enumerated landscapes, or state explicitly which forecasts would change under alternative proxy choices.","section":"§3, Limitations"},{"comment":"The parity falsification is a strong empirical result, but the statement that 'the pre-flight predicts this boundary before the search comparison is run' (Section 6) is difficult to audit because the decision bands appear to be defined after the fact. Please provide the pre-registered or timestamped forecast for the parity landscape, or make explicit that this case was used to set, rather than test, the thresholds. Without this information, the reader cannot tell whether the parity result is a confirmation of the protocol or an input to it.","section":"§6, Table 5"}],"minor_comments":[{"comment":"Protocol 1, step 1 says to 'measure variance, tie density, and evaluator resolution,' but the paper does not define how evaluator resolution is quantified; please provide a formula or a reference to a defined measure.","section":"§2, Protocol 1"},{"comment":"Theorem A.2 says 'Under the conditions of Theorem A.1,' but the preceding statement is Proposition A.1; the cross-reference should be corrected.","section":"Appendix A"},{"comment":"The rows for 'OPRO ordering search' and 'Wording level change' would benefit from stating the exact evaluation budgets and number of runs; Section 8 mentions five shared runs for the OPRO comparison, but this is not carried in the table.","section":"Table 4"},{"comment":"The caption states that the shaded regions 'summarize the conservative rule in Table 1,' but the numerical thresholds are not given in either place; please add the boundary values in the caption or table so the regions are reproducible.","section":"Figure 4"},{"comment":"The abstract says 'cross-model experiments show that useful ordering structure can transfer,' while Section 8 reports that 'fine rank transfer varies'; please align the abstract's wording with the more qualified finding.","section":"Abstract vs. §8"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the manuscript's strongest contributions are the exact enumeration, the honest failure accounting, and the retained forecast ledger. The central forecasting claim needs either reframing or out-of-sample validation; the six-instruction 'guiding' case with overlapping search intervals is the clearest illustration. I see no citation or scope problems beyond the usual need to keep the 24-domain roadmap clearly separated from validated results, which the paper mostly does."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the PRISM paper. The thing worth knowing: the exact enumeration of all 720 orderings of six fixed instructions is a real contribution. Accuracy swings from 6.3% to 96.9% purely from order, with interpretable position effects, and it survives automated rewriting. The author handles the negative parity result honestly: evolutionary search loses to uniform sampling on a 5,040-order landscape, and the protocol predicts it. The forecast ledger, with misses retained, is exactly the right habit.\n\nWhat's new: the earlier literature knew prompt order mattered and that rho1/FDC are useful landscape statistics. The new content is the operational pre-flight — measure two cheap diagnostics, decide whether to enumerate, sample, or evolve. The exact map and the cross-family transfer are solid. Reproducibility is good: under $20 of API spend, cached responses, bootstrapped intervals.\n\nThe soft spot is the paper's central inference. On the flagship six-instruction surface, the pre-flight says \"guiding,\" insertion selected, but the 40-run audit gives evolution 9.9 [7,13] evaluations vs sampling 10.8 [8,14] — overlapping intervals. Table 5 records the verdict as \"scoped,\" not correct. So regime classification works, but the abstract's promise to \"identify when structured search is likely to outperform random sampling\" is not supported. What is supported is identifying operator alignment and near-random/deceptive regimes.\n\nSecond issue: the decision bands in Table 1 and Figure 4 are empirical operating bands, with no calibration holdout reported. The 0.925 selector accuracy at B=100 in Section B.2 could be fit rather than prediction, since the shaded regions track the same points. This is addressable: pre-register thresholds before forecasting. Minor: the ordering standard deviation in the OPRO section appears without a full definition, and the Limitations section properly admits the proxy FDC can miss the relevant basin.\n\nCitation pattern is fair — classical statistics credited to Weinberger and Jones/Forrest, prompt sensitivity to Lu et al.\n\nOverall: worth refereeing. The empirical centerpiece is robust, the protocol is useful, and the author is honest about failures. A serious editor should send it out with requests to define thresholds prospectively and temper the C1 claim to align with the evidence. I'd bring it to a reading group, and I'd cite the ordering map.","headline":"The exhaustive 720-order instruction map is a genuine empirical find; the pre-flight protocol is promising but its core claim that it predicts when search beats sampling fails on the paper's flagship case.","tokens_in":12689,"tokens_out":2300,"would_cite":true,"duration_ms":19836,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two cheap landscape statistics forecast when permutation search beats sampling.","keywords":["permutation optimization","landscape diagnostics","fitness autocorrelation","fitness-distance correlation","instruction ordering","language model prompting","search strategy selection","pre-flight protocol"],"falsifier":"Take a fresh set of permutation optimization problems with known ground truth, such as random typed synthetic landscapes or new language-model instruction sets; run the pre-flight with the Table 1 bands unchanged and compare the predicted regime against the observed advantage of elitist search over uniform sampling at equal distinct-evaluation budgets. If forecast accuracy is at chance on these held-out problems, the central claim collapses.","tokens_in":11597,"feed_emoji":"🧭","tokens_out":7615,"duration_ms":60683,"temperature":0.7,"pith_summary":"This paper argues that the difficulty of a permutation optimization problem is not set by the size of the search space, but by measurable landscape structure. It introduces a pre-flight protocol, PRISM, that spends a small budget estimating one-move autocorrelation and fitness–distance correlation, then forecasts whether structured search, uniform sampling, or enumeration is the right executor. The protocol is validated on exact 720- and 5,040-order landscapes, on language-model instruction ordering where order alone moves accuracy from 6.3% to 96.9%, and on scientific and neural architecture benchmarks. A sympathetic reader would care because, if correct, it replaces universal optimizer claims with a cheap, recorded decision rule for when search is worth running at all.","feed_headline":"Two cheap stats predict when search beats random sampling","feed_subtitle":"The protocol also mapped 720 orderings of reasoning instructions, where order alone moves accuracy from 6.3% to 96.9%.","key_machinery":"The central object is the two-statistic pre-flight. The one-move autocorrelation $\\rho_1 = \\operatorname{corr}(F(\\pi), F(o(\\pi)))$ estimates local fitness preservation under a move operator $o$, and the fitness–distance correlation $\\operatorname{FDC} = \\operatorname{corr}(F(\\pi), d(\\pi, \\Omega))$ estimates global guidance toward the optimum set $\\Omega$. These two numbers route the search to an executor—exact enumeration, uniform sampling, an elitist evolutionary search, a surrogate, or transfer—under a distinct-evaluation budget, with a forecast ledger that retains null results and misses. The paper also uses exact enumeration of small spaces ($7!$ and $6!$) to obtain ground-truth optima and audited hit rates rather than method-dependent best observations.","core_discovery":"The central claim is that two inexpensive diagnostics—the one-move fitness autocorrelation $\\rho_1$ and the fitness–distance correlation FDC—separate permutation landscapes into guiding, near-random, and deceptive regimes, and that a protocol using these measurements can forecast, before the main budget is spent, whether an evolutionary searcher will beat uniform sampling. The paper shows this in exact ground truth: on the 5,040-order parity landscape, elitist search hits an optimum in 19 of 40 runs versus 30 of 40 for uniform sampling, matching the near-zero $\\rho_1$ and weak FDC forecast. On the 720-order instruction landscape, $\\rho_1 = 0.547$ and FDC $= -0.346$ predict insertion as the matched operator, yet dense optima make sampling competitive; both the success and the scoped null result are recorded as forecasts. The paper further claims that ordering structure transfers across model families and survives automated wording optimization, making ordering a distinct optimization coordinate.","pith_inferences":["The same two-statistic protocol could be applied to other fixed-component ordering problems listed in the paper, such as compiler pass ordering, database join plans, or laboratory automation, where an evaluator can be defined; these are testable extensions, not validated findings.","On larger landscapes where exact optima are unavailable, the proxy FDC may degrade exactly where the protocol is most useful; an uncertainty-aware probe that allocates extra evaluations to stabilise the proxy would be a natural follow-up.","If the thresholds generalize, PRISM could become a standard pre-registration step for optimization studies, with $\\rho_1$ and FDC reported before the search budget is spent.","The instruction-ordering result implies that prompt-optimization pipelines should explicitly search over order after content is fixed, rather than treating order as noise or calibrating it away."],"forward_implications":["On landscapes with near-zero local correlation, structured search can be worse than uniform sampling, so a correct sampling forecast counts as a success rather than a failed optimizer.","Practitioners can spend a small probe budget before the main search to choose between enumeration, sampling, and evolution, and record the forecast for later audit.","Ordering of fixed components should be treated as a separate optimization coordinate, alongside wording or content optimization.","The regime thresholds are empirical operating bands, not universal constants, so each new landscape family requires recalibration.","Exact ground truth at small $n$ exposes circular comparisons in which the optimizer's best observation is mistaken for the answer key."],"supporting_citations":[{"why":"Supplies the one-step autocorrelation measure of local fitness preservation used as the first pre-flight statistic.","marker":"[1]"},{"why":"Supplies the fitness–distance correlation measure of global guidance used as the second pre-flight statistic.","marker":"[2]"},{"why":"Connects permutation distance metrics and mutation operators to landscape structure, grounding the operator-alignment claim.","marker":"[3]"},{"why":"Supplies the reachability and almost-sure convergence guarantees for the elitist executor.","marker":"[4]"},{"why":"Establishes that few-shot prompt order can produce large performance differences, serving as the baseline for the instruction-landscape study.","marker":"[5]"},{"why":"Provides the presentation-order mechanism used to interpret why later instruction positions behave differently.","marker":"[12]"},{"why":"Supplies the GSM8K benchmark from which the 32-question subset for the exhaustive instruction study is drawn.","marker":"[18]"},{"why":"Underlies the six scientific pipeline systems used for cross-domain calibration of the protocol.","marker":"[20]"},{"why":"Supplies the NAS-Bench-201 backend for the registered tabular architecture forecasts.","marker":"[21]"}],"fun_headline_variants":["Two cheap stats predict when search beats random","Landscape metrics forecast search success upfront","Autocorrelation and FDC pick the right operator","Instruction order swings LLM accuracy from 6% to 97%","Measure two stats to know if optimization pays"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole forecast rests on the assumption that the empirical operating bands in Table 1 generalize to new landscapes, and that the proxy fitness–distance correlation, which uses a frozen best sample as the target, does not miss the true basin.","fun_headline_variants_meta":{"raw":{"variants":["Two cheap stats predict when search beats random","Landscape metrics forecast search success upfront","Autocorrelation and FDC pick the right operator","Instruction order swings LLM accuracy from 6% to 97%","Measure two stats to know if optimization pays"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000174,"raw_usage":{"total_tokens":1279,"prompt_tokens":941,"completion_tokens":338,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":265}},"tokens_in":557,"tokens_out":338,"duration_ms":4059,"temperature":1.0,"reasoning_tokens":265,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:07:35.912244+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh set of permutation optimization problems with known ground truth, such as random typed synthetic landscapes or new language-model instruction sets; run the pre-flight with the Table 1 bands unchanged and compare the predicted regime against the observed advantage of elitist search over uniform sampling at equal distinct-evaluation budgets. If forecast accuracy is at chance on these held-out problems, the central claim collapses.","supporting_citations":[{"cited_title":"Correlated and Uncorrelated Fitness Landscapes and How to Tell the Difference","cited_arxiv_id":null,"evidence_quote":"Supplies the one-step autocorrelation measure of local fitness preservation used as the first pre-flight statistic."},{"cited_title":"Fitness Distance Correlation as a Measure of Problem Difficulty for Genetic Algorithms","cited_arxiv_id":null,"evidence_quote":"Supplies the fitness–distance correlation measure of global guidance used as the second pre-flight statistic."},{"cited_title":"On Fitness Landscape Analysis of Permutation Problems: From Distance Metrics to Mutation Operator Selection","cited_arxiv_id":"2208.11188","evidence_quote":"Connects permutation distance metrics and mutation operators to landscape structure, grounding the operator-alignment claim."},{"cited_title":"Convergence Analysis of Canonical Genetic Algorithms","cited_arxiv_id":null,"evidence_quote":"Supplies the reachability and almost-sure convergence guarantees for the elitist executor."},{"cited_title":"Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity","cited_arxiv_id":null,"evidence_quote":"Establishes that few-shot prompt order can produce large performance differences, serving as the baseline for the instruction-landscape study."},{"cited_title":"The Curse of Verbalization: How Presentation Order Constrains LLM Reasoning","cited_arxiv_id":null,"evidence_quote":"Provides the presentation-order mechanism used to interpret why later instruction positions behave differently."}],"review_version":1}