{"id":"51a59e3c-b4c6-47b5-b45a-160e0904b6e0","arxiv_id":"2608.10694","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"Searching for prompts with a cheap evaluator and a strong reflector matches or beats optimizing directly on the deployment tier, at 5.6 to 14 times lower search cost.","lead":"Cheap language models can grade the many candidate prompts during evolutionary search while a strong model makes only the rare, high-value edits, and the final prompt transfers to stronger deployment models unchanged. This cut search cost by 5.6 to 14 times (up to 54 times on reasoning-heavy tiers) while matching or beating same-tier optimization.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'beats' half of the central claim rests on an unweighted pooled mean; Appendix G concedes the variance-weighted mean sits below zero, so positive transfer may not be robust.","rationale":"The reader's weakest assumption (cheap evaluator competence) is a scoped boundary condition explicitly acknowledged in Section 9; it explains when the method fails on new tasks but does not threaten the measured claim on the four tested tasks. A more immediate, load-bearing issue is the statistical support for the 'beats' component of the headline. Appendix G itself reports that a variance-weighted mean sits slightly below zero, which directly undercuts the pooled positive-transfer result. The unweighted +2.8% mean and its CI exclude zero, but only because high-variance, large-margin points (concentrated in the Mixed Qwen family, half the sample) receive equal weight. A proper hierarchical or variance-weighted analysis could easily shift the mean to zero or negative, reducing the claim to 'matches' rather than 'beats.' The paper deserves credit for disclosing this caveat, but the abstract and Section 5 do not carry the caveat forward, so the central claim overstates the evidence. The 'not worse' half remains solid via the sign test and bounded downside, so the reader's CONDITIONAL verdict is unchanged; the paper should either soften the positive-transfer language or supply the variance-weighted analysis. This is a different concern from the reader's competence assumption, hence 'disagree.'","tokens_in":30713,"tokens_out":8594,"duration_ms":78644,"concrete_test":"Recompute the pooled transfer residual δ% (Eq. 3) with a linear mixed-effects model: δ% ~ 1 + (1|setup) + (1|dataset), and also with inverse-variance weights (1/SE² per cell). Report the 95% CI for the overall mean and for each family excluding the Mixed Qwen arm. If the overall CI includes zero or the per-family paid-ladder CIs include zero, revise the abstract and Section 5 to say 'matches' rather than 'beats,' and mark the zero-shot positive-transfer claim (Section 2) as not established. A simpler check: compute the sign of the variance-weighted mean from the per-cell means and seed stds already reported in Tables 8–11; if it is negative, the unweighted +2.8% is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is that cheap-tier search 'matches or beats' same-tier optimization (Section 5) and specifically exhibits zero-shot positive transfer (Rs→t<0, Section 2). The evidence for the positive part is the pooled mean residual δ̄=+2.8% with 95% CI [+1.3%, +4.4%] over 48 points (Figure 3, Appendix G). This inference depends on treating the 48 residuals as independent and equally weighted, but they are clustered by 4 datasets and 12 setups, with widely differing variances (per-cell seed std up to ±6.7 points on HotpotQA). Appendix G concedes: 'a variance-weighted mean, which downweights the noisier large-margin points, sits slightly below zero.' That concession means the sign of the pooled mean flips under a more appropriate weighting, so the data do not robustly establish positive transfer. Moreover, of the four families, only Mixed Qwen (which contributes 24 of the 48 points) has a family-specific interval that excludes zero; the paid ladders (GPT +1.0%, Gemini +1.5%, Mixed Claude +3.0%) have intervals overlapping zero. The unweighted pooling therefore lets the most favorable, highest-variance family carry the 'beats' conclusion. What remains robust is 'not worse': 36/48 residuals are non-negative and the worst observed shortfall is 6.12% (≈3.8 points on the relevant scales). The abstract's 'matches or exceeds' is thus only half-supported; 'exceeds' is not a statistically robust finding.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a cost-aware reformulation of evolutionary LLM prompt optimization (instantiated on GEPA, with a MIPROv2 ablation) in which the fitness-evaluation role, the reflective variation role, and the deployment role are assigned to different model tiers. A cheap answerer scores candidates, a strong reflector proposes edits, and the evolved prompt is deployed zero-shot on stronger tiers. Across four tasks and eleven models in four families, the paper reports that this configuration matches or beats full same-tier optimization in 36 of 48 deployment cells, at 5.6–14× (25–54× on Gemini) lower search cost, with over 96% of search tokens on the cheapest tier. The paper also presents role ablations locating the gain in the reflector, an explicitness analysis of the evolved prompts, break-even volume analysis, and a pooled transfer-residual analysis.","tokens_in":30891,"tokens_out":6273,"duration_ms":110074,"significance":"If the central claim held, the paper would be a practically valuable result: it would decouple search cost from deployment-tier capability and supply a characterization of when cheap-tier search substitutes for target-tier search. The experimental design is a strength: shared per-task metric-call budgets, n=3 seeds, zero-cost seed baselines on every tier, a 2×2 role ablation, an optimizer ablation with MIPROv2, a neutral cross-family deploy target, and honest reporting of search traces that rank the cheap arms below the full-cost ones. The cost accounting is unusually careful (per-call token logs at published prices, break-even price ratios, deployment break-even volumes). The main weakness is that the 'beats' half of the claim rests on an unweighted pooled residual whose robustness is explicitly conceded in Appendix G; the data robustly support 'not worse,' and the positive-transfer contribution needs either stronger evidence or re-scoping.","major_comments":[{"comment":"The claim in the abstract and Section 5 that cheap search 'exceeds' or 'beats' same-tier optimization is not statistically robust. The evidence is the pooled mean δ̄=+2.8% with 95% CI [+1.3%, +4.4%] over 48 residuals, but those residuals are clustered by 12 setups and 4 datasets with heterogeneous variances, and Appendix G concedes that 'a variance-weighted mean, which downweights the noisier large-margin points, sits slightly below zero.' Only the Mixed Qwen family (24 of 48 points) has a family-level CI excluding zero; GPT (+1.0%), Gemini (+1.5%), and Mixed Claude (+3.0%) intervals overlap zero. The data therefore establish 'matches' (36/48 non-negative; worst shortfall 6.12%) but not 'exceeds.' Please either provide a robustness analysis (hierarchical model, family-level inference, or pre-specified weighting) that supports positive transfer, or rephrase the central claim and conclusion as 'at least matches' and characterize the positive tail as suggestive.","section":"§6, Appendix G, Fig. 3"},{"comment":"The method's boundary condition is stated but not operationalized: Section 9 says that if the cheap score is approximately 0, the fitness landscape is flat and search stagnates, so the cheap evaluator 'must achieve at least marginal success.' This is structurally distinct from the transfer claim, and the abstract's unconditional phrasing ('the resulting prompt matches or exceeds') and Section 5 headline do not carry this scope condition. Over the tested tasks the cheap answerers are all clearly above floor, but the paper does not report how close to zero a cheap tier can be before the method degrades, nor provide a diagnostic (e.g., seed-prompt score on the cheap tier) that practitioners could use to decide whether the method applies. Please either add such an analysis or explicitly qualify the headline claim as applying only when the cheap tier achieves non-trivial baseline competence.","section":"Section 9, Section 5"}],"minor_comments":[{"comment":"The phrase 'matches or beats ... at every deploy tier we tested' is stronger than the cell-by-cell data; the paper's own count is 36 of 48 at or above parity, and some cells (e.g., LiveBench-Math at luna, 75.2 vs 76.4; IFBench@3.5-flash, 69.7 vs 71.2) are below parity. Please qualify the headline with the 36/48 count and the worst observed shortfall.","section":"Section 5, first paragraph"},{"comment":"The cost normalization used in the pts/$ comparisons (dividing raw totals by measured token-log coverage) is essential to the headline '236× difference in return per search dollar' but only appears in the appendix. Consider stating in the main text that the cost figures in Table 2 are normalized so readers do not misread them as raw measured spend.","section":"Appendix C, Table 2"},{"comment":"The caption 'never worse than 3.8 points' could be misread as an absolute score; it means 3.8 points below the full same-tier baseline on the quality-gap axis. Please make explicit that the bound is relative to the baseline.","section":"Figure 1 caption"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it is the rare empirical study that takes cost seriously as a design axis: it splits the LLM in an evolutionary loop into three roles, cheap answerer, strong reflector, deployment target, and measures, with controlled budgets and seeds, when optimizing on the cheap tier transfers up. Second, the \"beats\" half of the headline does not survive a close look. Appendix G concedes that a variance-weighted pooled mean sits slightly below zero, and only the Mixed Qwen family's interval excludes zero. The unweighted +2.8% mean is carried in part by that family. So the robust claim is \"not worse at 5-54x lower cost,\" and the positive-transfer claim is suggestive but not established.\n\nWhat is genuinely new: the prior weak-to-strong observations were incidental single-pair findings. Here the authors turn it into a cost strategy and characterize where it holds across tasks, model families, price tiers, and an alternate optimizer (MIPROv2). The role ablation cleanly shows the gain comes from the strong reflector, not the cheap evaluator. The zero-cost seed baseline and the reported negative instances (e.g., IFBench@3.5-flash where the seed beats all optimized prompts) make the headroom boundaries visible. That is honest and useful.\n\nThe soft spots, in proportion. The lack of released code, data, and token logs is the biggest one, nothing here is independently checkable without re-implementation, and the per-cell seed spreads on HotpotQA are wide enough that several cells sit within noise. The luna parse fix is disclosed and plausible, but it is a manual pipeline intervention. The structural-explicitness explanation is clearly labeled post-hoc, which is fine. The limitation that the cheap tier must have baseline competence is real, load-bearing, and properly scoped.\n\nWho this is for: anyone working on prompt or program optimization, and anyone who pays for LLM inference at scale. The recipe is concrete and the cost accounting is unusually careful. I would send it to peer review, but require artifact release and a rewrite of the \"exceeds\" language before acceptance. The practical message, search cheap and deploy strong without losing quality, is likely to hold.","headline":"Well-controlled empirical study showing cheap-tier search with a strong reflector is not worse than same-tier optimization at 5-54x lower cost; the 'beats' half is overstated because the positive transfer claim flips under variance weighting.","tokens_in":31592,"tokens_out":2555,"would_cite":true,"duration_ms":29216,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Decoupling search tier from deployment tier makes prompt optimization 5.6–14× cheaper without losing quality.","keywords":["evolutionary prompt optimization","cost-aware LLM search","cross-tier transfer","fitness evaluation cost","reflective mutation","zero-shot prompt transfer","LLM tiers"],"falsifier":"Run the same protocol on a task where the cheap tier scores near zero on the seed prompt and on all random variations (e.g., a specialized reasoning task far from the cheap model's competence). If the cheap search then fails to beat the untrained seed on the strong deployment tier, while same-tier full-cost optimization succeeds, the central claim fails.","tokens_in":30358,"feed_emoji":"⚙️","tokens_out":3276,"duration_ms":25556,"temperature":0.7,"pith_summary":"This paper claims that the dominant cost of evolutionary prompt optimization, the fitness evaluation of each candidate prompt, can be moved from the target (deployment) model to the cheapest available model without sacrificing final quality, provided a strong model is kept for the rare edit-proposing role. The authors demonstrate this with a cost-aware cross-tier transfer method across four tasks and eleven models, showing that a prompt evolved with a cheap answerer and strong reflector matches or beats the prompt optimized directly on the target tier at full price, with over 96% of search tokens spent on the cheap tier. Why it matters: if true, practitioners can decouple search cost from deployment capability, making prompt optimization roughly an order of magnitude cheaper and shifting the cost to a one-time search expense rather than a per-query cost.","feed_headline":"Cheap search, strong deploy: prompt optimization at 5-14x less cost","feed_subtitle":"A prompt evolved on the cheapest model, with a strong editor, matches same-tier optimization on 36 of 48 deployments.","key_machinery":"The central object is the decoupling of three LLM roles that standard practice fuses: the answering model that scores candidates during search (the fitness evaluator), the variation operator that reads traces and proposes edits (the reflector), and the deployment model that serves the final prompt. The paper assigns the cheap answerer to fitness evaluation, the strong reflector to variation, and any target tier to deployment, with zero mapping or recalibration. The formal identity is the decomposition of the deployment objective into the cheap surrogate objective plus a cross-tier transfer residual, with the target regret as the key measured quantity contrasting cheap-tier search with same-tier search. The load-bearing mechanism is the volume asymmetry: fitness evaluation dominates token spend, so cheapening that role moves most of the cost, while the reflector's rarity makes its strength affordable.","core_discovery":"The central claim is that a prompt searched on a cheap model with a strong reflector, then deployed zero-shot on a stronger tier, matches or exceeds the prompt that same tier optimized for itself at full price. The paper formalizes the trade-off with target regret: the difference between the deployment-tier quality of the prompt optimized on the target and the prompt optimized cheaply and transferred upward. Across 48 (task, search arm, deploy tier) deployments, the mean transfer residual is +2.8% in favor of the cheap search, with 36 of 48 at or above parity and no scenario worse than 6.12% short. The cost reduction is 5.6–14× on the Mixed Claude and GPT ladders and 25–54× on Gemini, rising to 63–114× when the answerer is a self-hosted local model. The saving comes from the volume asymmetry: over 96% of search tokens are fitness-evaluation tokens, so moving that role to the cheapest tier moves almost the whole bill, while the strong reflector, though expensive per token, is rare. Upward transfer is positive because the variation operator is strong: a role ablation shows the reflector, not the evaluator, produces the transferable prompt. The cheap search also writes more explicit prompts, implying the mechanism that a weak evaluator rewards spelled-out structure that a stronger model can exploit.","pith_inferences":["The explicitness account suggests a testable prediction: if a cheap answerer is made artificially weaker or more literal, the evolved prompts should become even more explicit and transfer even better, while a cheap answerer that is already strong may yield less transferable prompts.","The method's scope likely extends beyond prompt text to agentic program modules, demonstrations, and tool-use policies, since the decoupling applies to any artifact whose fitness is evaluated by an LLM and whose variation is proposed by an LLM.","The measured break-even volume implies a practical decision rule for practitioners: the cheap-search strategy is best for one-time or moderate-volume deployments, while for extremely high-volume serving, a longer prompt's per-query cost can erode the one-time saving.","The finding that a cheap evaluator need only rank, not estimate, suggests that noisy or quantized evaluation metrics, or sub-sampled validation sets, might preserve much of the transfer benefit while further cutting cost."],"forward_implications":["If the central claim is correct, practitioners can run one cheap search and deploy the resulting prompt on any stronger tier in the same family, with no per-tier re-optimization.","Search cost becomes a one-time tuning expense rather than a per-query cost, so prompt optimization becomes viable for tasks and teams where full same-tier search was previously unaffordable.","The saving is robust to price changes: in 15 of 24 cells the cheap composition stays cheaper even at price parity, because it emits fewer output tokens overall.","The effect is not tied to a specific optimizer: repeating the construction with MIPROv2 instead of GEPA keeps the results, indicating the decoupling is the operative ingredient rather than one mutation operator.","The direction of transfer matters: upward transfer is reliable, whereas lateral transfer between similar-capability models degrades, so prompts should be evolved on a weak model and deployed on a strong one."],"supporting_citations":[{"why":"Provides the GEPA reflective prompt evolution method and its Eq. (2) optimization problem that this paper restructures, and the weak-to-strong transfer observation that motivates systematic study.","marker":"Agrawal et al. (2026)"},{"why":"Establishes the baseline of lateral cross-model prompt degradation ('model drift') that upward transfer is contrasted against.","marker":"Wang et al. (2025b)"},{"why":"Reports another weak-to-strong transfer observation and the prompt-insensitive ceiling cases that bound the method's headroom.","marker":"Gao et al. (2026)"},{"why":"Baseline cost-aware prompt optimization that cheapens within the target tier, against which this paper positions its cross-tier approach.","marker":"Zehle et al. (2025)"},{"why":"Establishes LLMs as prompt variation operators, the lineage this paper's strong-reflector component builds on.","marker":"Fernando et al. (2024)"},{"why":"Supplies MIPROv2, the alternative optimizer used in the ablation showing the decoupling is optimizer-independent.","marker":"Opsahl-Ong et al. (2024)"}],"fun_headline_variants":["Evolve prompts on cheap tier, deploy on strong: 5-14x cost cut","Weak evaluator, strong reflector: cheap prompt search beats same-tier","Cost-aware transfer: 96% tokens on cheap tier, 5-14x savings, same quality","From cheap to strong: prompt transfer matches same-tier at 5-14x less"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The cheap evaluator must be able to rank candidate prompts at least marginally well on the task; if it scores near zero everywhere, the fitness landscape is flat and search stagnates.","fun_headline_variants_meta":{"raw":{"variants":["Evolve prompts on cheap tier, deploy on strong: 5-14x cost cut","Weak evaluator, strong reflector: cheap prompt search beats same-tier","Cost-aware transfer: 96% tokens on cheap tier, 5-14x savings, same quality","From cheap to strong: prompt transfer matches same-tier at 5-14x less"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001242,"raw_usage":{"total_tokens":5133,"prompt_tokens":1018,"completion_tokens":4115,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":4022}},"tokens_in":634,"tokens_out":4115,"duration_ms":39670,"temperature":1.0,"reasoning_tokens":4022,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:13:33.454287+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same protocol on a task where the cheap tier scores near zero on the seed prompt and on all random variations (e.g., a specialized reasoning task far from the cheap model's competence). If the cheap search then fails to beat the untrained seed on the strong deployment tier, while same-tier full-cost optimization succeeds, the central claim fails.","supporting_citations":[],"review_version":2}