{"id":"2f3d9d13-ed60-4d7c-9a5a-91efc432822b","arxiv_id":"2505.08303","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Black-box prompt optimization yields small or negative gains on very large LLMs, and its benefit appears to shrink as model scale increases.","lead":"This paper tests whether automated prompt optimization still helps when large language models become very large. It reports that on DeepSeek V3 and Gemini 2.0 Flash, three popular optimization methods yield only small gains, and that gains shrink as model size grows.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Inverse-scaling claim rests on single runs of one optimizer, no variance estimates, and a model-family confound; the reported trend may be noise.","rationale":"The reader's rejection is well-founded. My independent stress-test converges on the same load-bearing weakness: the inverse-scaling conclusion in Section 4.2 is not supported by the reported measurements because there are no variance estimates, no repeated optimization runs, and no control for model family when extrapolating from Qwen 2.5 to DeepSeek V3. These are not stylistic issues; they directly affect whether the claimed monotone decline is real. The paper itself concedes the 72B-to-671B gap is untested, and the single-run EvoPrompt comparisons in Table 1 show differences as small as 0.4 accuracy points on 500 examples, which is within sampling noise. A more rigorous version of the study—multiple seeds, confidence intervals, and ideally a method-agnostic analysis across several model families—could still reveal a genuine trend, but the current evidence does not establish the central claim. Therefore I agree with the reader that rejection is appropriate, and I do not see any support that would move the verdict in the other direction.","tokens_in":6651,"tokens_out":2348,"duration_ms":25704,"concrete_test":"Run EvoPrompt with at least 5 independent random seeds per model (Qwen 2.5 7B, 14B, 32B, 72B, and DeepSeek V3) on SST-5 and AG News under identical optimization budgets and with the same initial prompt. Report the mean and 95% bootstrap confidence interval for each accuracy improvement, and test for a monotone decrease across model sizes using a permutation or rank-based test. If confidence intervals for adjacent sizes overlap substantially, or if seed variation reverses the ordering, the inverse-scaling claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim in Section 4.2—that black-box prompt optimization gains decline significantly as model scale increases—is supported only by EvoPrompt on Qwen 2.5 (7B–72B) plus a single external point from DeepSeek V3 (671B). No error bars, repeated seeds, or significance tests are reported. On a 500-example test set, an accuracy change such as 56.0 to 56.4 is just two additional correct examples, well within binomial sampling noise (Section 3.1, Table 1). EvoPrompt is stochastic, so one optimization trajectory per model cannot establish a monotone trend; the 12% to 5.9% to 1.1% progression cited for SST-5 in Section 4.2 may reflect optimizer variance rather than scale. In addition, the final point is not a Qwen 2.5 671B model but DeepSeek V3, a different architecture and training pipeline, so any decline from Qwen 72B to DeepSeek V3 confounds scale with model family. The paper's own Limitations section acknowledges the untested gap between 72B and 671B. Table 3 also shows qualitatively different optimized prompts across model sizes, meaning the optimization process is not held constant while scale varies. Together, these issues mean the evidence does not distinguish the claimed inverse-scaling law from random variation, optimizer instability, or model-family differences.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates three black-box prompt optimization methods (EvoPrompt, ProTeGi, BPO) on two large-scale LLMs (DeepSeek V3 and Gemini 2.0 Flash) across four NLU and NLG datasets, reporting only limited performance gains. To explain these limited gains, the authors hypothesize that model scale is the primary factor, and they test this by applying EvoPrompt to the Qwen 2.5 series (7B–72B) plus a single external point from DeepSeek V3 (671B). Based on these experiments, the paper claims an 'inverse scaling law' in which the effectiveness of black-box prompt optimization declines as model size increases.","tokens_in":6899,"tokens_out":5165,"duration_ms":48492,"significance":"If robustly established, the claimed inverse-scaling law would be practically important: it would suggest that black-box prompt optimization yields diminishing returns as LLMs grow, guiding practitioners to allocate optimization effort only for smaller models. The paper also contributes a useful preliminary data point on the behavior of three optimizers on very large models. However, the current evidence is far from establishing a law. The central claim rests on unrepeated single runs without uncertainty estimates, on a single optimizer for the scaling curve, and on a cross-family comparison that confounds model scale with model family and training pipeline. These issues are load-bearing for the paper's main conclusion.","major_comments":[{"comment":"The central inverse-scaling claim is supported only by single runs of EvoPrompt on each Qwen 2.5 model, with no standard deviations, confidence intervals, or significance tests. On SST-5, the reported improvement at the DeepSeek V3 point is 56.0 to 56.4, which is just two additional correct answers on a 500-example test set and lies within binomial sampling noise; the monotone decline from 12% to 5.9% to 1.1% could therefore reflect optimizer stochasticity rather than model scale. Please repeat the optimization with multiple seeds and report variance estimates and, where possible, paired significance tests.","section":"Section 4.2, Figure 1"},{"comment":"The scale comparison appends DeepSeek V3 (671B) to the Qwen 2.5 series, so model scale is confounded with model family, architecture, and training pipeline. The paper's own Limitations section explicitly acknowledges the untested gap between 72B and 671B and the differing model family; consequently, the final point in Figure 1 cannot be attributed to scale, and the label 'inverse-scaling phenomenon' is not warranted. The claim should either be restricted to the Qwen 2.5 family or supported by a controlled multi-family design that accounts for family identity.","section":"Section 4.2, Limitations"},{"comment":"The scaling analysis uses only EvoPrompt, yet the abstract and Section 4.2 generalize to 'black-box prompt optimization methods'; this generalization is unsupported by the data. In addition, Table 3 shows that the optimized prompts differ qualitatively across model sizes (e.g., the AG's News prompts contain different instructions and wording), so the optimization process is not held constant while scale varies. The observed decline in gains could stem from differences in the discovered prompts rather than from model scale. Applying all three methods across the scale range and reporting prompt similarity or edit distance would help disentangle these factors.","section":"Section 4.1, Table 3"},{"comment":"The claim that improvements on large-scale models are 'limited' is made without a comparison to smaller models under an identical setup within this section, and several entries in Table 1 are within the noise floor for a 500-example test set (e.g., DeepSeek V3 with BPO on AG's News decreases from 84.6 to 83.8, and SST-5 improvements of 0.4 points correspond to two examples). The average improvements of 0.86% to 2.03% need confidence intervals or statistical tests before they can be interpreted as nonzero effects.","section":"Section 3.3, Table 1"}],"minor_comments":[{"comment":"The Limitations section contains a typo: 'treasoning-oriented LLMs' should be 'reasoning-oriented LLMs.'","section":"Limitations"},{"comment":"The references for Anil et al., Brown et al., and Hurst et al. contain 'and 1 others' placeholders; the full author lists should be provided.","section":"References"},{"comment":"The dataset references for SST-5 and AG's News point to URLs with access dates; proper bibliographic entries (e.g., ACL Anthology or a formal publisher citation) should be used instead.","section":"References"},{"comment":"The phrase 'a distinct inverse-scaling phenomenon' is an interpretive label rather than a fitted scaling law; 'observed trend' would be more precise in the absence of a quantitative fit.","section":"Section 4.2"},{"comment":"Table 1 does not explain how the 'Average % Increase' is computed; the authors should state whether it is relative to the initial score or an average of per-method relative changes.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable preliminary study but currently reads more like a workshop report than a journal article. The central inverse-scaling claim needs substantially stronger evidence: repeated seeds, error bars, multiple optimizers on the scaling curve, and either a controlled multi-family design or a narrowed claim. The authors should also consider whether the current venue is appropriate unless the scaling analysis is substantially strengthened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou'll want to know: this paper asks a good question and reports a clean-sounding answer—black-box prompt optimization gains shrink as LLMs grow—but the evidence behind the answer is thinner than the framing suggests. The Qwen 2.5 scaling curve is new, but it rests on single runs of one optimizer with no variance estimates, and the top point comes from a different model family.\n\nWhat's genuinely useful: the authors evaluate three black-box methods on DeepSeek V3 and Gemini 2.0 Flash across four datasets, and they show concrete examples of how optimized prompts differ by scale. The observation that large models barely benefit from lexical tweaks is consistent with prior work (Shirafuji et al.), and the paper extends that to a within-family trend. The writing is clear, and the Limitations section is honest about the 72B–671B gap.\n\nThe soft spots are load-bearing. Every number in Table 1 comes from a single run. A change like 56.0 to 56.4 on a 500-example test set is two examples; that's sampling noise. Figure 1 has no error bars or significance tests, so the 12% → 5.9% → 1.1% progression could easily be optimizer instability. EvoPrompt is stochastic; one trajectory per model doesn't establish a monotone trend. And attaching DeepSeek V3 as the 671B data point confounds scale with architecture and training pipeline. The authors also note the optimized prompts themselves differ across sizes, so the intervention changes, not just the model.\n\nThat said, the inverse scaling hypothesis is plausible and worth testing. The paper just doesn't have the measurement to support calling it a law. I'd assign it to a serious referee: the question is timely for practitioners, and the work is a reasonable pilot. But the referee should require repeated runs, confidence intervals, and a clean scaling study within a single family before accepting the central claim. As it stands, I'd file this as a preliminary report, not a citable result.\n\nMy recommendation for peer review: send it out—the question matters enough that a rigorous referee could push the authors toward a more defensible design. I'd expect heavy revision, or rejection as-is if the authors can't provide variance estimates and address the confounds.","headline":"Useful preliminary measurement, but the 'inverse scaling law' is not supported by the evidence; treat the claim as a hypothesis, not a result.","tokens_in":7429,"tokens_out":2991,"would_cite":false,"duration_ms":28723,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Black-box prompt optimization delivers shrinking gains as language models grow, an inverse-scaling effect.","keywords":["black-box prompt optimization","inverse scaling law","large language models","EvoPrompt","prompt optimization","model scale","NLU benchmarks","NLG benchmarks"],"falsifier":"Run EvoPrompt with, say, ten independent optimization runs on Qwen2.5-7B and Qwen2.5-72B on SST-5, computing paired bootstrap confidence intervals for the mean gain; if the 7B gain is not significantly larger than the 72B gain, the inverse-scaling claim is unsupported. Alternatively, repeat the Qwen-scale experiment on another LLM family spanning 7B to 72B and show flat or increasing gains with model size.","tokens_in":6447,"feed_emoji":"📉","tokens_out":7184,"duration_ms":63255,"temperature":0.7,"pith_summary":"Black-box prompt optimization—automatically rewriting an instruction by treating the model as an unopenable box—has mostly been tested on 7B–14B models. This paper asks whether the same tricks still pay off on frontier-scale models and reports that they do not, by much: on DeepSeek V3 and Gemini 2.0 Flash, three optimizers improve four benchmarks by only about 0.86%–2.03% on average. It then argues that model scale itself is the cause, showing that EvoPrompt's gain on SST-5 drops from 12% at Qwen 7B to 5.9% at Qwen 72B to 1.1% at DeepSeek V3 671B. If this inverse-scaling pattern holds, automatic prompt optimization will matter less as models grow, shifting the practical value of prompt engineering toward smaller models and richer prompt content rather than lexical tweaks.","feed_headline":"Black-box prompt optimization pays off less on big AI models","feed_subtitle":"Across the Qwen 2.5 line, optimization gains drop from 12% at 7B to under 6% at 72B.","key_machinery":"The central object is the scaling curve produced by applying EvoPrompt—an evolutionary algorithm that rewrites prompts through crossover and mutation—to the Qwen 2.5 family at 7B, 14B, 32B, and 72B, with DeepSeek V3 (671B) as a further endpoint. The mechanism claimed is that the optimized prompts differ from initial prompts only in lexical detail, and such superficial changes matter less when the model's alignment is already strong. The argument treats the trend across model sizes as evidence that model scale, rather than optimizer choice or dataset, drives the diminishing returns.","core_discovery":"The paper's central claim is an inverse scaling law: the performance gain from black-box prompt optimization declines as LLM scale increases. The key observations are the small average gains on two large-scale models (DeepSeek V3 and Gemini 2.0 Flash) across SST-5, AG's News, SAMSum, and ASSET, and the monotonic decline of EvoPrompt gains across the Qwen 2.5 series, illustrated by SST-5 accuracy: 12% at 7B, down to 5.9% at 72B, and 1.1% at DeepSeek V3 (671B). The authors propose a mechanism: larger models have stronger alignment and domain understanding, so explicit domain cues and synonym-level rephrasing, which help smaller models, become redundant.","pith_inferences":["A direct test would be to run EvoPrompt with multiple independent seeds on Qwen2.5-7B and Qwen2.5-72B, computing paired bootstrap confidence intervals; if the 12% versus 5.9% gap on SST-5 does not survive, the inverse-scaling claim reduces to noise.","The same decline may partly reflect benchmark saturation: larger models already sit near the ceiling on these tasks, leaving less headroom for any optimizer to add.","If robust alignment is the cause, similar declines should appear for white-box prompt-tuning methods that access model gradients, not just for black-box optimizers.","The authors' own limitations note that reasoning-oriented models were excluded; a natural next test is whether inverse scaling holds for chain-of-thought-style models, which may be more prompt-sensitive."],"forward_implications":["For frontier models, automatic black-box prompt optimization may no longer justify its API cost, since the measured gains are often under one or two percent.","Benchmark comparisons across model sizes should control for prompt sensitivity, because the same optimized prompt yields very different gains at 7B versus 72B.","Smaller models, around 7B, remain the most promising targets for black-box prompt optimization in practical applications.","Prompt engineering effort may need to shift from lexical rewording toward richer context, demonstrations, or reasoning scaffolds, which the paper does not test.","Researchers evaluating new prompt optimizers should report results on large models separately, since averaging across scales can mask an inverse-scaling effect."],"supporting_citations":[{"why":"Supplies EvoPrompt, the evolutionary algorithm whose gains across Qwen model sizes form the paper's inverse-scaling curve.","marker":"Guo et al., 2023"},{"why":"Supplies ProTeGi, one of the three black-box optimizers evaluated on DeepSeek V3 and Gemini 2.0 Flash.","marker":"Pryzant et al., 2023"},{"why":"Supplies BPO, the sequence-to-sequence prompt optimizer used as the third comparison method.","marker":"Cheng et al., 2023"},{"why":"Defines DeepSeek V3, the 671B-parameter model that serves as the largest endpoint in the scaling comparison.","marker":"DeepSeek-AI, 2024"},{"why":"Defines Gemini 2.0 Flash, the other large-scale model on which limited optimization gains are reported.","marker":"Pichai et al., 2024"},{"why":"Provides SST-5, the sentiment benchmark whose headline numbers (12%, 5.9%, 1.1%) anchor the inverse-scaling claim.","marker":"Socher et al., 2013"},{"why":"Supports the explanation that superficial prompt changes have little effect on larger models.","marker":"Shirafuji et al., 2023"}],"fun_headline_variants":["Bigger LLMs shrink prompt optimization gains","Prompt tuning payoff drops as model scale grows","Inverse scaling found for prompt optimization","Large models resist black-box prompt tweaks","Prompt optimization loses edge on giant LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion rests on treating one run of EvoPrompt per model size, without error bars, as an accurate measure of optimization effectiveness, and on assuming that the Qwen 2.5 family's size differences, rather than dataset headroom or optimizer choices, are what drive the declining gains.","fun_headline_variants_meta":{"raw":{"variants":["Bigger LLMs shrink prompt optimization gains","Prompt tuning payoff drops as model scale grows","Inverse scaling found for prompt optimization","Large models resist black-box prompt tweaks","Prompt optimization loses edge on giant LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1345,"prompt_tokens":962,"completion_tokens":383,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":318}},"tokens_in":578,"tokens_out":383,"duration_ms":4611,"temperature":1.0,"reasoning_tokens":318,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:57:33.183105+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run EvoPrompt with, say, ten independent optimization runs on Qwen2.5-7B and Qwen2.5-72B on SST-5, computing paired bootstrap confidence intervals for the mean gain; if the 7B gain is not significantly larger than the 72B gain, the inverse-scaling claim is unsupported. Alternatively, repeat the Qwen-scale experiment on another LLM family spanning 7B to 72B and show flat or increasing gains with model size.","supporting_citations":[],"review_version":1}