{"id":"7fec0484-e69f-4f96-b77d-d0b72f29ca1a","arxiv_id":"2501.03486","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Prompt optimization for frozen LLMs is formulated as KL-regularized reward maximization, and the alignment gap to RLHF is bounded by total-variation terms minus a KL term.","lead":"This paper introduces Align-Pro, a framework that trains a small prompter model to rewrite prompts for a frozen language model, and it derives an upper bound on how far this prompt-based alignment can fall behind full RLHF fine-tuning. The bound and proof-of-concept experiments suggest prompt optimization is a viable alignment method when model parameters cannot be updated.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 6.1 is false as stated for signed rewards: the proof's TV bounds require r* ∈ [0, rmax], an assumption the paper never states; a two-prompt counterexample violates Eq. (11).","rationale":"The paper's framework is coherent and Lemma 5.1 follows the standard KL-regularized optimization derivation, but the headline theorem has a genuine sign-error in the total-variation bound. The proof treats rmax as a bound on the reward magnitude when the inequalities actually need a bound on the reward range; with signed rewards, the expectation difference can be twice the magnitude. My counterexample is small and computable, so the flaw is not a matter of interpretation. The reader's concern about the PPO-trained prompter not being connected to the closed-form ρ* is valid and remains a serious empirical gap, but it is secondary: even an exactly optimized ρ* does not satisfy the stated bound for signed rewards. The right resolution is conditional acceptance: the authors should add the missing nonnegativity or range assumption, correct the proof accordingly, and either connect the empirical PPO training to ρ* or explicitly reframe the experiments as a separate proof-of-concept. This does not require rejecting the overall idea, since the bound is repairable, but the theorem as currently written cannot stand.","tokens_in":16641,"tokens_out":12651,"duration_ms":115107,"concrete_test":"Run the explicit two-prompt counterexample: x, a, b with r*(x,y1)=1, r*(x,y2)=-1, π_F(y1|x)=π_F(y1|a)=1, π_F(y2|b)=1, ρ_sft(a|x)=ρ_sft(b|x)=0.5, and λ=1. Compute ρ* from Eq. (10), then evaluate both sides of Theorem 6.1; if the left side 0.238 exceeds the right side 0.172, the theorem as stated is refuted. Then check whether adding the assumption r* ∈ [0, rmax], or replacing rmax by (sup r - inf r), restores the inequalities in Appendix B; if so, the result is salvageable but requires a stated assumption.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central theoretical result, Theorem 6.1, relies on bounding expectation differences by rmax d_TV in Eqs. (26) and (31). This inequality is valid when r* is nonnegative, but not when rewards may be negative; if r* ∈ [-rmax, rmax], the sharp constant is (sup r - inf r)·TV, which can be 2 rmax·TV. The paper defines no such assumption. Concretely, take a degenerate prompt distribution P at x, two prompt states a and b with ρ_sft(a|x)=ρ_sft(b|x)=0.5, responses y1 and y2, rewards r*(x,y1)=1, r*(x,y2)=-1, and π_F(y1|x)=π_F(y1|a)=1, π_F(y2|b)=1, with λ=1. Then π*=δ_y1, so the first TV term in Eq. (11) is 0. The second term is 0.5. The optimal prompter from Eq. (10) is ρ*(a)=e/(e+e^{-1})≈0.881, ρ*(b)≈0.119, giving D_KL(ρ*||ρ_sft)≈0.328. The RHS of Eq. (11) is therefore 0.172, but J(π*) - J(e_ρ*) = 1 - 0.762 = 0.238, violating the theorem. The failure is precisely the proof's Eq. (31): Δ3 = 1, while rmax E_{ρ_sft}[d_TV(πF(x),πF(x'))] = 0.5. Thus the stated bound is not merely loose; it is false for the general bounded-reward setting the theorem claims. A corrected theorem must either assume r* ∈ [0, rmax] or use the reward range in the TV bounds.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Align-Pro, a prompt optimization framework for aligning frozen LLMs without parameter fine-tuning. It formulates prompter learning as maximizing the expected reward of the frozen model under a KL-regularization penalty relative to a reference prompter, derives a closed-form optimal prompter (Lemma 5.1), and proves a suboptimality bound comparing the resulting policy with the RLHF optimal policy (Theorem 6.1). Experiments on UltraFeedback, HelpSteer, and Orca with PPO-trained prompters report improved mean reward and win rate over a no-fine-tuning baseline.","tokens_in":17068,"tokens_out":9170,"duration_ms":84075,"significance":"If Theorem 6.1 were correct as stated, the decomposition would be a useful conceptual contribution: it separates the prompt-optimization gap into frozen-model quality, reference-prompt sensitivity, and a KL-budget term. The closed-form prompter and the accompanying proof effort are valuable. However, the theorem currently omits a necessary reward-range assumption, and the proof's total-variation bounds fail for signed rewards. In addition, the experiments train the prompter with PPO against a proxy reward model and never report the KL coefficient lambda, so they do not test the object analyzed in the theorem. These issues are local and repairable, but the central theoretical claim needs correction before publication.","major_comments":[{"comment":"The central bound is not valid for the general bounded-reward setting stated. The proof uses the inequality |E_{pi1}[r] - E_{pi2}[r]| <= rmax * d_TV(pi1, pi2), which holds for nonnegative rewards with supremum rmax, but not for signed rewards. For r in [-rmax, rmax], the sharp constant is (sup r - inf r) = 2 rmax, and the paper never states a nonnegativity assumption or defines rmax. A concrete counterexample: take a degenerate prompt distribution at x, two prompts a and b with rho_sft(a|x)=rho_sft(b|x)=0.5, responses y1 and y2, rewards r*(x,y1)=1 and r*(x,y2)=-1, pi_F(y1|x)=pi_F(y1|a)=1, pi_F(y2|b)=1, and lambda=1. Then pi* = delta_{y1}, the first TV term in Eq. (11) is 0, the second is 0.5, D_KL(rho*||rho_sft) is approximately 0.328, and the RHS is 0.172, while J(pi*) - J(e_rho*) = 1 - (0.881 - 0.119) = 0.238, violating the inequality. The theorem can be repaired by replacing rmax with the reward range (sup r - inf r) or by explicitly assuming r* in [0, rmax] and defining rmax, but as written the statement is false.","section":"Theorem 6.1 / Eq. (11) and Appendix B, Eqs. (26), (31)"},{"comment":"The experiments do not instantiate the object analyzed in Theorem 6.1. The theorem concerns the exact maximizer rho* of Eq. (9) with a known lambda and reference rho_sft. The experiments instead train the prompter with PPO using a separate reward model (RM-Gemma-2B); the value of lambda in Eq. (9) is never reported, and no convergence or approximation argument shows that the PPO-trained prompter approaches rho*. Consequently, the win rates in Table 1 and the mean/variance results in Figures 2 and 3 do not test the theoretical claim. The authors should either train with the Eq. (9) objective and report lambda, or provide a quantitative bound connecting the PPO solution to rho*.","section":"Section 7.1"},{"comment":"The negative KL term in Eq. (11) is imported directly from the optimality condition used to define rho* (Appendix B, Eq. (32)). It is therefore a restatement of the prompter's own objective rather than an independently measured quantity. The text should say this explicitly; as written, the interpretation that prompt optimization 'reduces the suboptimality bound' through the third term may mislead readers into thinking the bound relies on an external improvement rather than on the deliberate choice of the KL-regularized objective in Eq. (9).","section":"Equation (11) and discussion following Theorem 6.1"}],"minor_comments":[{"comment":"The proof of Lemma 5.1 differentiates the objective without enforcing the normalization constraint sum_{x'} rho(x'|x)=1, then normalizes the resulting expression; the final formula is correct, but the derivation should use a Lagrangian multiplier or cite the standard KL-regularized optimization argument.","section":"Appendix A, Eqs. (16)-(19)"},{"comment":"The text lists both F1 and F2 as Llama-3.1-8B-Instruct, but Table 1 and Figures 2-3 indicate that the second frozen model is Qwen-2.5-7B-Instruct; please correct this typo.","section":"Section 7.1, first paragraph"},{"comment":"Equation (4) uses pi_F as the reference policy in the closed-form RLHF solution, while Eq. (2) defines the KL penalty relative to pi_sft; please clarify whether pi_F = pi_sft or redefine the reference appropriately.","section":"Equation (4) versus Eq. (2)"},{"comment":"Win rates are computed on 100 samples without confidence intervals or significance tests; given the small sample, the claim of consistent improvement should be accompanied by error bars or a statistical test.","section":"Table 1"},{"comment":"The term 'log partition function' for Z(x) is a misnomer; Z(x) is the partition function, while log Z(x) is the log-partition function.","section":"Lemma 5.1, statement"}],"recommendation":"major_revision","confidential_remarks":"The central theorem is repairable with a local change (reward-range assumption or reward shifting), and the contribution is potentially suitable for the journal. However, the experimental section needs to be repositioned as exploratory proof-of-concept unless the PPO-trained prompter is connected to the closed-form rho*; at minimum, the PPO objective and lambda should be reported. I am not asking for new large-scale experiments, but the theory-experiment gap must be acknowledged and addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the central theoretical claim has a missing assumption. The proof of Theorem 6.1 bounds the TV terms by rmax·d_TV, which is only valid when r* is nonnegative. For signed rewards, the sharp constant is (sup r − inf r), and the stated bound fails. I checked the stress-test counterexample: two prompt states, symmetric rho_sft, rewards +1 and −1, lambda=1. The RHS of Eq. (11) comes out to 0.172, but the actual suboptimality gap is 0.238. So the theorem is not merely loose; it is false for the general bounded-reward setting it claims. The fix is easy: either state r* ∈ [0, r_max] explicitly or use 2r_max in the TV bounds. But this is load-bearing, because the 'budget' interpretation in Section 6 relies on the negative KL term being enough to offset the first two terms. With the wrong constant, that interpretation is not supported.\n\nWhat is genuinely new and useful: the framing of prompt optimization as KL-regularized reward maximization against a frozen model, and the decomposition of the suboptimality gap into frozen-model quality, reference-prompt sensitivity, and prompter KL. The closed-form optimal prompter in Eq. (10) is the standard exponential-tilt solution, which the authors acknowledge; the decomposition in Theorem 6.1 is new, even though it follows quickly from standard TV bounds once the correct assumption is in place. The proof of Lemma 5.1 is algebraically correct.\n\nThe experiments are proof-of-concept, not a test of the theorem. The prompter is trained with PPO against a proxy reward model, no lambda is reported, and there is no argument that the trained prompter converges to the closed-form rho*. Win rates are computed on 100 samples without error bars or released code. There is also a naming inconsistency where F1 and F2 are both described as Llama-3.1-8B-Instruct, though the table implies one is Qwen-2.5-7B. These are real but secondary.\n\nBottom line: with the reward nonnegativity assumption added, this is a modest but publishable contribution. Without it, the main theorem is technically false. I would send it to a serious referee, with a clear request to fix the theorem statement and to reframe the experiments as an illustration rather than a validation of the bound.","headline":"The framework is sensible and the decomposition is new, but Theorem 6.1 is false as stated without a nonnegativity assumption on the reward; the experiments don't test the theorem anyway.","tokens_in":17593,"tokens_out":2663,"would_cite":false,"duration_ms":23298,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68T50"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proves a quantitative upper bound on the alignment gap left by prompt optimization of a frozen language model, and supplies a closed-form optimal prompter.","keywords":["prompt optimization","LLM alignment","RLHF","frozen language model","suboptimality bound","total variation distance","KL-regularized optimization","closed-form optimal prompter"],"falsifier":"Train the prompter on a single dataset as in Section 7.1, record the realized $D_{\\mathrm{KL}}(\\rho_{\\mathrm{ppo}}\\|\\rho_{\\mathrm{sft}})$, estimate the two total-variation terms from samples, and compare the measured gap $J(\\pi^*) - J(\\tilde{\\pi}_{\\rho_{\\mathrm{ppo}}})$ with the right-hand side of Theorem 6.1. A violation of the inequality, or a large mismatch between the PPO-trained prompter and the closed-form $\\rho^*$ computed from Eq. (10), would show the experiments do not test the theorem.","tokens_in":16473,"feed_emoji":"🎯","tokens_out":8665,"duration_ms":70018,"temperature":0.7,"pith_summary":"This paper tries to establish that prompt optimization is a theoretically grounded alternative to RLHF for aligning language models whose weights cannot be changed. Treating the prompter as a trainable distribution $\\rho(x'|x)$ in front of a frozen policy $\\pi_F$, it derives the closed-form optimal prompter and then bounds the suboptimality gap $J(\\pi^*) - J(\\tilde{\\pi}_{\\rho^*})$ relative to the RLHF-optimal policy. The bound splits the gap into three readable pieces: how far the frozen model sits from $\\pi^*$, how much the reference prompter moves the frozen model's output distribution, and the KL divergence budget the optimal prompter spends. If this is right, alignment can be pursued, measured, and partly closed on frozen or black-box models instead of by parameter fine-tuning.","feed_headline":"Prompt optimization gets a provable performance bound vs RLHF","feed_subtitle":"A closed-form optimal prompter splits the alignment gap into frozen-model error, prompt sensitivity, and KL budget.","key_machinery":"The load-bearing object is the closed-form optimal prompter of Lemma 5.1, a Gibbs-style distribution that reweights the reference prompter by the exponentiated expected reward of the frozen model on the rewritten prompt. It is what makes the suboptimality bound in Theorem 6.1 possible: the optimality inequality for $\\rho^*$ converts the prompter's contribution into a negative KL term, so the bound shrinks exactly as the prompter spends KL budget to move away from $\\rho_{\\mathrm{sft}}$. The remaining two terms are total-variation distances that capture, respectively, the baseline quality of $\\pi_F$ relative to $\\pi^*$ and the sensitivity of $\\pi_F$ to prompt changes under $\\rho_{\\mathrm{sft}}$.","core_discovery":"Align-Pro's central assertion is that the best prompter for a frozen model is the solution to a KL-regularized reward maximization, and that this prompter makes the RLHF gap quantitative rather than empirical. Concretely, Lemma 5.1 gives $\\rho^*(x'|x) \\propto \\rho_{\\mathrm{sft}}(x'|x)\\exp(R(x,x')/\\lambda)$, where $R(x,x')$ is the expected reward of the frozen model on the rewritten prompt, and Theorem 6.1 bounds the gap by $r_{\\max}\\mathbb{E}[d_{\\mathrm{TV}}(\\pi^*,\\pi_F)] + r_{\\max}\\mathbb{E}[d_{\\mathrm{TV}}(\\pi_F(\\cdot|x),\\pi_F(\\cdot|x'))] - \\lambda \\mathbb{E}[D_{\\mathrm{KL}}(\\rho^*\\|\\rho_{\\mathrm{sft}})]$. The paper claims this decomposition shows exactly where prompt optimization loses to fine-tuning: in the frozen model's intrinsic misalignment and in the reference prompter's destabilizing rewrites, offset by the KL budget that the optimized prompter is allowed to spend. The experiments with PPO-trained prompters on three datasets are presented as proof of concept that a real prompter can realize the effect and beat the no-fine-tuning baseline.","pith_inferences":["The bound turns $\\lambda$ into a measurable alignment budget: one could estimate the two total-variation terms from data and choose $\\lambda$ so that the negative KL term offsets them, which the paper leaves implicit.","The theorem assumes the same reward $r^*$ defines both $\\pi^*$ and the prompter objective; a natural stress test is to train the prompter with one reward and evaluate with another, probing whether the inequality survives reward misspecification.","The prompt-sensitivity term predicts that a low-variance reference prompter, or an ensemble of prompters whose rewrites cancel, should shrink the second term; comparing Align-Pro's gap under different reference prompters would test that prediction."],"forward_implications":["If the frozen model already equals the RLHF-optimal policy, the first total-variation term vanishes and the bound reduces to prompt sensitivity minus KL budget.","If the reference prompter is a delta distribution, the prompt-sensitivity term vanishes, isolating the frozen-model quality and the KL budget.","Because the third term is negative, spending KL divergence budget on the prompter can only shrink the worst-case gap, giving a quantitative rationale for prompt engineering on frozen models.","The win-rate and mean-reward experiments on three datasets are offered as evidence that a PPO-trained prompter realizes this effect, beating no fine-tuning and approaching the RLHF oracle."],"supporting_citations":[{"why":"Supplies the derivation technique used to obtain the closed-form optimal prompter in Lemma 5.1.","marker":"[48]"},{"why":"Gives the reward-weighted regression form of the RLHF optimal policy used as the benchmark $\\pi^*$.","marker":"[46,47]"},{"why":"Defines the Bradley-Terry preference model underlying the reward $r^*$ in the alignment objective.","marker":"[44]"},{"why":"Provides the empirical prompt-rewriting-with-RL baseline that motivates the Align-Pro formulation.","marker":"[15]"},{"why":"Supplies the UltraFeedback dataset used to evaluate mean reward and win rate.","marker":"[49]"},{"why":"Supplies the HelpSteer dataset used in the experiments.","marker":"[50]"},{"why":"Supplies the Orca dataset used in the experiments.","marker":"[51]"}],"fun_headline_variants":["Prompt optimization gets a provable bound vs RLHF","Closed-form optimal prompter reveals alignment gap for frozen LLMs","Theory: how good prompts can align frozen LLMs without fine-tuning","Prompt optimization: provable bound on alignment gap vs RLHF","Optimal prompter formula splits alignment gap into three parts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The bound and the experiments both rest on the assumption that the PPO-trained prompter described in Section 7.1 converges to the closed-form $\\rho^*$ of Eq. (10), and that the reward used in training is the same $r^*$ defining $\\pi^*$; the paper does not report $\\lambda$ or verify this convergence.","fun_headline_variants_meta":{"raw":{"variants":["Prompt optimization gets a provable bound vs RLHF","Closed-form optimal prompter reveals alignment gap for frozen LLMs","Theory: how good prompts can align frozen LLMs without fine-tuning","Prompt optimization: provable bound on alignment gap vs RLHF","Optimal prompter formula splits alignment gap into three parts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000235,"raw_usage":{"total_tokens":1523,"prompt_tokens":991,"completion_tokens":532,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":446}},"tokens_in":607,"tokens_out":532,"duration_ms":4976,"temperature":1.0,"reasoning_tokens":446,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:53:07.065658+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the prompter on a single dataset as in Section 7.1, record the realized $D_{\\mathrm{KL}}(\\rho_{\\mathrm{ppo}}\\|\\rho_{\\mathrm{sft}})$, estimate the two total-variation terms from samples, and compare the measured gap $J(\\pi^*) - J(\\tilde{\\pi}_{\\rho_{\\mathrm{ppo}}})$ with the right-hand side of Theorem 6.1. A violation of the inequality, or a large mismatch between the PPO-trained prompter and the closed-form $\\rho^*$ computed from Eq. (10), would show the experiments do not test the theorem.","supporting_citations":[{"cited_title":"Direct preference optimization: Your language model is secretly a reward model","cited_arxiv_id":null,"evidence_quote":"Supplies the derivation technique used to obtain the closed-form optimal prompter in Lemma 5.1."},{"cited_title":"Rank analysis of incomplete block designs: I","cited_arxiv_id":null,"evidence_quote":"Defines the Bradley-Terry preference model underlying the reward $r^*$ in the alignment objective."},{"cited_title":"PRewrite: Prompt Rewriting with Reinforcement Learning","cited_arxiv_id":null,"evidence_quote":"Provides the empirical prompt-rewriting-with-RL baseline that motivates the Align-Pro formulation."},{"cited_title":"ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback","cited_arxiv_id":null,"evidence_quote":"Supplies the UltraFeedback dataset used to evaluate mean reward and win rate."},{"cited_title":"Helpsteer: Multi-attribute helpfulness dataset for steerlm","cited_arxiv_id":null,"evidence_quote":"Supplies the HelpSteer dataset used in the experiments."},{"cited_title":"Orca: Progressive learning from complex explanation traces of gpt-4","cited_arxiv_id":null,"evidence_quote":"Supplies the Orca dataset used in the experiments."}],"review_version":1}