{"id":"b7c4a78f-d9fc-4e8c-9d00-93a0f1db4e38","arxiv_id":"2506.17828","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"IRO trains a sequence of small value functions that reweight a frozen LLM's own samples, achieving test-time alignment gains that outperform Best-of-N search on AlpacaEval 2.0 and TL;DR.","lead":"IRO is a method for aligning a frozen large language model by training small guide models that score partial outputs, without changing the big model's weights. It matters because it offers an API-friendly way to customize models, similar to reinforcement fine-tuning, while claiming strong test-time gains with lower generation cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 2's value-only chunk scoring does not implement the softmax policy of Eq. (10), so Theorem 1's convergence guarantee does not apply to the deployed decoder.","rationale":"I read the paper in good faith. The proposed method is coherent and the experiments, while single-run, show consistent improvements over BoN and other baselines. The theoretical derivations are standard TRPO-style arguments, and Proposition 1 is a reasonable ideal-case analysis. However, the load-bearing weakness is the disconnect between the theory and the implemented decoder. Theorem 1's policy update (Eq. 10) is a token-level softmax that includes log π_base, whereas Algorithm 2 scores whole chunks with a value-only sum. This is not a minor implementation detail: the convergence guarantee is for a different policy. The paper's own Remark 4.1 concedes that learned value functions are imperfect and that reweighting is over a sampled subset, further weakening the link. Proposition 1's exponential-efficiency claim additionally assumes exact V* and uniform subset sampling, which are strong 'ideal' conditions, not the 'mild' conditions stated in the abstract. The reader's weakest assumption identified the log π_base omission, and I agree; I would only broaden it to include the chunk-level scoring mismatch. The concern is addressable by restating the theory or modifying the decoder, and it does not invalidate the empirical findings, so the conditionality of the reader's verdict is appropriate.","tokens_in":47722,"tokens_out":14279,"duration_ms":151240,"concrete_test":"Modify Algorithm 2's score to include the log-policy term and to accumulate per-token values: score(y_j) = log π_base(y_j|x) + Σ_{h in chunk} Σ_{i=0}^{t-1} (1/β_i) V̂_i(s_{h+1}), reusing the same value functions and hyperparameters. Rerun the AlpacaEval 2.0 200-sample subset and the TL;DR experiments for iterations 1-3, comparing LC win rates against BoN-E with the same evaluation protocol. If the LC win rates shift by more than about 1 point, the omitted log π_base term and chunk-level scoring materially change search behavior, confirming that the current theorems do not describe the reported results; if they are statistically unchanged, the omission is benign in practice, but the theoretical claims should still be restated to cover the value-only decoder.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central theoretical claim is that IRO is policy iteration converging to the optimal policy, and that it matches Best-of-N with exponentially fewer tokens. Theorem 1 is proved for the token-level softmax policy in Eq. (10)/(33): π̂_t(a|s) ∝ π̂_{t-1}(a|s) exp( V̂_{π̂_{t-1}}(s,a)/β_{t-1}) ∝ π_base(a|s) exp(Σ_i V̂_{π̂_i}(s,a)/β_i). The update explicitly includes the log-policy term log π_base. However, the deployed decoder (Algorithm 2) scores each candidate chunk as V(y_j) = Σ_i (1/β_i) V̂_i(x,y_j), with no log π_base term, and it scores a whole chunk by the value at the end of the chunk rather than by summing per-token values. Appendix G.1.5 confirms that log π_base is deliberately omitted. Consequently, Algorithm 2 is not sampling from, or optimizing, the policy π̂_t of Eq. (10); it is a separate heuristic. Theorem 1's convergence and regret bounds therefore do not cover the algorithm whose results are reported. Proposition 1 analyzes a greedy value-based decoder, but it assumes (i) the weighted sum of value functions equals V* exactly, and (ii) each step samples U candidates uniformly from the full action space A^L. The implementation samples candidates from π_base, not uniformly, so even Proposition 1's cost ratios are not derived for the actual candidate-generation process. The result is that neither theorem covers the implementation: Theorem 1 covers a token-level softmax policy that is not decoded, while Proposition 1 covers an idealized uniform-proposal, perfect-value decoder. The empirical win rates may be valid, but the theoretical support for the headline claim is not connected to the shipped method.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Iterative Reweight-then-Optimize (IRO), a test-time alignment method that leaves the base LLM frozen and instead trains a sequence of lightweight value functions, each regressed on Monte Carlo returns from data generated by the previous reweighted policy. At test time, these value functions guide a beam-search-style decoder over the frozen base policy, and the final response is selected by an outcome reward model. The authors claim that IRO is a form of policy iteration converging to the optimal policy (Theorem 1) and that, under idealized conditions, it matches Best-of-N performance with exponentially fewer tokens and reward queries (Proposition 1). Experiments on TL;DR summarization and AlpacaEval 2.0 show consistent win-rate gains over BoN, ARGS, and weak-to-strong search when using 1B or 7B value models to guide 1B, 6.9B, 8B, or 70B base models.","tokens_in":48037,"tokens_out":3590,"duration_ms":38044,"significance":"If the theoretical claims held for the implemented algorithm, the paper would make a substantial contribution: it would provide the first test-time alignment method that provably performs policy iteration on a frozen model, with a quantitative efficiency argument against Best-of-N, and it would open the door to API-only reinforcement fine-tuning. The empirical results are extensive and encouraging, including weak-to-strong generalization and ablations over chunk length, data size, and search budget, and the authors provide a code link. However, the paper's central theoretical statements are currently proved for a softmax policy that the implemented decoder does not run, and the efficiency proposition relies on assumptions about uniform candidate sampling and exact value-function approximation that are not satisfied by the deployment. The convergence and exponential-efficiency claims therefore do not yet cover the algorithm whose results are reported.","major_comments":[{"comment":"Theorem 1 is stated and proved for the token-level softmax policy in Eq. (10) and Eq. (33), namely π̂_t(a|s) ∝ π̂_{t-1}(a|s) exp(V̂_{π̂_{t-1}}(s,a)/β_{t-1}) ∝ π_base(a|s) exp(Σ_i V̂_{π̂_i}(s,a)/β_i). This update explicitly contains the log-policy term log π_base. The implemented decoder in Algorithm 2, however, scores each chunk by V(y_j) = Σ_i (1/β_i) V̂_i(x,y_j) with no log π_base term and then deterministically keeps the top-K beams, rather than sampling from the softmax policy. Appendix G.1.5 confirms that the log π_t term is deliberately omitted during scoring. Consequently, Theorem 1's convergence and regret bounds do not apply to the deployed search procedure, and the reported empirical gains are not protected by the paper's main theoretical guarantee.","section":"§4.1, Theorem 1"},{"comment":"Assumption 2 posits concentrability of π_base to π* (d^{π*}_h/d^{π_base}_h ≤ C_ST), but the value functions in Eq. (9) are fitted on data collected under the sequence of policies π̂_t. The proof of Theorem 1 needs a distribution-mismatch bound between d^{π̂_t}_h and d^{π_base}_h (or between d^{π̂_t}_h and d^{π*}_h) to control the ϵ2(t) term in Eq. (34). In Eq. (44) the bound is written as C_ST E_{s∼d^{π_1}_h}(...), which is not justified by Assumption 2 as stated, since π_1 (or π̂_t) is not π_base. As written, the estimation-error term in the theorem is not rigorously controlled.","section":"§4.1, Assumption 2"},{"comment":"Proposition 1's cost ratios in Eq. (14) are derived under two modeling assumptions that the implementation does not satisfy. First, the proof in Appendix B.2 assumes that at each chunk step the algorithm samples U candidates uniformly from the full action space A^L (see Eq. (24)), while Algorithm 2 samples successors from the base policy π_base and uses beam selection, not uniform proposals. Second, the proof computes BoN's success probability as N/|A|^H in Eq. (21), which holds only for uniform action draws; for a non-uniform base policy the probability of sampling the optimal trajectory under Best-of-N depends on π_base's probability mass on τ*, and the ratio in Eq. (14) does not follow. The exponential-efficiency claim therefore has not been established for the actual candidate-generation process.","section":"§4.2, Proposition 1"},{"comment":"Proposition 1 takes as input the condition that the weighted sum of the I learned value functions approximates V*, i.e. Σ_{i=0}^{I-1} V_i/β_i ≈ V*. This is precisely the object that the training procedure (Eq. (9)) is supposed to produce, so the exponential-efficiency claim is conditional on the training having already succeeded, rather than derived from the stated value-learning objective with its finite-sample error. Remark 4.1 concedes that with a small number of iterations the value functions may not approximate V* well, and that reweighting over a sampled subset adds approximation error. The proposition's conclusion therefore holds only under an idealized condition that the paper does not show is met by the empirical setup.","section":"§4.2, Proposition 1"},{"comment":"Algorithm 2's pseudocode describes only value-based beam selection (line 6) and does not include the diversity-first principle introduced in Remark 3.1 or implemented in the experiments. Since the ablation in Fig. 7 shows that diversity-first is important for the reported gains, the pseudocode does not fully specify the evaluated algorithm. The paper should either add the clustering/diversity step to Algorithm 2 or state in the pseudocode that diversity-first is applied as a separate search heuristic.","section":"§3, Algorithm 2"}],"minor_comments":[{"comment":"The index range in Eq. (10) is written as 't = 0, 2, ···', which should presumably be 't = 0, 1, 2, ···'.","section":"§4.1, Eq. (10)"},{"comment":"Algorithm 1 returns 'π̂_T constructed according to (3)', but (3) defines the optimal value function V*, not a policy; the reference should presumably be to Eq. (10).","section":"§3, Algorithm 1"},{"comment":"The statement 'we omit the term log π_t(a_h|s_h) when scoring the candidates, as it is already revealed during candidate generation' is misleading: in beam search, the log-probability of a candidate is not a constant across candidates and affects their relative ranking, so omitting it changes the induced policy. This point should be discussed explicitly.","section":"Appendix G.1.5"},{"comment":"The heading 'Example 1 in the TL;DR Taks' contains a typo: 'Taks' should be 'Task'.","section":"Appendix H"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising idea and a thorough empirical study, but the theory section and the implementation are not yet aligned. The main risk for the journal is that a reader will take the convergence and exponential-efficiency claims at face value even though they are proved for a softmax policy and an idealized uniform-proposal decoder that are not what the experiments run. A revision that either modifies the decoder to include the log-policy term (making it match the theory) or else re-states the theoretical results as applying to a modified objective with explicit approximation terms for the beam-search mismatch would substantially strengthen the paper. I would also ask the authors to clarify the concentrability assumption in the proof of Theorem 1."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper: the empirical work is solid and the iterative value-reweighting idea is genuinely new, but the theory as written does not cover the algorithm that actually runs. If you read it for the method and the experiments, you'll be satisfied. If you read it for the guarantees, you'll be frustrated.\n\nWhat I like: training a sequence of value functions on self-generated, reweighted data and then using that sequence to guide a frozen base policy via beam search is a clean, practical contribution. The API-only deployment angle is real, and the gains over BoN and weak-to-strong search on TL;DR and AlpacaEval look credible. The ablations on chunk length, beta schedule, and search budget are useful. The weak-to-strong results (1B guiding 6.9B, 7B guiding 70B) are a nice demonstration.\n\nThe soft spot: the theory and the decoder are different objects. Theorem 1 proves convergence for the token-level softmax policy pi_t proportional to pi_base times exp(sum of value functions), but Algorithm 2 scores whole chunks with only the value sum and deliberately omits log pi_base, as Appendix G.1.5 confirms. So the polynomial-time convergence guarantee does not apply to the deployed heuristic. Proposition 1's exponential-efficiency claim assumes the weighted value sum is exactly V* and that candidates are sampled uniformly over the action space, which the implementation does not do. The authors call these 'ideal conditions,' but the gap between the analyzed object and the shipped method is wide. There is also a concentrability mismatch: Theorem 1 assumes pi_base covers pi*, but the data is collected under pi_t, and the proof does not cleanly bridge that.\n\nMinor issues: the headline win rates are single-run point estimates without error bars, and the code link has no commit hash. These are addressable.\n\nOverall, the empirical method likely works as advertised, and the iterative-reweighting idea is worth pursuing. The theory needs to be either reworked to analyze the actual beam-search decoder or the decoder should be changed to match the theoretical policy. That is a significant revision, but not a fatal flaw.\n\nI would send this to peer review with the expectation of major revision. It is a serious paper with a real contribution, but the current form overstates what is proven.","headline":"IRO is a promising empirical method, but the theory is disconnected from the implemented decoder; the paper deserves serious review with revisions to close that gap.","tokens_in":48675,"tokens_out":2734,"would_cite":true,"duration_ms":28943,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","90C40"],"pacs":[],"model":"deepseek-v4-flash","headline":"IRO converges to the optimal aligned policy for a frozen LLM by iteratively training and applying small value functions, matching Best-of-N quality with exponentially fewer test-time tokens.","keywords":["test-time alignment","frozen LLMs","reinforcement learning","value functions","iterative reweighting","Best-of-N","token-level MDP","beam search"],"falsifier":"On a synthetic token-level MDP with a known optimal policy and value function, run Algorithm 2 exactly as implemented and count how often the returned continuation is optimal; compare this with the predicted probability $(U/|\\mathcal{A}|)^{H/L}$ and with Best-of-N's success rate at $N=U^{H/L}$. If the implemented scorer, which omits the base-model likelihood term, succeeds materially below the idealized reweighted policy, the exponential-efficiency claim is tied to the update in Eq. (10) rather than to the deployed decoder.","tokens_in":47454,"feed_emoji":"🎯","tokens_out":10785,"duration_ms":104251,"temperature":0.7,"pith_summary":"IRO, Iterative Reweight-then-Optimize, treats the alignment of a frozen language model as an RL problem whose solution can be written as the base policy reweighted by the exponentials of a sequence of value functions. The paper argues that training one small value model after another on self-generated completions is a policy-iteration loop: each new value function reflects the current reweighted policy, and applying the whole sequence at decoding time yields successively better aligned outputs without ever modifying the base model's weights. Under standard concentrability and function-class assumptions, IRO converges to the optimal policy; under an idealized condition in which the learned value functions approximate the optimal value function, it reaches the same success probability as Best-of-N with exponentially fewer generated tokens and reward/value queries. If correct, this would let a user align any frozen or API-only model to a private reward signal using only small value models and a reward model, which is what the paper's AlpacaEval 2.0 results (length-controlled win rates rising from 30.7% to 43.8% on an 8B base model, and from 43.1% to 49.8% on a 70B base model) are offered as evidence for.","feed_headline":"Value-guided rewrites lift frozen LLM win rate from 30.7% to 43.8%","feed_subtitle":"Iterative reweight-then-optimize aligns a frozen model at test time, matching Best-of-N with exponentially fewer tokens.","key_machinery":"The load-bearing object is the iterated reweighting identity $\\pi_t(a\\mid s)\\propto \\pi_{\\mathrm{base}}(a\\mid s)\\exp\\left(\\sum_{i=0}^{t-1}\\hat V_{\\pi_i}(s,a)/\\beta_i\\right)$, which is the closed-form solution of a KL-constrained surrogate objective in the style of trust-region policy optimization and turns policy improvement into value-function learning. Each step regresses a lightweight value model to the terminal rewards of completions drawn from the current reweighted policy; at test time, the same value models score chunks during a value-guided beam search, a diversity-first rule keeps redundant candidates from crowding out diverse high-value prefixes, and the reward model picks the final continuation. This identity is what connects a purely weight-free decoding procedure to the theory of policy iteration.","core_discovery":"The paper's central claim is that the optimal aligned policy over a frozen base model has the explicit form $\\pi_t(a\\mid s)\\propto \\pi_{\\mathrm{base}}(a\\mid s)\\exp\\left(\\sum_{i=0}^{t-1}\\hat V_{\\pi_i}(s,a)/\\beta_i\\right)$, and that IRO realizes this form by alternating regression of a value function to Monte Carlo returns with reweighted guided decoding. Theorem 1 states that, with bounded value-function classes and a concentrability assumption, choosing $\\beta_t=\\sqrt{t+1}/\\omega$ makes the gap to the optimal policy shrink as $O(T^{-1/2})$ plus a least-squares estimation term, so IRO is a policy-iteration method. Proposition 1 states that if the accumulated value functions are close to the optimal value function $V^*$, then IRO with beam width $U$ has success probability $(U/|\\mathcal{A}|)^{H/L}$; matching Best-of-N forces $U=N^{1/H}$, giving token-cost and query-cost ratios $(BK)^{H/L-1}$ and $(L/HI)(BK)^{H/L-1}$, respectively. The paper reads these bounds as: Best-of-N quality can be obtained with exponentially fewer tokens at test time.","pith_inferences":["Because the theorems are proven for the full reweighted policy in Eq. (10) while the deployed decoder scores candidates with only the value-function sum, a direct ablation that re-adds the base model's own likelihood term would isolate whether the implemented decoder is the algorithm being analyzed.","The same reweighting identity suggests an online variant: continue training value functions on reward feedback collected during deployment, turning IRO into a continually adapting alignment loop for a model whose weights remain hidden.","The diversity-first principle implies that candidate redundancy, not just value accuracy, limits greedy guided search; a controlled test of clustering-aware selection with the same value functions would quantify that contribution separately from the iterative value updates."],"forward_implications":["A frozen or deployed model can be re-aligned to a new reward signal without weight access, since only small value models are trained by regression on sampled completions.","Alignment improves across iterations: later value functions correct biases left by earlier ones, rather than relying on a single imperfect guidance pass.","At equal success probability, the token cost of IRO relative to Best-of-N scales like $(BK)^{H/L-1}$, so the advantage grows as the generation horizon lengthens.","Small value models can guide much larger base models (1B guiding 6.9B and 7B guiding 70B in the paper), which is what makes the frozen-model setting practical."],"supporting_citations":[{"why":"Supplies the KL-constrained surrogate objective whose closed-form solution is the exponential reweighting update that IRO iterates.","marker":"[38]"},{"why":"Supplies the value-function regression objective and the controlled-decoding idea used to train and apply lightweight value models.","marker":"[32]"},{"why":"Supplies the performance-difference lemma and the concentrability-based KL bound used in the proof of Theorem 1.","marker":"[9]"},{"why":"Supplies the least-squares concentration guarantee that bounds the value-function estimation error in Theorem 1.","marker":"[44]"},{"why":"Identifies the update as a natural policy gradient, the convergence context against which Theorem 1 is stated.","marker":"[24]"},{"why":"Provides the policy-gradient convergence rates that IRO's adaptive-beta analysis extends and compares with.","marker":"[46]"},{"why":"Supplies the value-guided beam-search decoding procedure that IRO adapts for its guided-generation step.","marker":"[53]"}],"fun_headline_variants":["Iterative reweight-then-optimize aligns frozen LLMs without weight updates","Test-time RL alignment for frozen LLMs: 30.7% to 43.8% win rate","IRO: Reweight-then-optimize lifts frozen LLM performance, cheaper than Best-of-N","Value-guided rewrites align frozen LLMs, matching Best-of-N with fewer tokens","Frozen LLM alignment via iterative reweighting, no weight access needed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the small learned value functions approximate the optimal value function closely enough to steer the reweighted policy, and that the deployed scorer—which omits the base model's own likelihood term—still behaves like the theoretical reweighted policy; if either gap is substantial, the convergence and exponential-efficiency guarantees do not transfer to the implemented decoder.","fun_headline_variants_meta":{"raw":{"variants":["Iterative reweight-then-optimize aligns frozen LLMs without weight updates","Test-time RL alignment for frozen LLMs: 30.7% to 43.8% win rate","IRO: Reweight-then-optimize lifts frozen LLM performance, cheaper than Best-of-N","Value-guided rewrites align frozen LLMs, matching Best-of-N with fewer tokens","Frozen LLM alignment via iterative reweighting, no weight access needed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000805,"raw_usage":{"total_tokens":3590,"prompt_tokens":1052,"completion_tokens":2538,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":668,"completion_tokens_details":{"reasoning_tokens":2422}},"tokens_in":668,"tokens_out":2538,"duration_ms":17786,"temperature":1.0,"reasoning_tokens":2422,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:00:48.190428+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a synthetic token-level MDP with a known optimal policy and value function, run Algorithm 2 exactly as implemented and count how often the returned continuation is optimal; compare this with the predicted probability $(U/|\\mathcal{A}|)^{H/L}$ and with Best-of-N's success rate at $N=U^{H/L}$. If the implemented scorer, which omits the base-model likelihood term, succeeds materially below the idealized reweighted policy, the exponential-efficiency claim is tied to the update in Eq. (10) rather than to the deployed decoder.","supporting_citations":[],"review_version":2}