{"id":"7c4ba7ec-03bd-42ac-a945-b8c059b96440","arxiv_id":"2501.18099","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"EvalPlanner trains LLM judges to separate planning from execution via iterative DPO on synthetic chains of thought, reaching 93.9 on RewardBench with only 22K synthetic preference pairs.","lead":"This paper introduces EvalPlanner, a training method that teaches an LLM judge to first draft an evaluation plan, then execute it step by step, and only then deliver a verdict. It reports state-of-the-art reward-model scores while using far fewer, fully synthetic preference pairs than prior models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The RewardBench/PPE SOTA claim rests on unexamined overlap between MATH training prompts (Section 3.1) and the math portions of both evaluation benchmarks; no decontamination is reported, so the headline scores are conditional on leakage being absent.","rationale":"The method itself is well-structured: decoupling planning from execution, constructing synthetic preference pairs from verdict correctness, and applying iterative DPO are plausible, and the ablations in Tables 8 through 11 support the mechanism. My concern is not internal inconsistency; it is an unexamined train/evaluation overlap that specifically targets the categories driving the headline gains. The single most consequential instance is RewardBench Reasoning, and Table 11 shows MATH training is the ingredient that boosts that category. PPE Correctness adds a second, even more direct collision because one of its columns is literally MATH. The central data-efficiency/SOTA claim is therefore only as strong as the decontamination assumption. This is checkable with public artifacts; if the authors release the 4,141 training prompts and filtered evaluation scores, the concern is either resolved or confirmed. The reader's weakest assumption identified the same risk for RewardBench, which is why my agreement is partial rather than full: the PPE Correctness MATH column makes the collision even more explicit. Until the decontamination check is run, CONDITIONAL is the appropriate verdict, and my read does not move the reader's verdict.","tokens_in":19685,"tokens_out":8246,"duration_ms":84595,"concrete_test":"Obtain the public RewardBench reasoning prompts and PPE Correctness MATH prompts, plus the exact 4,141 MATH training instructions used in Section 3.1 (request release if needed). Normalize by stripping whitespace, standardizing LaTeX, and removing equations; then compute exact-match and near-duplicate overlap using 10-gram Jaccard similarity or an embedding similarity threshold. Re-run EvalPlanner scoring on the non-overlapping subset and compare RewardBench overall/reasoning against 93.3/95.5 and PPE Correctness MATH against 81.7. Report both overlap counts and filtered scores; if filtering drops RewardBench below Skywork-Critic, the SOTA claim is leakage-driven, whereas if scores hold, the objection is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 trains EvalPlanner on 4,141 MATH triples (plus 17,588 WildChat triples), using correct/incorrect solutions to MATH problems as chosen/rejected responses. The two claimed SOTA results are then measured on benchmarks with explicit math components: RewardBench's Reasoning category (95.5/96.1, Table 1) and PPE Correctness's MATH column (79.2/81.7, Table 2). PPE Correctness explicitly includes a MATH column, and the training source is MATH (Hendrycks et al., 2021); the paper never shows that the two MATH item sets are disjoint. RewardBench's Reasoning category also contains mathematical problems, and no overlap analysis is provided for it either. The math categories are exactly where EvalPlanner outperforms baselines: Table 11 shows that MATH 2.5K training alone lifts RewardBench Reasoning to 93.4 versus 89.3 for WildChat 2.5K. If any training MATH questions appear in either evaluation set, the reported gains reflect memorized or near-memorized MATH content rather than the planning-and-execution recipe. Given the weight of the Reasoning category in RewardBench, removing leaked items could drop the 93.9 overall score below Skywork-Critic's 93.3. The absence of any decontamination report makes the central claim conditional.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EvalPlanner, a preference-optimization method for training LLM-as-a-Judge models. The judge is trained to generate (1) an unconstrained evaluation plan for the input instruction, (2) a step-by-step execution of that plan over the candidate responses, and (3) a final pairwise verdict. Training data are constructed synthetically: response pairs come from Self-Taught Evaluators (WildChat) and from correct/incorrect MATH solutions; plans and executions are sampled from the current model; and correct/incorrect (plan, execution, verdict) triples are used as chosen/rejected pairs. The model is trained with SFT followed by two iterations of DPO. Experiments on RewardBench, PPE, RM-Bench, JudgeBench, and a new FollowBenchEval benchmark report state-of-the-art generative reward model results (e.g., 93.9 on RewardBench) with only 22K synthetic preference pairs.","tokens_in":20123,"tokens_out":6371,"duration_ms":54590,"significance":"If the results hold, EvalPlanner is a convincing demonstration that decoupling planning from execution and iteratively optimizing the resulting thoughts via DPO is a data-efficient route to strong reward models. The paper includes informative ablations (Tables 8-11) showing the value of thought-level DPO, unconstrained plans, scaling the number of plans/executions, and the complementary roles of WildChat and MATH prompts. However, the headline SOTA claim is contingent on the absence of train/test overlap between the MATH training source and the math-containing evaluation benchmarks, which the paper does not establish. The new FollowBenchEval benchmark is a useful addition, though small.","major_comments":[{"comment":"The paper trains on 4,141 MATH triples (Section 3.1) and then reports its two strongest results on benchmarks with explicit math components: RewardBench Reasoning (Table 1, 95.5/96.1) and PPE Correctness MATH (Table 2, 79.2/81.7). PPE Correctness is documented as containing MATH (Hendrycks et al., 2021) examples, and RewardBench's Reasoning category includes mathematical problems. No decontamination, exact-match, or near-duplicate analysis is reported between the MATH training prompts and the math items in these evaluation sets. Because the improvements over baselines are concentrated in the math categories (Table 11: MATH training alone gives 93.4 Reasoning vs 89.3 for WildChat), even a small overlap could account for the margin separating EvalPlanner (93.9) from Skywork-Critic (93.3) on RewardBench. The authors should report overlap statistics and, if overlap exists, re-run the evaluation on the non-overlapping subset or retrain without the overlapping prompts.","section":"§3.1, Tables 1 and 2"},{"comment":"The main SOTA claim rests on a 93.9 overall RewardBench score, which is only 0.5 points above Skywork-Critic (93.3) and 0.5 above LMUnit (93.4). The paper reports a single run per configuration and gives no confidence intervals, standard errors, or checkpoint-selection details beyond a 150-sample validation set. Given the small margin, the authors should report the variance across at least a few seeds/checkpoints or a significance test (e.g., bootstrap over RewardBench items) to support the claim that the difference is reproducible rather than within noise.","section":"§4.1, Table 1"}],"minor_comments":[{"comment":"There is a typo in the example: 'nameänd' should read 'name and', and the phrase 'Assistant B's response.' is duplicated in Step 2. The figure should be corrected before publication.","section":"Figure 1"},{"comment":"Model names are inconsistent across the paper ('Llama3.1-70B-Instruct', 'Llama-3.1-70B-Instruct', 'LLama-3.1-70B-Instruct'). Please standardize to the official naming, e.g., 'Llama-3.1-70B-Instruct'.","section":"Tables 1-7"},{"comment":"The abstract and Section 4.1 state that EvalPlanner achieves state-of-the-art on RewardBench and PPE. Table 2 shows that EvalPlanner does not exceed DeepSeek-GRM (MetaRM voting@32) on the PPE Preference subset (65.8/65.6 vs 67.2). Please qualify the claim to 'PPE Correctness' or 'PPE overall'.","section":"Abstract"},{"comment":"The definitions of D(a,b) and D(b,a) use the notation (x,a,b) -> (z, e_c, y_c) without explicitly specifying the domain of e_c and e_r or how correct/incorrect executions are selected. Clarify that e_c and e_r are executions from E(a,b) leading to correct and incorrect verdicts, respectively.","section":"§2.2"},{"comment":"FollowBenchEval is introduced as a new benchmark, but the paper does not report how the 205 samples were selected from FollowBench or release the constructed data. A reproducibility statement or a link to the data would be valuable.","section":"§3.2"},{"comment":"The text says 'Mixtral 22Bx8 Instruct model'; the standard name is 'Mixtral-8x22B-Instruct'. Please use the standard name.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"The decontamination issue is the main technical risk. Because the training data are public MATH problems and the evaluation benchmarks contain MATH items, the authors must provide a decontamination analysis before the SOTA claim can be accepted. If the overlap is substantial, the paper would need re-evaluation with a decontaminated set. The small margins over Skywork-Critic and LMUnit make this particularly important. The paper otherwise is well-written and the method is clean."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"EvalPlanner is worth reading for the recipe: generate an unconstrained evaluation plan, execute it, then DPO the whole (plan, execution, verdict) triple against synthetic positives and negatives. The ablations are the strongest part. Unconstrained plans beat criteria-based and verification-question plans (Table 9), iterative DPO adds a real jump over one iteration (92.3 to 93.9, Table 4), and the MATH-vs-WildChat source split in Table 11 is informative. The method itself is plausible and clearly described.\n\nThe soft spot is exactly the one flagged in the stress-test: no decontamination is reported between the MATH training data and the math-heavy evaluation categories. Section 3.1 trains on 4,141 MATH triples; Table 11 shows MATH prompts drive the RewardBench Reasoning score (93.4 vs 89.3 from WildChat alone). RewardBench Reasoning is partly math, and PPE Correctness has an explicit MATH column. The paper never states that the training MATH split is disjoint from the test problems, or reports n-gram overlap. If the same or similar problems appear, the 93.9 headline reflects memorized verification of MATH solutions, not the planning recipe. I would not call this fatal; the method still has value, but the central SOTA claim is conditional until the authors show the overlap numbers.\n\nI disagree with any circularity worry. The 'chosen' thoughts are defined by agreement with a synthetic preference label, but that label comes from external correctness (MATH final answers) or a noisy-instruction perturbation, and the evaluation benchmarks are independent. The self-training loop is internal and not fitted to the test sets.\n\nMinor caveat: FollowBenchEval is self-constructed, so the 13% gain there is on their own benchmark. It is a secondary result, but the caveat should be stated.\n\nBottom line: this deserves a serious referee. I would send it out with a request for a decontamination analysis and, if possible, release of training and evaluation splits. If the overlap is clean, the paper is a solid advance; if not, the reasoning numbers need recalibration. The recipe itself is a genuine contribution that I'd cite.","headline":"A solid training recipe with convincing ablations, but the RewardBench SOTA claim hinges on a missing decontamination check between MATH training data and the math-heavy evaluation sets.","tokens_in":20506,"tokens_out":2791,"would_cite":true,"duration_ms":28688,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EvalPlanner, a judge model that plans before it judges, reaches state-of-the-art 93.9 on RewardBench using only 22K synthetic preference pairs.","keywords":["LLM-as-a-Judge","reward modeling","evaluation planning","chain-of-thought","preference optimization","DPO","synthetic data","generative reward model"],"falsifier":"Compute exact and near-duplicate overlap between the 2.5K MATH prompts used in training and the RewardBench reasoning subset; if any significant portion of the reasoning benchmark matches training prompts, the reported 95.5/96.1 reasoning scores and 93.9 overall would be inflated by leakage.","tokens_in":19530,"feed_emoji":"⚖️","tokens_out":7866,"duration_ms":73137,"temperature":0.7,"pith_summary":"EvalPlanner is a training recipe for LLM judges that separates the judge's thinking into two stages: first write an evaluation plan tailored to the instruction, then execute that plan step by step on the two candidate responses before issuing a verdict. The paper's central claim is that jointly optimizing plans and executions with preference optimization, using only synthetically generated preference pairs, produces a state-of-the-art generative reward model: 93.9 on RewardBench, with stronger results on RM-Bench, PPE Correctness, and multi-constraint instruction following. This matters because prior judges hand-design the content of reasoning traces per domain and train on up to 30 times more preference data, often human-annotated. If correct, EvalPlanner shows that planning and reasoning for evaluation can be learned, not engineered, and that test-time thinking alone is enough to make a judge accurate and reliable.","feed_headline":"Judge model that plans before judging tops RewardBench at 93.9","feed_subtitle":"EvalPlanner hits state-of-the-art on RewardBench using 22K synthetic preference pairs, a fraction of rival training data.","key_machinery":"The load-bearing mechanism is the three-part Chain-of-Thought with generative process $$p_\\$\\theta$(y\\mid x,a,b)=\\sum_{z\\in P}\\sum_{e\\in E} p_\\$\\theta$(y\\mid e,z,x,a,b)\\,p_\\$\\theta$(e\\mid z,x,a,b)\\,p_\\$\\theta$(z\\mid x),$$ where $z$ is an evaluation plan (a free-form recipe, possibly including reference answers, criteria, or unit-test-like checks), $e$ is the execution of that plan on the two responses, and $y$ is the final [[A]]/[[B]] verdict. Training data is built by sampling $|P|=5$ plans and $|E|=8$ executions per instruction (with both response orders), labeling a thought correct only if it reaches the correct verdict, and forming chosen/rejected pairs that differ in plan or execution. The model is then trained with SFT on chosen thoughts followed by two iterations of DPO, the second using fresh instructions and thoughts sampled from the previous iteration's model. This lets the optimization act on both plan quality and execution fidelity simultaneously, which the paper argues is what prior hand-designed criteria or verification-question methods cannot do.","core_discovery":"On the paper's own terms, the discovery is that an LLM-as-a-Judge trained to generate a Chain-of-Thought with three explicit components—an unconstrained evaluation plan $z$, a plan execution $e$, and a final verdict $y$—and iteratively optimized with DPO on correct-versus-incorrect (plan, execution, verdict) triples, outperforms all prior generative reward models on RewardBench (93.9 with a Llama-3.1-70B seed) while using only 22K synthetic preference pairs. The authors attribute the gains to the decoupling of planning from reasoning: the model learns both what to check and how to check it, rather than entangling the two in a single hand-structured rationale. They further report that a second DPO iteration on fresh instructions is what converts a competitive result (92.3 with 5K pairs) into the state of the art, and that the same recipe generalizes across two seed models and to benchmarks requiring objective correctness judgments.","pith_inferences":["If the plan–execution decoupling is the active ingredient, the same structure could be grafted onto scalar or verifier-style reward models, where the plan acts as a generated test harness rather than a free-text rationale.","A transfer experiment that fixes a reference plan and varies only executions would separate plan improvement from execution improvement; the current ablations vary both together.","Because MATH prompts are used for training and RewardBench's reasoning category contains math problems, a decontamination study is the first test that should accompany the state-of-the-art claim.","Plans generated for coding questions already resemble unit tests, so EvalPlanner points toward a unified treatment of evaluation planning and test-generation for verifiable tasks."],"forward_implications":["Generative reward models can be trained without human-annotated preferences or rationales; the only requirements are seed-model instructions and a way to label response pairs correct or incorrect.","A single judge model can cover chat, safety, code, math, and multi-constraint instruction following because the plan is generated per instruction rather than specified by a human per domain.","Iterative self-improvement is the active scaling axis: two DPO iterations on 22K pairs (93.9) beat one iteration on the same total data (92.5), so additional iterations are the natural next step.","Planning-based judges are less vulnerable to position bias, subtle content changes, and style biases, as shown by gains on RM-Bench and position-consistent FollowBenchEval."],"supporting_citations":[{"why":"Supplies the synthetic response-pair generation method (noisy-instruction modification) that EvalPlanner reuses for WildChat prompts, and is the main generative-reward-model baseline on RewardBench.","marker":"(Wang et al., 2024c)"},{"why":"Provides the Direct Preference Optimization objective used to contrast correct and incorrect plan–execution–verdict chains.","marker":"(Rafailov et al., 2024)"},{"why":"Defines the pairwise LLM-as-a-Judge setup, the judgment prompt template, and the handling of position bias that EvalPlanner adopts.","marker":"(Zheng et al., 2023)"},{"why":"Defines RewardBench, the primary benchmark whose 93.9 overall score is the paper's headline state-of-the-art claim.","marker":"(Lambert et al., 2024)"},{"why":"Supplies the MATH prompts used for training the reasoning half of EvalPlanner; these prompts overlap in subject matter with RewardBench's reasoning category, making it the key leakage sensitivity.","marker":"(Hendrycks et al., 2021)"},{"why":"Supplies WildChat instructions used for the general instruction-following training half.","marker":"(Zhao et al., 2024)"},{"why":"Defines the PPE benchmark used as a second state-of-the-art evaluation for the method.","marker":"(Frick et al., 2025)"}],"fun_headline_variants":["Judge plans before verdict, hits 93.9","Plan-first evaluator tops at 93.9","Synthetic data, clear plans: judge 93.9","EvalPlanner splits plan from reasoning for 93.9","Decouple plan and judge: SOTA 93.9"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim depends on the RewardBench reasoning test problems not overlapping with the MATH problems used for training, since the paper reports no decontamination check.","fun_headline_variants_meta":{"raw":{"variants":["Judge plans before verdict, hits 93.9","Plan-first evaluator tops at 93.9","Synthetic data, clear plans: judge 93.9","EvalPlanner splits plan from reasoning for 93.9","Decouple plan and judge: SOTA 93.9"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00029,"raw_usage":{"total_tokens":1723,"prompt_tokens":995,"completion_tokens":728,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":645}},"tokens_in":611,"tokens_out":728,"duration_ms":7946,"temperature":1.0,"reasoning_tokens":645,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T00:40:04.370044+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute exact and near-duplicate overlap between the 2.5K MATH prompts used in training and the RewardBench reasoning subset; if any significant portion of the reasoning benchmark matches training prompts, the reported 95.5/96.1 reasoning scores and 93.9 overall would be inflated by leakage.","supporting_citations":[],"review_version":1}