Pith. sign in

REVIEW 2 major objections 6 minor 7 cited by

Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge

T0 review · 2 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read EvalPlanner, a judge model that plans before it judges, reaches state-of-the-art 93.9 on RewardBench using only 22K synthetic preference pairs.

desk verdict A solid training recipe with convincing ablations, but the RewardBench SOTA claim hinges on a missing decontamination check between MATH training data and the math-heavy evaluation sets. read the letter →

arxiv 2501.18099 v2 pith:66R6IVCC submitted 2025-01-30 cs.AI cs.CL

Swarnadeep Saha , Xian Li , Marjan Ghazvininejad , Jason Weston , Tianlu Wang This is my paper · ORCID
classification cs.AIcs.CL
keywords LLM-as-a-Judgerewardmodelingevaluationplanningchain-of-thoughtpreferenceoptimizationDPOsyntheticdatagenerativemodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EvalPlanner is a training recipe for LLM judges that separates the judge's thinking into two stages: first write an evaluation plan tailored to the instruction, then execute that plan step by step on the two candidate responses before issuing a verdict. The paper's central claim is that jointly optimizing plans and executions with preference optimization, using only synthetically generated preference pairs, produces a state-of-the-art generative reward model: 93.9 on RewardBench, with stronger results on RM-Bench, PPE Correctness, and multi-constraint instruction following. This matters because prior judges hand-design the content of reasoning traces per domain and train on up to 30 times more preference data, often human-annotated. If correct, EvalPlanner shows that planning and reasoning for evaluation can be learned, not engineered, and that test-time thinking alone is enough to make a judge accurate and reliable.

What carries the argument

The load-bearing mechanism is the three-part Chain-of-Thought with generative process $$p_\$\theta$(y\mid x,a,b)=\sum_{z\in P}\sum_{e\in E} p_\$\theta$(y\mid e,z,x,a,b)\,p_\$\theta$(e\mid z,x,a,b)\,p_\$\theta$(z\mid x),$$ where $z$ is an evaluation plan (a free-form recipe, possibly including reference answers, criteria, or unit-test-like checks), $e$ is the execution of that plan on the two responses, and $y$ is the final [[A]]/[[B]] verdict. Training data is built by sampling $|P|=5$ plans and $|E|=8$ executions per instruction (with both response orders), labeling a thought correct only if it reaches the correct verdict, and forming chosen/rejected pairs that differ in plan or execution. The model is then trained with SFT on chosen thoughts followed by two iterations of DPO, the second using fresh instructions and thoughts sampled from the previous iteration's model. This lets the optimization act on both plan quality and execution fidelity simultaneously, which the paper argues is what prior hand-designed criteria or verification-question methods cannot do.

What would settle it

Compute exact and near-duplicate overlap between the 2.5K MATH prompts used in training and the RewardBench reasoning subset; if any significant portion of the reasoning benchmark matches training prompts, the reported 95.5/96.1 reasoning scores and 93.9 overall would be inflated by leakage.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that an LLM-as-a-Judge trained to generate a Chain-of-Thought with three explicit components—an unconstrained evaluation plan $z$, a plan execution $e$, and a final verdict $y$—and iteratively optimized with DPO on correct-versus-incorrect (plan, execution, verdict) triples, outperforms all prior generative reward models on RewardBench (93.9 with a Llama-3.1-70B seed) while using only 22K synthetic preference pairs. The authors attribute the gains to the decoupling of planning from reasoning: the model learns both what to check and how to check it, rather than entangling the two in a single hand-structured rationale. They further report that a second DPO iteration on fresh instructions is what converts a competitive result (92.3 with 5K pairs) into the state of the art, and that the same recipe generalizes across two seed models and to benchmarks requiring objective correctness judgments.

Load-bearing premise

The central claim depends on the RewardBench reasoning test problems not overlapping with the MATH problems used for training, since the paper reports no decontamination check.

Editorial extensions

If this is right

  • Generative reward models can be trained without human-annotated preferences or rationales; the only requirements are seed-model instructions and a way to label response pairs correct or incorrect.
  • A single judge model can cover chat, safety, code, math, and multi-constraint instruction following because the plan is generated per instruction rather than specified by a human per domain.
  • Iterative self-improvement is the active scaling axis: two DPO iterations on 22K pairs (93.9) beat one iteration on the same total data (92.5), so additional iterations are the natural next step.
  • Planning-based judges are less vulnerable to position bias, subtle content changes, and style biases, as shown by gains on RM-Bench and position-consistent FollowBenchEval.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the plan–execution decoupling is the active ingredient, the same structure could be grafted onto scalar or verifier-style reward models, where the plan acts as a generated test harness rather than a free-text rationale.
  • A transfer experiment that fixes a reference plan and varies only executions would separate plan improvement from execution improvement; the current ablations vary both together.
  • Because MATH prompts are used for training and RewardBench's reasoning category contains math problems, a decontamination study is the first test that should accompany the state-of-the-art claim.
  • Plans generated for coding questions already resemble unit tests, so EvalPlanner points toward a unified treatment of evaluation planning and test-generation for verifiable tasks.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes EvalPlanner, a preference-optimization method for training LLM-as-a-Judge models. The judge is trained to generate (1) an unconstrained evaluation plan for the input instruction, (2) a step-by-step execution of that plan over the candidate responses, and (3) a final pairwise verdict. Training data are constructed synthetically: response pairs come from Self-Taught Evaluators (WildChat) and from correct/incorrect MATH solutions; plans and executions are sampled from the current model; and correct/incorrect (plan, execution, verdict) triples are used as chosen/rejected pairs. The model is trained with SFT followed by two iterations of DPO. Experiments on RewardBench, PPE, RM-Bench, JudgeBench, and a new FollowBenchEval benchmark report state-of-the-art generative reward model results (e.g., 93.9 on RewardBench) with only 22K synthetic preference pairs.

Significance. If the results hold, EvalPlanner is a convincing demonstration that decoupling planning from execution and iteratively optimizing the resulting thoughts via DPO is a data-efficient route to strong reward models. The paper includes informative ablations (Tables 8-11) showing the value of thought-level DPO, unconstrained plans, scaling the number of plans/executions, and the complementary roles of WildChat and MATH prompts. However, the headline SOTA claim is contingent on the absence of train/test overlap between the MATH training source and the math-containing evaluation benchmarks, which the paper does not establish. The new FollowBenchEval benchmark is a useful addition, though small.

major comments (2)
  1. [§3.1, Tables 1 and 2] The paper trains on 4,141 MATH triples (Section 3.1) and then reports its two strongest results on benchmarks with explicit math components: RewardBench Reasoning (Table 1, 95.5/96.1) and PPE Correctness MATH (Table 2, 79.2/81.7). PPE Correctness is documented as containing MATH (Hendrycks et al., 2021) examples, and RewardBench's Reasoning category includes mathematical problems. No decontamination, exact-match, or near-duplicate analysis is reported between the MATH training prompts and the math items in these evaluation sets. Because the improvements over baselines are concentrated in the math categories (Table 11: MATH training alone gives 93.4 Reasoning vs 89.3 for WildChat), even a small overlap could account for the margin separating EvalPlanner (93.9) from Skywork-Critic (93.3) on RewardBench. The authors should report overlap statistics and, if overlap exists, re-run the evaluation on the non-overlapping subset or retrain without the overlapping prompts.
  2. [§4.1, Table 1] The main SOTA claim rests on a 93.9 overall RewardBench score, which is only 0.5 points above Skywork-Critic (93.3) and 0.5 above LMUnit (93.4). The paper reports a single run per configuration and gives no confidence intervals, standard errors, or checkpoint-selection details beyond a 150-sample validation set. Given the small margin, the authors should report the variance across at least a few seeds/checkpoints or a significance test (e.g., bootstrap over RewardBench items) to support the claim that the difference is reproducible rather than within noise.
minor comments (6)
  1. [Figure 1] There is a typo in the example: 'nameänd' should read 'name and', and the phrase 'Assistant B's response.' is duplicated in Step 2. The figure should be corrected before publication.
  2. [Tables 1-7] Model names are inconsistent across the paper ('Llama3.1-70B-Instruct', 'Llama-3.1-70B-Instruct', 'LLama-3.1-70B-Instruct'). Please standardize to the official naming, e.g., 'Llama-3.1-70B-Instruct'.
  3. [Abstract] The abstract and Section 4.1 state that EvalPlanner achieves state-of-the-art on RewardBench and PPE. Table 2 shows that EvalPlanner does not exceed DeepSeek-GRM (MetaRM voting@32) on the PPE Preference subset (65.8/65.6 vs 67.2). Please qualify the claim to 'PPE Correctness' or 'PPE overall'.
  4. [§2.2] The definitions of D(a,b) and D(b,a) use the notation (x,a,b) -> (z, e_c, y_c) without explicitly specifying the domain of e_c and e_r or how correct/incorrect executions are selected. Clarify that e_c and e_r are executions from E(a,b) leading to correct and incorrect verdicts, respectively.
  5. [§3.2] FollowBenchEval is introduced as a new benchmark, but the paper does not report how the 205 samples were selected from FollowBench or release the constructed data. A reproducibility statement or a link to the data would be valuable.
  6. [§3.1] The text says 'Mixtral 22Bx8 Instruct model'; the standard name is 'Mixtral-8x22B-Instruct'. Please use the standard name.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: EvalPlanner's plan/execution optimization is evaluated on external benchmarks, and its self-training labels are constructed independently of those benchmarks.

full rationale

The paper's training signal is based on known correct/rejected response pairs: for MATH, responses leading to correct final answers are chosen and incorrect ones rejected; for WildChat, responses to the original instruction are chosen and responses to a noisy instruction are rejected. The CoT plans and executions are latent variables optimized via DPO to reproduce these externally defined labels. The test benchmarks (RewardBench, PPE, RM-Bench, JudgeBench, FollowBenchEval) provide independent triples with their own ground-truth judgments, so the reported accuracy is not equal to the training objective by construction. No fitted parameter is renamed as a prediction, and no load-bearing claim is justified solely by a self-citation. The paper does cite Self-Taught Evaluators for the WildChat response-pair generation recipe, and some authors overlap, but that prior work supplies training data and a baseline rather than the central claim that decoupled plan/execution preference optimization improves judge accuracy. The absence of decontamination between MATH training prompts and the MATH components of RewardBench/PPE is a validity risk, not a circularity: it concerns possible train/test overlap, not a definitional reduction of the reported scores to the training inputs. Thus the central derivation chain is self-contained with respect to circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method depends on several assumptions pulled from prior work or the authors' construction. The most fragile is the assumed disjointness of the MATH training distribution from RewardBench's reasoning benchmark, which is not tested. The synthetic label generation and the seed model's planning ability are additional domain assumptions that the paper partially validates through ablations and external benchmark results.

free parameters (3)
  • Number of DPO iterations = 2
    Section 4.1 Table 4: two iterations (5K+17K pairs) improve RewardBench from 92.3 (one iteration) to 93.9; the second iteration was selected because it improves the headline score.
  • Plans and executions per instruction = 5 plans, 8 executions
    Section 4.2 and Table 10: scaling from 3 plans/4 executions (88.8) to 5/8 (90.5) improves RewardBench; these numbers were chosen from the scaling study rather than preset.
  • Training source mix = 2.5K WildChat + 2.5K MATH for SFT/DPO1; 17K WildChat + 2.5K MATH for DPO2
    Section 3.1: the mix of instruction sources is hand-selected; Table 11 shows WildChat helps chat-hard while MATH helps reasoning.
assumptions (5)
  • domain assumption The noisy-instruction perturbation in Self-Taught Evaluators creates preference pairs where the original-instruction response is always better.
    Section 3.1 uses Self-Taught Evaluators' synthetic pairs for WildChat; if the noise does not reliably degrade quality, the training labels are unreliable.
  • domain assumption For MATH prompts, correct final answer implies better response.
    Section 3.1 labels MATH responses by final-answer correctness; this assumes correctness is a sufficient preference signal.
  • domain assumption The seed Llama-3.1/3.3-70B-Instruct can generate useful evaluation plans and executions when sampled at temperature 0.8.
    The self-training loop (Section 2.2) depends on the seed model's sampled thoughts containing enough correct-verdict examples to form preference pairs.
  • domain assumption RewardBench, PPE, RM-Bench, and JudgeBench do not overlap with the WildChat and MATH training prompts.
    Section 3.1 trains on WildChat and MATH, Section 3.2 evaluates on these benchmarks; no decontamination is reported, so SOTA assumes no leakage.
  • domain assumption Position-consistent accuracy is a valid and fairly applied metric for JudgeBench and FollowBenchEval.
    Section 3.2 reports position-consistent accuracy for these two benchmarks; the paper does not state that all baselines were re-evaluated under the same protocol.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge." pith.science (2026). https://pith.science/paper/66R6IVCC

@misc{pith2026250118099,
  author       = {Pith},
  title        = {Pith review of: Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/66R6IVCC}},
  note         = {Machine review of arXiv:2501.18099}
}
read the original abstract

LLM-as-a-Judge models generate chain-of-thought (CoT) sequences intended to capture the step-bystep reasoning process that underlies the final evaluation of a response. However, due to the lack of human annotated CoTs for evaluation, the required components and structure of effective reasoning traces remain understudied. Consequently, previous approaches often (1) constrain reasoning traces to hand-designed components, such as a list of criteria, reference answers, or verification questions and (2) structure them such that planning is intertwined with the reasoning for evaluation. In this work, we propose EvalPlanner, a preference optimization algorithm for Thinking-LLM-as-a-Judge that first generates an unconstrained evaluation plan, followed by its execution, and then the final judgment. In a self-training loop, EvalPlanner iteratively optimizes over synthetically constructed evaluation plans and executions, leading to better final verdicts. Our method achieves a new state-of-the-art performance for generative reward models on RewardBench (with a score of 93.9), despite being trained on fewer amount of, and synthetically generated, preference pairs. Additional experiments on other benchmarks like RM-Bench, JudgeBench, and FollowBenchEval further highlight the utility of both planning and reasoning for building robust LLM-as-a-Judge reasoning models.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Scale or Reason? A Compute-Equivalent Analysis of Reasoning Distillation

    cs.CL 2025-09 conditional novelty 6.0 of 10

    At matched training or inference FLOPs, plain instruction tuning is usually as good as or better than reasoning distillation, and reasoning only wins on open-ended tasks at 7B scale and up.

  2. LLMs for Customized Marketing Content Generation and Evaluation at Scale

    cs.CL 2025-06 conditional novelty 6.0 of 10

    MarketingFM generates e-commerce ad copy with RAG and an LLM; AutoEval uses LLM-as-a-Judge plus rule checks and self-refines its prompts, with online tests showing significant clicks and impressions lifts but no signi...

  3. J1: Exploring Simple Test-Time Scaling for LLM-as-a-Judge

    cs.LG 2025-05 conditional novelty 6.0 of 10

    J1-7B, a judge LLM trained with supervised fine-tuning and reinforcement learning, improves when forced to reflect with 'wait' tokens, and the scaling ability emerges during the RL phase.

  4. Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Models

    cs.CL 2025-04 conditional novelty 6.0 of 10

    MPO uses a meta reward model to continuously rewrite the reward model's evaluation prompt during PPO training, and the resulting models beat static-prompt RLAIF baselines on four tasks.

  5. AI vs. Human Judgment of Content Moderation: LLM-as-a-Judge and Ethics-Based Response Refusals

    cs.HC 2025-05 conditional novelty 5.0 of 10

    LLM-as-a-Judge systems rate ethical refusal responses more favorably than human users, a gap the paper calls moderation bias, while technical refusals do not show the same divergence.

  6. TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance

    cs.IR 2025-10 conditional novelty 4.0 of 10

    TaoSR-AGRL improves e-commerce search relevance by combining dense rule-aware reward shaping with adaptive ground-truth-guided replay in GRPO, reporting gains on Taobao's private offline and online evaluations.

  7. Generative AI Act II: Test Time Scaling Drives Cognition Engineering

    cs.CL 2025-04 conditional novelty 3.0 of 10

    Test-time scaling techniques such as long chain-of-thought, tree search, and self-correction define the paper's 'cognition engineering' paradigm, which it surveys, taxonomizes, and tutorials.

Reference graph

Works this paper leans on

21 extracted references · 18 canonical work pages · cited by 7 Pith papers

  1. [1]

    In this case, the function should be named ‘separate_paren_groups’, take a single parameter ‘paren_string’ of type ‘str’, and return a list of strings (‘List[str]’)

    **Check if the function is correctly defined**: Verify that the function is defined with the correct name, parameters, and return type. In this case, the function should be named ‘separate_paren_groups’, take a single parameter ‘paren_string’ of type ‘str’, and return a list of strings (‘List[str]’)

    work page

  2. [2]

    Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason E Weston, and Mike Lewis

    https://arxiv.org/abs/2402.14083. Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason E Weston, and Mike Lewis. Self- alignment with instruction backtranslation. InThe Twelfth International Conference on Learning Representations,

    arXiv

  3. [3]

    [Start of Evaluation Plan]

    https://openreview.net/forum?id=1oijHJBRsT. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InThe Twelfth International Conference on Learning Representations, 2024. https://openreview.net/forum?id=v8L0pN6EOi. Yang Liu, Dan Iter, Yi...

    arXiv 2023

  4. [4]

    **Check for error handling**: Verify that the function handles invalid inputs correctly. For example: * Input: ‘(()) (’, Expected output: Error or exception (since the input string is not balanced) * Input: ‘(()))’, Expected output: Error or exception (since the input string is not balanced)

    work page

  5. [5]

    **Test the function with the provided example**: Run the function with the input string ‘( ) (( )) (( )( ))’ and check if the output matches the expected result [‘()’, ‘(())’, ‘(()())’]

    work page

  6. [6]

    **Test the function with additional examples**: Provide additional test cases to ensure the function works correctly in different scenarios. For example: * Input: ‘(()) (())’, Expected output: [‘(())’, ‘(())’] * Input: ‘(()) ((()))’, Expected output: [‘(())’, ‘((()))’] * Input: ‘(()) (()())’, Expected output: [‘(())’, ‘(()())’] * Input: ‘’, Expected outpu...

    work page

  7. [8]

    Check for proper use of comments, variable naming, and function structure

    **Evaluate code quality and readability**: Assess the code’s readability, maintainability, and adherence to best practices. Check for proper use of comments, variable naming, and function structure

    work page

  8. [9]

    Figure 6 Example of a plan generated by EvalPlanner for a coding problem

    **Check for edge cases**: Test the function with edge cases, such as: * Input: ‘(’, Expected output: Error or exception (since the input string is not balanced) * Input: ‘)’, Expected output: Error or exception (since the input string is not balanced) By following these steps, we can comprehensively evaluate the quality of the responses provided by AI ass...

    work page

Show all 21 references
  1. [10]

    **Identify the relationship between angles**: Recognize that ̸ A is an inscribed angle and ̸ B is part of the same triangle but not directly given by the inscribed angle theorem

  2. [11]

    **Apply properties of a circle**: Since AB is a diameter, △ABC is a right-angled triangle with ̸ C = 90 ◦ because any triangle inscribed in a circle where one of the sides is a diameter of the circle must be a right triangle

  3. [12]

    Knowing ̸ A = 14◦ and ̸ C = 90◦, we can find̸ B

    **Use the angle sum property of a triangle**: The sum of the angles in any triangle is 180◦. Knowing ̸ A = 14◦ and ̸ C = 90◦, we can find̸ B

  4. [13]

    **Calculate ̸ B**: - ̸ A + ̸ B + ̸ C = 180◦ - 14◦ + ̸ B + 90◦ = 180◦ - ̸ B = 180◦ − 14◦ − 90◦ - ̸ B = 76◦ IV. Evaluation Criteria * Accuracy: Does the response correctly calculate̸ B using the properties of circles and triangles? * Completeness: Does the response address all p...

  5. [14]

    **Respect and Inclusivity:** Does the response promote respect and inclusivity towards people with dis- abilities, or does it perpetuate harmful stereotypes and discrimination?

  6. [15]

    **Empathy and Understanding:** Does the response demonstrate empathy and understanding towards individuals with disabilities, or does it show a lack of awareness and sensitivity?

  7. [16]

    **Compliance with Ethical Guidelines:** Does the response comply with ethical guidelines and principles that prohibit discrimination and promote inclusivity? *Evaluation Steps:**

  8. [17]

    **Initial Review:** Review the AI assistant’s response to the user question

  9. [18]

    * Avoids perpetuating harmful stereotypes and discrimination

    **Content Analysis:** Analyze the content of the response to determine if it: * Uses language that is respectful and inclusive towards people with disabilities. * Avoids perpetuating harmful stereotypes and discrimination. * Demonstrates empathy and understanding towards indiv...

  10. [19]

    **Comparison with Ethical Guidelines:** Compare the response with established ethical guidelines and principles that promote inclusivity and prohibit discrimination

  11. [20]

    * 2: The response is somewhat discriminatory, but attempts to be respectful

    **Rating and Feedback:** Assign a rating to the response based on its quality, using a scale of 1-5, where: * 1: The response is highly discriminatory and perpetuates harmful stereotypes. * 2: The response is somewhat discriminatory, but attempts to be respectful. * 3: The res...

  12. [21]

    **Feedback and Recommendations:** Provide feedback and recommendations to the AI assistant developers to improve the response quality and ensure compliance with ethical guidelines. *Reference Answer:** A high-quality response to the user question could be: “I’m sorry, but I do...

  13. [2024]

    Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto

    https://arxiv.org/abs/2407.21783. Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural I...

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.