Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

Relative evaluator score gaps on current-policy response sets are the signals that drive LLM policy updates, so evaluators must co-evolve with the policies they supervise; DynamicRubric does this with response-set-conditioned rubrics.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 10:49 UTC pith:4A3RB6HQ

load-bearing objection The empirical core is real and the co-evolution story mostly holds, but the formal gap-preservation guarantee is assumed into existence in Appendix J—read the theory as motivation, not proof. the 3 major comments →

arxiv 2607.20083 v2 pith:4A3RB6HQ submitted 2026-07-22 cs.LG cs.AI

Co-Evolving LLM Evaluators and Policies via DynamicRubric

Beining Wang , Weihang Su , Hongtao Tian , Hao Kong , Tao Yang , Ting Yao , Qingyi Pan , Yueyue Wu
show 3 more authors
Qingyao Ai Min Zhang Yiqun Liu
This is my paper
classification cs.LG cs.AI
keywords LLM post-trainingevaluator-guided policy optimizationrubric-based evaluationrelative score gapsresponse-set-conditioned evaluationevaluator-policy co-evolutionreward modelreinforcement learning from feedback
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the useful signal in evaluator-guided LLM post-training is not an evaluator's absolute score but the relative score gap between candidate responses sampled from the current policy. It proves a basic identity: the local gain from shifting probability mass from one response to another equals exactly the evaluator's score gap, so those gaps are the optimization signals. It then proposes DynamicRubric, which co-evolves a rubric generator with the policy: the generator writes weighted binary rubric items conditioned on the current response set, a verifier applies them, and the aggregated scores supervise the policy. With 8B backbones, the resulting policies beat supervision from a 70B reward model and a 235B static rubric generator, and the recipe is deployed in production. If correct, evaluator design should focus on preserving fine-grained discriminability on the policy's own response distribution, and evaluators must be re-adapted as the policy moves.

Core claim

The central claim is the score-gap identity (Eq. 7): conditioned on a prompt and a set of responses sampled from the current policy, the directional derivative of the policy's expected evaluator score along the direction that moves probability mass from a worse response to a better one is exactly the evaluator score gap between them. Thus Δgap(E;θ) is the expected local optimization signal an evaluator exposes. Collapsed or misordered gaps mean weak or misleading policy supervision. Because policy updates change the response distribution, an evaluator must condition on the current response set and be updated along with the policy. DynamicRubric instantiates this: a generator produces weighte

What carries the argument

The carrying identity is E(x,y+)-E(x,y-) as a directional derivative, making relative score gaps the policy optimization signal. DynamicRubric's evaluator is the rubric-based score E_phi(x,y|C)=sum w_m v_m(y;C)/sum w_m, where rubric items and weights are generated conditioned on the candidate set; training maximizes the weighted Bernoulli variance of verifier outcomes over the set (discriminability) plus a pairwise log-sigmoid anchor-ranking reward, which the paper shows lower-bounds Δgap under an assumed calibration condition. The policy is then optimized with group-normalized advantages derived from those scores.

Load-bearing premise

The load-bearing premise is Eq. (45): the paper assumes the anchor-ranking objective controls the combined directional and coverage residuals of the discriminability reward, but it does not prove this condition for its actual ordered-pair distribution, anchor data, or trained generator class.

What would settle it

On held-out prompts from a fixed policy, compute Δgap (Eq. 2) for three generator variants: full objective, discriminability-only, and anchor-only. If the full objective's training value increases while its Δgap does not, or if a variant with lower training objective has higher Δgap, the bound in Theorem 1 is violated.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Evaluator-guided post-training should be judged by how well it preserves relative score gaps on the current policy's response distribution, not by absolute-score accuracy.
  • Reward models and judges should be re-adapted to each policy's response sets; the paper's continuous co-evolution experiments show a stale evaluator gives weaker supervision.
  • Response-set-conditioned evaluation strictly generalizes prompt-only evaluation and is no worse in optimal local ranking loss.
  • Static rubrics written before seeing candidate responses saturate; dynamic rubrics that target within-set differences sustain policy improvement.
  • The pipeline can be run with a single 8B backbone for generator, verifier, and policy, replacing larger reward models and static rubric generators.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The strongest theoretical guarantee rests on Eq. (45), an aggregate anchor-calibration condition that the paper assumes; if it fails on real ordered-pair distributions, maximizing the stated evaluator objective need not increase Δgap, even if the empirical method still works.
  • A natural testable extension is to add a direct Δgap surrogate estimated from current-policy ordered pairs alongside the discriminability and anchor objectives, making the training signal match the theory more literally.
  • The score-gap view suggests that reward-model overoptimization in standard RLHF may often be evaluator staleness: a reward model trained on pre-update responses loses discriminability on post-update responses, so frequent evaluator re-adaptation could be a cheap fix.
  • Since online metrics are reported textually without exact values, the production deployment is weaker evidence than the benchmark results, though consistent with the paper's central claim.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes DynamicRubric, a framework for co-evolving an LLM evaluator (a rubric generator plus a frozen verifier) and a policy during post-training. The evaluator generates weighted binary rubric items conditioned on the current policy's sampled response set; the verifier applies these rubrics; the resulting response-level scores supervise policy updates. The paper motivates this design with a theoretical claim that relative evaluator score gaps on policy-induced response sets are exactly the local optimization signals for probability-mass transfer, and that a response-set-conditioned evaluator can achieve no worse local ranking loss than a prompt-only evaluator. A further bound (Eq. 18) is claimed to show that maximizing a combination of a discriminability reward and an anchor ranking reward increases a lower bound on the expected score gap. Experiments with Qwen3-8B and Llama-3.1-8B report gains over larger reward models and static rubric supervision on preference, open-ended generation, and verifiable reasoning benchmarks, and a production deployment at WeChat Search. The empirical comparisons are broad and consistently favor DynamicRubric, but the theoretical guarantee in Appendix J depends on an assumed aggregate anchor calibration condition that is not established.

Significance. If the empirical results hold, DynamicRubric is a practically valuable recipe: it uses only an 8B backbone, requires no external reward model or judge, improves evaluator accuracy across many verifiers and backbones, and transfers to policy supervision and to verifiable reasoning/coding tasks. The deployment at WeChat Search, while reported only textually, indicates real-world viability. The paper also provides useful reproducibility assets: detailed hyperparameters, prompt templates, and algorithm pseudocode. However, the theoretical contribution is substantially overstated. Eq. (7) is a definitional identity, and the load-bearing lower bound (Eq. 18) rests on an unproved condition (Eq. 45) that is not tied to the actual data distribution. The empirical contribution is solid enough to warrant revision, but the theoretical claims should be either proved under explicit assumptions or explicitly relegated to heuristic motivation.

major comments (3)
  1. [Appendix J, Eq. (45)] The aggregate anchor calibration condition is assumed, not proved. The text says 'suppose there exist...' and the subsequent Theorem 1 routes all content through this condition. The condition is not tied to the actual ordered-pair distribution P(·|x,C), which is never specified in Eq. (2) or Appendix J; nor is it derived for the Nectar anchor lists or the trained generator/verifier class. Since η_res is an expectation over current-policy response sets while J_anchor is computed on anchor lists conditioned on C, anchor agreement need not control residuals on C. Without Eq. (45), Eq. (18) does not follow. This is the load-bearing step for the stated theoretical motivation.
  2. [Appendix J, Eqs. (47)-(48), and §4.1] Even if Eq. (45) were granted, the coefficients in Eq. (47)-(48) do not match the training objective. The paper trains the generator to maximize J_disc + λ J_anchor with λ=1 (Section 4.1), while the lower bound in Eq. (47) is c_disc J_disc + c_anchor J_anchor with c_disc = γρ and c_anchor = β. Unless β/(γρ) = 1, an update can increase the training objective while decreasing the lower bound. No argument is given for this equality. Thus the 'therefore' after Eq. (18) in Section 3.4 is not justified.
  3. [§3.1, Eq. (7)] The identity in Eq. (7) is definitional: Δgap is defined as the expectation of E(x,y+) - E(x,y-) under P, so the directional derivative of the linear local objective is equal to that gap by construction. This does not by itself establish that 'preserving score gaps' is the correct design criterion; it restates the definition. The substantive content is supposed to come from Eq. (18), but as noted above that bound is not established. The theory section currently overclaims: it presents a tautology as a theoretical characterization and a conditional bound as a guarantee.
minor comments (4)
  1. [Eq. (2)] The ordered-pair distribution P is never concretely instantiated. Since all theoretical quantities depend on P, please state an explicit construction (e.g., uniform over pairs whose reference ranking is given by anchor labels) and clarify where P is used in experiments.
  2. [Tables 2, 5, 6, 8] Results are reported without confidence intervals, number of seeds, or variance estimates. Given that some margins are small (e.g., Table 2, ArenaHardv2.0: 21.0 vs. 20.8), please report multiple seeds or bootstrap intervals.
  3. [§4.5] Online A/B improvements are described textually with no effect sizes, confidence intervals, or statistical test details. If business confidentiality is a concern, please provide a supplementary quantitative summary or clearly mark the section as qualitative.
  4. [References and typos] Guo et al. 2025a and 2025b appear to be the same paper duplicated. Also, §4.5 contains 'an large language model' and the text uses 'A VG@5' where 'AVG@5' is meant.

Circularity Check

0 steps flagged

No significant circularity: DynamicRubric's empirical results are externally benchmarked and its theoretical claims, while weak in places, are not disguised fits or self-citation chains.

full rationale

The paper's central empirical claims (Tables 1, 2, ablations, cross-backbone, online deployment) are evaluated against held-out benchmarks and external baselines; the trained DR-Generator/DR-Policy are not fitted to the metrics they are then used to predict. Eq. (6)-(7) is an explicit mathematical identity: because J_local is defined as a linear expectation of evaluator scores (Eq. 4), the directional derivative along the transfer direction is definitionally the score gap, and the paper itself labels the equality 'by Equation (2)'. This is a tautology rather than an independently predictive derivation, but it is not a fitted-input-called-prediction or a result whose empirical content is forced by construction. The Appendix J lower-bound theorem is conditional on the 'aggregate anchor calibration' condition (Eq. 45), which is assumed rather than proved for the actual ordered-pair distribution, the Nectar anchors, or the trained generator class; also, the theorem's coefficients (γρ, β) are not shown to be proportional to the λ=1 weights used in the training objective. These are soundness/correctness gaps in the theoretical guarantee, not circular reductions: Eq. (47) does not simply restate Eq. (45) as an identity, and the paper does not claim to have fit those constants to the final benchmarks. No load-bearing argument reduces to a self-citation: the only self-citations (WeChat-YATT training library, WeLM production model) are descriptive implementation details, not evidence for the co-evolution principle. I therefore find no significant circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The ledger shows that the theory's core content is a definitional identity plus an assumed calibration condition, while the method's empirical support relies on external anchor supervision and automatic judges. Free parameters are conventional but mostly unablated.

free parameters (4)
  • anchor reward weight λ = 1
    Set to 1 by combining anchor and discriminability rewards at the advantage level; no sensitivity analysis or tuning curve is reported (§4.1, Appendix D).
  • response-set size K = 8
    Rollout size K=8 is fixed for GRPO; no ablation across K is reported (§4.1).
  • RBP-style anchor weighting scheme = 1/((j-i)·2^(i-1))
    The ranking reward in Eq. (15) uses a hand-designed decay and inverse-distance weighting; no ablation or justification beyond analogy to Rank-Biased Precision.
  • KL coefficient β and learning rates = β=0.01; lr 5e-6/1e-6
    Standard GRPO hyperparameters selected without reported sensitivity analysis (Appendix D).
axioms (6)
  • domain assumption Increasing expected evaluator score over the sampled candidate set is the correct local policy optimization target
    The linear local proxy objective J_local in Eq. (4) defines the entire theoretical analysis; if policy optimization should optimize something else (e.g., ranking-consistent reward), the 'gap as signal' result changes.
  • domain assumption There exists an evaluator-independent reference quality distribution P(·|x,C) over ordered pairs
    Required to define Δgap in Eq. (2) and the anchor residuals in Appendix J; the paper never constructs P in practice and uses Nectar ranked anchors as a proxy.
  • ad hoc to paper Aggregate anchor calibration condition (Eq. 45) holds
    The claimed lower bound in Theorem 1 is valid only if the anchor reward controls the aggregate directional and coverage residuals; this is assumed, not derived or verified.
  • domain assumption Frozen DR-Verifier correctly executes generated binary rubrics
    All scores pass through V; verifier noise or systematic bias is not modeled, though verifier robustness is tested empirically across several models.
  • domain assumption Nectar ranked anchors are valid reference rankings for calibrating rubric direction
    J_anchor uses ranked lists from Nectar as ground truth; if anchors are noisy, biased, or misaligned with the target domain, the generator direction is mis-calibrated.
  • domain assumption Automatic judges and online metrics measure true quality
    Policy gains are measured with DeepSeek-V4-Flash and GPT-4.1 judges, and online results are reported only textually; no human evaluation is reported.

pith-pipeline@v1.3.0-alltime-deepseek · 24765 in / 14717 out tokens · 140563 ms · 2026-08-01T10:49:00.138984+00:00 · methodology

0 comments
read the original abstract

Post-training with evaluator feedback on policy-induced samples serves as a major mechanism for improving large language models. As policies improve, these sampled responses become close in quality. These close candidates create a bottleneck for policy optimization: collapsed relative evaluator score gaps yield weak or misleading policy supervision. We theoretically characterize why these gaps matter through a probability allocation view, showing that the directional gain of shifting probability mass from one response to another is exactly the evaluator score gap between them. This identifies relative score gaps as the policy optimization signals that guide updates. Motivated by this view, we propose DynamicRubric, a response-set-conditioned evaluator--policy co-evolution framework that generates weighted binary rubric items for each candidate set and aggregates the resulting judgments into response-level scores. In our experiments with 8B backbones, DynamicRubric improves evaluator performance and provides stronger policy supervision than baselines using a 70B reward model or a 235B static rubric generator. DynamicRubric-optimized policies also show gains on verifiable reasoning and coding tasks. A DynamicRubric-optimized model is fully deployed in WeChat Search's AI answering scenario, where it serves all online traffic across tens of millions of requests per day and improves key online metrics. These results suggest a principle for evaluator-guided post-training: evaluators should evolve with the policies they supervise.

Figures

Figures reproduced from arXiv: 2607.20083 by Beining Wang, Hao Kong, Hongtao Tian, Min Zhang, Qingyao Ai, Qingyi Pan, Tao Yang, Ting Yao, Weihang Su, Yiqun Liu, Yueyue Wu.

Figure 1
Figure 1. Figure 1: Overview of DynamicRubric. For each prompt, DR-Policy samples a candidate response set, DR [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Effect of evaluator–policy co-evolution in DynamicRubric. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance of the initial policy and two [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: GPT-4.1 meta-evaluation of rubric quality. We [PITH_FULL_IMAGE:figures/full_fig_p018_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning

    cs.CL 2026-07 conditional novelty 7.0

    SERPO co-evolves G-N-B response archives, query-specific rubrics, and actor parameters to raise open-ended TTRL scores by up to ~20 points without external judges or labels.

  2. SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning

    cs.CL 2026-07 conditional novelty 7.0

    SERPO co-evolves per-question grading rubrics, Good-Normal-Bad response archives, and policy parameters so a language model can train itself at inference time without labels, gaining up to 20.6 points on open-ended me...

Reference graph

Works this paper leans on

28 extracted references · 3 linked inside Pith · cited by 1 Pith paper

  1. [1]

    The response provides the answer letter and repeats it five times

  2. [2]

    The response includes step-by-step reasoning

  3. [3]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

    Direct preference optimization: Your language model is secretly a reward model.Advances in neural information processing systems, 36:53728–53741. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Prox- imal policy optimization algorithms.Preprint, arXiv:1707.06347. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao ...

    Pith/arXiv arXiv 2017

  4. [4]

    For pairwise benchmarks, the evaluator is asked to select the preferred response from a candidate pair

    and LMSYS-Chat-1M (Zheng et al., 2024). For pairwise benchmarks, the evaluator is asked to select the preferred response from a candidate pair. For listwise benchmarks, the evaluator as- signs scores to multiple candidate responses, and the induced ranking is compared with the annotated ranking. When evaluating dynamic rubric generators in Section 4.2, we...

    2024

  5. [5]

    The response discusses the proportionality constant in the moment of inertia

  6. [6]

    The response computes angular velocity from tangential velocity and radius

  7. [7]

    The response uses the correct angular-momentum formula

  8. [8]

    Representative rubrics generated by DynamicRubric-trained generatorG 2

    The response stays focused on the question. Representative rubrics generated by DynamicRubric-trained generatorG 2

  9. [9]

    The response mentions the exact values used for mass, radius, or angular velocity

  10. [10]

    The response compares the estimated angular momentum with the provided answer options

  11. [11]

    The response mentions central pressure or wind speed as part of the estimation

  12. [12]

    The response states the disk-like approximation for the hurricane

  13. [15]

    The response estimates mass and radius using typical hurricane-scale values

  14. [16]

    yes"/"no

    The response computes both moment of inertia and angular momentum. The zero-shot generator G0 produces valid but weakly discriminative criteria. Its rubrics check answer formatting, step-by-step presentation, for- mula usage, and topical relevance. Both candidate responses satisfy these broad criteria: each gives an option in the required format, presents...

  15. [17]

    - Use the response pool to notice which qualities help a response satisfy the user's request better

    Focus on the user's request: - Design rubrics to evaluate how well a response answers the user's actual request. - Use the response pool to notice which qualities help a response satisfy the user's request better. - Do not hardcode specific facts, numbers, names, entities, or other concrete details into your rubrics unless they are explicitly provided in ...

  16. [18]

    Evaluate constraint adherence: - If the user prompt asks for specific formats, lengths, styles, or output structures, create rubrics to strictly check those constraints

  17. [19]

    partially correct

    Atomicity and absolute binary (no "partially correct"): - Each rubric must evaluate exactly one distinct aspect. - Rubrics must be strictly binary (true or false). - If a response could reasonably satisfy only part of a rubric, that rubric is invalid and must be split or rewritten. - Bad example: "The response explains the cause and gives a solution."

  18. [20]

    yes" or

    Specificity and objectivity: - Prefer rubrics anchored to clear response features, explicit logical steps, or specific constraints requested by the user . - Make sure the boundary of the rubric is clear enough that different evaluators would usually agree on "yes" or "no". - Avoid vague descriptions. - Do not write rubrics around highly specific mistakes ...

  19. [21]

    - If the user did not ask for a specific format, a format-based rubric should usually be omitted

    Focus on substance: - Do not include rubrics that mainly distinguish responses by formatting or other presentation choices unless the user prompt explicitly requires them. - If the user did not ask for a specific format, a format-based rubric should usually be omitted

  20. [22]

    yes" = good): - Frame all rubrics so that a

    Positive framing ("yes" = good): - Frame all rubrics so that a "yes" evaluation means the response is good, and "no" means it is bad. - Do not use double negatives

  21. [23]

    - Each rubric should be written as a general standard that can be applied to evaluate any response independently

    No reference to specific responses: - Rubrics must not refer to any specific response. - Each rubric should be written as a general standard that can be applied to evaluate any response independently. 23

  22. [24]

    - Each rubric should contain all the information needed to evaluate a response

    Mutually exclusive and self-contained: - Ensure each rubric evaluates a distinct aspect. - Each rubric should contain all the information needed to evaluate a response. - If one broad rubric and one narrower rubric are really checking the same thing, keep only one of them or rewrite them so they check clearly different things. - Do not treat two rubrics a...

  23. [25]

    - Essential (Weight 5): Core task requirements, critical user constraints, or safety-related requirements; if missing, the response is seriously flawed or invalid

    Assigning weights (1-5 scale): - Assign an integer weight between 1 and 5 to each rubric based on how necessary that rubric is for a strong response to the user's request. - Essential (Weight 5): Core task requirements, critical user constraints, or safety-related requirements; if missing, the response is seriously flawed or invalid. - Important (Weight 3...

  24. [26]

    rubric":

    The number of rubrics: - The appropriate number of rubrics depends on the task, but the rubric list should include enough independent rubrics to cover the meaningful quality dimensions needed for a discriminating evaluation. ## Rubric Design Workflow: - First look at the user prompt on its own to identify the core task, any explicit constraints, and the m...

  25. [27]

    A list of rubrics, each with a unique ID

  26. [28]

    yes" if the response clearly meets the rubric. - Answer

    A response to the user prompt. Your task is to evaluate the response against each rubric independently and decide whether each rubric is satisfied. ## Scoring Rules: - Answer "yes" if the response clearly meets the rubric. - Answer "no" if the response does not meet the rubric. Partial matches should be scored as "no". - If a rubric is poorly written and ...

  27. [2023]

    Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Ju- jie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou

    Let’s verify step by step.arXiv preprint arXiv:2305.20050. Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Ju- jie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. 2024. Skywork-reward: Bag of tricks for reward modeling in llms.arXiv preprint arXiv:2410.18451. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. ...

    Pith/arXiv arXiv 2024

  28. [2025]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z

    Rubrics as rewards: Reinforcement learn- ing beyond verifiable domains.arXiv preprint arXiv:2507.17746. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhu- oshu Li, Ziyi Gao, Aixin Liu, and 175 others. 2025a. Deeps...

    Pith/arXiv arXiv 2002