{"id":"c4b49b83-11a6-4cb6-9066-b37199f4039e","arxiv_id":"2607.20083","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A response-set-conditioned rubric generator, co-trained with the policy it supervises, provides stronger evaluator and policy supervision than larger static reward models.","lead":"This paper introduces DynamicRubric, a post-training method in which an LLM evaluator generates weighted yes/no rubric items for each set of candidate responses, and the evaluator and policy are updated together. The authors report that an 8B model trained this way outperforms larger reward-model and static-rubric baselines on evaluation and generation benchmarks, and that a deployed version improved WeChat Search's AI answering metrics.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The theoretical bridge from Δgap to DynamicRubric's objective depends on the unproved Eq. (45) and on lower-bound coefficients matching λ; neither step is established.","rationale":"The reader's weakest-assumption analysis and mine both center on Appendix J. I agree that Eq. 45 is the unsecured premise: it asserts that anchor ranking controls directional/coverage residuals on the current-policy pair distribution, and the paper does not prove it for P, Nectar anchors, or the trained generator class. I add a sharper observation: the distribution mismatch between η_res (over current-policy C) and J_anchor (over anchor lists C_anchor) is not described, and P is left unspecified, so the condition is not even empirically checkable as written. I also flag a separate inference gap: Eq. 47's lower-bound coefficients are not shown to match the λ=1 combination actually optimized; without that, 'maximizing the combination increases a lower bound' does not follow. These are theory-level concerns; the experimental evidence—co-evolution ablations, cross-backbone repeats, verifier/judge robustness, and the production deployment—is substantial and supports the method's empirical value. That is why the verdict should remain CONDITIONAL rather than moving to REJECT: the central practical claim may survive, but the theoretical guarantee should be re-derived or empirically checked.","tokens_in":25083,"tokens_out":14607,"duration_ms":142400,"concrete_test":"Estimate Eq. (45) on held-out training prompts: fix θ_{t-1}, sample C~π_θ, define P(·|x,C) by Nectar rankings restricted to C, and compute η_res (Eq. 44), J_anchor (Eq. 16), and J_disc (Eq. 14) for several generator checkpoints in the update class across rounds. Check whether there exist β>0 with β=γρ (so the lower-bound coefficients align with λ=1) and round-dependent offsets ε_t satisfying -η_res ≥ β J_anchor + ε_t for all checkpoints in that round's candidate class. The binding regime is late training, where J_anchor is near zero and the condition requires -η_res ≥ ε_t. If the inequality fails, or no such β exists, Theorem 1's premise is false and the theoretical justification collapses; if it holds, the concern is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is not Eq. (7) (a definitional identity) or Eq. (10) (function-class inclusion), but Appendix J's move from Lemma 1 to Theorem 1 via the 'aggregate anchor calibration' condition, Eq. (45): -η_res^t(ϕ) ≥ β J_anchor^t(ϕ;θ_{t-1}) + ε_t. The paper assumes Eq. (45); it does not prove it for the actual ordered-pair distribution P, the Nectar anchors, or the trained generator class. The gap is structural: η_res (Eq. 44) is an expectation over ordered pairs within current-policy response sets C, whereas J_anchor (Eq. 16) is computed on Nectar anchor lists C_anchor, with rubrics generated from C but applied to C_anchor; anchor agreement need not control residuals on C. P is also never specified, so the condition cannot be checked as stated. Moreover, even granting Eq. (45), the paper's 'therefore' is not automatic: Eq. (47) has coefficients c_disc=γρ and c_anchor=β, while training maximizes J_disc + λ J_anchor (λ=1). Unless c_disc/c_anchor = 1/λ, a parameter step can increase the objective and decrease the lower bound. Without these links, the DynamicRubric evaluator objective is not shown to preserve Δgap; the empirical claims must stand alone.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DynamicRubric, a framework for co-evolving an LLM evaluator (a rubric generator plus a frozen verifier) and a policy during post-training. The evaluator generates weighted binary rubric items conditioned on the current policy's sampled response set; the verifier applies these rubrics; the resulting response-level scores supervise policy updates. The paper motivates this design with a theoretical claim that relative evaluator score gaps on policy-induced response sets are exactly the local optimization signals for probability-mass transfer, and that a response-set-conditioned evaluator can achieve no worse local ranking loss than a prompt-only evaluator. A further bound (Eq. 18) is claimed to show that maximizing a combination of a discriminability reward and an anchor ranking reward increases a lower bound on the expected score gap. Experiments with Qwen3-8B and Llama-3.1-8B report gains over larger reward models and static rubric supervision on preference, open-ended generation, and verifiable reasoning benchmarks, and a production deployment at WeChat Search. The empirical comparisons are broad and consistently favor DynamicRubric, but the theoretical guarantee in Appendix J depends on an assumed aggregate anchor calibration condition that is not established.","tokens_in":25442,"tokens_out":4386,"duration_ms":45040,"significance":"If the empirical results hold, DynamicRubric is a practically valuable recipe: it uses only an 8B backbone, requires no external reward model or judge, improves evaluator accuracy across many verifiers and backbones, and transfers to policy supervision and to verifiable reasoning/coding tasks. The deployment at WeChat Search, while reported only textually, indicates real-world viability. The paper also provides useful reproducibility assets: detailed hyperparameters, prompt templates, and algorithm pseudocode. However, the theoretical contribution is substantially overstated. Eq. (7) is a definitional identity, and the load-bearing lower bound (Eq. 18) rests on an unproved condition (Eq. 45) that is not tied to the actual data distribution. The empirical contribution is solid enough to warrant revision, but the theoretical claims should be either proved under explicit assumptions or explicitly relegated to heuristic motivation.","major_comments":[{"comment":"The aggregate anchor calibration condition is assumed, not proved. The text says 'suppose there exist...' and the subsequent Theorem 1 routes all content through this condition. The condition is not tied to the actual ordered-pair distribution P(·|x,C), which is never specified in Eq. (2) or Appendix J; nor is it derived for the Nectar anchor lists or the trained generator/verifier class. Since η_res is an expectation over current-policy response sets while J_anchor is computed on anchor lists conditioned on C, anchor agreement need not control residuals on C. Without Eq. (45), Eq. (18) does not follow. This is the load-bearing step for the stated theoretical motivation.","section":"Appendix J, Eq. (45)"},{"comment":"Even if Eq. (45) were granted, the coefficients in Eq. (47)-(48) do not match the training objective. The paper trains the generator to maximize J_disc + λ J_anchor with λ=1 (Section 4.1), while the lower bound in Eq. (47) is c_disc J_disc + c_anchor J_anchor with c_disc = γρ and c_anchor = β. Unless β/(γρ) = 1, an update can increase the training objective while decreasing the lower bound. No argument is given for this equality. Thus the 'therefore' after Eq. (18) in Section 3.4 is not justified.","section":"Appendix J, Eqs. (47)-(48), and §4.1"},{"comment":"The identity in Eq. (7) is definitional: Δgap is defined as the expectation of E(x,y+) - E(x,y-) under P, so the directional derivative of the linear local objective is equal to that gap by construction. This does not by itself establish that 'preserving score gaps' is the correct design criterion; it restates the definition. The substantive content is supposed to come from Eq. (18), but as noted above that bound is not established. The theory section currently overclaims: it presents a tautology as a theoretical characterization and a conditional bound as a guarantee.","section":"§3.1, Eq. (7)"}],"minor_comments":[{"comment":"The ordered-pair distribution P is never concretely instantiated. Since all theoretical quantities depend on P, please state an explicit construction (e.g., uniform over pairs whose reference ranking is given by anchor labels) and clarify where P is used in experiments.","section":"Eq. (2)"},{"comment":"Results are reported without confidence intervals, number of seeds, or variance estimates. Given that some margins are small (e.g., Table 2, ArenaHardv2.0: 21.0 vs. 20.8), please report multiple seeds or bootstrap intervals.","section":"Tables 2, 5, 6, 8"},{"comment":"Online A/B improvements are described textually with no effect sizes, confidence intervals, or statistical test details. If business confidentiality is a concern, please provide a supplementary quantitative summary or clearly mark the section as qualitative.","section":"§4.5"},{"comment":"Guo et al. 2025a and 2025b appear to be the same paper duplicated. Also, §4.5 contains 'an large language model' and the text uses 'A VG@5' where 'AVG@5' is meant.","section":"References and typos"}],"recommendation":"major_revision","confidential_remarks":"The empirical core of this paper is strong and broad, and I would not reject it on the basis of the theory alone. However, the theoretical claims in Section 3.4 and Appendix J are presented as established when they depend on an assumed condition (Eq. 45) and on a coefficient-matching equality that is not argued. I recommend major revision: the authors should either prove Eq. (45) under explicit, checkable assumptions (including a concrete P), or clearly downgrade the theoretical section to a motivating heuristic and let the empirical results stand on their own. The deployment claim is not independently verifiable from the text, but it is not grounds for rejection by itself."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid empirical paper with a theory section that is weaker than it looks. What's actually new: DynamicRubric generates weighted binary rubrics conditioned on the current candidate response set, trains the generator with a Bernoulli-variance discriminability reward plus ranked-anchor calibration, and co-evolves generator and policy in a bilevel loop. That combination is not present in RaR, G-Eval, or FLASK, and the experiments back it. An 8B DR-Generator beats zero-shot 32B dynamic rubric generation and large scalar reward models on the evaluator benchmarks, and the DR-Policy consistently outperforms supervision from a 70B reward model and a 235B static rubric generator on AlpacaEval2, ArenaHard, WildBench, and WritingBench. The verifier robustness table and the Llama-3.1-8B cross-backbone replication are genuine evidence. The WeChat deployment claim is impressive but unverifiable, and the authors are appropriately honest about that.\n\nThe soft spot is the theory. Equation (7) is definitional: the directional derivative of a linear local proxy along a transfer direction equals the score gap. Fine as framing, but it is not a substantive theorem. The load-bearing lower bound in Appendix J depends on the aggregate anchor calibration condition, Eq. (45), which asserts that the anchor objective controls the combined directional and coverage residuals. That condition is assumed, not proven, for the actual ordered-pair distribution P, the Nectar anchors, or the trained generator class—and P is never even specified. Even granting Eq. (45), there is a coefficient mismatch: the bound has c_disc = γρ and c_anchor = β, while training maximizes J_disc + λ J_anchor with λ = 1. Nothing ensures that a step improving the training objective increases the lower bound. So Eq. (18) in the main text overstates what has been shown. The co-evolution experiments in §4.4 are the strongest support for the paper's principle, and they do not rely on the bound.\n\nMinor but worth saying: offline tables have no error bars, no code or data are shipped, and the production metrics are confidential. Those are fixable in revision. The limitations section is candid about verifier scope and training cost.\n\nOverall: the empirical contribution is real, the method is clearly described, and the central practical claim—that evaluators should adapt to current-policy response sets—is supported by direct experiments. The theory should be reframed as motivation or made rigorous. This deserves a serious referee, not a desk reject.","headline":"The empirical core is real and the co-evolution story mostly holds, but the formal gap-preservation guarantee is assumed into existence in Appendix J—read the theory as motivation, not proof.","tokens_in":760,"tokens_out":1558,"would_cite":true,"duration_ms":42830,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Relative evaluator score gaps on current-policy response sets are the signals that drive LLM policy updates, so evaluators must co-evolve with the policies they supervise; DynamicRubric does this with response-set-conditioned rubrics.","keywords":["LLM post-training","evaluator-guided policy optimization","rubric-based evaluation","relative score gaps","response-set-conditioned evaluation","evaluator-policy co-evolution","reward model","reinforcement learning from feedback"],"falsifier":"On held-out prompts from a fixed policy, compute Δgap (Eq. 2) for three generator variants: full objective, discriminability-only, and anchor-only. If the full objective's training value increases while its Δgap does not, or if a variant with lower training objective has higher Δgap, the bound in Theorem 1 is violated.","tokens_in":24985,"feed_emoji":"🎯","tokens_out":5400,"duration_ms":107226,"temperature":0.7,"pith_summary":"This paper argues that the useful signal in evaluator-guided LLM post-training is not an evaluator's absolute score but the relative score gap between candidate responses sampled from the current policy. It proves a basic identity: the local gain from shifting probability mass from one response to another equals exactly the evaluator's score gap, so those gaps are the optimization signals. It then proposes DynamicRubric, which co-evolves a rubric generator with the policy: the generator writes weighted binary rubric items conditioned on the current response set, a verifier applies them, and the aggregated scores supervise the policy. With 8B backbones, the resulting policies beat supervision from a 70B reward model and a 235B static rubric generator, and the recipe is deployed in production. If correct, evaluator design should focus on preserving fine-grained discriminability on the policy's own response distribution, and evaluators must be re-adapted as the policy moves.","feed_headline":"8B dynamic-rubric supervisor beats 70B reward-model training","feed_subtitle":"Relative score gaps between close responses drive policy updates, so evaluators should evolve with the policy they supervise.","key_machinery":"The carrying identity is E(x,y+)-E(x,y-) as a directional derivative, making relative score gaps the policy optimization signal. DynamicRubric's evaluator is the rubric-based score E_phi(x,y|C)=sum w_m v_m(y;C)/sum w_m, where rubric items and weights are generated conditioned on the candidate set; training maximizes the weighted Bernoulli variance of verifier outcomes over the set (discriminability) plus a pairwise log-sigmoid anchor-ranking reward, which the paper shows lower-bounds Δgap under an assumed calibration condition. The policy is then optimized with group-normalized advantages derived from those scores.","core_discovery":"The central claim is the score-gap identity (Eq. 7): conditioned on a prompt and a set of responses sampled from the current policy, the directional derivative of the policy's expected evaluator score along the direction that moves probability mass from a worse response to a better one is exactly the evaluator score gap between them. Thus Δgap(E;θ) is the expected local optimization signal an evaluator exposes. Collapsed or misordered gaps mean weak or misleading policy supervision. Because policy updates change the response distribution, an evaluator must condition on the current response set and be updated along with the policy. DynamicRubric instantiates this: a generator produces weighte","pith_inferences":["The strongest theoretical guarantee rests on Eq. (45), an aggregate anchor-calibration condition that the paper assumes; if it fails on real ordered-pair distributions, maximizing the stated evaluator objective need not increase Δgap, even if the empirical method still works.","A natural testable extension is to add a direct Δgap surrogate estimated from current-policy ordered pairs alongside the discriminability and anchor objectives, making the training signal match the theory more literally.","The score-gap view suggests that reward-model overoptimization in standard RLHF may often be evaluator staleness: a reward model trained on pre-update responses loses discriminability on post-update responses, so frequent evaluator re-adaptation could be a cheap fix.","Since online metrics are reported textually without exact values, the production deployment is weaker evidence than the benchmark results, though consistent with the paper's central claim."],"forward_implications":["Evaluator-guided post-training should be judged by how well it preserves relative score gaps on the current policy's response distribution, not by absolute-score accuracy.","Reward models and judges should be re-adapted to each policy's response sets; the paper's continuous co-evolution experiments show a stale evaluator gives weaker supervision.","Response-set-conditioned evaluation strictly generalizes prompt-only evaluation and is no worse in optimal local ranking loss.","Static rubrics written before seeing candidate responses saturate; dynamic rubrics that target within-set differences sustain policy improvement.","The pipeline can be run with a single 8B backbone for generator, verifier, and policy, replacing larger reward models and static rubric generators."],"fun_headline_variants":["Why evaluator score gaps drive LLM policy gains","Co-evolving evaluators beat 70B static reward models","The score-gap identity: key to training better LLMs","DynamicRubric: evaluators that adapt with policies","8B adaptive evaluator outranks 70B reward model"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is Eq. (45): the paper assumes the anchor-ranking objective controls the combined directional and coverage residuals of the discriminability reward, but it does not prove this condition for its actual ordered-pair distribution, anchor data, or trained generator class.","fun_headline_variants_meta":{"raw":{"variants":["Why evaluator score gaps drive LLM policy gains","Co-evolving evaluators beat 70B static reward models","The score-gap identity: key to training better LLMs","DynamicRubric: evaluators that adapt with policies","8B adaptive evaluator outranks 70B reward model"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000808,"raw_usage":{"total_tokens":3395,"prompt_tokens":768,"completion_tokens":2627,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":2545}},"tokens_in":512,"tokens_out":2627,"duration_ms":17213,"temperature":1.0,"reasoning_tokens":2545,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T10:49:00.138984+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On held-out prompts from a fixed policy, compute Δgap (Eq. 2) for three generator variants: full objective, discriminability-only, and anchor-only. If the full objective's training value increases while its Δgap does not, or if a variant with lower training objective has higher Δgap, the bound in Theorem 1 is violated.","supporting_citations":[],"review_version":1}