Pith. sign in

REVIEW 2 major objections 5 minor 35 references

CoRE: Consensus Rewards via Equilibrium for Test-Time Reinforcement Learning

T0 review · 2 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CoRE replaces majority voting in test-time reinforcement learning with an equilibrium-based consensus over a graph of sampled roll-outs, yielding a refined pseudo-label, graded rewards, and a cohesiveness gate.

desk verdict A promising consensus-reward method with broad empirical results, but the headline theory of confidence lowering the recovery threshold does not hold as proven. read the letter →

arxiv 2608.09324 v1 pith:ZKLGGKAN submitted 2026-08-10 cs.AI

Ambuj Mehrish , Sebastiano Vascon This is my paper · ORCID
classification cs.AI
keywords test-timereinforcementlearningconsensusrewardsdominantsetsreplicatordynamicsmajorityvotingconfidencecalibrationself-supervisedrewardlanguagemodelreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Test-time reinforcement learning has to invent its own reward on unlabeled questions, and the standard choice—majority voting over sampled roll-outs—throws away the model's confidence and the structure of its reasoning, rewarding every majority-matching roll-out identically and discarding correct minorities. CoRE replaces the vote with dominant-set extraction on a graph whose edges combine answer agreement, reasoning similarity, and generation confidence, producing a refined pseudo-label, a graded per-roll-out reward, and a per-question cohesiveness gate. The paper claims this strictly generalizes majority voting, gives a sharp threshold for when a correct minority can overturn a larger wrong plurality, and shows that confidence calibration lowers that threshold multiplicatively. Across seven backbones and five benchmarks, CoRE improves the untrained base by +21.7 points on average versus +20.4 for majority-vote TTRL, and reaches the vote's plateau accuracy in 54–70% fewer steps.

What carries the argument

The machinery is the roll-out graph plus dominant-set extraction. Roll-outs are nodes; an edge exists only between roll-outs with the same extracted answer, weighted by a reasoning-similarity kernel (TF-IDF n-gram cosine) with a floor $\kappa$, and the affinity matrix is then confidence-calibrated by $A' = D^{1/2} A D^{1/2}$, where node weights are exponentials of relative mean token log-probability. A dominant set—the weighted-graph generalization of a maximal clique that appears as the support of a strict local maximizer of $x^\top A' x$ on the simplex—is found by iterating replicator dynamics from the simplex barycenter, so initial mass is proportional to answer-class size. The equilibrium mass $x^*$ gives the pseudo-label (answer with greatest mass), graded rewards $R_i = (A x^*)_i / \max_j (A x^*)_j$, and a cohesiveness gate $w_q = x^{*\top} A x^*$ that scales the GRPO loss. Block-diagonality of the affinity over answer classes lets the analysis decompose the dynamics into within-class refinement and between-class mass competition, which is what yields the extraction threshold.

What would settle it

Take a benchmark where the model's wrong answers are engineered to be more confident than its correct ones (for example, by adding confident-looking plausible distractors) and run CoRE and majority-vote TTRL under the same GRPO loop; if the mean-confidence gap $\bar w_w - \bar w_*$ is consistently positive, the theory predicts CoRE's accuracy should fall below the vote's, and the measured gap should correlate with the size of the loss.

Watch

Extended reading notes

Core claim

The central claim is that consensus should be found, not counted: the $N$ roll-outs for a question form a weighted graph, and the equilibrium of replicator dynamics on that graph selects a dominant set of mutually supporting trajectories. With answer-only affinity and uniform confidence, the selected class is exactly the plurality, so majority voting is a special case (Prop. 1). For a correct clique of size $n^*$ and affinity $c^*$ facing a wrong clique of size $m > n^*$ and affinity $c_w$, barycenter-initialized replicator dynamics recover the correct class iff $c^* > \frac{m-1}{n^*-1} c_w$ (Prop. 2). Confidence calibration rescales each class's internal score by the mean of its square-root weights, multiplying the recovery threshold by the ratio of the wrong-to-correct mean weights; when correct roll-outs are more confident, the threshold drops multiplicatively, turning otherwise infeasible minority recoveries into feasible ones (Prop. 3). The paper tests these predictions across seven backbones and five benchmarks, reporting average gains of +21.7 points over the untrained base versus +20.4 for majority-vote TTRL, wins of up to +7.5 points where agreement is contested, and 54–70% faster convergence to the vote's plateau.

Load-bearing premise

The recovery threshold in Proposition 3 assumes that correct answer clusters are on average more confident than incorrect clusters; if wrong answers are systematically overconfident, confidence weighting amplifies the wrong cluster and CoRE can underperform majority-vote TTRL.

Editorial extensions

If this is right

  • Test-time RL can use CoRE's graded rewards and cohesiveness gate with the same $N=64$ roll-outs and the same GRPO loop, so the better supervision costs no extra sampling.
  • Where a coherent, confident correct minority exists, CoRE recovers questions the vote gets wrong; where roll-outs are nearly unanimous, CoRE reduces to voting and does not hurt.
  • The extraction threshold gives a diagnostic for when to trust the pseudo-label: compare $n^*$, $c^*$, $m$, $c_w$, and the confidence gap before adapting on a question.
  • Because CoRE reaches the vote's plateau in 54–70% fewer steps, it also reduces the compute budget of test-time adaptation.
  • Confidence-weighted voting alone (CISC) and graph structure alone (EC) each perform near the vote; the multiplicative combination is what creates the recovery regime.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the confidence premise is weak in a target domain, a practitioner could test the sign of $\bar w_* - \bar w_w$ on a held-out set before committing to CoRE; this is a direct, cheap check the paper's theory implies.
  • The same equilibrium-consensus read-out could be reused outside GRPO, for example to build verifier-free rerankers or to select which unlabeled questions are informative enough to train on at all.
  • Replacing the TF-IDF reasoning kernel with hidden-state similarity might extend CoRE's recovery to semantically fluent but lexically dissimilar wrong answers, a direction the paper flags as future work.
  • The threshold formula suggests CoRE's gains should transfer to new models and benchmarks only when the measured operating point (size and affinity ratios, confidence gap) lies inside the recovery region; reporting those four quantities alongside accuracy would make the method's claims easier to test.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes CoRE, a test-time reinforcement learning reward that replaces majority voting over N rollouts with dominant-set extraction on a graph whose edges combine answer agreement, lexical reasoning similarity, and generation confidence. The method produces a refined pseudo-label, graded per-rollout rewards, and a question-level cohesiveness gate, and plugs into GRPO. The theoretical section claims that majority voting is a special case (Prop. 1), gives a threshold for a correct minority to overturn a wrong plurality under two-class uniform-clique assumptions (Prop. 2), and claims that confidence calibration multiplicatively lowers that threshold (Prop. 3). The evaluation covers seven backbones and five benchmarks (42 model-benchmark cells, three seeds each) and reports average gains of +21.7 points over the untrained base versus +20.4 for majority-vote TTRL, plus faster convergence to the vote's plateau accuracy.

Significance. If the main claims held, CoRE would be a useful and well-motivated contribution: it upgrades the consensus function at no extra rollout cost, includes a clean reduction to voting at kappa=1, and provides an extensive benchmark comparison with offline pseudo-label evidence that the gains are not purely optimization noise. The monotone ascent and constant-shift lemmas are sound, and the authors are explicit about competence-floor and saturation regimes. However, the headline theoretical claim, Proposition 3 and the associated exponential threshold reduction, is not established, and the counterexample in this report shows it can fail under the paper's own confidence premise. The empirical method may still be valuable, but the paper currently overclaims its theoretical support.

major comments (2)
  1. [Appendix G / §3.3, Eqs. (6)–(7)] Proposition 3 as stated is not established. The proof evaluates the calibrated quadratic form at the uniform vector on S, but with heterogeneous w_i the uniform vector is not a fixed point of the within-class replicator flow (Lemma 5, Eq. (8)); the score that enters the mass-competition threshold of Prop. 2 is the limit of that flow, not the uniform-vector value in Eq. (6). Even taking Eq. (6) as the score, substituting it into n_* mu'_* > m mu'_w gives the exact threshold c_*/c_w > [m bar_mu_w^2 - bar_nu_w] / [n_* bar_mu_*^2 - bar_nu_*], not Eq. (7), which drops the -bar_nu_S/|S| terms. Concrete counterexample in the alpha=0 setting of Props. 2-3: n_*=8, m=24, c_*=3.3 c_w; correct-clique weights are four at 1 and four at about 0.0004, wrong-clique weights constant 0.25. Then bar_mu_*^2 = 0.2601 > 0.25 = bar_mu_w^2, so the paper's confidence premise holds; Prop. 2's threshold is 23/7 ≈ 3.29 and Eq. (7) gives ≈ 3.16, both predicting recovery, but the exact inequality requires c_*/c_w > ≈ 3.64 (and if the flow concentrates on the four high-confidence nodes the required ratio is even larger, ≈ 3.83). The wrong plurality wins, so the claimed e^{-delta/tau} multiplicative reduction and the '8-vs-24 threshold falls below c_w' statement in §3.3 are not guaranteed. This directly contradicts the abstract's claim that confidence calibration provably lowers the threshold multiplicatively.
  2. [§6 / Fig. 4] The recovery-region analysis applies the two-class, alpha=0 threshold of Prop. 2 to experiments that use alpha=0.1 and many answer classes per question. The only support for this transfer is the sentence in Appendix F stating that the threshold was 'verified to four decimal places on the exact pipeline'; no verification details, diagnostics, or code are provided. Because the headline claim that CoRE's gains and failures follow the predicted regimes depends on this transfer, the paper should either provide the numerical verification or a robustness bound, or explicitly re-label Fig. 4 and the surrounding discussion as heuristic.
minor comments (5)
  1. [§5 / Table 1] The sentence 'CoRE improves the no-RL base by +25.0 points on average' conflicts with Table 1's reported CoRE mean Δ of +21.6; please correct the number or clarify which subset it refers to.
  2. [§5] The claim that CoRE is the strongest RL method in every model family should be qualified: in several individual cells CISC or Majority is higher (e.g., Qwen2.5-Math-7B on MATH-500, Qwen3-8B on MATH-500, Llama-3.1-8B on AIME), so 'best arm on every model' overstates the tables.
  3. [Appendix A] The reproducibility claim cannot be checked because the public repository is listed as 'TBA'; please provide the actual URL or a statement of when it will be available.
  4. [§2] The description of CISC as 'pre-registered' is asserted without evidence; either provide the registration document or soften the wording to 'fixed in advance'.
  5. [Appendix H] Example A includes a diagram that is omitted from the manuscript, making the geometry question impossible to verify; please include the diagram or remove the example.

Circularity Check

1 steps flagged · score 2.0 of 10

No significant circularity: majority-voting reduction (Prop. 1) and extraction threshold (Prop. 2) are genuine theorems of the method's own equations, and the empirical arms (Majority/EC/CISC) are independent baselines on external benchmarks.

  1. fitted input called prediction [§6 Recovery region / Fig. 4 and Appendix H (Qualitative recovery analysis)]
    "confidence calibration lowers it by e−δ/τ , yielding ≈0.29r here. Model–benchmark pairs below this operative threshold are recoverable; those above it are not. ... Measured cells below their bar are recovered wins; those above are losses. ... On the recovered questions the correct cluster is on average +0.78 nats more confident than the wrong plurality."

    The operative threshold used to classify cells as recoverable is fixed by the confidence gap δ measured on the same data the analysis then validates: 'yielding ≈0.29r here' uses per-cell measured gaps, and Appendix H measures the gap specifically on the recovered questions (+0.78 nats). The cell positions (r, c*/c_w) and the win/loss outcomes are computed from the same roll-out statistics. Thus the claimed prediction that 'cells below their bar are recovered wins' is partly an in-sample restatement: the quantity the theory says should explain recovery is measured on the very instances whose recovery it is invoked to explain. The circularity is mild because δ is not fit to the win/loss labels and the functional form of the threshold is derived, not post-hoc; the main derivation (Props.

full rationale

The central derivation chain is not circular. CoRE is defined by its own equations (§3.2): the affinity kernel (1), the confidence congruence (2), the self-penalty and shift, and the replicator extraction (3); the announced predictions (i)–(iii) in §3.3 are consequences of the construction rather than equivalences to its inputs. Prop. 1 (majority voting as the κ=1, w≡e special case) is a genuine special-case theorem, not a covert premise, and the paper verifies it as a unit test. Props. 2–3 are derived through the block-decomposition mass competition (Lemmas 4–5), not assumed. The empirical claims are tested against independent controls (Majority TTRL, EC, CISC) trained in the same GRPO loop on external benchmarks (AMC/AIME/MATH/GPQA), so the reported +21.7 vs +20.4 improvement is not self-referential. There are no load-bearing self-citations: the authors cite external work (Pavan & Pelillo 2007, Baum 1967, Zuo et al. 2026) and invoke no uniqueness theorem from their own prior work. The mild circularity is confined to §6 and Fig. 4, where the operative threshold's free quantity (the confidence gap δ) is measured on the same cells, and in Appendix H on the same recovered questions, whose recoverability it is then used to explain; this is an in-sample consistency check presented with predictive language, but it is not statistically forced because the outcome labels do not determine δ. Additionally, a correctness issue (not circularity) exists in Prop. 3: Eq. (6) is exact only at the uniform vector, and Eq. (7) follows only after dropping the -ν̄_S/|S| terms, as the proof's own '≈' and Jensen remark concede. The constructed 8-vs-24 weight profile shows the exact uniform-score threshold can exceed both Prop. 2's threshold and Eq. (7), so 'confidence provably lowers the threshold' is not guaranteed in full generality. This is a proof gap affecting the stated theorem, not a reduction of the result to its inputs, and therefore does not raise the circularity score above 2.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper's central claim rests on three hand-chosen hyperparameters (kappa, tau, alpha) plus N and the support threshold, none of which are fitted to hold-out data but all of which could have been tuned. The theory uses standard dominant-set results and simplifying uniform-clique assumptions; the key empirical premise is that correct roll-outs are, on average, more confident than incorrect ones.

free parameters (6)
  • kappa (reasoning kernel floor) = 0.1
    Floor in edge weight A_ij = K_ans (kappa + (1-kappa) K_rsn); controls how much reasoning similarity matters. kappa=1 recovers majority voting (Prop. 1).
  • tau (confidence temperature) = 0.25
    Temperature in node weights w_i = exp((c_i - max_j c_j)/tau); controls strength of confidence calibration. Smaller tau amplifies confidence gaps.
  • alpha (singleton self-penalty) = 0.1
    Diagonal penalty -alpha I used to avoid singleton consensus; alpha>0 required by Lemma 3.
  • C (nonnegativity shift) = alpha (0.1)
    Shift C e e^T ensures nonnegative affinity matrix; set equal to alpha in all experiments.
  • N (roll-outs per question) = 64
    Number of samples per question; follows TTRL baseline. Pseudo-label accuracy improves with N (Fig. 2).
  • support threshold rho = 1/(10N)
    Threshold on equilibrium mass for defining consensus support; used in read-outs.
assumptions (5)
  • standard math Baum-Eagon inequality and dominance of replicator dynamics
    Used in Lemma 2 to guarantee monotone ascent and convergence of the replicator update; cited to Pavan and Pelillo (2007).
  • domain assumption Answer kernel gates all edges (block-diagonal affinity)
    Assumed in Eq. (1) and throughout §3.3; different answers share no edge regardless of prose similarity.
  • domain assumption TF-IDF n-gram vectors capture reasoning similarity
    The reasoning kernel K_rsn uses cosine similarity of character 4-gram TF-IDF vectors; assumes lexical overlap reflects shared derivation steps.
  • domain assumption Correct clusters have higher mean confidence than incorrect clusters
    Required for Prop. 3's multiplicative threshold reduction; acknowledged in §6 as the premise that can fail when incorrect answers are overconfident.
  • domain assumption Two uniform cliques approximation for the threshold analysis
    Props. 2-3 analyze the ideal case of two answer classes, each a uniform clique with constant affinity and confidence; real roll-outs have heterogeneous within-class structure.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CoRE: Consensus Rewards via Equilibrium for Test-Time Reinforcement Learning." pith.science (2026). https://pith.science/paper/ZKLGGKAN

@misc{pith2026260809324,
  author       = {Pith},
  title        = {Pith review of: CoRE: Consensus Rewards via Equilibrium for Test-Time Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZKLGGKAN}},
  note         = {Machine review of arXiv:2608.09324}
}
abstract

On unlabeled test data, reinforcement learning lacks a ground-truth reward; test-time RL methods derive one from the model's own roll-outs, rewarding those that match the majority vote over $N$ sampled answers. That vote discards a correct answer whenever it is a minority and scores every majority-matching roll-out identically. We replace it with \emph{CoRE} (Consensus Rewards via Equilibrium): the $N$ roll-outs form a graph whose edges combine answer agreement, reasoning similarity, and generation confidence, and replicator dynamics extract its dominant set, yielding a refined pseudo-label, a graded per-roll-out reward, and a per-question cohesiveness gate. CoRE strictly generalizes voting: majority voting is recovered as a special case; a block-value analysis gives a sharp threshold for when consensus recovers a correct minority against a larger wrong plurality; and confidence calibration provably lowers that threshold multiplicatively. Across seven backbones and five benchmarks (42 model--benchmark cells, three seeds each), \emph{CoRE} improves the untrained base by $+21.7$ points on average versus $+20.4$ for majority-vote TTRL, wins wherever agreement is contestable with margins over the vote of up to $+7.5$ points, and reaches the voting baseline's plateau accuracy in $54$--$70$\% fewer steps. Consensus, not counting: treating the roll-out group as a graph rather than a ballot box turns a brittle vote into a calibrated, graded, self-supervised reward at no extra roll-out cost.

Figures

Figures reproduced from arXiv: 2608.09324 by the authors.

Figure 1
Figure 1. CoRE overview. Roll-outs become a graph whose edges combine answer agreement, reasoning similarity, and calibrated confidence; replicator dynamics extract the dominant set, yielding a refined pseudo-label, graded per-roll-out rewards, and a cohesiveness gate that plug directly into GRPO. reduces exposure to reward over-optimization (Gao et al., 2023; Skalse et al., 2022; Pan et al., 2022; Amodei et al., 2016). 3 CoR… view at source ↗
Figure 2
Figure 2. Offline pseudo-label accuracy versus roll-out count 𝑁. CoRE outperforms majority voting at every 𝑁, indicating a better training signal rather than optimization noise [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Validation pass@1 vs. training step (means over 3 seeds). CoRE reaches TTRL’s plateau accuracy in 54–70% fewer steps. 57% fewer on DeepSeek-Math-7B (AMC), and 54% fewer on Qwen2.5-Math-1.5B (MATH). Unlike the binary majority reward, which assigns the same target to every consensus member, the equilibrium reward 𝑅𝑖 = (𝐴𝑥∗ )𝑖/max𝑗(𝐴𝑥∗ )𝑗 grades each roll-out by its support within the consensus. This denser signal appe… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The recovery region: outnumbering ratio 𝑟 = 𝑚/𝑛 ∗ vs. required coherence advantage 𝑐 ∗ /𝑐𝑤. The majority-vote start raises the bar to ≈𝑟 (red); confidence pulls it back by 𝑒 −𝛿/𝜏 (green). Measured cells below their bar are recovered wins; those above are losses. raise …
Figure 5
Figure 5. Figure 5: (a) On MATH L4/L5, CoRE beats the confidence-only control CISC on 13/14 cells. (b) Graph and confidence interact synergistically: adding the graph to CISC helps, removing confidence from CoRE hurts. of confidence and graph structure in CoRE: each signal is most informa…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 3 canonical work pages

  1. [3]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    arXiv

  2. [6]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    arXiv

  3. [8]

    Openai o1 system card.arXiv preprint arXiv:2412.16720,

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720,

    arXiv

  4. [9]

    Mistral 7b

    AlbertQJiang,AlexandreSablayrolles,ArthurMensch,ChrisBamford,DevendraSinghChaplot,Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825,

    arXiv

  5. [10]

    Language models (mostly) know what they know.arXiv preprint arXiv:2207.05221,

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know.arXiv preprint arXiv:2207.05221,

    arXiv

  6. [12]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InInternational Conference on Learning Representations, volume 2024, pages 39578–39601,

    work page 2024

  7. [13]

    Teaching models to express their uncertainty in words

    Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334,

    arXiv

  8. [14]

    Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783,

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783,

    arXiv

Show all 35 references
  1. [16]

    The effects of reward misspecification: Mapping and mitigating misaligned models.arXiv preprint arXiv:2201.03544,

    14 CoRE: Consensus Rewards via Equilibrium Preprint Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models.arXiv preprint arXiv:2201.03544,

    arXiv

  2. [18]

    Gpqa: A graduate-level google-proof q&a benchmark.arXiv preprint arXiv:2311.12022,

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark.arXiv preprint arXiv:2311.12022,

    arXiv

  3. [21]

    Spurious rewards: Rethinking training signals in rlvr.arXiv preprint arXiv:2506.10947,

    Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, et al. Spurious rewards: Rethinking training signals in rlvr.arXiv preprint arXiv:2506.10947,

    arXiv

  4. [22]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024a

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024a. Zhihong Shao, Peiyi Wang, Qihao Zhu, R...

    arXiv

  5. [23]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,

    arXiv

  6. [25]

    Solving math word problems with process-and outcome-based feedback.arXiv preprint arXiv:2211.14275,

    Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback.arXiv preprint arXiv:2211.14275,

    arXiv

  7. [26]

    Tent: Fully test-time adaptation by entropy minimization.arXiv preprint arXiv:2006.10726,

    Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization.arXiv preprint arXiv:2006.10726,

    2006 arXiv

  8. [28]

    Self-trainingwithnoisystudentimproves imagenet classification.arXiv preprint arXiv:1911.04252,

    QizheXie,Minh-ThangLuong,EduardHovy,andQuocVLe. Self-trainingwithnoisystudentimproves imagenet classification.arXiv preprint arXiv:1911.04252,

    1911 arXiv

  9. [29]

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement.arXiv preprint arXiv:2409.12122,

    arXiv

  10. [30]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    arXiv

  11. [31]

    Self-rewarding language models.arXiv preprint arXiv:2401.10020, 3:2,

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models.arXiv preprint arXiv:2401.10020, 3:2,

    arXiv

  12. [33]

    Learning to reason without external rewards.arXiv preprint arXiv:2505.19590,

    Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning to reason without external rewards.arXiv preprint arXiv:2505.19590,

    arXiv

  13. [34]

    B Use of Large Language Models Large language models were used only to improve grammar and clarity in author-written text

    17 CoRE: Consensus Rewards via Equilibrium Preprint A Code and Reproducibility For reproducibility, we release the full codebase, including the roll-out graph construction, the replicator- dynamics consensus and read-outs, the GRPO training loop, and all evaluation scripts, in...

    2007

  14. [35]

    role": "user

    □ Equation(9)is the object the paper’s slogan refers to: the classes play a coordination game for mass, and the barycenter start enters through𝑠𝑎(0)=𝑛 𝑎/𝑁—the initialization encodes the ballot. Proof of Proposition 1.At 𝜅=1, 𝑤≡𝑒 , every class is a unit-affinity clique, so by L...

    2024

  15. [1965]

    Training language models to follow instructions with human feedback.arXiv preprint arXiv:2203.02155,

    LongOuyang,JeffWu,XuJiang,DiogoAlmeida,CarrollLWainwright,PamelaMishkin,ChongZhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.arXiv preprint arXiv:2203.02155,

    arXiv

  16. [1978]

    KimiTeam,AngangDu,BofeiGao,BoweiXing,ChangjiuJiang,ChengChen,ChengLi,ChenjunXiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599,

    arXiv

  17. [1988]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    arXiv

  18. [1997]

    Large language monkeys: Scaling inference compute with repeated sampling.arXiv preprint arXiv:2407.21787,

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling.arXiv preprint arXiv:2407.21787,

    arXiv

  19. [2007]

    Maximizing confidence alone improves reasoning.arXiv preprint arXiv:2505.22660,

    Mihir Prabhudesai, Lili Chen, Alex Ippoliti, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. Maximizing confidence alone improves reasoning.arXiv preprint arXiv:2505.22660,

    arXiv

  20. [2017]

    Can large reasoning models self-train?arXiv preprint arXiv:2505.21444,

    Sheikh Shafayat, Fahim Tajwar, Ruslan Salakhutdinov, Jeff Schneider, and Andrea Zanette. Can large reasoning models self-train?arXiv preprint arXiv:2505.21444,

  21. [2020]

    Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171,

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171,

    arXiv

  22. [2021]

    The entropy mechanism of reinforcement learning for reasoning language models.arXiv preprint arXiv:2505.22617,

    Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models.arXiv preprint arXiv:2505.22617,

    arXiv

  23. [2022]

    Making language models better reasoners with step-aware verifier

    Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making language models better reasoners with step-aware verifier. InProceedings of the 61st annual meeting of the ACL (volume 1: long papers), pages 5315–5333, 2023a. Ziniu Li, Tian Xu, Yus...

    arXiv

  24. [2023]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    13 CoRE: Consensus Rewards via Equilibrium Preprint Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    arXiv

  25. [2024]

    Concrete problems in ai safety.arXiv preprint arXiv:1606.06565,

    Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety.arXiv preprint arXiv:1606.06565,

    arXiv

  26. [2025]

    Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    arXiv

  27. [2026]

    Co-rewarding: Stable self-supervised rl for eliciting reasoning in large language models

    16 CoRE: Consensus Rewards via Equilibrium Preprint Zizhuo Zhang, Jianing Zhu, Xinmu Ge, Zihua Zhao, Zhanke Zhou, Xuan Li, Xiao Feng, Jiangchao Yao, and Bo Han. Co-rewarding: Stable self-supervised rl for eliciting reasoning in large language models. arXiv preprint arXiv:2508.00410,

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.