Pith. sign in

REVIEW 3 major objections 4 minor 69 references

SERPO claims that open-ended test-time RL can improve without labels by co-evolving response archives, query-specific rubrics, and policy weights, reporting in-domain gains of roughly 20 points on HealthBench and ResearchQA.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 01:41 UTC pith:MULZLS7H

load-bearing objection A serious, empirically careful TTRL paper for open-ended generation whose main caveat is the self-referential reward loop; worth refereeing, but the authors should measure internal reward vs external quality agreement. the 3 major comments →

arxiv 2607.26873 v2 pith:MULZLS7H submitted 2026-07-29 cs.CL

SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning

Jianze Wang , Kunwang Zheng , Ying Liu , Yu Cao , Qilong Zhang , Jinlong Chen , Hua Yang , Qianglong Chen This is my paper
classification cs.CL
keywords test-time reinforcement learningopen-ended generationrubric evolutionself-evolving rewardslabel-free adaptationGood-Normal-Bad archivesprobabilistic judgingtransductive adaptation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Test-time reinforcement learning is currently built on answer voting, which assumes responses can be reduced to a shared canonical answer. SERPO claims this assumption is unnecessary: for open-ended prompts, rewards can be built from a closed loop that co-evolves ordered Good–Normal–Bad response archives, query-specific rubrics (checklists of atomic quality criteria), and the policy itself, with no labels, no reference answers, and no external judge during adaptation. On two base models and two in-domain benchmarks — medical advice (HealthBench) and scholarly QA (ResearchQA) — the loop improves scores by up to 20.63 and 20.31 points over the base models, raises the six-benchmark macro-average by up to 8.06 points, and transfers positively to all four held-out benchmarks. If true, test-time RL no longer needs a canonical-answer assumption to improve open-ended generation, and self-generated criteria can substitute for most of the gain of a privileged external judge.

Core claim

The central claim is that a policy can keep improving on open-ended test prompts without any external signal if the reward itself evolves. From each prompt's rollouts, SERPO stores the most separated Good–Normal–Bad triple (maximising the product of score gaps), lets a frozen generator propose atomic criteria that explain the contrasts, keeps criteria that vary across the archive and agree with its ordering, deletes persistently weak criteria, and converts the surviving criteria's Pass/Fail token probabilities into a calibration-weighted scalar reward for a GRPO actor update. The new rollouts refresh the archive and the rubric, closing the loop. Across four in-domain model–benchmark configur

What carries the argument

The load-bearing mechanism is the three-way co-evolution loop. (1) Response evolution: each prompt keeps bounded Good–Normal–Bad archives; at each encounter the judge scores rollouts with the current rubric and the most separated ordered triple (maximising (s_G−s_N)(s_N−s_B)(s_G−s_B)) enters the archives. (2) Rubric evolution: a frozen generator proposes atomic criteria that explain G–N–B contrasts; each criterion gets utility d_m = v_m·a_m, where v_m is response-score variance and a_m is G–N–B order agreement, and the bottom quarter by utility is at risk of deletion after three consecutive strikes. (3) Policy evolution: the judge's True/False token log-probabilities give a [0,1] satisfactio

Load-bearing premise

The whole loop assumes that a rubric which the model itself generates and judges, using the model's own outputs as evidence, tracks true response quality — not merely the frozen judge's stylistic preferences — as the policy drifts away from the base model; there is no external check inside the loop.

What would settle it

Run SERPO on the same HealthBench prompts but replace the frozen judge with a deliberately biased one (e.g., rewards short, non-escalating answers, or reverses one criterion's polarity). If the gain on the official reporting rubric collapses or flips while the biased judge's satisfaction rises by the same margin, the loop is optimizing self-consistency with the judge, not underlying quality.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Open-ended generation becomes a target for test-time RL: any domain with no canonical answer (medical advice, scientific synthesis, dialogue, planning) can in principle be adapted on unlabeled test prompts alone.
  • Self-generated rubrics recover most of the benefit of an external judge: the 4B model attains roughly three-quarters of the privileged-reference in-domain gain, and the method beats the privileged reference on all eight OOD settings.
  • Gains are not a one-shot epoch-30 artifact: the 45-epoch run keeps improving past the standard budget, and switching evolution from HealthBench to ResearchQA improves both the new and the old benchmark.
  • Rubric evolution and policy evolution are complementary: removing either roughly halves or eliminates the in-domain gain (36% loss from freezing the actor, 12–13% from freezing rubrics or archives), while static rubric guidance alone hurts.
  • Fixed evaluator roles matter: training the judge or the rubric generator reduces performance, supporting a stable frozen reference as the anchor for self-evolution.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The self-referential signal is the soft spot: because the same evolving rubric orders the evidence and defines the reward, the loop could converge to what the frozen judge likes rather than what is true; the paper's own limitation note (§6) concedes this. A testable consequence: if the judge favors a style (verbosity, urgency, hedging), SERPO should drift that way even when it hurts a different ev
  • The recipe generalises beyond two benchmarks: any task where a model can articulate verifiable atomic criteria and a judge can give graded verdicts is a candidate, suggesting extensions to code review, legal analysis, or long-form debate, though the paper does not test these.
  • The query-specific rubrics seem to encode transferable quality dimensions: positive OOD transfer with no extra adaptation implies the rubrics learned on HealthBench/ResearchQA capture domain-general criteria; a natural extension is transferring the rubric pool itself to new prompt sets to accelerate adaptation.
  • The descriptive linear projection (epoch 99 to reach the privileged reference) is not a prediction; it assumes the late-horizon slope holds, but the paper's own curves show rubric-only methods plateau, so sustained improvement likely requires the full three-way loop plus periodic refresh of evidence.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. SERPO is a test-time reinforcement learning method for open-ended generation that operates without reference answers, external reward models, or stronger judges. For each prompt it maintains Good–Normal–Bad response archives, a query-specific rubric pool, and a shared actor. A frozen copy of the deployed model serves both as rubric generator and as probabilistic judge. The loop works as follows: sample rollouts, order them by the current rubric, store the most separated triple in G-N-B archives, refresh criteria that discriminate this ordering, convert criterion-satisfaction probabilities into calibrated scalar rewards, and update the actor with GRPO. The paper evaluates on two Qwen models, two in-domain benchmarks (HealthBench, ResearchQA), and four OOD benchmarks, reporting large gains over base models, a strengthened claim-consensus voting baseline, and a privileged external-judge reference. It also includes ablations, long-horizon evolution, sequential cross-benchmark evolution, and a public code repository.

Significance. If the reported gains are genuine, this is a meaningful step toward TTRL for open-ended tasks, where answer voting is not applicable. The empirical package is solid in several respects: three evaluation seeds with standard deviations (Table 6), complete component ablations (Table 2), a privileged external-judge/official-rubric reference that anchors the gain magnitude, official benchmark splits, and a public code repository. The central weakness is that the internal reward path is fully self-referential: archive ordering, criterion utility, and reward calibration all derive from the frozen base model's own judgments. The paper explicitly concedes (§6) that rubric generation and judging may inherit biases. What is missing is any measurement of how well SERPO's internal reward correlates with an independent quality signal over the course of evolution. This concern is real and load-bearing, because the paper interprets the 20-point gains as improved response quality rather than convergence to the self-consistent preferences of the frozen judge.

major comments (3)
  1. [§4, Eqs. (5)–(7) and Algorithm 1] The reward path is self-referential. Eq. (5) computes archive-ordering scores from the current active rubric, Eq. (6) selects the maximally separated Good/Normal/Bad triple under that score, and Eq. (7) assigns high utility to criteria that order the resulting buckets concordantly. Since the same frozen base model generates and judges criteria, this loop can converge to any self-consistent preference of that model. The paper's own §6 limitation ('Rubric generation and judging may inherit biases from the deployed model') does not resolve the issue. This is load-bearing because the central claim is that SERPO improves response quality, not just self-agreement. I request an analysis reporting the correlation between SERPO's internal rewards (or per-criterion utilities) and an independent quality signal (e.g., GPT-5.1 official-rubric scores on a held-out sample, or human labels) across evolu
  2. [§5, RQ2 and Table 6] The paper claims that all eight OOD settings improve over Base, but several differences are within evaluation noise. For example, Qwen3-4B MedQA is +0.92 (54.73±0.70 vs 53.81±0.55), and Qwen3.5-9B RaR-Science is +2.06 (79.93±0.64 vs 77.87±0.65) with overlapping or near-overlapping standard deviations across three seeds. The paper should report significance tests or a paired/gain analysis across benchmarks, or soften the claim from 'all eight improve' to a transfer trend. This does not undermine the in-domain result, but it is necessary to substantiate RQ2 as stated.
  3. [§5, Baseline Comparison and Table 4] The strengthened voting baselines use G=16 rollouts per prompt while SERPO uses G=8, but the methods differ far more in total judge/verifier calls: SERPO scores each of 8 responses against up to 15 active criteria and periodically invokes rubric generation, so its reward signal is derived from substantially more evidence. Reporting total judge calls or FLOP-matched evolution curves would clarify whether the gains come from the self-evolving rubric mechanism or from additional inference. This is important for the headline comparison to TTRL baselines and for the claim that SERPO is effective under a fixed information budget.
minor comments (4)
  1. [Algorithm 1, line 12] There is a typo: 'i =r^arc' should be 'r_{t,i}^{arc}'.
  2. [Table 1] The table refers to 'light-blue rows' and 'bold marks the best label-free result'; these visual cues may be lost in monochrome print. Please use a more robust encoding.
  3. [§2 Related Work] The novelty statement 'first to combine post-reasoning Boolean verdict probabilities with evolving, query-specific rubrics' is hedged by a concurrent reference to LLM-as-a-Verifier. Please clarify what exactly is claimed as novel relative to that work and to the other rubric-evolution methods listed in the same paragraph.
  4. [§5, Table 6] Some OOD gains are labeled as improvements despite standard deviations overlapping zero effect; consider adding a confidence interval or a footnote for the OOD columns.

Circularity Check

0 steps flagged

No significant circularity: SERPO's benchmark gains are externally measured; the self-referential reward loop is an acknowledged design feature and limitation, not a derived prediction.

full rationale

The paper's central empirical claim—that SERPO improves HealthBench, ResearchQA, and OOD benchmarks—is not derived from the method's internal equations but from a reporting evaluation that is explicitly disjoint from the adaptation signal: GPT-5.1 with official rubrics for HealthBench, ResearchQA, LLMEval-Med, and RaR-Science, and direct answer extraction for MedQA and GPQA-Diamond (§5, Table 1, Appendix D.1). The internal reward path is self-referential by design: Eq. (5) computes archive-ordering scores from the current active rubric, Eq. (6) selects the Good/Normal/Bad triple as the maximally separated triple under that score, and Eq. (7) assigns utility to criteria that agree with that ordering. This is a legitimate concern about reward validity and the paper concedes it in §6: 'Rubric generation and judging may inherit biases from the deployed model.' However, this is not a circular derivation of the reported results. The paper does not claim that internal rubric utility is a proof of external quality; it uses external benchmarks, ablations (Table 2), OOD transfer, length controls, and a privileged external-judge reference as independent checks. There is no load-bearing self-citation chain, no imported uniqueness theorem, and no fitted parameter renamed as a prediction. The self-consistency of the G-N-B loop is a stated design property, not a hidden assumption that makes the headline result true by construction.

Axiom & Free-Parameter Ledger

8 free parameters · 6 axioms · 3 invented entities

The ledger has no continuous parameter fitted to the evaluation benchmarks: §A.3 states the optimization and reward parameters come from the Qwen3 training recipe and standard GRPO practice and are kept fixed across benchmarks. The load-bearing content lives in the assumptions: the frozen self-judge's criterion scores track quality (Eq. 3), the rubric-induced archive ordering is a valid quality ladder (Eqs. 5-7), and the external graders faithfully measure the target quality (§D.1). G-N-B archives, rubric pools, and the verdict-probability interface are internal bookkeeping with no falsifiable handle outside the final benchmark outcomes.

free parameters (8)
  • Rollout group size G = 8 (SERPO); 16 (voting baselines)
    Group size sets GRPO advantage resolution; voting baselines get twice the rollouts (§A.2, App. C).
  • Elimination fraction ζ = 0.25
    Bottom quartile of criteria marked at risk per refresh (§A.3, Algorithm 1).
  • G-N-B archive width W; rubric refresh interval F = 3 visits; 3 visits
    Archive depth and refresh cadence determine the evidence the rubric evolves on (§A.3).
  • Calibration minimum range δ = 0.05
    Below this Good-minus-Bad archive mean range, calibrated scores fall back to raw oriented scores (Eq. 8).
  • Utility tie margin τ; weight floor ε_u; admission margin = 0.05; 0.01; 0.02
    Shape concordance counts (Eq. 7), reward weights, and challenger admission (§B.3).
  • Initial/active criterion caps; new candidates per refresh = 8/15; 5
    Rubric pool size bounds; set by hand (§A.3).
  • Max missing-grade rate; dedup agreement = 0.20; 3 common grades / 0.90
    Fallback-to-zero trigger in reward construction; semantic-deduplication threshold (§B).
  • Claim-consensus threshold κ (baseline) = 0.5
    Fraction of G=16 rollouts that must support a claim for it to enter the consensus set (Eq. 9, App. C.1).
axioms (6)
  • domain assumption A frozen initial-weight copy of the deployed model, prompted as an expert evaluator (rubric generator and judge), yields criterion-satisfaction scores that track true response quality.
    The entire reward signal (Eqs. 3-8) comes from frozen copies of the base model (§4, §A.1); no human or external verification of rubric-criterion quality is reported during adaptation.
  • domain assumption The two-token softmax over the judge's True/False verdict logprobs (Eq. 3) is calibrated enough to serve as a satisfaction probability for GRPO.
    q_J is normalized over only the two verdict tokens; the rest of the distribution is ignored (§4 'Criterion Scoring').
  • ad hoc to paper The archive ordering produced by the current rubric's provisional scores (Eqs. 5-6) is a reliable quality ladder, so selecting criteria concordant with that ordering (Eq. 7) improves real quality.
    This is the self-referential premise of rubric evolution; its only external check is the reporting evaluation, and the paper's Limitations admit the judge may inherit model biases.
  • domain assumption The transductive TTRL protocol — adapting on the very prompts whose final scores are reported — is a valid measure of improvement.
    Defined in §3; ID scores are reported on the adaptation prompts; OOD sets mitigate but are separate.
  • domain assumption GPT-5.1 with official rubrics (HealthBench, ResearchQA, LLMEval-Med, RaR-Science) and direct answer extraction (MedQA, GPQA) faithfully measures the targeted quality.
    All ID and most OOD numbers rest on this external grader (§5, §D.1).
  • standard math Max variance of a [0,1]-valued score is 1/4, justifying the factor 4 in Eq. (7).
    Popoviciu's inequality; elementary.
invented entities (3)
  • G-N-B response archives (per-prompt Good/Normal/Bad FIFO) no independent evidence
    purpose: Store maximally separated rollouts to serve as ordered evidence for rubric evolution and calibration (§4).
    Internal bookkeeping; no external falsifiable handle apart from final benchmark outcomes.
  • Query-specific rubric pools with utilities and elimination strikes no independent evidence
    purpose: Evolving reward criteria per prompt; criteria deleted after three at-risk rounds (§4, §B.3).
    Internal state; verified only through end-task evaluation.
  • Probabilistic verdict-token reward interface (q_J, calibration to archive means) no independent evidence
    purpose: Convert judge's True/False logprobs into continuous rewards (§4, Eqs. 3-8).
    Reward-shaping function, not an independently observed quantity.

pith-pipeline@v1.3.0-alltime-deepseek · 22614 in / 30157 out tokens · 318752 ms · 2026-08-03T01:41:58.504400+00:00 · methodology

0 comments
read the original abstract

Test-time reinforcement learning (TTRL) enables language models to self-evolve at inference time without labeled feedback. Existing methods rely on answer voting and therefore do not extend naturally to open-ended generation, where valid responses cannot be mapped to a shared canonical answer. Without external reward models or stronger judges, adaptation must instead construct reliable rewards from the model's own outputs. We introduce SERPO (Self-Evolving Rubric Policy Optimization), which replaces answer voting with a closed loop that co-evolves response evidence, query-specific rubrics, and policy parameters. Good-Normal-Bad (G-N-B) response evolution organizes maximally separated rollouts into ordered archives; rubric evolution retains criteria that discriminate these archives; probabilistic criterion scoring converts verdict-token likelihoods into reward signals; and policy evolution optimizes the actor with the resulting signals. New actor rollouts then refresh both the archives and rubrics, closing the three-way evolution loop. Across two model configurations, two in-domain benchmarks, and four OOD benchmarks, SERPO improves HealthBench and ResearchQA by up to 20.63 and 20.31 points over the corresponding base models, raises the six-benchmark macro-average by up to 8.06 points, and supports OOD transfer and continued cross-benchmark evolution.

Figures

Figures reproduced from arXiv: 2607.26873 by Hua Yang, Jianze Wang, Jinlong Chen, Kunwang Zheng, Qianglong Chen, Qilong Zhang, Ying Liu, Yu Cao.

Figure 1
Figure 1. Figure 1: Criterion-level evidence for open-ended TTRL. Claim consensus can retain frequent but incomplete advice, whereas SERPO forms G-N-B evidence using evolving query-specific criteria without reference answers. reasoning Boolean verdict probabilities with evolving, query￾specific rubrics for policy optimization; concurrent LLM￾as-a-Verifier independently studies a related fixed-criterion interface (Kwok et al. … view at source ↗
Figure 1
Figure 1. Figure 1: SERPO improves across all in-domain settings, with further gains beyond the standard 30-epoch budget. (a) Evaluation scores for Base, response-vote TTRL, and SERPO after 30 epochs in four in-domain model–benchmark configurations. Error bars show the standard deviation across three evaluation runs; percentages above the SERPO bars report relative gains over Base, and the break indicates a truncated y-axis. … view at source ↗
Figure 2
Figure 2. Figure 2: Overview of SERPO. Here q is the input query, while oi and ri are the i-th policy rollout and its scalar reward. Et, Rt, and θt denote the query-local G-N-B archive state, query-specific rubric state, and shared actor parameters at encounter t, respectively. Criterion utility combines response-score variance vm and G-N-B order agreement am; ∆R denotes the low-utility elimination region. Snowflakes mark fix… view at source ↗
Figure 2
Figure 2. Figure 2: Criterion-level evidence for open-ended TTRL. Claim consensus can retain frequent but incomplete advice, whereas SERPO forms G-N-B evidence using evolving query-specific criteria without reference answers. 1 Introduction Test-time reinforcement learning (TTRL) enables a deployed language model to self-evolve on the test distribution through rewards constructed at inference time. We study open-ended generat… view at source ↗
Figure 3
Figure 3. Figure 3: Qwen3-4B evolution on HealthBench (HB). The [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of SERPO. Here q is the input query, while oi and ri are the i-th policy rollout and its scalar reward. Et, Rt, and θt denote the query-local G-N-B archive state, query-specific rubric state, and shared actor parameters at encounter t, respectively. Criterion utility combines response-score variance vm and G-N-B order agreement am; ∆R denotes the low-utility elimination region. Snowflakes mark fix… view at source ↗
Figure 5
Figure 5. Figure 5: Response-length ablations on Qwen3-4B. Both panels share the same variant rows. Left: the three-benchmark mean [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qwen3-4B evolution from HealthBench (HB) to [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Response-length behavior of Qwen3-4B. Left: six-benchmark macro-average versus mean length over the five bench [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 6
Figure 6. Figure 6: Response-length ablations on Qwen3-4B. Both panels share the same variant rows. Left: the three-benchmark mean [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Response-length behavior of Qwen3-4B. Left: six-benchmark macro-average versus mean length over the five bench [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

69 extracted references · 35 linked inside Pith

  1. [2]

    K.; Wei, J.; Hicks, R

    Arora, R. K.; Wei, J.; Hicks, R. S.; Bowman, P.; Qui \ n onero-Candela, J.; Tsimpourlas, F.; Sharman, M.; Shah, M.; Vallone, A.; Beutel, A.; Heidecke, J.; and Singhal, K. 2025. HealthBench : Evaluating Large Language Models Towards Improved Human Health. arXiv:2505.08775

    Pith/arXiv arXiv 2025

  2. [3]

    Y.; and Yearick, K

    Bay, Y. Y.; and Yearick, K. A. 2026. When More Sampling Hurts: The Modal Ceiling and Correlation Ceiling of Test-Time Scaling. arXiv:2606.28661

    Pith/arXiv arXiv 2026

  3. [6]

    Fang, J.; Hong, Z.; Zheng, M.; Song, M.; Li, G.; Jiang, H.; Zhang, D.; Guo, H.; Wang, X.; and Chua, T.-S. 2026. Rubric-based On-policy Distillation. arXiv:2605.07396

    Pith/arXiv arXiv 2026

  4. [8]

    Gunjal, A.; Wang, A.; Lau, E.; Nath, V.; He, Y.; Liu, B.; and Hendryx, S. 2025. Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. arXiv:2507.17746

    Pith/arXiv arXiv 2025

  5. [9]

    Huang, C.; Chou, S.-Y.; Zhang, Z.; and Cardie, C. 2026 a . Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Text. arXiv:2604.20051

    Pith/arXiv arXiv 2026

  6. [11]

    Jin, D.; Pan, E.; Oufattole, N.; Weng, W.-H.; Fang, H.; and Szolovits, P. 2021. What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical Exams. Applied Sciences, 11(14): 6421

    2021

  7. [12]

    Y.; Shin, J.; Welleck, S.; Neubig, G.; Lee, M.; Lee, K.; and Seo, M

    Kim, S.; Suk, J.; Longpre, S.; Lin, B. Y.; Shin, J.; Welleck, S.; Neubig, G.; Lee, M.; Lee, K.; and Seo, M. 2024. Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models. arXiv:2405.01535

    Pith/arXiv arXiv 2024

  8. [17]

    Yifei ; Chang, A.; Malaviya, C.; and Yatskar, M

    Li S. Yifei ; Chang, A.; Malaviya, C.; and Yatskar, M. 2026. ResearchQA : Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubrics. Transactions of the Association for Computational Linguistics, 14: 1344--1368

    2026

  9. [18]

    Lin, H.; Kuai, Z.; Xue, E.; and Wang, L. 2026. Detecting and Mitigating the Correct-Answer Extinction Window in Test-Time Reinforcement Learning with Majority Voting. arXiv:2605.19444

    Pith/arXiv arXiv 2026

  10. [22]

    P.; Hermann, K.; Welleck, S.; Yazdanbakhsh, A.; and Clark, P

    Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; Gupta, S.; Majumder, B. P.; Hermann, K.; Welleck, S.; Yazdanbakhsh, A.; and Clark, P. 2023. Self-Refine: Iterative Refinement with Self-Feedback. arXiv:2303.17651

    Pith/arXiv arXiv 2023

  11. [24]

    Nguyen, M.; Nguyen, Q.; and Vuong, P. 2026. Recursive Self-Evolving Agents via Held-Out Selection. arXiv:2606.28374

    Pith/arXiv arXiv 2026

  12. [25]

    OpenAI . 2025. GPT-5.1 Instant and GPT-5.1 Thinking System Card Addendum. https://openai.com/index/gpt-5-system-card-addendum-gpt-5-1/. Accessed: 2026-07-29

    2025

  13. [26]

    Qwen Team . 2025. Qwen3-4B-Instruct-2507 Model Card. https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507. Accessed: 2026-07-29

    2025

  14. [27]

    Qwen Team . 2026 a . Qwen3.5-9B Model Card. https://huggingface.co/Qwen/Qwen3.5-9B. Accessed: 2026-07-29

    2026

  15. [28]

    Qwen Team . 2026 b . Qwen3.6-27B Model Card. https://huggingface.co/Qwen/Qwen3.6-27B. Accessed: 2026-07-29

    2026

  16. [30]

    Rezaei, M.; Mahmoud, A.; Wang, Z.; Tyagi, U.; Gosai, A.; Dumitru, R.-G.; Sabharwal, A.; Liu, B.; and He, Y. 2026. Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers. arXiv:2606.12507

    Pith/arXiv arXiv 2026

  17. [31]

    Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal Policy Optimization Algorithms. arXiv:1707.06347

    Pith/arXiv arXiv 2017

  18. [34]

    Sheng, L.; Ma, W.; Hong, R.; Wang, X.; Zhang, A.; and Chua, T.-S. 2026. Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubrics. arXiv:2602.10885

    arXiv 2026

  19. [35]

    Shinn, N.; Cassano, F.; Berman, E.; Gopinath, A.; Narasimhan, K.; and Yao, S. 2023. Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv:2303.11366

    Pith/arXiv arXiv 2023

  20. [37]

    Wang, X.; Wei, J.; Schuurmans, D.; Le, Q.; Chi, E.; Narang, S.; Chowdhery, A.; and Zhou, D. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. arXiv:2203.11171

    Pith/arXiv arXiv 2023

  21. [39]

    Wu, S.; Tan, H.; Zhang, X.; Xia, B.; Zhang, S.; Qi, X.; Yu, B.; and Jia, J. 2026. Beyond Majority Voting: Self-Reflective Test-Time Reinforcement Learning for LLM Reasoning. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research. To appear

    2026

  22. [42]

    L.; Cao, Y.; and Narasimhan, K

    Yao, S.; Yu, D.; Zhao, J.; Shafran, I.; Griffiths, T. L.; Cao, Y.; and Narasimhan, K. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. arXiv:2305.10601

    Pith/arXiv arXiv 2023

  23. [44]

    Zhang, M.; Shen, Y.; Li, Z.; Sha, H.; Hu, B.; Wang, Y.; Huang, C.; Liu, S.; Tong, J.; Jiang, C.; Chai, M.; Xi, Z.; Dou, S.; Gui, T.; Zhang, Q.; and Huang, X. 2025 a . LLMEval-Med : A Real-world Clinical Benchmark for Medical LLM s with Physician Validation. In Findings of the Association for Computational Linguistics: EMNLP 2025, 4888--4914. Association f...

    2025

  24. [45]

    Zhang, Q.; Hu, C.; Upasani, S.; Ma, B.; Hong, F.; Kamanuru, V.; Rainton, J.; Wu, C.; Ji, M.; Li, H.; Thakker, U.; Zou, J.; and Olukotun, K. 2025 b . Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models. arXiv:2510.04618

    Pith/arXiv arXiv 2025

  25. [46]

    Zuo, Y.; Zhang, K.; Sheng, L.; Qu, S.; Cui, G.; Zhu, X.; Li, H.; Zhang, Y.; Long, X.; Hua, E.; Qi, B.; Sun, Y.; Ma, Z.; Yuan, L.; Ding, N.; and Zhou, B. 2025. TTRL: Test-Time Reinforcement Learning. arXiv:2504.16084

    Pith/arXiv arXiv 2025

  26. [47]

    2025 , eprint=

    TTRL: Test-Time Reinforcement Learning , author=. 2025 , eprint=

    2025

  27. [48]

    2026 , eprint=

    Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers , author=. 2026 , eprint=

    2026

  28. [49]

    Beyond Majority Voting: Self-Reflective Test-Time Reinforcement Learning for

    Wu, Sitong and Tan, Haoru and Zhang, Xichen and Xia, Bin and Zhang, Shaofeng and Qi, Xiaojuan and Yu, Bei and Jia, Jiaya , booktitle=. Beyond Majority Voting: Self-Reflective Test-Time Reinforcement Learning for. 2026 , note=

    2026

  29. [50]

    2025 , eprint=

    Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains , author=. 2025 , eprint=

    2025

  30. [51]

    Rein, David and Hou, Betty Li and Stickland, Asa Cooper and Petty, Jackson and Pang, Richard Yuanzhe and Dirani, Julien and Michael, Julian and Bowman, Samuel R. , year=. 2311.12022 , archivePrefix=

    Pith/arXiv arXiv

  31. [52]

    Applied Sciences , volume=

    What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical Exams , author=. Applied Sciences , volume=. doi:10.3390/app11146421 , year=

    doi:10.3390/app11146421

  32. [53]

    2025 , pages=

    Zhang, Ming and Shen, Yujiong and Li, Zelin and Sha, Huayu and Hu, Binze and Wang, Yuhui and Huang, Chenhao and Liu, Shichun and Tong, Jingqi and Jiang, Changhao and Chai, Mingxu and Xi, Zhiheng and Dou, Shihan and Gui, Tao and Zhang, Qi and Huang, Xuanjing , booktitle=. 2025 , pages=

    2025

  33. [54]

    and Zhang, Hao and Gonzalez, Joseph E

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric P. and Zhang, Hao and Gonzalez, Joseph E. and Stoica, Ion , year=. Judging. 2306.05685 , archivePrefix=

    Pith/arXiv arXiv

  34. [55]

    2026 , eprint=

    Detecting and Mitigating the Correct-Answer Extinction Window in Test-Time Reinforcement Learning with Majority Voting , author=. 2026 , eprint=

    2026

  35. [56]

    2026 , eprint=

    When More Sampling Hurts: The Modal Ceiling and Correlation Ceiling of Test-Time Scaling , author=. 2026 , eprint=

    2026

  36. [57]

    2607.05391 , archivePrefix=

    Kwok, Jacky and Li, Shulu and Atreya, Pranav and Liu, Yuejiang and Jiang, Yixing and Finn, Chelsea and Pavone, Marco and Stoica, Ion and Mirhoseini, Azalia , year=. 2607.05391 , archivePrefix=

    Pith/arXiv arXiv

  37. [58]

    Shao, Zhihong and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Song, Junxiao and Bi, Xiao and Zhang, Haowei and Zhang, Mingchuan and Li, Y. K. and Wu, Y. and Guo, Daya , year=. 2402.03300 , archivePrefix=

    Pith/arXiv arXiv

  38. [59]

    2505.02387 , archivePrefix=

    Chen, Xiusi and Li, Gaotang and Wang, Ziqi and Jin, Bowen and Qian, Cheng and Wang, Yu and Wang, Hongru and Zhang, Yu and Zhang, Denghui and Zhang, Tong and Tong, Hanghang and Ji, Heng , year=. 2505.02387 , archivePrefix=

    arXiv

  39. [60]

    Shao, Rulin and Asai, Akari and Shen, Shannon Zejiang and Ivison, Hamish and Kishore, Varsha and Zhuo, Jingming and Zhao, Xinran and Park, Molly and Finlayson, Samuel G. and Sontag, David and Murray, Tyler and Min, Sewon and Dasigi, Pradeep and Soldaini, Luca and Brahman, Faeze and Yih, Wen-tau and Wu, Tongshuang and Zettlemoyer, Luke and Kim, Yoon and Ha...

    Pith/arXiv arXiv

  40. [61]

    2026 , eprint=

    Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubrics , author=. 2026 , eprint=

    2026

  41. [62]

    2601.08430 , archivePrefix=

    Li, Sunzhu and Zhao, Jiale and Wei, Miteto and Ren, Huimin and Zhou, Yang and Yang, Jingwen and Liu, Shunyu and Zhang, Kaike and Chen, Wei , year=. 2601.08430 , archivePrefix=

    arXiv

  42. [63]

    2026 , eprint=

    Rubric-based On-policy Distillation , author=. 2026 , eprint=

    2026

  43. [64]

    2505.10320 , archivePrefix=

    Whitehouse, Chenxi and Wang, Tianlu and Yu, Ping and Li, Xian and Weston, Jason and Kulikov, Ilia and Saha, Swarnadeep , year=. 2505.10320 , archivePrefix=

    arXiv

  44. [65]

    Step-wise Rubric Rewards for

    Xie, Weichu and Zhao, Haozhe and Liu, Wenpu and Zhu, Yongfu and Chen, Liang and Ye, Minghao and Chen, Zirong and Xu, Yuqi and Dong, Shuai and Wang, Ziyue and Xu, Xinbo and Shi, Kean and Wu, Ruoyu and Zhang, Xiaoying and Shao, Wenqi and Chang, Baobao and Duan, Nan and Wang, Jiaqi , year=. Step-wise Rubric Rewards for. 2605.17291 , archivePrefix=

    Pith/arXiv arXiv

  45. [66]

    2025 , eprint=

    Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models , author=. 2025 , eprint=

    2025

  46. [67]

    and Wei, Jason and Hicks, Rebecca Soskin and Bowman, Preston and Qui

    Arora, Rahul K. and Wei, Jason and Hicks, Rebecca Soskin and Bowman, Preston and Qui. 2025 , eprint=

    2025

  47. [68]

    The Unreasonable Effectiveness of Entropy Minimization in

    Agarwal, Shivam and Zhang, Zimin and Yuan, Lifan and Han, Jiawei and Peng, Hao , year=. The Unreasonable Effectiveness of Entropy Minimization in. 2505.15134 , archivePrefix=

    Pith/arXiv arXiv

  48. [69]

    2026 , eprint=

    Recursive Self-Evolving Agents via Held-Out Selection , author=. 2026 , eprint=

    2026

  49. [70]

    2601.22628 , archivePrefix=

    Yang, Chengyi and Xiang, Zhishang and Tang, Yunbo and Teng, Zongpei and Huang, Chengsong and Long, Fei and Liu, Yuhan and Su, Jinsong , year=. 2601.22628 , archivePrefix=

    arXiv

  50. [71]

    2606.03239 , archivePrefix=

    Liu, Zheng and Zhang, Longxiang and Wang, Xintong and Xu, Zhiang and Zhan, Shaoxiong and Shan, Xin and Huang, Wen and Dai, Tao and Xia, Shu-Tao and Huo, Chengfu and Ding, Liang , year=. 2606.03239 , archivePrefix=

    Pith/arXiv arXiv

  51. [72]

    2606.23038 , archivePrefix=

    Ding, Hongxin and Huang, Baixiang and Fang, Yue and Liao, Weibin and Li, Zheng and Zhang, Jinyang and Wu, Zhijing and Zhao, Junfeng and Wang, Yasha , year=. 2606.23038 , archivePrefix=

    Pith/arXiv arXiv

  52. [73]

    2605.03871 , archivePrefix=

    Li, Shuyue Stella and Xin, Rui and Xiao, Teng and Wang, Yike and Shao, Rulin and Hao, Zoey and Sclar, Melanie and Oh, Sewoong and Brahman, Faeze and Koh, Pang Wei and Tsvetkov, Yulia , year=. 2605.03871 , archivePrefix=

    Pith/arXiv arXiv

  53. [74]

    2605.29847 , archivePrefix=

    Guan, Xin and Hu, Xiaomeng and Huang, Shen and Wang, Zhenyi and Zhang, Bo and Li, Zijian and Xie, Pengjun and Liu, Bo and Cao, Jiuxin , year=. 2605.29847 , archivePrefix=

    Pith/arXiv arXiv

  54. [75]

    Co-Evolving

    Wang, Beining and Su, Weihang and Tian, Hongtao and Kong, Hao and Yang, Tao and Yao, Ting and Pan, Qingyi and Wu, Yueyue and Ai, Qingyao and Zhang, Min and Liu, Yiqun , year=. Co-Evolving. 2607.20083 , archivePrefix=

    Pith/arXiv arXiv

  55. [76]

    2605.09959 , archivePrefix=

    Huang, Chengsong and Liu, Haolin and Zheng, Tong and Dai, Runpeng and Huang, Langlin and Li, Jinyuan and Li, Zongxia and Wei, Zhepei and Meng, Yu and Huang, Jiaxin , year=. 2605.09959 , archivePrefix=

    Pith/arXiv arXiv

  56. [77]

    2026 , eprint=

    Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Text , author=. 2026 , eprint=

    2026

  57. [78]

    2605.31433 , archivePrefix=

    Kwan, Wai-Chung and Gema, Aryo Pradipta and Leang, Joshua Ong Jun and Minervini, Pasquale , year=. 2605.31433 , archivePrefix=

    Pith/arXiv arXiv

  58. [79]

    2023 , eprint=

    Self-Consistency Improves Chain of Thought Reasoning in Language Models , author=. 2023 , eprint=

    2023

  59. [80]

    2023 , eprint=

    Tree of Thoughts: Deliberate Problem Solving with Large Language Models , author=. 2023 , eprint=

    2023

  60. [81]

    2023 , eprint=

    Self-Refine: Iterative Refinement with Self-Feedback , author=. 2023 , eprint=

    2023

  61. [82]

    2023 , eprint=

    Reflexion: Language Agents with Verbal Reinforcement Learning , author=. 2023 , eprint=

    2023

  62. [83]

    2303.16634 , archivePrefix=

    Yang Liu and Dan Iter and Yichong Xu and Shuohang Wang and Ruochen Xu and Chenguang Zhu , year=. 2303.16634 , archivePrefix=

    Pith/arXiv arXiv

  63. [84]

    2024 , eprint=

    Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models , author=. 2024 , eprint=

    2024

  64. [85]

    2307.10928 , archivePrefix=

    Seonghyeon Ye and Doyoung Kim and Sungdong Kim and Hyeonbin Hwang and Seungone Kim and Yongrae Jo and James Thorne and Juho Kim and Minjoon Seo , year=. 2307.10928 , archivePrefix=

    Pith/arXiv arXiv

  65. [86]

    2311.08788 , archivePrefix=

    Minqian Liu and Ying Shen and Zhiyang Xu and Yixin Cao and Eunah Cho and Vaibhav Kumar and Reza Ghanadan and Lifu Huang , year=. 2311.08788 , archivePrefix=

    Pith/arXiv arXiv

  66. [87]

    2305.14251 , archivePrefix=

    Sewon Min and Kalpesh Krishna and Xinxi Lyu and Mike Lewis and Yih, Wen-tau and Pang Wei Koh and Mohit Iyyer and Luke Zettlemoyer and Hannaneh Hajishirzi , year=. 2305.14251 , archivePrefix=

    Pith/arXiv arXiv

  67. [88]

    2017 , eprint=

    Proximal Policy Optimization Algorithms , author=. 2017 , eprint=

    2017

  68. [89]

    2505.09388 , archivePrefix=

    An Yang and Anfeng Li and Baosong Yang and Beichen Zhang and Binyuan Hui and Bo Zheng and Bowen Yu and Chang Gao and Chengen Huang and Chenxu Lv and Chujie Zheng and Dayiheng Liu and Fan Zhou and Fei Huang and Feng Hu and Hao Ge and Haoran Wei and Huan Lin and Jialong Tang and Jian Yang and Jianhong Tu and Jianwei Zhang and Jianxin Yang and Jiaxi Yang and...

    Pith/arXiv arXiv

  69. [90]

    2026 , doi=

    Transactions of the Association for Computational Linguistics , volume=. 2026 , doi=

    2026