Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

ARCO: a rubric model that writes per-step natural-language criteria, scores each action against them, and co-evolves with the policy, reporting the highest Exact Match in all six multi-hop QA settings tested.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 10:39 UTC pith:ZROJB42X

load-bearing objection A genuinely new assembly for per-step rubric rewards, backed by a clean and honest empirical study, but the headline credit-assignment mechanism is under-determined — the real step-level signal is inherited from GPT warmup projections, not derived by the decomposition objective. the 3 major comments →

arxiv 2606.21262 v2 pith:ZROJB42X submitted 2026-06-19 cs.AI cs.CL

ARCO: Adaptive Rubrics with Co-Evolution for Multi-Step LLM-Based Agents

Zihang Tian , Jingsen Zhang , Rui Li , Xiaohe Bo , Yuanzi Li , Xu Chen This is my paper
classification cs.AI cs.CL
keywords reinforcement learningLLM agentsrubric-based rewardsprocess reward modelscredit assignmentco-evolutionmulti-hop question answeringinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that a reward model for multi-step LLM agents should be rubric-based, step-level, and co-evolved with the policy, and that all three properties can be realized by one same-scale open-source model without an external judge. ARCO attaches a short natural-language checklist to every action, scores the action against that checklist, and constrains the step scores to add up to the final binary outcome. The rubric writer and scorer share a backbone with the policy and are updated together on on-policy rollouts, so the criteria track the agent's changing failure modes. On three multi-hop QA benchmarks with two backbones, the paper reports the highest Exact Match in all six settings against outcome-reward, rubric-based, and process-reward baselines, and its binding diagnostic shows rubrics identify the scored action at roughly two times chance. A sympathetic reader would care because interpretable step-level credit assignment is usually obtained only with gold step labels or a frozen closed-source judge; ARCO claims neither is needed.

Core claim

The paper's central claim is that per-step rubric criteria can do real credit-assignment work even when the only training signal is the terminal outcome reward. ARCO's rubric model, built from the same backbone as the policy, first writes K criteria for the state-action pair and then predicts K dimension scores conditioned on those criteria; the step score is the mean. A trajectory-decomposition objective (the sum of step scores must equal the terminal reward) plus KL regularization to a warmup reference lets the rubric model and the policy be optimized on the same on-policy rollouts. The paper reports that this design beats outcome-level binary-reward RL, frozen rubric-based judges, and an

What carries the argument

The key mechanism is the hierarchical rubric model built on a shared causal-LM backbone. One head generates K=3 natural-language criteria per action after seeing the trajectory prefix; the second head, the score head, maps the mean-pooled hidden states of the full scoring sequence to K criterion scores in [-1,1], and the step reward is the mean of those scores. The load-bearing objective is the trajectory decomposition constraint, which requires the sum of step rewards to equal the terminal binary reward; this is what lets the model learn per-step credit without gold step labels. A position-bucketed reward-to-go baseline groups steps by search rank or terminal status, and a KL term keeps gen

Load-bearing premise

Everything rests on whether the warmup teacher's per-step criteria and projected scores—which are only constrained to sum to the final outcome—actually carry information about which step caused success or failure; if they do not, the decomposition objective alone cannot recover genuine per-step credit, and the claimed step-level interpretability and gains do not follow from the stated mechanism.

What would settle it

Take one benchmark setting, keep the warmup rubric text, and replace the teacher's projected per-step scores with random or equal scores that still sum to the terminal reward. If the reported EM advantage persists and the step-binding accuracy stays near two times chance, the step-level scoring is not doing the claimed credit-assignment work; if both collapse, the warmup annotations are the true carrier. A second check: compare ARCO's per-step scores to human step-quality judgments on a small sample; near-zero correlation with high EM would show the rubrics are post-hoc text rather than measur

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Step-level rewards that add up to the terminal outcome can be trained without gold step labels, using only success/failure and a warmup teacher.
  • The evaluator can keep pace with a changing policy, so reward criteria track newly emerging failure modes instead of going stale.
  • Natural-language criteria carry real signal: removing them costs 3.0–3.5 EM points on HotpotQA, and prefix-conditioned rubrics improve EM in every setting tested.
  • A small, same-scale rubric model suffices; scaling the rubric model does not improve final policy quality, and cross-family evaluators still transfer useful scoring signals.
  • Increasing rubric width beyond K=3 mostly adds paraphrastic duplicates rather than new evaluation dimensions, so rubric budgets should emphasize importance over diversity.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: because trajectories are only 3–6 steps and the only per-step supervision in RL is the sum constraint, the informative step-level signal likely comes from the warmup teacher's projected scores; a clean test would decouple warmup quality from the co-evolution objective.
  • Inference: the 'no external judge' property is post-warmup; the method still needs a strong teacher at initialization, so an open question is whether weaker or self-generated warmup preserves the gains.
  • Inference: the binding results at roughly two times chance suggest the rubrics encode partial, not full, action identity; contrastive training that names the distinguishing entity or relation for each step could make step-level criteria more discriminative.
  • Inference: the framework should transfer to longer-horizon tool-use and coding agents, but whether rubrics remain stable and informative over dozens of steps is untested; drift-aware scheduling or rubric caching may be needed.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ARCO, a framework for multi-step LLM-based agents in which a rubric model mu generates per-step natural-language criteria and predicts rubric-conditioned step-level scores, and co-evolves with the policy at the parameter level on on-policy rollouts. A trajectory-decomposition loss (Eq. 5) ties the sum of step scores to the binary terminal reward, and a supervised warmup (Appendix B) initializes the rubric model using GPT-annotated trajectories and scores projected to satisfy that same constraint. Experiments on HotpotQA, 2WikiMultiHopQA, and MuSiQue with Qwen3-4B and Llama-3.2-3B report the highest Exact Match in all six (dataset, backbone) cells, with ablations, a rubric-action binding diagnostic, a K-sensitivity sweep, and backbone-asymmetry analyses.

Significance. ARCO is a well-motivated attempt to make process rewards for LLM agents both interpretable and adaptable. The hierarchical rubric model with a shared backbone is clean, the ablation suite is informative, and the empirical scope (three benchmarks, two backbones) is broader than many contemporary papers. The authors provide code and data and are admirably explicit about limitations. If the reported gains hold under proper statistical testing, ARCO would be a useful contribution to interpretable RL for language agents. However, the central mechanistic claim — that trajectory decomposition alone enables step-level credit assignment from terminal outcomes — is not supported by the evidence. The step-level signal is inherited from the GPT warmup, not derived by the proposed loss, and the reported single-run, dev-selected numbers do not yet justify the strength of the empirical claims.

major comments (3)
  1. [§3.2, Eq. (5); Appendix B/E; Limitations] The trajectory-decomposition loss (Eq. 5) imposes a single equality per trajectory, sum_t s_t = R(τ). With at most 3–6 steps and no step-level labels, this leaves per-step scores badly underdetermined; any vector summing to R(τ) minimizes the MSE. The gradient of this term w.r.t. the score head is a shared scalar times the sum of per-step Jacobians, so it cannot by itself differentiate step quality. The actual step-level signal comes from the warmup targets in Eq. (6), which are GPT-annotated and explicitly constructed to satisfy the same sum constraint (Appendix E: 'adjust magnitudes ... so the exact sum is {reward}'). The abstract's claim that trajectory decomposition 'enabl[es] credit assignment without step-level labels' is therefore an overstatement; the per-step content is inherited from a closed-source annotator. The Limitations section concedes that the loss 'only ties the sum of
  2. [§4.1, Table 1] All cells report a single run at the best dev-EM checkpoint, with no seeds, standard deviations, or significance tests. Several winning margins are small (e.g., MuSiQue/Qwen EM 27.40 vs. 26.60 for RaR; HotpotQA/Llama EM 36.40 vs. 35.00 for R1-Searcher). With one run and dev-based checkpoint selection, the central empirical claim — 'highest EM in all six settings' — is not statistically robust. Please report at least three seeds with mean±std, specify the checkpoint-selection rule across seeds, and if possible report on a held-out test split (the current 500-example evaluation set is used for selection).
  3. [§4.4, Table 3; RQ4] The RQ3 binding diagnostic demonstrates that the natural-language rubric text is action-specific (Bind 42–55% vs. 25% chance), but it does not evaluate the numeric step scores s_t. The paper's claims of step-level interpretability and diagnostic utility (§7) rest on those scores being meaningful. The only score-related evidence is internal (RQ4's within-cluster score range). This is insufficient: a uniform redistribution of the terminal reward would also satisfy the decomposition constraint and could produce similar task performance. To support the interpretability claim, compare ARCO's step scores with independent step-level annotations, or include an ablation where s_t is replaced by a constant share of R(τ) per step; such an experiment would isolate the contribution of score informativeness from the rubric text and sum constraint.
minor comments (5)
  1. [Abstract; §1; Appendix B/E] The paper states 'no external judge,' but the warmup uses a GPT-API teacher for trajectory and rubric annotation and RQ3 uses gpt-4o-mini. Clarify that 'no external judge' refers only to the RL stage after warmup.
  2. [§4.1] Define 'best post-transition dev-EM checkpoint' precisely. How is the transition epoch η chosen per dataset/backbone, and what happens if the best checkpoint occurs before the transition?
  3. [Eq. (3), Eq. (4)] The position-bucketed baseline is defined over the batch. State explicitly whether the baseline is recomputed per gradient step and whether the current trajectory is included in its own baseline bucket.
  4. [§4.5] The K-sensitivity sweep is performed only on HotpotQA/Qwen. State whether the qualitative conclusions (peak at K=3, saturation of themes) are expected to transfer to other datasets/backbones, or add a smaller sweep on another setting.
  5. [§4.3, Figure 2] The matched-step comparison is described in the text but the selection procedure (how the same (h_t,a_t) is identified across policies) could be made more explicit in the caption or a paragraph; currently it is easy to miss.

Circularity Check

1 steps flagged

Step-level credit assignment is constructed in warmup projection, not derived from Eq. 5.

specific steps
  1. fitted input called prediction [Abstract; Sec. 3.2 (Eq. 5); Appendix B (Eq. 6); Appendix E (Fig. 10 warmup prompt)]
    "Scores are parsed, clipped to [-1,1], padded or truncated to width K, and projected so that the summed step-score means satisfy the trajectory-level decomposition constraint, sum_t (1/K) sum_j s-hat_{t,j} = R(tau). ... This hard decomposition constraint overrides the local scoring guide: adjust magnitudes, including negative scores when necessary, so the exact sum is {reward:.3f}. ... A trajectory decomposition constraint ties the sum of step rewards to the terminal outcome, enabling credit assignment without step-level labels."

    By construction, Eq. (6) forces the warmup score targets to satisfy exactly the equality that Eq. (5) minimizes. Any per-step signal therefore enters through GPT-annotated pseudo-scores that were projected to be decomposition-consistent, not through Eq. (5)'s ability to split R(tau) among steps. The Eq. (5) gradient is a single shared correction over the whole trajectory, so with no step-level labels it cannot identify which step caused success or failure; it only preserves a sum constraint already satisfied by the warmup data. Thus the paper's claim that trajectory decomposition 'enables credit assignment without step-level labels' is a renamed property of the warmup projection, and the claimed step-level interpretability is inherited from a closed-source annotator rather than derived fro

full rationale

ARCO's headline EM results are empirical and are measured against external baselines, so the paper is not wholly circular; the self-citations (Bo et al. 2024, Tian et al. 2026) are used only for environment conventions and are not load-bearing. The specific circularity is in the mechanistic claim of step-level credit assignment. Eq. (5) imposes only sum_t s_t = R(tau), which leaves per-step scores under-determined. The only step-level information comes from GPT warmup scores that Eq. (6) and the Appendix E prompt force to satisfy that exact same sum constraint. The paper's own Limitations admit that the decomposition loss 'only ties the sum of step scores to the terminal outcome,' and the RQ3 binding diagnostic tests rubric text, not the numeric score values, so it does not close this gap. The per-step reward prediction is therefore shaped by construction, while the overall EM comparisons retain independent content.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 1 invented entities

The method introduces one framework-level construct (µ) and several free design parameters (K, loss scales, schedule, baseline bucketing, warmup projection). The load-bearing assumptions are the decomposition constraint’s informativeness and the quality of GPT-annotated warmup labels. The benchmark environment, the retriever, and the base models are imported from prior work. No parameter-free derivation of any constant is claimed.

free parameters (5)
  • K (number of rubric criteria per step) = 3
    Chosen by hand and validated by the RQ4 sweep on HotpotQA/Qwen: K=3 gives 42.80 EM, K=9 recovers to 42.40, so the default is not sharply distinguished from alternatives.
  • Loss scales α, β, λ_π, λ (dense reward scale, KL weights, criterion-score loss weight) = reported in Appendix B
    Hyperparameters chosen by hand/validation; they set the balance between outcome reward, rubric KL, and decomposition in Eqs. 3, 5, and 7.
  • η (dense transition epoch) = early sparse phase then dense
    Sparse-to-dense schedule knob in Algorithm 1; a design choice affecting when rubric scores start shaping the policy.
  • Position-bucketed baseline buckets = search_1..terminal
    Baseline design in Eq. 4; chosen by the authors and it directly shapes the advantage estimates used to train π.
  • Warmup score projection = projected so Σ_t step-mean = R(τ)
    Warmup criterion scores are constructed to satisfy the decomposition constraint (Appendix B, Eq. 6); this manufactures the supervision signal rather than measuring it.
axioms (5)
  • domain assumption Binary terminal EM reward R(τ) ∈ {0,1} is a sufficient learning signal for multi-hop QA agents
    Adopted from the cited ORM/PRM environment protocol (§2, §4.1); standard in the field but still an assumption about task learnability.
  • ad hoc to paper Sum-of-step-scores = terminal reward (decomposition) yields valid per-step credit without step labels
    Eq. 5; this is the paper’s central mechanism and is assumed, not derived. With T ≤ 6, the constraint leaves per-step scores under-determined; the paper’s own Limitations section concedes richer process supervision could help.
  • ad hoc to paper Warmup GPT-annotated criteria and projected scores are a sound initialization for co-evolution
    §3.2 and Appendix B; the ‘no external judge’ property holds only after warmup, and the whole pipeline inherits the closed-source teacher’s annotation quality and prompt compliance.
  • domain assumption Rubric text generated by the LM head is informative to the score head through the shared backbone
    §3.1; supported by the w/o Rubric ablation (3.0–3.5 EM drop on HotpotQA) but not independently verified against human judgments of criterion quality.
  • ad hoc to paper KL regularization to the warmup reference keeps rubric text coherent while still allowing adaptation
    Eq. 5 KL term; assumes the warmup reference is good enough that rubric drift remains bounded during co-evolution.
invented entities (1)
  • Rubric model µ with shared backbone plus generation and score heads independent evidence
    purpose: Generates per-step natural-language criteria and predicts criterion-conditioned step scores; co-evolves with the policy on on-policy data
    A model component rather than a new physical entity. It has a falsifiable handle in the RQ3 binding diagnostic (rubric text should identify the action it scored; 42–55% vs 25% chance), which the paper itself reports.

pith-pipeline@v1.3.0-alltime-deepseek · 17059 in / 18063 out tokens · 173208 ms · 2026-08-02T10:39:30.649769+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning for multi-step LLM agents often relies on scalar rewards that indicate success but cannot explain why a trajectory is good or bad. Rubric-based rewards improve interpretability through natural-language criteria, but existing methods share two limitations: they score at the trajectory level, offering no guidance for individual steps; and their scorer is closed-source and static, so it cannot adapt as the agent evolves during training. We propose ARCO (Adaptive Rubric CO-evolution), which generates a per-step rubric and predicts a rubric-conditioned step-level reward for each action, and continually updates this rubric model on on-policy rollouts so that its criteria and scores co-evolve with the agent's improving behavior. Across HotpotQA, 2WikiMultiHopQA, and MuSiQue with two open-source backbones, ARCO achieves the highest EM in all settings over outcome-, rubric-, and process-reward baselines, and analyses show its rubrics are step-specific, robust to design choices, and useful for diagnosing agent behavior. Code and data are available at https://github.com/zihangtian/ARCO.

Figures

Figures reproduced from arXiv: 2606.21262 by Jingsen Zhang, Rui Li, Xiaohe Bo, Xu Chen, Yuanzi Li, Zihang Tian.

Figure 1
Figure 1. Figure 1: ARCO architecture on a multi-hop QA example. The rubric model [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Three single-step score comparisons on MuSiQue dev. In each panel, ARCO and one [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: RQ4 K-sensitivity diagnostics on HotpotQA / Qwen for the sweep on fixed policy-warmup trajectories (K ∈ {1, 3, 5, 7, 9}). (A) Dev EM/F1 at each best-dev-EM checkpoint. (B) Distinct semantic themes per step (blue, left) and distinct-theme ratio #themes/K (red, right). (C) Step-level total semantic duplicate rate (orange, left) and score range (purple, right; max–min scores within same-step duplicate semanti… view at source ↗
Figure 4
Figure 4. Figure 4: Three K=7 HotpotQA / Qwen dev steps with different redundancy patterns. Colored dots show the theme of each criterion. Exact duplicate: 7 byte-identical criteria. Semantic duplicate: 7 different wordings all scoring the same theme. Diverse: 7 criteria span 5 distinct themes. through paraphrase. This suggests allocating criterion slots by importance rather than diversity. We adopt K=3 as default: it achieve… view at source ↗
Figure 5
Figure 5. Figure 5: A HotpotQA dev trajectory under the four-way binding protocol (Qwen3-4B, ARCO epoch [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Two RQ3 rubric-binding failures. In Case 1, the gold action is [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Policy prompt for the HotpotQA agent. The forced-finish variant is used when the search [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Shared rubric system prompt that defines the evaluator role and output constraints. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Step-level rubric prompt that generates local criteria for an individual policy action. [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Trajectory-level rubric prompt used during warmup annotation to produce criteria and [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: RQ3 four-way step-binding judge prompt ( [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: RQ3 step-specificity judge prompt (Spec). The judge sees the question, trajectory prefix, gold action, and the rubric, and rates how specifically the rubric evaluates the gold action on a 1–5 scale. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_12.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. DecoEvo: Score-Decoupled Co-Evolution of Solver and Rubric-Generator Skills in Text Space

    cs.AI 2026-07 conditional novelty 6.0

    DecoEvo co-evolves solver and rubric-generator skills with decoupled objectives, beating SkillOpt on 15 benchmark-backbone settings by 1.7–3.1 points without gold rubrics or weight updates.

  2. Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading

    cs.AI 2026-07 conditional novelty 6.0

    A 46-task terminal benchmark with subtask-level dense rewards shows frontier agents rarely finish long workflows, with the best model at 28.3% pass@1 (R≥0.95).

  3. Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading

    cs.AI 2026-07 conditional novelty 5.5

    A 46-task long-horizon terminal benchmark with dense subtask rewards finds even the best frontier agent fully solves only ~15% of tasks under a 0.95 reward threshold.

Reference graph

Works this paper leans on

23 extracted references · 11 linked inside Pith · cited by 2 Pith papers

  1. [7]

    Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen

    J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.ArXiv, abs/2106.09685,

    Pith/arXiv arXiv

  2. [8]

    Changxin Huang, Yanbin Chang, Junfan Lin, Junyang Liang, Runhao Zeng, and Jianqiang Li

    URL https://api.semanticscholar.org/CorpusID:235458009. Changxin Huang, Yanbin Chang, Junfan Lin, Junyang Liang, Runhao Zeng, and Jianqiang Li. Efficient language-instructed skill acquisition via reward-policy co-evolution. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 14576–14584, 2025a. Zenan Huang, Yihong Zhuang, Guosh...

    Pith/arXiv arXiv

  3. [9]

    Reinforcement learning for long-horizon multi-turn search agents

    Vivek Kalyan and Martin Andrews. Reinforcement learning for long-horizon multi-turn search agents. arXiv preprint arXiv:2510.24126,

    arXiv

  4. [10]

    Openrubrics: Towards scalable synthetic rubric generation for reward modeling and llm alignment.arXiv preprint arXiv:2510.07743, 2025a

    Tianci Liu, Ran Xu, Tony Yu, Ilgee Hong, Carl Yang, Tuo Zhao, and Haoyu Wang. Openrubrics: Towards scalable synthetic rubric generation for reward modeling and llm alignment.arXiv preprint arXiv:2510.07743, 2025a. Xiaoqian Liu, Ke Wang, Yuchuan Wu, Fei Huang, Yongbin Li, Junge Zhang, and Jianbin Jiao. Agentic reinforcement learning with implicit step rewa...

    arXiv

  5. [11]

    Online rubrics elicitation from pairwise comparisons.arXiv preprint arXiv:2510.07284,

    MohammadHossein Rezaei, Robert Vacareanu, Zihao Wang, Clinton Wang, Bing Liu, Yunzhong He, and Afra Feyza Akyürek. Online rubrics elicitation from pairwise comparisons.arXiv preprint arXiv:2510.07284,

    arXiv

  6. [12]

    Dr tulu: Reinforcement learning with evolving rubrics for deep research.arXiv preprint arXiv:2511.19399,

    Rulin Shao, Akari Asai, Shannon Zejiang Shen, Hamish Ivison, Varsha Kishore, Jingming Zhuo, Xinran Zhao, Molly Park, Samuel G Finlayson, David Sontag, et al. Dr tulu: Reinforcement learning with evolving rubrics for deep research.arXiv preprint arXiv:2511.19399,

    Pith/arXiv arXiv

  7. [13]

    R1-Searcher: Incentivizing the search capability in LLMs via reinforcement learning.arXiv preprint arXiv:2503.05592,

    Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, et al. R1-Searcher: Incentivizing the search capability in LLMs via reinforcement learning.arXiv preprint arXiv:2503.05592,

    Pith/arXiv arXiv

  8. [14]

    Haps: Hierarchical llm routing with joint architecture and parameter search.arXiv preprint arXiv:2601.05903,

    Zihang Tian, Rui Li, Jingsen Zhang, Xiaohe Bo, Wei Huo, and Xu Chen. Haps: Hierarchical llm routing with joint architecture and parameter search.arXiv preprint arXiv:2601.05903,

    arXiv

  9. [16]

    Watch every step! LLM agent learning via iterative step-level process refinement

    Weimin Xiong, Yifan Song, Xiutian Zhao, Wenhao Wu, Xun Wang, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. Watch every step! LLM agent learning via iterative step-level process refinement. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,

    2024

  10. [17]

    Alternating reinforcement learning for rubric-based reward modeling in non- verifiable LLM post-training.arXiv preprint arXiv:2602.01511,

    Ran Xu, Tianci Liu, Zihan Dong, Tony Yu, Ilgee Hong, Carl Yang, Linjun Zhang, Tao Zhao, and Haoyu Wang. Alternating reinforcement learning for rubric-based reward modeling in non- verifiable LLM post-training.arXiv preprint arXiv:2602.01511,

    arXiv

  11. [18]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  12. [19]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. InProceedings of the 2018 conference on empirical methods in natural language processing, pp. 2369–2380,

    2018

  13. [20]

    Reinforcing multi-turn reasoning in llm agents via turn-level credit assignment

    Siliang Zeng, Quan Wei, William Brown, Oana Frunza, Yuriy Nevmyvaka, Yang Katie Zhao, and Mingyi Hong. Reinforcing multi-turn reasoning in llm agents via turn-level credit assignment. In ICML 2025 Workshop on Computer Use Agents,

    2025

  14. [21]

    Chasing the tail: Effective rubric-based reward modeling for large language model post-training.arXiv preprint arXiv:2509.21500,

    Junkai Zhang, Zihao Wang, Lin Gui, Swarnashree Mysore Sathyendra, Jaehwan Jeong, Victor Veitch, Wei Wang, Yunzhong He, Bing Liu, and Lifeng Jin. Chasing the tail: Effective rubric-based reward modeling for large language model post-training.arXiv preprint arXiv:2509.21500,

    arXiv

  15. [22]

    R-search: Empowering llm reasoning with search via multi-reward reinforcement learning.arXiv preprint arXiv:2506.04185,

    Qingfei Zhao, Ruobing Wang, Dingling Xu, Daren Zha, and Limin Liu. R-search: Empowering llm reasoning with search via multi-reward reinforcement learning.arXiv preprint arXiv:2506.04185,

    Pith/arXiv arXiv

  16. [23]

    the missing person’s birth date

    12 Preprint A TRAININGALGORITHM Algorithm 1 details the full ARCO training loop, covering rollout, rubric generation and scoring, and the co-evolution updates ofπandµ. Algorithm 1ARCO: Adaptive Rubric Co-Evolution Require: Policy π (SFT-initialized), rubric model µ (SFT-initialized), training examples E, retriever R, max stepsT, dense transition epochη 1:...

    2004

  17. [2020]

    Cooper: Co-optimizing policy and reward models in reinforcement learning for large language models.arXiv preprint arXiv:2508.05613,

    10 Preprint Haitao Hong, Yuchen Yan, Xingyu Wu, Guiyang Hou, Wenqi Zhang, Weiming Lu, Yongliang Shen, and Jun Xiao. Cooper: Co-optimizing policy and reward models in reinforcement learning for large language models.arXiv preprint arXiv:2508.05613,

    Pith/arXiv arXiv

  18. [2021]

    The Llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aieleen Letman, Akhil Mathur, Alan Schelten, Amy Yang, et al. The Llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Pith/arXiv arXiv

  19. [2022]

    Co-evolution of pol- icy and internal reward for language agents.arXiv preprint arXiv:2604.03098, 2026a

    Xinyu Wang, Hanwei Wu, Jingwei Song, Shuyuan Zhang, Jiayi Zhang, Fanqi Kong, Tung Sum Thomas Kwok, Xiao-Wen Chang, Yuyu Luo, Chenglin Wu, et al. Co-evolution of pol- icy and internal reward for language agents.arXiv preprint arXiv:2604.03098, 2026a. Yinjie Wang, Tianbao Xie, Ke Shen, Mengdi Wang, and Ling Yang. Rlanything: Forge environment, policy, and r...

    Pith/arXiv arXiv

  20. [2023]

    SimCSE: Simple contrastive learning of sentence embeddings

    Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6894–6910,

    2021

  21. [2024]

    Rubrics as rewards: Reinforcement learning beyond verifiable domains.arXiv preprint arXiv:2507.17746,

    Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains.arXiv preprint arXiv:2507.17746,

    Pith/arXiv arXiv

  22. [2025]

    Carmo: Dynamic criteria generation for context aware reward modelling

    Taneesh Gupta, Shivam Shandilya, Xuchao Zhang, Rahul Madhavan, Supriyo Ghosh, Chetan Bansal, Huaxiu Yao, and Saravan Rajmohan. Carmo: Dynamic criteria generation for context aware reward modelling. InFindings of the Association for Computational Linguistics: ACL 2025, pp. 2202–2261,

    2025

  23. [2026]

    Process reward models for LLM agents: Practical framework and directions

    Sanjiban Choudhury. Process reward models for LLM agents: Practical framework and directions. arXiv preprint arXiv:2502.10325,

    Pith/arXiv arXiv