Pith. sign in

REVIEW 2 major objections 7 minor 3 cited by

VerIF: Verification Engineering for Reinforcement Learning in Instruction Following

T0 review · 2 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VERIF is a hybrid verification method that combines rule-based code checks for hard constraints with a single-pass LLM judgment for soft constraints, and reinforcement learning with this reward substantially improves instruction following…

desk verdict Hybrid code+LLM reward for RL instruction following is a useful, novel recipe, but the headline IFEval is inflated by checkpoint selection and the soft verifier's reliability needs direct evidence. read the letter →

arxiv 2506.09942 v1 pith:CQWJ3FH2 submitted 2025-06-11 cs.CL cs.AI

Hao Peng , Yunjia Qi , Xiaozhi Wang , Bin Xu , Lei Hou , Juanzi Li This is my paper
classification cs.CLcs.AI
keywords reinforcementlearningwithverifiablerewardsinstructionfollowingLLM-as-a-judgeverificationengineeringconstraintGRPOrewardmodeldataconstruction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes VERIF, a verification method that turns instruction following into a reinforcement-learning reward without human preference annotations or reference answers. The idea is to split each instruction's constraints into hard ones, such as length, format, and keyword presence, that are checked by generated Python code, and soft ones, such as style and content, that are judged in a single pass by a large reasoning model such as QwQ-32B, then average the two binary verdicts into one reward. To support training, the authors construct VerInstruct, about 22,000 complex instructions with an average of 6.2 constraints and verification code for hard constraints. GRPO training with this reward substantially improves instruction following on IFEval, Multi-IF, SysBench, FollowBench, and CFBench, while general math and language benchmarks stay roughly level. If correct, this gives a scalable way to add an instruction-following reinforcement-learning stage to existing post-training recipes.

What carries the argument

VerIF is the central object: $\text{VERIF}(x,y)=F(\text{Code}(y,C_h),\text{LLM}(y,C_s))$, with $F$ as averaging in the reported experiments. $\text{Code}(y,C_h)\in\{0,1\}$ is the output of LLM-generated Python functions that check hard constraints such as length, keyword presence, and format; $\text{LLM}(y,C_s)\in\{0,1\}$ is the single-pass binary judgment of a reasoning LLM, QwQ-32B or the distilled IF-Verifier-7B, over all soft constraints such as style and content. The mechanism works by routing each constraint type to the verifier best suited to it, compensating the LLM's known weaknesses in numerical counting and keyword detection and compensating code's inability to judge semantic constraints. The final averaged score is used online as the reward for GRPO, with 16 rollouts per prompt.

What would settle it

Run the same GRPO training on VerInstruct but replace the QwQ-32B soft-constraint verdicts with verdicts from a held-out set of human raters or an independently trained judge for the same responses; if the IFEval and Multi-IF gains shrink or vanish, the reported gains depend on the specific judge rather than on better constraint satisfaction.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that a hybrid verification signal, not a single verifier, is the reliable reward for reinforcement learning in instruction following. Code verification alone handles hard constraints accurately but misses soft constraints, reaching only 48.6 percent overall accuracy on IFBench, while LLM-only verification handles soft constraints but fails on keyword and length constraints, reaching 37.4 percent. Averaging code verdicts on $C_h$ with a reasoning LLM's single-pass verdict on $C_s$ reaches 58.1 percent on IFBench and, when used as the GRPO reward on VerInstruct, lifts TULU 3 SFT from 68.4 to 84.5 prompt-level strict IFEval and Multi-IF Turn 3 from 40.3 to 54.0, surpassing TULU 3 itself. The authors also show that a distilled 7B verifier matches most of this gain, and that ablations removing either component lower the final benchmark scores.

Load-bearing premise

The load-bearing premise is that a single-pass binary judgment from a large reasoning model is a reliable reward for soft constraints, which make up 77.7 percent of the training data; if that judge is biased, noisy, or reward-hackable, the policy may learn to satisfy the judge rather than the user's intent.

Editorial extensions

If this is right

  • RL with VERIF can be added as an extra post-training stage on top of SFT, improving instruction following without needing preference pairs or reference answers.
  • Because hard constraints are checked exactly by code and soft constraints by an LLM, the reward is cheap to compute automatically, making large-scale RL data construction more practical than SFT data pipelines.
  • The same recipe transfers to unseen constraint types, multilingual prompts, multi-turn conversations, and system-prompt following, even though the training data is English and single-turn only.
  • A distilled 7B verifier can replace a 32B verifier with comparable RL performance and far lower online reward latency, making the method practical on limited GPU budgets.
  • Instruction-following RL with VERIF does not degrade general, mathematical, or NLU capabilities and can even slightly improve some of them, so it can be integrated into existing RL pipelines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's pass@$k$ evidence suggests that base models can already sample many correct responses, so VERIF's main contribution is to make the reward reliable enough for RL to exploit that latent sample space; combining it with search or higher sampling temperatures could amplify the gains.
  • Because 77.7 percent of VerInstruct constraints are soft, the quality of the LLM judge is the bottleneck, so ensembled judges, multi-pass verification, or calibrated confidence thresholds are direct robustness extensions the paper leaves implicit.
  • The evaluation suite is largely automated, so a human-evaluation study on a subset of Multi-IF and FollowBench would test whether the learned policies truly follow constraints rather than satisfy the judge.
  • Since the data and code are released, running VERIF on a larger, more diverse constraint set, including multi-turn and non-English instructions, would show whether the training-distribution mismatch is the only limit on generalization.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. The paper proposes VERIF, a hybrid verification method for reinforcement learning with verifiable rewards (RLVR) in instruction following. Hard constraints (length, format, keyword) are checked by generated Python code; soft constraints (content, style, etc.) are judged by a large reasoning LLM (QwQ-32B) in a single pass. The authors construct VERINSTRUCT, roughly 22k instruction instances with constraint annotations and verification code/LLM tags, and train TULU 3 SFT and DeepSeek-R1-Distill-Qwen-7B with GRPO. They report large gains on IFEval, Multi-IF, SysBench, FollowBench, and CFBench, with no degradation on general benchmarks, and also distill a 7B verifier (IF-Verifier-7B) that approaches QwQ-32B's training performance. The central claim is that the hybrid verification approach is an effective, automatic reward for RLVR in instruction following and that it generalizes to unseen constraint types.

Significance. If the claims hold, this is a valuable contribution: it provides a scalable recipe for RLVR in instruction following, an under-explored area compared with math and code, and it releases data, code, and models. The multi-benchmark improvements over the base models, particularly the large gains on Multi-IF and SysBench, are credible and are not simply artifacts of validation selection. The ablation study is informative in showing that both the code and the LLM components contribute. The small-verifier distillation addresses practical deployment. However, the two issues below—the use of IFEval as the validation set for checkpoint selection and the lack of evidence for the reliability of the LLM soft-constraint reward—must be resolved before the central claims can be fully accepted.

major comments (2)
  1. [Appendix B, Table 2] Appendix B states that 'we use IFEval as the validation set to select the best checkpoint.' Table 2 then reports IFEval as the headline result (TULU 3 SFT +VERIF reaches 84.5 prompt-level strict vs 68.4 for the base). Because the reported checkpoint is selected on the test set, the IFEval number is not an unbiased estimate of the method's performance, and it directly supports the claim of state-of-the-art among comparable-size models. Please either select checkpoints on a held-out set that is not used for reporting (for example, a distinct validation split or another benchmark such as Multi-IF), report the final checkpoint's performance together with variance across checkpoints, or explicitly reframe the IFEval result as validation-selected.
  2. [Section 2.2, Table 1; Section 3.3; Table 4] The reliability of the LLM soft-constraint verifier is load-bearing but not established. In Section 2.2, Table 1, QwQ-32B achieves 48.1% soft-constraint accuracy on IFBench pairwise judgments, which is essentially at chance, yet the text concludes that soft constraints 'can be reliably verified by advanced LLMs.' The reward used in RL (Section 3.3) is a different task: a single-pass binary judgment of all soft constraints together, for which no accuracy, calibration, or human agreement is reported. Table 4 shows that removing the LLM component lowers IFEval from 84.5 to 74.7, so this signal drives a large part of the reported gain. The paper's own Limitations section acknowledges LLM-as-a-judge bias and adversarial vulnerability but does not quantify them for this specific reward. Please provide evidence that the single-pass binary judgment is reliable (for example, agreement with human labels on a sample of VERINSTRUCT), or show that the trained policy's gains persist when the soft judge is replaced by a more robust or calibrated one.
minor comments (7)
  1. [Section 4.5, Table 4] The ablation definitions are ambiguous: the text says 'w/o code*' uses only the LLM to verify all constraints, and 'w/o code' uses only the LLM for soft constraints, but since VERIF already uses code for hard constraints, 'w/o code' as written also removes code from hard constraints and thus also uses only the LLM for all constraints. The two conditions appear indistinguishable as described, yet they produce different results (81.7 vs 76.2). Please clarify what actually differs between these ablations and correct the descriptions.
  2. [Appendix B, Table 2] The Conifer entry in Table 2 is from a model retrained by the authors on Conifer's released SFT and DPO data, not the actual Conifer model. This reproduction should be clearly flagged in the table caption or footnote, as it affects interpretation of the comparison.
  3. [References] The reference entry for QwQ-32B ('Qwq-32b: Reward model for factuality and harmlessness') appears incorrect; QwQ-32B is a general reasoning model, not a reward model. Please correct or remove this citation.
  4. [References] The duplicated reference for Yue et al. (2025a and 2025b) should be merged, and the 'TULU 3' naming appears as 'TULU' inconsistently in some places.
  5. [Figure 3] Figure 3 is difficult to read in the text version due to small axes and missing numeric value labels; please ensure high resolution and add explicit value labels so the per-constraint accuracies are legible.
  6. [Throughout] There are several typos, including 'decription' in Section 3.1, 'OpenAssitant' in Section 3.2, and 'but but' in Related Work. A careful proofreading pass is needed.
  7. [General evaluation] The paper reports single runs for all RL training and evaluation. Reporting multiple seeds or at least confidence intervals would strengthen the reliability of the headline numbers, especially given the checkpoint selection procedure.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the VerIF reward combines an external code check with an external LLM judge, and the claimed improvements are measured on external benchmarks; the paper's self-citations are minor and do not force the results.

full rationale

VerIF's reward function is defined as VERIF(x,y) = F(Code(y, Ch), LLM(y, Cs)) with F as averaging; the code component is a deterministic checker and the LLM component is an external model (QwQ-32B), not the policy being trained. The policy models (TULU 3 SFT, DeepSeek-R1-Distill-Qwen-7B) are distinct from the verifier, so the reward is not self-referential. The central experimental claims are evaluated on external instruction-following benchmarks (IFEval, Multi-IF, SysBench, FollowBench, CFBench) and general benchmarks (AlpacaEval 2.0, MT-Bench, GSM8K, etc.), not on the training reward itself. The paper discloses that IFEval is used as a validation set to select the best checkpoint, so reporting IFEval results is a form of model selection rather than an independent prediction; this is a methodological caveat, but not a circular derivation. The self-citations to IFBench (Peng et al., 2025) and constraint back-translation (Qi et al., 2024) motivate design choices, but the hybrid reward's benefit is independently supported by the ablation study in Table 4, which shows that removing either component degrades performance on external benchmarks. No fitted parameter is renamed as a prediction, and no uniqueness theorem or ansatz is smuggled in via self-citation. The main weakness is reward reliability: Table 1 reports only 48.1% soft-constraint accuracy for QwQ-32B on IFBench while the text claims soft constraints 'can be reliably verified by advanced LLMs'; this is a validity concern, not circularity. Overall, the paper is not circular; it receives a low score only for minor self-citation in the design rationale.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

No invented physical entities are proposed; the dataset and verifier are concrete released artifacts. Free parameters are standard RL hyperparameters and design choices; none are fitted to the final evaluation benchmarks, but hyperparameters like the aggregation function and rollout count shape the training signal. Axioms are the domain assumptions about verifiability and judge reliability, plus the verification-selection choice for IFEval.

free parameters (6)
  • number of rollouts per prompt = 16
    Chosen for GRPO value estimation; affects reward variance and compute, not fitted to data.
  • KL loss coefficient = 1e-3
    GRPO KL penalty; standard choice.
  • learning rate (RL) = 1e-6
    Learning rate for GRPO training.
  • learning rate (verifier SFT) = 2e-5
    Learning rate for IF-Verifier-7B training.
  • aggregation function F = average
    Reward combination of code and LLM scores; could be multiplication, choice affects training signal.
  • minimum constraints per instance = 2
    Dataset filtering threshold in Section 3.2.
assumptions (4)
  • domain assumption Hard constraints (length, keyword, formatting) can be reliably checked by automatically generated Python code
    Pilot results on IFBench (Table 1) support this, but accuracy on VerInstruct itself is not measured.
  • domain assumption Soft constraints (style, content, etc.) can be reliably judged by a single-pass binary decision from QwQ-32B
    Assumed in Section 3.3; the authors acknowledge LLM-judge biases and adversarial vulnerability in Limitations.
  • ad hoc to paper IFEval is a valid validation set for selecting the final RL checkpoint
    Appendix B; this introduces selection bias when the same IFEval result is reported as a headline result in Table 2.
  • domain assumption Constraint back-translation produces constraints that are actually satisfied by the given responses
    Section 3.2 relies on Qi et al. (2024); no manual verification on the full dataset.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VerIF: Verification Engineering for Reinforcement Learning in Instruction Following." pith.science (2026). https://pith.science/paper/CQWJ3FH2

@misc{pith2026250609942,
  author       = {Pith},
  title        = {Pith review of: VerIF: Verification Engineering for Reinforcement Learning in Instruction Following},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CQWJ3FH2}},
  note         = {Machine review of arXiv:2506.09942}
}
read the original abstract

Reinforcement learning with verifiable rewards (RLVR) has become a key technique for enhancing large language models (LLMs), with verification engineering playing a central role. However, best practices for RL in instruction following remain underexplored. In this work, we explore the verification challenge in RL for instruction following and propose VerIF, a verification method that combines rule-based code verification with LLM-based verification from a large reasoning model (e.g., QwQ-32B). To support this approach, we construct a high-quality instruction-following dataset, VerInstruct, containing approximately 22,000 instances with associated verification signals. We apply RL training with VerIF to two models, achieving significant improvements across several representative instruction-following benchmarks. The trained models reach state-of-the-art performance among models of comparable size and generalize well to unseen constraints. We further observe that their general capabilities remain unaffected, suggesting that RL with VerIF can be integrated into existing RL recipes to enhance overall model performance. We have released our datasets, codes, and models to facilitate future research at https://github.com/THU-KEG/VerIF.

Figures

Figures reproduced from arXiv: 2506.09942 by the authors.

Figure 1
Figure 1. A simplified illustration of VERIF. The in￾struction constraints are categorized as soft or hard and verified using different methods in VERIF. In this work, we explore verification engineering for reinforcement learning in instruction following. Specifically, this work focuses on the following of constraints in the instruction (Zhou et al., 2023), such as response length, as shown in [PITH_FULL_IMAGE:figures/full_… view at source ↗
Figure 3
Figure 3. Accuracy (%) of code-only or LLM-only verification in verifying compliance with different types of constraints. LLM-only adopts QwQ-32B. constraints and LLM verification performs better on soft constraints, supporting the rationale for the code+LLM verification approach. The detailed re￾sults across different constraint types are shown in [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Left: The construction process for VERINSTRUCT, including complex instruction generation and verification construction. Right: Our verification method, VERIF, providing verification for instruction following. Code(y, Ch) ∈ {0, 1} denotes whether y satis￾fies all hard constraints in Ch, and LLM(y, Cs) ∈ {0, 1} indicates whether y satisfies all soft con￾straints in Cs. F denotes the aggregation method used to combine … view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Prompt-level strict scores (%) across different [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Reward curves during RL training with dif [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: shows the distribution of 22, 000 in￾stances in the VERINSTRUCT dataset. Following IFBench (Peng et al., 2025), we categorize con￾straints into five types: length, keyword, format, content, and style. The left chart presents the pro￾portional distribution of constraint…

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LsrIF: Enhancing Logic-Structured Instruction Following of Large Language Models

    cs.AI 2026-01 conditional novelty 6.0 of 10

    Logic-structured rewards—averaging parallel constraints, decaying rewards after sequential failures, rewarding only the active conditional branch—improve instruction-following and transfer to reasoning.

  2. Reasoning Up the Instruction Ladder for Controllable Language Models

    cs.CL 2025-10 conditional novelty 6.0 of 10

    RLVR on ~7K verifiable system/user conflict examples teaches LLMs to prioritize higher-priority instructions, improving instruction-hierarchy and safety benchmarks.

  3. Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Models

    cs.LG 2025-10 conditional novelty 6.0 of 10

    A dynamic replay and reweighting scheduler (RECAP) preserves general capabilities during RLVR while keeping reasoning performance at least as good as reasoning-only finetuning.

Reference graph

Works this paper leans on

66 extracted references · 22 canonical work pages · cited by 3 Pith papers

  1. [1]

    Kaikai An, Li Sheng, Ganqu Cui, Shuzheng Si, Ning Ding, Yu Cheng, and Baobao Chang. 2025. Ultraif: Advancing instruction following from the wild. arXiv preprint arXiv:2502.04153

    work page arXiv 2025

  2. [2]

    BAAI. 2024. Infinity instruct. arXiv preprint arXiv:2406.XXXX

    work page 2024

  3. [3]

    Thomas Ball, Shuo Chen, and Cormac Herley. 2024. Can we count on llms? the fixed-effect fallacy and claims of gpt-4 capabilities. Transactions on Machine Learning Research

    work page 2024

  4. [4]

    ByteDance-Seed. 2025. https://arxiv.org/abs/2504.13914 Seed1.5-thinking: Advancing superb reasoning models with reinforcement learning . Preprint, arXiv:2504.13914

    arXiv 2025

  5. [5]

    Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. 2024. Huatuogpt-o1, towards medical complex reasoning with llms. arXiv preprint arXiv:2412.18925

    arXiv 2024

  6. [6]

    Jiale Cheng, Xiao Liu, Cunxiang Wang, Xiaotao Gu, Yida Lu, Dan Zhang, Yuxiao Dong, Jie Tang, Hongning Wang, and Minlie Huang. 2024. Spar: Self-play with tree-search refinement to improve instruction-following in large language models. arXiv preprint arXiv:2412.11605

    arXiv 2024

  7. [7]

    Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. 2025. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.17161

    arXiv 2025

  8. [8]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

Show all 66 references
  1. [9]

    Guanting Dong, Keming Lu, Chengpeng Li, Tingyu Xia, Bowen Yu, Chang Zhou, and Jingren Zhou. 2024. Self-play with execution feedback: Improving instruction-following capabilities of large language models. arXiv preprint arXiv:2406.13542

    2024 arXiv

  2. [10]

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of NAACL-HLT, pages 2368--2378

    2019

  3. [11]

    Yann Dubois, Bal \'a zs Galambosi, Percy Liang, and Tatsunori B Hashimoto. 2024. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475

    2024 arXiv

  4. [12]

    Shahul Es. 2023. Orca-chat: A high-quality explanation-style chat dataset

    2023

  5. [13]

    Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. 2025. Retool: Reinforcement learning for strategic tool use in llms. arXiv preprint arXiv:2504.11536

    2025 arXiv

  6. [14]

    Tairan Fu, Raquel Ferrando, Javier Conde, Carlos Arriaga, and Pedro Reviriego. 2024. Why do large language models (llms) struggle to count letters? arXiv preprint arXiv:2412.18626

    2024 arXiv

  7. [15]

    Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2024. Bias and fairness in large language models: A survey. Computational Linguistics, 50(3):1097--1179

    2024

  8. [16]

    Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Chenghao Ma, Shanghaoran Quan, Liang Chen, Qingxiu Dong, Runxin Xu, and 1 others. 2025. Omni-math: A universal olympiad level mathematic benchmark for large language models. In Proceedings of ICLR

    2025

  9. [17]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  10. [18]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  11. [19]

    Minggui He, Yilun Liu, Shimin Tao, Yuanchang Luo, Hongyong Zeng, Chang Su, Li Zhang, Hongxia Ma, Daimeng Wei, Weibin Meng, and 1 others. 2025. R1-t1: Fully incentivizing translation capability in llms via reasoning learning. arXiv preprint arXiv:2502.19735

    2025 arXiv

  12. [20]

    Qianyu He, Jie Zeng, Qianxi He, Jiaqing Liang, and Yanghua Xiao. 2024 a . From complex to simple: Enhancing multi-constraint complex instruction following ability of large language models. In Findings of ACL, pages 10864--10882

    2024

  13. [21]

    Yun He, Di Jin, Chaoqi Wang, Chloe Bi, Karishma Mandyam, Hejia Zhang, Chen Zhu, Ning Li, Tengyu Xu, Hongjiang Lv, Shruti Bhosale, Chen Zhu, Karthik Abinav Sankararaman, Eryk Helenowski, Melissa Hall Melanie Kambadur, Aditya Tayade, Hao Ma, Han Fang, and Si-Ying Wang. 2024 b . ...

    2024 arXiv

  14. [22]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  15. [23]

    Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. 2024. Followbench: A multi-level fine-grained constraints following benchmark for large language models. In Proceedings of ACL, pages 4667--4688

    2024

  16. [24]

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516

    2025 arXiv

  17. [25]

    o pf, Yannic Kilcher, Dimitri von R \

    Andreas K \"o pf, Yannic Kilcher, Dimitri von R \"u tte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Rich \'a rd Nagyfi, and 1 others. 2024. Openassistant conversations-democratizing large language model alignment. Advances i...

    2024

  18. [26]

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, and 1 others. 2024. T " ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124

    2024 arXiv

  19. [27]

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, and 1 others. 2024. From generation to judgment: Opportunities and challenges of llm-as-a-judge. arXiv preprint arXiv:2411.16594

    2024

  20. [28]

    Xuefeng Li, Haoyang Zou, and Pengfei Liu. 2025. Torl: Scaling tool-integrated rl. arXiv preprint arXiv:2503.23383

    2025 arXiv

  21. [29]

    Zhaowei Liu, Xin Guo, Fangqi Lou, Lingfeng Zeng, Jinyi Niu, Zixuan Wang, Jiajie Xu, Weige Cai, Ziwei Yang, Xueqian Zhao, and 1 others. 2025 a . Fin-r1: A large language model for financial reasoning through reinforcement learning. arXiv preprint arXiv:2503.16252

    2025

  22. [30]

    Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. 2025 b . Inference-time scaling for generalist reward modeling. arXiv preprint arXiv:2504.02495

    2025

  23. [31]

    Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. 2025 a . Deepcoder: A fully open-source 14b coder at o3-mini level. Notion Blog

    2025

  24. [32]

    Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica

    Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. 2025 b . Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl. Notion Blog

    2025

  25. [33]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. In Proceedings of NeurIPs

    2022

  26. [34]

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277

    2023 arXiv

  27. [35]

    Hao Peng, Yunjia Qi, Xiaozhi Wang, Zijun Yao, Bin Xu, Lei Hou, and Juanzi Li. 2025. Agentic reward modeling: Integrating human preferences with verifiable correctness signals for reliable reward systems. arXiv preprint arXiv:2502.19328

    2025 arXiv

  28. [36]

    Chau Pham, Simeng Sun, and Mohit Iyyer. 2024. Suri: Multi-constraint instruction following in long-form text generation. In Findings of ACL, pages 1722--1753

    2024

  29. [37]

    Yunjia Qi, Hao Peng, Xiaozhi Wang, Bin Xu, Lei Hou, and Juanzi Li. 2024. Constraint back-translation improves complex instruction following of large language models. arXiv preprint arXiv:2410.24175

    2024 arXiv

  30. [38]

    Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-T \"u r, Gokhan Tur, and Heng Ji. 2025 a . Toolrl: Reward is all tool learning needs. arXiv preprint arXiv:2504.13958

    2025 arXiv

  31. [39]

    Lingfei Qian, Weipeng Zhou, Yan Wang, Xueqing Peng, Han Yi, Jimin Huang, Qianqian Xie, and Jianyun Nie. 2025 b . Fino1: On the transferability of reasoning enhanced llms to finance. arXiv preprint arXiv:2502.08127

    2025 arXiv

  32. [40]

    Yanzhao Qin, Tao Zhang, Yanjun Shen, Wenjing Luo, Haoze Sun, Yan Zhang, Yujing Qiao, Weipeng Chen, Zenan Zhou, Wentao Zhang, and 1 others. 2024. Sysbench: Can large language models follow system messages? arXiv preprint arXiv:2408.10943

    2024

  33. [41]

    Team Qwen. 2025. https://qwenlm.github.io/blog/qwq-32b/ Qwq-32b: Reward model for factuality and harmlessness . Accessed: 2025-05-13

    2025

  34. [42]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Proceedings of NeurIPs

    2023

  35. [43]

    Qingyu Ren, Jie Zeng, Qianyu He, Jiaqing Liang, Yanghua Xiao, Weikang Zhou, Zeye Sun, and Fei Yu. 2025. Step-by-step mastery: Enhancing soft constraint following ability of large language models. arXiv preprint arXiv:2501.04945

    2025 arXiv

  36. [44]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Jun-Mei Song, Mingchuan Zhang, Y. K. Li, Yu Wu, and Daya Guo. 2024. https://api.semanticscholar.org/CorpusID:267412607 Deepseekmath: Pushing the limits of mathematical reasoning in open language models . ArXiv, abs/2402.03300

    2024 arXiv

  37. [45]

    Jiawen Shi, Zenghui Yuan, Yinuo Liu, Yue Huang, Pan Zhou, Lichao Sun, and Neil Zhenqiang Gong. 2024. Optimization-based prompt injection attack to llm-as-a-judge. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 660--674

    2024

  38. [46]

    Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. 2022. Defining and characterizing reward hacking. Advances in Neural Information Processing Systems, 35:9460--9471

    2022

  39. [47]

    Haoran Sun, Lixin Liu, Junjie Li, Fengyu Wang, Baohua Dong, Ran Lin, and Ruohui Huang. 2024. Conifer: Improving complex constrained instruction-following ability of large language models. arXiv preprint arXiv:2404.02823

    2024 arXiv

  40. [48]

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, and 1 others. 2023. Challenging big-bench tasks and whether chain-of-thought can solve them. In Findings of ACL

    2023

  41. [49]

    Rush, and Thomas Wolf

    Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alvaro Bartolome, Alexander M. Rush, and Thomas Wolf. https://github.com/huggingface/alignment-handbook The Alignment Handbook

  42. [50]

    Bingning Wang, Haizhou Zhao, Huozhi Zhou, Liang Song, Mingyu Xu, Wei Cheng, Xiangrong Zeng, Yupeng Zhang, Yuqi Huo, Zecheng Wang, and 1 others. 2025. Baichuan-m1: Pushing the medical capability of large language models. arXiv preprint arXiv:2502.12671

    2025 arXiv

  43. [51]

    Jiaan Wang, Fandong Meng, Yunlong Liang, and Jie Zhou. 2024 a . Drt-o1: Optimized deep reasoning translation via long chain-of-thought. arXiv e-prints, pages arXiv--2412

    2024

  44. [52]

    Junqiao Wang, Zeng Zhang, Yangfan He, Yuyang Song, Tianyu Shi, Yuchen Li, Hengyuan Xu, Kunyu Wu, Guangwu Qian, Qiuwu Chen, and 1 others. 2024 b . Enhancing code llms with reinforcement learning in code generation. arXiv preprint arXiv:2412.20367

    2024 arXiv

  45. [53]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, and 1 others. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Informa...

  46. [54]

    Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. 2025. Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2502.14768

    2025 arXiv

  47. [55]

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244

    2023 arXiv

  48. [56]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  49. [57]

    Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, and 1 others. 2024. Justice or prejudice? quantifying biases in llm-as-a-judge. In Neurips Safe Generative AI Workshop 2024

    2024

  50. [59]

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. 2025 b . https://arxiv.org/abs/2504.13837 Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? Preprint, arXiv:2504.13837

    2025 arXiv

  51. [60]

    Tao Zhang, Yanjun Shen, Wenjing Luo, Yan Zhang, Hao Liang, Fan Yang, Mingan Lin, Yujing Qiao, Weipeng Chen, Bin Cui, and 1 others. 2024 a . Cfbench: A comprehensive constraints-following benchmark for llms. arXiv preprint arXiv:2408.01122

    2024

  52. [61]

    Xinghua Zhang, Haiyang Yu, Cheng Fu, Fei Huang, and Yongbin Li. 2024 b . Iopo: Empowering llms with complex instruction following via input-output preference optimization. arXiv preprint arXiv:2411.06208

    2024 arXiv

  53. [62]

    Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. Wildchat: 1m chatgpt interaction logs in the wild. In Proceedings of ICLR

    2024

  54. [63]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

  55. [64]

    Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. 2025. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments. arXiv preprint arXiv:2504.03160

    2025 arXiv

  56. [65]

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911

    2023 arXiv

  57. [66]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  58. [67]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.