Pith. sign in

REVIEW 4 major objections 5 minor 35 references

This paper claims that preference cycles in LLM judge feedback are measurement noise, and removing a minimum feedback arc set before policy optimization yields more stable, better-performing RLAIF than pairwise, listwise, pointwise, or ELO

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 09:22 UTC pith:KYYVXZVZ

load-bearing objection A real idea—FAS deconfliction as a reward front-end—but as submitted the method is under-specified (minimum FAS is non-unique) and the empirical claims outrun the evidence. the 4 major comments →

arxiv 2510.15514 v3 pith:KYYVXZVZ submitted 2025-10-17 cs.AI

Voting with the Graph: Stable RLAIF via Topological Consistency Maximization

Boyin Liu , Zhuo Zhang , Sen Huang , Lipeng Xie , Qingxu Fu , Haoran Chen , LI YU , Tianyi Hu
show 3 more authors
Zhaoyang Liu Bolin Ding Dongbin Zhao
This is my paper
classification cs.AI
keywords RLAIFLLM judgepreference cyclesfeedback arc setdirected acyclic graphreward signaltransitivitytraining stability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper is trying to establish that preference cycles—A beats B, B beats C, C beats A—in LLM judge feedback are largely stochastic measurement errors, and that deleting the smallest set of edges needed to break them turns conflicted judgments into a reward signal that makes reinforcement learning from AI feedback (RLAIF) more stable and more accurate. It introduces a diagnostic metric, CDR, that measures how often a judge's pairwise verdicts contain a cycle, and a reward front-end, DGR, that builds a directed preference graph, removes a minimum feedback arc set to make it acyclic, and scores each response by net wins (out-degree minus in-degree) in the cleaned graph. On Arena-Hard, MT-Bench, and WritingBench, DGR integrated with GRPO and GSPO outperforms pointwise, listwise, pairwise win-rate, and ELO baselines, and remains stable across different judge models and group sizes. A sympathetic reader would care because the fix is cheap and modular: it does not require retraining the judge or changing the policy optimizer.

Core claim

The paper's central claim is that logical coherence of AI feedback is a separable, addressable dimension of reward quality, and it operationalizes this through topological consistency maximization. Specifically, it reports that current LLM judges produce preference cycles in roughly 1.5 to 6.7 percent of evaluated samples, that these cycles violate the transitivity assumptions underlying preference learning, and that explicitly resolving them by minimum-feedback-arc-set deletion—not by random cycle-breaking or edge reversal—improves downstream policy performance. The resulting reward is a net-win score computed on a DAG, which the paper argues preserves the maximum amount of original prefere

What carries the argument

Deconflicted Graph Rewards (DGR): a three-stage reward front-end that (1) constructs a semicomplete directed preference graph from all pairwise judge judgments, with ties contributing no edge; (2) converts the graph into a Directed Acyclic Graph by removing a minimum feedback arc set, solved exactly by exhaustive search when the group has at most 10 responses and approximated for larger groups; and (3) assigns each response a reward equal to out-degree minus in-degree in the DAG. These net-win scores are normalized into advantages for group-based optimizers such as GRPO and GSPO. The companion diagnostic, Conflict Detection Rate (CDR), uses strongly connected components of the preference gra

Load-bearing premise

The claim rests on treating observed preference cycles as stochastic measurement errors rather than genuine intransitivity; if judges' cycles are real (for instance, reflecting multi-criteria trade-offs), DGR removes true signal and the reported gains would be an artifact of discarding information the model disagreed with.

What would settle it

Collect many repeated, re-prompted judgments for the same response triples and check whether the cycle edges that DGR removes survive at high confidence. If a meaningful fraction are stable and reproducible, or if a dataset with known genuine intransitivity shows DGR hurting rather than helping downstream performance, the noise model and the central claim would be falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • DGR can be inserted in front of any group-based preference optimizer, so existing RLAIF pipelines can gain stability without retraining the judge or altering the learning algorithm.
  • CDR gives a label-free diagnostic that complements accuracy, revealing a trade-off in which more accurate judge prompts tend to produce more preference cycles.
  • If the noise model is right, reducing CDR during prompt or judge selection should reduce reward noise and improve RL stability in practice.
  • The ablation shows the resolution strategy matters: random cycle-breaking barely helps and edge reversal can hurt, so the minimum-feedback-arc-set choice carries the improvement.
  • DGR's advantage over win-rate rewards holds across different judge models and grows with a weaker judge, suggesting the clean-up matters most exactly when judgments are noisiest.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the paper's noise assumption: re-query the judge on the edges DGR removes, repeatedly and with varied prompts. If removed edges are reproduced at high confidence, some cycles encode genuine preference structure and DGR is deleting real signal rather than noise.
  • The net-win score weights every surviving edge equally; a natural extension would weight edges by judge confidence or margin, potentially improving reward calibration and training stability further.
  • CDR could be used adaptively during training—for example, shrinking the comparison group or switching prompts when the measured conflict rate in a batch spikes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a diagnostic metric, Conflict Detection Rate (CDR), and a reward-generation front-end, Deconflicted Graph Rewards (DGR), for RLAIF. DGR constructs a directed preference graph from all pairwise LLM judgments over G responses, removes a minimum feedback arc set to make the graph a DAG, and assigns each response a net-win score (out-degree minus in-degree). These scores are normalized into advantages and used inside GRPO or GSPO. Experiments on Qwen3-14B/8B trained on 1,000 WildChat queries and evaluated on Arena-Hard, MT-Bench, and WritingBench report gains over pointwise, listwise, pairwise, and ELO baselines, along with ablations over conflict-resolution strategy, judge prompt, judge model, and graph size.

Significance. If the central claim is confirmed, DGR would be a cheap, model-agnostic preprocessing layer for preference-based RL, and CDR would be a useful diagnostic for judge consistency. The paper has real strengths: the graph-construction idea is simple and modular; the final evaluation uses GPT-4.1 and Claude-3.7 as judges, different from the Qwen3-32B training judge, which avoids the most direct circularity; and the code is promised to be public. However, the evidence as reported is not yet sufficient. The reward signal is not a well-defined function of the raw judgments because the minimum FAS is generally non-unique and no tie-breaking rule is given; the headline results are 'peak scores' from two runs with no variance or significance testing; and the paper's central assumption that cycles are stochastic noise rather than genuine intransitivity is never validated. These issues are substantive but appear fixable within the manuscript's scope.

major comments (4)
  1. [§3.2.1, Stage 2, and Eq. (3)] DGR's reward signal is not well-defined as stated. A minimum feedback arc set is generally non-unique. For a 3-cycle A≻B≻C≻A, deleting any one edge is a minimum FAS, and the three resulting DAGs give net-win vectors (A=1,B=0,C=-1), (A=-1,B=1,C=0), and (A=0,B=-1,C=1), respectively. The paper does not specify a tie-breaking rule, an aggregation over all optimal FAS sets, or a deterministic implementation. Therefore Eq. (3) can return different reward vectors for identical raw judgments, and the 'DGR (Optimal Resolution)' entry in Table 2 is not reproducible from the text. Please specify a canonical tie-break, average over all optimal FAS sets, or provide the exact deterministic algorithm and seed used in the experiments.
  2. [§4.1, Evaluation Protocol; Tables 1-3] The paper reports 'the peak score achieved across all training steps from two independent experimental runs.' Peak-over-training is an optimistically biased statistic rather than an estimate of expected performance, and with n=2 there is no variance, confidence interval, or significance test. Table differences of 0.5-1.5 points are therefore not sufficient to support claims such as 'significantly improves' or 'dominant performance.' Please report full training curves and mean ± standard deviation or standard error over at least 3-5 seeds, with paired significance tests or bootstrap intervals for the headline comparisons.
  3. [§4.4.2, Table 3] The correlation analysis uses only four prompts (P2-P5). With n=4, the reported correlations, including r=0.98 between CDR and accuracy and the near-zero r for DGR, are highly sensitive to a single data point and cannot support the paper's 'accuracy-consistency dilemma' or 'DGR is uncorrelated with signal quality' conclusions. Moreover, P5 is also the prompt used in Table 1 and Table 2, so the four-point set is not an independent spread. Please add more prompt conditions or replace the correlation claim with paired per-prompt comparisons and confidence intervals.
  4. [Abstract and §3.2.1] The method's justification is that observed preference cycles arise mainly from stochastic measurement error, not genuine intransitivity. This is stated as an assumption but never tested. If LLM or human preferences are legitimately non-transitive (e.g., multi-criteria trade-offs), then removing a minimum FAS deletes true signal and the reported gains could be an artifact of discarding disagreeing evidence. The paper should validate this assumption directly, for example by comparing the judge accuracy of removed FAS edges versus retained edges on RewardBench2 or another ground-truth preference dataset, and should discuss what happens when the noise model is violated.
minor comments (5)
  1. [Abstract vs. main text] The abstract refers to 'Topological Consensus Rewards (TCR)' and 'Cycle Incidence Rate (CIR),' while the full text consistently uses 'Deconflicted Graph Rewards (DGR)' and 'Conflict Detection Rate (CDR).' This suggests a version mismatch and should be fixed.
  2. [§4.1 and Figure 1] CDR depends on the number of candidates G in each sample, but the paper does not state G for the RewardBench2 evaluation. The reader must infer that one chosen plus three rejected responses gives G=4. Please state this explicitly.
  3. [Appendix C.1.3, Algorithm 7] The ELO baseline iterates over 'each comparison pair' without specifying the update order. Different orderings can lead to different final ratings. Please state the exact ordering or use a batch update.
  4. [§4.2 and Table 1] The pointwise baseline maps scores directly to rewards, but the DGR/ELO/PREF methods normalize within a group. The comparison would be cleaner if all methods used the same normalization before computing advantages.
  5. [§4.4.1, Table 2] DGR-RandomResolve and DGR-ReverseResolve use random edge selection without a specified seed. Since the reader cannot reproduce these ablations, please provide seeds or a deterministic protocol.

Circularity Check

0 steps flagged

No load-bearing circularity; DGR's benchmark claims are externally grounded and the pipeline is an explicit graph-theoretic construction.

full rationale

DGR is an algorithmic reward front-end, not a fitted predictor. Raw pairwise judgments form a semicomplete preference graph (§3.2.1 Stage 1); a minimum feedback arc set is removed to obtain a DAG (Stage 2, Eq. 2); net-win scores are computed (Stage 3, Eq. 3); and the headline results are measured on Arena-Hard, MT-Bench, and WritingBench by judges (GPT-4.1, Claude-3.7) different from the training judge (Qwen3-32B). There are no fitted parameters later relabeled as predictions, no imported uniqueness theorem from the authors, and the cited graph-theoretic tools (Slater 1961; Eades et al. 1993) are external and standard. The abstract's statement 'Under our noise model, these cycles arise primarily from stochastic measurement errors rather than genuine intransitivity' is a modeling premise, not a derived result; even if cycles are genuine intransitivity, the external benchmark comparisons would still be independent evidence about whether enforcing acyclicity helps RLAIF. The paper does not test whether removed edges were actually wrong judgments, and it does not specify tie-breaking among equally minimal feedback arc sets (so Eq. (3) is implementation-dependent on cyclic graphs); these are correctness/reproducibility limitations, not circular derivations. The supplied abstract (TCR/CIR) also does not match the full text's method (CDR/DGR), a manuscript-consistency issue that does not affect the external grounding of the experiments. I therefore find no load-bearing circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central algorithm depends on no fitted constants, but it depends on several untested domain assumptions: cycles equal noise, minimum FAS equals optimal denoising, true preferences are transitive, and LLM judge judgments are a valid preference proxy. These are not independently validated in the paper.

free parameters (3)
  • Group size G (rollouts per query) = not stated; sensitivity n=4-7
    Determines graph size and conflict frequency; the main results likely use a fixed G but the value is not reported in the main text.
  • GRPO/GSPO hyperparameters (epsilon, beta, mu) = not reported
    Policy clipping, KL coefficient, and update epochs are hand-set and not given, making exact reproduction impossible from the paper alone.
  • Reward prompt selection (P1-P5) = P1 for main results, P5 for ablation
    Prompt choice changes the accuracy/CDR trade-off and may affect the size of the reported DGR advantage.
axioms (4)
  • domain assumption Preference cycles in LLM judge feedback are stochastic measurement errors rather than genuine intransitivity.
    Abstract states this under 'our noise model'; the full text contains no noise model and no ground-truth test. DGR deletes cycle edges based on this assumption.
  • ad hoc to paper A minimum feedback arc set deletion preserves the true preference signal while removing only noise.
    Stage 2 (§3.2.1) equates minimum FAS with signal purification; the paper does not verify that the deleted edges are actually incorrect judgments.
  • domain assumption The true preference relation is transitive (a strict partial order).
    Definition 3.1 and Stage 2 treat any cycle as a conflict, ruling out genuinely non-transitive human preferences that may arise from multi-criteria trade-offs.
  • domain assumption LLM judge pairwise judgments are a usable proxy for human preferences.
    Standard RLAIF assumption adopted without supporting evidence beyond citing prior RLAIF work.

pith-pipeline@v1.3.0-alltime-deepseek · 17411 in / 13269 out tokens · 108352 ms · 2026-08-04T09:22:03.039734+00:00 · methodology

0 comments
read the original abstract

Reinforcement Learning from AI Feedback (RLAIF) relies on LLM judges as preference measurement instruments, yet these instruments are fundamentally limited by random measurement errors -- stochastic fluctuations that manifest as preference cycles (e.g., $A \succ B \succ C \succ A$), occurring in 5-9% of evaluations across state-of-the-art models. While repeated sampling mitigates noise by averaging multiple judgments, it treats each comparison in isolation and fails to exploit the structural constraints that distinguish systematic signals from random noise. We introduce Topological Consensus Rewards (TCR), a framework that leverages transitivity as a denoising mechanism via topological majority voting: systematic signals reinforce each other through transitive chains, while random errors cluster into topologically exposed cycles. TCR approximates the Maximum Acyclic Subgraph to filter stochastic noise from preference signals. We also propose Cycle Incidence Rate (CIR) as a diagnostic metric that measures the proportion of samples containing preference cycles. Under our noise model, these cycles arise primarily from stochastic measurement errors rather than genuine intransitivity. Experiments on Arena-Hard, MT-Bench, and WritingBench demonstrate that TCR consistently outperforms pairwise baselines and classical ranking algorithms, while exhibiting robust performance across different judge models.

Figures

Figures reproduced from arXiv: 2510.15514 by Bolin Ding, Boyin Liu, Dongbin Zhao, Haoran Chen, Lipeng Xie, Li Yu, Qingxu Fu, Sen Huang, Tianyi Hu, Zhaoyang Liu, Zhuo Zhang.

Figure 1
Figure 1. Figure 1: Conflict detection and mitigation results. (a) CDR and accuracy comparison across dif [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Sensitivity analysis of DGR compared to PREF on Arena-Hard. (a) Performance across [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 15 linked inside Pith

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Introducing claude 3.7 sonnet

    Anthropic. Introducing claude 3.7 sonnet. Anthropic Technical Report, 2025. URL https://www.anthropic.com/news/claude-3-7-sonnet

    2025

  3. [3]

    Constitutional ai: Harmlessness from ai feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022

    Pith/arXiv arXiv 2022

  4. [4]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39: 0 324--345, 1952

    1952

  5. [5]

    Less is more: Improving llm alignment via preference data selection

    Yongchao Deng, Yang Liu, Ruochen Zhang, Jiahao Wu, Tao Luo, and Jingang Li. Less is more: Improving llm alignment via preference data selection. arXiv preprint arXiv:2502.14560, 2025

    arXiv 2025

  6. [6]

    A fast and effective heuristic for the feedback arc set problem

    Peter Eades, Xuemin Lin, and William F Smyth. A fast and effective heuristic for the feedback arc set problem. Information processing letters, 47 0 (6): 0 319--323, 1993

    1993

  7. [7]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. arXiv preprint arXiv:2210.10760, 2023

    Pith/arXiv arXiv 2023

  8. [8]

    Writing-zero: Bridge the gap between non-verifiable tasks and verifiable rewards, 2025

    Ruipeng Jia, Yunyi Yang, Yongbo Gai, Kai Luo, Shihao Huang, Jianhe Lin, Xiaoxi Jiang, and Guanjun Jiang. Writing-zero: Bridge the gap between non-verifiable tasks and verifiable rewards, 2025. URL https://arxiv.org/abs/2506.00103

    Pith/arXiv arXiv 2025

  9. [9]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pp.\ 611--626, 2023

    2023

  10. [10]

    Rlaif: Scaling reinforcement learning from human feedback with ai feedback

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267, 2023

    Pith/arXiv arXiv 2023

  11. [11]

    From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    Aligning with logic: Measuring, evaluating and improving logical preference consistency in large language models

    Yinhong Liu, Zhijiang Guo, Tianya Liang, Ehsan Shareghi, Ivan Vuli \'c , and Nigel Collier. Aligning with logic: Measuring, evaluating and improving logical preference consistency in large language models. arXiv preprint arXiv:2410.02205, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Rewardbench 2: Advancing reward model evaluation

    Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A Smith, Hannaneh Hajishirzi, and Nathan Lambert. Rewardbench 2: Advancing reward model evaluation. arXiv preprint arXiv:2506.01937, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Gpt-4 technical report, 2024

    OpenAI. Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774

    Pith/arXiv arXiv 2024

  15. [15]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 27730--27744, 2022

    2022

  16. [16]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  17. [17]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pp.\ 1279--1297, 2025

    2025

  18. [18]

    Inconsistencies in a schedule of paired comparisons

    Patrick Slater. Inconsistencies in a schedule of paired comparisons. Biometrika, 48 0 (3/4): 0 303--312, 1961

    1961

  19. [19]

    Preference ranking optimization for human alignment

    Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 18990--18998, 2024

    2024

  20. [20]

    Depth-first search and linear graph algorithms

    Robert Endre Tarjan. Depth-first search and linear graph algorithms. SIAM Journal on Computing, 1 0 (2): 0 146--160, 1972

    1972

  21. [21]

    Kimi k2: Open agentic intelligence

    Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    Gram: A generative foundation reward model for reward generalization

    Chenglong Wang, Yang Liu, Zhihong Xu, Ruochen Zhang, Jiahao Wu, Tao Luo, Jingang Li, Xunliang Liu, Weiran Qi, Yujiu Yang, et al. Gram: A generative foundation reward model for reward generalization. arXiv preprint arXiv:2506.14175, 2025 a

    arXiv 2025

  23. [23]

    Gram-r²: Self-training generative foundation reward models for reward reasoning

    Chenglong Wang, Yang Liu, Zhihong Xu, Ruochen Zhang, Jiahao Wu, Tao Luo, Jingang Li, Xunliang Liu, Weiran Qi, Yujiu Yang, et al. Gram-r²: Self-training generative foundation reward models for reward reasoning. arXiv preprint arXiv:2509.02492, 2025 b

    arXiv 2025

  24. [24]

    Pref-grpo: Pairwise preference reward-based grpo for stable text-to-image reinforcement learning

    Yibin Wang, Zhimin Li, Yuhang Zang, Yujie Zhou, Jiazi Bu, Chunyu Wang, Qinglin Lu, Cheng Jin, and Jiaqi Wang. Pref-grpo: Pairwise preference reward-based grpo for stable text-to-image reinforcement learning. arXiv preprint arXiv:2508.20751, 2025 c

    Pith/arXiv arXiv 2025

  25. [25]

    A comprehensive survey of llm alignment techniques: Rlhf, rlaif, ppo, dpo and more

    Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Xiang-Bo Mao, Sitaram Asur, et al. A comprehensive survey of llm alignment techniques: Rlhf, rlaif, ppo, dpo and more. arXiv preprint arXiv:2407.16216, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    Writingbench: A comprehensive benchmark for generative writing

    Yuning Wu, Jiahao Mei, Ming Yan, Chenliang Li, Shaopeng Lai, Yuran Ren, Zijia Wang, Ji Zhang, Mengyue Wu, Qin Jin, et al. Writingbench: A comprehensive benchmark for generative writing. arXiv preprint arXiv:2503.05244, 2025

    arXiv 2025

  27. [27]

    On the algorithmic bias of aligning large language models with rlhf: Preference collapse and matching regularization

    Jiancong Xiao, Ziniu Li, Xingyu Xie, Emily Getzen, Cong Fang, Qi Long, and Weijie J Su. On the algorithmic bias of aligning large language models with rlhf: Preference collapse and matching regularization. Journal of the American Statistical Association, 0 (just-accepted): 0 1--21, 2025

    2025

  28. [28]

    A unified pairwise framework for rlhf: Bridging generative reward modeling and policy optimization

    Wenyuan Xu, Shujian Li, Haoqin Wang, Jiahui Chen, Xing Zhou, Xu Wang, et al. A unified pairwise framework for rlhf: Bridging generative reward modeling and policy optimization. arXiv preprint arXiv:2504.04950, 2025

    Pith/arXiv arXiv 2025

  29. [29]

    Qwen3 technical report, 2025

    An Yang et al. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388

    Pith/arXiv arXiv 2025

  30. [30]

    Wildchat: 1m chatgpt interaction logs in the wild

    Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: 1m chatgpt interaction logs in the wild. arXiv preprint arXiv:2405.01470, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    Group sequence policy optimization

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36: 0 46595--46623, 2023

    2023

  33. [33]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  34. [34]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  35. [35]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...