REVIEW 5 major objections 5 minor 32 references
Token-level Accept or Reject: A Micro Alignment Approach for Large Language Models
T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A 4-million-parameter external network can align LLMs by accepting or rejecting tokens one at a time, outperforming RLHF and DPO on safety and helpfulness benchmarks.
desk verdict A genuinely new token-level decoupled alignment idea, but the shared reward models between training and evaluation make the headline gains uninterpretable as alignment. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the accept-reject policy network, a three-layer fully-connected model that takes the prompt, partial response, and one candidate token's representation and outputs a binary decision. Around it, the machinery consists of: (1) an MDP in which states are prompt-plus-partial-response-plus-candidate-token and actions are accept/reject; (2) a hybrid top-k and top-p truncated candidate set from the frozen reference model, sorted by probability; (3) a token-level reward that is zero for non-terminal decisions and equals the reward-and-cost-model score minus a KL divergence penalty at the end; and (4) distributed Soft Actor-Critic training with two critic heads. The greedy decoding loop accepts the highest-probability token unless rejected, with a fallback that accepts the last candidate so generation always terminates.
What would settle it
Compute the preference-rate gains of MARA over the upstream LLM using a held-out reward model or human labels that were not used in training; if the gains shrink, vanish, or reverse on the same prompts, the reported improvements are largely an artifact of reward overoptimization rather than genuine alignment.
Extended reading notes
Core claim
The central claim is that sentence-level preference alignment can be decomposed into per-token accept/reject decisions, and that a compact three-layer fully-connected network trained by soft actor-critic on this token-level Markov decision process can steer a frozen supervised fine-tuned model's generation toward responses that score better on both helpfulness and harmlessness. The authors argue that the token-level reward in Eq. (8), which gives the reward-model score only at the terminal state and applies a KL penalty at every step, is a faithful token-level decomposition of the RLHF objective, enabling a micro model to learn where to intervene. The end-of-sequence reward comes from the beaver reward and cost models combined as $r(x,y)=\alpha_r R(x,y)-\alpha_c C(x,y)$. Their evidence is the preference-rate gains over the upstream LLM alone and over RLHF, DPO, and Aligner baselines, including cross-family transfer of an alignment model trained on one LLM to other LLMs.
Load-bearing premise
The central claim assumes the beaver-7b-reward and beaver-7b-cost models are a valid proxy for human preference, because the same two models define the training reward and the evaluation metric, and no human evaluation or held-out reward model is reported.
Editorial extensions
If this is right
- Aligning an 8B-parameter LLM would require training only a 4M-parameter alignment network, cutting the compute and memory footprint of RLHF and DPO by orders of magnitude.
- A single alignment model trained on one LLM can be transferred to other LLMs, including across the Llama and Mistral families, while retaining substantial preference-rate gains.
- Token-level accept/reject produces explicit decisions at each generation step, and the reported visualizations show that most acceptances occur within the top three candidate positions.
- MARA makes alignment an inference-time operation on a frozen model, so preference updates could be shipped by replacing the small network rather than re-running fine-tuning.
- Compared with Aligner, MARA reaches competitive preference rates with 4M parameters versus 7B parameters and higher token throughput.
Reading between the lines
- Editorial inference: because the training reward and the evaluation metric are the same beaver reward and cost models, part of the reported gains may reflect reward hacking rather than genuine alignment, and a human evaluation would be the decisive check.
- Editorial inference: MARA is effectively a learned inference-time sampling filter over a frozen model's top-k/top-p candidate sets, so it can in principle be combined with other decoding-time controls such as best-of-n sampling or explicit content filters.
- Editorial inference: the acceptance-distribution pattern suggests the gate learns a compact decision window, which could enable cheap auditing of exactly which tokens the alignment model modifies during generation.
- Editorial inference: a natural stress test is to apply the same accept-reject gate to adversarial, out-of-distribution, or multilingual prompts; if gains degrade there, the method's safety behavior is tied to the training distribution.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes MARA (Micro token-level Accept-Reject Aligning), a small three-layer fully-connected network that is trained, separately from the LLM, to make binary accept/reject decisions over candidate tokens generated by the reference model. The alignment task is formulated as an MDP with a sparse terminal reward defined by a weighted combination of the beaver reward and cost models, and the network is trained with a distributed Soft Actor-Critic algorithm. Experiments on PKU-SafeRLHF, BeaverTails, and HarmfulQA across seven Llama/Mistral models report large preference-rate improvements over the base LLMs, RLHF, DPO, and Aligner, as well as cross-model transfer of the trained alignment model.
Significance. The decoupled, parameter-efficient design is conceptually attractive: if a 4M-parameter model could indeed provide alignment that transfers across LLM families, it would be a practically useful contribution with clear computational advantages over RLHF/DPO-style fine-tuning. The paper explicitly claims full reproducibility with public code and trained models. However, the current validation is undermined by a circular evaluation setup in which the same reward/cost models define both the training reward and the preference-rate metric, by an unexplained internal inconsistency between Table 1 and Table 3, and by a contradiction between the promised public code and the ethics statement that withholds all materials. The practical significance is therefore contingent on a substantially revised evaluation with independent or human judgments.
major comments (5)
- [Section 4.1, Eqs. (12) and (13)] The evaluation metric and the training reward are defined by the same two beaver models: Eq. (12) computes the preference rate from beaver-7b-v1.0-reward and beaver-7b-v1.0-cost scores, while Eq. (13) defines the training reward r(x,y) as a weighted combination of exactly those two models. MARA is therefore explicitly optimized to maximize the very scores that determine win/tie/lose outcomes in the evaluation. The headline gains in Table 1 and the baseline comparisons in Table 2 are thus at least partially explained by reward overoptimization rather than by improved alignment with human preferences. The paper provides no held-out reward model, no independent safety classifier, and no human evaluation to break this loop. This is a load-bearing flaw because the central claim that MARA 'improves alignment' is not supported by the reported metric.
- [Table 1 vs. Table 3] The diagonal entry for Mistral-7B-v0.3 (training and inference on the same model) reports SafeRLHF +17.09%, BeaverTails +16.00%, and HarmfulQA +11.07% in Table 1, but the same configuration in Table 3 reports +9.05%, +9.14%, and +6.97%, respectively. The two tables cannot both describe the same experiment, and the paper does not explain the discrepancy. The corresponding diagonal entry for Llama-3.1-8B is consistent, which suggests the Mistral numbers involve a change in setup or a reporting error. This inconsistency directly affects the claimed cross-model compatibility results and must be resolved before the experimental claims can be accepted.
- [Abstract, Section 1, and Ethics Statement] The abstract and the introduction promise that the source code and trained models are publicly available at https://github.com/IAAR-Shanghai/MARA and https://huggingface.co/IAAR-Shanghai/MARA_AGENTS. However, the Ethics Statement says: 'To maintain anonymity during the review process, we temporarily withhold our trained models and training logs... Upon acceptance, we will release all materials.' The appendices do not contain the promised complete implementation source code. This contradiction makes the 'full reproducibility' claim unverifiable in the submitted form and is a direct breach of the paper's own stated commitment.
- [Sections 2.2 and 3.3, Eq. (8)] The claim that Eq. (8) is 'a token-level decomposition of the RLHF objective in Eq. 2' is asserted without proof. In particular, the accept-reject policy in Eq. (6) is restricted to a truncated candidate set obtained by top-k/top-p sampling, so the token distribution of the MARA policy is not the same as the full-vocabulary distribution of the reference model. No derivation shows that optimizing the SAC objective in Eqs. (9)-(10) with the sparse terminal reward in Eq. (8) is equivalent to optimizing the RLHF objective in Eq. (2), nor that the accept-reject policy class can represent the optimal RLHF policy. Additionally, the KL terms in Eq. (8) are written as DKL of two probability values for a single token rather than as divergences between full distributions, which is mathematically ill-defined as written.
- [Section 4.1 and Appendix B.1] The reported preference-rate improvements are presented without error bars, multiple seeds, or significance tests. The underlying win/tie/lose counts in Tables 7-10 imply small evaluation sets (e.g., roughly 200 comparisons for SafeRLHF), but the exact number of evaluation instances per dataset is never stated. Given the scale of the reported differences (sometimes over 60 percentage points), the absence of any variability measure makes it impossible to judge whether the improvements are stable or within the noise of the evaluation protocol.
minor comments (5)
- [Eq. (7)] The expression 't∼eTi' appears to be a typo and should probably read 't ∈ eTi' to match the notation used in Eq. (6).
- [Figures 3, 5, 6, and 7] The token-acceptance figures contain garbled '/uni0000001a/...' text that appears to be corrupted font encoding; as rendered, the figures are not interpretable and need to be regenerated.
- [Table 5] The column header 'Perference' is a typo and should read 'Preference'.
- [Section 4.2] The sentence 'MARA demonstrates competitive advantages on SafeRLHF (+5.60%) and BeaverTails (+4.37%) evaluation datasets, while showing stronger ablation effects on HH-RLHF and Ultra-Feedback datasets' uses 'ablation effects' where 'results' or 'improvements' is meant.
- [Section 4.1] The paper never states the number of evaluation examples per dataset; this should be specified so that the win/tie/lose statistics in Appendix B.1 can be interpreted.
Circularity Check
Training and evaluation share the same beaver reward/cost models, so headline preference-rate gains partly reflect reward overoptimization rather than independent alignment.
-
fitted input called prediction
[Section 4.1 (Reward Models, Eq. 13) and Appendix A.2 (Evaluation Metrics)]
"Reward Models. Our alignment training utilizes a combination of two specialized models: beaver-7b-v1.0-reward for utility evaluation and beaver-7b-v1.0-cost for safety evaluation ... The final reward r(x, y) is computed as: r(x, y) = αrR(x, y) − αcC(x, y) (13) ... [A.2] we utilize the scores from reward models: beaver-7b-v1.0-reward for helpfulness evaluation and beaver-7b-v1.0-cost for harmlessness assessment."
The preference-rate metric used for every headline gain is computed from beaver-7b-v1.0-reward and beaver-7b-v1.0-cost, and the training reward in Eq. (13) is the same weighted combination of exactly those two models. MARA's policy is explicitly optimized to maximize r(x, y), so beating the reference LLM on this scalar is a predictable consequence of optimization, not independent evidence of human alignment. The paper provides no human evaluation, no held-out reward model, and no re-scoring with an independent safety or helpfulness classifier. Table 5 confirms the policy tracks the chosen alpha_r:alpha_c weights (e.g., alpha_r:alpha_c=1:0 gives +37.57% helpfulness but -22.57% harmlessness), i.e., the method is optimizing the beaver scores that define the evaluation.
full rationale
The method's derivation chain itself is not circular: the token-level decomposition of Eq. (4) is a formal identity, and the MARA training objective in Eqs. (8)-(11) is a standard SAC-style RL objective with a stated reward. The circularity is in the empirical support for the central claim. The paper trains MARA to maximize r(x,y) = alpha_r R(x,y) - alpha_c C(x,y) using beaver-7b-v1.0-reward and beaver-7b-v1.0-cost, and then evaluates 'preference rate' with those same two models, declaring a win only when a response scores higher on both. A policy optimized against a scalar will tend to score higher on that scalar than an unoptimized reference policy, so the headline improvements are expected even if the reward models are imperfect proxies for human preference. This is not a self-citation issue: the beaver models are external, but their dual role as training reward and evaluation metric creates a closed loop. An independent evaluation (human study, held-out reward model, or a different safety/helpfulness classifier) would be needed to break the loop. The many ties in the win/tie/lose tables and the large trade-offs in Table 5 further indicate that the aggregate preference-rate improvements can be driven by reward-model-specific behavior rather than genuine alignment. Because the central empirical claim reduces to optimizing the same scoring function used for measurement, the circularity score is high, though the proposed architecture and training method remain independently meaningful contributions.
Assumptions & free parameters
free parameters (4)
- Reward weighting alpha_r and alpha_c =
Llama family: 1:1; Mistral family: 2:1
- KL divergence coefficient lambda =
0.1
- Top-k and Top-p candidate truncation =
k=50, p=0.95
- Minimum entropy threshold H =
2
assumptions (4)
- domain assumption The last hidden state of the reference LLM, together with the candidate token, contains enough information for a 3-layer MLP to make accept/reject decisions.
- domain assumption Beaver-7b reward and cost models are adequate proxies for human helpfulness and harmlessness.
- ad hoc to paper The token-level MDP with sparse terminal reward is a valid decomposition of the RLHF objective.
- domain assumption Candidate set truncation with top-k/top-p does not bias the alignment outcome.
Cite this review
Pith. "Pith review of Token-level Accept or Reject: A Micro Alignment Approach for Large Language Models." pith.science (2026). https://pith.science/paper/7KO2IX4Q
@misc{pith2026250519743,
author = {Pith},
title = {Pith review of: Token-level Accept or Reject: A Micro Alignment Approach for Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/7KO2IX4Q}},
note = {Machine review of arXiv:2505.19743}
}
read the original abstract
With the rapid development of Large Language Models (LLMs), aligning these models with human preferences and values is critical to ensuring ethical and safe applications. However, existing alignment techniques such as RLHF or DPO often require direct fine-tuning on LLMs with billions of parameters, resulting in substantial computational costs and inefficiencies. To address this, we propose Micro token-level Accept-Reject Aligning (MARA) approach designed to operate independently of the language models. MARA simplifies the alignment process by decomposing sentence-level preference learning into token-level binary classification, where a compact three-layer fully-connected network determines whether candidate tokens are "Accepted" or "Rejected" as part of the response. Extensive experiments across seven different LLMs and three open-source datasets show that MARA achieves significant improvements in alignment performance while reducing computational costs. The source code and implementation details are publicly available at https://github.com/IAAR-Shanghai/MARA, and the trained models are released at https://huggingface.co/IAAR-Shanghai/MARA_AGENTS.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
- [1]
-
[3]
[Accessed 24-01-2025]. [Anwar et al., 2024] Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al. Foundational challenges in assuring alignment and safety of large language models. arXiv preprint arXiv:2404.09932,
arXiv 2025
-
[4]
A general theo- retical paradigm to understand learning from human pref- erences
[Azar et al., 2024] Mohammad Gheshlaghi Azar, Zhao- han Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theo- retical paradigm to understand learning from human pref- erences. In International Conference on Artificial Intelli- gence and Statistics, pages 4447–4455. PMLR,
work page 2024
-
[5]
Training a helpful and harmless assistant with reinforce- ment learning from human feedback
[Bai et al., 2022a] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforce- ment learning from human feedback. arXiv preprint arXiv:2204.05862,
-
[7]
Red-teaming large language models using chain of utterances for safety-alignment
[Bhardwaj and Poria, 2023] Rishabh Bhardwaj and Sou- janya Poria. Red-teaming large language models using chain of utterances for safety-alignment. arXiv preprint arXiv:2308.09662,
arXiv 2023
-
[9]
Safe rlhf: Safe reinforcement learning from human feedback
[Dai et al., 2024] Josef Dai, Xuehai Pan, Ruiyang Sun, Ji- aming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. In The Twelfth International Con- ference on Learning Representations,
work page 2024
-
[10]
Raft: Reward ranked finetuning for generative foundation model alignment
[Dong et al., 2023] Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, SHUM KaShun, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research,
work page 2023
-
[11]
Helping or herding? re- ward model ensembles mitigate but do not eliminate re- ward hacking
[Eisenstein et al., 2023] Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D’Amour, DJ Dvi- jotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping or herding? re- ward model ensembles mitigate but do not eliminate re- ward hacking. arXiv preprint arXiv:2312.09244,
arXiv 2023
Show all 32 references
-
[12]
Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor
[Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on ma- chine learning, pages 1861–1870. PMLR,
2018
-
[13]
Pku-saferlhf: Towards multi-level safety alignment for llms with human preference
[Ji et al., 2024b] Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Qiu, Boxun Li, and Yaodong Yang. Pku-saferlhf: Towards multi-level safety alignment for llms with human preference. arXiv preprint arXiv:2406.15513,
-
[14]
Mistral 7b
[Jiang et al., 2023] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825,
2023 arXiv
-
[15]
Dynamic context selection for document-level neural machine translation via rein- forcement learning
[Kang et al., 2020] Xiaomian Kang, Yang Zhao, Jiajun Zhang, and Chengqing Zong. Dynamic context selection for document-level neural machine translation via rein- forcement learning. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Process- ing (...
2020
-
[17]
Rain: Your language mod- els can align themselves without finetuning
[Li et al., 2024] Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. Rain: Your language mod- els can align themselves without finetuning. InThe Twelfth International Conference on Learning Representations ,
2024
-
[18]
Webgpt: Browser-assisted question-answering with human feedback
[Nakano et al., 2021] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332,
2021 arXiv
-
[20]
Discovering language model behaviors with model-written evaluations
[Perez et al., 2023] Ethan Perez, Sam Ringer, Kamil ˙e Lukoˇsi¯ut˙e, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. Discovering language model behaviors with model-written evaluations. In 61st Annual Meeting of ...
2023
-
[21]
From r to q∗: Your lan- guage model is secretly a q-function
[Rafailov et al., 2024a] Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q∗: Your lan- guage model is secretly a q-function. arXiv preprint arXiv:2404.12358,
-
[22]
Self-critiquing models for assisting human eval- uators
[Saunders et al., 2022] William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human eval- uators. arXiv preprint arXiv:2206.05802,
2022 arXiv
-
[23]
Learning to summarize with human feedback
[Stiennon et al., 2020] Nisan Stiennon, Long Ouyang, Jef- frey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021,
2020
-
[24]
Llama 2: Open foundation and fine-tuned chat models
[Touvron et al., 2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288,
2023 arXiv
-
[25]
Beyond reverse kl: Gen- eralizing direct preference optimization with diverse diver- gence constraints
[Wang et al., 2023a] Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse kl: Gen- eralizing direct preference optimization with diverse diver- gence constraints. arXiv preprint arXiv:2309.16240,
-
[26]
Aligning large lan- guage models with human: A survey
[Wang et al., 2023b] Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large lan- guage models with human: A survey. arXiv preprint arXiv:2307.12966,
-
[27]
Recursively summarizing books with human feed- back
[Wu et al., 2021] Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Chris- tiano. Recursively summarizing books with human feed- back. arXiv preprint arXiv:2109.10862,
2021 arXiv
-
[28]
Rrhf: Rank responses to align language models with human feedback without tears
[Yuan et al., 2023] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302,
2023 arXiv
-
[29]
Token- level direct preference optimization
[Zeng et al., 2024] Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token- level direct preference optimization. In Forty-first Interna- tional Conference on Machine Learning,
2024
-
[30]
Llamafactory: Unified efficient fine- tuning of 100+ language models
[Zheng et al., 2024] Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine- tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Lingui...
2024
-
[31]
[Zhong et al., 2024] Han Zhong, Guhao Feng, Wei Xiong, Li Zhao, Di He, Jiang Bian, and Liwei Wang
Association for Computational Linguistics. [Zhong et al., 2024] Han Zhong, Guhao Feng, Wei Xiong, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf. arXiv preprint arXiv:2404.18922,
2024 arXiv
-
[32]
Fine-tuning lan- guage models from human preferences
[Ziegler et al., 2019] Daniel M Ziegler, Nisan Stiennon, Jef- frey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning lan- guage models from human preferences. arXiv preprint arXiv:1909.08593,
2019 arXiv
-
[2020]
Rlaif: Scaling reinforcement learning from human feedback with ai feedback
[Lee et al., 2023] Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267,
2023 arXiv
-
[2021]
Training language models to follow instruc- tions with human feedback
[Ouyang et al., 2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instruc- tions with human feedback. Advances in neural informa- tion p...
2022
-
[2022]
Constitutional ai: Harmlessness from ai feedback
[Bai et al., 2022b] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073,
-
[2023]
Ultrafeedback: Boosting language models with high-quality feedback
[Cui et al., 2023] Ganqu Cui, Lifan Yuan, Ning Ding, Guan- ming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv e-prints, pages arXiv–2310,
2023
-
[2024]
aligner/aligner-7b-v1.0 · Hug- ging Face — huggingface.co
[Aligners, 2024] Aligners. aligner/aligner-7b-v1.0 · Hug- ging Face — huggingface.co. https://huggingface.co/ aligner/aligner-7b-v1.0,
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.