Pith. sign in

REVIEW 6 major objections 7 minor 1 cited by

Activation Reward Models for Few-Shot Model Alignment

T0 review · 6 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Activation Reward Models claim that steering a handful of attention-head activations turns any open LLM or LMM into a few-shot reward model that outperforms prompting-based judges and beats GPT-4o on most reward-hacking splits.

desk verdict A useful extension of SAV-style steering to few-shot reward modeling with a worthwhile new benchmark, but the core mechanism is underspecified and the reported gains can't yet be attributed to criterion-specific preference encoding. read the letter →

arxiv 2507.01368 v1 pith:PGFZW4RP submitted 2025-07-02 cs.CV cs.LG

Tianning Chai , Chancharik Mitra , Brandon Huang , Gautam Rajendrakumar Gare , Zhiqiu Lin , Assaf Arbelle , Leonid Karlinsky , Rogerio Feris
show 3 more authors
Trevor Darrell Deva Ramanan Roei Herzig
This is my paper
classification cs.CVcs.LG
keywords activationsteeringfew-shotrewardmodelinghackingpreferencealignmentattentionheadstokenprobabilityscoringmultimodalmodelsHack
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a reward model can be built on the fly from a small number of labeled preference examples, without training a separate model, by steering the internal activations of an off-the-shelf LLM or LMM. The method extracts mean attention-head activations from few-shot examples, selects a sparse set of heads, and scores a response by the probability that the steered model says "Yes" to a binary criteria question. The authors claim this beats existing few-shot judges, voting-based scoring, token-probability scoring, and Sparse Attention Vectors on standard benchmarks, and surpasses GPT-4o on most splits of their new PreferenceHack benchmark. If right, it offers a lightweight, rapid route to adapting alignment signals to new preferences and newly discovered biases.

What carries the argument

The load-bearing mechanism is activation steering at the level of attention heads, paired with a token-probability readout. Activation extraction computes the mean activation $\mu_{l,j} = \frac{1}{n}\sum_i E[z_{l,j} \mid p_i, r_i, y_i]$ for each layer-head location; head selection uses a Bernoulli distribution over head locations optimized by REINFORCE to maximize held-out preference accuracy, producing the sparse set $\lambda^{\mathrm{ARM}}_j$; reward scoring injects the mean activations at those heads and reads out the probability of the "Yes" token. The paper edits attention-head outputs before the projection layer and makes no weight updates.

What would settle it

Reshuffle the labels of the few-shot examples before computing the mean activations, keeping the same prompts and responses; if the resulting Activation RM still scores near its reported accuracy on PreferenceHack, the signal is not the preference criterion. A complementary experiment: run the method with head sets chosen randomly instead of by REINFORCE; if random heads perform about as well, head selection is not doing the work the paper assigns it.

Watch

Extended reading notes

Core claim

The paper's central claim is that a preference criterion can be encoded directly in the activations of a frozen LLM or LMM. Given a few labeled examples of what counts as a good or bad response, it computes the mean activation of every attention head at the last input token during correct evaluations, selects a sparse set of attention heads with REINFORCE, and then scores any new response $r$ to prompt $p$ as $s(r|p) = P_F(\text{``Yes''} \mid \text{``Does this response meet the specified criteria?''}, \lambda_{\mathrm{ARM}}, \mu_{\mathrm{ARM}})$, where $\mu_{\mathrm{ARM}}$ is the mean activation and $\lambda_{\mathrm{ARM}}$ the selected head set. The paper claims this Activation RM outperforms zero- and few-shot LLM-as-a-judge prompting, voting-based scoring, token probability scoring, and Sparse Attention Vectors on RewardBench and MultimodalRewardBench, and surpasses GPT-4o on most splits of its new PreferenceHack benchmark. It also claims the method is sample-efficient, matching LoRA finetuning without weight updates, and that well-specified tasks such as safety benefit the most.

Load-bearing premise

The whole comparison depends on the premise that adding the mean activation vectors at the selected attention heads, before the projection layer, changes the model's "Yes" probability in a way that tracks the intended preference criterion on held-out prompts; the paper does not specify the exact injection rule, its scaling, or an ablation isolating the edit's effect from other ways the steering could shift the output.

Editorial extensions

If this is right

  • Standard reward modeling benchmarks can be tackled without training a separate reward model: Activation RMs beat zero- and few-shot judge prompting, voting, token-probability scoring, and Sparse Attention Vectors on RewardBench and MultimodalRewardBench for both tested backbones.
  • Reward-hacking defenses become cheap to update: on the new PreferenceHack benchmark, Activation RMs outperform GPT-4o on most bias splits, including the length-bias splits where judge baselines are near chance.
  • Few-shot steering is sample efficient: with roughly 12 to 130 labeled examples it matches LoRA finetuning on 130 examples, and its scores improve as more examples are added.
  • The method's advantage concentrates on well-specified criteria such as safety and specific biases, suggesting few-shot reward modeling is most reliable when the preference is narrow and clearly defined.
  • The same steered model can serve as a general evaluator, a best-of-N ranker, or a source of scalar rewards for RL-based preference optimization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Implicitly, the selected attention-head locations and mean offsets could serve as an inspectable record of what criterion the model was steered toward, giving the reward signal an interpretability that a prompted judge lacks.
  • An immediate extension is online re-steering: when a new reward hack is discovered, a few newly labeled pairs could update the steering vectors without retraining, which is the practical use case PreferenceHack is designed to simulate.
  • A testable extension is to paraphrase or translate the criteria question; if scores shift materially under paraphrases, the reward depends on the surface wording of the question rather than the preference criterion alone.
  • Because the method requires white-box access to attention heads, its reach is limited to open models; whether equivalent steering can be recovered from black-box APIs is a question this paper leaves open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 7 minor

Summary. The paper proposes Activation Reward Models (Activation RMs), a few-shot reward modeling approach that uses activation steering on selected attention heads of an LLM/LMM to encode a preference criterion, then scores a candidate response by the probability of the model generating "Yes" to a criteria query. The method is evaluated on RewardBench and Multimodal RewardBench against zero/few-shot baselines including LLM-as-a-judge, generative scoring, voting, and Sparse Attention Vectors, and on a new PreferenceHack benchmark designed to test reward hacking with paired preferences. The paper reports that Activation RMs outperform the included baselines on most benchmarks and exceed GPT-4o on most PreferenceHack splits.

Significance. If the mechanism is as claimed, Activation RMs would be a lightweight and rapidly adaptable alternative to trained reward models, and PreferenceHack would be a useful testbed for reward-hacking robustness. The core idea of pairing activation steering with generative verification is interesting and the reported results are suggestive. However, the current manuscript leaves the central mechanism underspecified: the extracted steering vector as defined is label-agnostic, the injection operation and scaling are not described, and the REINFORCE head-selection procedure lacks key details. The empirical evidence also lacks error bars, and some ablations contradict the text. The contribution is potentially valuable but requires substantial clarification and additional evidence.

major comments (6)
  1. [Section 3.3, Eq. (1)] The mean activation μ_{l,j} is a label-agnostic average over all few-shot examples and does not contrast preferred and rejected responses, so it cannot encode a preference direction as written. Please specify whether the actual implementation uses a contrastive difference (e.g., mean(chosen) − mean(rejected)) or per-class means; if the plain mean is used, explain how injecting it yields criterion-specific reward scores rather than a uniform prior over "Yes". This is load-bearing for the claimed preference-encoding mechanism.
  2. [Section 5.1, first paragraph] The activation injection rule is unspecified: the text states that activations are "directly edit[ed] ... before the projection layer" with no statement of the operation (add, subtract, replace), any scaling coefficient, or an ablation of edit strength. An unscaled additive shift of a mean vector could increase P("Yes") for all responses, inflating RewardBench and PreferenceHack scores through a response-independent bias. Please provide the exact injection recipe and an experiment that varies edit strength.
  3. [Section 3.4] The REINFORCE-based head selection is described only as maximizing "the model's ability"; the reward signal, validation set size v, number of selected heads, and optimization hyperparameters (learning rate, temperature, baseline) are not reported. Since the selected head set λ is part of the reward score in Eq. (3), the method cannot be reimplemented from the paper. Please specify these details or release code.
  4. [Section 6.3, Table 3, and Conclusion] The claim that "performance scales with increasing number of examples" is not supported: the macro averages are 76.22 (12), 76.66 (20), 76.52 (40), 76.42 (80), and 77.17 (130), which is non-monotonic with differences near the likely noise level; Table 5 in the supplement shows the same non-monotonic pattern (Qwen 8-shot at 68.00 vs 12-shot at 76.56). Additionally, §6.3 states CoT has "little effect on our results" but Table 3 contains no CoT row for Activation RM, while the Conclusion states CoT "enhances results." These statements should be reconciled and the missing CoT ablation reported.
  5. [Tables 1-3] No error bars, seeds, or significance tests are reported, and the gaps to the SAV baseline are often small (e.g., 1-4 points on 920-example PreferenceHack splits). Please report standard deviations over at least 3 seeds and, where appropriate, significance tests, so the "consistently outperforms" claim can be evaluated.
  6. [Section 4.2 and Appendix C.2] PreferenceHack's few-shot training examples and test examples are generated with the same GPT-4o-mini bias-injection functions (e.g., generate_tedious_with_gpt4, hacky_list_rewrite, make_positive_spin), so the training set reveals the exact bias template. The benchmark therefore measures adaptation to a known injection pattern rather than robustness to novel reward-hacking strategies; please qualify the generalization claim and, if possible, add a held-out split with a different injection protocol.
minor comments (7)
  1. [Section 5.1] Grammatical error: "the activation steering procedure directly edit the output" should read "the activation steering procedure directly edits the output."
  2. [Section 4.1] Typo: "To the best of our best knowledge" should be "To the best of our knowledge."
  3. [Section 5.3] Section 5.3 states PreferenceHack has "seven splits" and mentions numerical and orientation biases, but Section 4.1 and Table 2 describe six splits (length, format, positivity in language and multimodal). Please correct this inconsistency.
  4. [Section 6.1] "Taskness" is introduced as an explanation for safety gains but is not defined or measured; if kept, please define it and provide evidence.
  5. [Conclusion] The Conclusion says "chain-of-thought reasoning enhances results," contradicting §6.3's statement that it has little effect; see major comment 4.
  6. [Abstract and Section 1] The phrase "no additional model finetuning" is slightly misleading because the REINFORCE head selection trains a selection policy; consider rephrasing to "no finetuning of the base model weights."
  7. [References] The reference list contains duplicates and incomplete entries (e.g., references [3] and [4] are the same work, and several entries lack arXiv identifiers); please clean up.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reward score is an explicit supervised function of few-shot activations and head selection, evaluated on held-out examples; self-citations are not load-bearing.

full rationale

I walked the derivation chain from the few-shot labeled examples through activation extraction (Section 3.3), attention head selection (Section 3.4), and reward score generation (Section 3.5). The reward score is defined, not derived, as the probability of the 'Yes' token after injecting the mean activations at the selected heads; both the mean activations and the selected heads are fit to labeled and validation examples and then evaluated on held-out prompts and responses. This is standard supervised few-shot learning: the prediction is a function of the training labels, but it is not equivalent to those labels by construction because the test inputs are new. The paper's own equations do not reduce any claimed prediction to its inputs; no fitted parameter is renamed as a prediction, and no uniqueness theorem or other load-bearing premise is imported from the authors' prior work. The authors' prior Sparse Attention Vectors work is used as a baseline and as related work, not as justification for the central mechanism. PreferenceHack is self-authored and its training and test splits share the same bias-injection distribution, which is a legitimate external-validity concern for the 'adapting to new attacks' framing, but it is not circularity in the derivation chain because the test labels are never derived from the fitted quantities. The Limitations section explicitly acknowledges the method's dependence on internal model access and task specificity, further confirming that the claims are empirical rather than definitional. Since no specific equation-to-equation reduction or fitted-input-called-prediction step could be exhibited, the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 6 assumptions · 1 invented entities

The central claims rest on several domain assumptions: that last-token attention-head activations encode a preference criterion, that a REINFORCE-selected sparse head subset generalizes, that additive activation editing is non-destructive, that 'Yes' token probability is a meaningful reward, and that the GPT-4o-mini-generated PreferenceHack biases are clean. The paper provides empirical outputs but not isolated evidence for these premises. The only invented construct is 'taskness', which is a post hoc label rather than a measured quantity.

free parameters (4)
  • Mean activation vectors mu_ARM_j = Per-head mean of n labeled-example activations
    Defined in Section 3.3 and used directly in the reward score formula in Section 3.5; these are the core fitted quantities.
  • Selected attention head set lambda_ARM_j = Sparse subset chosen by REINFORCE, exact heads not reported
    Chosen by optimizing a Bernoulli distribution over heads on v validation examples (Section 3.4); the selection changes per criterion and is not disclosed.
  • REINFORCE hyperparameters = 600 optimization steps; learning rate, entropy weight, head count not reported
    Section 5.1 reports 600 steps but omits the other hyperparameters that determine which heads are selected.
  • Number of few-shot examples per split = 80 or 130 depending on benchmark
    Set by convention in Section 5.1 and Supplementary Section C; performance varies with this number as shown in Table 5.
assumptions (6)
  • domain assumption Mean last-token attention-head activations from labeled preference examples encode the evaluation criterion.
    Section 3.3 defines mu_l,j as this mean and relies on it as the steering signal; no evidence is given that the representation is stable across prompts or criteria.
  • domain assumption A sparse subset of attention heads, selected by REINFORCE on a validation set, is sufficient to steer reward behavior.
    Section 3.4 and Figure 1 depend on this sparsity assumption; the validation set size and selection reward are not specified.
  • domain assumption Adding mean activations to selected attention-head outputs is a faithful and non-destructive intervention.
    Section 5.1 says activations are edited before the projection layer, but no ablation verifies that the edit preserves general generation quality.
  • domain assumption The probability of the 'Yes' token after the steered query is a valid scalar reward.
    Section 3.5 defines the reward score this way; no calibration against human scores or trained reward models is provided.
  • domain assumption GPT-4o-mini injections in PreferenceHack preserve factual errors and isolate the target bias.
    Section 4.2 and Supplementary C.2 rely on this; there is no human check or automatic verification that the biased responses introduce no additional confounds.
  • standard math REINFORCE gives an unbiased estimate of head importance for the validation reward.
    Section 3.4 cites Williams's REINFORCE; the estimator is standard, but the reward signal used for optimization is not defined in the paper.
invented entities (1)
  • taskness
    purpose: Post hoc explanation for why Activation RM shows larger gains on safety tasks than on broad domains like chat or math.
    Introduced in Section 6.1; no definition, measurement, or falsifiable handle is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Activation Reward Models for Few-Shot Model Alignment." pith.science (2026). https://pith.science/paper/PGFZW4RP

@misc{pith2026250701368,
  author       = {Pith},
  title        = {Pith review of: Activation Reward Models for Few-Shot Model Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PGFZW4RP}},
  note         = {Machine review of arXiv:2507.01368}
}
read the original abstract

Aligning Large Language Models (LLMs) and Large Multimodal Models (LMMs) to human preferences is a central challenge in improving the quality of the models' generative outputs for real-world applications. A common approach is to use reward modeling to encode preferences, enabling alignment via post-training using reinforcement learning. However, traditional reward modeling is not easily adaptable to new preferences because it requires a separate reward model, commonly trained on large preference datasets. To address this, we introduce Activation Reward Models (Activation RMs) -- a novel few-shot reward modeling method that leverages activation steering to construct well-aligned reward signals using minimal supervision and no additional model finetuning. Activation RMs outperform existing few-shot reward modeling approaches such as LLM-as-a-judge with in-context learning, voting-based scoring, and token probability scoring on standard reward modeling benchmarks. Furthermore, we demonstrate the effectiveness of Activation RMs in mitigating reward hacking behaviors, highlighting their utility for safety-critical applications. Toward this end, we propose PreferenceHack, a novel few-shot setting benchmark, the first to test reward models on reward hacking in a paired preference format. Finally, we show that Activation RM achieves state-of-the-art performance on this benchmark, surpassing even GPT-4o.

Figures

Figures reproduced from arXiv: 2507.01368 by the authors.

Figure 1
Figure 1. Activation Reward Models. The Activation RMs pipeline has two high-level steps. First, few-shot examples are used to steer specific attention heads within the model. Second, using this edited model, downstream inference for reward modeling is done via token probability scoring. effective on general tasks, they can underperform on specific applications such as protecting against prompting attacks and hijacking strate… view at source ↗
Figure 2
Figure 2. PreferenceHack Examples. We show samples based on two images of our PreferenceHack benchmark. Each sample would consist of a ground truth response paired with a biased incorrect response. The reward model is tasked with preferring the correct description over the biased one. preserving their factual errors. This approach creates a test bed for assessing multimodal reward hacking vulnerabilities. 4.3 Evaluation Proto… view at source ↗
Figure 3
Figure 3. Additional PreferenceHack Examples We provide more examples from our Preference￾Hack dataset. 11 [PITH_FULL_IMAGE:figures/full_fig_p026_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multimodal Reward Hacking in Reinforcement Learning

    cs.AI 2026-07 conditional novelty 6.5 of 10

    Imperfect multimodal RL rewards systematically create new failures (NRFR > RHR); scaling and answer-aware rewards help but do not eliminate hacking, and unreliable visual verifiers actively increase it.

Reference graph

Works this paper leans on

75 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [1]

    The claude 3 model family: Opus, sonnet, haiku

  2. [2]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. A...

    arXiv 2025

  3. [4]

    Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, ...

    arXiv 2022

  4. [6]

    Constitutional AI: Harmlessness from AI feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, and Others. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073, 2022

    arXiv 2022

  5. [7]

    Capturing individual human preferences with reward features

    André Barreto, Vincent Dumoulin, Yiran Mao, Nicolás Pérez-Nieves, Bobak Shahriari, Yann Dauphin, Doina Precup, and Hugo Larochelle. Capturing individual human preferences with reward features. arXiv preprint arXiv:2503.17338, 2025

    arXiv 2025

  6. [8]

    Network dissection: Quantify- ing interpretability of deep visual representations

    David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantify- ing interpretability of deep visual representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6541–6549, 2017

    work page 2017

  7. [9]

    Understanding the role of individual units in a deep neural network

    David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Understanding the role of individual units in a deep neural network. In Proceedings of the National Academy of Sciences, pages 30071–30077, 2020

    work page 2020

  8. [10]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    arXiv 2005

Show all 75 references
  1. [11]

    RRHF-V: Ranking responses to mitigate hallucinations in multimodal large language models with human feedback

    Guoqing Chen, Fu Zhang, Jinghao Lin, Chenglong Lu, and Jingwei Cheng. RRHF-V: Ranking responses to mitigate hallucinations in multimodal large language models with human feedback. In Proceedings of the 31st International Conference on Computational Linguistics, pages 6798–6815...

  2. [12]

    Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik, Sam Bowman, Jan Leike, Jared Kaplan, and Ethan Perez

    Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson E. Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik, Sam Bowman, Jan Leike, Jared Kaplan, and Ethan Perez. Reasoning models don’t always say what they think. 2025

    2025

  3. [13]

    Christiano, Jan Leike, Tom B

    Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017

    2017

  4. [14]

    Bowman, Ethan Perez, and Evan Hubinger

    Carson Denison, Monte MacDiarmid, Fazl Barez, David Duvenaud, Shauna Kravec, Samuel Marks, Nicholas Schiefer, Ryan Soklaski, Alex Tamkin, Jared Kaplan, Samuel R. Bowman, Ethan Perez, and Evan Hubinger. Sycophancy to subterfuge: Investigating reward-tampering in large language ...

    2024 arXiv

  5. [15]

    Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant

    Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D’Amour, Dj Dvijotham, Adam Fisch, Katherine Heller, Stephen R. Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant. Helping or herding? reward model ensembles mitigate but do not eliminate reward hack...

    2023 arXiv

  6. [16]

    Paint by word

    Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. Paint by word. In arXiv preprint arXiv:2103.10951, 2023

    2023 arXiv

  7. [17]

    A survey on llm-as-a-judge

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594, 2024

    2024 arXiv

  8. [18]

    Sahil Gureja, Zifan Xu, Aditi Chaudhary, Yuxuan Yao, Ajay Saini, Sabyasachi Ghosh, Gaurav Sahu, Preksha Nema, Barnabás Póczos, and Zachary C. Lipton. M-rewardbench: Evaluating reward models in multilingual settings. arXiv preprint arXiv:2410.15522, 2024

    arXiv

  9. [19]

    In-context learning creates task vectors

    Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9318–9333, Singapore, 2023. Association for Computational Linguistics

    2023

  10. [20]

    In-context learning creates task vectors

    Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. arXiv preprint arXiv:2310.15916, 2023

    2023 arXiv

  11. [21]

    Li, and Jacob Andreas

    Evan Hernandez, Belinda Z. Li, and Jacob Andreas. Inspecting and editing knowledge representations in language models. arXiv preprint arXiv:2304.00740, 2023

    2023 arXiv

  12. [22]

    Finding visual task vectors

    Alberto Hojel, Yutong Bai, Trevor Darrell, Amir Globerson, and Amir Bar. Finding visual task vectors. In Proceedings of the European Conference on Computer Vision (ECCV), 2024

    2024

  13. [23]

    Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality

    Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. ArXiv, abs/2306.14610, 2023

    2023 arXiv

  14. [24]

    Multimodal task vectors enable many-shot multimodal in-context learning

    Brandon Huang, Chancharik Mitra, Assaf Arbelle, Leonid Karlinsky, Trevor Darrell, and Roei Herzig. Multimodal task vectors enable many-shot multimodal in-context learning. In Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  15. [25]

    Multimodal task vectors enable many-shot multimodal in-context learning

    Brandon Huang, Chancharik Mitra, Assaf Arbelle, Leonid Karlinsky, Trevor Darrell, and Roei Herzig. Multimodal task vectors enable many-shot multimodal in-context learning. In Advances in Neural Information Processing Systems, pages 22124–22153, 2024

    2024

  16. [26]

    Rag-rewardbench: Benchmarking reward models in retrieval augmented generation for preference alignment

    Zhiyang Jin, Prakhar Gupta, Chun Kai Ling, Fuli Luo, Congzheng Song, Yuxi Yang, Xiang Ren, and Yuandong Tian. Rag-rewardbench: Benchmarking reward models in retrieval augmented generation for preference alignment. arXiv preprint arXiv:2412.13746, 2024

    2024 arXiv

  17. [27]

    Few-shot steerable alignment: Adapting rewards and llm policies with neural processes

    Katarzyna Kobalczyk, Claudio Fanconi, Hao Sun, and Mihaela van der Schaar. Few-shot steerable alignment: Adapting rewards and llm policies with neural processes. arXiv preprint arXiv:2412.13998, 2024

    2024 arXiv

  18. [28]

    Smith, and Hannaneh Hajishirzi

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, Liane Lovitt, Bill Yuchen Lin, Khyathi Raghavi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:...

    2024 arXiv

  19. [29]

    RLAIF vs

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor C˘arbune, Abhinav Rastogi, and Sushant Prakash. RLAIF vs. RLHF: Scaling reinforcement learning from human feedback with ai feedback. In International Confer...

    2024

  20. [30]

    RLAIF vs

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Ren Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. RLAIF vs. RLHF: Scaling reinforcement learning from human feedback with AI feedback. In Proceedings of th...

    2024

  21. [31]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3045–3059, Online and Punta Cana, Dominican Republic, 2021. Associa...

    2021

  22. [32]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. ArXiv, abs/2408.03326, 2024

    2024 arXiv

  23. [33]

    BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023

    2023

  24. [34]

    Evaluating text-to-visual generation with image-to-text generation

    Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. In Computer Vision – ECCV 2024, pages 366–384. Springer, 2024

    2024

  25. [35]

    Inform: Mitigating reward hacking in rlhf via information-theoretic reward modeling

    Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, and Dacheng Tao. Inform: Mitigating reward hacking in rlhf via information-theoretic reward modeling. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  26. [36]

    Sparse attention vectors: Generative multimodal model features are discriminative vision-language classifiers

    Chancharik Mitra, Brandon Huang, Tianning Chai, Zhiqiu Lin, Assaf Arbelle, Rogerio Feris, Leonid Kar- linsky, Trevor Darrell, Deva Ramanan, and Roei Herzig. Sparse attention vectors: Generative multimodal model features are discriminative vision-language classifiers. arXiv pre...

    2024 arXiv

  27. [37]

    Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng

    Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian D. Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. Rule based rewards for language model safety. ArXiv, abs/2411.01111, 2024

    2024 arXiv

  28. [38]

    In-context learning and induction heads

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022

    2022 arXiv

  29. [39]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023

    2023 arXiv

  30. [40]

    OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M ˛ adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex ...

    2024

  31. [41]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, and Others. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems (NeurIPS), 35, 2022

    2022

  32. [42]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, ...

    2022

  33. [43]

    Steering llama 2 via contrastive activation addition

    Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681, 2023

    2023 arXiv

  34. [44]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019

    2019

  35. [45]

    Red teaming language models with language models

    Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.03286, 2022

    2022 arXiv

  36. [46]

    Improving language understanding by generative pre-training

    Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training. 2018

    2018

  37. [47]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023

    2023 arXiv

  38. [48]

    GPT-4 jailbreaks itself with near-perfect success using self- explanation

    Govind Ramesh, Yao Dou, and Wei Xu. GPT-4 jailbreaks itself with near-perfect success using self- explanation. arXiv preprint arXiv:2405.13077, 2024

    2024 arXiv

  39. [49]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 13

    2017 arXiv

  40. [50]

    Finding alignments between interpretable causal variables and distributed neural representations

    Sarah Schwettmann. Finding alignments between interpretable causal variables and distributed neural representations. In arXiv preprint arXiv:2303.02536, 2023

    2023 arXiv

  41. [51]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  42. [52]

    FSPO: Few-shot preference optimization of synthetic preference data in LLMs elicits effective personalization to real users

    Anikait Singh, Sheryl Hsu, Kyle Hsu, Eric Mitchell, Stefano Ermon, Tatsunori Hashimoto, Archit Sharma, and Chelsea Finn. FSPO: Few-shot preference optimization of synthetic preference data in LLMs elicits effective personalization to real users. arXiv preprint arXiv:2502.19312, 2025

    2025 arXiv

  43. [53]

    Alpaca: A strong, replicable instruction- following model

    Stanford Center for Research on Foundation Models (CRFM). Alpaca: A strong, replicable instruction- following model. https://crfm.stanford.edu/2023/03/13/alpaca.html, 2023. Accessed: 2025- 05-20

    2023

  44. [54]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008–3021, 2020

    2020

  45. [55]

    Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, Paul Christiano, Jan Leike, and Others

    Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, Paul Christiano, Jan Leike, and Others. Learning to summarize from human feedback. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020

    2020

  46. [56]

    Extracting latent steering vectors from pretrained language models

    Nishant Subramani, Nivedita Suresh, and Matthew Peters. Extracting latent steering vectors from pretrained language models. In Findings of the Association for Computational Linguistics: ACL 2022, pages 566–581, Dublin, Ireland, 2022. Association for Computational Linguistics

    2022

  47. [57]

    Granite vision: a lightweight, open-source multimodal model for enterprise intelligence

    Granite Vision Team, Leonid Karlinsky, Assaf Arbelle, Abraham Daniels, Ahmed Nassar, Amit Alfassi, Bo Wu, Eli Schwartz, Dhiraj Joshi, Jovana Kondic, et al. Granite vision: a lightweight, open-source multimodal model for enterprise intelligence. arXiv preprint arXiv:2502.09927, 2025

    2025 arXiv

  48. [58]

    Li, Arnab Sen Sharma, Aaron Mueller, Byron C

    Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function vectors in large language models. In Proceedings of the International Conference on Learning Representations (ICLR), 2024

    2024

  49. [59]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language...

    2023 arXiv

  50. [60]

    Vazquez, Ulisse Mini, and Monte MacDiarmid

    Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248, 2024

    2024 arXiv

  51. [61]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  52. [62]

    Large language models are not fair evaluators

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 5605–5620, 2024

    2024

  53. [63]

    Williams

    Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229–256, 2004

    2004

  54. [64]

    rewordbench: Benchmarking and improving the robustness of reward models with transformed inputs

    Zhaofeng Wu, Michihiro Yasunaga, Andrew Cohen, Yoon Kim, Asli Celikyilmaz, and Marjan Ghazvinine- jad. rewordbench: Benchmarking and improving the robustness of reward models with transformed inputs. arXiv preprint arXiv:2503.11751, 2025

    2025

  55. [65]

    Multimodal rewardbench: Holistic evaluation of reward models for vision-language models

    Michihiro Yasunaga, Luke Zettlemoyer, and Marjan Ghazvininejad. Multimodal rewardbench: Holistic evaluation of reward models for vision-language models. arXiv preprint arXiv:2502.14191, 2025

    2025 arXiv

  56. [66]

    Zettlemoyer, and Marjan Ghazvininejad

    Michihiro Yasunaga, Luke S. Zettlemoyer, and Marjan Ghazvininejad. Multimodal rewardbench: Holistic evaluation of reward models for vision language models. ArXiv, 2025

    2025

  57. [67]

    Which attention heads matter for in-context learning? In Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025

    Kayo Yin and Jacob Steinhardt. Which attention heads matter for in-context learning? In Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025. 14

    2025

  58. [68]

    ICPL: Few-shot in-context preference learning via LLMs

    Chao Yu, Qixin Tan, Hong Lu, Jiaxuan Gao, Xinting Yang, Yu Wang, Yi Wu, and Eugene Vinitsky. ICPL: Few-shot in-context preference learning via LLMs. arXiv preprint arXiv:2410.17233, 2024

    2024 arXiv

  59. [69]

    Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Co...

    2024

  60. [70]

    Rrhf: Rank responses to align language models with human feedback without tears

    Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302, 2023

    2023 arXiv

  61. [71]

    Rag-reward: Optimizing rag with reward modeling and rlhf

    Hanning Zhang, Juntong Song, Juno Zhu, Yuanhao Wu, Tong Zhang, and Cheng Niu. Rag-reward: Optimizing rag with reward modeling and rlhf. arXiv preprint arXiv:2501.13264, 2025

    2025

  62. [72]

    Generative verifiers: Reward modeling as next-token prediction

    Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. ArXiv, 2024

    2024

  63. [73]

    Generative verifiers: Reward modeling as next-token prediction

    Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. In Proceedings of the International Conference on Learning Representations (ICLR), 2025

    2025

  64. [74]

    Mm-rlhf: The next step forward in multimodal llm alignment

    Yi-Fan Zhang, Tao Yu, Haochen Tian, Chaoyou Fu, Peiyan Li, Jianshu Zeng, Wulin Xie, Yang Shi, Huanyu Zhang, Junkang Wu, et al. Mm-rlhf: The next step forward in multimodal llm alignment. arXiv preprint arXiv:2502.10391, 2025

    2025 arXiv

  65. [75]

    Interpreting deep visual representations via network dissection

    Bolei Zhou, David Bau, Aude Oliva, and Antonio Torralba. Interpreting deep visual representations via network dissection. In IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 2131–2145, 2018

    2018

  66. [76]

    Activation Reward Models

    Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. 15 Supplementary Material for “Activation Reward Models” He...

    1909 arXiv

  67. [2025]

    Association for Computational Linguistics. 10

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.