Pith. sign in

REVIEW 3 major objections 5 minor 62 references

Icon$^{2}$: Aligning Large Language Models Using Self-Synthetic Preference Data via Inherent Regulation

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that human preference criteria are linearly encoded in a language model's hidden states, so a single steering direction per criterion lets the model generate its own chosen and rejected response pairs, and training on…

desk verdict Genuinely novel method for self-synthetic preference data via representation steering, with solid ablations and a real efficiency win, but the core premise that PCA directions encode human preference is only indirectly validated and the reporting lacks error bars and code release. read the letter →

arxiv 2509.05605 v1 pith:HJUHWMM5 submitted 2025-09-06 cs.CL cs.AI

Qiyuan Chen , Hongsen Huang , Qian Shao , Jiahe Chen , Jintai Chen , Hongxia Xu , Renjie Hua , Ren Chuan
show 1 more author
Jian Wu
This is my paper · ORCID
classification cs.CLcs.AI
keywords preferencedataconstructionrepresentationengineeringinherentcontrolself-syntheticcontrastivesystempromptsinstructionfilteringdirectoptimizationalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes ICON$^{2}$, a framework that builds preference datasets without human annotation, external judges, or multiple sampled responses. It extracts layer-wise direction vectors from contrastive system prompts representing criteria like helpfulness, honesty, and harmlessness, then uses these vectors both to filter self-synthesized instructions and to steer token representations during decoding, directly producing preferred and dispreferred response pairs. On AlpacaEval 2.0, Llama3-8B reaches a length-controlled win rate of 23.22% (up from 5.59% after SFT) and Qwen2-7B reaches 20.00% (up from 9.95%), with Arena-Hard gains of 13.7 and 13.2 points. The claimed efficiency payoff is up to 48.1% lower compute cost than Self-Rewarding, because only two generation passes per instruction are needed instead of sampling and ranking many candidates.

What carries the argument

The central mechanism is the layer-wise preference direction vector $u^l_c$, defined as the first principal component of the contrastive representations $h^{l,+}_{i,c} - h^{l,-}_{i,c}$ obtained from positive and negative system prompts for criterion $c$. This vector carries the argument in two places: it computes an inherent-consistency score $\text{consistency}_{i,c} = \text{meanpool}(\{h^l_i^\top u^l_c\}_{l=1}^N)$ that filters self-synthesized instructions, and it steers token representations during decoding through $\hat{z}^l_k = z^l_k + \gamma_{c^*} u^l_{c^*}$, with positive $\gamma$ producing the chosen response and negative $\gamma$ producing the rejected response. The paper names the overall method ICON$^{2}$ and shows that middle layers (10--20) give the best steering results.

What would settle it

Take a model held out from the paper's procedure, extract the helpfulness direction from 1024 unrelated instructions using the paper's contrastive methodology, and generate 200 response pairs by positive and negative steering. If human annotators do not prefer the positive-steered responses over the negative-steered ones at a rate reliably above chance (say, at least 80%, comparable to the paper's implied separation), then the extracted direction is not encoding a human preference. A second, sharper test: fine-tune a model to be deliberately unhelpful; if the extracted 'helpfulness' direction from that model does not flip or disappear, the vector is capturing a superficial prompt artifact rather than the intended preference.

Watch

Extended reading notes

Core claim

The central claim is that sophisticated human preferences can be captured as linear directions in the representation space of an SFT model, and that adding or subtracting these directions during decoding creates response pairs with clear alignment differences. Formally, for each criterion $c$, the paper computes the contrastive vector $v^l_{i,c} = h^{l,+}_{i,c} - h^{l,-}_{i,c}$ between hidden states under positive and negative system prompts, aggregates over a small feature set ($D_{\text{feat}}$ of 1024 Alpaca instructions), and takes the first principal component $u^l_c$ per layer as the direction vector. These directions then serve two purposes: instruction filtering by mean-pooled dot-product consistency (max over criteria), and token-level steering via $\hat{z}^l_k = z^l_k + \gamma_{c^*} u^l_{c^*}$ during decoding at layers 10--20. The paper reports that training with direct preference optimization (DPO) on the resulting pairs substantially beats sampling-based baselines on length-controlled win rate and raw win rate on AlpacaEval 2.0 and Arena-Hard, and improves MT-Bench scores, including second-turn gains above 1.05 points.

Load-bearing premise

The load-bearing premise is that the first principal component of the difference between hidden states under 'helpful', 'honest', and 'harmless' prompts and their negations actually points along a direction that human judges would consider better-to-worse; if that direction is not aligned with human preferences, ICON2 merely makes the model more consistent with its own internal biases.

Editorial extensions

If this is right

  • Preference dataset construction becomes a two-pass generation problem: one positive-steered pass and one negative-steered pass per instruction, eliminating the need to sample, score, and rank five or more responses.
  • The reported cost reduction (up to 48.1% versus Self-Rewarding on Llama3-8B, with total cost \$149.1 versus \$287.2) implies that large-scale preference data can be synthesized on commodity hardware within hours rather than days.
  • Instruction synthesis from pre-query templates without seed instructions, combined with inherent-consistency filtering, yields tailored instructions that outperform manual collections and prior synthetic sets when evaluated under the same DPO pipeline.
  • Hyperparameters for steering (layer range and $\gamma$ values) can be selected from 100 reward-scored responses in under one GPU hour, without running DPO, making the approach easy to port to new base models.
  • MT-Bench second-turn improvements of more than 1.05 points suggest that even single-turn preference pairs improve multi-turn dialogue quality, likely by making the model's early responses more aligned.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the method's premise would be cross-model transfer: extract direction vectors from one model family (say, a Qwen model) and steer a different family (say, Llama) to see whether the direction encodes a general human preference or is idiosyncratic to the model that produced it; the paper only measures same-model steering.
  • Because the steering strength $\gamma$ is a continuous parameter, the method can generate graded preference pairs with controlled margins, which could be useful for preference-optimization variants that need fine-grained reward differences rather than binary chosen/rejected labels.
  • The paper's own Limitations section notes that online DPO settings and multi-turn dialogue generalization are unverified; whether the extracted direction vectors remain stable when the model weights change during online training is an open empirical question.
  • The consistency filter selects each instruction based on its maximum alignment with a single criterion; a natural extension would be to explicitly balance the filtered set across honesty, harmlessness, helpfulness, and general quality to avoid over-specialization in one direction.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ICON2, a framework that constructs preference data for DPO without external annotations or multi-response sampling. It extracts layer-wise direction vectors as the first PCA component of contrastive hidden-state differences between positive and negative system prompts for criteria C = {honesty, harmlessness, helpfulness, general}; filters self-synthesized instructions by a mean-pooled dot-product consistency score; and generates chosen/rejected response pairs by adding/subtracting the direction vector during decoding. The resulting pairs are used for offline DPO. Experiments on Llama3-8B and Qwen2-7B report improvements over SFT and several baselines on AlpacaEval 2.0, Arena-Hard, and MT-Bench, with reduced GPU cost. Appendices provide pseudocode, leakage analysis, sensitivity analysis, a direct-prompting comparison, and a no-DPO hyperparameter selection scheme.

Significance. If the extracted directions genuinely encode human preferences, the method is a novel and efficient route to tailored preference data: it avoids multi-response sampling, is model-specific, and ships detailed pseudocode, a leakage analysis (Appendix D), a stability analysis (Appendix G), an explicit comparison with direct prompting (Appendix I), and a reward-based hyperparameter selection procedure (Appendix J). These are real strengths. However, the paper's central scientific claim—that the PCA directions represent human preferences rather than stylistic correlates—is not yet supported by any human or independent validation. Combined with single-run headline numbers, the current evidence is suggestive rather than conclusive. The contribution would be significantly strengthened by a human pairwise-preference study on generated pairs and multi-seed evaluation.

major comments (3)
  1. [Section 3.1, Eq. (1); Appendix G] The load-bearing premise that the first principal component of contrastive hidden-state differences encodes human preference is not validated. Appendix G shows only that the vectors are stable across D_feat size and source (Table 9) and that downstream results are stable (Table 11); it does not show semantic alignment with human judgment. Because Eq. (4) uses these directions to label chosen and rejected responses, all DPO gains could in principle reflect steering of stylistic properties (length, politeness, hedging) rather than genuine alignment. The authors' own Ethical Considerations acknowledge this risk. I request a human-annotation study on a sample of generated pairs, or an independent pre-registered preference probe, reporting agreement rates, together with a comparison of the extracted directions against established behavioral preference directions.
  2. [Tables 1 and 2; Section 4.2] All headline results are single runs without error bars, confidence intervals, or significance tests. For example, the 23.22 versus 18.38 LC gap between ICON2(General+3H) and Self-Refine in Table 1 is presented as a strong improvement, but no variance or paired comparison is reported. AlpacaEval 2.0 and Arena-Hard win rates are stochastic, and MT-Bench scores are known to be noisy; I request at least three seeds or bootstrap confidence intervals over judge samples to support the comparative claims. Without this, the reported gains cannot be distinguished from run-to-run noise.
  3. [Section 3.3, Eq. (4); Appendix I] The preference labels are produced in a self-referential loop: the same SFT model provides the contrastive directions, the steered chosen and rejected responses, and the DPO training signal, with no external human or independent reward-model verification. While self-rewarding pipelines also use model-generated labels, they at least rely on an LLM judge scoring responses; here the label is an unvalidated geometric direction. Table 12 shows that IC pairs do not saturate reward accuracy, but that only indicates the pairs are harder to classify, not that the labels reflect human preferences. I request an external evaluation of the generated pairs—for example, a human pairwise-preference sample or a held-out reward model—to show that the chosen/rejected distinction corresponds to actual preference rather than to the model's internal bias.
minor comments (5)
  1. [Abstract; Table 1] The abstract states an average win-rate improvement of 13.89% on AlpacaEval 2.0, but Table 1 gives (17.63 + 10.05) / 2 = 13.84 for length-controlled win rates, and the abstract does not specify whether the number refers to LC or raw WR. Please align the text with the table and define the metric explicitly.
  2. [Section 3.2, Eq. (2)] The consistency score is an unnormalized mean-pooled dot product between hidden states and direction vectors. Although the max over criteria is unaffected by the shared hidden-state norm, the absolute score used for thresholding and ranking can be dominated by instruction length or layer-wise norm differences; please normalize the vectors or justify that the filtering is invariant to these scale effects.
  3. [Table 3] The experiments in Table 3 use 20k instructions for each instruction-construction method, while the main experiments use 100k filtered instructions. Please clarify whether the 20k subset is drawn from the same 1M raw pool and whether the same consistency threshold is applied, so the comparison is apples-to-apples.
  4. [Table 1; Section 4.1] The row labeled 'Manual Collection' uses UltraFeedback, whose preference annotations come from GPT-4 rather than human annotators; the label is misleading and should be changed to something like 'UltraFeedback (GPT-4 labels)' to avoid implying human annotation.
  5. [Section J; Table 14] The no-DPO hyperparameter selection procedure is described for γ values but the paper does not state whether the same γ is used for all four criteria in the main experiments; since Eq. (4) defines γ_{c*}, please clarify whether per-criterion coefficients were tuned or a single global coefficient was used.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ICON2's preference pairs are self-synthetic, but the central empirical claims are validated against external benchmarks, and the unvalidated direction-premise is a correctness/validity limitation, not a circular derivation.

full rationale

ICON2's pipeline is self-referential in the sense that the direction vectors of Eq. (1) are extracted from the SFT model's own hidden states and then used in Eq. (4) to generate chosen/rejected pairs by adding or subtracting gamma * u. That is a data-generation procedure, not a reduction of a predicted result to a fitted input: the paper never defines human preference as the model's own direction, nor does it claim to derive benchmark scores from the vectors. The preference-direction premise is assumed rather than human-validated, and the paper itself flags this in the Ethical Considerations section, stating that 'steering responses via unsupervised representation vectors could inadvertently prioritize harmful but superficially plausible outputs without explicit safety filtering' and acknowledging 'the lack of human oversight in instruction synthesis.' Those are limitations and correctness risks, not circularity. The empirical evaluation is externally anchored: AlpacaEval 2.0, Arena-Hard, and MT-Bench use GPT-4 as an independent judge, and the paper compares against multiple baselines (Manual Collection, Sampling-Ranking, Self-Rewarding, Self-Refine). If the extracted direction did not track useful response quality, the benchmark comparisons would not automatically favor ICON2, so the central claim is falsifiable and not equivalent to the method's own construction. There is no load-bearing self-citation chain, no imported uniqueness theorem, and no fitted parameter renamed as a prediction: the hyperparameters are tuned on a reward model and then evaluated externally, which is standard practice. The sensitivity analysis in Appendix G confirms stability of the vectors across different feature datasets but does not establish semantic preference content; that missing support weakens the paper's framing but does not make the derivation circular. Overall, the method is self-synthetic but externally benchmarked, so no step reduces to its own input by construction.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the linear representation hypothesis and on the assumption that the model's own contrastive directions are valid proxies for human preferences. The method introduces several manually set hyperparameters (gamma, layer range, criteria set, retention threshold) that are tuned using a reward model and not derived from first principles.

free parameters (7)
  • gamma_positive (gamma+) = 0.1
    Steering strength for chosen responses; selected via ArmoRM reward scores on 20k synthesized instructions (Table 14, Appendix J.1).
  • gamma_negative (gamma-) = -0.05
    Steering strength for rejected responses; selected via reward score and the proportion of negative rewards below positive (Tables 14-15, Appendix J.2).
  • control_layer_interval = [10, 20] for 8B models
    Layer range for token steering; selected based on hyperparameter analysis in Section 4.6.1 where middle layers were best.
  • criteria_set C = {honesty, harmlessness, helpfulness, general}
    Manual decomposition of human preferences; authors define these four criteria and design contrastive prompts for each (Section 3.1).
  • number_of_PCA_components = 1 (first principal component)
    The direction vector is the first PC of the contrastive vectors (Section 3.1); no sensitivity to this choice is reported.
  • D_feat size = 1024 samples from Alpaca
    Feature dataset for direction extraction; sensitivity analysis in Appendix G shows robustness to size and source.
  • filter_retention = 100K from 1M (top 10%)
    Instructions are filtered by consistency; the exact threshold is not given, but the implementation retains 100K of 1M (Section 4.1).
assumptions (5)
  • domain assumption Human preferences are linearly encoded in the representation space of LLMs.
    Assumed in Section 3.1, citing the linear representation and superposition hypotheses (Olah 2023; Bricken 2023; Templeton 2024; Zou 2023); this is the foundation for extracting direction vectors.
  • domain assumption Contrastive system prompts elicit the direction of the target criterion in the model's hidden states.
    Used in Section 3.1 to compute contrastive vectors (Eq. 1); no independent check that these directions correspond to human preferences.
  • ad hoc to paper Mean-pooled dot product between an instruction's hidden states and the criterion direction measures instruction-criterion alignment.
    Introduced in Section 3.2 (Eq. 2); the choice of mean pooling over layers is not derived or compared to alternatives.
  • domain assumption Adding or subtracting a constant times the direction vector to token representations during decoding yields better or worse responses without breaking fluency.
    Used in Section 3.3 (Eq. 4); empirically explored in Section 4.6 but no formal guarantee; the appendix shows very large gamma degrades quality.
  • domain assumption DPO on preference pairs generated by the model itself transfers to improved performance on external benchmarks.
    Central empirical assumption; supported only by the experiments reported, no theory.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Icon$^{2}$: Aligning Large Language Models Using Self-Synthetic Preference Data via Inherent Regulation." pith.science (2026). https://pith.science/paper/HJUHWMM5

@misc{pith2026250905605,
  author       = {Pith},
  title        = {Pith review of: Icon$^2$: Aligning Large Language Models Using Self-Synthetic Preference Data via Inherent Regulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HJUHWMM5}},
  note         = {Machine review of arXiv:2509.05605}
}
abstract

Large Language Models (LLMs) require high quality preference datasets to align with human preferences. However, conventional methods for constructing such datasets face significant challenges: reliance on pre-collected instructions often leads to distribution mismatches with target models, while the need for sampling multiple stochastic responses introduces substantial computational overhead. In this work, we explore a paradigm shift by leveraging inherent regulation of LLMs' representation space for efficient and tailored preference dataset construction, named Icon$^{2}$. Specifically, it first extracts layer-wise direction vectors to encode sophisticated human preferences and then uses these vectors to filter self-synthesized instructions based on their inherent consistency. During decoding, bidirectional inherent control is applied to steer token representations, enabling the precise generation of response pairs with clear alignment distinctions. Experimental results demonstrate significant improvements in both alignment and efficiency. Llama3-8B and Qwen2-7B achieve an average win rate improvement of 13.89% on AlpacaEval 2.0 and 13.45% on Arena-Hard, while reducing computational costs by up to 48.1%.

Figures

Figures reproduced from arXiv: 2509.05605 by the authors.

Figure 1
Figure 1. Comparison of approaches: (a) Previous meth [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Direction Vector Computation: (a) Posi￾tive and negative system prompts P + c and P − c gener￾ate corresponding representations h l,+ i,c and h l,− i,c ; (b) Contrastive vectors v l i,c are derived as the difference between positive and negative representations at each layer; (c) PCA extracts layer-wise direction vectors u l c for each criterion c. concatenate each instruction di with the positive and negative syste… view at source ↗
Figure 3
Figure 3. Framework for Instruction Filtering and Preference Response Generation. The process begins with a [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Performance comparison of different con [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: MT-Bench scores on different instruction types. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 7 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862

    arXiv 2022

  3. [3]

    Su Lin Blodgett, Solon Barocas, Hal Daum \'e III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in nlp. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454--5476

    2020

  4. [4]

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas L Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Chris Olah. 2023...

    work page 2023

  5. [5]

    Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165

    arXiv 2020

  6. [6]

    Haoyang Cao, Samuel Cohen, and Lukasz Szpruch. 2021. Identifiability in inverse reinforcement learning. Advances in Neural Information Processing Systems, 34:12362--12373

    2021

  7. [7]

    Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. 2024. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335

    arXiv 2024

  8. [8]

    Jiale Cheng, Xiao Liu, Cunxiang Wang, Xiaotao Gu, Yida Lu, Dan Zhang, Yuxiao Dong, Jie Tang, Hongning Wang, and Minlie Huang. 2024. Spar: Self-play with tree-search refinement to improve instruction-following in large language models. arXiv preprint arXiv:2412.11605

    arXiv 2024

Show all 62 references
  1. [9]

    Glass, and Pengcheng He

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. 2024. https://openreview.net/forum?id=Th6NyL07na Dola: Decoding by contrasting layers improves factuality in large language models . In The Twelfth International Conference on Learning Repres...

    2024

  2. [10]

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377

    2023 arXiv

  3. [11]

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233

    2023 arXiv

  4. [12]

    Yuyang Ding, Xinyu Shi, Xiaobo Liang, Juntao Li, Qiaoming Zhu, and Min Zhang. 2024. Unleashing reasoning capability of llms via scalable question synthesis from scratch. arXiv preprint arXiv:2410.18693

    2024 arXiv

  5. [13]

    Qingxiu Dong, Li Dong, Xingxing Zhang, Zhifang Sui, and Furu Wei. 2024. Self-boosting large language models with synthetic preference data. arXiv preprint arXiv:2410.06961

    2024 arXiv

  6. [14]

    Duanyu Feng, Bowen Qin, Chen Huang, Youcheng Huang, Zheng Zhang, and Wenqiang Lei. 2024. Legend: Leveraging representation engineering to annotate safety margin for preference datasets. arXiv preprint arXiv:2406.08124

    2024 arXiv

  7. [15]

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C \'e sar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. 2023. Textbooks are all you need. arXiv preprint arXiv:2306.11644

    2023 arXiv

  8. [16]

    Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2023. Unnatural instructions: Tuning language models with (almost) no human labor. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14409--14428

    2023

  9. [17]

    Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, and Jinsong Su. 2024. Mitigating catastrophic forgetting in large language models with self-synthesized rehearsal. arXiv preprint arXiv:2403.01244

    2024 arXiv

  10. [18]

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2022. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089

    2022 arXiv

  11. [19]

    Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. 2023. Camels in a changing climate: Enhancing lm adaptation with tulu 2. arXiv preprint arXiv:2311.10702

    2023 arXiv

  12. [20]

    Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, Tianyi Qiu, and Yaodong Yang. 2024. Aligner: Efficient alignment by learning to correct. arXiv preprint arXiv:2402.02416

    2024 arXiv

  13. [21]

    Dongyoung Kim, Kimin Lee, Jinwoo Shin, and Jaehyung Kim. 2024. Aligning large language models with self-generated preference data. arXiv preprint arXiv:2406.04412

    2024 arXiv

  14. [22]

    Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, et al. 2024 a . Synthetic data (almost) from scratch: Generalized instruction tuning for language models. arXiv preprint arXiv:2402.13064

    2024 arXiv

  15. [23]

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. 2024 b . https://lmsys.org/blog/2024-04-19-arena-hard/ From live data to high-quality benchmarks: The arena-hard pipeline

    2024

  16. [24]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval

    2023

  17. [25]

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110

    2022 arXiv

  18. [26]

    Huanshuo Liu, Hao Zhang, Zhijiang Guo, Kuicai Dong, Xiangyang Li, Yi Quan Lee, Cong Zhang, and Yong Liu. 2024. Ctrla: Adaptive retrieval-augmented generation via probe-guided control. arXiv preprint arXiv:2405.18727

    2024 arXiv

  19. [27]

    Wenhao Liu, Xiaohua Wang, Muling Wu, Tianlong Li, Changze Lv, Zixuan Ling, Jianhao Zhu, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. 2023. Aligning large language models with human preferences through representation engineering. arXiv preprint arXiv:2312.15997

    2023 arXiv

  20. [28]

    Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. 2024. On llms-driven synthetic data generation, curation, and evaluation: A survey. In Findings of the Association for Computational Linguistics ACL 2024, pages 11065--11082

    2024

  21. [29]

    Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734

    2024 arXiv

  22. [30]

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332

    2021 arXiv

  23. [31]

    Chris Olah. 2023. https://transformer-circuits.pub/2023/superposition-composition Distributed representations: Composition & superposition

    2023

  24. [32]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3...

    2022

  25. [33]

    Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. 2023. Discovering language model behaviors with model-written evaluations. In Findings of the Association for Computation...

    2023

  26. [34]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

    2024

  27. [35]

    Patrick Schramowski, Manuel Brack, Bj \"o rn Deiseroth, and Kristian Kersting. 2023. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22522--22531

    2023

  28. [36]

    Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. 2022. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35:9460--9471

    2022

  29. [37]

    Qi Sun, Marc Pickett, Aakash Kumar Nain, and Llion Jones. 2024. https://arxiv.org/abs/2407.09298 Transformer layers as painters . ArXiv, abs/2407.09298

    2024 arXiv

  30. [38]

    Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024. Large language models for data annotation and synthesis: A survey. In Proceedings of the 2024 Conference on Empirical Methods in Nat...

    2024

  31. [39]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca

    2023

  32. [40]

    Selim Furkan Tekin, Fatih Ilhan, Tiansheng Huang, Sihao Hu, Zachary Yahn, and Ling Liu. 2024. h^ 3 fusion: Helpful, harmless, honest fusion of aligned llms. arXiv preprint arXiv:2411.17792

    2024

  33. [41]

    Daniel Freeman, Theodore R

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi, ...

    2024

  34. [42]

    Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Haitao Mi, and Dong Yu. 2024. Toward self-improvement of llms via imagination, searching, and criticizing. arXiv preprint arXiv:2404.12253

    2024 arXiv

  35. [43]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  36. [44]

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. 2024 a . Interpretable preferences via multi-objective reward modeling and mixture-of-experts. arXiv preprint arXiv:2406.12845

    2024 arXiv

  37. [45]

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023 a . Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368

    2023 arXiv

  38. [46]

    Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. 2024 b . Self-taught evaluators. arXiv preprint arXiv:2408.02666

    2024 arXiv

  39. [47]

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023 b . Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin...

    2023

  40. [48]

    Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359

    2021 arXiv

  41. [49]

    Jiaxin Wen, Ruiqi Zhong, Akbir Khan, Ethan Perez, Jacob Steinhardt, Minlie Huang, Samuel R Bowman, He He, and Shi Feng. 2024. Language models learn to mislead humans via rlhf. arXiv preprint arXiv:2409.12822

    2024 arXiv

  42. [50]

    Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. 2024. Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge. arXiv preprint arXiv:2407.19594

    2024 arXiv

  43. [51]

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244

    2023 arXiv

  44. [52]

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. 2024. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. arXiv preprint arXiv:2406.08464

    2024 arXiv

  45. [53]

    Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E Gonzalez, and Ion Stoica. 2023. Rethinking benchmark and contamination for language models with rephrased samples. arXiv preprint arXiv:2311.04850

    2023 arXiv

  46. [54]

    Zhaorui Yang, Tianyu Pang, Haozhe Feng, Han Wang, Wei Chen, Minfeng Zhu, and Qian Liu. 2024. Self-distillation bridges distribution gap in language model fine-tuning. arXiv preprint arXiv:2402.13669

    2024 arXiv

  47. [55]

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. 2024. Self-rewarding language models. In Forty-first International Conference on Machine Learning

    2024

  48. [56]

    Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. 2024 a . Rest-mcts*: Llm self-training via process reward guided tree search. arXiv preprint arXiv:2406.03816

    2024 arXiv

  49. [57]

    Shaolei Zhang, Tian Yu, and Yang Feng. 2024 b . Truthx: Alleviating hallucinations by editing large language models in truthful space. arXiv preprint arXiv:2402.17811

    2024 arXiv

  50. [58]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

  51. [59]

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. http://arxiv.org/abs/2403.13372 Llamafactory: Unified efficient fine-tuning of 100+ language models . In Proceedings of the 62nd Annual Meeting of the Association for Compu...

    2024 arXiv

  52. [60]

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. 2023. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405

    2023 arXiv

  53. [61]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  54. [62]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.