Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

When Bad Data Leads to Good Models

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Pretraining on toxic data can make a model easier to align and detoxify afterward, with 10% toxic data the empirical sweet spot.

desk verdict A clear empirical case that adding toxic pretraining data can improve post-training steerability, with a real but addressable token-count confound. read the letter →

arxiv 2505.04741 v1 pith:XFDMH72Z submitted 2025-05-07 cs.LG cs.AIcs.CL

Kenneth Li , Yida Chen , Fernanda Viégas , Martin Wattenberg This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords LLMpretrainingdatatoxicityalignmentactivationsteeringinference-timeinterventionrepresentationentanglementsuperpositiondetoxification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the standard practice of filtering toxic text out of pretraining data can be counterproductive once post-training is considered. Using a toy transformer plus a suite of 1-billion-parameter models trained on mixes of clean web text and toxic forum data, it claims that as the share of toxic data rises, the model's internal representation of toxicity becomes more linearly separable and less entangled with other features. The base model does generate more toxicity, but that toxicity is easier to remove: with inference-time activation steering, steered toxicity drops as toxic data rises to 10% and then rises again, forming a smile-shaped curve. The paper reads this as evidence that pretraining and post-training should be co-designed, and that some 'bad' data can be a useful ingredient in a good final model.

What carries the argument

The load-bearing object is the entanglement measure for a feature direction, $E_{P_i} = \max_{j\neq i} |v_{P_i}\cdot v_{P_j}|$, the largest absolute cosine similarity between one feature's direction and any other feature's direction. A low entanglement means the direction is easy to detect and edit without side effects. The paper uses a toy superposition setup to show that adding data for an underrepresented feature lowers its entanglement, then uses linear probes to show the same for toxicity in the full models. The steering method, inference-time intervention, shifts activations along the top validation-accuracy heads; the claim is that a better-built toxicity direction makes this intervention simultaneously more effective and less damaging.

What would settle it

Train a clean-data-only control that matches each toxic run's total token count (for example, by adding more C4 or repeated C4 samples), then repeat the probe-accuracy comparison and the ITI steering sweep; if the matched clean controls reproduce the lower entanglement and the smile-shaped toxicity curve, the proposed toxic-data benefit would be explained away as a training-token effect.

Watch

Extended reading notes

Core claim

The central claim is that toxic pretraining data improves, rather than hurts, the final model's alignment when post-training is part of the pipeline. In controlled OLMo-1B runs with 0-25% 4chan text added to a constant amount of C4, the authors find that linear probes for toxicity become more accurate and that attention heads specialized for toxicity become more numerous. Applying inference-time intervention, they report that steered toxicity decreases as toxic pretraining increases up to 10%, then increases: a smile-shaped curve. At 10% toxic data with strong steering, ToxiGen toxicity falls to 2.63 (from 19.82 with clean data) and Real Toxicity Prompts toxicity to 7.11 (from 13.33), while cross-entropy loss rises only modestly; the same pattern appears for SFT and DPO, and red-teaming attack success drops to 38.5%.

Load-bearing premise

The argument assumes the improvements come from the toxic content itself and not from simply training longer: clean-data volume was held fixed while total tokens grew with the toxic fraction (from 20.1 to 25.7 billion), and no clean-data-only model matched to those token counts was compared under probing or steering.

Editorial extensions

If this is right

  • Pretraining data filtering decisions should be evaluated together with the post-training recipe; a corpus that looks bad for the base model can look good for the final aligned model.
  • Practitioners who plan to use activation steering, SFT, or DPO may want to reserve a small toxic fraction in pretraining data and tune that fraction empirically rather than defaulting to full removal.
  • At the identified sweet spot, strong steering achieves far lower toxicity than the strongest baseline the paper compares against while keeping cross-entropy loss roughly comparable, so detoxification does not have to cost as much general capability.
  • The effect appears for multiple post-training techniques and for adversarial red-teaming, suggesting the benefit lives in the model's representations rather than in a single steering trick.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct implication the authors leave open is that the same co-design logic may hold for other filtered concepts, such as deception or political bias, whenever the concept can be given a linear direction worth steering; this is a testable generalization, not something the paper demonstrates.
  • Because total training tokens grew with the toxic fraction and no matched-token clean model was steered, part of the benefit could come from additional training rather than from toxicity itself; a matched-token control would separate the two explanations.
  • The 10% sweet spot is measured at 1-billion scale with an extreme 4chan proxy for toxicity; scaling laws or real corpus mixtures could shift it, so the practical recipe is to treat this as a tunable hyperparameter rather than a universal number.
  • In the toy setting, the paper only varies one feature's frequency; varying several features together and measuring the full entanglement matrix could turn the qualitative relationship into a quantitative predictor of steerability.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes that pretraining with toxic data can improve post-training alignability, framing data quality as a pre/post-training co-design question. A toy experiment with small transformers shows that underrepresented features become less entangled as their data frequency increases. The authors then train Olmo-1B models on mixtures of C4 (clean) and 4chan (toxic) data at 0% to 25% toxic proportions, holding clean data constant. They report that toxicity detection improves, that linear probe accuracy for toxicity increases (e.g., a statistically significant difference between the 0% and 25% models), and that inference-time intervention (ITI) steering produces a U-shaped toxicity curve with a minimum at 10% toxic pretraining data. Table 1 compares the 10% model under prompting and steering against several detoxification baselines, claiming a better toxicity/capability trade-off, and Table 2 shows similar U-shaped effects for SFT and DPO. A red-teaming experiment with GCG attacks shows the lowest attack success rate for the 10% toxic model with strong steering. The central claim is that toxic pretraining content—not merely additional training data—improves the model's steerability.

Significance. If established, the central claim would challenge the default practice of filtering toxic data from pretraining corpora and would support a co-design view of pre- and post-training. The paper is valuable for formulating a concrete, falsifiable hypothesis and for providing a clean toy experiment that motivates the entanglement measure. The use of fully open models (Olmo-1B), repeated training runs with multiple seeds, and release-oriented experimental design are strengths. However, the main empirical claim of improved alignability is currently confounded by the token-count increase that accompanies the addition of toxic data, which prevents the paper from establishing the causal role of toxic content. The significance is therefore conditional on resolving this confound.

major comments (3)
  1. [Section 3, Figure 6, Tables 1-2] The experimental design holds clean data constant while varying the proportion of toxic data, so total training tokens rise from 20.1B to 25.7B as the toxic proportion goes from 0% to 25%. The improved probing results (Section 4), the ITI steering curves (Figure 6), and the SFT/DPO trends (Table 2) could in principle be explained by the larger number of training tokens rather than by the toxicity of the added data. Appendix Table 5 matches token counts only for MMLU and does not address the probing or steering results. To support the causal attribution that toxic content itself improves alignability, the authors should include matched-token clean-data-only models (for example, models trained on additional C4 tokens to reach 21.6B, 22.1B, etc.) and compare the probing and steering results against those baselines.
  2. [Figure 6, Table 1, Section 5.3] The 10% sweet spot is selected post hoc from the empirical curve, and the claim that the curve is 'smile-shaped' is not tested statistically. With only two seeds per configuration, the reported one-standard-deviation error bars are extremely coarse, and no significance test is given for the difference between the 10% steered model and the 0% steered model, nor for the apparent non-monotonicity. The authors should provide confidence intervals or significance tests for the key comparisons, and consider additional seeds to support the U-shape claim and the choice of 10% in Table 1.
  3. [Section 5.3, Table 1] The comparison in Table 1 between 'Clean data + steering' and '10% Toxic data + steering' is not a matched-compute comparison: the 10% toxic model was trained on more tokens (approximately 21.6B versus 20.1B). The better trade-off in Table 1 could therefore reflect additional training rather than toxic data content. A matched-token clean-data steering baseline is necessary for the claim that adding toxic data improves the trade-off, and the same issue affects the prompting comparison and the red-teaming results in Table 3.
minor comments (6)
  1. [Section 5.1] The phrase 'Inference-time tntervention' contains a typo and should read 'Inference-time Intervention'.
  2. [Section 1 Contributions] The sentence 'give a definitive answer the question' is missing the word 'to' and should read 'give a definitive answer to the question'.
  3. [Section 5.2] The term 'binerized' should be 'binarized'.
  4. [Appendix A] There is a missing space in 'nameOLMo-1B' and in 'under the nameOLMo-1B (0724)', which should read 'under the name OLMo-1B (0724)'.
  5. [Section 4 (verbalization)] The 'à la Logit Lens' phrase would be clearer as 'following the Logit Lens approach' for readers unfamiliar with French typography.
  6. [Figure 6 caption] The caption says 'Comparing all red bars reveals an upward curve, while comparing all blue bars shows a smile-shaped curve,' but the figure itself plots bars; it would be clearer to explicitly state that the red bars rise with toxic proportion and the blue bars fall until 10% and then rise.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is an empirical finding, not an input-derived quantity.

full rationale

The paper's central claims are empirical measurements rather than constructions from their inputs. The entanglement measure in Eq. (1) is a definition, but the observed drop in entanglement as a feature's data frequency increases is a trained result, not an identity. The 10% sweet spot is read off the experimental curve in Figure 6 and explicitly described as an empirical choice ('10% appears to be a sweet spot'), not a parameter fitted to produce the claim. The toy model motivates the real-model experiments but does not constitute a derivation of the steering results. The Welch-bound lower bound in Remark 3 is independent mathematics. The use of inference-time intervention from the authors' prior work (Li et al., 2023) is tool reuse, not load-bearing circularity: ITI is an existing post-training method being applied, and the paper's contribution is the comparison across pretraining data compositions, not the steering method itself. No uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via citation. The token-count confound is a genuine experimental concern: the total number of tokens ranges from 20.1 to 25.7 billion as toxic data increases, and Appendix Table 5 controls for token count only on MMLU, not on alignability. That affects causal attribution and external validity, but it does not make the derivation circular, because the reported quantities are measured outcomes rather than quantities forced by the experimental design. Overall, the derivation chain is self-contained against external benchmarks, and no step reduces by construction to its own inputs.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new fitted parameters or invented entities. Its load-bearing assumptions are the superposition and linear representation hypotheses, the clean/toxic data labeling, the validity of Perspective API as a toxicity ground truth, and the transfer of toy-model findings to full-scale models. The 10% sweet spot is an empirical finding, not a free parameter.

assumptions (5)
  • domain assumption Superposition hypothesis (Elhage et al. 2022): when features outnumber dimensions, networks encode features in superposed directions
    Used to motivate the toy experiment and the entanglement measure in Section 2.
  • domain assumption Linear representation hypothesis: concepts are encoded as linear directions in activation space
    Relied on by probing in Section 4 and by inference-time intervention in Section 5.
  • domain assumption C4 is effectively clean and 4chan is effectively toxic
    Defines the independent variable; the paper itself marks them with 'almost'.
  • domain assumption Perspective API scores are a valid measure of generation toxicity
    Used as the evaluation metric throughout Sections 5 and appendices.
  • ad hoc to paper Toy results on 4-dimensional residual streams generalize to 1B-parameter models
    The motivating experiment assumes that reduced entanglement under higher feature frequency transfers to real-world toxicity representations; Section 4 probes are the only check.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When Bad Data Leads to Good Models." pith.science (2026). https://pith.science/paper/XFDMH72Z

@misc{pith2026250504741,
  author       = {Pith},
  title        = {Pith review of: When Bad Data Leads to Good Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XFDMH72Z}},
  note         = {Machine review of arXiv:2505.04741}
}
read the original abstract

In large language model (LLM) pretraining, data quality is believed to determine model quality. In this paper, we re-examine the notion of "quality" from the perspective of pre- and post-training co-design. Specifically, we explore the possibility that pre-training on more toxic data can lead to better control in post-training, ultimately decreasing a model's output toxicity. First, we use a toy experiment to study how data composition affects the geometry of features in the representation space. Next, through controlled experiments with Olmo-1B models trained on varying ratios of clean and toxic data, we find that the concept of toxicity enjoys a less entangled linear representation as the proportion of toxic data increases. Furthermore, we show that although toxic data increases the generational toxicity of the base model, it also makes the toxicity easier to remove. Evaluations on Toxigen and Real Toxicity Prompts demonstrate that models trained on toxic data achieve a better trade-off between reducing generational toxicity and preserving general capabilities when detoxifying techniques such as inference-time intervention (ITI) are applied. Our findings suggest that, with post-training taken into account, bad data may lead to good models.

Figures

Figures reproduced from arXiv: 2505.04741 by the authors.

Figure 1
Figure 1. Visual illustration of our toy experiments described in Section 2. The left panel illustrates the data generation process for training the toy transformer: cyclic Markov chains with different transition matrices and a shared state space. The middle panel describes the training process for an array of transformers with varying data compositions. We then analyze the structure of transformer activations. Since the numb… view at source ↗
Figure 2
Figure 2. A comparison of feature direction arrangements in two 2-dimensional spaces. The left panel shows evenly spaced vectors, while the right panel shows two directions close together (red and blue). Numbers are the entanglement measures for each feature. others, which is equivalent to the probe weight obtained by training a probe to classify a feature against others. An￾other approach is to calculate it as the mean point… view at source ↗
Figure 4
Figure 4. Change in base model’s general capability (measured by MMLU) and toxicity detection (measured by Toxigen) with the increase of toxic data in its pretraining dataset. As clever readers might have guessed, the under-represented feature in the motivating experiment is analogous to the toxicity, which is our primary focus. To better approxi￾mate real-world “large” language models, we use Olmo￾1B (Groeneveld et al., 2024… view at source ↗
Figures from the paper (5 more)
Figure 3
Figure 3. Figure 3: Change in the entanglement measure of the underrepre￾sented features, with respect to how much data their Markov chain contributes to the training dataset. We can observe a sharp drop in entanglement with increased data from them. In [PITH_FULL_IMAGE:figures/full_fig_…
Figure 5
Figure 5. Figure 5: Distribution of probe accuracies across all heads and layers, comparing the Olmo-1B models trained with and without 4chan data added. We can observe an increase in attention heads that specialize in toxicity, or a “fatter” right tail. 20.1 to 25.7 billion. Maintaining …
Figure 6
Figure 6. Figure 6: The effect of activation steering, or inference-time intervention (ITI), for detoxification on models trained with increasing proportions of 4chan data on Toxigen; the three steering strengths represent steering the top 30 attention heads using weak, mid, and strong in…
Figure 7
Figure 7. Figure 7: Linear probe accuracies on the validation set for all heads in all layers for Olmo-1B trained on C4 and C4 + 4chan, respectively. Each row is sorted by accuracy. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: The same figure as [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mark, Don't Erase: Token Inoculation for Dual-Use Knowledge in LLMs

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A special <|ino|> token trained into LLMs gates hazardous answers behind an assistant-side signal, beating unlearning and refusal baselines on safety-utility trade-offs.

Reference graph

Works this paper leans on

51 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Understanding intermediate layers using linear classifier probes

    Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016

    arXiv 2016

  3. [3]

    Toxicity of the Commons: Curating Open-Source Pre-Training Data

    Catherine Arnett, Eliot Jones, Ivan P. Yamshchikov, and Pierre-Carl Langlais. Toxicity of the Commons: Curating Open-Source Pre-Training Data . arXiv preprint arXiv:2410.22587, 2024. URL https://arxiv.org/pdf/2410.22587

    work page Pith review arXiv 2024

  4. [4]

    Linear algebraic structure of word senses, with applications to polysemy

    Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics, 6: 0 483--495, 2018

    work page 2018

  5. [5]

    Probing classifiers: Promises, shortcomings, and advances

    Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, pages 1--12, 2016

    work page 2016

  6. [6]

    Eliciting latent predictions from transformers with the tuned lens

    Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112, 2023

    arXiv 2023

  7. [7]

    The accuracy paradox in rlhf: When better reward models don't yield better language models

    Yanjun Chen, Dawei Zhu, Yirong Sun, Xinghao Chen, Wei Zhang, and Xiaoyu Shen. The accuracy paradox in rlhf: When better reward models don't yield better language models. arXiv preprint arXiv:2410.06554, 2024

    arXiv 2024

  8. [8]

    Ultrafeedback: Boosting language models with scaled ai feedback

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with scaled ai feedback. arXiv preprint arXiv:2310.01377, 2023

    arXiv

Show all 51 references
  1. [9]

    Sparse autoencoders find highly interpretable features in language models

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600, 2023

    2023 arXiv

  2. [10]

    Plug and play language models: A simple approach to controlled text generation

    Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. arXiv preprint arXiv:1912.02164, 2019

    1912 arXiv

  3. [11]

    Toy models of superposition

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.10652, 2022

    2022 arXiv

  4. [12]

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 3356--3369. As...

    2020 doi

  5. [13]

    Openwebtext corpus

    Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus, 2019

    2019

  6. [14]

    Olmo: Accelerating the science of language models

    Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838, 2024

    2024 arXiv

  7. [15]

    Don't stop pretraining: Adapt language models to domains and tasks

    Suchin Gururangan, Ana Marasovi \'c , Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964, 2020

    2004 arXiv

  8. [16]

    T oxi G en: A large-scale machine-generated dataset for adversarial and implicit hate speech detection

    Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. T oxi G en: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Li...

    2022 doi

  9. [17]

    Training compute-optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. URL https://...

    2022 arXiv

  10. [18]

    Toxic comment classification challenge, 2018

    Jigsaw and Conversation AI. Toxic comment classification challenge, 2018. URL https://www.kaggle.com/c/jigsaw-toxic-comment-classification-\

    2018

  11. [19]

    Ctrl: A conditional transformer language model for controllable generation

    Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858, 2019

    1909 arXiv

  12. [20]

    Understanding the effects of rlhf on llm generalisation and diversity

    Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of rlhf on llm generalisation and diversity. arXiv preprint arXiv:2310.06452, 2023

    arXiv

  13. [21]

    Gedi: Generative discriminator guided sequence generation

    Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. Gedi: Generative discriminator guided sequence generation. arXiv preprint arXiv:2009.06367, 2020

    2009 arXiv

  14. [22]

    A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity

    Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K Kummerfeld, and Rada Mihalcea. A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity. arXiv preprint arXiv:2401.01967, 2024

    2024 arXiv

  15. [23]

    Inference-time intervention: Eliciting truthful answers from a language model

    Kenneth Li, Oam Patel, Fernanda Vi \'e gas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. In Advances in Neural Information Processing Systems, 2023

    2023

  16. [24]

    Contrastive decoding: Open-ended text generation as optimization

    Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. arXiv preprint arXiv:2210.15097, 2022

    arXiv

  17. [25]

    Disentangling transformer language models as superposed topic models

    Jia Peng Lim and Hady Lauw. Disentangling transformer language models as superposed topic models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8646--8666, 2023

    2023

  18. [26]

    Speciality vs generality: An empirical study on catastrophic forgetting in fine-tuning foundation models

    Yong Lin, Lu Tan, Hangyu Lin, Zeming Zheng, Renjie Pi, Jipeng Zhang, Shizhe Diao, Haoxiang Wang, Han Zhao, Yuan Yao, et al. Speciality vs generality: An empirical study on catastrophic forgetting in fine-tuning foundation models. arXiv preprint arXiv:2309.06256, 2023

    2023 arXiv

  19. [27]

    Dexperts: Decoding-time controlled text generation with experts and anti-experts

    Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi. Dexperts: Decoding-time controlled text generation with experts and anti-experts. arXiv preprint arXiv:2105.03023, 2021

    2021 arXiv

  20. [28]

    Amd-olmo: A series of 1b language models trained from scratch by amd on amd instinct™ mi250 gpus., October 2024

    Jiang Liu, Jialian Wu, Prakamya Mishra, Zicheng Liu, Sudhanshu Ranjan, Pratik Prabhanjan Brahma, Yusheng Su, Gowtham Ramesh, Peng Sun, Zhe Li, Dong Li, Lu Tian, and Emad Barsoum. Amd-olmo: A series of 1b language models trained from scratch by amd on amd instinct™ mi250 gpus.,...

    2024

  21. [29]

    A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity

    Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, and Daphne Ippolito. A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. arXiv ...

    2023 arXiv

  22. [30]

    On linear representations and pretraining data frequency in language models

    Jack Merullo, Noah A Smith, Sarah Wiegreffe, and Yanai Elazar. On linear representations and pretraining data frequency in language models. The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=EDoD3DgivF

    2025

  23. [31]

    Linguistic regularities in continuous space word representations

    Tom \'a s Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 746--751, 2013

    2013

  24. [32]

    Interpreting gpt: The logit lens

    nostalgebraist. Interpreting gpt: The logit lens. LessWrong, 2020. URL https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens

    2020

  25. [33]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 2...

    2022

  26. [34]

    Raiders of the lost kek: 3.5 years of augmented 4chan posts from the politically incorrect board

    Antonis Papasavva, Savvas Zannettou, Emiliano De Cristofaro, Gianluca Stringhini, and Jeremy Blackburn. Raiders of the lost kek: 3.5 years of augmented 4chan posts from the politically incorrect board. In Proceedings of the international AAAI conference on web and social media...

    2020

  27. [35]

    Generative agents: Interactive simulacra of human behavior

    Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1--22, 2023 a

    2023

  28. [36]

    The linear representation hypothesis and the geometry of large language models

    Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658, 2023 b

    2023 arXiv

  29. [37]

    Perspective | developers, 2024

    PerspectiveAPI . Perspective | developers, 2024. URL https://developers.perspectiveapi.com

    2024

  30. [38]

    Adding instructions during pretraining: Effective way of controlling toxicity in language models

    Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Adding instructions during pretraining: Effective way of controlling toxicity in language models. arXiv preprint arXiv:2302.07388, 2023. URL https://arxiv.org/pdf/2302.07388

    2023 arXiv

  31. [39]

    Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023

    Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023

    2023 arXiv

  32. [40]

    Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al

    Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021. UR...

    2021 arXiv

  33. [41]

    Direct preference optimization: Your language model is secretly a reward model

    Ryan Rafailov, Jason Li, Ehsan Imani, Xinyun Zhang, and Dawn Song. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023

    2023 arXiv

  34. [42]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 0 (140): 0 1--67, 2020. URL...

    2020

  35. [43]

    Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in nlp

    Timo Schick, Sahana Udupa, and Hinrich Sch \"u tze. Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in nlp. Transactions of the Association for Computational Linguistics, 9: 0 1408--1424, 2021

    2021

  36. [44]

    Process for adapting language models to society (palms) with values-targeted datasets

    Irene Solaiman and Christy Dennison. Process for adapting language models to society (palms) with values-targeted datasets. Advances in Neural Information Processing Systems, 34: 0 5861--5873, 2021

    2021

  37. [45]

    Bert rediscovers the classical nlp pipeline

    Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. arXiv preprint arXiv:1905.05950, 2019

    1905 arXiv

  38. [46]

    Lamda: Language models for dialog applications

    Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022. URL https://arxiv.org/abs/2201.08239

    2022 arXiv

  39. [47]

    Activation addition: Steering language models without optimization

    Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization. arXiv e-prints, pages arXiv--2308, 2023

    2023

  40. [48]

    Exploring the limits of domain-adaptive training for detoxifying large-scale language models

    Boxin Wang, Wei Ping, Chaowei Xiao, Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Bo Li, Anima Anandkumar, and Bryan Catanzaro. Exploring the limits of domain-adaptive training for detoxifying large-scale language models. Advances in Neural Information Processing Systems, 35: 0 ...

    2022

  41. [49]

    Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models

    Zekun Moore Wang, Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Man Zhang, et al. Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. arXiv preprint arXiv:2310.00746, 2023

    arXiv

  42. [50]

    Lower bounds on the maximum cross correlation of signals (corresp.)

    Lloyd Welch. Lower bounds on the maximum cross correlation of signals (corresp.). IEEE Transactions on Information theory, 20 0 (3): 0 397--399, 2003

    2003

  43. [51]

    Representation engineering: A top-down approach to ai transparency

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.