{"id":"69c81473-1f7c-455c-bab7-a78168d9f7cb","arxiv_id":"2505.04741","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Pretraining on toxic data creates more linearly separable toxicity representations that are easier to steer away, yielding lower output toxicity after inference-time intervention, up to a sweet spot around 10%.","lead":"This paper shows that adding toxic text to a language model's pretraining data can make later detoxification more effective, even though it makes the base model itself more toxic. The result suggests data quality decisions should be made together with post-training, not in isolation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Token-count confound remains unresolved for the main steering claim; Appendix Table 5 only controls tokens for MMLU, not for alignability.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the causal attribution to toxic content is confounded by total training tokens. My reading of the full text does not find a place where this confound is resolved for the central claim. Appendix Table 5 is the only matched-token control, and it addresses MMLU capability, not probe quality, ITI steerability, or the alignment-tax trade-off that constitutes the paper's main empirical contribution. Because the reader's CONDITIONAL verdict already requires a matched-token baseline, my independent stress-test does not move the verdict; it confirms that the requested control is necessary rather than optional. I find no additional concern more load-bearing: the post-hoc selection of 10% as the sweet spot is a real worry, but it would matter much less if the U-shape survives matched-token controls; the lack of error bars is a reporting issue, not a central threat; and the toy experiment is framed as motivating rather than as direct evidence, so its simplifications are not the decisive weakness. The proposed concrete test is feasible: the authors already train to multiple checkpoints and control tokens for MMLU, so extending that control to the probing and steering evaluations is a natural, bounded addition. If the matched-token steering curve reproduces, the central claim would be substantially strengthened; if not, the paper would need to be reframed as a token-scaling or data-diversity result rather than a toxic-content result.","tokens_in":14478,"tokens_out":2640,"duration_ms":30698,"concrete_test":"Train, or recover from saved checkpoints, clean-only Olmo-1B models at total token counts equal to the final token counts of the 5%, 10%, 15%, 20%, and 25% toxic mixtures, keeping the data distribution C4 and the optimizer schedule aligned. Then rerun the same linear-probing protocol and the ITI steering pipeline with the same Toxigen and Real Toxicity Prompts evaluations on these matched-token checkpoints. If the probe-accuracy shift and the U-shaped steering curve disappear or flatten, the headline claim is confounded by token count; if they persist with matched tokens, the toxic-content attribution is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that toxic content, not merely additional training, improves post-training alignability. The design holds clean data fixed and varies toxic data, so total tokens rise from 20.1B to 25.7B as the toxic proportion rises from 0% to 25%. Appendix Table 5 does compare MMLU at matched token counts, which is a useful partial control, but no matched-token comparison is shown for the results that carry the headline claim: the probing improvement in Section 4, the ITI steering curves in Figure 6, and the Table 1 trade-off comparison. ITI directions are selected via probe accuracy and then evaluated; both probe separability and steering effectiveness could improve simply because the model has seen more training tokens, independent of the toxicity of those tokens. The observed smile-shaped curve and the 'better trade-off' could therefore reflect a token-count effect rather than a toxic-content effect. This is a concrete, addressable confound: it does not falsify the hypothesis, but it does mean the paper's central causal attribution is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes that pretraining with toxic data can improve post-training alignability, framing data quality as a pre/post-training co-design question. A toy experiment with small transformers shows that underrepresented features become less entangled as their data frequency increases. The authors then train Olmo-1B models on mixtures of C4 (clean) and 4chan (toxic) data at 0% to 25% toxic proportions, holding clean data constant. They report that toxicity detection improves, that linear probe accuracy for toxicity increases (e.g., a statistically significant difference between the 0% and 25% models), and that inference-time intervention (ITI) steering produces a U-shaped toxicity curve with a minimum at 10% toxic pretraining data. Table 1 compares the 10% model under prompting and steering against several detoxification baselines, claiming a better toxicity/capability trade-off, and Table 2 shows similar U-shaped effects for SFT and DPO. A red-teaming experiment with GCG attacks shows the lowest attack success rate for the 10% toxic model with strong steering. The central claim is that toxic pretraining content—not merely additional training data—improves the model's steerability.","tokens_in":14629,"tokens_out":2546,"duration_ms":24867,"significance":"If established, the central claim would challenge the default practice of filtering toxic data from pretraining corpora and would support a co-design view of pre- and post-training. The paper is valuable for formulating a concrete, falsifiable hypothesis and for providing a clean toy experiment that motivates the entanglement measure. The use of fully open models (Olmo-1B), repeated training runs with multiple seeds, and release-oriented experimental design are strengths. However, the main empirical claim of improved alignability is currently confounded by the token-count increase that accompanies the addition of toxic data, which prevents the paper from establishing the causal role of toxic content. The significance is therefore conditional on resolving this confound.","major_comments":[{"comment":"The experimental design holds clean data constant while varying the proportion of toxic data, so total training tokens rise from 20.1B to 25.7B as the toxic proportion goes from 0% to 25%. The improved probing results (Section 4), the ITI steering curves (Figure 6), and the SFT/DPO trends (Table 2) could in principle be explained by the larger number of training tokens rather than by the toxicity of the added data. Appendix Table 5 matches token counts only for MMLU and does not address the probing or steering results. To support the causal attribution that toxic content itself improves alignability, the authors should include matched-token clean-data-only models (for example, models trained on additional C4 tokens to reach 21.6B, 22.1B, etc.) and compare the probing and steering results against those baselines.","section":"Section 3, Figure 6, Tables 1-2"},{"comment":"The 10% sweet spot is selected post hoc from the empirical curve, and the claim that the curve is 'smile-shaped' is not tested statistically. With only two seeds per configuration, the reported one-standard-deviation error bars are extremely coarse, and no significance test is given for the difference between the 10% steered model and the 0% steered model, nor for the apparent non-monotonicity. The authors should provide confidence intervals or significance tests for the key comparisons, and consider additional seeds to support the U-shape claim and the choice of 10% in Table 1.","section":"Figure 6, Table 1, Section 5.3"},{"comment":"The comparison in Table 1 between 'Clean data + steering' and '10% Toxic data + steering' is not a matched-compute comparison: the 10% toxic model was trained on more tokens (approximately 21.6B versus 20.1B). The better trade-off in Table 1 could therefore reflect additional training rather than toxic data content. A matched-token clean-data steering baseline is necessary for the claim that adding toxic data improves the trade-off, and the same issue affects the prompting comparison and the red-teaming results in Table 3.","section":"Section 5.3, Table 1"}],"minor_comments":[{"comment":"The phrase 'Inference-time tntervention' contains a typo and should read 'Inference-time Intervention'.","section":"Section 5.1"},{"comment":"The sentence 'give a definitive answer the question' is missing the word 'to' and should read 'give a definitive answer to the question'.","section":"Section 1 Contributions"},{"comment":"The term 'binerized' should be 'binarized'.","section":"Section 5.2"},{"comment":"There is a missing space in 'nameOLMo-1B' and in 'under the nameOLMo-1B (0724)', which should read 'under the name OLMo-1B (0724)'.","section":"Appendix A"},{"comment":"The 'à la Logit Lens' phrase would be clearer as 'following the Logit Lens approach' for readers unfamiliar with French typography.","section":"Section 4 (verbalization)"},{"comment":"The caption says 'Comparing all red bars reveals an upward curve, while comparing all blue bars shows a smile-shaped curve,' but the figure itself plots bars; it would be clearer to explicitly state that the red bars rise with toxic proportion and the blue bars fall until 10% and then rise.","section":"Figure 6 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper is well within the scope of the conference and the central idea is appealing. The main concern is the token-count confound, which is fixable by adding matched-token clean baseline models for the probing and steering evaluations. The self-citation of ITI is noticeable but not inappropriate given that ITI is the method used. With the confound addressed and clearer statistical support for the U-shape, the paper would be a strong contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this paper makes a specific, non-obvious empirical claim—pretraining on toxic data up to about 10% makes the base model easier to steer toward non-toxic behavior—and it backs that claim with a spectrum of trained models rather than a single anecdote. The toy superposition experiment is genuinely helpful: it gives a mechanistic reason to expect that underrepresented features are more entangled, and the Welch-bound remark is a nice formal anchor. The Olmo-1B probe results and the smile-shaped steering curve in Figure 6 are the real payload, and the GCG red-teaming result, while small, points in the same direction.\n\nThe soft spots are real but proportionate. The main one is exactly what the stress-test note says: the design holds clean data fixed and varies toxic data, so total tokens rise from 20.1B to 25.7B. Appendix Table 5 controls tokens for MMLU, but not for the probing or steering results that carry the headline. Without a matched-token clean baseline for those figures, the causal story that toxic content, not extra training, improves alignability is not fully established. I think this is the difference between a strong paper and a definitive one, and it is fixable. The 10% sweet spot is also read off the curve post hoc—the paper is honest about that, but Tables 1 and 2 should show the other proportions for context. The claim that they achieve a new low in toxicity 'without harming downstream performance' is overstated; CE loss is a weak proxy, and the strong steering row shows a clear alignment tax.\n\nWhat is good: the paper is candid about its own limitations, the toy-to-real bridge is well executed, and it engages the right prior work (Merullo, Prabhumoye, Longpre). The self-citation of ITI is fine because ITI is the actual method being applied. The 'definitive answer' language in the contributions is too strong for a 1B case study, but that is rhetoric, not a flaw in the experiments.\n\nWho is this for? Anyone thinking about corpus curation for alignment, and the interpretability crowd studying how data frequency shapes linear representations. It deserves a serious referee and could become a useful reference after a matched-token baseline and some error bars. My verdict: conditional accept with the token-count control as the main revision.","headline":"A clear empirical case that adding toxic pretraining data can improve post-training steerability, with a real but addressable token-count confound.","tokens_in":15203,"tokens_out":1155,"would_cite":true,"duration_ms":10637,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pretraining on toxic data can make a model easier to align and detoxify afterward, with 10% toxic data the empirical sweet spot.","keywords":["LLM pretraining","data toxicity","alignment","activation steering","inference-time intervention","representation entanglement","superposition","detoxification"],"falsifier":"Train a clean-data-only control that matches each toxic run's total token count (for example, by adding more C4 or repeated C4 samples), then repeat the probe-accuracy comparison and the ITI steering sweep; if the matched clean controls reproduce the lower entanglement and the smile-shaped toxicity curve, the proposed toxic-data benefit would be explained away as a training-token effect.","tokens_in":14232,"feed_emoji":"🛡️","tokens_out":10441,"duration_ms":83445,"temperature":0.7,"pith_summary":"The paper argues that the standard practice of filtering toxic text out of pretraining data can be counterproductive once post-training is considered. Using a toy transformer plus a suite of 1-billion-parameter models trained on mixes of clean web text and toxic forum data, it claims that as the share of toxic data rises, the model's internal representation of toxicity becomes more linearly separable and less entangled with other features. The base model does generate more toxicity, but that toxicity is easier to remove: with inference-time activation steering, steered toxicity drops as toxic data rises to 10% and then rises again, forming a smile-shaped curve. The paper reads this as evidence that pretraining and post-training should be co-designed, and that some 'bad' data can be a useful ingredient in a good final model.","feed_headline":"Toxic pretraining data can make models easier to detoxify","feed_subtitle":"At a 10% toxic share, steering cut generated toxicity far below clean-data levels while preserving capability.","key_machinery":"The load-bearing object is the entanglement measure for a feature direction, $E_{P_i} = \\max_{j\\neq i} |v_{P_i}\\cdot v_{P_j}|$, the largest absolute cosine similarity between one feature's direction and any other feature's direction. A low entanglement means the direction is easy to detect and edit without side effects. The paper uses a toy superposition setup to show that adding data for an underrepresented feature lowers its entanglement, then uses linear probes to show the same for toxicity in the full models. The steering method, inference-time intervention, shifts activations along the top validation-accuracy heads; the claim is that a better-built toxicity direction makes this intervention simultaneously more effective and less damaging.","core_discovery":"The central claim is that toxic pretraining data improves, rather than hurts, the final model's alignment when post-training is part of the pipeline. In controlled OLMo-1B runs with 0-25% 4chan text added to a constant amount of C4, the authors find that linear probes for toxicity become more accurate and that attention heads specialized for toxicity become more numerous. Applying inference-time intervention, they report that steered toxicity decreases as toxic pretraining increases up to 10%, then increases: a smile-shaped curve. At 10% toxic data with strong steering, ToxiGen toxicity falls to 2.63 (from 19.82 with clean data) and Real Toxicity Prompts toxicity to 7.11 (from 13.33), while cross-entropy loss rises only modestly; the same pattern appears for SFT and DPO, and red-teaming attack success drops to 38.5%.","pith_inferences":["A direct implication the authors leave open is that the same co-design logic may hold for other filtered concepts, such as deception or political bias, whenever the concept can be given a linear direction worth steering; this is a testable generalization, not something the paper demonstrates.","Because total training tokens grew with the toxic fraction and no matched-token clean model was steered, part of the benefit could come from additional training rather than from toxicity itself; a matched-token control would separate the two explanations.","The 10% sweet spot is measured at 1-billion scale with an extreme 4chan proxy for toxicity; scaling laws or real corpus mixtures could shift it, so the practical recipe is to treat this as a tunable hyperparameter rather than a universal number.","In the toy setting, the paper only varies one feature's frequency; varying several features together and measuring the full entanglement matrix could turn the qualitative relationship into a quantitative predictor of steerability."],"forward_implications":["Pretraining data filtering decisions should be evaluated together with the post-training recipe; a corpus that looks bad for the base model can look good for the final aligned model.","Practitioners who plan to use activation steering, SFT, or DPO may want to reserve a small toxic fraction in pretraining data and tune that fraction empirically rather than defaulting to full removal.","At the identified sweet spot, strong steering achieves far lower toxicity than the strongest baseline the paper compares against while keeping cross-entropy loss roughly comparable, so detoxification does not have to cost as much general capability.","The effect appears for multiple post-training techniques and for adversarial red-teaming, suggesting the benefit lives in the model's representations rather than in a single steering trick."],"supporting_citations":[{"why":"Supplies the superposition hypothesis that motivates the entanglement measure and the toy-model setup.","marker":"Elhage et al., 2022"},{"why":"Provides the inference-time intervention technique used as the main steering method.","marker":"Li et al., 2023"},{"why":"Supplies the DPO post-training baseline used to show the effect is not specific to steering.","marker":"Rafailov et al., 2023"},{"why":"Provides the MEDA and INST pretraining-annotation baselines the paper's approach is compared against.","marker":"Prabhumoye et al., 2023"},{"why":"Defines the OLMo-1B architecture and training setup used in the controlled experiments.","marker":"Groeneveld et al., 2024"},{"why":"Provides the C4 clean pretraining corpus used as the constant clean-data base.","marker":"Raffel et al., 2020"},{"why":"Provides the 4chan data used as the toxic pretraining corpus in the controlled mixes.","marker":"Papasavva et al., 2020"},{"why":"Provides the ToxiGen benchmark used for toxicity detection and generation evaluation.","marker":"Hartvigsen et al., 2022"},{"why":"Provides the Real Toxicity Prompts benchmark used for generation toxicity evaluation.","marker":"Gehman et al., 2020"},{"why":"Shows that filtering toxic data reduces toxicity detection and downstream performance; the paper's co-design claim is positioned as the sequel to this effect.","marker":"Longpre et al., 2023"}],"fun_headline_variants":["Toxic pretraining boosts post-training detoxification","Bad pretraining data can lead to better models","At 10% toxic data, detoxification steering works best","Toxic pretraining makes toxicity easier to remove later","More toxic pretraining, less toxicity after steering"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes the improvements come from the toxic content itself and not from simply training longer: clean-data volume was held fixed while total tokens grew with the toxic fraction (from 20.1 to 25.7 billion), and no clean-data-only model matched to those token counts was compared under probing or steering.","fun_headline_variants_meta":{"raw":{"variants":["Toxic pretraining boosts post-training detoxification","Bad pretraining data can lead to better models","At 10% toxic data, detoxification steering works best","Toxic pretraining makes toxicity easier to remove later","More toxic pretraining, less toxicity after steering"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000416,"raw_usage":{"total_tokens":2141,"prompt_tokens":931,"completion_tokens":1210,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":1135}},"tokens_in":547,"tokens_out":1210,"duration_ms":12116,"temperature":1.0,"reasoning_tokens":1135,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:21:37.371128+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a clean-data-only control that matches each toxic run's total token count (for example, by adding more C4 or repeated C4 samples), then repeat the probe-accuracy comparison and the ITI steering sweep; if the matched clean controls reproduce the lower entanglement and the smile-shaped toxicity curve, the proposed toxic-data benefit would be explained away as a training-token effect.","supporting_citations":[{"cited_title":"Raiders of the lost kek: 3.5 years of augmented 4chan posts from the politically incorrect board","cited_arxiv_id":null,"evidence_quote":"Provides the 4chan data used as the toxic pretraining corpus in the controlled mixes."}],"review_version":1}