REVIEW 4 major objections 5 minor 27 references
Subliminal Learning is Non-Semantic Distillation
T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read Subliminal learning is non-semantic: teacher biases pass to students through spurious weight-level structure, and adding Gaussian noise to the weights makes the transfer stronger.
desk verdict Valuable mechanistic findings on subliminal learning, but the headline noise experiment doesn't fully support the 'non-semantic' title claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the subliminal distribution shift — the measurable difference between what a biased teacher and an unbiased parent output when both are asked to produce lists of integers. The load-bearing lever is Gaussian weight noising, $W \leftarrow W + \sigma \, \mathrm{std}(W) \, \varepsilon$ with $\varepsilon_{ij} \sim \mathcal{N}(0,1)$, which enlarges the shift if spurious structure mediates it and would shrink it if semantic associations did. The paper then uses residual-stream steering vectors (mean-centered activation differences), LoRA contribution vectors compared by cosine similarity, and mean-gradient alignment to trace where and how the bias is encoded.
What would settle it
Run the noise experiment with independent noise draws for teacher and student (or noise applied only to the teacher); if the transfer increase disappears when the teacher alone is noised and the student stays clean, the claim that teacher weight noise carries the non-semantic signal is wrong.
Extended reading notes
Core claim
Subliminal learning is non-semantic distillation: the teacher's bias enters the student through a distribution shift in generated numbers that is caused by spurious, noise-like connections in the weights shared between teacher and student, not by any meaningful association between, say, 'owls' and particular integers. The paper establishes this by showing that adding Gaussian noise to the weights of both models strengthens the transfer by a factor of 1.9 in Gemma and 1.3 in Llama, and that the student's LoRA contribution in the residual stream closely imitates the exact steering vector used on a steered teacher, while no such pattern appears for prompted teachers. It further shows that steering vectors trained on steered data reproduce the bias, whereas those trained on prompted data fail, and that mean gradients of the parent model on steered data correlate linearly with the teacher's steering vector.
Load-bearing premise
The noise experiment applies the same Gaussian noise to the copy used as both teacher and student, and the larger preference shift is read as amplified non-semantic signal; if the gain instead comes from the noised student being easier to move or the noised teacher emitting more stereotyped lists, the causal conclusion does not follow.
Editorial extensions
If this is right
- Adding noise to teacher and student weights strengthens subliminal transfer, indicating that spurious, non-semantic structure in shared weights is a causal carrier of the bias.
- Students do not just inherit the semantic bias; they inherit the intervention type (steering vs. prompting) at the level of individual activation patterns.
- Steering vectors trained from steered subliminal data transfer the bias, while steering vectors trained from prompted data fail, confirming activation-level imitation.
- Mean gradients of the parent on steered data give a detectable, linear signal of the teacher's steering vector, offering a partial auditing method; activations do not.
- The teacher-as-random-network account explains why shared initialization is required and why small dataset mutations such as shuffling or paraphrasing destroy the latent signal.
Reading between the lines
- If non-semantic shared structure is the carrier, then models that share pretrained weights but diverge during RLHF or fine-tuning might stop transferring each other's subliminal biases; this is a testable consequence the paper does not run.
- The gradient-correlation result suggests a practical screening test: compute mean gradients on suspect data and compare against a library of candidate steering directions, but this would miss prompted attacks, which show no gradient or activation signal.
- The mechanistic split between steered and prompted students implies that auditing or defence methods validated on one intervention type may be ineffective on the other, so evaluations should cover both.
- If noise amplification is symmetric, tuning noise magnitude could become a deliberate attack amplifier, making adversarial subliminal poisoning easier rather than harder; the paper does not explore this risk.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies subliminal learning (SL) in Gemma-2B-Instruct and Llama-3.1-8B-Instruct, where a bias such as 'loves owls' is transferred from a teacher to an initially identical student through seemingly unrelated integer-list completions. It reports three main contributions: (i) adding Gaussian noise to the weights of a copy used as both teacher and student increases the measured preference transfer by 1.9x in Gemma and 1.3x in Llama, which the paper interprets as evidence that SL is mediated by non-semantic weight structure; (ii) steering vectors can serve as the teacher intervention, and students trained on steered data show LoRA activation contributions aligned with the teacher's steering vector while prompted students do not; and (iii) mean gradients of the parent model on steered subliminal data correlate linearly with the teacher's steering vector, suggesting a possible auditing signal. The paper frames the teacher as a 'random network' mapping interventions to subliminal datasets.
Significance. If the noise result were clean, this would be a valuable contribution to the emerging SL literature: it would provide a direct weight-level manipulation isolating non-semantic mediation, and the gradient-auditing result offers a falsifiable, practically motivated diagnostic. The paper is also commendable for releasing code, testing two model families, reporting 10-seed confidence intervals for the noise experiment, and for the novel steered-versus-prompted comparison, which is partly independent of the noise claim. However, the headline non-semantic conclusion currently rests on a confounded protocol, and the 'type of intervention' claim lacks the necessary baseline; both points need rework before the central claims can be accepted.
major comments (4)
- [Section 3, Eq. (2)] The central claim that 'non-semantic weight structures play a crucial role' is not established by the reported experiment, because the same noised copy is used as both teacher and student. The 1.9x/1.3x increase could equally arise from (i) a noised student being easier to move, e.g. because noise weakens its priors or reduces effective capacity, (ii) a noised teacher emitting more degenerate, repetitive, or narrowly distributed lists that are trivially learnable, or (iii) shared random idiosyncrasy of the common noised initialization. None of these isolates 'structure in the teacher's weights' as the mediator. Please add ablations with a noised teacher and clean student, a clean teacher and noised student, and independent noise realizations for teacher and student, and report transfer sizes for each; the abstract and findings should be revised to match the outcome.
- [Section 3, sigma selection] The noise scale was selected as the strongest level at which models 'remained coherent enough to follow instructions' (sigma = 0.10 for Gemma and 0.15 for Llama). This selection criterion is a confound: if noising degrades instruction following, the generated lists may become more repetitive or narrower in vocabulary, and the increased transfer could reflect output statistics rather than non-semantic weight structure. Please report distributional statistics of the noised versus clean teacher outputs (e.g., repetition rate, unique tokens, token entropy, format compliance) and show that the effect persists when these statistics are controlled for or across a sigma sweep.
- [Section 4.1, Eq. (3), Figure 4] The conclusion that 'students inherit ... the type of intervention' is supported only by comparing steered students' LoRA contributions with the ground-truth steering vector va and observing that prompted students do not align with va. But a prompted student would not be expected to align with va under any account, including the paper's own random-network model. To demonstrate that prompted students fail to imitate the prompt intervention, the LoRA contribution should be compared against the actual prompt-induced activation difference (e.g., the mean residual-stream difference between prompted and unprompted teacher on the same prompts). The steering-vector training experiment in Figure 5 partially addresses this by showing prompted data cannot be compressed into a rank-1 vector, but it does not test whether a full LoRA's activation pattern is prompt-like.
- [Section 3, Figure 3] The 1.9x and 1.3x factors are reported without a formal significance test or a confidence interval on the ratio itself; the per-animal variance is high, and only 8 animals are evaluated per setup, as the Limitations acknowledge. Please report per-animal ratios, a paired test across animals, and a confidence interval for the multiplicative effect, so readers can assess whether the effect is robust rather than driven by one or two animals.
minor comments (5)
- [Figure 2] The caption should state whether error bars are plotted; as shown, the figure appears to show single runs per animal, making it hard to assess variance in the baseline transfer results.
- [Abstract and Section 3] The abstract says the noise result 'suggest[s] that non-semantic weight structures play a crucial role,' while Section 3 and the Limitations concede that causality is uncertain and only one noise variant was tested; please align the abstract and the Findings paragraph with these caveats.
- [Section 4.2, Appendix G] The 'promise for data auditing' conclusion should be tempered by the facts that the Llama gradient correlation is weaker, gradient steering fails for Llama, and the method requires knowing the ground-truth vector va in advance; the paper notes the last point, but the abstract's wording is stronger than the evidence.
- [Section 3, Results] The '1.9x' and '1.3x' factors should be defined precisely (ratio of means versus mean of per-animal ratios) and accompanied by a confidence interval.
- [Appendix C] The claim that noise-induced preference changes are 'statistically insignificant' is made without reporting a test; please add the test or rephrase the statement.
Circularity Check
No significant circularity: the empirical claims are tested against controls and independently constructed probes; the sole self-referential probe choice does not reduce to a tautology.
full rationale
The paper's derivation chain is not circular. The central claims are empirically tested, not derived from their own definitions. (1) The noise experiment (Section 3, Eq. 2) compares noised vs un-noised subliminal transfer; the reported 1.9x (Gemma) and 1.3x (Llama) increases are observed effects against an un-noised baseline, not quantities fitted to match the conclusion. The protocol does use the same noised copy as both teacher and student, which is a confound, but a confounded experiment is not a circular derivation; the paper's Limitations section (Section 5) explicitly concedes 'it is not certain that the relationship is causal, nor that noise is the driving factor,' so the paper does not present the result as a logical necessity. (2) The steering-vector analyses (Section 4.1, Eq. 3) measure the student's LoRA contribution against the independently constructed steering vector v_a from Eq. (1); the observed cosine-similarity peak is an empirical finding, not a consequence of data generation by construction. (3) The gradient-audit result (Section 4.2) compares mean gradients to v_a and finds a diagonal correlation, while activation differences show no such correlation; this contrast gives the finding independent content. The only self-referential aspect is that the same v_a used to generate steered data is also used as the probe, but that is a probe choice rather than a fitted parameter or definitional identity. No load-bearing self-citations appear: the cited prior work (Cloud et al. 2025, Schrodi et al. 2026, Minder et al. 2026, Wang et al. 2025) is by other authors and serves as external background, not as a justification that presupposes this paper's conclusions. The limitations passage is appropriately candid and weighs against any claim of forced circularity.
Assumptions & free parameters
free parameters (4)
- Gaussian noise scale sigma (teacher and student weights) =
0.10 (Gemma), 0.15 (Llama)
- Steering layer index =
14 (Gemma), 21 (Llama)
- Steering strength =
8
- LoRA training hyperparameters (rank, alpha, learning rate, epochs, batch size) =
rank 8, alpha 8, lr 1e-4, epochs 1-3, batch 8-12
assumptions (4)
- domain assumption Adding Gaussian noise to weights amplifies non-semantic signal more than it destroys semantic signal.
- domain assumption The same noised model can serve as both teacher and student without changing the mechanism of SL.
- domain assumption LoRA adapter contributions can stand in for full finetune activation changes when comparing students to steering vectors.
- domain assumption Random network theory from Poole et al. applies to trained language models treated as noisy maps from interventions to outputs.
Cite this review
Pith. "Pith review of Subliminal Learning is Non-Semantic Distillation." pith.science (2026). https://pith.science/paper/4NS6QVZN
@misc{pith2026260805734,
author = {Pith},
title = {Pith review of: Subliminal Learning is Non-Semantic Distillation},
year = {2026},
howpublished = {\url{https://pith.science/paper/4NS6QVZN}},
note = {Machine review of arXiv:2608.05734}
}
read the original abstract
Subliminal Learning (SL) is a surprising type of generalization displayed by modern language models. It allows the transfer of a bias or behavior from a teacher model to a student by distilling from seemingly unrelated or random synthetic data from the teacher. This presents challenges in ensuring AI systems remain predictable and are trained safely, as standard auditing of the input data would not catch the hidden subliminal signal. Here, we investigate several open questions as to the enabling mechanisms and drivers of SL. First is the nature of the process by which biases are encoded in the data. We find that by adding Gaussian noise to the weights of the teacher and student models, the magnitude of subliminal transfer is increased by a factor of 1.9 in Gemma and 1.3 in Llama, suggesting that non-semantic weight structures play a crucial role. We show that steering vectors can be applied to the teacher to produce subliminal data, in addition to prompting and finetuning as used in previous studies. Analysis of the activations of the student models that have been trained on steered and prompted data demonstrates that students inherit not just the semantic meaning of the teacher's bias, but also the type of intervention that was used to apply it: steered students imitate steering vectors, prompted students do not. Additionally, the gradients of steered subliminal data show a linear correlation with the teacher's steering vectors, showing promise for data auditing. More broadly, as synthetic data becomes central to frontier training pipelines, being able to see the latent signals hidden in training data becomes paramount.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
2025 , eprint=
Subliminal Learning: Language models transmit behavioral traits via hidden signals in data , author=. 2025 , eprint=
2025
-
[2]
Chunky Post-Training: Data Driven Failures of Generalization , author=. 2026 , eprint=
work page 2026
-
[3]
Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data , author=. 2024 , eprint=
work page 2024
-
[4]
Simple Mechanistic Explanations for Out-Of-Context Reasoning
Simple mechanistic explanations for out-of-context reasoning , author=. arXiv preprint arXiv:2507.08218 , year=
-
[5]
Towards Understanding Subliminal Learning: When and How Hidden Biases Transfer , author=. 2026 , eprint=
work page 2026
-
[6]
Subliminal Effects in Your Data: A General Mechanism via Log-Linearity , author=. 2026 , eprint=
work page 2026
-
[7]
arXiv preprint arXiv:2504.02922 , year=
Overcoming sparsity artifacts in crosscoders to interpret chat-tuning , author=. arXiv preprint arXiv:2504.02922 , year=
-
[8]
Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences , author=. 2026 , eprint=
work page 2026
Show all 27 references
-
[9]
Mechanistic Interpretability Workshop at NeurIPS 2025 , year=
Token Entanglement in Subliminal Learning , author=. Mechanistic Interpretability Workshop at NeurIPS 2025 , year=
2025
-
[10]
Grattafiori, Aaron and Dubey, Abhimanyu and Jauhri, Abhinav and Pandey, Abhinav and Kadian, Abhishek and Al-Dahle, Ahmad and Letman, Aiesha and Mathur, Akhil and Schelten, Alan and Vaughan, Alex and others , journal=. The
-
[11]
arXiv preprint arXiv:2403.08295 , year=
Gemma: Open Models Based on. arXiv preprint arXiv:2403.08295 , year=
-
[12]
arXiv preprint arXiv:2512.09742 , year=
Weird generalization and inductive backdoors: New ways to corrupt llms , author=. arXiv preprint arXiv:2512.09742 , year=
-
[13]
arXiv preprint arXiv:2512.10092 , year=
Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit , author=. arXiv preprint arXiv:2512.10092 , year=
-
[14]
arXiv preprint arXiv:2507.16795 , year=
Steering out-of-distribution generalization with concept ablation fine-tuning , author=. arXiv preprint arXiv:2507.16795 , year=
-
[15]
2026 , month = feb, url =
McGrath, Thomas , title =. 2026 , month = feb, url =
2026
-
[16]
2022 , journal =
Toy Models of Superposition , author =. 2022 , journal =
2022
-
[17]
arXiv preprint arXiv:2312.03813 , year=
Improving Activation Steering in Language Models with Mean-Centring , author=. arXiv preprint arXiv:2312.03813 , year=
-
[18]
arXiv preprint arXiv:2308.10248 , year=
Steering language models with activation engineering , author=. arXiv preprint arXiv:2308.10248 , year=
-
[19]
Findings of the Association for Computational Linguistics: ACL 2022 , pages=
Extracting Latent Steering Vectors from Pretrained Language Models , author=. Findings of the Association for Computational Linguistics: ACL 2022 , pages=
2022
-
[20]
2021 , eprint=
LoRA: Low-Rank Adaptation of Large Language Models , author=. 2021 , eprint=
2021
-
[21]
2020 , eprint=
HuggingFace's Transformers: State-of-the-art Natural Language Processing , author=. 2020 , eprint=
2020
-
[22]
2020 , howpublished =
nostalgebraist , title =. 2020 , howpublished =
2020
-
[23]
2015 , eprint=
Distilling the Knowledge in a Neural Network , author=. 2015 , eprint=
2015
-
[24]
arXiv preprint arXiv:2209.15189 , year=
Learning by distilling context , author=. arXiv preprint arXiv:2209.15189 , year=
-
[25]
arXiv preprint arXiv:2112.00861 , year=
A general language assistant as a laboratory for alignment , author=. arXiv preprint arXiv:2112.00861 , year=
-
[26]
2020 , eprint=
Linear Mode Connectivity and the Lottery Ticket Hypothesis , author=. 2020 , eprint=
2020
-
[27]
2016 , eprint=
Exponential expressivity in deep neural networks through transient chaos , author=. 2016 , eprint=
2016
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.