Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

On Linear Representations and Pretraining Data Frequency in Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read How often subjects and objects co-occur in pretraining largely determines whether a language model forms a linear representation of a factual relation, and that structure can reveal hidden training-data frequencies.

desk verdict A credible, useful empirical link between LRE quality and pretraining co-occurrence frequency—but the headline thresholds are post hoc fits to 24 relations and the arXiv text is unfinished. read the letter →

arxiv 2504.12459 v1 pith:NMECJJPH submitted 2025-04-16 cs.CL cs.AI

Jack Merullo , Noah A. Smith , Sarah Wiegreffe , Yanai Elazar This is my paper · ORCID
classification cs.CLcs.AI
keywords linearrepresentationspretrainingdatafrequencylanguagemodelinterpretabilityRelationalEmbeddingssubject-objectco-occurrencetraininginferencein-contextlearningfactualrelations
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that whether a language model encodes a factual relation linearly is largely determined by how often the subject and object of that relation appear together in the pretraining corpus. It reports strong correlations (r = 0.82) between co-occurrence counts and the quality of Linear Relational Embeddings, with thresholds around 1,000 co-occurrences for OLMo-7B, 2,000 for GPT-J, and 4,000 for OLMo-1B, above which linear structure consistently forms. The same relationship holds at every pretraining checkpoint, so total exposure matters more than when the exposure happened. It then reverses the direction of the link: a regression trained on linear-representation quality can estimate how often a term appeared in pretraining data, and this regression transfers from OLMo-7B to GPT-J, suggesting that closed-data models leak frequency information through their linear structure. If this is right, linear-representation quality becomes a predictable, corpus-driven property rather than an unexplained emergent one.

What carries the argument

The load-bearing object is the Linear Relational Embedding (LRE), an affine map W and bias b fitted to approximate the model's computation F(s, c) = o from a subject representation s to an object representation o in a few-shot context, with W estimated as the expected Jacobian of the model with respect to s across examples. The paper measures LRE quality with causality (whether editing the subject representation changes the prediction to another object of the same relation) and faithfulness (whether the LRE reproduces the model's own prediction), and pairs those measurements with exact co-occurrence counts obtained by searching tokenized training batches with a released counting tool. The argument is carried by correlating the two: co-occurrence frequency predicts causality at every checkpoint, and causality then predicts frequency in the regression analyses.

What would settle it

Re-run the main OLMo-7B analysis with co-occurrence counts restricted to sequences whose templates express the relation (e.g., 'X plays the Y'), then check whether the roughly 2,000-count threshold and the r = 0.82 correlation with causality survive; if they vanish, the reported frequency–linearity link is an artifact of the co-occurrence proxy.

Watch

Extended reading notes

Core claim

Across the 25 factual relations studied in OLMo-7B, OLMo-1B, and GPT-J, the quality of linear representations is strongly correlated with how often subjects and objects co-occur in pretraining data (Pearson r = 0.82). For each model there is a consistent co-occurrence threshold above which average causality exceeds 0.9: about 1,998 for OLMo-7B, 1,097 for GPT-J, and 4,447 for OLMo-1B, and this threshold holds regardless of which pretraining stage the co-occurrences fall in. The same linear-representation metrics, fed into a regression model, predict how often a term appeared in pretraining more accurately than log-probability features alone, and a regression fit on one model transfers to another model with different training data. The paper concludes that linear representational structure contains signal about the pretraining corpus and that frequency is a primary determinant of when such structure forms.

Load-bearing premise

All frequency counts treat a subject and object appearing in the same tokenized training sequence as a mention of the factual relation, so if most co-occurrences are incidental, the thresholds and the frequency regressions measure the counting proxy rather than actual knowledge of the fact.

Editorial extensions

If this is right

  • A relation whose average subject-object co-occurrence exceeds the model-specific threshold will almost always show a high-quality linear representation, even at very early pretraining checkpoints; the exact training stage does not matter.
  • Causality correlates with subject-object co-occurrence (r = 0.82) more strongly than with subject frequency alone (r = 0.66) or object frequency alone (r = 0.59), making co-occurrence the operative statistic for linear-structure formation.
  • Regression models using LRE metrics predict how often an object appeared in pretraining within one order of magnitude about 70% of the time, a level the paper shows log-probability features alone do not reach.
  • A regression fit on one model's LREs transfers to another model trained on different data, providing a way to estimate term frequencies in open-weight models whose pretraining data is not public.
  • High few-shot accuracy can coexist with weak linear structure for low-frequency relations, so accuracy and linearity are related but not identical properties of a factual relation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested corollary is that a low causality score for a relation may simply mean the subject-object pair was undertrained, not that the model stores that fact non-linearly; frequency-normalized interpretability comparisons would settle this.
  • The thresholds suggest a data-curation recipe the paper does not causally test: deliberately adding subject-object co-occurrences to pretraining data might create or strengthen steerable linear directions, and a controlled repetition experiment would be the direct test.
  • If the cross-model transfer is a general phenomenon, term-frequency estimation could eventually apply to entirely closed-weight models with no released data at all, though the paper only demonstrates transfer between two open-weight models.
  • Counting relation mentions by templated contexts rather than raw co-occurrence would sharpen or shift the reported thresholds, making template-filtered counts the natural next experiment.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper investigates whether the formation of linear representations of factual relations in transformer LMs is predictable from pretraining corpus statistics. Using Linear Relational Embeddings (LREs) and the causality metric on 24 factual relations, the authors report a strong correlation (r = 0.82) between subject-object co-occurrence frequency in pretraining data and LRE quality in OLMo-7B, OLMo-1B, and GPT-J. They further claim that LREs consistently form above co-occurrence thresholds of roughly 1k, 2k, and 4k for GPT-J, OLMo-7B, and OLMo-1B respectively, regardless of when in pretraining the threshold is reached. Finally, they train a random-forest regression on LRE features to predict term frequencies from linear representation quality and report that the regression transfers across models with different pretraining corpora. The paper also releases the Batch Search counting tool.

Significance. If the central claims hold, this is a valuable empirical contribution: it connects interpretability findings about linear representations to pretraining data statistics and suggests a new, representation-based signal for inferring properties of closed pretraining corpora. The paper has concrete strengths: it uses external corpus counts rather than constructing counts from model probabilities, it studies multiple checkpoints across pretraining, and the cross-model regression is evaluated on held-out relations in a genuinely different setting. The released counting tool is a useful artifact. However, the headline threshold numbers are estimated post hoc from the same scatterplots they describe, with a small number of relations and autocorrelated checkpoint observations, and the frequency proxy is noisy. These issues make the strong version of the threshold claim conditional rather than established.

major comments (4)
  1. [Sec. 4.2, Fig. 2] The threshold values (GPT-J 1,097; OLMo-7B 1,998; OLMo-1B 4,447) are read off the same scatterplots that are used to assert them: the text defines the threshold as the co-occurrence count above which average causality exceeds 0.9, with no confidence intervals, no leave-one-relation-out analysis, and no held-out validation. With only 24 relations, a single low-frequency, low-causality relation such as star-constellation-name can move the crossing point substantially, so the abstract's claim that linear representations 'consistently ... form when ... co-occur at least 1k and 2k times' is not supported by the reported evidence. Please add bootstrap or leave-one-relation-out estimates of the threshold and report their variability; if the threshold does not transfer to held-out relations, the strong claim should be weakened to the correlation statement.
  2. [Sec. 3.2] The load-bearing definition of a relation occurrence is 'any time two terms appear in the same sequence within a batch', which is used both for the threshold finding and for the frequency-prediction regression. The paper cites Elsahar et al. (2018) for this proxy but provides no validation within this study that same-sequence co-occurrence corresponds to a textual mention of the factual relation. Co-occurrences in lists or unrelated contexts are counted as relation occurrences, so the causal interpretation that 'frequency of factual relation mentions' drives LRE formation is conflated with generic co-occurrence frequency. Please quantify the share of counted co-occurrences that are true relation mentions (e.g., by template or dependency matching) and rerun the central analyses on the relation-mention subset; the threshold numbers may change.
  3. [Sec. 4.1, Sec. 4.2] The reported Pearson correlation r = 0.82 pools observations across eight OLMo checkpoints per relation, but these checkpoints are not independent: they share the same architecture, the same corpus, and sequential training steps. Treating each (relation, checkpoint) pair as an independent observation inflates the effective sample size for both the correlation and the threshold estimation. Please report within-relation correlations, cluster the analysis by checkpoint or by relation, and include a non-parametric correlation (e.g., Spearman) to show that the headline association is not an artifact of pooling autocorrelated points.
  4. [Sec. 5.3, Table 1] The cross-model frequency prediction relies on scaling GPT-J counts by 'the ratio of total tokens trained between the two models', but the paper does not state how this ratio is computed, whether it is known a priori, or whether it was fitted to improve the results. Because the applied claim about predicting counts of closed-data models depends on this scaling, please specify the scaling, justify it independently of the evaluation data, and test sensitivity to the chosen ratio. In addition, Table 1 shows that for subject-object co-occurrence prediction evaluated on OLMo, LRE features (0.68) are essentially at the mean-frequency baseline (0.67); the paper's summary that LREs 'encode a rich signal' should be qualified for this setting.
minor comments (5)
  1. [Throughout] The manuscript contains unfinished editorial markers that should be removed before publication: 'TODO: What to do about awkward spacing' in the Figure 2 caption, 'TODO: I was going to define causality and faithfulness here' in Section 3, and 'TODO: make more concise, fill in other sections' in the Related Work heading. The main text also never actually defines causality and faithfulness despite promising to do so; please add the definitions or point to a precise appendix location.
  2. [Fig. 2 caption] The caption text includes a stray incomplete sentence ('We highlight Even at very early training steps...'), and the figure label 'Frequency v . Efficacy' contains typographical errors. Please proofread the figure and caption carefully.
  3. [Sec. 5.1] The sentence 'We fit 24 models such that each relation is held out once per random seed across 4 seeds' is ambiguous: if there are 24 relations and 4 seeds, the reader would expect 96 fits, not 24. Please clarify the exact protocol.
  4. [Appendix F] The figure captions label the accuracy columns 'N Shot' instead of '5 Shot' in several places; please correct the label and explain whether 'N' denotes the number of demonstrations.
  5. [Eq. (1)] The notation in Equation (1) is not self-contained: the reader is told that F(s, c) = o is a forward pass, but the symbols s, c, o, and the Jacobian are not defined precisely enough to reproduce the computation without the Hernandez et al. paper. Please add a short formal definition of these objects.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: corpus co-occurrence counts and LRE metrics are measured independently, and the regression is evaluated on held-out relations and cross-model transfer.

full rationale

The paper's derivation chain is not circular. The central empirical link is between LRE causality/faithfulness, computed from model activations using the Hernandez et al. (2024) method, and subject-object co-occurrence counts, measured independently from the pretraining corpora (Dolma via the authors' Batch Search tool, and the Pile via WIMBD). These are distinct quantities: one is a corpus statistic, the other is a function of model weights. No equation defines LRE quality in terms of co-occurrence counts or vice versa. The Section 4.2 thresholds (GPT-J 1,097, OLMo-7B 1,998, OLMo-1B 4,447) are post hoc descriptive summaries of the scatterplots, defined operationally as the counts above which the mean causality exceeds a fixed 0.9 bar; they are not used as a held-out prediction, so any concern about their generality is a statistical validity issue rather than a circularity. The Section 5 regression predicts external corpus counts from LRE features with leave-one-relation-out evaluation and cross-model transfer, so the prediction target is external to the model features. Self-citations (Elazar et al. 2022 for the co-occurrence proxy, and Elazar et al. 2024/WIMBD for counting) are not load-bearing: the proxy is independently attributed to Elsahar et al. (2018), and WIMBD counts are validated against the Batch Search counts with slope 0.94 and r = 0.99. No uniqueness theorem or ansatz is imported from the authors' prior work. The reported correlations and regression results therefore stand as independent empirical measurements rather than reductions to their own inputs.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The empirical claims rest on corpus counters (BatchSearch and WIMBD), the LRE metric family from Hernandez et al. (2024), and hand-chosen thresholds. No new theoretical entities are introduced. The main uncharged costs are the identification of sequence co-occurrence with relation mentions, fidelity of OLMo batch reconstruction, and the post-hoc fitting of the 0.9 causality bar and the per-model co-occurrence cutoffs.

free parameters (5)
  • Per-model co-occurrence thresholds for LRE formation = GPT-J: 1,097; OLMo-7B: 1,998; OLMo-1B: 4,447
    Computed as the co-occurrence count above which mean causality exceeds 0.9 in the same scatterplots the claim is based on (Section 4.2, Figure 2 table); not validated on held-out relations.
  • Causality quality threshold = 0.9
    Chosen by hand as the definition of "nearly perfectly linear" (Section 4.2); this choice determines where the fitted co-occurrence cutoffs land.
  • LRE hyperparameters per relation and checkpoint (layer, beta, rank) = Per-relation values from sweeps in Appendix C
    Layer and rank for causality and beta for faithfulness are selected per relation on the same checkpoints used for the main analysis, which risks overfitting the causality metric.
  • Cross-model token-count scaling ratio = Not stated
    In Section 5.3, GPT-J features are scaled by the ratio of total tokens trained between the two models when transferring the regression; this constant is not reported and acts as an unmeasured fitted factor.
  • Random forest regression hyperparameters = 100 trees, default settings
    Standard choices for the frequency-prediction regressions (Section 5.1); they affect the reported transfer accuracies but are not central to the main frequency-linearity correlation.
assumptions (4)
  • domain assumption Subject-object co-occurrence in a sequence is an accurate proxy for mentions of the factual relation.
    Adopted from Elsahar et al. (2018) and Elazar et al. (2022) in Section 3.2; the authors verify counts only against WIMBD (Appendix D), not against relation-level ground truth. The commonsense-relation analysis in Appendix G shows the proxy is weaker outside factual relations.
  • domain assumption The LRE first-order Taylor approximation F(s,c) approximately equal to W s + b faithfully captures how the LM decodes relation objects.
    Taken from Hernandez et al. (2024), Section 2.1; the paper inherits this operationalization of 'linear representation' without re-deriving it.
  • domain assumption For OLMo, the provided tooling accurately recreates the exact pretraining batches from Dolma.
    Section 4.1: 'The OLMo family of models provides tools for accurately recreating the batches from Dolma'; the co-occurrence counts are only as accurate as this batch reconstruction.
  • domain assumption Dolma and the Pile, as reconstructed, are the actual pretraining data of OLMo and GPT-J.
    Section 3 identifies these as 'their corresponding datasets'; GPT-J counts use WIMBD over the entire Pile, which may differ from GPT-J's effective training order and sequence segmentation (the paper notes WIMBD overestimates co-occurrences with slope 0.94 in Appendix D).

how reviews work

0 comments
Cite this review

Pith. "Pith review of On Linear Representations and Pretraining Data Frequency in Language Models." pith.science (2026). https://pith.science/paper/NMECJJPH

@misc{pith2026250412459,
  author       = {Pith},
  title        = {Pith review of: On Linear Representations and Pretraining Data Frequency in Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NMECJJPH}},
  note         = {Machine review of arXiv:2504.12459}
}
read the original abstract

Pretraining data has a direct impact on the behaviors and quality of language models (LMs), but we only understand the most basic principles of this relationship. While most work focuses on pretraining data's effect on downstream task behavior, we investigate its relationship to LM representations. Previous work has discovered that, in language models, some concepts are encoded `linearly' in the representations, but what factors cause these representations to form? We study the connection between pretraining data frequency and models' linear representations of factual relations. We find evidence that the formation of linear representations is strongly connected to pretraining term frequencies; specifically for subject-relation-object fact triplets, both subject-object co-occurrence frequency and in-context learning accuracy for the relation are highly correlated with linear representations. This is the case across all phases of pretraining. In OLMo-7B and GPT-J, we discover that a linear representation consistently (but not exclusively) forms when the subjects and objects within a relation co-occur at least 1k and 2k times, respectively, regardless of when these occurrences happen during pretraining. Finally, we train a regression model on measurements of linear representation quality in fully-trained LMs that can predict how often a term was seen in pretraining. Our model achieves low error even on inputs from a different model with a different pretraining dataset, providing a new method for estimating properties of the otherwise-unknown training data of closed-data models. We conclude that the strength of linear representations in LMs contains signal about the models' pretraining corpora that may provide new avenues for controlling and improving model behavior: particularly, manipulating the models' training data to meet specific frequency thresholds.

Figures

Figures reproduced from arXiv: 2504.12459 by the authors.

Figure 1
Figure 1. Overview of this work. Given a dataset of subject-relation-object factual relation triplets, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 4
Figure 4. Average Causality and Faithfulness results across relations depending on if the LRE was [PITH_FULL_IMAGE:figures/full_fig_p017_4.png] view at source ↗
Figure 5
Figure 5. Causality and Faithfulness results for each relation depending on if the LRE was fit with [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figures from the paper (10 more)
Figure 6
Figure 6. Figure 6: OLMo 0424 7B per layer faithfulness scores as a function of the choice of layer at which [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: OLMo 0424 7B per layer causality scores as a function of the choice of layer at which to [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: OLMo 0424 7B LRE Beta hyperparameter sweep at highest performing layer. [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: OLMo 0424 7B LRE Rank hyperparameter sweep at highest performing layer. [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: Comparison between WIMBD and Batch Search subject-object co-occurrences [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: Correlations between each feature in our regression analysis. Because of the high cor [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: Hard causality is by far the most important feature for generalizing to new relations when [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]
Figure 13
Figure 13. Figure 13: Zero shot, 5-shot accuracies against causality for each relation across training time in [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]
Figure 14
Figure 14. Figure 14: Zero shot, 5-shot accuracies against causality for each relation across training time in 2 :25000 [PITH_FULL_IMAGE:figures/full_fig_p025_14.png]
Figure 15
Figure 15. Figure 15: Commonsense relations compared to pretraining time in OLMo-7B. [PITH_FULL_IMAGE:figures/full_fig_p025_15.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Bridging Compute- and Data-Optimal Pretraining

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Pretraining loss obeys a single law in which repeated or paraphrased tokens count as η(N, data-per-parameter, expansion-ratio) fresh tokens, with total effective data saturating as derived tokens grow.

  2. Graded Entity-Familiarity Readouts in Language Models: Polish Adaptation, Cross-Language Robustness, and Refusal Steering

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Prompt-point activations carry a graded, steerable entity-familiarity signal that is robust to Polish/English stem changes and is stronger in Polish-adapted models than in base models.

Reference graph

Works this paper leans on

62 extracted references · 19 canonical work pages · cited by 2 Pith papers

  1. [1]

    To code or not to code? exploring impact of code in pre-training

    Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr Locatelli, Marzieh Fadaee, Ahmet \"U st \"u n, and Sara Hooker. To code or not to code? exploring impact of code in pre-training. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=zSfeN1uAcx

    work page 2025

  2. [2]

    Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers

    Giuseppe Ateniese, Luigi V Mancini, Angelo Spognardi, Antonio Villani, Domenico Vitali, and Giovanni Felici. Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers. International Journal of Security and Networks, 10 0 (3): 0 137--150, 2015. URL https://dl.acm.org/doi/10.1504/IJSN.2015.071829

    arXiv 2015

  3. [3]

    Interpreting neural networks through the polytope lens, 2022

    Sid Black, Lee Sharkey, Leo Grinsztajn, Eric Winsor, Dan Braun, Jacob Merizian, Kip Parker, Carlos Ramón Guevara, Beren Millidge, Gabriel Alfour, and Connor Leahy. Interpreting neural networks through the polytope lens, 2022. URL https://arxiv.org/abs/2211.12312

    arXiv 2022

  4. [4]

    Membership inference attacks from first principles

    Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramèr. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pp.\ 1897--1914, 2022. URL https://ieeexplore.ieee.org/document/9833649/

    work page arXiv 2022

  5. [5]

    Quantifying memorization across neural language models

    Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=TatRHT_1cK

    2023

  6. [6]

    How do large language models acquire factual knowledge during pretraining? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    Hoyeon Chang, Jinho Park, Seonghyeon Ye, Sohee Yang, Youngkyung Seo, Du-Seong Chang, and Minjoon Seo. How do large language models acquire factual knowledge during pretraining? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=TYdzj1EvBP

    work page 2024

  7. [7]

    Identifying Linear Relational Concepts in Large Language Models

    David Chanin, Anthony Hunter, and Oana-Maria Camburu. Identifying Linear Relational Concepts in Large Language Models . In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics : Human Language Technologies ( Volume 1: Long Papers ) , pp.\ 1524-...

    doi:10.18653/v1/2024.naacl-long.85 2024

  8. [8]

    Recurrent neural networks learn to store and generate sequences using non-linear representations

    R \'o bert Csord \'a s, Christopher Potts, Christopher D Manning, and Atticus Geiger. Recurrent neural networks learn to store and generate sequences using non-linear representations. In Yonatan Belinkov, Najoung Kim, Jaap Jumelet, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen (eds.), Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpretin...

    work page doi:10.18653/v1/2024.bl 2024

Show all 62 references
  1. [9]

    Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals

    Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals . Transactions of the Association for Computational Linguistics, 9: 0 160--175, 03 2021. URL https://doi.org/10.1162/tacl\_a\_00359

    2021 doi

  2. [10]

    Measuring causal effects of data statistics on language model's `factual' predictions

    Yanai Elazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Marius Mosbach, Yonatan Belinkov, Hinrich Sch \"u tze, and Yoav Goldberg. Measuring causal effects of data statistics on language model's `factual' predictions. arXiv preprint arXiv:2207.14251, 202...

    2022 arXiv

  3. [11]

    Smith, and Jesse Dodge

    Yanai Elazar, Akshita Bhagia, Ian Helgi Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Evan Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hannaneh Hajishirzi, Noah A. Smith, and Jesse Dodge. What's in my big data? In The Twelfth International Confere...

    2024

  4. [12]

    A mathematical framework for transformer circuits

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. URL https://transformer-circuits.pub/2021/frame...

    2021

  5. [13]

    T - RE x: A large scale alignment of natural language with knowledge base triples

    Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. T - RE x: A large scale alignment of natural language with knowledge base triples. In Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Dec...

    2018

  6. [14]

    Towards Understanding Linear Word Analogies

    Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. Towards Understanding Linear Word Analogies . In Anna Korhonen, David Traum, and Llu\'is M\`arquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp.\ 3253--3262. Association...

    2019 doi

  7. [15]

    The pile: An 800gb dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. URL https://arxiv.org/abs/2101.00027

    2020 arXiv

  8. [16]

    Scaling and evaluating sparse autoencoders

    Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/for...

    2025

  9. [17]

    What can transformers learn in-context? a case study of simple function classes

    Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022...

    2022

  10. [18]

    Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn`t

    Anna Gladkova, Aleksandr Drozd, and Satoshi Matsuoka. Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn`t. In Jacob Andreas, Eunsol Choi, and Angeliki Lazaridou (eds.), Proceedings of the NAACL Student Research Work...

    2016 doi

  11. [19]

    OLM o: Accelerating the science of language models

    Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. OLM o: Accelerating the science of language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of th...

    2024 doi

  12. [20]

    Jonathan Hayase, Alisa Liu, Yejin Choi, Sewoong Oh, and Noah A. Smith. Data mixture inference: What do BPE tokenizers reveal about their training data? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=EHXyeImux0

    2024

  13. [21]

    In- Context Learning Creates Task Vectors

    Roee Hendel, Mor Geva, and Amir Globerson. In- Context Learning Creates Task Vectors . In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics : EMNLP 2023 , pp.\ 9318--9333. Association for Computational Linguistics, 2023...

    2023 doi

  14. [22]

    Linearity of relation decoding in transformer language models

    Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://op...

    2024

  15. [23]

    Sparse autoencoders find highly interpretable features in language models

    Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=F76bwRSLeK

    2024

  16. [24]

    On the origins of linear representations in large language models

    Yibo Jiang, Goutham Rajendran, Pradeep Kumar Ravikumar, Bryon Aragam, and Victor Veitch. On the origins of linear representations in large language models. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=otuTw4Mghk

    2024

  17. [25]

    Marzena Karpinska, Bofang Li, Anna Rogers, and Aleksandr Drozd. Subcharacter information in J apanese embeddings: When is it worth it? In Georgiana Dinu, Miguel Ballesteros, Avirup Sil, Sam Bowman, Wael Hamza, Anders Sogaard, Tahira Naseem, and Yoav Goldberg (eds.), Proceeding...

    2018 doi

  18. [26]

    Multilingual reliability and semantic structure of continuous word spaces

    Maximilian K \"o per, Christian Scheible, and Sabine Schulte im Walde. Multilingual reliability and semantic structure of continuous word spaces. In Matthew Purver, Mehrnoosh Sadrzadeh, and Matthew Stone (eds.), Proceedings of the 11th International Conference on Computational...

    2015

  19. [27]

    A pretrainer`s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity

    Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, and Daphne Ippolito. A pretrainer`s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. In Kev...

    2024

  20. [28]

    At which training stage does code data help LLM s reasoning? In The Twelfth International Conference on Learning Representations, 2024

    Yingwei Ma, Yue Liu, Yue Yu, Yuanliang Zhang, Yu Jiang, Changjian Wang, and Shanshan Li. At which training stage does code data help LLM s reasoning? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=KIPJKST4gw

    2024

  21. [29]

    When not to trust language models: Investigating effectiveness of parametric and non-parametric memories

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings o...

    2023 doi

  22. [30]

    Embers of autoregression show how large language models are shaped by the problem they are trained to solve

    R Thomas McCoy, Shunyu Yao, Dan Friedman, Mathew D Hardy, and Thomas L Griffiths. Embers of autoregression show how large language models are shaped by the problem they are trained to solve. Proceedings of the National Academy of Sciences, 121 0 (41): 0 e2322420121, 2024. URL ...

    2024 doi

  23. [31]

    Language models implement simple W ord2 V ec-style vector arithmetic

    Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Language models implement simple W ord2 V ec-style vector arithmetic. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational...

    2024 doi

  24. [32]

    Efficient estimation of word representations in vector space

    Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013 a . URL https://arxiv.org/abs/1301.3781

    2013 arXiv

  25. [33]

    Distributed representations of words and phrases and their compositionality

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Syst...

    2013

  26. [34]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference o...

    2022 doi

  27. [35]

    Zoom in: An introduction to circuits

    Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5 0 (3): 0 e00024--001, 2020. URL https://distill.pub/2020/circuits/zoom-in/

    2020

  28. [36]

    Chatterji, Faisal Ladhak, and Tatsunori Hashimoto

    Yonatan Oren, Nicole Meister, Niladri S. Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. Proving test set contamination in black-box language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=KS8mIvetg2

    2024

  29. [37]

    Learning Hierarchical Structures with Linear Relational Embedding

    Alberto Paccanaro and Geoffrey E Hinton. Learning Hierarchical Structures with Linear Relational Embedding . In Advances in Neural Information Processing Systems , volume 14. MIT Press, 2001. URL https://papers.nips.cc/paper_files/paper/2001/hash/814a9c18f5abff398787c9cfcbf3d8...

    2001

  30. [38]

    The Linear Representation Hypothesis and the Geometry of Large Language Models

    Kiho Park, Yo Joong Choe, and Victor Veitch. The Linear Representation Hypothesis and the Geometry of Large Language Models . In Forty-First International Conference on Machine Learning , 2024. URL https://openreview.net/forum?id=UGpGkLzwpP

    2024

  31. [39]

    G lo V e: Global vectors for word representation

    Jeffrey Pennington, Richard Socher, and Christopher Manning. G lo V e: Global vectors for word representation. In Alessandro Moschitti, Bo Pang, and Walter Daelemans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing ( EMNLP ) , pp.\...

    2014 doi

  32. [40]

    Null it out: Guarding protected attributes by iterative nullspace projection

    Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.\ 7237--7256, Online, July...

    2020

  33. [41]

    Impact of pretraining term frequencies on few-shot numerical reasoning

    Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot numerical reasoning. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp.\ 84...

    2022 doi

  34. [42]

    Backtracking mathematical reasoning of language models to the pretraining data

    Yasaman Razeghi, Hamish Ivison, Sameer Singh, and Yanai Elazar. Backtracking mathematical reasoning of language models to the pretraining data. In NeurIPS Workshop on Attributing Model Behavior at Scale, 2023. URL https://openreview.net/forum?id=EKvqw9k3lC

    2023

  35. [43]

    Steering llama 2 via contrastive activation addition

    Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. Steering llama 2 via contrastive activation addition. pp.\ 15504--15522, August 2024. doi:10.18653/v1/2024.acl-long.828. URL https://aclanthology.org/2024.acl-long.828/

    2024 doi

  36. [44]

    Salton, A

    G. Salton, A. Wong, and C. S. Yang. A vector space model for automatic indexing. Commun. ACM, 18 0 (11): 0 613–620, November 1975. ISSN 0001-0782. doi:10.1145/361219.361220. URL https://doi.org/10.1145/361219.361220

    1975

  37. [45]

    Naomi Saphra and Sarah Wiegreffe. Mechanistic? In Yonatan Belinkov, Najoung Kim, Jaap Jumelet, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen (eds.), Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp.\ 480--498, Miami, Florida,...

    2024 doi

  38. [46]

    The bias amplification paradox in text-to-image generation

    Preethi Seshadri, Sameer Singh, and Yanai Elazar. The bias amplification paradox in text-to-image generation. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistic...

    2024 doi

  39. [47]

    Detecting pretraining data from large language models

    Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/fo...

    2024

  40. [48]

    Membership inference attacks against machine learning models

    Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pp.\ 3--18, 2017. doi:10.1109/SP.2017.41. URL https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&a...

    2017 doi

  41. [49]

    The curious case of hallucinatory (un)answerability: Finding truths in the hidden states of over-confident large language models

    Aviv Slobodkin, Omer Goldman, Avi Caciularu, Ido Dagan, and Shauli Ravfogel. The curious case of hallucinatory (un)answerability: Finding truths in the hidden states of over-confident large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of th...

    2023 doi

  42. [50]

    Dolma: an open corpus of three trillion tokens for language model pretraining research

    Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: an open corpus of three trillion tokens for language model pretraining research. In Lun-Wei Ku, Andre Martins, a...

    2024

  43. [51]

    Extracting Latent Steering Vectors from Pretrained Language Models

    Nishant Subramani, Nivedita Suresh, and Matthew Peters. Extracting Latent Steering Vectors from Pretrained Language Models . In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics : ACL 2022 , pp.\ 566--581...

    2022 doi

  44. [52]

    Formalizing and estimating distribution inference risks

    Anshuman Suri and David Evans. Formalizing and estimating distribution inference risks. Proceedings on Privacy Enhancing Technologies, 2022. URL https://arxiv.org/abs/2109.06024

    2022 arXiv

  45. [53]

    Scaling Monosemanticity : Extracting Interpretable Features from Claude 3 Sonnet

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, et al. Scaling Monosemanticity : Extracting Interpretable Features from Claude 3 Sonnet . 2024. URL https://transformer-circuits.pub...

    2024

  46. [54]

    Function vectors in large language models

    Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=AwyxtyMwaG

    2024

  47. [55]

    GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model

    Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model . https://github.com/kingoflolz/mesh-transformer-jax, May 2021

    2021

  48. [56]

    Understanding reasoning ability of language models from the perspective of reasoning paths aggregation

    Xinyi Wang, Alfonso Amayuelas, Kexun Zhang, Liangming Pan, Wenhu Chen, and William Yang Wang. Understanding reasoning ability of language models from the perspective of reasoning paths aggregation. In Forty-first International Conference on Machine Learning, 2024. URL https://...

    2024

  49. [57]

    Generalization v.s

    Xinyi Wang, Antonis Antoniades, Yanai Elazar, Alfonso Amayuelas, Alon Albalak, Kexun Zhang, and William Yang Wang. Generalization v.s. memorization: Tracing language models capabilities back to pretraining data. In The Thirteenth International Conference on Learning Representa...

    2025

  50. [58]

    Doremi: Optimizing data mixtures speeds up language model pretraining

    Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. UR...

    2023

  51. [59]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  52. [60]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  53. [61]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  54. [62]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.