Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Beyond the Rosetta Stone: Unification Forces in Generalization Dynamics

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that cross-lingual fact recall fails because models learn language as a shortcut, and that a training-time unification score can predict and guide transfer.

desk verdict Useful synthetic study and a practical unification metric, but the 'essential' claim overreaches against the paper's own zero-parallel results. read the letter →

arxiv 2508.11017 v2 pith:WWBTLEBQ submitted 2025-08-14 cs.CL cs.AI

Carter Blum , Katja Filippova , Ann Yuan , Asma Ghandeharioun , Julian Zimmert , Fred Zhang , Jessica Hoffmann , Tal Linzen
show 3 more authors
Martin Wattenberg Lucas Dixon Mor Geva
This is my paper · ORCID
classification cs.CLcs.AI
keywords cross-lingualtransferfactualrecallrepresentationalunificationspuriouscorrelationspretrainingdynamicssyntheticlanguagesscorelanguagefeature
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks why a language model can answer a fact correctly in the language it was trained on but hallucinate when asked the same fact in another language. Using small Transformers trained from scratch on synthetic two-language datasets, the authors show that the failure is not a gap in the facts themselves: the same facts are present in both languages, yet the model develops separate, language-specific representations instead of one shared representation. They identify a specific early learning phase in which this separation happens, and trace it to a single mechanism: when the language of an example is both easy to detect and predictive of the answer distribution, the model learns language as a shortcut and that shortcut dominates its internal representations. They show that unification of representations is necessary for cross-lingual transfer, introduce a unification score computed on training representations that predicts cross-lingual accuracy with Pearson correlation above 0.95, and demonstrate that data interventions that make language less informative or less extractable improve transfer without adding parallel data. This matters because it turns an apparently intractable failure of multilingual models into a measurable, modifiable property of pretraining data.

What carries the argument

The load-bearing object is the 'language feature': the identity of the language in which a training example is written, treated as a potentially spurious predictor. Its influence is measured by a language feature footprint, the $R^2$ of a linear regression of hidden-state representations on language identity. The central quantitative tool is the unification score, defined for two languages as the cosine similarity between representations of the same fact in the two languages divided by the baseline similarity between representations of different facts in the same language; it is computed on the parallel (bilingual) training examples using either residual-stream activations or gradients. The mechanism is feature-learning order: following prior theoretical results, features that explain more variance in the training objective are learned earlier and dominate representations, so an easy-to-extract and informative language feature is learned before the fact identity and leaves a persistent footprint. The paper visualizes this as 'checkerboarding' in pairwise similarity matrices, where same-language blocks become more similar than same-fact cross-language blocks during an early checkpoint phase.

What would settle it

A concrete falsifier: train two models on identical synthetic facts with the same amount of parallel data, one with attribute frequencies balanced across languages and one with language highly informative about attributes; if the balanced model does not show higher unification scores and cross-lingual accuracy, the core claim fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that cross-lingual factual recall fails when a model's internal representations of the same fact in two languages remain separate, and that this separation is caused by language operating as a spurious predictive feature during pretraining. In the controlled setup, every model sees the same facts, the same two languages, and the same number of examples; only distributional properties change. Separation occurs when the language of an example is easy to extract (distinct vocabularies, word-level tokens) and informative about the attribute to be predicted (because attribute frequencies differ across languages). Under those conditions the model enters a 'checkerboarding' phase in which same-language examples with different facts look more similar than cross-lingual examples of the same fact, and this phase predicts poor transfer in the final model. The paper further claims that unification is essential for transfer, that a unification score measured on training representations correlates with cross-lingual accuracy above 0.95 and can replace a small cross-lingual test set for model selection, and that the same mechanism appears in Gemma-2-2B, where unification scores predict answer accuracy with an ROC of about 0.65 and vocabulary overlap between languages explains more variance in translation accuracy than whether a fact appears in both languages.

Load-bearing premise

The whole argument depends on the assumption that what happens with synthetic, invented languages that share no words and no grammar is the same mechanism that makes real multilingual models fail; the Gemma results are suggestive but not proof.

Editorial extensions

If this is right

  • Balancing the frequency of attribute values across languages in monolingual data improves cross-lingual transfer without increasing the amount of parallel data, by reducing the informativeness of the language feature.
  • Making language harder to detect—through character-level tokenization, lowercasing, more templates, or shared vocabulary and scripts—shrinks the language footprint and improves cross-lingual generalization.
  • Training-time unification scores can be used to select model checkpoints or hyperparameter runs, performing comparably to a small cross-lingual test set, while in-language accuracy alone is a misleading selection signal.
  • In large models, vocabulary overlap between a source and target language explains a substantial share of variance in cross-lingual factual accuracy, more than whether individual facts have parallel pages.
  • The same mechanism likely governs transfer across paraphrases and related surface forms, not only across languages, since the separation is driven by any easy, informative surface feature.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the unification score could be monitored during real pretraining as an early-warning signal for the formation of knowledge silos, before a cross-lingual evaluation set is available.
  • Beyond the paper: the results suggest a concrete testable intervention for production pretraining—obscuring script-level or casing-level language cues early in training, or curriculum-balancing attribute distributions—though the paper notes language is a legitimate prior for culture-specific facts.
  • Beyond the paper: if the mechanism is generic, then any salient, informative surface feature (formatting, domain markers, template identity) should induce similar siloing; this could be tested by re-running the same synthetic setup with domains instead of languages.
  • Beyond the paper: the strong correlation between unification and transfer in synthetic data raises the possibility of using unification as a training objective or regularizer, something the paper does not propose.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper trains small Transformer models from scratch on synthetic multilingual template languages to study the causes of cross-lingual factual transfer failure. It reports that parallel training data ("celebrity" events) promotes unified representations of the same fact across languages, while monolingual data whose language identity is both informative about the attribute distribution and easy to extract induces representational separation. The authors introduce a unification score computed on parallel training examples and show it correlates strongly with cross-lingual accuracy in the synthetic setting and, more weakly, with cross-lingual factual accuracy in Gemma-2-2B and Gemma-3-4B on the ECLeKTic dataset. They also present interventions (balancing attribute frequencies, obscuring language identity via character tokenization or more templates) that improve cross-lingual transfer, and a toy regression model linking language-feature variance to transfer failure.

Significance. If the causal story holds, this is a valuable controlled, interventional account of cross-lingual transfer: it connects concrete dataset statistics (mutual information between language and attribute, extractability of language identity) to representation geometry and to training dynamics, and it offers a representation-based metric for model and checkpoint selection that does not require an OOD test set. The paper is careful in several respects: dataset sizes are matched across parallel-data fractions, the balanced and imbalanced manipulations preserve the amount of parallel data, and the extractability comparison is made between two character-tokenizer conditions that differ only in casing. The findings would be a useful step toward mechanistic, data-side explanations of cross-lingual hallucinations, and the proposed interventions are actionable for pretraining data curation. However, the central causal claim is currently stated more strongly than the evidence supports, and one statistical claim in the LLM validation is contradicted by the paper's own appendix.

major comments (3)
  1. [Abstract; Sec. 2.1; Sec. 5; App. B.11] The claim that "unification is essential for cross-lingual transfer" is not supported by the paper's own zero-parallel results. Section 2.1 reports that models trained with 0% parallel data reach ~99% cross-lingual accuracy on birth-year facts, yet the unification score of Sec. 5 is defined only over "the parallel examples from the training set," so it is undefined or empty in exactly that regime. Appendix B.11 itself concedes that at celebrities=0 there is "no formal basis for mapping language A templates to language B," and the accuracy metric used throughout does not distinguish genuine attribute recall from language-prior guessing. To maintain the essentiality claim, the authors must show that the 99% zero-parallel performance is not genuine transfer (e.g., by removing the language-specific attribute prior with the balanced construction at 0% parallel and showing the accuracy collapses), or they must explicitly restrict the claim to settings with parallel data and treat zero-parallel transfer as a separate mechanism. As written, the correlation results in Fig. 7 and the causal language in the abstract overstate what the experiments establish.
  2. [Sec. 6; App. B.13.5] The main text states that unification predicts OOD accuracy with "ROC score of around 0.65, regardless of the layer" and that "this is statistically significant at all layers via a t-test with Bonferroni correction," but the detailed t-test results in App. B.13.5 report p=0.107 at layer 0 and p=0.105 at layer 41, both not significant even before any correction. The claim of significance at all layers is therefore factually incorrect. This is load-bearing because the LLM validation is the main evidence that the synthetic findings extend to real models; the corrected picture is a modest, layer-dependent signal that should be reported accurately.
  3. [Sec. 5; Fig. 7] The strong correlation (PCC > 0.95) between the unification score and cross-lingual accuracy is potentially a proxy for the fraction of parallel data, since the Fig. 7 caption notes that the celebrity fraction "also correlates strongly with generalization ability." The paper does not provide a partial correlation controlling for the parallel fraction, nor a fixed-fraction analysis showing that within runs with the same celebrity fraction, the unification score still tracks accuracy. Because the interventions in Sec. 4 vary the parallel fraction as well as informativeness/extractability, the unique predictive value of the representation-based score over simple dataset statistics is not established. The model-selection experiments in Fig. 8 are suggestive but do not resolve this confound.
minor comments (5)
  1. [Sec. 5] The definition of the unification score uses notation that conflates a fact with a set of datapoints: in Unification(θ,D) := E_{X,Y∼Facts(D)} [simθ(X,Y)/simθ(X,X)], please define precisely what X and Y are (e.g., the sets of training examples for a given fact in the two languages) and what simθ(X,X) denotes, so that the ratio is unambiguous.
  2. [App. B.13.2, Table 1] In the 'fr' row, the R² for Vocabulary Overlap alone is 99.53 while the R² for Both is 30.16, which is implausibly lower than either single feature and suggests a numerical error or an unusual suppression effect; please check and explain this value.
  3. [Sec. 4.1] The 'src' condition is not size-matched with the balanced and imbalanced conditions; the equal-footing statement applies only to the latter two. Please state explicitly that 'src' is a smaller dataset and restrict the controlled comparison to balanced vs. imbalanced, or match the sizes.
  4. [Sec. 4.2] The comparison between the word-tokenizer baseline and the character-tokenizer conditions confounds language extractability with tokenization granularity; the paper correctly restricts the primary extractability comparison to the two character-tokenizer conditions, but the abstract and introduction should avoid implying that the word-vs-char contrast alone isolates extractability.
  5. [App. B.3] The alternative unification probe (training a language classifier on celebrity representations) is reported to be substantially less discriminative; a sentence explaining why the similarity-ratio metric is preferable to this linear-probe formulation would help readers interpret the choice of unification metric.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the unification score is measured from residual-stream representations and cross-lingual accuracy from held-out labels; the two are not equated by construction.

full rationale

The paper's central predictive quantity, the unification score in Section 5, is defined as the expected ratio of cosine similarities between same-fact cross-language representational sets and within-set similarities, with D taken from the parallel training examples. This definition contains no label or accuracy term. Cross-lingual accuracy is measured on reserved held-out verbalizations of non-celebrity facts (Section 2), so the reported PCC > 0.95 is an empirical correlation between distinct quantities, not an identity imposed by the formalism. The interventions in Sections 4 and 4.2 (balancing attribute frequencies, changing tokenizer type, varying template count) manipulate dataset properties and then measure accuracy and representational footprint; no parameter is fitted to the cross-lingual accuracy values. The Gemma-2-2B validation (ROC 0.65, Bonferroni-corrected t-tests) is an external benchmark and does not restate the synthetic results. The self-citations present (Chang et al. 2025 for gradient approximations, Geva et al. 2023 for factual recall background, Ghandeharioun et al. 2024 for representation analysis) are implementation or background references, not load-bearing justifications of the main claim. The zero-parallel 99% birth-year result and the Appendix B.11 caveat that 'there is no formal basis for mapping language A templates to language B' weaken the causal wording 'unification is essential,' but that is a scope and correctness concern, not definitional circularity. No equation in the paper reduces a prediction to its inputs by construction, so no circular step is exhibited.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central mechanistic claims rest on three pillars: the synthetic template-language assumption, the validity of residual-stream similarity as a proxy for unification, and the transfer of the shortcut-learning story from controlled data to natural language. The paper explicitly acknowledges the first pillar's limitation but does not provide independent evidence for it.

free parameters (2)
  • Attribute pool sizes = 20 birth years, 30 death years, 100 cities
    Chosen by hand to create different attribute frequency distributions, which yields different mutual information between language and attribute; these sizes are not fit to a target but are experimental design choices.
  • Language variance fraction (toy regression) = 0%, 5%, 10%
    In Appendix B.14, the fraction of label variance explained by the language feature is manually set to illustrate the effect on cross-lingual generalization; it is a hand-chosen parameter of the toy model.
assumptions (5)
  • domain assumption Features are learned in descending order of the variance they explain, and more extractable features dominate representations.
    Invoked in Sec. 4 to motivate why language, when informative and extractable, is learned early and dominates the residual stream.
  • domain assumption Representational similarity in the residual stream at the pre-attribute token is a valid measure of whether facts are unified across languages.
    All unification metrics and footprint analyses in Secs. 3 and 5 rely on cosine similarity of residual stream activations.
  • domain assumption The synthetic template-based languages with disjoint vocabularies capture the essential structure of cross-lingual transfer failure.
    The entire Petri dish methodology depends on this; the paper acknowledges in Limitations that structural and lexical similarities are ignored.
  • domain assumption Mutual information between language and attribute computed on the generated dataset is the correct measure of language informativeness.
    Sec. 4.1 uses MI(Language; Attribute) to quantify the spurious correlation, assuming MI is the operative quantity.
  • domain assumption ECLeKTic and OpenSubtitles are adequate proxies for natural-language cross-lingual factual transfer and vocabulary overlap.
    Sec. 6 uses these datasets to validate the synthetic findings on Gemma-2-2B.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond the Rosetta Stone: Unification Forces in Generalization Dynamics." pith.science (2026). https://pith.science/paper/WWBTLEBQ

@misc{pith2026250811017,
  author       = {Pith},
  title        = {Pith review of: Beyond the Rosetta Stone: Unification Forces in Generalization Dynamics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WWBTLEBQ}},
  note         = {Machine review of arXiv:2508.11017}
}
read the original abstract

Large language models (LLMs) struggle with cross-lingual knowledge transfer: they hallucinate when asked in one language about facts expressed in a different language during training. This work introduces a controlled setting to study the causes and dynamics of this phenomenon by training small Transformer models from scratch on synthetic multilingual datasets. We identify a learning phase wherein a model develops either separate or unified representations of the same facts across languages, and show that unification is essential for cross-lingual transfer. We also show that the degree of unification depends on mutual information between facts and training data language, and on how easy it is to extract that language. Based on these insights, we develop methods to modulate the level of cross-lingual transfer by manipulating data distribution and tokenization, and we introduce metrics and visualizations to formally characterize their effects on unification. Our work shows how controlled settings can shed light on pre-training dynamics and suggests new directions for improving cross-lingual transfer in LLMs.

Figures

Figures reproduced from arXiv: 2508.11017 by the authors.

Figure 1
Figure 1. Birth and death events are created for every entity by sampling from a set of birth and death years (disjoint) and cities (same pool). A dataset is comprised of monolingual (events expressed in either EN or ES) and parallel (events expressed in both languages) data. Arrows point from a particular event to the training templates (solid line) or in-language test set templates (dashed). All verbalizations in the other … view at source ↗
Figure 2
Figure 2. Left: Same-language (in-distribution, ID) versus cross-language (out-of-distribution, OOD) test label probability across models (trained from scratch with a Pythia config on datasets expressing the same facts using the same languages) with a range of parallel data ratios. ID per￾formance provides a misleading signal regarding OOD performance. Right: Accuracy curves for in-language (ID) and cross-language (OOD) perfo… view at source ↗
Figure 3
Figure 3. Pairwise cosine similarities between activation-based representations of examples from a [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Pairwise similarity matrices between activation-based representations at the token pre [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Cross-lingual accuracy for models trained on [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Cross-lingual accuracy for models trained with the character ( [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Activation unification scores (left) and gradient unification scores (right) correlate strongly [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: (left) Unification score is competitive with using a small test set to select the best model [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: We also experiment with an alternative measure of how similar the representations of [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Layerwise results for activation unification scores vs OOD. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 12
Figure 12. Figure 12: Cased vs all-lowercased (by language) at checkpoint-8,000 [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: [Models with 30% and 8% celebrity events, layers 0-1, final checkpoint, activations [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: [Models with 30% and 4% celebrity events, final checkpoint] Also with [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: Increasing the number of templates substantially improves cross-lingual generalization, [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: Frequencies of the birth-year attribute values in the KG (left) and in a dataset, split [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]
Figure 17
Figure 17. Figure 17: Representational variance explained by language (“Lang ID” - red) versus fact (“Fact ID” [PITH_FULL_IMAGE:figures/full_fig_p018_17.png]
Figure 18
Figure 18. Figure 18: Green line shows proportion of errors where the predicted attributed belongs to a different [PITH_FULL_IMAGE:figures/full_fig_p019_18.png]
Figure 20
Figure 20. Figure 20: ROC plots of how much unification score is predictive of translation accuracy for [PITH_FULL_IMAGE:figures/full_fig_p020_20.png]
Figure 21
Figure 21. Figure 21: < x, y > indicates going from source language x to target language y. Lower left includes languages like Hebrew and Hindi that have no to little vocab overlap with English. Top right includes languages with higher vocab overlap with English, such as Indonesian and Por…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Rethinking Cross-lingual Gaps from a Statistical Viewpoint

    cs.CL 2025-10 conditional novelty 6.0 of 10

    Cross-lingual accuracy gaps in LLMs are dominated by higher response variance in target languages, not missing knowledge; ensembling and variance-reduction prompts shrink the gap.

Reference graph

Works this paper leans on

32 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  3. [3]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  4. [4]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    work page

  5. [5]

    Language models' factuality depends on the language of inquiry, 2025

    Tushar Aggarwal, Kumar Tanmay, Ayush Agrawal, Kumar Ayush, Hamid Palangi, and Paul Pu Liang. Language models' factuality depends on the language of inquiry, 2025. URL https://arxiv.org/abs/2502.17955

    arXiv 2025

  6. [6]

    ICML 2024 Tutorial: Physics of Language Models , July 2024

    Zeyuan Allen-Zhu . ICML 2024 Tutorial: Physics of Language Models , July 2024. Project page: https://physics.allen-zhu.com/

    work page 2024

  7. [7]

    Chang, Dheeraj Rajagopal, Tolga Bolukbasi, Lucas Dixon, and Ian Tenney

    Tyler A. Chang, Dheeraj Rajagopal, Tolga Bolukbasi, Lucas Dixon, and Ian Tenney. Scalable influence and fact tracing for large language model pretraining. In Proceedings of the 13th International Conference on Learning Representations, 24--28 Apr 2025. URL https://arxiv.org/abs/2410.17413

    arXiv 2025

  8. [8]

    Wichmann, and Wieland Brendel

    Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNN s are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bygh9j09KX

    work page 2019

Show all 32 references
  1. [9]

    Dissecting recall of factual associations in auto-regressive language models

    Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Proce...

    2023 doi

  2. [10]

    Patchscopes: A unifying framework for inspecting hidden representations of language models

    Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscopes: A unifying framework for inspecting hidden representations of language models. In International Conference for Machine Learning, 2024. URL https://arxiv.org/abs/2401.06102

    2024 arXiv

  3. [11]

    Eclektic: a novel challenge set for evaluation of cross-lingual knowledge transfer, 2025

    Omer Goldman, Uri Shaham, Dan Malkin, Sivan Eiger, Avinatan Hassidim, Yossi Matias, Joshua Maynez, Adi Mayrav Gilady, Jason Riesa, Shruti Rijhwani, Laura Rimell, Idan Szpektor, Reut Tsarfaty, and Matan Eyal. Eclektic: a novel challenge set for evaluation of cross-lingual knowl...

    2025

  4. [12]

    On the foundations of shortcut learning

    Katherine Hermann, Hossein Mobahi, Thomas FEL, and Michael Curtis Mozer. On the foundations of shortcut learning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Tj3xLVuE9f

    2024

  5. [13]

    Understanding black-box predictions via influence functions

    Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp.\ 1885--1894. PML...

    2017

  6. [14]

    Chan, and Katherine Hermann

    Andrew Kyle Lampinen, Stephanie C.Y. Chan, and Katherine Hermann. Learned feature representations are biased by complexity, learning order, position, and more. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=aY2nsgE97a

    2024

  7. [15]

    P re A lign: Boosting cross-lingual transfer by early establishment of multilingual alignment

    Jiahuan Li, Shujian Huang, Aarron Ching, Xinyu Dai, and Jiajun Chen. P re A lign: Boosting cross-lingual transfer by early establishment of multilingual alignment. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Meth...

    2024 doi

  8. [17]

    Understanding cross-lingual inconsistency in large language models, 2025 b

    Zheng Wei Lim, Alham Fikri Aji, and Trevor Cohn. Understanding cross-lingual inconsistency in large language models, 2025 b . URL https://arxiv.org/abs/2505.13141

    2025

  9. [18]

    Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles

    Pierre Lison and J \"o rg Tiedemann. Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pp.\ 923--929, 2016

    2016

  10. [19]

    Tracing multilingual factual knowledge acquisition in pretraining, 2025

    Yihong Liu, Mingyang Wang, Amir Hossein Kargaran, Felicia Körner, Ercong Nie, Barbara Plank, François Yvon, and Hinrich Schütze. Tracing multilingual factual knowledge acquisition in pretraining, 2025. URL https://arxiv.org/abs/2505.14824

    2025

  11. [20]

    Thomas McCoy, Ellie Pavlick, and Tal Linzen

    R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Anna Korhonen, David Traum, and Llu \'i s M \`a rquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computatio...

    2019 doi

  12. [21]

    Interpreting gtp: T he logit lens, 2020

    nostalgebraist. Interpreting gtp: T he logit lens, 2020. URL https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens

    2020

  13. [22]

    Overlap-based vocabulary generation improves cross-lingual transfer among related languages

    Vaidehi Patil, Partha Talukdar, and Sunita Sarawagi. Overlap-based vocabulary generation improves cross-lingual transfer among related languages. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for C...

    2022 doi

  14. [23]

    The Book of Why: The New Science of Cause and Effect

    Judea Pearl and Dana Mackenzie. The Book of Why: The New Science of Cause and Effect. Penguin Books, 2018

    2018

  15. [24]

    Cross-lingual consistency of factual knowledge in multilingual language models

    Jirui Qi, Raquel Fern \'a ndez, and Arianna Bisazza. Cross-lingual consistency of factual knowledge in multilingual language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp....

    2023 doi

  16. [25]

    Gemma 2: Improving open language models at a practical size

    Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024

    2024 arXiv

  17. [26]

    Procedural knowledge in pretraining drives reasoning in large language models

    Laura Ruis, Maximilian Mozes, Juhan Bae, Siddhartha Rao Kamalakara, Dwaraknath Gnaneshwar, Acyr Locatelli, Robert Kirk, Tim Rockt \"a schel, Edward Grefenstette, and Max Bartolo. Procedural knowledge in pretraining drives reasoning in large language models. In The Thirteenth I...

    2025

  18. [27]

    A mathematical theory of semantic development in deep neural networks

    Andrew M Saxe, James L McClelland, and Surya Ganguli. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116 0 (23): 0 11537--11546, 2019

    2019

  19. [28]

    Theoretical and practical perspectives on what influence functions do

    Andrea Schioppa, Katja Filippova, Ivan Titov, and Polina Zablotskaia. Theoretical and practical perspectives on what influence functions do. In Neural Information Processing Systems, 2023. URL https://arxiv.org/abs/2305.16971

    2023 arXiv

  20. [29]

    John schulman - reinforcement learning from human feedback: Progress and challenges

    John Schulman. John schulman - reinforcement learning from human feedback: Progress and challenges. https://www.youtube.com/watch?v=hhiLw5Q_UFg, 2023. Accessed: July 2025

    2023

  21. [30]

    Do multilingual llms think in english? arXiv preprint arXiv:2502.15603, 2025

    Lisa Schut, Yarin Gal, and Sebastian Farquhar. Do multilingual llms think in english? arXiv preprint arXiv:2502.15603, 2025

    2025 arXiv

  22. [31]

    Lost in multilinguality: Dissecting cross-lingual factual inconsistency in transformer language models, 2025

    Mingyang Wang, Heike Adel, Lukas Lange, Yihong Liu, Ercong Nie, Jannik Strotgen, and Hinrich Schutze. Lost in multilinguality: Dissecting cross-lingual factual inconsistency in transformer language models, 2025. URL https://arxiv.org/pdf/2504.04264. To appear in ACL 2025

    2025 arXiv

  23. [32]

    Kankanhalli

    Ziwei Xu, Sanjay Jain, and Mohan S. Kankanhalli. Hallucination is inevitable: An innate limitation of large language models. CoRR, abs/2401.11817, 2024. URL https://doi.org/10.48550/arXiv.2401.11817

    2024 doi arXiv

  24. [33]

    Converging to a lingua franca: Evolution of linguistic regions and semantics alignment in multilingual large language models

    Hongchuan Zeng, Senyu Han, Lu Chen, and Kai Yu. Converging to a lingua franca: Evolution of linguistic regions and semantics alignment in multilingual large language models. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schock...

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.