{"id":"98760efe-5fc7-486a-82b1-bc1b7c7445ae","arxiv_id":"2508.11017","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Cross-lingual fact recall in transformers depends on whether the model unifies the same fact across languages, and unification is controlled by how informative and how easy-to-detect the language signal is in the training data.","lead":"This paper trains small AI models on made-up languages to watch how they store the same facts in different languages, and finds that knowledge transfers across languages only when the model builds a shared internal representation. It then shows that data design, such as making languages harder to tell apart, can push models toward that shared representation and improve cross-lingual fact recall.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero-parallel birth-year transfer (99%) contradicts 'unification is essential'; high cross-lingual accuracy may be attribute-prior guessing, and the unification score is undefined at 0% parallel.","rationale":"The reader's weakest assumption was external validity: synthetic template languages may not preserve the mechanism of real cross-lingual transfer. That concern is legitimate and partially acknowledged in the Limitations section. However, I found a more immediate, internally checkable tension: the paper's own zero-parallel results show high cross-lingual accuracy in a regime where the unification score is undefined and where Appendix B.11 concedes transfer may happen 'for the wrong reasons.' This directly affects the abstract's 'unification is essential' claim, independent of whether the synthetic setup generalizes to real LLMs. The proposed test distinguishes genuine retrieval from prior guessing, which is exactly the distinction the paper needs. Because the paper's existing evidence is still largely sound for the parallel-data regime and the central mechanism is well motivated, the conditional verdict remains appropriate; no harsher adjustment is needed, but the claim should be reworded or the additional analysis should be supplied.","tokens_in":15450,"tokens_out":9113,"duration_ms":106304,"concrete_test":"Re-analyze the 0%-parallel (and low-parallel) models from Sec. 2.1 and Fig. 2b: compute cross-lingual accuracy separately for birth-year facts whose correct year is NOT the most frequent year in the target language's training marginal, and compare against a baseline that always predicts the target-language marginal mode. If accuracy on non-majority attributes is at chance, the 99% figure is prior guessing, so the 'unification is essential' claim remains untested for genuine transfer and should be reworded; if accuracy remains high for non-majority attributes, genuine transfer without unification occurs and the claim is directly contradicted. Either outcome settles the concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's causal claim ('unification is essential for cross-lingual transfer') is undercut by the paper's own zero-parallel results. Section 2.1 reports that models trained with 0% celebrity/parallel events still reach ~99% cross-lingual accuracy on birth-year facts; Section 3 (Fig. 2b) shows a checkpoint attaining perfect OOD accuracy on birth-year while other fact types fail. At 0% parallel data, the unification score of Sec. 5 is computed on 'the parallel examples from the training set,' so it is undefined or empty; there are no same-fact cross-lingual training pairs to unify. Yet high cross-lingual accuracy occurs. The paper acknowledges this in Appendix B.11: 'Cross-lingual generalization can take place for the wrong reasons,' noting there is 'no formal basis for mapping language A templates to language B' at celebrities=0. This means the accuracy metric used throughout does not distinguish genuine unification-mediated retrieval from guessing based on a language-specific attribute prior (e.g., always predicting the frequent year). Because all headline correlations (Fig. 7, Fig. 8) use this same accuracy measure, they cannot establish that unification is necessary; they may only characterize a regime where parallel data is present and the prior is weak. The central claim needs to be restricted to genuine attribute recall, and the paper needs to show that apparent transfer without parallel data disappears when the language prior is removed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper trains small Transformer models from scratch on synthetic multilingual template languages to study the causes of cross-lingual factual transfer failure. It reports that parallel training data (\"celebrity\" events) promotes unified representations of the same fact across languages, while monolingual data whose language identity is both informative about the attribute distribution and easy to extract induces representational separation. The authors introduce a unification score computed on parallel training examples and show it correlates strongly with cross-lingual accuracy in the synthetic setting and, more weakly, with cross-lingual factual accuracy in Gemma-2-2B and Gemma-3-4B on the ECLeKTic dataset. They also present interventions (balancing attribute frequencies, obscuring language identity via character tokenization or more templates) that improve cross-lingual transfer, and a toy regression model linking language-feature variance to transfer failure.","tokens_in":15719,"tokens_out":8194,"duration_ms":92111,"significance":"If the causal story holds, this is a valuable controlled, interventional account of cross-lingual transfer: it connects concrete dataset statistics (mutual information between language and attribute, extractability of language identity) to representation geometry and to training dynamics, and it offers a representation-based metric for model and checkpoint selection that does not require an OOD test set. The paper is careful in several respects: dataset sizes are matched across parallel-data fractions, the balanced and imbalanced manipulations preserve the amount of parallel data, and the extractability comparison is made between two character-tokenizer conditions that differ only in casing. The findings would be a useful step toward mechanistic, data-side explanations of cross-lingual hallucinations, and the proposed interventions are actionable for pretraining data curation. However, the central causal claim is currently stated more strongly than the evidence supports, and one statistical claim in the LLM validation is contradicted by the paper's own appendix.","major_comments":[{"comment":"The claim that \"unification is essential for cross-lingual transfer\" is not supported by the paper's own zero-parallel results. Section 2.1 reports that models trained with 0% parallel data reach ~99% cross-lingual accuracy on birth-year facts, yet the unification score of Sec. 5 is defined only over \"the parallel examples from the training set,\" so it is undefined or empty in exactly that regime. Appendix B.11 itself concedes that at celebrities=0 there is \"no formal basis for mapping language A templates to language B,\" and the accuracy metric used throughout does not distinguish genuine attribute recall from language-prior guessing. To maintain the essentiality claim, the authors must show that the 99% zero-parallel performance is not genuine transfer (e.g., by removing the language-specific attribute prior with the balanced construction at 0% parallel and showing the accuracy collapses), or they must explicitly restrict the claim to settings with parallel data and treat zero-parallel transfer as a separate mechanism. As written, the correlation results in Fig. 7 and the causal language in the abstract overstate what the experiments establish.","section":"Abstract; Sec. 2.1; Sec. 5; App. B.11"},{"comment":"The main text states that unification predicts OOD accuracy with \"ROC score of around 0.65, regardless of the layer\" and that \"this is statistically significant at all layers via a t-test with Bonferroni correction,\" but the detailed t-test results in App. B.13.5 report p=0.107 at layer 0 and p=0.105 at layer 41, both not significant even before any correction. The claim of significance at all layers is therefore factually incorrect. This is load-bearing because the LLM validation is the main evidence that the synthetic findings extend to real models; the corrected picture is a modest, layer-dependent signal that should be reported accurately.","section":"Sec. 6; App. B.13.5"},{"comment":"The strong correlation (PCC > 0.95) between the unification score and cross-lingual accuracy is potentially a proxy for the fraction of parallel data, since the Fig. 7 caption notes that the celebrity fraction \"also correlates strongly with generalization ability.\" The paper does not provide a partial correlation controlling for the parallel fraction, nor a fixed-fraction analysis showing that within runs with the same celebrity fraction, the unification score still tracks accuracy. Because the interventions in Sec. 4 vary the parallel fraction as well as informativeness/extractability, the unique predictive value of the representation-based score over simple dataset statistics is not established. The model-selection experiments in Fig. 8 are suggestive but do not resolve this confound.","section":"Sec. 5; Fig. 7"}],"minor_comments":[{"comment":"The definition of the unification score uses notation that conflates a fact with a set of datapoints: in Unification(θ,D) := E_{X,Y∼Facts(D)} [simθ(X,Y)/simθ(X,X)], please define precisely what X and Y are (e.g., the sets of training examples for a given fact in the two languages) and what simθ(X,X) denotes, so that the ratio is unambiguous.","section":"Sec. 5"},{"comment":"In the 'fr' row, the R² for Vocabulary Overlap alone is 99.53 while the R² for Both is 30.16, which is implausibly lower than either single feature and suggests a numerical error or an unusual suppression effect; please check and explain this value.","section":"App. B.13.2, Table 1"},{"comment":"The 'src' condition is not size-matched with the balanced and imbalanced conditions; the equal-footing statement applies only to the latter two. Please state explicitly that 'src' is a smaller dataset and restrict the controlled comparison to balanced vs. imbalanced, or match the sizes.","section":"Sec. 4.1"},{"comment":"The comparison between the word-tokenizer baseline and the character-tokenizer conditions confounds language extractability with tokenization granularity; the paper correctly restricts the primary extractability comparison to the two character-tokenizer conditions, but the abstract and introduction should avoid implying that the word-vs-char contrast alone isolates extractability.","section":"Sec. 4.2"},{"comment":"The alternative unification probe (training a language classifier on celebrity representations) is reported to be substantially less discriminative; a sentence explaining why the similarity-ratio metric is preferable to this linear-probe formulation would help readers interpret the choice of unification metric.","section":"App. B.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a strong empirical study with careful synthetic controls, but the advertised central claim ('unification is essential') is not supported by the paper's own zero-parallel results, and the 'significant at all layers' statement for the Gemma experiments is contradicted by the appendix. These issues are fixable within the scope of a revision, but they require re-framing the main claim and correcting the statistical reporting. The paper would also benefit from a direct check of whether the unification score adds predictive signal beyond the celebrity fraction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe thing to know: this is a genuinely useful controlled study of when language identity becomes a shortcut that blocks cross-lingual factual recall, and the unification score is a practical model-selection metric. But the abstract's punchline—'unification is essential for cross-lingual transfer'—is not supported by the paper's own zero-parallel data, and the Gemma significance claim has an internal inconsistency.\n\nWhat's new: the synthetic Petri dish is well built. Data size is matched across parallel-data ratios, held-out templates test genuine generalization rather than memorization, and the informativeness/extractability manipulations are clean causal interventions. The result that balancing attribute frequencies across languages, or making language harder to extract (character tokenizer, more templates), improves transfer is solid and nicely explained. The unification score, computed from residual-stream or gradient similarities on parallel training examples, predicts cross-lingual accuracy with PCC >0.95 in the synthetic runs and is competitive with small OOD test sets for model selection. That is a real contribution.\n\nSoft spots, in order. First, the zero-parallel results: models reach ~99% cross-lingual accuracy on birth-year facts with no parallel data. The paper itself notes in Appendix B.11 that this can be 'for the wrong reasons'—a language-specific attribute prior. That directly undercuts 'unification is essential'; the metric is undefined at 0% parallel, so the headline correlation only covers regimes where parallel examples exist. The claim needs restating: unification is the dominant mechanism when parallel data is present and the attribute prior is weak, not a necessary condition for all apparent transfer. Second, the LLM validation: the main text says unification predicts OOD accuracy with ROC 0.65 and is 'statistically significant at all layers,' but Appendix B.13.5 shows layers 0 and 41 with p≈0.1. That is an internal inconsistency that should be fixed. ROC 0.65 is modest but real; the layerwise caveat matters. Third, no code or data is released, which slows reproducibility of the synthetic setup and the score.\n\nWho should read it: anyone working on multilingual pretraining, shortcut learning, or mechanistic model selection. It deserves a serious referee—the synthetic core is careful and novel, and the overclaim is fixable. I'd engage with it.","headline":"Useful synthetic study and a practical unification metric, but the 'essential' claim overreaches against the paper's own zero-parallel results.","tokens_in":16268,"tokens_out":2798,"would_cite":true,"duration_ms":27322,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that cross-lingual fact recall fails because models learn language as a shortcut, and that a training-time unification score can predict and guide transfer.","keywords":["cross-lingual transfer","factual recall","representational unification","spurious correlations","pretraining dynamics","synthetic languages","unification score","language feature"],"falsifier":"A concrete falsifier: train two models on identical synthetic facts with the same amount of parallel data, one with attribute frequencies balanced across languages and one with language highly informative about attributes; if the balanced model does not show higher unification scores and cross-lingual accuracy, the core claim fails.","tokens_in":15250,"feed_emoji":"🌐","tokens_out":6612,"duration_ms":65391,"temperature":0.7,"pith_summary":"This paper asks why a language model can answer a fact correctly in the language it was trained on but hallucinate when asked the same fact in another language. Using small Transformers trained from scratch on synthetic two-language datasets, the authors show that the failure is not a gap in the facts themselves: the same facts are present in both languages, yet the model develops separate, language-specific representations instead of one shared representation. They identify a specific early learning phase in which this separation happens, and trace it to a single mechanism: when the language of an example is both easy to detect and predictive of the answer distribution, the model learns language as a shortcut and that shortcut dominates its internal representations. They show that unification of representations is necessary for cross-lingual transfer, introduce a unification score computed on training representations that predicts cross-lingual accuracy with Pearson correlation above 0.95, and demonstrate that data interventions that make language less informative or less extractable improve transfer without adding parallel data. This matters because it turns an apparently intractable failure of multilingual models into a measurable, modifiable property of pretraining data.","feed_headline":"Cross-lingual recall fails when language becomes a training shortcut","feed_subtitle":"A training-time unification score predicts which models transfer facts across languages, with correlation above 0.95.","key_machinery":"The load-bearing object is the 'language feature': the identity of the language in which a training example is written, treated as a potentially spurious predictor. Its influence is measured by a language feature footprint, the $R^2$ of a linear regression of hidden-state representations on language identity. The central quantitative tool is the unification score, defined for two languages as the cosine similarity between representations of the same fact in the two languages divided by the baseline similarity between representations of different facts in the same language; it is computed on the parallel (bilingual) training examples using either residual-stream activations or gradients. The mechanism is feature-learning order: following prior theoretical results, features that explain more variance in the training objective are learned earlier and dominate representations, so an easy-to-extract and informative language feature is learned before the fact identity and leaves a persistent footprint. The paper visualizes this as 'checkerboarding' in pairwise similarity matrices, where same-language blocks become more similar than same-fact cross-language blocks during an early checkpoint phase.","core_discovery":"On its own terms, the paper's central claim is that cross-lingual factual recall fails when a model's internal representations of the same fact in two languages remain separate, and that this separation is caused by language operating as a spurious predictive feature during pretraining. In the controlled setup, every model sees the same facts, the same two languages, and the same number of examples; only distributional properties change. Separation occurs when the language of an example is easy to extract (distinct vocabularies, word-level tokens) and informative about the attribute to be predicted (because attribute frequencies differ across languages). Under those conditions the model enters a 'checkerboarding' phase in which same-language examples with different facts look more similar than cross-lingual examples of the same fact, and this phase predicts poor transfer in the final model. The paper further claims that unification is essential for transfer, that a unification score measured on training representations correlates with cross-lingual accuracy above 0.95 and can replace a small cross-lingual test set for model selection, and that the same mechanism appears in Gemma-2-2B, where unification scores predict answer accuracy with an ROC of about 0.65 and vocabulary overlap between languages explains more variance in translation accuracy than whether a fact appears in both languages.","pith_inferences":["Beyond the paper: the unification score could be monitored during real pretraining as an early-warning signal for the formation of knowledge silos, before a cross-lingual evaluation set is available.","Beyond the paper: the results suggest a concrete testable intervention for production pretraining—obscuring script-level or casing-level language cues early in training, or curriculum-balancing attribute distributions—though the paper notes language is a legitimate prior for culture-specific facts.","Beyond the paper: if the mechanism is generic, then any salient, informative surface feature (formatting, domain markers, template identity) should induce similar siloing; this could be tested by re-running the same synthetic setup with domains instead of languages.","Beyond the paper: the strong correlation between unification and transfer in synthetic data raises the possibility of using unification as a training objective or regularizer, something the paper does not propose."],"forward_implications":["Balancing the frequency of attribute values across languages in monolingual data improves cross-lingual transfer without increasing the amount of parallel data, by reducing the informativeness of the language feature.","Making language harder to detect—through character-level tokenization, lowercasing, more templates, or shared vocabulary and scripts—shrinks the language footprint and improves cross-lingual generalization.","Training-time unification scores can be used to select model checkpoints or hyperparameter runs, performing comparably to a small cross-lingual test set, while in-language accuracy alone is a misleading selection signal.","In large models, vocabulary overlap between a source and target language explains a substantial share of variance in cross-lingual factual accuracy, more than whether individual facts have parallel pages.","The same mechanism likely governs transfer across paraphrases and related surface forms, not only across languages, since the separation is driven by any easy, informative surface feature."],"supporting_citations":[{"why":"Supplies the held-out verbalization methodology for measuring generalization rather than string memorization, and the observation that multiple paraphrases improve recall.","marker":"Allen-Zhu, 2024"},{"why":"Provides the theoretical grounding that features are learned in descending order of variance explained, which underpins the language-feature mechanism.","marker":"Saxe et al., 2019"},{"why":"Supplies the notion of feature footprint measured as R^2 of linear regression on representations, used to quantify the language feature's dominance.","marker":"Lampinen et al., 2024"},{"why":"Provides the computationally efficient gradient approximation used to form gradient-based representations and influence estimates.","marker":"Chang et al., 2025"},{"why":"Provides the ECLeKTic dataset used to validate the unification score and vocabulary-overlap analysis on Gemma models.","marker":"Goldman et al., 2025"},{"why":"Defines the Gemma 2 model family in which the paper tests whether unification predicts cross-lingual answer accuracy.","marker":"Riviere et al., 2024"},{"why":"Documents the cross-lingual factual inconsistency and shared-script effects that the paper's mechanism aims to explain.","marker":"Qi et al., 2023"},{"why":"Shows vocabulary overlap improves cross-lingual transfer, a phenomenon the paper reinterprets as reduced language extractability.","marker":"Patil et al., 2022"}],"fun_headline_variants":["Language shortcuts block cross-lingual fact transfer","Separate representations doom cross-lingual recall","Unification score beats test set for transfer prediction","Checkerboarding: when same facts look different across languages","Facts don't travel if language is a trainable cue"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole argument depends on the assumption that what happens with synthetic, invented languages that share no words and no grammar is the same mechanism that makes real multilingual models fail; the Gemma results are suggestive but not proof.","fun_headline_variants_meta":{"raw":{"variants":["Language shortcuts block cross-lingual fact transfer","Separate representations doom cross-lingual recall","Unification score beats test set for transfer prediction","Checkerboarding: when same facts look different across languages","Facts don't travel if language is a trainable cue"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000412,"raw_usage":{"total_tokens":2144,"prompt_tokens":969,"completion_tokens":1175,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":1100}},"tokens_in":585,"tokens_out":1175,"duration_ms":11761,"temperature":1.0,"reasoning_tokens":1100,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:29:00.086015+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete falsifier: train two models on identical synthetic facts with the same amount of parallel data, one with attribute frequencies balanced across languages and one with language highly informative about attributes; if the balanced model does not show higher unification scores and cross-lingual accuracy, the core claim fails.","supporting_citations":[{"cited_title":"ICML 2024 Tutorial: Physics of Language Models , July 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the held-out verbalization methodology for measuring generalization rather than string memorization, and the observation that multiple paraphrases improve recall."},{"cited_title":"A mathematical theory of semantic development in deep neural networks","cited_arxiv_id":null,"evidence_quote":"Provides the theoretical grounding that features are learned in descending order of variance explained, which underpins the language-feature mechanism."},{"cited_title":"Chan, and Katherine Hermann","cited_arxiv_id":null,"evidence_quote":"Supplies the notion of feature footprint measured as R^2 of linear regression on representations, used to quantify the language feature's dominance."}],"review_version":2}