REVIEW 10 cited by
Do Llamas Work in English? On the Latent Language of Multilingual Transformers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We ask whether multilingual language models trained on unbalanced, English-dominated corpora use English as an internal pivot language -- a question of key importance for understanding how language models function and the origins of linguistic bias. Focusing on the Llama-2 family of transformer models, our study uses carefully constructed non-English prompts with a unique correct single-token continuation. From layer to layer, transformers gradually map an input embedding of the final prompt token to an output embedding from which next-token probabilities are computed. Tracking intermediate embeddings through their high-dimensional space reveals three distinct phases, whereby intermediate embeddings (1) start far away from output token embeddings; (2) already allow for decoding a semantically correct next token in the middle layers, but give higher probability to its version in English than in the input language; (3) finally move into an input-language-specific region of the embedding space. We cast these results into a conceptual model where the three phases operate in "input space", "concept space", and "output space", respectively. Crucially, our evidence suggests that the abstract "concept space" lies closer to English than to other languages, which may have important consequences regarding the biases held by multilingual language models.
Forward citations
Cited by 10 Pith papers
-
ChiKhaPo: A Large-Scale Multilingual Benchmark for Evaluating Lexical Comprehension and Generation in Large Language Models
ChiKhaPo is an 8-subtask benchmark that measures word-level comprehension and generation in 2,700+ languages and shows state-of-the-art models perform poorly on low-resource languages.
-
Semantic Primes as Explanans for Emotion in Large Language Models
NSM semantic primes are more recoverable, more causally effective, and behaviorally more interchangeable with emotions than appraisal dimensions in four instruction-tuned LLMs.
-
Controlling the Risk of Corrupted Contexts for Language Models via Early-Exiting
An early-exit rule with a zero-shot fallback, calibrated by Learn-then-Test risk control, keeps the average loss from corrupted in-context demonstrations under a preset bound.
-
BASIC: Boosting Visual Alignment with Intrinsic Refined Embeddings in Multimodal Large Language Models
A self-supervision method makes multimodal LLMs align their input image embeddings with the model's own refined internal representations, improving visual QA scores over LLaVA baselines.
-
CLAIM: Mitigating Multilingual Object Hallucination in Large Vision-Language Models with Cross-Lingual Attention Intervention
An inference-time attention-shift intervention aligns non-English queries' cross-modal attention with English, cutting multilingual object hallucination in LVLMs on POPE and MME.
-
How Programming Concepts and Neurons Are Shared in Code Language Models
In Llama-based code models, programming languages are represented through an English-like intermediate token space, with language-specific neurons concentrated in bottom layers and exclusive PL neurons in top layers; ...
-
Disentangling Language and Culture for Evaluating Multilingual Large Language Models
A new dual-axis evaluation framework shows multilingual LLMs answer culture-specific questions best when the question language matches the cultural context, with partial neuron-level evidence for the effect.
-
Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipeline
LLMs recall facts through an English-centric internal path and then translate the answer; injecting a translation vector and a recall vector raises accuracy by over 35 percentage points in the weakest language.
-
GRAIT: Gradient-Driven Refusal-Aware Instruction Tuning for Effective Hallucination Mitigation
GRAIT selects and reweights refusal-training examples using gradient influence, reporting lower hallucination rates and better helpfulness scores than prior refusal-aware tuning baselines.
-
Language-Specific Layer Matters: Efficient Multilingual Enhancement for Large Vision-Language Models
Selective fine-tuning of language-specific shallow layers identified by neuron activation analysis improves multilingual vision-language performance with only 14% of parameters tuned.
Discussion (0). Continue with ORCID to comment.