Improving Multilingual Language Models by Aligning Representations through Steering

Buddhika Laknath Semage; Omar Mahmoud; Santu Rana; Thommen George Karimpanal

REVIEW 5 major objections 8 minor 44 references

Improving Multilingual Language Models by Aligning Representations through Steering

T0 review · 5 major / 8 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single learned vector added at one layer aligns multilingual LLMs

desk verdict The SFT comparison is internally inconsistent and the layer-selection protocol is undisclosed, so the headline claim is not reliable; the per-model data and layer analysis are still worth a serious look. read the letter →

arxiv 2505.12584 v2 pith:5R6MM4LR submitted 2025-05-19 cs.CL

Omar Mahmoud , Buddhika Laknath Semage , Thommen George Karimpanal , Santu Rana This is my paper · ORCID

classification cs.CL

keywords multilinguallanguagemodelsrepresentationsteeringactivationinterventionresidualstreamcross-lingualtransferdirectpreferenceoptimizationlow-resourcelanguages

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that multilingual failures in large language models are largely a representational alignment problem, and that correcting it does not require further training. Its method learns one steering vector per language and adds it to the residual stream at a single layer during inference. Across 14 languages and five tasks, this intervention beats supervised fine-tuning, 5-shot in-context learning, cross-lingual prompting, and the NLLB translation pipeline, and comes within about three points of Google Translate in aggregate. If true, the result recasts a major piece of multilingual capability as a single-layer geometric correction rather than a data-hungry training problem.

What carries the argument

The steering vector $v \in \mathbb{R}^d$ is a single learnable direction added to the residual stream at the entry of layer $L$, so that each token's activation becomes $x_{\text{target}} + v$. It is learned from English-target sentence pairs either by a direct preference optimization objective that raises the probability of English-aligned responses and lowers the probability of target-language responses, or by minimizing mean squared error between the aligned and English activations. Its work is to shift the whole target-language activation distribution into the model's English-representation subspace in one step, which the paper claims mimics the representational alignment that fine-tuning produces over many epochs.

What would settle it

Re-run the headline comparison selecting the steering layer and vector hyperparameters only on a held-out validation split, never on the test benchmarks; if the reported margins over supervised fine-tuning vanish or reverse under that protocol, the central comparison is an artifact of optimistic layer selection. A second check is to train the steering vector without any English translations and see whether task gains remain, which would reveal how dependent the method is on English supervision.

Watch

Extended reading notes

Core claim

The paper claims that a fixed additive vector $v$, applied as $x_{\text{altered}} = x_{\text{target}} + v$ to every token's residual stream at one selected layer, aligns non-English representations with English closely enough to improve task performance across five open-source LLM families and 14 languages. The vector learned by a directional preference objective (DPO) consistently outperforms MSE-based alignment, transfers between linguistically related languages, and produces internal alignment effects similar to full supervised fine-tuning, as shown by logit-lens inspection. The paper also reports that the best layer is architecture-specific and diagnostic: multilingual models like Aya23 benefit most from early-layer steering, while Llama2 needs early or middle intervention for low-resource languages and is disrupted by late-layer changes.

Load-bearing premise

A single additive vector, identical for every token position and every prompt in a language, can capture the mapping from that language's internal representations to English well enough to improve task behavior; the authors concede that this fixed linear direction limits the mapping's capacity.

Editorial extensions

If this is right

Multilingual improvement becomes possible without gradient updates: only a small bilingual corpus, a few forward passes, and one stored vector are needed at inference time.
A steering vector learned on one language transfers to related languages within the same family, so a small set of vectors could cover language families rather than individual languages.
The layer at which steering helps is diagnostic of how a model organizes languages: early-layer effectiveness indicates a shared language-neutral space, while late-layer ineffectiveness indicates the model has already committed to an interpretation.
Because steering and fine-tuning both act by aligning non-English representations with English, steering can be composed with fine-tuning, or used in settings where fine-tuning data are scarce.
The method offers a cheaper, model-native alternative to external translation pipelines for low- and mid-resource languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If one vector per language suffices, multilingual models may encode a shared English-like subspace plus language-specific offsets, making cross-lingual alignment a low-rank correction problem rather than a deep architectural one.
The layer-sensitivity result could be turned into a pre-training diagnostic: measuring how far a language's early activations are from the model's dominant-language subspace might predict downstream performance gaps without running full benchmarks.
A natural extension is to replace the fixed vector with a per-token or per-prompt vector; the paper's own limitation section notes that a fixed linear direction misses contextual variation, so context-dependent steering should close more of the gap to translation-based systems.
Since the method works best on languages whose representations already partially overlap English, it is most plausibly paired with data interventions for low-resource languages, not treated as a complete substitute for multilingual pretraining.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my read. The paper has a genuinely interesting idea—a single fixed steering vector added to one layer's residual stream—and the per-model tables give it some empirical support. But the headline comparison is not reliable as stated. Table 2's SFT row does not match Table 6: I re-averaged Table 6's SFT numbers per task and got (40.5, 68.7, 55.9, 47.8, 43.2), while Table 2 lists (34.7, 55.2, 46.1, 39.9, 35.4). Every other method (Base, Google, NLLB, 5-shot, XLT, DPO, MSE) is consistent between the two tables. So either the SFT row is from a different run or it's a typo, and it's the exact row that supports the 26.7% claim. With the consistent numbers, DPO beats SFT by about 4–5%, not 26.7%.

The layer-selection gap is also real. The paper never says which layer L produced Table 2. Section 6.1 shows the effect is sharply layer-dependent, and the Limitations concede the fixed linear vector is limited. If L was chosen on the test benchmarks, the margins are upper bounds. The stress-test note holds up.

What I like: the layer-sensitivity analysis is informative and mechanistic—Aya23 works best early, Llama2 early/middle for low-resource languages, final layers useless. The cross-lingual transfer results (Spanish vector helps German/French/Russian but not Japanese) are a clean sanity check. The logit-lens comparison between steering and fine-tuning is a useful qualitative observation. The paper is also honest in its limitations section about the fixed-direction constraint.

What's missing: no direct baseline against Wang et al. 2024, so the single-layer vs all-layer question is not actually answered; no code/data; no error bars. Those are minor-to-moderate, but they add uncertainty.

Overall, this deserves a serious referee: the per-model data are substantial and the layer analysis is valuable, so it would be a shame to desk-reject. But the authors need to fix the SFT row, disclose how L was chosen, and ideally add error bars and code before the claims can be trusted. If the SFT row is a simple error, the paper lands as a modest improvement over prompting/translation baselines rather than a leap over fine-tuning.

Referee Report

5 major / 8 minor

Summary. The paper proposes a lightweight multilingual intervention: for each target language, a single steering vector is learned from paired English/non-English corpora (MSVAMP for math, Tatoeba for general text) and added to the residual stream at one selected layer during inference (Eqs. 4 and 6). The method is evaluated on five open-source LLMs (LLaMA2-7B, Aya23-8B, LLaMA3-8B, Gemma-7B, Qwen1.5-7B) across MGSM, XNLI, XCOPA, M3Exam, and MMLU, against seven baselines including basic prompting, Google Translate, NLLB, 5-shot ICL, XLT, and SFT. The authors report that DPO-based steering improves over SFT by about 26.7%, over 5-shot ICL by about 38.8%, over NLLB by about 25.4%, and approaches Google Translate within about 3%. They also analyze layer sensitivity, cross-lingual transfer of steering vectors, and internal alignment via logit lens, concluding that single-layer steering can reproduce part of the effect of multilingual fine-tuning.

Significance. If the empirical claims hold, the paper is a useful contribution to multilingual representation engineering: a single fixed additive vector at one layer can recover a meaningful portion of the gains from full multilingual fine-tuning, at much lower computational cost, and the layer-dependence results are informative for mechanistic interpretability. The paper is commendably transparent in its Limitations section about the fixed linear direction, the cost of layer sweeping, and the distribution sensitivity of the method, and it provides per-model tables (Table 6) that make the aggregate claims checkable. The main risk is experimental protocol: the headline results are trustworthy only if the steering layer was chosen on a proper validation split and if the DPO objective is specified consistently; both points currently require clarification.

major comments (5)

[Section 6.1 and Table 2] The headline results in Table 2 do not state which layer L was used for each model or language, nor how L was chosen. Section 6.1 shows that the effect of steering is sharply layer-dependent: Aya23 benefits most from early layers, LLaMA2 from early and middle layers for low-resource languages, and final layers give little or no improvement. If L was selected by inspecting the same benchmarks reported in Table 2, the reported margins over SFT, NLLB, and the 3.08% gap to Google Translate are upper bounds from test-set selection rather than estimates of the method with a pre-specified or validation-selected layer. The authors should disclose L for every model and language group, choose L on a held-out validation split (for example, a portion of MSVAMP/Tatoeba not used for training the vector), and report the sensitivity of Table 2 to the choice of L, such as the mean and standard deviation over layers or over a small set of pre-registered layers.
[Appendix A, Eq. (7) versus Algorithm 1] The paper defines the DPO objective in Eq. (7) with a reference-model probability ratio, a temperature beta, and a sign flip d, but Algorithm 1 (line 12) optimizes L_DPO = -E_t log sigma(ell_en_t - ell_t), which contains no reference model, no beta, and no d, and treats logits as if they were probabilities. These two objectives are not equivalent, so the learned steering vector is not reproducible from the text as written. Please provide a single, precise loss with all terms defined, and clarify how the logits ell and ell_en are obtained from the altered and English hidden states, including whether the model is run from layer L+1 onward after the intervention.
[Section 3, Eqs. (4)-(6) and Algorithms 1-2] The additive intervention is underspecified at the sequence level. The paper defines token-wise residual activations x_i^(l), but Eqs. (4) and (6) add a single vector v to 'x_target', and Algorithms 1-2 call HiddenState(M,q,L) and add v to the whole hidden state. If v is added to every token position, the training pairs (q, q_en) have different lengths and the loss must specify a per-position or pooled alignment between the two sequences. If v is instead added to a pooled representation, that contradicts Figure 1, which states that v is added to each token's residual stream. The authors should specify the exact tensor shape, the pooling or alignment procedure, and how the altered hidden state is propagated through the remaining layers.
[Section 4.1, training and evaluation datasets] The MGSM result may overestimate the method because the math steering vector is trained on MSVAMP, a multilingual math-word-problem corpus that is distributionally very close to MGSM. The paper itself shows in Section 5.2 and Figure 8 that steering gains are largest when the test distribution closely matches the training distribution, so the MGSM row in Table 2 is not an out-of-domain test of the method. The authors should quantify the overlap between the MSVAMP training subset and MGSM, for example by checking for duplicate or near-duplicate questions, and should report MGSM results with a vector trained on Tatoeba only, or on another non-math corpus, as a robustness check.
[Section 4.2 and Table 6] The 5-shot and XLT baselines produce implausibly low scores that are inconsistent with typical behavior of these methods, for example Aya23 XLT on XCOPA scores 12.1 versus 81.6 for the basic prompt, and LLaMA2 5-shot on MGSM scores 12.2 versus 19.6 for the basic prompt. Because one headline claim is a 38.8% improvement over 5-shot ICL, the construction of these baselines must be described in detail, including the number and language of the examples, the prompt templates, and how the translation step is performed for XLT. As reported, the comparison to 'competitive baselines' is not yet credible.

minor comments (8)

[Table 6 caption] The caption says 'Three models' but the table contains five models (Aya23-8B, LLaMA2-7B, LLaMA3-8B, Gemma-7B, Qwen1.5-7B); correct the caption and state the language set used for each row.
[Tables 1 and 6] Table 1 and Table 6 report different MGSM averages for the same models, for example LLaMA2-7B Basic Prompt is 13.4 in Table 1 but 19.6 in Table 6; clarify whether these use different language subsets or different task configurations.
[Figure 4 caption] The caption refers to 'Aya32-8B' and should read 'Aya23-8B'.
[Table 1] Values such as '.36' and '.01' should be written as '0.36' and '0.01' for readability.
[Algorithm 1] The notation Logits(M, ~h) is undefined; specify whether the model is run from layer L+1 onward with the altered hidden state replacing the residual stream at that layer.
[Throughout] The term 'probing' is used to describe steering in the Limitations section and elsewhere; since probing usually means reading or analyzing representations, clarify the terminology to avoid confusion.
[Throughout] Spelling of the model name is inconsistent ('LLama2', 'Llama2', 'LLaMA2'); choose one convention and use it consistently.
[Appendix A, Eq. (7)] The symbol d is used both as a direction coefficient and as the hidden dimension; use a different symbol, such as s, for the direction coefficient.

Circularity Check

1 steps flagged · score 1.0 of 10

Central benchmark claim is not circular; the only self-definitional moment is the logit-lens alignment illustration, which restates the training objective.

self definitional [Section 7 (logit-lens analysis), using Eqs. (4)-(6) and Appendix A Eq. (7)]
"Both fine-tuning and steering improve representational alignment with English: fine-tuning achieves this gradually over many steps, while steering provides an immediate, targeted correction."

The steering vector is learned precisely to move target-language representations toward English: Eq. (5) minimizes MSE(x_en, x_target + v), and Eq. (7) optimizes v to favor English responses r_T over target-language responses r_O. The Section 7 logit-lens experiment then reports that adding v makes the internal state more English-like. That is a check that the optimization objective was achieved, not an independent confirmation that single-layer steering reproduces fine-tuning. The downstream benchmark results in Table 2 remain independent because their labels never enter the vector fit, so this self-definitional demonstration is not load-bearing for the headline claim.

full rationale

The main derivation chain is self-contained: the steering vector v is fit on MSVAMP/Tatoeba bilingual pairs (Algorithms 1-2) and evaluated on MGSM, XNLI, XCOPA, M3Exam, and MMLU, whose labels do not enter the fit. The gains over SFT, 5-shot, XLT, and NLLB are therefore empirical outcomes rather than algebraic consequences of the training loss. The only reduction I identified is the Section 7 logit-lens demonstration, which verifies that a vector trained to maximize English alignment yields English-aligned representations; it is a sanity check of the objective and I score it as minor. The MSVAMP/MGSM domain overlap is acknowledged by the authors in Section 5.2 and Figure 8 and is a distribution-match issue, not label leakage. The undisclosed choice of steering layer for Table 2 is a transparency/selection risk, but the paper never states that the layer was selected on the test benchmarks, so under the no-speculation rule it is not a circularity finding. Prior-work citations (Cao et al. 2024; Park et al. 2023; Wang et al. 2024) are external, and no load-bearing self-citation chain appears.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The steering vector itself is the main fitted object: one vector per language per model, learned on 1,000 bilingual pairs. The injection layer is a per-model choice made by sweeping, with no disclosed validation. Hyperparameters beta, learning rates, and epoch counts are adopted or hand-tuned (Appendix C). The method rests on three transferred domain assumptions: the linear-representation hypothesis, the validity of a preference-optimization objective for alignment, and cosine similarity as a proxy for translation quality. No invented entities, such as new forces, particles, or conserved quantities, are introduced.

free parameters (5)

Steering vector v per language per model = not reported (dimension equals model hidden size)
Learned by DPO or MSE on 1,000 MSVAMP/Tatoeba pairs per language (Equations 5 and 7); the fitted vector is the mechanism behind every reported gain.
Steering layer L per model = not stated for headline results; ablation shows early layers for Aya23 and early-to-middle layers for Llama2…
Selected by sweeping layers (Section 6.1); if chosen on test performance, the gains in Tables 2 and 6 are optimistic.
DPO temperature beta = 0.1
Borrowed from Cao et al. 2024 (Appendix C); no sensitivity analysis is provided.
MSE training epochs and learning rate = learning rate 1e-8; epochs varied per language in the range [3, 5, 8, 12]
Chosen by hand per language (Appendix C); the range of epochs indicates tuning that is not fully specified.
Training sample size = 1,000 instances per language
Fixed sampling choice from MSVAMP and Tatoeba; no sensitivity analysis is reported.

assumptions (4)

domain assumption A single additive vector at one layer can align a target language's representations with English (linear representation hypothesis).
Invoked at Equations 4 and 6; the paper's own Limitations state the fixed linear direction limits its capacity to capture the intricate mapping relationships between languages fully.
domain assumption The DPO preference objective, designed for behavior steering (Cao et al. 2024), is a valid objective for learning representational language alignment.
Equation 7 transfers a personalization objective to the multilingual setting without a derivation that its optimum corresponds to the desired representational alignment.
domain assumption Cosine similarity between a language's activations and English activations is a valid proxy for internal translation quality and performance readiness.
Used in Section 5.1 and Figure 2 to explain performance gaps; the paper does not validate the proxy against task accuracy per language.
domain assumption The five benchmarks (MGSM, XNLI, XCOPA, M3Exam, MMLU) provide comparable cross-lingual measurements of the claimed capability.
Standard benchmark-practice assumption; the paper averages over tasks and languages without reporting per-language or per-task agreement.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Improving Multilingual Language Models by Aligning Representations through Steering." pith.science (2026). https://pith.science/paper/5R6MM4LR

@misc{pith2026250512584,
  author       = {Pith},
  title        = {Pith review of: Improving Multilingual Language Models by Aligning Representations through Steering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5R6MM4LR}},
  note         = {Machine review of arXiv:2505.12584}
}

read the original abstract

This paper investigates how Large Language Models (LLMs) represent non-English tokens -- a question that remains underexplored despite recent progress. We propose a lightweight intervention method using representation steering, where a learned vector is added to the residual stream at a single model layer to enhance multilingual performance. Through extensive experiments across seven competitive baselines -- including prompt optimization, supervised fine-tuning (SFT), in-context learning, cross-lingual transfer, and translation-based methods-we show that our approach consistently outperforms most alternatives. In particular, it achieves performance on par with production-grade translation systems while requiring far fewer resources. We further explore the complementarity between our method and SFT, demonstrating that steering offers a direct, efficient way to realign internal representations. These findings underscore the potential of activation-level interventions as a powerful tool for improving the multilingual capabilities of LLMs.

Figures

Figures reproduced from arXiv: 2505.12584 by the authors.

**Figure 2.** Similarity scores between language and En [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Scores after applying steering vectors trans [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: demonstrate that using learnable steering vectors surpasses PCA and the Mean Difference approaches [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]

**Figure 5.** Figure 5: illustrates the processing of multilingual tokens in models of varying capabilities within the same family. [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]

**Figure 6.** Figure 6: : Distribution of non-English token values across model layers at three different stages: pre-fine-tuning [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]

**Figure 7.** Figure 7: illustrates the layers most sensitive to probing across two models. Aya23 demonstrates high sensitivity in [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

**Figure 8.** Figure 8: The sub-figures illustrate the distribution of the training and testing datasets across various tasks, [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 3 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Annah and shash42. 2023. https://www.lesswrong.com/posts/JCgs7jGEvritqFLfR/evaluating-hidden-directions-on-the-utility-dataset Evaluating hidden directions on the utility dataset . Accessed: 2025-02-13

work page 2023
[4]

Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, Kelly Marchisio, Max Bartolo, Sebastian Ruder, Acyr Locatelli, Julia Kreutzer, Nick Frosst, Aidan Gomez, Phil Blunsom, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker. 2024. https://arxiv.org/abs/2405.15032 Ay...

arXiv 2024
[5]

Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. 2023. https://arxiv.org/abs/2303.08112 Eliciting latent predictions from transformers with the tuned lens . Preprint, arXiv:2303.08112

arXiv 2023
[6]

Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165

arXiv 2020
[7]

Yuanpu Cao, Tianrong Zhang, Bochuan Cao, Ziyi Yin, Lu Lin, Fenglong Ma, and Jinghui Chen. 2024. Personalized steering of large language models: Versatile steering vectors through bi-directional preference optimization. arXiv preprint arXiv:2406.00045

arXiv 2024
[8]

Nuo Chen, Zinan Zheng, Ning Wu, Linjun Shou, Ming Gong, Yangqiu Song, Dongmei Zhang, and Jia Li. 2023 a . https://arxiv.org/abs/2310.20246 Breaking language barriers in multilingual mathematical reasoning: Insights and observations . Preprint, arXiv:2310.20246

arXiv 2023

Show all 44 references

[9]

Pinzhen Chen, Shaoxiong Ji, Nikolay Bogoychev, Andrey Kutuzov, Barry Haddow, and Kenneth Heafield. 2023 b . Monolingual or multilingual instruction tuning: Which makes a better alpaca. arXiv preprint arXiv:2309.08958

2023 arXiv
[10]

Bowman, Holger Schwenk, and Veselin Stoyanov

Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Asso...

2018
[11]

Marta R Costa-juss \`a , James Cross, Onur C elebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672

2022 arXiv
[12]

John Dang, Arash Ahmadian, Kelly Marchisio, Julia Kreutzer, Ahmet \"U st \"u n, and Sara Hooker. 2024. Rlhf can speak many languages: Unlocking multilingual preference optimization for llms. arXiv preprint arXiv:2407.02552

2024 arXiv
[13]

Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez de Lacalle, and Mikel Artetxe. 2023. Do multilingual language models think better in english? arXiv preprint arXiv:2308.01223

2023 arXiv
[14]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, and Abhishek Kadian. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

2024 arXiv
[15]

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

2020 arXiv
[16]

Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting. arXiv preprint arXiv:2305.07004

2023 arXiv
[17]

Zixian Huang, Wenhao Zhu, Gong Cheng, Lei Li, and Fei Yuan. 2024. Mindmerger: Efficient boosting llm reasoning in non-english languages. arXiv preprint arXiv:2405.17386

2024 arXiv
[18]

Tianlong Li, Xiaoqing Zheng, and Xuanjing Huang. 2024 a . Open the pandora's box of llms: Jailbreaking llms through representation engineering. arXiv preprint arXiv:2401.06824

2024 arXiv
[19]

Zihao Li, Yucheng Shi, Zirui Liu, Fan Yang, Ali Payani, Ninghao Liu, and Mengnan Du. 2024 b . Quantifying multilingual performance of large language models across languages. arXiv preprint arXiv:2404.11553

2024 arXiv
[20]

Chaoqun Liu, Wenxuan Zhang, Yiran Zhao, Anh Tuan Luu, and Lidong Bing. 2024. Is translation all you need? a study on solving multilingual tasks with large language models. arXiv preprint arXiv:2403.10258

2024 arXiv
[21]

Dawn Lu and Nina Rimsky. 2024. https://arxiv.org/abs/2402.00402 Investigating bias representations in llama 2 chat via activation steering . Preprint, arXiv:2402.00402

2024 arXiv
[22]

nostalgebraist. 2020. interpreting gpt: the logit lens

2020
[23]

Ayomide Odumakinde, Daniel D'souza, Pat Verga, Beyza Ermis, and Sara Hooker. 2024. Multilingual arbitrage: Optimizing data pools to accelerate multilingual progress. arXiv preprint arXiv:2408.14960

2024 arXiv
[24]

Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. 2024. https://arxiv.org/abs/2312.06681 Steering llama 2 via contrastive activation addition . Preprint, arXiv:2312.06681

2024 arXiv
[25]

Kiho Park, Yo Joong Choe, and Victor Veitch. 2023. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658

2023 arXiv
[26]

Edoardo Maria Ponti, Goran Glava s , Olga Majewska, Qianchu Liu, Ivan Vuli \'c , and Anna Korhonen. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.185 XCOPA : A multilingual dataset for causal commonsense reasoning . In Proceedings of the 2020 Conference on Empirical Method...

2020 doi
[27]

Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.163 Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages . In Proceedings of the 2023 Conference on Empirical Methods in Na...

2023 doi
[28]

Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip S Yu. 2024. Multilingual large language model: A survey of resources, taxonomy and frontiers. arXiv preprint arXiv:2404.04925

2024 arXiv
[29]

Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. 2022. Language models are multilingual chain-of-thought reasoners. arXiv preprint arXiv:2210.03057

2022 arXiv
[30]

Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex...

2024 arXiv
[31]

Qwen Team. 2024. https://qwenlm.github.io/blog/qwen1.5/ Introducing qwen1.5

2024
[32]

J \"o rg Tiedemann. 2020. https://aclanthology.org/2020.wmt-1.139 The tatoeba translation challenge -- realistic data sets for low resource and multilingual MT . In Proceedings of the Fifth Conference on Machine Translation, pages 1174--1182, Online. Association for Computatio...

2020
[33]

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

2023 arXiv
[34]

Haoran Wang and Kai Shu. 2024. https://arxiv.org/abs/2311.09433 Trojan activation attack: Red-teaming large language models using activation steering for safety-alignment . Preprint, arXiv:2311.09433

2024 arXiv
[35]

Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do llamas work in english? on the latent language of multilingual transformers. arXiv preprint arXiv:2402.10588

2024 arXiv
[36]

Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Yixuan Gao, Annan Wang, Erli Zhang, Wenxiu Sun, Qiong Yan, Xiongkuo Min, Guangtao Zhai, and Weisi Lin. 2023. https://arxiv.org/abs/2312.17090 Q-align: Teaching lmms for visual scoring via discrete t...

2023 arXiv
[37]

Dongkeun Yoon, Joel Jang, Sungdong Kim, Seungone Kim, Sheikh Shafayat, and Minjoon Seo. 2024. Langbridge: Multilingual reasoning without multilingual supervision. arXiv preprint arXiv:2401.10695

2024 arXiv
[38]

Wenxuan Zhang, Sharifah Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2023. https://arxiv.org/abs/2306.05179 M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models . Preprint, arXiv:2306.05179

2023 arXiv
[39]

Yuanchi Zhang, Yile Wang, Zijun Liu, Shuo Wang, Xiaolong Wang, Peng Li, Maosong Sun, and Yang Liu. 2024. https://arxiv.org/abs/2402.12204 Enhancing multilingual capabilities of large language models through self-distillation from resource-rich languages . Preprint, arXiv:2402.12204

2024 arXiv
[40]

Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. 2024. https://arxiv.org/abs/2402.18815 How do large language models handle multilingualism? Preprint, arXiv:2402.18815

2024 arXiv
[41]

Chengzhi Zhong, Fei Cheng, Qianying Liu, Junfeng Jiang, Zhen Wan, Chenhui Chu, Yugo Murawaki, and Sadao Kurohashi. 2024. https://arxiv.org/abs/2408.10811 Beyond english-centric llms: What language do multilingual language models think in? Preprint, arXiv:2408.10811

2024 arXiv
[42]

Wenhao Zhu, Shujian Huang, Fei Yuan, Shuaijie She, Jiajun Chen, and Alexandra Birch. 2024. Question translation training for better multilingual reasoning. arXiv preprint arXiv:2401.07817

2024 arXiv
[43]

Wenhao Zhu, Yunzhe Lv, Qingxiu Dong, Fei Yuan, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023. Extrapolating large language models to non-english by aligning languages. arXiv preprint arXiv:2308.04948

2023 arXiv
[44]

Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. 2023. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405

2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Annah and shash42. 2023. https://www.lesswrong.com/posts/JCgs7jGEvritqFLfR/evaluating-hidden-directions-on-the-utility-dataset Evaluating hidden directions on the utility dataset . Accessed: 2025-02-13

work page 2023

[4] [4]

Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, Kelly Marchisio, Max Bartolo, Sebastian Ruder, Acyr Locatelli, Julia Kreutzer, Nick Frosst, Aidan Gomez, Phil Blunsom, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker. 2024. https://arxiv.org/abs/2405.15032 Ay...

arXiv 2024

[5] [5]

Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. 2023. https://arxiv.org/abs/2303.08112 Eliciting latent predictions from transformers with the tuned lens . Preprint, arXiv:2303.08112

arXiv 2023

[6] [6]

Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165

arXiv 2020

[7] [7]

Yuanpu Cao, Tianrong Zhang, Bochuan Cao, Ziyi Yin, Lu Lin, Fenglong Ma, and Jinghui Chen. 2024. Personalized steering of large language models: Versatile steering vectors through bi-directional preference optimization. arXiv preprint arXiv:2406.00045

arXiv 2024

[8] [8]

Nuo Chen, Zinan Zheng, Ning Wu, Linjun Shou, Ming Gong, Yangqiu Song, Dongmei Zhang, and Jia Li. 2023 a . https://arxiv.org/abs/2310.20246 Breaking language barriers in multilingual mathematical reasoning: Insights and observations . Preprint, arXiv:2310.20246

arXiv 2023