Pith. sign in

REVIEW 4 major objections 6 minor 33 references

Uncovering Cross-Linguistic Disparities in LLMs using Sparse Autoencoders

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that sparse-autoencoder features reveal systematic under-activation of medium-to-low resource languages in Gemma-2-2B, and that fine-tuning to align those activations with English reduces the gap but only modestly…

desk verdict Descriptive SAE comparison is new and worth a look, but the causal claim collapses under probe-selection circularity and benchmark evidence that contradicts the abstract. read the letter →

arxiv 2507.18918 v1 pith:IACZMG77 submitted 2025-07-25 cs.CL cs.AI

Richmond Sin Jing Xuan , Jalil Huseynov , Yang Zhang This is my paper · ORCID
classification cs.CLcs.AI
keywords sparseautoencodersmultilingualLLMsactivationdisparitieslow-resourcelanguagesactivation-awarefine-tuningLoRAGemma-2-2Bcross-lingualbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that multilingual language models can look language-neutral at the embedding level while still being systematically weaker for medium-to-low resource languages inside the model. Using sparse-autoencoder features of Gemma-2-2B across all 26 layers and 10 languages, it reports that non-English languages receive lower feature activations, up to 26.27% lower in early layers and a persistent gap near 20% in deep layers, and that these activation gaps track benchmark gaps on ARC-Challenge, MMLU, and HellaSwag. It then shows that fine-tuning with an activation-alignment loss raises activations for under-resourced languages by large percentages while keeping English activations mostly intact, but downstream accuracy improves only modestly and unevenly. If the paper is right, SAE activation values are a practical diagnostic for cross-linguistic inequality and a target for partial correction, not a guarantee of task-level gains.

What carries the argument

The load-bearing object is the sparse autoencoder (SAE) feature: a decomposition that turns each layer's dense activation vector into a small set of sparse, interpretable unit features. The paper anchors each feature using English phrases whose activation exceeds 80% of the feature's maximum, then measures how strongly the same feature fires on translations of those phrases in nine languages. The quantity doing the work is the per-layer activation gap, the percentage by which the mean activation for medium-to-low resource languages falls below that for high-resource languages; the same gap feeds the fine-tuning loss, which drives target-language activations toward English activations under a stability regularizer.

What would settle it

Select SAE features independently from non-English phrases using the same 80% activation threshold and measure activation of English translations on those indices. If the cross-linguistic activation gap vanishes or becomes symmetric, the reported disparity is an artifact of English-anchored feature selection rather than a model-level representational deficit. A second check: fine-tune with the activation-alignment loss but freeze all English-retention terms and evaluate MMLU; if MMLU still declines, the loss itself, not knowledge drift, is causing the drop.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that Gemma-2-2B systematically activates learned features less for medium-to-low resource languages than for English and other high-resource languages, despite near-equal residual embeddings. The gap peaks at layer 6 with 26.27% lower activation, narrows but never closes deeper in the network at 19.89% at layer 25, and is strongly negatively correlated with benchmark accuracy, with Pearson coefficients around -0.93 to -0.95 across ARC-Challenge, HellaSwag, and MMLU. Fine-tuning with a LoRA objective that penalizes the absolute difference between English and target-language activations while adding a stability term for original English activations produced activation gains of 87.69% for Malayalam and 86.32% for Hindi with roughly 91% English retention. The downstream effect was modest and inconsistent: ARC-Challenge for Malayalam rose 1.44 percentage points, while MMLU and HellaSwag declined, so the paper claims activation alignment is a key factor but not a sufficient one.

Load-bearing premise

The argument assumes that features selected from English phrases are language-neutral sensors, so a lower activation on a translated phrase means the model genuinely represents that content more weakly, rather than the feature simply matching English wording better.

Editorial extensions

If this is right

  • SAE activation magnitude can be used as a layer-wise diagnostic: a 20% or larger activation gap flags a language that is likely to underperform, before running expensive benchmarks.
  • Activation-aware LoRA fine-tuning is a viable cheap intervention: it raised Malayalam activations by 87.69% and Hindi by 86.32% while keeping English activations at about 91%, showing the gap is partially movable.
  • High embedding similarity is not sufficient evidence of equitable multilingual ability; activation-level measurements reveal disparities that cosine similarity hides.
  • Since deeper-layer gaps persist near 20%, shared representations do not automatically erase language-specific deficits; targeted early-layer interventions may be needed.
  • Activation alignment alone does not guarantee task gains: MMLU and HellaSwag declined for Malayalam, so the goal should be alignment plus knowledge preservation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the activation gap is causal rather than merely correlated, then fine-tuning only the early layers where the gap is largest should produce larger benchmark gains than aligning all layers, at lower cost; this is directly testable.
  • The same English-anchored SAE-probe protocol could be run on other model families; comparing gap sizes would show whether the disparity is architecture-driven or data-driven, without needing new benchmark suites.
  • A stricter test of measurement validity would select features independently from each language and compare feature sets; if the same features are not activated across languages, the gap is partly a probe artifact rather than pure under-representation.
  • The MMLU decline after alignment suggests that forcing activation toward English may overwrite factual knowledge; a knowledge-retention term in the loss could separate alignment effects from forgetting effects.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper investigates cross-linguistic activation disparities in Gemma-2-2B using the Gemma Scope sparse autoencoders. English phrases activating each SAE feature above 80% of its maximum are extracted from Neuronpedia and machine-translated into nine languages; activations of the translations are then read at the same English-selected feature indices across all 26 layers (§2.1–§2.5). The authors report that medium-to-low-resource languages receive up to 26.27% lower activations at layer 6 and a persisting 19.89% gap at layer 25 (Table 1), that embedding cosine similarity is high across languages despite large benchmark gaps on ARC-C, MMLU, and HellaSwag, and that the activation gap correlates strongly with benchmark accuracy (r ≈ -0.93 to -0.95; Figures 9–11). A LoRA fine-tuning procedure with the loss f(u_l, v_l) = |u_l - v_l| + α||u_l - u_orig,l||² targeted at layer 20 is then reported to yield activation gains of up to 87.69% (Malayalam) with roughly 91% English activation retention, and the abstract claims 'modest but consistent' benchmark improvements. The central claim is that SAE activation values can diagnose and partially correct cross-linguistic disparities, with activation alignment as a key factor in multilingual performance.

Significance. If the measured disparities were validated, the paper's descriptive finding—that embedding-level similarity coexists with large SAE-activation gaps and large benchmark gaps—would be a useful contribution to multilingual interpretability, and the openly documented, layer-wise activation tables (Table 1) would support follow-up work. The paper makes good use of public resources (Gemma Scope, Neuronpedia, OPUS-MT) and reports full per-language benchmark and activation numbers, and it deserves credit for explicitly documenting in Appendix C.6 that post-fine-tuning benchmark results are inconsistent rather than suppressing them. The falsifiable core—that embedding similarity does not guarantee equal task performance in multilingual LLMs—is clearly and testably stated. However, the significance claimed in the abstract is not currently established: the activation measurement rests on unvalidated English-derived probes, the headline activation gains are the direct optimum of the fine-tuning loss, and the abstract's 'consistent improvements' claim is contradicted by the paper's own Table 9.

major comments (4)
  1. [§2.1, §2.4, Appendix A.1] The activation measurements assume that feature indices selected from English phrases (top-activated phrases above 80% of each feature's maximum, §2.1) are language-neutral probes when activations of machine-translated phrases are read at those same indices (§2.4). The paper supplies no per-language feature-selection control, no translation-invariance check, and no analysis using each language's own top-activating phrases. Appendix A.1 explicitly concedes that SAEs learn composed features that may be language-specific, and Section 4 concedes that Helsinki-NLP translation errors may introduce bias in cross-lingual analysis. Under these conditions, the headline gaps (26.27% at layer 6 and 19.89% at layer 25 in Table 1) and the correlations in Figures 9–11 could reflect a mismatch between the English-derived probe directions and the translated content rather than a systematic language-level activation deficit. Because the activation-gap measurement is the paper's central diagnostic contribution, a per-language validation of the probes is required before the gaps can be interpreted as evidence of under-representation.
  2. [§2.6, Appendix B.1, §3.3, Table 5] The reported activation gains after fine-tuning are the direct optimum of the training objective, so they cannot serve as independent evidence for the paper's intervention claim. The loss f(u_l, v_l) = |u_l - v_l| + α||u_l - u_orig,l||² in §2.6 is exactly the activation gap that Table 5 and Figure 4 report as the main fine-tuning result; with target layer 20 and fine-tuned layers 0–20 (Appendix B.1), the 87.69% Malayalam and 86.32% Hindi gains at layer 20 measure how well gradient descent minimized the training loss, and the fine-tuning dataset is the same as the analysis dataset described in §2.1. These numbers therefore confirm the optimizer's behavior, not the value of activation alignment. The paper should report held-out activation alignment (on phrases and layers not used in training) and should present any layer-20 gain as an optimization sanity check rather than as a discovered effect.
  3. [Abstract, §3.3, Table 9, Appendix C.6] The abstract's claim of 'modest but consistent improvements' is contradicted by the paper's own data. For Malayalam, Table 9 shows +1.44 points on ARC-C but -3.95 points on MMLU and -2.41 points on HellaSwag after fine-tuning, so two of three benchmarks decline and the improvements are not consistent. Appendix C.6 acknowledges this inconsistency and Section 5 states that the effect on task performance 'remains limited,' but the abstract and the §3.3 framing ('demonstrating the positive impact of activation alignment on downstream tasks') do not reflect that qualifier. In addition, the pre-fine-tuning MMLU score for Malayalam is 29.44% in Table 4 but 29.05% in Table 9, an internal inconsistency in the only table that supports the causal claim. As written, the evidence does not support the statement that activation alignment is a key factor in enhancing multilingual performance.
  4. [§3.2, Figures 9–11] The correlations between activation gap and benchmark accuracy (r ≈ -0.95, -0.93, and -0.95 in Figures 9–11) are computed over nine or ten languages and are reported without confidence intervals, p-values, or any check that the relationship survives control for the probe-selection issue in §2.1–§2.4. Because the same English-selected probe set determines the activation values and the language grouping, these correlations cannot be distinguished from a selection artifact, and the very high r-values are implausible for such small samples unless the underlying variable is nearly a monotone transform of benchmark difficulty. The authors should report uncertainty quantification and a per-language feature-control analysis before using these correlations as evidence of a systematic link between activation magnitude and performance.
minor comments (6)
  1. [Tables 4 and 9] Tables 4 and 9 report the pre-fine-tuning Malayalam MMLU accuracy as 29.44% and 29.05%, respectively; the discrepancy should be resolved.
  2. [§2.2, Figure 7] Section 2.2 samples 1,000 feature indices (indices 16i for i = 0..999), while the caption of Figure 7 refers to 4,000 feature indices; the sampling description should be reconciled.
  3. [§2.5, §2.6] Sections 2.5 and 2.6 do not say whether A_l denotes SAE latent activations or residual activations, whether values are mean-pooled over tokens, or how A_high and A_med-low are aggregated over feature indices; the measurement definitions should be stated explicitly for reproducibility.
  4. [Appendix B.1] Appendix B.1 reports 4,000 fine-tuning samples, two iterations, and α = 1.00 but gives no learning rate, LoRA rank, or number of seeds; given that the benchmark deltas are on the order of 1–4 points, standard errors or multiple runs are needed to rule out noise.
  5. [References, §2.3] Okapi is cited via Lai et al. (2023), which describes instruction-tuned models; the reference does not clearly identify an 'evaluation framework,' so the citation should be corrected or replaced with the actual benchmark source.
  6. [Table 5, §3.3] Table 5's caption says 'across ten languages' but the table lists nine non-English languages, and Section 3.3 claims gains 'across all ten languages' while reporting English retention separately; the counting and the meaning of 'retention' should be clarified.

Circularity Check

1 steps flagged · score 6.0 of 10

Activation gains after fine-tuning are the optimized loss value; benchmark evidence is mixed and partly negative, so the central causal claim is only partially supported.

  1. fitted input called prediction [Section 2.6 (fine-tuning loss) and Section 3.3, Table 5]
    "f(ul,v l) =|ul−vl| +α·||ul−uorig,l||2 ... The first term minimizes activation gaps between English and the target language ... Fine-tuning increased activations across all ten languages in layer 20 (Figure 4), with the highest improvements in Marathi (93.87%), Malayalam (87.69%), and Hindi ( 86.32%)."

    The reported activation "gains" are the same quantity that the fine-tuning objective directly minimizes. The loss's first term is |u_l - v_l|, the activation gap between English and the target language on the same SAE feature indices used for measurement; an optimizer that minimizes this term will, by construction, raise v_l toward u_l. Reporting an 87.69% Malayalam activation increase is therefore reporting how well the loss was optimized, not an independent discovery about activation alignment. The only independent evidence, benchmark deltas, is mixed and partly negative (Appendix C.6: ARC-C +1.44, MMLU -3.95, HellaSwag -2.41), so the central causal claim is supported mainly by a tautological quantity.

full rationale

The strongest circularity is in the fine-tuning evaluation. The paper's headline "substantial activation gains" (87.69% Malayalam, 86.32% Hindi) are the direct value of the minimized objective f(u_l,v_l)=|u_l-v_l|+alpha*||u_l-u_orig,l||^2; they are the optimizer's success, not an external confirmation that activation alignment causes better multilingual performance. This is a partial reduction of the paper's evidence to its own training target, warranting score 6. Separately, the SAE probe selection is a real validity risk but not a derivation-chain circularity: Section 2.1 selects features from English top-activating phrases and Section 2.4 reads activations of translated phrases on those same indices, while Appendix A.1 concedes SAEs may learn composed or language-specific features and Section 4 concedes Helsinki-NLP translation errors may introduce bias. If features are not language-neutral, the measured gaps could be artifacts, but this is a measurement-validity concern rather than an equation-level reduction. There is no load-bearing self-citation chain or imported uniqueness theorem; the correlations with benchmarks are independent but, given the mixed post-fine-tuning benchmark results and the unvalidated probe, the abstract's "modest but consistent improvements" and the conclusion that activation alignment is key are not established by the paper's own evidence.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The analysis depends on hand-chosen hyperparameters and on the assumption that English-selected SAE features are valid for cross-lingual comparison. The fine-tuning objective includes the measured activation gap itself, so the activation improvements are fitted rather than predicted. No new theoretical entities are introduced.

free parameters (7)
  • alpha (regularization weight) = 1.00
    Chosen for the fine-tuning loss in Section 2.6; no sensitivity analysis is reported.
  • Fine-tuning data size = 4000 samples
    Used in Section B.1; the paper does not explain how these samples were drawn from the ~6 million phrase dataset.
  • Fine-tuning iterations = 2
    Only two iterations are used, with no convergence check or ablations.
  • Target layer = 20
    Selected in Section B.1 to allow fine-tuning of layers 0-20 while preserving upper layers; the choice is ad hoc.
  • Activation threshold for phrase extraction = 0.80 of max activation
    Used in Section 2.1 to select English phrases for each feature; the threshold is not varied.
  • Feature index sampling = every 16th index, 1000 indices
    Section 2.2 samples indices {16i | i in 0..999}, but Figure 7 refers to 4000 feature indices, an inconsistency.
  • Language group assignment = high-resource: en, zh, ru, es, it; medium-low: id, ca, mr, ml, hi
    Groups are assigned in Section 2.5 based on assumed resource levels, which shapes the activation gap calculation.
assumptions (4)
  • domain assumption SAE features are language-neutral probes.
    Features are selected using English phrases (Section 2.1) and then used to measure activations on translated text (Section 2.4); if features are English-centric, the activation gap is an artifact.
  • domain assumption Helsinki-NLP translations preserve meaning and activation-relevant content.
    The paper acknowledges translation errors can misalign activation patterns in the Limitations section.
  • domain assumption Mean pooling of token activations yields comparable vectors across languages.
    Section 2.2 averages token activations per phrase; languages with different token counts or tokenizers may not be directly comparable.
  • domain assumption The fine-tuning loss does not degrade other model capabilities beyond the measured English retention.
    The benchmark declines for Malayalam on MMLU and HellaSwag suggest general capabilities can be hurt, so the assumption is questionable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Uncovering Cross-Linguistic Disparities in LLMs using Sparse Autoencoders." pith.science (2026). https://pith.science/paper/IACZMG77

@misc{pith2026250718918,
  author       = {Pith},
  title        = {Pith review of: Uncovering Cross-Linguistic Disparities in LLMs using Sparse Autoencoders},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IACZMG77}},
  note         = {Machine review of arXiv:2507.18918}
}
read the original abstract

Multilingual large language models (LLMs) exhibit strong cross-linguistic generalization, yet medium to low resource languages underperform on common benchmarks such as ARC-Challenge, MMLU, and HellaSwag. We analyze activation patterns in Gemma-2-2B across all 26 residual layers and 10 languages: Chinese (zh), Russian (ru), Spanish (es), Italian (it), medium to low resource languages including Indonesian (id), Catalan (ca), Marathi (mr), Malayalam (ml), and Hindi (hi), with English (en) as the reference. Using Sparse Autoencoders (SAEs), we reveal systematic disparities in activation patterns. Medium to low resource languages receive up to 26.27 percent lower activations in early layers, with a persistent gap of 19.89 percent in deeper layers. To address this, we apply activation-aware fine-tuning via Low-Rank Adaptation (LoRA), leading to substantial activation gains, such as 87.69 percent for Malayalam and 86.32 percent for Hindi, while maintaining English retention at approximately 91 percent. After fine-tuning, benchmark results show modest but consistent improvements, highlighting activation alignment as a key factor in enhancing multilingual LLM performance.

Figures

Figures reproduced from arXiv: 2507.18918 by the authors.

Figure 1
Figure 1. Cross-linguistic embedding similarity be [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Performance between high-resource and medium-to-low resource languages across all bench￾marks. 26.34% (Malayalam), and HellaSwag (10-shot) shows a 46.79% gap between English and Malay￾alam, indicating surface-level alignment does not guarantee equitable activation dynamics. Further￾more, medium-to-low resource languages consis￾tently underperform across all benchmarks, partic￾ularly ARC-Challenge and MMLU, where Hin… view at source ↗
Figure 3
Figure 3. Differences in activation between high￾resource and medium-to-low resource languages across layers [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Activation improvements and English reten [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Activation alignment for Malayalam across [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Benchmark performance across all tasks for [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Activation ratios across all layers and 4000 feature indices, comparing activation values for each language [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Comparison of activation differences between high-resource and medium-to-low resource language [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Correlation between activation difference with [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Correlation between activation difference [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Correlation between activation difference [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 8 canonical work pages

  1. [1]

    Ashish Sunil Agrawal, Barah Fazili, and Preethi Jyothi. 2024. https://aclanthology.org/2024.eacl-short.28/ Translation errors significantly impact low-resource languages in cross-lingual learning . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (EACL)

    work page 2024

  2. [2]

    Meta AI. 2024. https://ai.meta.com/blog/meta-llama-3/ Llama 3: Advances in large-scale multilingual training . Meta Research

    work page 2024

  3. [3]

    Evan Anders, Clement Neo, Jason Hoelscher-Obermaier, and Jessica N. Howard. 2024. https://www.lesswrong.com/posts/a5wwqza2cY3W7L9cj/sparse-autoencoders-find-composed-features-in-small-toy Sparse autoencoders find composed features in small toy models

    work page 2024

  4. [4]

    Damian Blasi, Antonios Anastasopoulos, and Graham Neubig. 2021. https://arxiv.org/abs/2110.06733 Systematic inequalities in language technology performance across the world's languages . arXiv preprint arXiv:2110.06733

    arXiv 2021

  5. [5]

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. 2023. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2

    2023

  6. [6]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. https://arxiv.org/abs/1803.05457 Think you have solved question answering? try arc, the ai2 reasoning challenge . arXiv preprint arXiv:1803.05457

    arXiv 2018

  7. [7]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://arxiv.org/abs/1911.02116 Unsupervised cross-lingual representation learning at scale . arXiv preprint arXiv:1911.02116

    arXiv 2020

  8. [8]

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. 2023. https://arxiv.org/abs/2309.08600 Sparse autoencoders find highly interpretable features in language models . arXiv preprint arXiv:2309.08600

    arXiv 2023

Show all 33 references
  1. [9]

    Google DeepMind. 2024. https://arxiv.org/abs/2403.08295 Gemma: Open models based on gemini research and technology . arXiv preprint arXiv:2403.08295

    2024 arXiv

  2. [10]

    Haim Dubossarsky, Ivan Vuli \'c , Roi Reichart, and Anna Korhonen. 2020. https://arxiv.org/abs/2001.11136 The secret is in the spectra: Predicting cross-lingual task performance with spectral similarity measures . arXiv preprint arXiv:2001.11136

    2020 arXiv

  3. [11]

    Philipp Dufter and Hinrich Schütze. 2020. https://arxiv.org/abs/2005.00396 Identifying necessary elements for bert's multilinguality . arXiv preprint arXiv:2005.00396

    2020 arXiv

  4. [12]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. https://arxiv.org/abs/2009.03300 Measuring massive multitask language understanding . arXiv preprint arXiv:2009.03300

    2020 arXiv

  5. [13]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. https://arxiv.org/abs/2106.09685 Lora: Low-rank adaptation of large language models . arXiv preprint arXiv:2106.09685

    2021 arXiv

  6. [14]

    Curt Tigges Joseph Bloom and David Chanin. 2024. Saelens. https://github.com/jbloomAus/SAELens

    2024

  7. [16]

    Rossi, and Thien Huu Nguyen

    Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2023. https://arxiv.org/abs/2307.16039 Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback ....

    2023 arXiv

  8. [17]

    Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. 2024. https://arxiv.org/abs/2408.05147 Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2 . arXiv pre...

    2024 arXiv

  9. [18]

    Weize Liu, Yinlong Xu, Hongxia Xu, Jintai Chen, Xuming Hu, and Jian Wu. 2024. https://arxiv.org/abs/2402.16367 Unraveling babel: Exploring multilingual activation patterns of llms and their applications . arXiv preprint arXiv:2402.16367

    2024 arXiv

  10. [19]

    Neel Nanda, Robert Krzyzanowski, Connor Kissane, Joseph Isaac Bloom, and Arthur Conmy. 2024. https://arxiv.org/abs/2406.17759 Interpreting attention layer outputs with sparse autoencoders . arXiv preprint arXiv:2406.17759

    2024 arXiv

  11. [20]

    OpenAI. 2023. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . arXiv preprint arXiv:2303.08774

    2023 arXiv

  12. [21]

    Indraneil Paul, Goran Glavaš, and Iryna Gurevych. 2024. https://arxiv.org/abs/2403.03894 Ircoder: Intermediate representations make language models robust multilingual code generators . arXiv preprint arXiv:2403.03894

    2024 arXiv

  13. [22]

    Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. https://arxiv.org/abs/1906.01502 How multilingual is multilingual bert? arXiv preprint arXiv:1906.01502

    2019 arXiv

  14. [23]

    Gemma Team: Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan...

    2024 arXiv

  15. [24]

    Robinson, Perez Ogayo, David R

    Nathaniel R. Robinson, Perez Ogayo, David R. Mortensen, and Graham Neubig. 2023. https://arxiv.org/abs/2309.07423 Chatgpt mt: Competitive for high- (but not low-) resource languages . arXiv preprint arXiv:2309.07423

    2023 arXiv

  16. [25]

    Tong Su, Xin Peng, Sarubi Thillainathan, David Guzmán, Surangika Ranathunga, and En-Shiun Annie Lee. 2024. https://arxiv.org/abs/2412.15726 Unlocking parameter-efficient fine-tuning for low-resource language translation . arXiv preprint arXiv:2412.15726

    2024 arXiv

  17. [26]

    Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi, C

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L. Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi,...

    2024

  18. [27]

    o rg Tiedemann, Mikko Aulamo, Daria Bakshandaeva, Michele Boggia, Stig-Arne Gr \

    J \"o rg Tiedemann, Mikko Aulamo, Daria Bakshandaeva, Michele Boggia, Stig-Arne Gr \"o nroos, Tommi Nieminen, Alessandro Raganato, Yves Scherrer, Raul Vazquez, and Sami Virpioja. 2023. https://doi.org/10.1007/s10579-023-09704-w Democratizing neural machine translation with OPU...

    2023 doi

  19. [28]

    J \"o rg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT — B uilding open translation services for the W orld. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation (EAMT), Lisbon, Portugal

    2020

  20. [29]

    true features

    Demian Till. 2024. https://www.lesswrong.com/posts/QoR8noAB3Mp2KBA4B/do-sparse-autoencoders-find-true-features Do sparse autoencoders find "true features"

    2024

  21. [30]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. https://arxiv.org/abs/1905.07830 Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830

    2019 arXiv

  22. [31]

    Hongchuan Zeng, Senyu Han, Lu Chen, and Kai Yu. 2024. https://arxiv.org/abs/2410.11718 Converging to a lingua franca: Evolution of linguistic regions and semantic alignment in multilingual large language models . arXiv preprint arXiv:2410.11718

    2024 arXiv

  23. [32]

    Yang Zhang, Yanfei Dong, and Kenji Kawaguchi. 2024. https://arxiv.org/abs/2409.14381 Investigating layer importance in large language models . arXiv preprint arXiv:2409.14381

    2024 arXiv

  24. [33]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  25. [34]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.