Pith. sign in

REVIEW 3 major objections 6 minor 44 references

Understanding Subword Compositionality of Large Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Most large language models compose subword representations almost exactly by vector addition, recoverable through a single linear map, and the six models tested fall into three distinct composition-strategy groups.

desk verdict Solid layer-wise geometry result on subword addition, but the three-group taxonomy rests on visual curve reading and the probing experiments never state a held-out split. read the letter →

arxiv 2508.17953 v1 pith:NLHJHOWS submitted 2025-08-25 cs.CL cs.AIcs.LG

Qiwei Peng , Yekun Chai , Anders S{o}gaard This is my paper · ORCID
classification cs.CLcs.AIcs.LG MSC 68T5068T07
keywords subwordcompositioncompositionalityProcrustesanalysisprobingclassifiersvectoradditionlargelanguagemodelsisometrywordlengthprediction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish how LLMs build word-level representations from subword pieces, and it reports two related findings. First, in most of the six models tested, the composed representation of a word is isometric to the simple sum of its two subword representations: after fitting an orthogonal linear map, adding subword vectors and looking up the nearest whole-word vector recovers the original word with high precision. Second, the layer-by-layer geometry and probing results sort the six models into three stable groups, suggesting systematic rather than idiosyncratic differences in composition strategy. If the paper is right, subword composition in mainstream LLMs is largely linear, and a model's composition group is a measurable, fairly fixed property that is set during pre-training rather than by instruction tuning.

What carries the argument

Procrustes analysis is the central instrument: given matrices $X$ of composed subword vectors and $Y$ of whole-word vectors, the orthogonal transformation $W^* = UV^T$ obtained from the SVD of $YX^T$ maps the composed space onto the word space, and recoverability is measured by Precision@1 over nearest neighbors on a held-out test split. This converts the claim 'composition behaves like addition' from a metaphor into a falsifiable geometric statement. The probing experiments use logistic regression to classify root versus non-root words and linear regression to predict word length, run layer by layer on both original and addition-composed representations.

What would settle it

Test the same six models on words with three or more subwords, or on an agglutinative language such as Turkish, and recompute Procrustes P@1 for the sum operation: if recoverability drops toward chance, the paper's claims are limited to the two-subword English setting. The design is fully specified in the paper, so the test is runnable without new machinery.

Watch

Extended reading notes

Core claim

The paper claims that, in most of the models tested, the representation a model builds for a whole word from its two subword pieces is, up to a single orthogonal linear transformation, the vector sum of the two subword representations. After fitting that transformation on a training split, summing subword vectors and retrieving the nearest whole-word vector achieves high Precision@1 on a held-out test split, and addition beats multiplication and absolute difference everywhere. The same experiments divide the six models into three groups: Llama3 and Llama3.1 show little linear recoverability outside the embedding layer; Aya-expanse and Gemma2 stay linearly recoverable across nearly all layers; Falcon and Qwen2.5 start recoverable, then drop in middle layers. Probing shows that the root/non-root distinction survives in the composed vectors for all models at all layers, while word length information is reliably recoverable only in some models and some layers, with a consistent late-layer re-emergence. Instruction tuning changes the scores slightly but not the grouping, so the strategies are argued to be fixed during pre-training.

Load-bearing premise

The 3,432-word benchmark only includes English words that split into exactly two subwords, with both subwords present in all six vocabularies; if that selection is not representative of how words are usually tokenized, the observed additivity and the three-group grouping may be artifacts of the sample.

Editorial extensions

If this is right

  • Word-level vectors in most tested LLMs can be approximated well by adding subword vectors and applying one linear map, making linear readouts a viable route to word-level meaning for those models.
  • Because instruction tuning barely changes the patterns, the composition strategy is largely fixed during pre-training, pointing to pre-training data and data mixture rather than fine-tuning as the causal factor.
  • Semantic decomposability is preserved in all models at all layers, whereas surface form is not; for the Aya/Gemma group both content and form survive, making the input more easily derivable from the composed representation.
  • The three-group taxonomy offers a compact diagnostic for tokenizer and representation design: whether a model retains subword traces throughout its depth or abstracts them away early is observable from the layer-wise P@1 curves.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If this additivity generalizes beyond the paper's narrow two-subword English sample, mean-pooling subword embeddings—a common practical shortcut—is a much better approximation of model-internal word representations for some model families than for others.
  • The grouping correlates with model families' data-mixture choices, suggesting a testable hypothesis the paper stops short of: training the same architecture on two different data mixtures should move its composition group.
  • The late-layer re-emergence of word-length information seen in several models may reflect pressure from next-token prediction to reconstruct surface form; one could test this by ablating the final output projection and re-measuring length accuracy.
  • Because root words are already less recoverable than non-root words, a natural extension is to check whether additivity degrades gracefully with subword count; the paper's mechanism predicts it would.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies how six instruction-tuned LLMs (Llama3, Llama3.1, Aya-expanse, Gemma2, Falcon, Qwen2.5) compose subword representations into whole-word representations. Using a parallel dataset of 3,432 English words whose two subword tokens are in all six vocabularies, it applies orthogonal Procrustes analysis to compare simple composition operations (addition, multiplication, absolute difference) against whole-word representations across layers, with the transformation fit on a train split and evaluated on a test split. It also trains logistic and linear probes to predict root/non-root class and word length from composed representations. The paper claims that (i) addition is near-isometric to LLM composition in most models, (ii) content information is preserved across layers and models while form information is preserved only in some models, and (iii) the six models form three compositional groups. The three-group taxonomy is presented as the main contribution.

Significance. If the results hold, the finding that a simple orthogonal alignment makes subword sums retrieve whole-word vectors across layers is a meaningful constraint on how transformer representations compose, and the proposed three-way grouping could guide further analysis of training-data and architectural effects. The geometry analysis is carefully set up in one respect: the Procrustes transformation is fit on a train split and evaluated on a held-out test split, and the comparison of addition, multiplication, and absolute difference is clean. The base-versus-instruction-tuned comparison and the root/non-root breakdown are also valuable. However, the probing experiments lack an explicit held-out evaluation, and the central three-group taxonomy rests on visual inspection without quantitative support, so the paper's broader claims currently outrun the evidence.

major comments (3)
  1. [§4.1–§4.2, Figures 6–7] The probing experiments are described as training logistic and linear probes for three epochs with Adam, but no held-out split or cross-validation is mentioned, unlike the geometry analysis in §3.2. With input dimensionality d≈4096 (3584 for Gemma2) and n=3432, a linear probe can memorize the training labels, so high weighted F1/accuracy on the same examples does not establish that root/non-root or length information is encoded. Since Section 5 uses these curves as corroborating evidence for the three-group taxonomy, the authors should report held-out evaluation or cross-validated performance for every layer and model, and state how the random baselines were computed under the same evaluation scheme.
  2. [§3.3, §5 (Figures 2, 5, 6, 7)] The central claim that the six models fall into three distinct groups is supported only by visual inspection of layer-wise curves; no clustering, pairwise curve-distance measure, or significance test is reported. For instance, in Figure 2 the Llama and Falcon/Qwen curves overlap in early layers, and in Figure 7 the 'same three groups' are asserted without quantifying between-group versus within-group differences. The authors should provide a quantitative grouping, for example silhouette scores over curve features or permutation tests on curve differences, so that the taxonomy is not an impressionistic reading of the plots.
  3. [§3.1 (dataset) and Limitations] The dataset construction—words with exactly two subword tokens and both tokens present in all six vocabularies—likely selects for short, high-frequency, morphologically transparent words, and the paper's own limitations section acknowledges the small, English-only coverage. Because the main claims ('most models', 'three groups') are about composition in general, the absence of any sensitivity analysis (for example, relaxing the two-subword constraint, using per-model vocabularies, or stratifying by frequency or morphological category) leaves open the possibility that the observed additivity and the three-group taxonomy are artifacts of the word sample. Please add at least one robustness check, or explicitly restrict the claims to this dataset in the abstract and contributions.
minor comments (6)
  1. [Abstract and author line] The abstract contains typos ('effective compose' should be 'effectively compose', 'pattens' should be 'patterns'), and the author line 'Y e kun Chai' has an unusual space that should be fixed.
  2. [§3.2] The candidate set for P@1 retrieval is not specified: it should state whether retrieval is over the test-split whole-word vectors, the full vocabulary, or some other set, and whether the target word itself is included in the candidates.
  3. [§3.3 and Figure 2 caption] The phrase 'significantly more similar' is used without a statistical test; if error bars are the only evidence, replace 'significantly' with 'consistently' or report an appropriate significance test over the three runs.
  4. [§4.1] The text says 'Table 3.1' but should refer to Table 1; also, the 'random baseline' should be described precisely, since a classifier matching class proportions and a majority-class baseline are different and the 56% figure should be reproducible from the class distribution.
  5. [§3.3 and Figure 5] The contextualized-subword experiment is underspecified: please describe the exact input format (are the two subwords fed as a two-token sequence?), which hidden states are extracted, and whether the same subword combinations are used as in the non-contextualized setting.
  6. [§5 and Appendix A] The reference to the data-mixture appendix appears only as a footnote; move the pointer into the main text so readers can find the appendix information easily.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the geometry result uses an explicit train/test split for the Procrustes mapping, and the probing analyses are independent measurements despite lacking a stated held-out split.

full rationale

The main claim that subword composition is close to simple addition is established by fitting an orthogonal Procrustes map on a train split and measuring retrieval P@1 on a held-out test split (§3.2: 'The train split is used to find the optimal linear transformation W∗ which will then be applied to the test split for evaluation'). The fitted transformation is therefore not being reused to predict the very data that determined it. The comparison among addition, multiplication, and absolute difference is a direct empirical comparison, not an equation that reduces to its own input. The three-group taxonomy is offered as an interpretive summary of layer-wise P@1, probing, and length-prediction curves; it is not derived by fitting a parameter to the taxonomy labels. The probing sections (§4.1, §4.2) do not state a held-out split, so their reported F1/accuracy should be treated with caution as evidence for content/form retention; however, this is a validation weakness rather than a circular step, because the probe weights are not claimed to be predictions of a quantity defined by those same weights. Self-citations (Chai et al., 2024a; Peng and Søgaard, 2024; Bertolini et al., 2021) occur only in background and method-motivation contexts and are not load-bearing for the experimental conclusions. The appended Limitations section explicitly flags the small English-only dataset and the open question of why the strategies differ, which further indicates that the paper is not hiding its input assumptions. Overall the derivation chain is self-contained with respect to the fitted parameters and external benchmarks.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper's claims rest on two methodological assumptions common in the probing and alignment literature, plus a dataset representativeness assumption that the authors themselves flag as a limitation. No new entities are postulated and no explicit free parameters are fit; the Procrustes transform and probes are trained on data splits.

assumptions (3)
  • domain assumption Procrustes alignment with P@1 retrieval is a valid measure of structural similarity between composed and whole-word vector spaces.
    Section 3.2 assumes that high test P@1 after fitting W implies the model's composition is near-isometric to the tested operation; the paper does not validate against a null model or alternative similarity measures.
  • domain assumption Linear probing (logistic regression and linear regression) reveals whether information is encoded in a representation.
    Sections 4.1 and 4.2 interpret probe accuracy as evidence about preservation of content and form; this is the standard linear-probing assumption, not independently verified here.
  • domain assumption The 3,432 words selected are representative of general two-subword composition behavior.
    Section 3.1 requires both subwords to be in all six vocabularies, which constrains the sample; the Limitations section acknowledges this but no robustness check is run.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding Subword Compositionality of Large Language Models." pith.science (2026). https://pith.science/paper/NLHJHOWS

@misc{pith2026250817953,
  author       = {Pith},
  title        = {Pith review of: Understanding Subword Compositionality of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NLHJHOWS}},
  note         = {Machine review of arXiv:2508.17953}
}
read the original abstract

Large language models (LLMs) take sequences of subwords as input, requiring them to effective compose subword representations into meaningful word-level representations. In this paper, we present a comprehensive set of experiments to probe how LLMs compose subword information, focusing on three key aspects: structural similarity, semantic decomposability, and form retention. Our analysis of the experiments suggests that these five LLM families can be classified into three distinct groups, likely reflecting difference in their underlying composition strategies. Specifically, we observe (i) three distinct patterns in the evolution of structural similarity between subword compositions and whole-word representations across layers; (ii) great performance when probing layer by layer their sensitivity to semantic decompositionality; and (iii) three distinct patterns when probing sensitivity to formal features, e.g., character sequence length. These findings provide valuable insights into the compositional dynamics of LLMs and highlight different compositional pattens in how LLMs encode and integrate subword information.

Figures

Figures reproduced from arXiv: 2508.17953 by the authors.

Figure 1
Figure 1. Illustration of the pipeline of our geometry analysis. All words and subwords exist in models’ vocabulary. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Structural similarity between LLM compo [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Structural similarity between LLM compo [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Structural similarity between LLM composi [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Structural similarity between LLM compo [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Performance (weighted F1) of different LLMs on word type classification across all layers. Orange [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Performance (Accuracy) of different LLMs on word length prediction across all layers. Orange indicates [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: The figure taken from Almazrouei et al. (2023b) that illustrates pre-training data mixture in Fal￾con models. Gemma2 Gemma 2 models (9B) are pre-trained on 8 trillion tokens. These tokens come from a variety of data sources, including web documents, [PITH_FULL_IMAGE:f…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 10 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023 a . Falcon-40B : an open large language model with state-of-the-art performance

    work page 2023

  3. [3]

    Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, M \'e rouane Debbah, \'E tienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. 2023 b . The falcon series of open language models. arXiv preprint arXiv:2311.16867

    arXiv 2023

  4. [4]

    Khuyagbaatar Batsuren, G \'a bor Bella, Aryaman Arora, Viktor Martinovic, Kyle Gorman, Zden e k Z abokrtsk \'y , Amarsanaa Ganbold, S \'a rka Dohnalov \'a , Magda S ev c \'i kov \'a , Kate r ina Pelegrinov \'a , Fausto Giunchiglia, Ryan Cotterell, and Ekaterina Vylomova. 2022. https://doi.org/10.18653/v1/2022.sigmorphon-1.11 The SIGMORPHON 2022 shared tas...

    doi:10.18653/v1/2022.sigmorphon-1.11 2022

  5. [5]

    Khuyagbaatar Batsuren, Ekaterina Vylomova, Verna Dankers, Tsetsuukhei Delgerbaatar, Omri Uzan, Yuval Pinter, and G \'a bor Bella. 2024. Evaluating subword tokenization: Alien subword composition and oov generalization challenge. arXiv preprint arXiv:2404.13292

    arXiv 2024

  6. [6]

    Lorenzo Bertolini, Julie Weeds, David Weir, and Qiwei Peng. 2021. https://doi.org/10.18653/v1/2021.findings-acl.296 Representing syntax and composition with geometric transformations . In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3343--3353, Online. Association for Computational Linguistics

    work page doi:10.18653/v1/2021.findings-acl.296 2021

  7. [7]

    Ned Block. 1981. https://doi.org/10.2307/2184371 Psychologism and behaviorism . Philosophical Review, 90(1):5--43

    work page doi:10.2307/2184371 1981

  8. [8]

    Qi Cao, Takeshi Kojima, Yutaka Matsuo, and Yusuke Iwasawa. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.550 Unnatural error correction: GPT -4 can almost perfectly handle unnatural scrambled text . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8898--8913, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.emnlp-main.550 2023

Show all 44 references
  1. [9]

    Yekun Chai, Yewei Fang, Qiwei Peng, and Xuhong Li. 2024 a . https://doi.org/10.18653/v1/2024.findings-emnlp.86 Tokenization falling short: On subword robustness in large language models . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1582--159...

    2024 doi

  2. [10]

    Yekun Chai, Qingyi Liu, Jingwu Xiao, Shuohuan Wang, Yu Sun, and Hua Wu. 2024 b . https://doi.org/10.18653/v1/2024.emnlp-main.182 Autoregressive pre-training on pixels and texts . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3...

    2024 doi

  3. [11]

    Alexis Conneau, German Kruszewski, Guillaume Lample, Lo \"i c Barrault, and Marco Baroni. 2018. https://doi.org/10.18653/v1/P18-1198 What you can cram into a single \ & ! \# * vector: Probing sentence embeddings for linguistic properties . In Proceedings of the 56th Annual Mee...

    2018 doi

  4. [12]

    John Dang, Shivalika Singh, Daniel D'souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztaj...

    2024 arXiv

  5. [13]

    Ishita Dasgupta, Demi Guo, Andreas Stuhlm \"u ller, Samuel J Gershman, and Noah D Goodman. 2018. Evaluating compositionality in sentence embeddings. arXiv preprint arXiv:1802.04302

    2018 arXiv

  6. [14]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  7. [15]

    Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, et al. 2024. Faith and fate: Limits of transformers on compositionality. Advances in Neural Information Processing Systems, 36

    2024

  8. [16]

    Allyson Ettinger, Ahmed Elgohary, Colin Phillips, and Philip Resnik. 2018. https://aclanthology.org/C18-1152/ Assessing composition in sentence vector representations . In Proceedings of the 27th International Conference on Computational Linguistics, pages 1790--1801, Santa Fe...

    2018

  9. [17]

    Leonidas Gee, Andrea Zugarini, Leonardo Rigutini, and Paolo Torroni. 2022. https://doi.org/10.18653/v1/2022.emnlp-industry.41 Fast vocabulary transfer for language model compression . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: In...

    2022 doi

  10. [18]

    Hongyu Gong, Suma Bhat, and Pramod Viswanath. 2017. Geometry of compositionality. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31

    2017

  11. [19]

    John Hewitt and Christopher D. Manning. 2019. https://doi.org/10.18653/v1/N19-1419 A structural probe for finding syntax in word representations . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Langu...

    2019 doi

  12. [20]

    Josef Klafka and Allyson Ettinger. 2020. https://doi.org/10.18653/v1/2020.acl-main.434 Spying on your neighbors: Fine-grained probing of contextual embeddings for information about surrounding words . In Proceedings of the 58th Annual Meeting of the Association for Computation...

    2020 doi

  13. [21]

    Taku Kudo. 2018. https://doi.org/10.18653/v1/P18-1007 Subword regularization: Improving neural network translation models with multiple subword candidates . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page...

    2018 doi

  14. [22]

    Jiaang Li, Yova Kementchedjhieva, Constanza Fierro, and Anders S gaard. 2024 a . Do vision and language models share concepts? a vector space alignment study. Transactions of the Association for Computational Linguistics, 12:1232--1249

    2024

  15. [23]

    Zhaoyi Li, Gangwei Jiang, Hong Xie, Linqi Song, Defu Lian, and Ying Wei. 2024 b . https://doi.org/10.18653/v1/2024.findings-acl.576 Understanding and patching compositional reasoning in LLM s . In Findings of the Association for Computational Linguistics: ACL 2024, pages 9668-...

    2024 doi

  16. [24]

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. 2024. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173

    2024 arXiv

  17. [25]

    Qiwei Peng and Anders S gaard. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.315 Concept space alignment in multilingual LLM s . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5511--5526, Miami, Florida, USA. Association fo...

    2024 doi

  18. [26]

    Lotz, Emanuele Bugliarello, Elizabeth Salesky, Miryam de Lhoneux, and Desmond Elliott

    Phillip Rust, Jonas F. Lotz, Emanuele Bugliarello, Elizabeth Salesky, Miryam de Lhoneux, and Desmond Elliott. 2023. https://openreview.net/forum?id=FkSp8VW8RjH Language modelling with pixels . In The Eleventh International Conference on Learning Representations

    2023

  19. [27]

    Peter H Sch \"o nemann. 1966. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31(1):1--10

    1966

  20. [28]

    Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5149--5152. IEEE

    2012

  21. [29]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. https://doi.org/10.18653/v1/P16-1162 Neural machine translation of rare words with subword units . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages ...

    2016 doi

  22. [30]

    Chen Shani, Jilles Vreeken, and Dafna Shahaf. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.877 Towards concept-aware large language models . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13158--13170, Singapore. Association for Comput...

    2023 doi

  23. [31]

    Yintao Tai, Xiyang Liao, Alessandro Suglia, and Antonio Vergari. 2024. https://doi.org/10.18653/v1/2024.findings-acl.874 PIXAR : Auto-regressive language modeling in pixel space . In Findings of the Association for Computational Linguistics: ACL 2024, pages 14673--14695, Bangk...

    2024 doi

  24. [32]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  25. [33]

    Gemma Team. 2024 a . https://doi.org/10.34740/KAGGLE/M/3301 Gemma

    2024 doi

  26. [34]

    Qwen Team. 2024 b . Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  27. [35]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  28. [36]

    Changhan Wang, Kyunghyun Cho, and Jiatao Gu. 2020. Neural machine translation with byte-level subwords. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 9154--9160

    2020

  29. [37]

    Dixuan Wang, Yanda Li, Junyuan Jiang, Zepeng Ding, Guochao Jiang, Jiaqing Liang, and Deqing Yang. 2024 a . Tokenization matters! degrading large language models through challenging their tokenization. arXiv preprint arXiv:2405.17067

    2024 arXiv

  30. [38]

    Siyuan Wang, Zhongyu Wei, Yejin Choi, and Xiang Ren. 2024 b . https://doi.org/10.18653/v1/2024.acl-long.406 Can LLM s reason with rules? logic scaffolding for stress-testing and improving LLM s . In Proceedings of the 62nd Annual Meeting of the Association for Computational Li...

    2024 doi

  31. [39]

    Di Wu, Yibin Lei, Andrew Yates, and Christof Monz. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.823 Representational isomorphism and alignment of multilingual large language models . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 14074...

    2024 doi

  32. [40]

    Ningyu Xu, Qi Zhang, Menghan Zhang, Peng Qian, and Xuanjing Huang. 2024. On the tip of the tongue: Analyzing conceptual representation in large language models with reverse-dictionary probe. arXiv preprint arXiv:2402.14404

    2024 arXiv

  33. [41]

    Zhaozhen Xu, Zhijin Guo, and Nello Cristianini. 2023. On compositionality in data embedding. In International Symposium on Intelligent Data Analysis, pages 484--496. Springer

    2023

  34. [42]

    Lang Yu and Allyson Ettinger. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.397 Assessing phrasal representation and composition in transformers . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4896--4907, Online. A...

    2020 doi

  35. [43]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  36. [44]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.