Pith. sign in

REVIEW 3 major objections 6 minor 46 references

Rethinking the Understanding Ability across LLMs through Mutual Information

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes that a language model's understanding is the mutual information between an input sentence and its embedding, and that token recoverability from the embedding is a computable lower bound on that quantity.

desk verdict Useful empirical probe, but the Fano lower-bound claim breaks on real text; fix the theory or call it a heuristic. read the letter →

arxiv 2505.23790 v1 pith:643I5SYC submitted 2025-05-25 cs.CL cs.AI

Shaojie Wang , Sirui Ding , Na Zou This is my paper
classification cs.CLcs.AI
keywords mutualinformationlanguageunderstandingtoken-levelrecoverabilitysentenceembeddingsencoder-onlymodelsdecoder-onlyFano'sinequalityinformation-theoreticevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes an information-theoretic definition of a language model's understanding: the mutual information between an input sentence and the model's latent representation of it, written $I(S;E)$. Because this quantity is hard to compute directly, the authors decompose it into token-level mutual information and derive a computable lower bound, namely token-level recoverability, the probability that a decoder can reconstruct the original tokens from the sentence embedding. Using this bound as a measurement, they report that encoder-only models preserve markedly more input information than decoder-only models, and that decoder-only models show an inverted-U curve across layers, first increasing then discarding token information. They further show that fine-tuning to maximize token-level recoverability improves performance on classification, retrieval, similarity, and clustering tasks without task-specific supervision.

What carries the argument

The central object is sentence-level mutual information $I(S;E)$ between a sentence $S$ and its embedding $E$, operationalized through token-level recoverability $P_{rec}(S,E)$, the maximum probability that any decoder can correctly recover original tokens from the embedding. The argument is carried by two inequalities: a compositional lower bound (Theorem 1) linking sentence-level MI to the average token-level MI, and a Fano-derived lower bound (Theorem 2) linking token-level MI to recoverability. In the experiments, a linear embedding decoder trained with binary cross-entropy supplies the recoverability values that stand in for MI; the linear choice is justified by prior evidence of high linearity across transformer layers.

What would settle it

Take a fixed test corpus, estimate the empirical entropy $H(t_i)$ of each token from its corpus frequency, and recompute the right side of Theorem 2 with $H(t_i)$ replacing $\log |V|$; if the inequality reverses or the model ranking changes materially, the uniform-token premise is doing the work. A second check: replace the linear decoder with a high-capacity nonlinear decoder on the same embeddings; if the ranking reverses, recoverability partly measures decoder capacity rather than representation fidelity.

Watch

Extended reading notes

Core claim

The paper's central claim is that language understanding capacity can be quantified as $I(S;E)$, the mutual information between a sentence and its embedding, and that this quantity is accessible in practice through token-level recoverability. Theorem 1 lower-bounds sentence-level MI by the average token-level MI plus the entropy-deviation term $\Delta$, while Theorem 2 uses Fano's inequality to lower-bound token-level MI by $P_{rec}(S,E)\log(|V|-1)-H_b(P_{rec}(S,E))$ under a uniform-token assumption. The authors interpret measured recoverability as an architecture-agnostic fidelity measure, and their experiments show that encoder-only models dominate decoder-only models on this measure, that decoder-only layers exhibit an inverted-U 'forgetting' pattern, and that recoverability-based fine-tuning transfers to classification, retrieval, semantic similarity, and clustering without task-specific supervision.

Load-bearing premise

The framework depends on the assumption in Theorem 2 that every token in the vocabulary is equally likely, so each token carries a fixed amount of information, $H(t_i)=\log |V|$; real language is strongly non-uniform, and if that assumption fails the recoverability numbers may not be true lower bounds on mutual information.

Editorial extensions

If this is right

  • A shared recoverability task can rank models of different architectures without any task labels, making MI a task-agnostic and model-agnostic evaluation.
  • Encoder-only models should retain more input information than decoder-only models across domains, matching the reported tables.
  • The intermediate layers of decoder-only models are the most information-rich, so embedding extraction for transfer should target those layers rather than the final layer.
  • Fine-tuning with the recoverability objective should raise downstream understanding performance without task-specific labels, as reported on four task families.
  • Comparisons must control for input length, because recoverability falls as texts grow and the encoder-decoder gap widens.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the uniform-token assumption in Theorem 2 is relaxed to real, non-uniform token distributions, the recoverability numbers are no longer guaranteed to be lower bounds on mutual information; the model ranking may still be meaningful but would need separate justification. This is an editorial inference.
  • Because the decoder is trained separately for each model and dataset, cross-model comparisons may partly reflect decoder trainability rather than representation fidelity; holding decoder capacity fixed or using a frozen universal decoder would separate the two. This is an editorial inference.
  • The late-layer forgetting pattern suggests that next-token prediction pressure discards token information; adding a reconstruction or recovery term to pretraining might retain that information and improve downstream understanding. This is a testable extension not explored in the paper.
  • The framework could be applied layer-wise as a diagnostic for knowledge editing or safety interventions: a model that has 'forgotten' token information in later layers may be less faithful to its inputs and more prone to hallucination. This is an editorial inference.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes an information-theoretic framework for evaluating the 'understanding ability' of LLMs. It defines understanding as the mutual information I(S;E) between an input sentence S and its latent embedding E, then decomposes this sentence-level MI into token-level MI terms. The main theoretical contribution is Theorem 2, which claims a computable lower bound on average token-level MI in terms of token-level recoverability, derived via Fano's inequality under an explicit assumption that tokens are uniformly distributed over the vocabulary. The authors implement recoverability with a linear decoder trained on DBpedia, measure it across seven encoder-only and decoder-only LLMs on several domains, study layer-wise patterns, and fine-tune three decoder-only LLMs to maximize recoverability, reporting improvements on classification, STS, retrieval, and clustering tasks. The paper concludes that encoder-only models preserve more input information, that decoder-only models exhibit a late-layer 'forgetting' pattern, and that recoverability-based fine-tuning enhances understanding.

Significance. If the theoretical link were valid, the framework would be attractive: it offers a task-agnostic, architecture-agnostic measure and a new unsupervised fine-tuning objective that does not require task labels. The empirical observations—especially the encoder/decoder asymmetry and the late-layer forgetting trajectory—are interesting and potentially useful for model analysis and representation learning. The paper includes clearly stated definitions, explicit proofs in appendices, and a reproducible experimental protocol. However, the central theoretical guarantee (Theorem 2) rests on an unrealistic uniformity assumption, and as shown below the derived bound can exceed the information-theoretic maximum, so the recoverability numbers are not actually lower bounds on token-level mutual information for natural language. This undermines the interpretation of the measurements and the claim that fine-tuning increases mutual information. The empirical findings may still be valuable as a heuristic proxy, but the paper's core theoretical contribution needs substantial revision.

major comments (3)
  1. [Section 3.2, Theorem 2 and Appendix B] The derivation of the recoverability lower bound assumes H(t_i)=log|V|, i.e., that tokens are uniformly distributed over the vocabulary. Natural language token distributions are strongly non-uniform; for typical subword vocabularies of size ~50k, log2|V| is about 15.6 bits while the marginal entropy of a token in context is typically 6-8 bits. The correct application of Fano's inequality gives I(t_i;E) >= H(t_i) - Hb(Pe,i) - Pe,i log(|V|-1). Replacing H(t_i) by log|V| is not a lower bound. For example, with |V|=50,257 and Prec=0.87 (reported for RoBERTa-large on AG_NEWS in Table 1), the paper's bound yields about 13.0 bits, which exceeds any possible I(t_i;E) because I(t_i;E) <= H(t_i) <= ~8 bits. The theorem is therefore false as stated for real text, and the empirical recoverability values cannot be interpreted as lower bounds on token-level MI. The authors must either estimate H(t_i) and use the corrected Fano bound, or explicitly reframe recoverability as a heuristic proxy without a lower-bound guarantee.
  2. [Section 5.4] The fine-tuning experiment optimizes token-level recoverability and then reports improved downstream performance. Even if Theorem 2 were valid, increasing a lower bound on MI does not imply increasing the actual MI. Because the theorem is invalid for natural language (see first major comment), the experiment demonstrates only that recoverability training transfers to tasks; it does not establish that mutual information itself was enhanced. To support the claim that MI enhancement causes the improvements, the authors should directly measure MI or a corrected lower bound before and after fine-tuning, or substantially soften the conclusion to an empirical statement about the recoverability proxy.
  3. [Section 1 and overall framing] The paper defines understanding as I(S;E), then uses recoverability as a proxy for MI, then validates the approach by showing that training to maximize recoverability improves tasks. This structure risks circularity: the operational measure is the same quantity being optimized and then measured. The theoretical lower bound is supposed to break the circularity, but since it fails, the architecture-agnostic and information-theoretic interpretation is not independently supported. The paper should separate the definitional claim from the operational measure and provide a validation that does not presuppose the identity between recoverability and understanding.
minor comments (6)
  1. [Table 1 caption] The training dataset is spelled 'Depedia' in the caption; it should be 'DBpedia' (the correct spelling appears in Section 4.1).
  2. [Section 5.4, first sentence] The phrase 'To empirically validate our theoretical framework empirically' is redundant; remove the second 'empirically'.
  3. [Figure 2] The x-axis 'Normalized Depth' is not precisely defined in the caption. Please state which layer corresponds to 0 and which to 1 (e.g., embedding layer vs. final transformer layer) for reproducibility, and whether the normalization is the same for encoder-only and decoder-only models.
  4. [Table 3] The STS improvements are very small (e.g., 0.8778 to 0.8784 for OPT-2.7B). Without standard deviations or significance tests, it is unclear whether these gains are meaningful; please report variance or confidence intervals, or mark statistically significant differences.
  5. [Tables 1 and 2] The test set size is only 200 samples, yet metrics are reported to four decimal places (e.g., COS 0.9544). This level of precision is misleading without confidence intervals; either report fewer decimals or provide error bars.
  6. [Equation (4)] The loss uses binary cross-entropy with logits for multi-class token prediction over a large vocabulary, with a one-hot label matrix. Please clarify why sigmoid-based BCE was chosen instead of softmax cross-entropy, since the labels are mutually exclusive by construction; this choice can affect the decoder's calibration and, consequently, the reported Prec values.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the MI–recoverability link is a substantive (though possibly flawed) Fano bound, not a definitional equivalence.

full rationale

The paper's derivation chain is not circular. Understanding capacity is defined as sentence-level mutual information (Section 1), then connected to token-level mutual information via a chain-rule bound (Theorem 1), and to token-level recoverability via Fano's inequality (Theorem 2). Recoverability is not defined as MI; it is related to MI through a mathematical theorem. The fine-tuning experiment optimizes recoverability directly and then measures held-out downstream task performance, which is external validation rather than a fitted parameter renamed as a prediction. There are no load-bearing self-citations or imported uniqueness claims. The main weaknesses are correctness concerns: Theorem 2 assumes H(t_i) = log|V|, which is false for natural language token distributions, and a lower bound does not by itself imply monotonic correspondence between recoverability and MI. These issues undermine the validity of the MI interpretation, but they do not make the derivation circular, because no equation is equivalent to another by construction and the empirical claims are not statistically forced by the training procedure.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on two domain assumptions that are plausible but not justified from first principles: the identification of understanding with mutual information, and the uniform-token assumption used in the Fano-based bound. The linear decoder and fine-tuning hyperparameters are fitted quantities that the empirical results depend on. No new physical or mathematical entities are introduced.

free parameters (2)
  • Embedding decoder weights W and b per model = Trained on DBPedia; values not reported
    The recoverability scores in Tables 1 and 2 depend on this trained linear decoder. No training hyperparameters or final weights are provided, so the fitted values are not independently reproducible.
  • Fine-tuning hyperparameters for recoverability training = Not reported
    The downstream improvements in Section 5.4 depend on learning rate, batch size, number of epochs, and optimizer choice, none of which are given. Without these values the enhancement results cannot be replicated or compared to standard fine-tuning.
assumptions (4)
  • domain assumption Understanding ability can be identified with sentence-level mutual information I(S;E).
    Section 1 states "we define the language understanding capacity of a model through the mutual information between input sentences and their latent representations." This is a definitional postulate, not a derived result.
  • domain assumption Tokens are uniformly distributed over the vocabulary, so H(ti) = log |V|.
    Theorem 2 in Section 3.2 uses this assumption to convert Fano's inequality into the recoverability lower bound. Natural language token distributions are strongly non-uniform, so the derived bound may not hold for real sentences.
  • domain assumption A linear decoder is sufficient because embeddings are highly linear across layers.
    Section 4.1 invokes prior work [29] to justify using a linear transformation f(E)=WE+b. If the embedding-to-token mapping is nonlinear, the linear recoverability measure underestimates the true recoverability and hence the mutual information lower bound.
  • standard math Fano's inequality and the entropy chain rule are valid background results.
    The proofs in Appendix A and B rely on Fano's inequality, the chain rule for entropy, conditioning reduces entropy, and Jensen's inequality for the concave binary entropy function.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking the Understanding Ability across LLMs through Mutual Information." pith.science (2026). https://pith.science/paper/643I5SYC

@misc{pith2026250523790,
  author       = {Pith},
  title        = {Pith review of: Rethinking the Understanding Ability across LLMs through Mutual Information},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/643I5SYC}},
  note         = {Machine review of arXiv:2505.23790}
}
read the original abstract

Recent advances in large language models (LLMs) have revolutionized natural language processing, yet evaluating their intrinsic linguistic understanding remains challenging. Moving beyond specialized evaluation tasks, we propose an information-theoretic framework grounded in mutual information (MI) to achieve this. We formalize the understanding as MI between an input sentence and its latent representation (sentence-level MI), measuring how effectively input information is preserved in latent representation. Given that LLMs learn embeddings for individual tokens, we decompose sentence-level MI into token-level MI between tokens and sentence embeddings, establishing theoretical bounds connecting these measures. Based on this foundation, we theoretically derive a computable lower bound for token-level MI using Fano's inequality, which directly relates to token-level recoverability-the ability to predict original tokens from sentence embedding. We implement this recoverability task to comparatively measure MI across different LLMs, revealing that encoder-only models consistently maintain higher information fidelity than their decoder-only counterparts, with the latter exhibiting a distinctive late-layer "forgetting" pattern where mutual information is first enhanced and then discarded. Moreover, fine-tuning to maximize token-level recoverability consistently improves understanding ability of LLMs on tasks without task-specific supervision, demonstrating that mutual information can serve as a foundation for understanding and improving language model capabilities.

Figures

Figures reproduced from arXiv: 2505.23790 by the authors.

Figure 1
Figure 1. Two paradigms for evaluating language understanding in LLMs. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Layer recoverability performance across model depths for RoBERTa-large (encoder-only) [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Classification performance comparison between original models and their mutual [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Retrieval performance comparison between original models and their mutual information [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 18 canonical work pages

  1. [1]

    Comprehensive analysis of falcon 7b: A state-of-the-art generative large language model

    Manimaran Aridoss, Khushwant Singh Bisht, and Arul Kumar Natarajan. Comprehensive analysis of falcon 7b: A state-of-the-art generative large language model. In Generative AI: Current Trends and Applications, pages 147–164. Springer, 2024

    work page 2024

  2. [2]

    The vulnerability of language model benchmarks: Do they accurately reflect true llm performance? arXiv preprint arXiv:2412.03597, 2024

    Sourav Banerjee, Ayushi Agarwal, and Eishkaran Singh. The vulnerability of language model benchmarks: Do they accurately reflect true llm performance? arXiv preprint arXiv:2412.03597, 2024

    arXiv 2024

  3. [3]

    LLMs' Reading Comprehension Is Affected by Parametric Knowledge and Struggles with Hypothetical Statements

    Victoria Basmov, Yoav Goldberg, and Reut Tsarfaty. Llms’ reading comprehension is af- fected by parametric knowledge and struggles with hypothetical statements. arXiv preprint arXiv:2404.06283, 2024

    work page Pith review arXiv 2024

  4. [4]

    Llm2vec: Large language models are secretly powerful text encoders

    Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. Llm2vec: Large language models are secretly powerful text encoders. arXiv preprint arXiv:2404.05961, 2024

    arXiv 2024

  5. [5]

    A survey on evaluation of large language models

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1–45, 2024

    2024

  6. [6]

    Are decoder-only large language models the silver bullet for code search? arXiv preprint arXiv:2410.22240, 2024

    Yuxuan Chen, Guangsheng Ou, Mingwei Liu, Yanlin Wang, and Zibin Zheng. Are decoder-only large language models the silver bullet for code search? arXiv preprint arXiv:2410.22240, 2024

    arXiv 2024

  7. [7]

    Elements of information theory

    Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999

    1999

  8. [8]

    Testing ai on language comprehension tasks reveals insensitivity to underlying meaning

    Vittoria Dentella, Fritz Günther, Elliot Murphy, Gary Marcus, and Evelina Leivada. Testing ai on language comprehension tasks reveals insensitivity to underlying meaning. Scientific Reports, 14(1):28083, 2024

    work page 2024

Show all 46 references
  1. [9]

    Information theoretic approaches to understanding circuit function

    Adrienne Fairhall, Eric Shea-Brown, and Andrea Barreiro. Information theoretic approaches to understanding circuit function. Current opinion in neurobiology, 22(4):653–659, 2012

    2012

  2. [10]

    Dissecting deep learning net- works—visualizing mutual information

    Hui Fang, Victoria Wang, and Motonori Yamaguchi. Dissecting deep learning net- works—visualizing mutual information. Entropy, 20(11):823, 2018

    2018

  3. [11]

    Benchmark performance is a poor measure of generalisable ai reasoning capabili- ties

    James Fodor. Benchmark performance is a poor measure of generalisable ai reasoning capabili- ties

  4. [12]

    Line goes up? inherent limitations of benchmarks for evaluating large language models

    James Fodor. Line goes up? inherent limitations of benchmarks for evaluating large language models. arXiv preprint arXiv:2502.14318, 2025

    2025 arXiv

  5. [13]

    Entropy and mutual information in models of deep neural networks

    Marylou Gabrié, Andre Manoel, Clément Luneau, Nicolas Macris, Florent Krzakala, Lenka Zdeborová, et al. Entropy and mutual information in models of deep neural networks. Advances in neural information processing systems, 31, 2018

    2018

  6. [14]

    The pile: An 800gb dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    2020 arXiv

  7. [15]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  8. [16]

    Mixtral of experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  9. [17]

    Estimating mutual information

    Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical Review E—Statistical, Nonlinear, and Soft Matter Physics, 69(6):066138, 2004. 12

    2004

  10. [18]

    Are chatgpt and gpt-4 general-purpose solvers for financial text analytics? a study on several typical tasks

    Xianzhi Li, Samuel Chan, Xiaodan Zhu, Yulong Pei, Zhiqiang Ma, Xiaomo Liu, and Sameena Shah. Are chatgpt and gpt-4 general-purpose solvers for financial text analytics? a study on several typical tasks. arXiv preprint arXiv:2305.05862, 2023

    2023 arXiv

  11. [19]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 364, 2019

    1907 arXiv

  12. [20]

    Maas, Raymond E

    Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y . Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pages 142–15...

    2011

  13. [21]

    Pointer sentinel mixture models, 2016

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016

    2016

  14. [22]

    Named entity recognition

    Behrang Mohit. Named entity recognition. In Natural language processing of semitic languages, pages 221–245. Springer, 2014

    2014

  15. [23]

    Mteb: Massive text embedding benchmark

    Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316, 2022

    2022 arXiv

  16. [24]

    Using an llm to help with code understanding

    Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. Using an llm to help with code understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–13, 2024

    2024

  17. [25]

    Encoder vs decoder: Comparative analysis of encoder and decoder language models on multilingual nlu tasks

    Dan Saattrup Nielsen, Kenneth Enevoldsen, and Peter Schneider-Kamp. Encoder vs decoder: Comparative analysis of encoder and decoder language models on multilingual nlu tasks. arXiv preprint arXiv:2406.13469, 2024

    2024 arXiv

  18. [26]

    How much a galaxy knows about its large-scale envi- ronment?: An information theoretic perspective

    Biswajit Pandey and Suman Sarkar. How much a galaxy knows about its large-scale envi- ronment?: An information theoretic perspective. Monthly Notices of the Royal Astronomical Society: Letters, 467(1):L6–L10, 2017

    2017

  19. [27]

    Reasoning with large language models, a survey

    Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Back. Reasoning with large language models, a survey. arXiv preprint arXiv:2407.11511, 2024

    2024

  20. [28]

    Muhammad Qorib, Geonsik Moon, and Hwee Tou Ng. Are decoder-only language models better than encoder-only language models in understanding word meaning? In Findings of the Association for Computational Linguistics ACL 2024, pages 16339–16347, 2024

    2024

  21. [29]

    Your transformer is secretly linear.arXiv preprint arXiv:2405.12250, 2024

    Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Nikolai Gerasimenko, Ivan Oseledets, Denis Dimitrov, and Andrey Kuznetsov. Your transformer is secretly linear.arXiv preprint arXiv:2405.12250, 2024

    2024 arXiv

  22. [30]

    Llms’ understanding of natural language revealed

    Walid S Saba. Llms’ understanding of natural language revealed. arXiv preprint arXiv:2407.19630, 2024

    2024 arXiv

  23. [31]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019

    1910 arXiv

  24. [32]

    Incremental mutual information: a new method for char- acterizing the strength and dynamics of connections in neuronal circuits

    Abhinav Singh and Nicholas A Lesica. Incremental mutual information: a new method for char- acterizing the strength and dynamics of connections in neuronal circuits. PLoS computational biology, 6(12):e1001035, 2010

    2010

  25. [33]

    Leveraging conditional mutual information to improve large language model fine-tuning for classification

    Thanushon Sivakaran and En-Hui Yang. Leveraging conditional mutual information to improve large language model fine-tuning for classification. arXiv preprint arXiv:2502.11258, 2025

    2025 arXiv

  26. [34]

    Layer by layer: Uncovering hidden representations in language models

    Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in language models. arXiv preprint arXiv:2502.02013, 2025

    2025 arXiv

  27. [35]

    Table meets llm: Can large language models understand structured table data? a benchmark and empirical study

    Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. Table meets llm: Can large language models understand structured table data? a benchmark and empirical study. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 645–654, 2024. 13

    2024

  28. [36]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  29. [37]

    Many of your dpos are secretly one: Attempting unification through mutual information

    Rasul Tutnov, Antoine Grosnit, and Haitham Bou-Ammar. Many of your dpos are secretly one: Attempting unification through mutual information. arXiv preprint arXiv:2501.01544, 2025

    2025 arXiv

  30. [38]

    Fact or fiction: Verifying scientific claims

    David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 7534–7550, On...

    2020

  31. [39]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. 2019. In the Proceedings of ICLR

    2019

  32. [40]

    Mutual information as a tool for identifying phase transitions in dynamical complex systems with limited data

    Robert T Wicks, Sandra C Chapman, and RO Dendy. Mutual information as a tool for identifying phase transitions in dynamical complex systems with limited data. Physical Review E—Statistical, Nonlinear, and Soft Matter Physics, 75(5):051125, 2007

    2007

  33. [41]

    Interpreting and steering llms with mutual information-based explanations on sparse autoencoders

    Xuansheng Wu, Jiayi Yuan, Wenlin Yao, Xiaoming Zhai, and Ninghao Liu. Interpreting and steering llms with mutual information-based explanations on sparse autoencoders. arXiv preprint arXiv:2502.15576, 2025

    2025 arXiv

  34. [42]

    Open, closed, or small language models for text classification?

    Hao Yu, Zachary Yang, Kellin Pelrine, Jean Francois Godbout, and Reihaneh Rabbany. Open, closed, or small language models for text classification?. arxiv.arXiv preprint arXiv:2308.10092, 2023

    2023 arXiv

  35. [43]

    When scaling meets llm finetuning: The effect of data, model and finetuning method

    Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets llm finetuning: The effect of data, model and finetuning method. arXiv preprint arXiv:2402.17193, 2024

    2024 arXiv

  36. [44]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  37. [45]

    Character-level Convolutional Networks for Text Classification

    Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level Convolutional Networks for Text Classification. arXiv:1509.01626 [cs], September 2015

    2015 arXiv

  38. [46]

    Character-level convolutional networks for text classification

    Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015. 14

    2015

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.