Pith. sign in

REVIEW 3 major objections 5 minor 55 references

Response Uncertainty and Probe Modeling: Two Sides of the Same Coin in LLM Interpretability?

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper reports a strong negative correlation between how consistently an LLM answers a factual question and how accurately a linear probe can decode the underlying concept from the model's hidden representations.

desk verdict A genuinely new empirical correlation between response uncertainty and probe performance, with a mechanistic story that outruns the evidence. read the letter →

arxiv 2505.18575 v1 pith:WUW4V4TW submitted 2025-05-24 cs.AI

Yongjie Wang , Yibo Wang , Xin Zhou , Zhiqi Shen This is my paper · ORCID
classification cs.AI
keywords LLMinterpretabilityprobingresponseuncertaintylinearrepresentationhypothesisfeatureattributionAttnLRPhallucinationdetectionsemanticentropy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that two seemingly separate diagnostics of large language models are actually two views of the same internal fact: the consistency of a model's repeated answers to the same question (response uncertainty) and how accurately a simple linear probe can decode the underlying concept from the model's hidden activations. Across six time- and space-related datasets and six instruction-tuned LLMs from 8 to 72 billion parameters, it reports a strong negative correlation: when responses are less variable, probe performance is higher, with Kendall and Spearman rank correlations frequently below -0.8. It then offers a mechanistic account: high response uncertainty corresponds to relevance spread across a broader set of features, which makes a low-dimensional linear probe a worse fit, a claim supported by feature-attribution experiments and a sparsity-based generalization bound. If the connection holds, response uncertainty can act as a cheap, label-free indicator of whether a concept is linearly decodable, and can guide where to look for interpretable representations.

What carries the argument

The load-bearing machinery is the uncertainty-binned probe analysis: query the model repeatedly for the same factual prompt, estimate per-sample response uncertainty (variance for continuous values, with entropy as a check), sort samples by uncertainty, divide them into overlapping sliding-window segments, train a separate ridge-regression probe on the hidden-activation representation of each segment, and correlate segment mean uncertainty with probe performance. The supporting mechanism is AttnLRP feature attribution, which assigns importance scores to latent features with respect to the generated response; combined with Remove-and-Retrain masking (RoAR), it shows the probe and the generation process rely on the same concentrated feature set. The theoretical piece is a sparsity-based argument: if responses vary more, the union of important features across response variants is larger, and the Lasso oracle inequality (a high-probability bound on the gap between empirical and true risk for sparse regression) makes the ridge probe's fit harder as the number of relevant features grows.

What would settle it

Re-run the uncertainty-sorted bin analysis while controlling for the variance of ground-truth labels within each bin (e.g., matching bins to have similar spread of death years or coordinates). If the negative correlation between response uncertainty and probe performance largely disappears under this control, the claimed relationship is an artifact of bin composition rather than the proposed feature-sharing mechanism. A second check: test the correlation on a base (non-instruction-tuned) model; the paper's own observation predicts it should be much weaker, and a null result there would refine the scope of the claim.

Watch

Extended reading notes

Core claim

The central claim is that 'when an LLM exhibits lower response uncertainty (i.e., less variability in its outputs), its corresponding internal representations can be more accurately modeled by a linear probe,' and conversely that high uncertainty signals a larger set of important features that degrade probe training. The paper demonstrates this by sorting samples by per-sample response variance or entropy, splitting the sorted data into overlapping sliding-window segments, training a ridge-regression probe on each segment, and correlating segment-level mean uncertainty with probe $R^2$ and Spearman rank scores; the correlation is strongly negative across six datasets and six models. Feature-attribution experiments using AttnLRP (attention-aware layer-wise relevance propagation) show that keeping only the top 20-30% of response-relevant features preserves probe performance, and that high-uncertainty subsets require a larger fraction of features to maintain performance, consistent with the proposed mechanism. The paper also shows that low-uncertainty examples yield interpretable visualizations, such as brand categories, a circular clock-time embedding, and a birth-year probe.

Load-bearing premise

The argument assumes that probe performance measured on a bin of samples cleanly reflects how well the concept is encoded in the representations of those samples, independent of confounds such as how spread out the ground-truth values are within the bin, how familiar the entities are, or how hard the subset is; if these differ systematically between low- and high-uncertainty bins, the correlation could arise without the proposed feature-sparsity mechanism.

Editorial extensions

If this is right

  • Response uncertainty can serve as a lightweight, label-free diagnostic: datasets or samples with limited uncertainty are likely to yield probeable concepts, reducing the need for exhaustive data labeling and trial-and-error probing.
  • Probe performance and generation consistency reflect the same representational substance: both degrade or improve together as the number of important features in the latent space grows or shrinks.
  • Low-uncertainty subsets can be mined for interpretable structure without training probes first, as demonstrated by the brand, clock-time, and birth-year examples.
  • The finding links the probing and hallucination research threads, suggesting that uncertainty measurements and probe analyses can be traded off depending on which is practically available.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested extension is whether the correlation is causal: a controlled experiment that intervenes on representation sparsity (e.g., by pruning or steering features) and measures both probe performance and response uncertainty would distinguish a shared mechanism from a common confound.
  • The paper leaves open whether bin-level correlations are distorted by the spread of ground-truth values within a bin; a follow-up that stratifies by ground-truth variance or entity frequency could settle whether the correlation is an artifact of sample composition.
  • The observation that the correlation appears mainly in instruction-tuned models suggests a testable prediction: base models should show a much weaker relationship, and instruction-tuning should strengthen it; this is grounded in the paper's own remark but not experimentally quantified.
  • If the sparsity mechanism is right, techniques that increase activation sparsity or concentrate feature importance should improve both linear probeability and answer consistency, which would open a practical route to more interpretable and more reliable LLMs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This manuscript investigates whether the uncertainty of an LLM's generated responses is correlated with how accurately a linear probe can recover a target concept from the LLM's internal representations. The authors measure response uncertainty as the variance (or entropy) across repeated generations, sort samples by uncertainty, partition them into overlapping sliding-window segments, train a ridge-regression probe on each segment, and report the rank correlation between average segment uncertainty and probe performance (R2 and Spearman). Across six time/space datasets and six public LLMs, they report strong negative correlations. They then propose a mechanistic explanation based on AttnLRP feature attribution: high-uncertainty responses draw on a larger set of important features, making the probe harder to fit, and they support this with masking experiments and a theoretical argument in Appendix G. The paper also presents case studies in which low-uncertainty concepts produce interpretable embedding structure. The central claim is framed as establishing a link between response uncertainty and probe performance that could enable cheaper probing diagnostics.

Significance. If the empirical correlation is genuine and survives appropriate controls, the result would be practically useful: response uncertainty is cheap to measure, and could serve as a proxy for whether a linear probe will succeed on a given sample set, connecting two currently separate literatures. The paper has clear strengths: the correlation is measured directly rather than fitted, the experiment spans six datasets and six LLM families, both variance and entropy uncertainty estimators are considered, and sensitivity analyses are reported for temperature, sliding-window parameters, and prompt variation. The mechanistic story is plausible and potentially interesting, and the case studies are illustrative. However, the central empirical claim depends on bin-level R2/Spearman comparability, which is confounded by within-bin target variance, and the theoretical support in Appendix G is not derived for the actual ridge probe. These issues are local and fixable, but they are load-bearing for the paper's main conclusion, so the manuscript needs substantial revision.

major comments (3)
  1. [Section 3.1, Table 1, Figure 2] The central correlation is computed between average response uncertainty and bin-level probe performance measured by R2 and Spearman rank. R2 is normalized by the segment's own total variance of the ground-truth target, and Spearman is sensitive to the range, tie structure, and distribution of distinct target values within the segment. The paper does not report within-segment standard deviations of Y or include them as covariates. If high-uncertainty bins happen to contain targets that are concentrated over a short time interval or small spatial region, the same absolute prediction error would yield a lower R2 and a lower rank correlation even if the linear encoding quality were identical. The authors should control for segment target variance (for example, by reporting RMSE evaluated under a global variance normalization, or by computing partial correlations with within-bin std(Y) as a covariate), or otherwise demonstrate that the correlation is not a bin-composition artifact. This is necessary to support the headline claim that lower response uncertainty implies more accurately probeable representations.
  2. [Section 5.1, Figure 6] The mechanistic claim that high-uncertainty responses are associated with a larger set of important features rests on comparing probe performance across Low/Mid/High uncertainty subsets, but the method described in the text trains the probe on the entire dataset and then evaluates per-subset performance after masking features selected by AttnLRP with respect to the LLM response, not the probe. The sentence stating that "the probe trained on the high-uncertainty subset requires a larger number of features" is inconsistent with this reported procedure. The observed differences could reflect distribution shift between subsets rather than the size of the feature set needed for probing. The authors should train independent probes on each subset, use consistent Remove-and-Retrain methodology for each subset, provide error bars across random seeds, and report results for more than the two datasets currently shown in the main text and appendix.
  3. [Appendix G, Eq. (3)] The theoretical argument applies a Lasso oracle inequality to what is, in the paper's experiments, a ridge-regression probe (Eq. (2)). The bound in Eq. (3) controls the excess risk of Lasso in a high-dimensional sparse linear model; it does not apply to the ridge objective, and it does not imply that a larger number of important features causes lower probe R2. In addition, the premise that higher response uncertainty implies a larger truly important feature set is asserted rather than derived. The authors should either derive a relevant bound for ridge/linear regression in the exact setting used (or for the probe metric actually reported), or remove the claim that the mechanism is theoretically supported. As written, the theoretical analysis does not rescue the empirical correlation from the bin-composition confound.
minor comments (5)
  1. [Table 1 and Section 4.1] Table 1 reports correlations without confidence intervals or significance tests; because sliding-window segments overlap, the effective number of independent observations is smaller than the number of segments. The authors should report bootstrap or other valid uncertainty estimates, or at least state the number of segments and their dependence structure.
  2. [Table 1 footnote] The decision to compute World-dataset correlations on the top 20,000 most uncertain samples is a data-dependent choice that is not accompanied by a sensitivity analysis on the cutoff. The paper should justify this cutoff and show that the conclusion is stable across reasonable cutoff choices, especially since the World dataset is one of the six datasets supporting the central claim.
  3. [Section 5.2 / Table 7] The text refers to "Section 5.3" in the Table 7 caption and in Appendix H, but the case-study section is numbered Section 5.2; the cross-references should be fixed.
  4. [Reference [5]] The cited reference is written as "Templeton Adly, Conerly Tom, ..." but the author is conventionally cited as Adly Templeton et al.; the entry should be corrected for consistency with the in-text citation "Adly et al.".
  5. [Figure 2 caption] The caption reads "Std R2 score Spearman rank correlation"; the intended labels are "R2 score" and "Spearman rank correlation" with a separator. Please clarify the caption and legend.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the headline correlation is measured directly, and the mechanistic analysis rests on independent masking experiments rather than on a fitted parameter renamed as a prediction.

full rationale

The paper's central claim is an observed correlation between bin-level response uncertainty (variance or entropy of repeated LLM answers) and bin-level probe performance (R2 or Spearman rank of a ridge probe on the same samples). This quantity is measured directly from experiments; no parameter is fitted to a subset of data and then used to predict the same or a closely related quantity. The sorting and sliding-window binning is a grouping device, not a fitted model. The proposed mechanism in Section 5 is supported by separate AttnLRP masking experiments that compare Low, Mid, and High uncertainty subsets; that evidence is independent of the headline correlation, even though the mechanism is offered post hoc. The Lasso-oracle bound in Appendix G is invoked as a loose theoretical motivation for why more important features could hinder ridge probes; it is not derived from the paper's own setup and may be a correctness risk because it concerns the generalization gap rather than R2 directly, but it does not define probe performance in terms of response uncertainty or vice versa. The only self-citation, reference [47], supports a standard 'remove and test' evaluation protocol and is not load-bearing. Concerns about within-bin ground-truth variance or entity difficulty are confounding and validity concerns, not circularity by construction. No circular step can be exhibited, so the appropriate finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central correlation result rests mainly on the datasets from Gurnee and Tegmark, on the choice of variance as an uncertainty measure, and on the linear probe setup. The mechanistic explanation additionally assumes sparsity of important features (from prior literature) and that AttnLRP relevance scores for generation reflect the features a linear probe would use. The mathematical result in Appendix G is a standard Lasso oracle bound whose connection to the probe's R2 is asserted rather than derived. No new entities are invented.

free parameters (2)
  • World dataset sample cutoff = 20,000 samples
    For the World dataset, correlations in Table 1 are computed only on the top 20,000 most uncertain samples, a disclosed post-hoc subset.
  • Sliding window size and stride = e.g., 6000/1000 for Llama 8B; varies by model
    Segment definitions for correlation analysis are hand-chosen; sensitivity is shown in Figure 4 and Appendix E, but different settings change the coefficient values.
assumptions (4)
  • domain assumption The true probe regression for each target concept is approximately s-sparse with s << d.
    Invoked in Section 5.1 and Appendix G to justify the Lasso oracle inequality; supported by sparse-representation citations [31,45], but not verified for the probe's target concept.
  • domain assumption AttnLRP feature importance scores for response generation identify the features that a linear probe relies on.
    Assumed in Section 5 ('both the probe model and the LLM generation rely on the same set of features'); the masking experiments provide evidence but do not establish causality.
  • domain assumption Response variance across 20 generations at temperature 1 is a valid measure of the model's uncertainty about a concept.
    Used to sort samples in Sections 3 and 4; entropy is added as a robustness check in Appendix C; Section 7 notes base models do not show the same behavior.
  • standard math Bühlmann and van de Geer's Lasso oracle inequality applies to the ridge regression probe setting.
    Appendix G Eq. (3); the probe uses ridge loss, not Lasso, and the bound is about the risk gap, not test R2, so the application is an informal transfer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Response Uncertainty and Probe Modeling: Two Sides of the Same Coin in LLM Interpretability?." pith.science (2026). https://pith.science/paper/WUW4V4TW

@misc{pith2026250518575,
  author       = {Pith},
  title        = {Pith review of: Response Uncertainty and Probe Modeling: Two Sides of the Same Coin in LLM Interpretability?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WUW4V4TW}},
  note         = {Machine review of arXiv:2505.18575}
}
read the original abstract

Probing techniques have shown promise in revealing how LLMs encode human-interpretable concepts, particularly when applied to curated datasets. However, the factors governing a dataset's suitability for effective probe training are not well-understood. This study hypothesizes that probe performance on such datasets reflects characteristics of both the LLM's generated responses and its internal feature space. Through quantitative analysis of probe performance and LLM response uncertainty across a series of tasks, we find a strong correlation: improved probe performance consistently corresponds to a reduction in response uncertainty, and vice versa. Subsequently, we delve deeper into this correlation through the lens of feature importance analysis. Our findings indicate that high LLM response variance is associated with a larger set of important features, which poses a greater challenge for probe models and often results in diminished performance. Moreover, leveraging the insights from response uncertainty analysis, we are able to identify concrete examples where LLM representations align with human knowledge across diverse domains, offering additional evidence of interpretable reasoning in LLMs.

Figures

Figures reproduced from arXiv: 2505.18575 by the authors.

Figure 1
Figure 1. Framework for correlating probe performance with response uncertainty. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Trend between response uncertainty and probe performance on the Llama 3.1 (8B) model [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. The correlation analysis with different temperatures [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (7 more)
Figure 6
Figure 6. Figure 6: Remove and test on differently uncertain sub￾sets, Llama 3.1 (8B). and reporting its performance on the masked test set—following the Remove and Retrain (RoAR) paradigm [4]. The results are reported in [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Three commonsense examples where LLM embeddings align well with human knowledge [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Trend between response uncertainty (entropy) and probe performance on the Llama 3.1 [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: The correlation analysis with different temperatures in generation for Mistral-Small (24B). [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Correlation coefficient vs sliding window parameters, Llama 3.1 (8B) on Figures. ‘Var’, [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 12
Figure 12. Figure 12: reports the remove and test results on different uncertainty subsets using Artworks dataset. From the bottom figure, we can see that preserving the top 5% of important features approximately achieves the similar Spearman rank correlation for low-uncertainty subset whi…
Figure 13
Figure 13. Figure 13: Remove and test on three subsets of varying uncertainty (measured by entropy), on Llama [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 22 canonical work pages

  1. [1]

    To believe or not to believe your llm: Iterative prompting for estimating epistemic uncertainty.Advances in Neural Information Processing Systems, 37:58077–58117, 2024

    Yasin Abbasi Yadkori, Ilja Kuzborskij, András György, and Csaba Szepesvari. To believe or not to believe your llm: Iterative prompting for estimating epistemic uncertainty.Advances in Neural Information Processing Systems, 37:58077–58117, 2024

    work page 2024

  2. [2]

    Can language models encode perceptual structure without grounding? a case study in color

    Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. Can language models encode perceptual structure without grounding? a case study in color. InProceedings of the 25th Conference on Computational Natural Language Learning, pages 109–132, 2021

    work page 2021

  3. [3]

    Attnlrp: attention-aware layer-wise relevance propagation for transformers

    Reduan Achtibat, Sayed Mohammad Vakilzadeh Hatefi, Maximilian Dreyer, Aakriti Jain, Thomas Wiegand, Sebastian Lapuschkin, and Wojciech Samek. Attnlrp: attention-aware layer-wise relevance propagation for transformers. InProceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024

    work page 2024

  4. [4]

    Sanity checks for saliency maps.Advances in neural information processing systems, 31, 2018

    Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps.Advances in neural information processing systems, 31, 2018

    2018

  5. [5]

    Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet.Trans- former Circuits Thread, 2024

    Templeton Adly, Conerly Tom, Marcus Jonathan, Lindsey Jack, Bricken Trenton, Chen Brian, et al. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet.Trans- former Circuits Thread, 2024

    work page 2024

  6. [6]

    Understanding intermediate layers using linear classifier probes

    Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. InThe Fifth International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=ryF7rTqgl

    work page 2017

  7. [7]

    A close look at decomposition-based xai-methods for transformer language models.arXiv preprint arXiv:2502.15886, 2025

    Leila Arras, Bruno Puri, Patrick Kahardipraja, Sebastian Lapuschkin, and Wojciech Samek. A close look at decomposition-based xai-methods for transformer language models.arXiv preprint arXiv:2502.15886, 2025

    arXiv 2025

  8. [8]

    On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.PloS one, 10(7):e0130140, 2015

    Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.PloS one, 10(7):e0130140, 2015

    2015

Show all 55 references
  1. [9]

    Probing classifiers: Promises, shortcomings, and advances.Com- putational Linguistics, 48(1):207–219, March 2022

    Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances.Com- putational Linguistics, 48(1):207–219, March 2022. doi: 10.1162/coli_a_00422. URL https://aclanthology.org/2022.cl-1.7/

    2022 doi

  2. [10]

    Bender and Alexander Koller

    Emily M. Bender and Alexander Koller. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors,Proceedings of the 58th Annual Meeting of the Association for Computational Linguistic...

    2020 doi

  3. [11]

    Springer Science & Business Media, 2011

    Peter Bühlmann and Sara Van De Geer.Statistics for high-dimensional data: methods, theory and applications. Springer Science & Business Media, 2011

    2011

  4. [12]

    Discovering latent knowledge in language models without supervision

    Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. InThe Eleventh International Conference on Learning Representations

  5. [13]

    An axiomatic assessment of entropy- and variance-based uncertainty quantification in regression,

    Christopher Bülte, Yusuf Sale, Timo Löhr, Paul Hofman, Gitta Kutyniok, and Eyke Hüllermeier. An axiomatic assessment of entropy- and variance-based uncertainty quantification in regression,

  6. [14]

    (a) i am not a lawyer, but...: engaging legal experts towards responsible llm policies for legal advice

    Inyoung Cheong, King Xia, KJ Kevin Feng, Quan Ze Chen, and Amy X Zhang. (a) i am not a lawyer, but...: engaging legal experts towards responsible llm policies for legal advice. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 2454–...

    2024

  7. [15]

    SaulLM-54b & saulLM-141b: Scaling up domain adaptation for the legal domain

    Pierre Colombo, Telmo Pires, Malik Boudiaf, Rui Filipe Coimbra Pereira de Melo, Gabriel Hautreux, Etienne Malaboeuf, Johanne Charpentier, Dominic Culver, and Michael Desa. SaulLM-54b & saulLM-141b: Scaling up domain adaptation for the legal domain. InThe Thirty-eighth Annual C...

    2024

  8. [16]

    Ensembles for uncertainty estimation: Benefits of prior functions and bootstrapping.Transactions on Machine Learning Research, 2023

    Vikranth Dwaracherla, Zheng Wen, Ian Osband, Xiuyuan Lu, Seyed Mohammad Asghari, and Benjamin Van Roy. Ensembles for uncertainty estimation: Benefits of prior functions and bootstrapping.Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.ne...

    2023

  9. [17]

    From local to global: A graph rag approach to query-focused summarization.arXiv preprint arXiv:2404.16130, 2024

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. From local to global: A graph rag approach to query-focused summarization.arXiv preprint arXiv:2404.16130, 2024

    2024 arXiv

  10. [18]

    A mathematical framework for transformer circuits.Transformer Circuits Thread, 1(1):12, 2021

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits.Transformer Circuits Thread, 1(1):12, 2021

    2021

  11. [19]

    Not all language model features are one-dimensionally linear

    Joshua Engels, Eric J Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. Not all language model features are one-dimensionally linear. InThe Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=d63a4AM4hb

    2025

  12. [20]

    Detecting hallucinations in large language models using semantic entropy.Nature, 630(8017):625–630, 2024

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy.Nature, 630(8017):625–630, 2024

    2024

  13. [21]

    Language models represent space and time

    Wes Gurnee and Max Tegmark. Language models represent space and time. InThe Twelfth International Conference on Learning Representations, 2024. URL https://openreview. net/forum?id=jE8xbmvFin

    2024

  14. [22]

    Bridging language and items for retrieval and recommendation.arXiv preprint arXiv:2403.03952, 2024

    Yupeng Hou, Jiacheng Li, Zhankui He, An Yan, Xiusi Chen, and Julian McAuley. Bridging language and items for retrieval and recommendation.arXiv preprint arXiv:2403.03952, 2024

    2024 arXiv

  15. [23]

    Survey of hallucination in natural language generation

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Comput. Surv., 55(12), March 2023. ISSN 0360-0300. doi: 10.1145/3571730. URL https://doi.or...

    2023 doi

  16. [24]

    Johnson, Daniel Tarlow, David Duvenaud, and Chris J

    Daniel D. Johnson, Daniel Tarlow, David Duvenaud, and Chris J. Maddison. Experts don’t cheat: Learning what you don’t know by predicting pairs. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors...

    2024

  17. [25]

    Language models (mostly) know what they know.arXiv preprint arXiv:2207.05221, 2022

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know.arXiv preprint arXiv:2207.05221, 2022

    2022 arXiv

  18. [26]

    Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav)

    Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). InInternational conference on machine learning, pages 2668–2677. PMLR, 2018

    2018

  19. [27]

    Concept bottleneck models

    Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. InInternational conference on machine learning, pages 5338–5348. PMLR, 2020

    2020

  20. [28]

    Semantic entropy probes: Robust and cheap hallucination detection in LLMs, 2025

    Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth A Malik, and Yarin Gal. Semantic entropy probes: Robust and cheap hallucination detection in LLMs, 2025. URL https://openreview.net/forum?id=YQvvJjLWX0. 11

    2025

  21. [29]

    Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. InThe Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum? id=VD-AYtP0dve

    2023

  22. [30]

    Emergent world representations: Exploring a sequence model trained on a synthetic task

    Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task. InThe Eleventh International Conference on Learning Representations, 2023. URL https://ope...

    2023

  23. [31]

    Sparsing law: Towards large language models with greater activation sparsity.arXiv preprint arXiv:2411.02335, 2024

    Yuqi Luo, Chenyang Song, Xu Han, Yingfa Chen, Chaojun Xiao, Zhiyuan Liu, and Maosong Sun. Sparsing law: Towards large language models with greater activation sparsity.arXiv preprint arXiv:2411.02335, 2024

    2024 arXiv

  24. [32]

    On faithfulness and factuality in abstractive summarization

    Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors,Proceedings of the 58th Annual Meeting of the Association for Computational L...

    1906 doi

  25. [33]

    Linguistic regularities in continuous space word representations

    Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Lucy Vanderwende, Hal Daumé III, and Katrin Kirchhoff, editors, Proceedings of the 2013 Conference of the North American Chapter of the Association for Computat...

    2013

  26. [34]

    Linguistic regularities in continuous space word representations

    Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. InProceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 746–751, 2013

    2013

  27. [35]

    Robustly identifying concepts introduced during chat fine-tuning using crosscoders.arXiv preprint arXiv:2504.02922, 2025

    Julian Minder, Clement Dumas, Caden Juang, Bilal Chugtai, and Neel Nanda. Robustly identifying concepts introduced during chat fine-tuning using crosscoders.arXiv preprint arXiv:2504.02922, 2025

    2025

  28. [36]

    Progress mea- sures for grokking via mechanistic interpretability

    Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress mea- sures for grokking via mechanistic interpretability. InThe Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=9XFSbDPmdW

    2023

  29. [37]

    Emergent linear representations in world models of self-supervised sequence models

    Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. In Yonatan Belinkov, Sophie Hao, Jaap Jumelet, Na- joung Kim, Arya McCarthy, and Hosein Mohebbi, editors,Proceedings of the 6th BlackboxNLP Worksh...

    2023 doi

  30. [38]

    Generative agents: Interactive simulacra of human behavior

    Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. InProceed- ings of the 36th annual acm symposium on user interface software and technology, pages 1–22, 2023

    2023

  31. [39]

    URLhttps://aclanthology.org/2023.blackboxnlp-1.2/

    2023

  32. [40]

    Llm-based agentic systems in medicine and healthcare.Nature Machine Intelligence, 6(12):1418–1420, 2024

    Jianing Qiu, Kyle Lam, Guohao Li, Amish Acharya, Tien Yin Wong, Ara Darzi, Wu Yuan, and Eric J Topol. Llm-based agentic systems in medicine and healthcare.Nature Machine Intelligence, 6(12):1418–1420, 2024. 12

    2024

  33. [41]

    Mapping language models to grounded conceptual spaces

    Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. In International Conference on Learning Representations, 2022. URL https://openreview. net/forum?id=gJcEM8sxHK

    2022

  34. [42]

    Re- thinking interpretability in the era of large language models.arXiv preprint arXiv:2402.01761, 2024

    Chandan Singh, Jeevana Priya Inala, Michel Galley, Rich Caruana, and Jianfeng Gao. Re- thinking interpretability in the era of large language models.arXiv preprint arXiv:2402.01761, 2024

    2024 arXiv

  35. [43]

    Ash, and Dipendra Misra

    Pratyusha Sharma, Jordan T. Ash, and Dipendra Misra. The truth is in there: Improving reasoning in language models with layer-selective rank reduction. InThe Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum? id=ozX92bu8VA

    2024

  36. [44]

    Large language models in medicine.Nature medicine, 29(8):1930–1940, 2023

    Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine.Nature medicine, 29(8):1930–1940, 2023

    1930

  37. [45]

    Gemma Team. Gemma 3. 2025. URLhttps://goo.gle/Gemma3Report

    2025

  38. [46]

    V oyager: An open-ended embodied agent with large language models.Transactions on Machine Learning Research, 2024

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. V oyager: An open-ended embodied agent with large language models.Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https: //openreview.net/forum?i...

    2024

  39. [47]

    Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned

    Elena V oita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797–...

    2019

  40. [48]

    Gaussian process probes (gpp) for uncertainty-aware probing.Advances in neural information processing systems, 36:63573–63594, 2023

    Zi Wang, Alexander Ku, Jason Baldridge, Tom Griffiths, and Been Kim. Gaussian process probes (gpp) for uncertainty-aware probing.Advances in neural information processing systems, 36:63573–63594, 2023

    2023

  41. [49]

    Gradient based feature attribution in explainable ai: A technical review.arXiv preprint arXiv:2403.10415, 2024

    Yongjie Wang, Tong Zhang, Xu Guo, and Zhiqi Shen. Gradient based feature attribution in explainable ai: A technical review.arXiv preprint arXiv:2403.10415, 2024

    2024 arXiv

  42. [50]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  43. [51]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  44. [52]

    On completeness-aware concept-based explanations in deep neural networks.Advances in neural information processing systems, 33:20554–20565, 2020

    Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-aware concept-based explanations in deep neural networks.Advances in neural information processing systems, 33:20554–20565, 2020

    2020

  45. [53]

    Tree of thoughts: Deliberate problem solving with large language models.Ad- vances in neural information processing systems, 36:11809–11822, 2023

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models.Ad- vances in neural information processing systems, 36:11809–11822, 2023

    2023

  46. [55]

    Cleopatra

    Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. Explainability for large language models: A survey.ACM Trans. Intell. Syst. Technol., 15(2), February 2024. ISSN 2157-6904. doi: 10.1145/3639372. URL https://do...

    2024 doi

  47. [2025]

    URLhttps://arxiv.org/abs/2504.18433

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.