REVIEW 3 major objections 6 minor 25 references
Measuring Concept Content in Text from LLM Activations: ESG Evidence from Concept Vectors and Linear Probes
T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A linear probe on frozen LLM activations measures ESG concept content within 0.6 accuracy points of fine-tuned classifiers.
desk verdict Useful head-to-head with an overstated headline: the probe-vs-RFM and probe-vs-output comparisons hold, but the 'within 0.6 points of fine-tuning' claim is best-of-N noise. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is linear probing of LLM activations, compared against RFM concept vectors. Each input sentence is passed through a frozen decoder-only model, and every layer's per-token hidden states are pooled (last-token, mean, or max) into one vector per layer; a ridge classifier is then fit on each layer's pooled activations, and a logistic regression meta-learner combines the per-layer decision scores via nested 5-fold cross-validation. The RFM alternative computes an Average Gradient Outer Product matrix from labelled activations and takes its top eigenvectors as concept directions, scoring new sentences by cosine similarity and thresholding the score. Token pooling and a task-framing wrapper are varied, and the best wrapper is selected on a validation split. The comparison against the embedding baseline, surface baselines, the model's own answer, and published fine-tuned classifiers carries the argument.
What would settle it
Re-run the published fine-tuned ESG classifiers on the exact 5-fold cross-validation splits used for the linear probes; if the accuracy gap widens beyond 0.6 points or reverses on any pillar, the near-parity claim is not established.
Extended reading notes
Core claim
The central claim is that a linear probe on the frozen activations of an off-the-shelf LLM measures ESG concept presence nearly as well as a classifier fine-tuned for the task. On the Environmental pillar the best probe reaches 0.951 accuracy against 0.957 for the best fine-tuned model; on Social it reaches 0.924 against 0.934; on Governance 0.876 against 0.897. The probe outscores the same model's own answer to the same binary question in eleven of twelve model-pillar combinations, by 0.043 AUC on average, so the internal signal is not merely echoing the output. Against the other activation-based extractor, the probe wins in eleven of twelve like-for-like comparisons even when both are trained on identical splits. The paper therefore proposes the frozen-model linear probe as a serious standing alternative to task-specific fine-tuning for measuring concept content.
Load-bearing premise
The load-bearing premise is that the published fine-tuned classifier accuracies are directly comparable to the probe and concept-vector numbers even though the fine-tuned models were not re-run on the same data splits; if the evaluation protocols differ in train/test composition or label distribution, the claimed 0.6-to-2.1-point margins do not follow.
Editorial extensions
If this is right
- Firms and researchers can measure concept presence in text with a few minutes of GPU time per concept and no task-specific fine-tuning, which lowers the cost of large-scale ESG and financial text monitoring.
- Because the probe beats the model's own answer, downstream applications that rely on LLM outputs alone may miss concept content that is accessible from internal states; reading activations can recover that content.
- The RFM continuous score, if validated on graded labels, would extend measurement from classification (present or not) to degree of concept presence, giving text-as-data researchers a graded measure.
- The 9.4-point accuracy swings from changing the prompt wrapper imply that any practitioner using these methods must treat wrapper design as a first-order hyperparameter, not a fixed detail.
Reading between the lines
- Editorial inference: the same probe-on-frozen-activations recipe should transfer to other regulated-text domains, such as risk disclosure, corporate culture, or contract clauses, where labelled sets are small and fine-tuning is expensive; the ESG result is evidence of mechanism, not a domain-bound accident.
- Editorial inference: because the probe outperforms the model's verbal yes/no answer, activation-based reading could serve as a monitoring channel for machine-generated corporate text, where surface wording and underlying concept content may drift apart.
- Editorial inference: the consistent superiority of the first RFM eigenvector over the mean of five suggests the AGOP concept vectors are capturing one dominant separation; inspecting what the second through fifth eigenvectors separate would clarify whether averaging adds signal or noise.
- Editorial inference: a direct test of the continuous-score claim would re-annotate a sample of ESG sentences on an ordinal scale, such as weak versus strong presence, and rank-correlate the RFM cosine scores against those grades; the paper already notes such validation awaits graded labels.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper asks whether the internal activations of frozen, out-of-the-box LLMs can measure concept content in text, using the ESG dataset of Schimanski et al. (2024) as a testbed. It compares two activation-based extractors—linear probing and RFM concept vectors—against an embedding baseline, surface baselines, the model's own yes/no answer, and the fine-tuned classifiers published with the dataset. The central reported result is that the best linear probe comes within 0.6, 1.0, and 2.1 accuracy points of the best fine-tuned model on the Environmental, Social, and Governance pillars, respectively, and beats the same model's own answer in eleven of twelve model-pillar combinations. The paper also reports that linear probing outperforms RFM in eleven of twelve like-for-like comparisons on a shared split, and it presents the RFM continuous score as an unvalidated graded measure.
Significance. If the headline claim were established, the paper would make a practically useful contribution: concept measurement without task-specific fine-tuning, at a few minutes of GPU time per pillar. The study has real strengths. It includes a like-for-like probe-versus-RFM comparison on an identical split, which the reader's preliminary report confirms as sound. It also tests the internal-versus-output premise directly by comparing activation-based measures with the model's own elicited answer, and it reports surface baselines (dictionary, topic model, TF-IDF) that put the activation-based results in context. The authors are admirably explicit about limitations: wrapper sensitivity, the unvalidated continuous score, the absence of Gemma from the probe tables, and the swept RFM threshold. The difficulty is that the strongest claim—that a frozen-model probe 'stands in for fine-tuning'—is not established by the current analysis because the comparison to fine-tuned classifiers is not a controlled one and the probe configuration is selected on the test outcome.
major comments (3)
- [Section 4.1 and Tables 3–5] The headline 'within 0.6, 1.0 and 2.1 accuracy points' compares the maximum of nine probe configurations (3 pooling strategies × 3 models) with the maximum of two fine-tuned architectures, with no validation-based model selection or multiplicity correction. On Environmental, the selected probe scores 0.951±0.009 against EnvRoBERTa's 0.957±0.010, a gap smaller than one standard error, and the expected maximum of nine positively correlated estimates is inflated. Consequently, the paper does not establish that a linear probe on frozen activations is as accurate as a fine-tuned classifier; it establishes only that some probe configuration came close on the test split. The authors should either select the probe configuration on a validation split before seeing the test results, apply a multiple-comparison correction, or reframe the claim as an exploratory upper bound rather than a head-to-head result.
- [Section 4 and Tables 3–5] The comparison to the fine-tuned models of Schimanski et al. (2024) is not like-for-like. The probe is scored by 5-fold cross-validation over all 2,000 sentences, RFM is scored once on a 500-sentence held-out split, and the fine-tuned numbers are taken from the original paper without rerunning those models on the same splits. Differences in train/test composition or label distribution across protocols can easily produce gaps of the size reported. The authors should either rerun the fine-tuned models under the same evaluation protocol or clearly label the fine-tuned numbers as an external benchmark rather than a controlled comparison. This is load-bearing because the abstract and conclusion use the 'within 0.6 points' language as the paper's main justification for frozen-model monitoring.
- [Section 3.3.1 and Tables 3–5] The RFM decision threshold is swept on the split being reported, so the RFM accuracies and F1 scores in Tables 3–5 are the best attainable on that split rather than the values a frozen threshold would produce. The paper's own supplementary check reports that refitting the threshold on validation and applying it to the test split lowers accuracy by 0.011 on average and F1 by 0.025, with worst cases of 0.038 and 0.136. This optimistic reporting should be corrected: the tables should present the fixed-threshold values, and all claims about RFM's relative performance should be based on those values. The issue is acknowledged in the text, but it still affects every RFM number in the results and the 'seven of twelve' comparison with the model's own answer.
minor comments (6)
- [Section 4.2] The sentence 'the single exception is a tie (0.975 against 0.978 for Llama-3.1-8B on Environmental)' is internally inconsistent: 0.975 and 0.978 are not equal, so this is not a tie. The numbers should be reconciled or the wording corrected.
- [Table 2] The column headers contain intrusive spacing ('Y es (T rue)', 'No (F alse)', 'T otal'); this is a typesetting artifact that should be fixed.
- [Appendix A, Table 6] Several wrapper rows are visually similar and hard to distinguish. Assigning each wrapper a label (e.g., W1–W6) would make the 9.4-point sensitivity range easier to follow.
- [Section 3.3.1] Please clarify whether the precision, recall, F1, and AUC values for RFM are computed at the accuracy-optimizing threshold or at some other operating point, since the text describes the threshold sweep only in terms of accuracy.
- [Section 5] The paper says 'a few minutes of GPU time per pillar' but later quantifies the cost as under four minutes for RFM and about five minutes for the full probe. These statements are consistent but could be consolidated to avoid seeming imprecise.
- [General] The paper does not state whether code and data will be publicly released. Given the reproducibility claims, an availability statement would be helpful.
Circularity Check
No circularity: the activation-based measures are trained on labeled activations and evaluated on held-out text; self-citations are background, and the disclosed test-split threshold sweep and best-of-N comparisons are statistical limitations, not definitional reductions.
full rationale
The paper's central claim—that a linear probe on frozen LLM activations approaches fine-tuned classifier accuracy—is not circular. The probe is trained on labeled activations within a nested 5-fold cross-validation and evaluated on held-out folds (Section 3.3.2), and the RFM concept vectors are extracted from a training split and scored on a separate test split (Section 3.3.1). The fine-tuned accuracies come from the external benchmark paper (Schimanski et al., 2024), not from the present authors' definitions; the like-for-like probe-vs-RFM split comparison and the probe-vs-own-answer comparisons are independent internal checks. Cited prior work by the authors (Zohrehvand et al. 2024; Choudhury et al. 2026) appears only as background or as a future-looking aside and is not load-bearing. The disclosed practice of sweeping the RFM threshold on the reported split is a form of test-set fitting that inflates the RFM numbers, and the headline 0.6-point gap is selected as the maximum over nine probe configurations against the maximum over two fine-tuned models, but these are statistical/selection concerns, not reductions of the result to its inputs. Similarly, the comparison to published fine-tuned models without a shared evaluation protocol is an external-validity limitation, not circularity. No equation or parameter is defined in terms of the quantity it is said to predict.
Assumptions & free parameters
free parameters (4)
- RFM decision threshold =
per pillar, optimized on the test split, not tabulated for the main results
- Number of RFM concept vectors N =
5
- Wrapper template =
best of 6 on validation; winner varies by model and pillar
- Ridge regularization strength =
not reported
assumptions (5)
- domain assumption Linear representation hypothesis: concepts are linearly accessible as directions in LLM activation space.
- domain assumption Pooled sentence vector preserves concept-relevant signal.
- domain assumption Human-annotated binary ESG labels are a valid ground truth for concept presence.
- domain assumption The published fine-tuned classifier metrics are directly comparable to the reported probe and RFM metrics.
- standard math Standard linear algebra and supervised learning assumptions: iid samples, train/test separation, eigendecomposition of the AGOP matrix.
Cite this review
Pith. "Pith review of Measuring Concept Content in Text from LLM Activations: ESG Evidence from Concept Vectors and Linear Probes." pith.science (2026). https://pith.science/paper/HMXDVGXD
@misc{pith2026260807208,
author = {Pith},
title = {Pith review of: Measuring Concept Content in Text from LLM Activations: ESG Evidence from Concept Vectors and Linear Probes},
year = {2026},
howpublished = {\url{https://pith.science/paper/HMXDVGXD}},
note = {Machine review of arXiv:2608.07208}
}
read the original abstract
Existing measures of how much a text is about a concept read the surface of the text: dictionary word shares, topic proportions, embedding similarities. They score the words a text uses, not the judgment a reader forms about it. Recent work has shown that a gap exists in what Large Language Models (LLMs) know internally versus what they express in their response. This paper asks whether that internal knowledge, read by monitoring the activations of frozen, out-of-the-box LLMs, can stand in for task-specific fine-tuning when measuring concept content, and which extraction method reads it best. We extract such measures via the Recursive Feature Machine (RFM) algorithm and via linear probing, and compare these against an embedding baseline, surface baselines, and the same model's own answer to the question. We demonstrate the approach on financial text, a domain studied extensively and served by established annotated resources, using a human-annotated Environmental, Social and Governance (ESG) dataset. The best linear probe comes within 0.6 percentage points of a fine-tuned domain classifier's accuracy without any task-specific fine-tuning, and outscores the same model's own answer to the question in eleven of twelve comparisons, so the activations carry concept content the response does not report. The simple probe consistently beats the RFM concept vectors, which in turn provide what classification alone does not: a continuous score intended to reflect how strongly a concept is present in a text, whose validation awaits graded labels.
Figures
Reference graph
Works this paper leans on
-
[1]
DBLP:conf/iclr/AlainB17 APACrefauthors Alain, G. \ Bengio, Y. APACrefauthors \ 2017 . Understanding intermediate layers using linear classifier probes Understanding intermediate layers using linear classifier probes . 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings . 5th In...
work page 2017
-
[2]
azaria-mitchell-2023-internal APACrefauthors Azaria, A. \ Mitchell, T. APACrefauthors \ 2023 12 . The Internal State of an LLM Knows When It ' s Lying The internal state of an LLM knows when it ' s lying . H. Bouamor, J. Pino \ K. Bali\ ( ), Findings of the Association for Computational Linguistics: EMNLP 2023 Findings of the Association for Computational...
-
[3]
beaglehole2026xrfm APACrefauthors Beaglehole, D. , Holzm \"u ller, D. , Radhakrishnan, A. \ Belkin, M. APACrefauthors \ 2026 . x RFM : Accurate, scalable, and interpretable feature learning models for tabular data x RFM : Accurate, scalable, and interpretable feature learning models for tabular data . The Fourteenth International Conference on Learning Re...
work page 2026
-
[4]
doi:10.1126/science.aea6792 APACrefauthors Beaglehole, D. , Radhakrishnan, A. , Boix-Adserà, E. \ Belkin, M. APACrefauthors \ 2026 . Toward universal steering and monitoring of AI models Toward universal steering and monitoring of AI models . Science 391 6787 787-792 . APACrefDOI doi:10.1126/science.aea6792 APACrefDOI
-
[5]
10.1162/coli_a_00422 APACrefauthors Belinkov, Y. APACrefauthors \ 2022 04 . Probing Classifiers: Promises, Shortcomings, and Advances Probing classifiers: Promises, shortcomings, and advances . Computational Linguistics 48 1 207-219 . APACrefDOI doi:10.1162/coli_a_00422 APACrefDOI
-
[6]
burns2023discovering APACrefauthors Burns, C. , Ye, H. , Klein, D. \ Steinhardt, J. APACrefauthors \ 2023 . Discovering Latent Knowledge in Language Models Without Supervision Discovering latent knowledge in language models without supervision . The Eleventh International Conference on Learning Representations ( ICLR ) . The Eleventh International Confere...
work page 2023
-
[7]
ChoudhuryEtAl2026 APACrefauthors Choudhury, P. , Vanneste, B. \ Zohrehvand, A. APACrefauthors \ 2026 . The Wade Test: Generative AI and a CEO bot. The wade test: Generative AI and a CEO bot. Harvard Business School Working Paper No. 25-008 APACrefDOI doi:10.2139/ssrn.4945933 APACrefDOI
-
[8]
DU2025102755 APACrefauthors Du, K. , Zhao, Y. , Mao, R. , Xing, F. \ Cambria, E. APACrefauthors \ 2025 . Natural language processing in finance: A survey Natural language processing in finance: A survey . Information Fusion 115 102755 . APACrefDOI doi:10.1016/j.inffus.2024.102755 APACrefDOI
arXiv 2025
Show all 25 references
-
[9]
, Kelly, B
gentzkow2019text APACrefauthors Gentzkow, M. , Kelly, B. \ Taddy, M. APACrefauthors \ 2019 . Text as Data Text as data . Journal of Economic Literature 57 3 535--574 . APACrefDOI doi:10.1257/jel.20181020 APACrefDOI
2019 doi
-
[10]
, Sofroniew, N
gurnee2026workspace APACrefauthors Gurnee, W. , Sofroniew, N. , Pearce, A. , Piotrowski, M. , Kauvar, I. , Chen, R. Lindsey, J. APACrefauthors \ 2026 . Verbalizable Representations Form a Global Workspace in Language Models Verbalizable representations form a global workspace ...
2026
-
[11]
\ Tegmark, M
gurnee2024language APACrefauthors Gurnee, W. \ Tegmark, M. APACrefauthors \ 2024 . Language models represent space and time Language models represent space and time . The Twelfth International Conference on Learning Representations ( ICLR ) . The Twelfth International Conferen...
2024 arXiv
-
[12]
\ Liang, P
hewitt-liang-2019-designing APACrefauthors Hewitt, J. \ Liang, P. APACrefauthors \ 2019 11 . Designing and Interpreting Probes with Control Tasks Designing and interpreting probes with control tasks . K. Inui, J. Jiang, V. Ng \ X. Wan\ ( ), Proceedings of the 2019 Conference o...
2019
-
[13]
\ Kennard, R W
Hoerl01021970 APACrefauthors Hoerl, A E. \ Kennard, R W. APACrefauthors \ 1970 . Ridge Regression: Biased Estimation for Nonorthogonal Problems Ridge regression: Biased estimation for nonorthogonal problems . Technometrics 12 1 55--67 . APACrefDOI doi:10.1080/00401706.1970.104...
1970
-
[14]
, Taddy, M
kozlowski2019geometry APACrefauthors Kozlowski, A C. , Taddy, M. \ Evans, J A. APACrefauthors \ 2019 . The geometry of culture: Analyzing the meanings of class through word embeddings The geometry of culture: Analyzing the meanings of class through word embeddings . American S...
2019 doi
-
[15]
, Mai, F
li2021measuring APACrefauthors Li, K. , Mai, F. , Shen, R. \ Yan, X. APACrefauthors \ 2021 . Measuring corporate culture using machine learning Measuring corporate culture using machine learning . The Review of Financial Studies 34 7 3265--3315 . APACrefDOI doi:10.1093/rfs/hha...
2021 doi
-
[16]
\ McDonald, B
loughran2011liability APACrefauthors Loughran, T. \ McDonald, B. APACrefauthors \ 2011 . When Is a Liability Not a Liability? Textual Analysis, Dictionaries, and 10-Ks When is a liability not a liability? textual analysis, dictionaries, and 10-ks . The Journal of Finance 66 1 ...
2011
-
[17]
\ Tegmark, M
marks2024geometry APACrefauthors Marks, S. \ Tegmark, M. APACrefauthors \ 2024 . The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets The geometry of truth: Emergent linear structure in large language model representat...
2024
-
[18]
, Choe, Y J
pmlr-v235-park24c APACrefauthors Park, K. , Choe, Y J. \ Veitch, V. APACrefauthors \ 2024 jul . The Linear Representation Hypothesis and the Geometry of Large Language Models The linear representation hypothesis and the geometry of large language models . R. Salakhutdinov\ . \...
2024
-
[19]
\ Gurevych, I
reimers-gurevych-2019-sentence APACrefauthors Reimers, N. \ Gurevych, I. APACrefauthors \ 2019 11 . Sentence- BERT : Sentence Embeddings using S iamese BERT -Networks Sentence- BERT : Sentence embeddings using S iamese BERT -networks . K. Inui, J. Jiang, V. Ng \ X. Wan\ ( ), P...
2019
-
[20]
, Reding, A
schimanski2024bridging APACrefauthors Schimanski, T. , Reding, A. , Reding, N. , Bingler, J. , Kraus, M. \ Leippold, M. APACrefauthors \ 2024 . Bridging the gap in ESG measurement: Using NLP to quantify environmental, social, and governance communication Bridging the gap in ES...
2024
-
[21]
, Das, D
tenney-etal-2019-bert APACrefauthors Tenney, I. , Das, D. \ Pavlick, E. APACrefauthors \ 2019 07 . BERT Rediscovers the Classical NLP Pipeline BERT rediscovers the classical NLP pipeline . A. Korhonen, D. Traum \ L. M \`a rquez\ ( ), Proceedings of the 57th Annual Meeting of t...
2019
-
[22]
, Shazeer, N
NIPS2017_3f5ee243 APACrefauthors Vaswani, A. , Shazeer, N. , Parmar, N. , Uszkoreit, J. , Jones, L. , Gomez, A N. Polosukhin, I. APACrefauthors \ 2017 . Attention is All you Need Attention is all you need . I. Guyon\ . \ ( ), Advances in Neural Information Processing Systems A...
2017
-
[23]
, Duan, H
yang2024llmmeasure APACrefauthors Yang, Y. , Duan, H. , Liu, J. \ Tam, K Y. APACrefauthors \ 2024 . LLM-Measure : Generating valid, consistent, and reproducible text-based measures for social science research. LLM-Measure : Generating valid, consistent, and reproducible text-b...
2024 arXiv
-
[24]
, Doshi, A R
zohrehvand2024event APACrefauthors Zohrehvand, A. , Doshi, A R. \ Vanneste, B S. APACrefauthors \ 2024 . Generalizing event studies using synthetic controls: An application to the Dollar Tree -- Family Dollar acquisition Generalizing event studies using synthetic controls: An ...
2024
-
[25]
, Phan, L
zou2023transparency APACrefauthors Zou, A. , Phan, L. , Chen, S. , Campbell, J. , Guo, P. , Ren, R. Hendrycks, D. APACrefauthors \ 2023 . Representation Engineering: A Top-Down Approach to AI Transparency. Representation engineering: A top-down approach to AI transparency
2023
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.