Pith. sign in

REVIEW 4 major objections 4 minor 20 references

LLaMAs Have Feelings Too: Unveiling Sentiment and Emotion Representations in LLaMA Models Through Probing

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read The paper claims that sentiment in LLaMA models is most detectable in mid-layer residual streams, emotions in early layers, and that a LLaMA truncated to its most representative layer plus a lightweight probe beats prompting.

desk verdict The qualitative layer map of sentiment in LLaMA is probably right, but the headline numbers are inflated by test-set selection; still worth a serious look. read the letter →

arxiv 2505.16491 v2 pith:G5CCOJBT submitted 2025-05-22 cs.CL cs.AI

classification cs.CLcs.AI
keywords sentimentanalysisemotionclassificationprobingclassifiersLLaMAlayer-wisepoolingstrategiesmodeltruncationefficientinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks where sentiment and emotion information lives inside LLaMA models and whether that location can be put to practical use. By training probe classifiers on residual-stream representations at each layer, it argues that binary positive/negative polarity is most detectable in middle layers, while fine-grained emotions are most detectable in early layers. It also claims that the common practice of using the last token as a sentence summary is suboptimal, with a concatenation of mean, max, and min pooling often doing better. Building on the layer map, the paper introduces SENTRILLAMA, a LLaMA truncated to the most representative layer and fitted with a lightweight classification head, which it reports matches or beats prompting-based LLaMA and approaches fine-tuned DeBERTa and RoBERTa accuracy while reducing memory use. A sympathetic reader would care because the result offers both a map of affective information in decoder-only transformers and a recipe for cheaper sentiment classifiers.

What carries the argument

The central object is the probe: a classifier (typically a linear SVM or logistic regression) trained on the residual-stream representation $x^{(l+1)}$ at a given layer $l$, pooled over tokens by one of six methods (last-token, mean, max, min, concatenation of mean-max-min, and attention-weighted mean). A forward pass through the LLaMA model produces the representation, the probe is trained on labeled sentiment data, and the layer with the highest test accuracy is taken to be where sentiment is most represented. SENTRILLAMA then keeps only the embedding layer and the transformer layers up to that best layer, replaces the language-model head with a lightweight classifier, and thereby defines a smaller model that inherits the sentiment-relevant representations.

What would settle it

A concrete control would be a label-shuffle test: train the same probes on the same residual-stream features with randomly permuted sentiment labels across all layers. If the mid-layer accuracy peak for binary polarity survives at high levels, then the reported localization is an artifact of the probe, pooling, and layer-selection procedure rather than a genuine concentration of sentiment information.

Watch

Extended reading notes

Core claim

The central claim is that sentiment and emotion concepts are encoded in LLaMA's hidden layers in a localizable way: linear classifiers reach roughly 90 percent accuracy on binary polarity when reading the residual stream at middle layers, and around 70 percent for six-way emotion classification when reading early layers. The paper further claims that the last token is not consistently the best pooling choice, and that a truncated model retaining only layers up to the most representative one plus a linear SVM matches or exceeds zero-shot, few-shot, and chain-of-thought prompting while approaching DeBERTa-large and RoBERTa-large performance with an average parameter reduction of about 57 percent. The reported best layer moves deeper as the model grows, from around layer 8 of 16 in the 1B model to around layer 20 of 32 in the 8B model on SST-2.

Load-bearing premise

The load-bearing premise is that probe accuracy at the most representative layer reflects where sentiment is actually encoded in the model, rather than dataset regularities or the probe-selection procedure itself; the paper selects the best layer, pooling, and classifier on the test set, so the localization claim is not an independent prediction.

Editorial extensions

If this is right

  • Layer-specific probing can serve as an alternative to prompting for sentiment classification, with reported accuracy gains of up to 14 percent.
  • Truncating LLaMA at the most representative layer cuts memory requirements by roughly 57 percent on average while retaining most of the accuracy, making large-model sentiment classifiers cheaper to deploy.
  • Pooling choice matters: concatenating mean, max, and min representations often outperforms the last token, which suggests that studies relying solely on last-token representations may be leaving signal unused.
  • Larger models encode sentiment more cleanly, with the best layer appearing deeper, so scale and layer choice interact when designing downstream probes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the localization holds, the same truncation recipe could extend to other classification tasks with labeled data, such as sarcasm or intent detection, though the paper does not test this.
  • If sentiment is indeed most readable in a single mid-layer, activation-steering and representation-engineering methods that target one layer could be aimed at that identified layer for more effective interventions.
  • A direct test of the localization claim would be a label-shuffle control: if the mid-layer accuracy peak survives when probe labels are randomly permuted, the result is far more likely to reflect genuine representation structure rather than dataset artifacts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper probes Llama-3 (1B, 3B, 8B, instruction-tuned) hidden states for binary sentiment and six-way emotion classification. It compares six pooling methods and twelve probe classifiers across layers, reports where sentiment/emotion information is 'most represented,' proposes SENTRILLAMA, a model truncated to the most representative layer plus a lightweight classifier, and argues that this approach outperforms prompting, approaches DeBERTa/RoBERTa on some datasets, and reduces memory usage by 57% on average. The central scientific claims are that binary sentiment is best detected in mid layers, emotions in early layers, that last-token pooling is not consistently best, and that layer-specific probing is a practical alternative to prompting.

Significance. If the claims are correct, the paper offers a practical, parameter-efficient recipe for sentiment classification and a useful map of where affective information resides in decoder-only LMs. The study has real strengths: a broad experimental grid across three model scales and four datasets, six pooling methods, twelve classifiers, shared code (stated), and deterministic seed configuration. The qualitative picture is suggestive and largely consistent across models and datasets. However, the headline quantitative claims rest on a protocol that selects the best layer/pooling/probe on the test set and then reports accuracy on that same test set, so the magnitudes of the claimed gains and the SENTRILLAMA comparisons are not established as stated. The contribution is potentially valuable but needs a corrected evaluation protocol before its central quantitative conclusions can be accepted.

major comments (4)
  1. [§4, §5, Tables 3–4] The central quantitative results are selected on the test set. Section 4 states that probe accuracy is measured on the test set, Table 3 is built by ranking configurations by test accuracy, and Section 5 defines the 'most representative layer' L_i as that same test-selected row, which Table 4 then reports as SENTRILLAMA's accuracy on the same test set. With a search space of roughly 15–32 layers × 6 pooling methods × 12 classifiers, taking the maximum on the held-out test set inflates the expected accuracy and invalidates the comparison against prompting, DeBERTa, and RoBERTa in Table 4. Please redo the selection with a proper validation split (or nested cross-validation), then report test accuracy only for the configuration selected on validation, with confidence intervals or repeated-seed statistics.
  2. [Abstract, §5, Table 4] The abstract's claim of 'detection accuracy increasing up to 14% over prompting techniques' is not traceable to Table 4. The reported SENTRILLAMA-versus-prompting gaps range from a few points (e.g., 0.9594 vs. 0.9154 on SST-2 for the 3B model) to more than 45 points (e.g., 0.8015 vs. 0.3475 on Emotion for the 1B model). No row yields a 14% gap. Either identify the exact comparison being summarized or correct/remove the claim.
  3. [§4, §6, Conclusion, Figures 1 and 6] The localization claims ('mid-layers for binary sentiment, early layers for emotions') are post-hoc selections rather than independent predictions. Because the best layer is chosen by maximizing test accuracy and the paper reports a single split with no error bars, the qualitative layer trends are suggestive but not statistically supported. Please add a validation-based selection procedure and report per-layer confidence intervals (bootstrap or repeated seeds) so the localization pattern can be distinguished from selection noise.
  4. [§5, Table 4, Appendix A] The comparison against DeBERTaV3-large and RoBERTa-large is not fully specified. The paper does not state whether these baselines are fine-tuned on the same reduced training splits used for SENTRILLAMA or on the original full IMDB and Emotion datasets. If the baselines use the full datasets while SENTRILLAMA is selected on the test set, the comparison is doubly favorable to SENTRILLAMA. Please specify the baseline training protocol, and ideally fine-tune the baselines on the same reduced splits used for the proposed method.
minor comments (4)
  1. [§3, Choice of Representation] The phrase 'we adopt the residuals stream' contains a typo; it should read 'the residual stream.'
  2. [Appendix G] The heading 'Compuattional Efficiency of SENTRILLAMA' contains a typo; it should be 'Computational Efficiency.'
  3. [Appendix F, Figure 7] The caption for subfigure (g) reads '1B-Instruct on Rotten Rotten'; the dataset name should be 'Rotten Tomatoes.'
  4. [§3, footnote 1] The code availability footnote states 'Sentiment Probing Toolkit' but does not include a URL. Please provide a working link or repository identifier.

Circularity Check

2 steps flagged · score 6.0 of 10

SENTRILLAMA's reported accuracy and best-layer localization are selections made on the test set, not independent predictions.

  1. fitted input called prediction [Section 4 (Sentiment detection results), Table 3]
    "For every probing datasets we train the classifier Cw on the training set, and evaluate its detection performance by measuring the accuracy on the test set. ... The results, summarized in Table 3, which shows the top-3 performers by datasets, reveal that concatenating the mean, max, and min of repθ is often the most effective method for detecting sentiment concepts."

    The best layer, pooling method, and probe classifier are selected by ranking test accuracy, and the reported findings (mid-layers for binary polarity, early layers for emotion, last-token not optimal) are summaries of these test-set maxima. Because the same test set is used both for selection and for reporting, the reported numbers are the selection objective rather than independent estimates. No held-out validation split, repeated-seed statistics, or error bars are provided, so the specific best-layer claims and their magnitudes are forced by the selection procedure rather than being predictions.

  2. fitted input called prediction [Section 5 (SENTRILLAMA), Tables 4-5]
    "Based on our earlier analysis, we identified the optimal layer Li, which retains the most salient features for downstream tasks. ... The proposed SENTRILLAMAapproach reduces model complexity by retaining only the input embedding and the layers up to the most representative layer (i), identified in previous experiments, while replacing the LM head with a lightweight classification head, such as a linear SVM. ... Table 4 presents a comparative analysis of the accuracy of SENTRILLAMA, DeBERTa, RoBERTa, and a prompt-based method."

    The SENTRILLAMA accuracy reported in Table 4 is the test accuracy of the configuration already selected as the top performer in Table 3; operationally, the 'most representative layer' is the layer that maximizes test accuracy on the same test set used for the comparison. Therefore the claimed gains over prompting and parity with DeBERTa/RoBERTa are the maximum over the searched grid of layers, pooling methods, and classifiers, evaluated on the very test set used for benchmarking. The parameter-reduction percentages are arithmetic facts, but the accuracy comparison that supports the 'state-of-the-art performance' claim is the selection criterion itself, not an independent prediction.

full rationale

This is not a case of load-bearing self-citation: the probing framework is attributed to external work (von Rütte et al., 2024), and the authors' self-citations are contextual rather than foundational. The core circularity is test-set selection presented as discovery. Section 4 ranks layer/pooling/probe configurations by accuracy on the test set and derives the localization findings from those maxima; Section 5 then reuses the top-ranked configuration as 'the most representative layer' and reports the same test-set numbers as SENTRILLAMA's accuracy in Table 4, comparing them with DeBERTa, RoBERTa, and prompting baselines. Thus the central quantitative claim (SENTRILLAMA approaches DeBERTa/RoBERTa while cutting memory by roughly half) is the fitted maximum of the search grid, not an independent evaluation. No validation split, repeated-seed statistics, or error bars are reported, and the Limitations section only says 'there is scope for exploring alternative methodologies within this selection mechanism' without flagging that selection was made on the test set. The qualitative pattern (binary sentiment peaks in middle layers, emotions in early layers) is more robust because it is a coarse reading of the full accuracy curves, which is why the score is 6 rather than higher; nevertheless, the headline best-layer numbers and the SENTRILLAMA comparison reduce to the selection objective, making the central applied claim partially circular by construction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the probing assumption that classifier accuracy on residual-stream representations indicates internal concept encoding, on the unreported preliminary choice of residual stream over alternatives, and on test-set-based selection of layers, pooling, and probes. The reduced datasets and manually designed prompts are additional uncontrolled choices.

free parameters (5)
  • best layer index L_i per dataset = e.g., SST2: 10 (1B), 5 or 12 (3B), 3 or 14 (8B); Emotion: 0 or 1
    Selected by the highest test accuracy per dataset in Table 3; no validation split, so this is a fitted hyperparameter.
  • pooling method per dataset = concat-mean-max-min, mean, attention, max, last-token depending on dataset
    Chosen by test accuracy from six alternatives; see Table 3.
  • probe classifier per dataset = linear SVM, non-linear SVM, logistic regression, MLP, LightGBM, etc.
    Chosen by test accuracy from twelve classifiers; see Table 3.
  • Optuna hyperparameters = not reported in text
    Five optimization trials per classifier; final parameter values and validation protocol are not given, so the tuning is not auditable.
  • reduced dataset sizes = IMDB 7000 train and 7000 test; Emotion 6000 train and 2000 test
    Subsampled to keep label distribution and mean length; other dataset properties are not preserved by construction.
assumptions (5)
  • domain assumption Probe classifier accuracy on residual-stream representations measures whether the model internally encodes the sentiment concept.
    Standard probing assumption used throughout Section 4; no control for lexical cues or positional artifacts.
  • ad hoc to paper The residual stream is the best representation substrate for sentiment readout.
    Section 3: 'Based on our preliminary experiments showing marginally higher detection accuracy, we adopt the residual stream.' The preliminary comparison is not reported.
  • domain assumption Reduced IMDB and Emotion datasets preserve the properties relevant to sentiment probing.
    Appendix A matches label distribution and average sentence length only; other distributional properties are unverified.
  • domain assumption The hand-written prompts are fair baselines for zero-shot, few-shot, and Chain-of-Thought performance.
    Prompt templates in Appendix C are not optimized and the CoT variant suppresses reasoning output, which may understate prompting performance.
  • domain assumption Layers after the most representative layer do not contribute useful information for the classification head.
    SENTRILLAMA truncates at L_i; this is the core premise of the efficiency gain and is validated only via the test-set-selected L_i.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLaMAs Have Feelings Too: Unveiling Sentiment and Emotion Representations in LLaMA Models Through Probing." pith.science (2026). https://pith.science/paper/G5CCOJBT

@misc{pith2026250516491,
  author       = {Pith},
  title        = {Pith review of: LLaMAs Have Feelings Too: Unveiling Sentiment and Emotion Representations in LLaMA Models Through Probing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G5CCOJBT}},
  note         = {Machine review of arXiv:2505.16491}
}
read the original abstract

Large Language Models (LLMs) have rapidly become central to NLP, demonstrating their ability to adapt to various tasks through prompting techniques, including sentiment analysis. However, we still have a limited understanding of how these models capture sentiment-related information. This study probes the hidden layers of Llama models to pinpoint where sentiment features are most represented and to assess how this affects sentiment analysis. Using probe classifiers, we analyze sentiment encoding across layers and scales, identifying the layers and pooling methods that best capture sentiment signals. Our results show that sentiment information is most concentrated in mid-layers for binary polarity tasks, with detection accuracy increasing up to 14% over prompting techniques. Additionally, we find that in decoder-only models, the last token is not consistently the most informative for sentiment encoding. Finally, this approach enables sentiment tasks to be performed with memory requirements reduced by an average of 57%. These insights contribute to a broader understanding of sentiment in LLMs, suggesting layer-specific probing as an effective approach for sentiment tasks beyond prompting, with potential to enhance model utility and reduce memory requirements.

Figures

Figures reproduced from arXiv: 2505.16491 by the authors.

Figure 1
Figure 1. Layer-wise probing accuracy using the Last-Token approach on SST-2; Appendix [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Mean Pooling Visually Explained Feature Activations the smell of freshly LastToken Pooling [PITH_FULL_IMAGE:figures/full_fig_p016_2.png] view at source ↗
Figure 3
Figure 3. Last Token Pooling Visually Explained Feature Activations the smell of freshly Max/Min Pooling [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Max and Min Pooling Visually Explained SoftMax Function 0.003 0.042 0.011 0.999 Importance Score 4 1 2 3 11 14 13 12 22 21 24 23 33 34 31 32 the smell of freshly 0.012 0.042 0.022 2.997 0.033 0.588 0.143 11.988 0.066 0.882 0.264 22.977 0.099 1.428 0.341 31.968 the smel…
Figure 6
Figure 6. Figure 6: Layer-wise probing accuracy using the last token on SST-2, IMDB, Rotten Tomatoes, and Emotion. [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Layer-wise pooling methods confidence accuracy on SST-2, IMDB, Rotten Tomatoes, and Emotion. [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 13 canonical work pages

  1. [1]

    2.Calculate Dataset Statistics: (a) Compute the label distribution to main- tain the proportion of samples for each label in the original dataset

    Initialize Random Seed: To achieve repro- ducible datasets, we initialized a random seed (Appendix B) at the beginning of the work- flow, ensuring that the sampling process could be replicated exactly in future runs. 2.Calculate Dataset Statistics: (a) Compute the label distribution to main- tain the proportion of samples for each label in the original da...

  2. [2]

    Association for Computational Linguistics

    Is compound aspect-based sentiment analysis addressed by LLMs? InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 7836–7861, Miami, Florida, USA. Association for Computational Linguistics. Giovanni Maria Biancofiore, Dario Di Palma, Claudio Pomo, Fedelucio Narducci, and Tommaso Di Noia

  3. [3]

    (b) Random sample to select the required number of samples for each label

    Generate the Dataset: For each label in the dataset, we followed a sampling process as follows: (a) Filter samples by sentence length for the given label. (b) Random sample to select the required number of samples for each label. (c) Combine the selected samples for all la- bels into a single dataset. To ensure ran- domness, the combined dataset was shuf-...

  4. [4]

    If the av- erage sentence length exceeded the specified constraint, an error was raised, indicating the need for further adjustments

    Validate Reduced Dataset: The label dis- tribution and average sentence length were recalculated for the reduced dataset. If the av- erage sentence length exceeded the specified constraint, an error was raised, indicating the need for further adjustments. To provide transparency and facilitate analysis, a summary of the preprocessing steps is reported:

  5. [16]

    The total number of samples in both the origi- nal and reduced datasets

  6. [17]

    The distribution of labels before and after re- duction

  7. [18]

    User:"Classify the sentiment of the following text: ‘{text}’

    The average sentence length in both datasets. This comprehensive preprocessing workflow en- sured that the reduced datasets retained the sta- tistical properties of the original datasets while adhering to predefined constraints. The code used to generate the datasets is provided in ‘dataset/reduce_dataset.py,‘ along with a de- tailed PDF report containing...

  8. [19]

    Weighted Sum of Adjusted Embeddings

Show all 20 references
  1. [20]

    Published as a conference paper at ACL 2025 Main F Layer-wise pooling methods confidence accuracy on SST-2, IMDB, Rotten Tomatoes, and Emotion datasets

    Average Across Dimension 41 23 11 1413 12 2221 2423 33 3431 32 the smell of freshly Mean Vector 2.5 22.5 12.5 32.5 Figure 5: Attention Pooling Visually Explained Published as a conference paper at ACL 2025 Main E Comprehensive Results for the Last-Token Approach 0 2 4 6 8 10 1...

  2. [751]

    Roser Morante and Eduardo Blanco

    The Association for Computational Linguistics. Roser Morante and Eduardo Blanco. 2012. *sem 2012 shared task: Resolving the scope and focus of nega- tion. In*SEM@NAACL-HLT, pages 265–274. Asso- ciation for Computational Linguistics. nostalgebraist. 2020. Interpreting gpt: The ...

  3. [1145]

    Fabian Pedregosa, Gaël Varoquaux, Alexandre Gram- fort, Vincent Michel, Bertrand Thirion, Olivier Grisel, and Mathieu Blondel et al

    ACM. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gram- fort, Vincent Michel, Bertrand Thirion, Olivier Grisel, and Mathieu Blondel et al. 2011. Scikit-learn: Ma- chine learning in python.J. Mach. Learn. Res., 12:2825–2830. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patr...

  4. [1373]

    Dario Di Palma, Felice Antonio Merra, Maurizio Sfilio, Vito Walter Anelli, Fedelucio Narducci, and Tom- maso Di Noia

    ACM. Dario Di Palma, Felice Antonio Merra, Maurizio Sfilio, Vito Walter Anelli, Fedelucio Narducci, and Tom- maso Di Noia. 2025. Do llms memorize recommen- dation datasets? a preliminary study on movielens- 1m. InProceedings of the 48th International ACM SIGIR Conference on Re...

  5. [1751]

    Filip Klubicka and John D

    ACL. Filip Klubicka and John D. Kelleher. 2022. Probing with noise: Unpicking the warp and weft of embed- dings. InBlackboxNLP@EMNLP, pages 404–417. Association for Computational Linguistics. Jan Ole Krugmann and Jochen Hartmann. 2024. Senti- ment analysis in the age of genera...

  6. [1804]

    Frank Xing

    Association for Computational Linguistics. Frank Xing. 2024. Designing heterogeneous llm agents for financial sentiment analysis.ACM Transactions on Management Information Systems. Yadollah Yaghoobzadeh, Katharina Kann, Timothy J. Hazen, Eneko Agirre, and Hinrich Schütze. 2019...

  7. [1997]

    InACL, pages 174–181

    Predicting the semantic orientation of adjec- tives. InACL, pages 174–181. Morgan Kaufmann Publishers / ACL. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. Debertav3: Improving deberta using electra-style pre- training with gradient-disentangled embedding shar- ing. InICLR...

  8. [2002]

    InEMNLP, pages 79– 86

    Thumbs up? sentiment classification using machine learning techniques. InEMNLP, pages 79– 86. Vincenzo Paparella, Dario Di Palma, Vito Walter Anelli, and Tommaso Di Noia. 2023. Broadening the scope: Evaluating the potential of recommender systems beyond prioritizing accuracy. ...

  9. [2013]

    InHLT-NAACL, pages 746–

    Linguistic regularities in continuous space word representations. InHLT-NAACL, pages 746–

  10. [2023]

    Representation engineering: A top-down ap- proach to AI transparency.CoRR, abs/2310.01405. Published as a conference paper at ACL 2025 Main Appendix A Detailed Preprocessing Workflow The IMDB and Emotion datasets were prepro- cessed due to their original sizes—50K for IMDB and...

  11. [2024]

    InICACS, pages 1–9

    Leveraging large language models and prompt settings for context-aware financial sentiment analy- sis. InICACS, pages 1–9. IEEE. Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next- generation hyperparameter optimization framewo...

  12. [2025]

    In Human-Centered AI: An Illustrated Scientific Quest, pages 399–438

    Conversational user interfaces and agents. In Human-Centered AI: An Illustrated Scientific Quest, pages 399–438. Springer. Monali Bordoloi and Saroj K. Biswas. 2023. Sentiment analysis: A survey on design framework, applications and future scopes.Artif. Intell. Rev., 56(11):12...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.