REVIEW 4 major objections 6 minor 46 references
Beyond Vision: How Large Language Models Interpret Facial Expressions from Valence-Arousal Values
T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read LLMs can translate two emotion numbers into human-like face descriptions
desk verdict Solid negative result on VA-to-category mapping; the 'close alignment' claim for free-text descriptions is not supported without a proper baseline. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanisms are valence-arousal values as structured numerical proxies for facial expressions and the semantic-similarity pipeline used to evaluate the descriptions. FaceChannel, an off-the-shelf deep network, extracts valence and arousal values in $[-1, 1]$ from each face image; these two numbers are inserted into a fixed prompt that asks the LLM to categorize or describe the expression. For the description task, LLM output is compared with human annotations in two ways: combined similarity (whole LLM description versus concatenated human category definitions) and separate similarity (an $n \times n$ matrix of sentence-level cosine similarities averaged). Scores are computed with three sentence-embedding models—Word2Vec, BERT, and a Transformer encoder—and tested against a baseline of .5 via one-sample t-tests. The evaluation's choice of embedding is decisive: Word2Vec and BERT produce high mean similarities, while the Transformer embedding produces scores statistically indistinguishable from chance.
What would settle it
Run Experiment 2 again with exactly the same prompts, datasets, and embedding models, but shuffle the valence-arousal pairs across images before inserting them into the prompts: if the mean Word2Vec and BERT cosine similarities remain near .8, the high alignment scores are driven by the prompt template and the human definitions, not by the VA values; if the scores drop to the .5 baseline, the paper's VA-grounded inference claim is supported.
Extended reading notes
Core claim
The central claim is that an LLM given only two scalar values, valence and arousal, can generate semantic descriptions of a facial expression that align closely with what human observers write, even though the same model cannot reliably map those values to discrete emotion names. In the semantic description experiment on the Emotic dataset, GPT-4o-mini reached a mean combined cosine similarity of $M = .81$ with Word2Vec embeddings and $M = .79$ with BERT embeddings, significantly above the .5 random-chance baseline; GPT-4o and LLAMA 3.2 8B Instruct showed similar patterns. In the categorization experiments, by contrast, GPT-4o-mini achieved 30.42 percent and 31.42 percent accuracy on the IIMI basic-emotion task, with near-perfect scores only for happiness and sadness, and exact multi-label matches on the 26-emotion Emotic task ranged from 6.91 to 18.32 percent. The authors interpret the asymmetry as evidence that LLMs reason about affect better in open-ended language than through rigid category labels, consistent with gradient rather than categorical theories of emotion perception.
Load-bearing premise
The whole argument that LLM descriptions 'align closely with human interpretations' rests on treating cosine similarity between LLM text and concatenated human category definitions, computed with a chosen sentence-embedding model, as a valid measure of alignment—and the same outputs score at chance when the embedding model changes.
Editorial extensions
If this is right
- Affective inference from facial expressions does not require pixels: once valence and arousal are known, an LLM can provide human-like textual descriptions, opening a privacy-preserving path for emotion analysis.
- Free-text description, not discrete classification, is the mode in which LLMs currently succeed at affective inference, so benchmarks that measure only category accuracy will understate their useful capability.
- The high similarity scores are not universal across embedding models, meaning any claim about 'alignment with human interpretations' must specify the metric and embedding used.
- LLM affective reasoning from valence-arousal values is heavily biased toward polarized emotions such as happiness and sadness, and largely fails for anger, surprise, and other categories, so the models cannot yet serve as general emotion readers.
- A text-only affective layer could complement vision-language or sensor-based systems by converting their continuous affect estimates into interpretable natural-language explanations.
Reading between the lines
- If the same prompts are run with VA values randomly permuted across images—or replaced by random values drawn from the same distribution—and the Word2Vec/BERT similarity stays near .8, then the high scores reflect linguistic priors and the wording of the human category definitions, not the specific VA inputs.
- The near-chance Transformer-embedding result suggests the embedding model is a hidden moderator: future comparisons should fix one embedding and vary only the input VA values to measure how much of the similarity is actually controlled by arousal and valence.
- A practical extension the paper only gestures at: any wearable or sensor system that already outputs continuous valence and arousal could delegate natural-language emotion description to an LLM, never storing or transmitting face images; the chief risk would be carrying over training-data stereotypes about which emotions go with which VA values.
- Because anger and disgust can share similar valence-arousal coordinates, the categorization failures hint at an upper bound: VA values alone may not be able to disambiguate every discrete emotion, so dimensional models feeding LLMs will need context or action-unit information to reach fine-grained labels.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper tests whether LLMs can interpret facial expressions from valence-arousal (VA) values alone, using two tasks: discrete emotion categorization (basic emotions on IIMI, complex emotions on Emotic) and free-text semantic description of expressions (on Emotic). VA values are obtained from images using FaceChannel. For categorization, GPT-4o-mini and GPT-4o are evaluated against human labels; accuracy is low overall, with some success for polarized emotions such as happiness and sadness. For semantic description, GPT-4o, GPT-4o-mini, and LLaMA 3.2 8B generate sentences from VA values, which are compared to human-assigned category definitions via cosine similarity computed with Word2Vec, BERT, and an unspecified 'Transformers' embedding. The paper reports high similarity for Word2Vec and BERT (combined M between .75 and .81) but near-chance similarity for the 'Transformers' embedding (combined M between .35 and .42). The authors conclude that LLMs are capable of free-text affective inference from VA values, despite poor classification performance.
Significance. If validated, the finding that LLMs can convert VA values into human-aligned affective descriptions would be a useful contribution to privacy-preserving affective computing, as it suggests that structured numerical affect representations can substitute for raw images in some applications. The paper is also commendable for using external benchmark datasets, for not fitting any quantity to the evaluation data, and for reporting results with confidence intervals and significance tests. The categorization experiments are clearly described and their conclusions are appropriately cautious. However, the central positive claim about semantic description alignment rests on a semantic-similarity measurement that lacks crucial controls and reproducibility details, and the near-chance results from the 'Transformers' embedding remain unexplained. The paper's contribution would be substantially strengthened by addressing these issues.
major comments (4)
- [Section IV-B (Analysis)] The 'Transformers' embedding model is never specified. The citation [41] refers to Vaswani et al.'s architecture paper, not to a concrete pretrained sentence-embedding model. Table III reports combined similarities of .42, .39, and .35 for the three LLMs under this embedding, yet the reader cannot tell which model was used, whether it was a sentence-transformer, or how its parameters were configured. This makes the near-chance results impossible to reproduce and prevents interpretation of the large gap between this embedding and Word2Vec/BERT. The authors should name the exact model (e.g., a specific sentence-transformers checkpoint) and, ideally, justify its selection.
- [Section IV-B and Section IV-C (Results)] The claim that LLM descriptions 'align closely with human-generated interpretations' is not supported by the operationalization. The comparison in Experiment 2 is between LLM-generated sentences and the concatenated definitions of human-assigned emotion categories (e.g., 'Happiness: feeling delighted...'), not between LLM text and human free-text descriptions. The Emotic dataset provides discrete emotion category labels, not free-text explanations. Thus, the abstract and RQ2 overstate what is measured: high cosine similarity with category definitions may reflect the LLM's lexical knowledge of emotion words rather than its ability to infer the specific expression from VA values. A control condition comparing descriptions generated from random or permuted VA values against the same category definitions is needed to establish that the similarity is due to VA-grounded inference.
- [Section IV-C (Results)] The use of .5 as a 'random chance' baseline for cosine similarity is not justified. The authors cite Corley and Mihalcea [9] for this threshold, but that reference does not establish .5 as the chance level for cosine similarity between emotion-related sentences. With contextual embeddings, even semantically unrelated texts often have positive cosine similarity well above zero. The one-sample t-tests against .5 only show that the mean differs from .5; they do not show that the similarity exceeds what would be obtained by a no-information baseline. The authors should include a permutation or random-text control to calibrate the similarity scale, and should avoid interpreting a fixed .5 threshold as 'chance.'
- [Section V (Discussion) and Table III] The large discrepancy between Word2Vec/BERT (combined M ≈ .75–.81) and 'Transformers' (combined M ≈ .35–.42) is not adequately addressed. The Discussion attributes the difference to 'pre-trained embeddings capturing contextual and semantic relationships' versus 'purely structural representations,' but BERT is itself a Transformer-based model, so the terminology is confusing. The discrepancy could indicate that the chosen 'Transformer' embedding is a poor model for this task, that the similarity results are highly sensitive to embedding choice, or that the Word2Vec/BERT scores are inflated by lexical overlap. The paper should either identify the Transformer model, analyze the discrepancy, or temper the conclusion that LLM descriptions 'align closely' with human interpretations. As written, the headline result is not robust across embedding methods.
minor comments (6)
- [Section III-A2] Experiment 1.1 evaluates only GPT-4o-mini for basic emotion classification, while Experiments 1.2 and 2 also use GPT-4o and LLaMA. The authors should justify this asymmetry or acknowledge it as a limitation.
- [Section IV-B] The bootstrap procedure for confidence intervals is not described; the authors should state the number of resamples, the seed, and whether confidence intervals are percentile-based or bias-corrected.
- [Table II] The 'Image' column is empty in the example table. If the images are omitted for privacy or copyright reasons, please state this in the caption or table notes.
- [Section V] The phrase 'BERT and Word2Vec performed better than Transformers' is misleading because BERT is a Transformer-based model. Consider naming the third embedding method explicitly (e.g., 'the unspecified sentence-Transformer') to avoid confusion.
- [Throughout] The spacing in 'V A values' is inconsistent; the paper uses both 'VA' and 'V A' (e.g., in the abstract vs. Section II). Please standardize the notation.
- [Section IV-C] The text says 'Bootstrap testing calculated average cosine similarity scores and 95% confidence intervals,' but the subsequent t-test is against a fixed value of .5; the connection between the bootstrap results and the t-test should be made explicit.
Circularity Check
No significant circularity: all evaluated quantities come from external datasets and an external face model; no parameter is fitted to the data being scored.
full rationale
The study is an empirical evaluation, not a derivation. VA values are extracted by the external FaceChannel model and are fed to LLMs whose outputs are scored against independent human annotations (IIMI emotion labels and EMOTIC category definitions/descriptions). No component of the LLM or of the scoring pipeline is trained or fitted on the evaluation items, and no result is defined in terms of the quantity it is said to predict. The central RQ2 measure, cosine similarity between LLM-generated descriptions and human category definitions, is an explicitly stated operationalization of 'alignment' rather than a hidden re-use of the target; the fact that different embedding models give different similarities is a measurement-property issue, not circularity. The authors' self-citations ([10], [23]-[25], [34]) appear only as background motivation or dataset-annotation references and are not load-bearing for the reported accuracies or similarities. No equation reduces to its own inputs, no fitted parameter is renamed as a prediction, and no uniqueness claim is imported from the authors' prior work. Therefore the derivation chain is self-contained, with no circular step to quote.
Assumptions & free parameters
assumptions (4)
- domain assumption FaceChannel's predicted valence-arousal values faithfully represent the affective content of facial expressions in IIMI and Emotic images.
- domain assumption Cosine similarity between LLM-generated descriptions and human category definitions, computed with Word2Vec, BERT, and the unspecified Transformer model, is a valid measure of semantic alignment with human interpretations.
- ad hoc to paper A cosine similarity of 0.5 represents random chance for this task.
- ad hoc to paper The 3,047-image manually inspected test subset of Emotic is representative of the full test set.
Cite this review
Pith. "Pith review of Beyond Vision: How Large Language Models Interpret Facial Expressions from Valence-Arousal Values." pith.science (2026). https://pith.science/paper/OUG2YIGK
@misc{pith2026250206875,
author = {Pith},
title = {Pith review of: Beyond Vision: How Large Language Models Interpret Facial Expressions from Valence-Arousal Values},
year = {2026},
howpublished = {\url{https://pith.science/paper/OUG2YIGK}},
note = {Machine review of arXiv:2502.06875}
}
read the original abstract
Large Language Models primarily operate through text-based inputs and outputs, yet human emotion is communicated through both verbal and non-verbal cues, including facial expressions. While Vision-Language Models analyze facial expressions from images, they are resource-intensive and may depend more on linguistic priors than visual understanding. To address this, this study investigates whether LLMs can infer affective meaning from dimensions of facial expressions-Valence and Arousal values, structured numerical representations, rather than using raw visual input. VA values were extracted using Facechannel from images of facial expressions and provided to LLMs in two tasks: (1) categorizing facial expressions into basic (on the IIMI dataset) and complex emotions (on the Emotic dataset) and (2) generating semantic descriptions of facial expressions (on the Emotic dataset). Results from the categorization task indicate that LLMs struggle to classify VA values into discrete emotion categories, particularly for emotions beyond basic polarities (e.g., happiness, sadness). However, in the semantic description task, LLMs produced textual descriptions that align closely with human-generated interpretations, demonstrating a stronger capacity for free text affective inference of facial expressions.
Reference graph
Works this paper leans on
-
[41]
A. Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017
2017
-
[9]
C. D. Corley and R. Mihalcea. Measuring the semantic similarity of texts. In Proceedings of the ACL workshop on empirical modeling of semantic equivalence and entailment , pages 13–18, 2005
work page 2005
-
[1]
D. G. Altman and j. Martin Bland. Statistics notes: The normal distribution. BMJ, 310:298, 2 1995
work page 1995
-
[2]
M. Apidianaki. From word types to tokens and back: A survey of approaches to word meaning representation and interpretation. Compu- tational Linguistics, 49:465–523, 6 2023
work page 2023
-
[3]
L. F. Barrett. Discrete emotions or dimensions? the role of valence focus and arousal focus. Cognition & Emotion , 12(4):579–599, 1998
work page 1998
-
[4]
L. F. Barrett. The theory of constructed emotion: an active inference account of interoception and categorization. Social Cognitive and Affective Neuroscience, 12:1, 1 2016
work page 2016
-
[5]
L. F. Barrett, R. Adolphs, S. Marsella, A. M. Martinez, and S. D. Pollak. Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements. Psychological science in the public interest : a journal of the American Psychological Society , 20:1, 7 2019
work page 2019
- [6]
Show all 46 references
-
[7]
I. Burkitt. Complex emotions: Relations, feelings and images in emotional experience. The sociological review, 50(S2):151–167, 2002
2002
-
[8]
K. W. Church. Word2vec. Natural Language Engineering , 23(1):155– 162, 2017
2017
-
[10]
F. I. Dogan, U. Ozyurt, G. Cinar, and H. Gunes. Grace: Generating socially appropriate robot actions leveraging llms and human explana- tions. In 2025 IEEE International Conference on Robotics & Automation (ICRA). IEEE, 9 2025
2025
-
[11]
S. Du, Y . Tao, and A. M. Martinez. Compound facial expressions of emotion. Proceedings of the National Academy of Sciences of the United States of America , 111:E1454–E1462, 4 2014
2014
-
[12]
P. Ekman. Are there basic emotions? Psychological review, 99:550–553, 1992
1992
-
[13]
P. Ekman. Facial expressions of emotion: New findings, new questions. Psychological Science, 3:34–38, 1 1992. 5
1992
-
[14]
M. W. Fagerland. T-tests, non-parametric tests, and large studiesa paradox of statistical practice? BMC Medical Research Methodology , 12:1–7, 6 2012
2012
-
[15]
A. Faiz, S. Kaneda, R. Wang, R. Osi, P. Sharma, F. Chen, and L. Jiang. Llmcarbon: Modeling the end-to-end carbon footprint of large language models. arXiv preprint arXiv:2309.14393 , 2023
2023 arXiv
-
[16]
Fujimura, Y
T. Fujimura, Y . T. Matsuda, K. Katahira, M. Okada, and K. Okanoya. Categorical and dimensional perceptions in decoding emotional facial expressions. Cognition & Emotion , 26:587, 6 2011
2011
-
[17]
Gendron and L
M. Gendron and L. F. Barrett. Reconstructing the past: A century of ideas about emotion in psychology. Emotion Review, 1:316–339, 9 2009
2009
-
[18]
Y . He, Q. Ai, and K. Chen. A memd method of human emotion recognition based on valence-arousal model. In 2017 9th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC), volume 2, pages 399–402. IEEE, 2017
2017
-
[19]
R. E. Jack, W. Sun, I. Delis, O. G. Garrod, and P. G. Schyns. Four not six: Revealing culturally common facial expressions of emotion. Journal of Experimental Psychology: General , 145:708–730, 6 2016
2016
-
[20]
J. D. M.-W. C. Kenton and L. K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, page 2. Minneapolis, Minnesota, 2019
2019
-
[21]
Knief and W
U. Knief and W. Forstmeier. Violating the normality assumption may be the lesser of two evils. Behavior Research Methods , 53:2576–2590, 12 2021
2021
-
[22]
Kosti, J
R. Kosti, J. M. Alvarez, A. Recasens, and A. Lapedriza. Emotic: Emotions in context dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops , pages 61–69, 2017
2017
-
[23]
Laban and E
G. Laban and E. S. Cross. Sharing our emotions with robots: Why do we do it and how does it make us feel? IEEE Transactions on Affective Computing, pages 1–18, 2024
2024
-
[24]
Laban, A
G. Laban, A. Kappas, V . Morrison, and E. S. Cross. Opening up to social robots: How emotions drive self-disclosure behavior. In IEEE International Workshop on Robot and Human Communication, RO- MAN, pages 1697–1704. IEEE Computer Society, 2023
2023
-
[25]
Laban, T
G. Laban, T. Laban, and H. Gunes. Lexi: Large language models exper- imentation interface. Proceedings of the 12th International Conference on Human-Agent Interaction , pages 250–259, 11 2024
2024
-
[26]
Lecker and H
M. Lecker and H. Aviezer. More than words? semantic emotion labels boost context effects on faces. Affective Science, 2:163, 6 2021
2021
-
[27]
Y . Lee, S. Kim, R. A. Rossi, T. Yu, and X. Chen. Learning to reduce: Towards improving performance of large language models on structured data. arXiv preprint arXiv:2407.02750 , 2024
2024 arXiv
-
[28]
Y . Lei, D. Yang, Z. Chen, J. Chen, P. Zhai, and L. Zhang. Large vision- language models as emotion recognizers in context awareness. arXiv preprint arXiv:2407.11300, 2024
2024 arXiv
-
[29]
W. Li, Q. Xu, S. Liu, L. Yu, Y . Yang, L. Zhang, and X. He. Emotion concept in perception of facial expressions: Effects of emotion-label words and emotion-laden words. Neuropsychologia, 174:108345, 9 2022
2022
-
[30]
Z. Lin, X. Chen, D. Pathak, P. Zhang, and D. Ramanan. Revisiting the role of language priors in vision-language models. Proceedings of Machine Learning Research , 235:29914–29934, 6 2023
2023
-
[31]
C. Liu, Z. Xie, S. Zhao, J. Zhou, T. Xu, M. Li, and E. Chen. Speak from heart: An emotion-guided llm-based multimodal method for emotional dialogue generation. In Proceedings of the 2024 International Conference on Multimedia Retrieval , pages 533–542, 2024
2024
-
[32]
M. Liu, Y . Duan, R. A. Ince, C. Chen, O. G. Garrod, P. G. Schyns, and R. E. Jack. Facial expressions elicit multiplexed perceptions of emotion categories and dimensions. Current Biology, 32:200–209.e6, 1 2022
2022
-
[33]
T. Luo, A. Cao, G. Lee, J. Johnson, and H. Lee. Probing visual language priors in vlms. 12 2024
2024
-
[34]
W. Mou, O. Celiktutan, and H. Gunes. Group-level arousal and valence recognition in static images: Face, body and context. In 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) , volume 5, pages 1–6. IEEE, 2015
2015
-
[35]
OpenAI. Gpt-4o. https://openai.com, 2024. Accessed: 2024-08-12
2024
-
[36]
Gpt-4o-mini
OpenAI. Gpt-4o-mini. https://openai.com, 2024. Accessed: 2024-08-12
2024
-
[37]
black and white
A. B. Satpute, E. C. Nook, S. Narayanan, J. Shu, J. Weber, and K. N. Ochsner. Emotions in “black and white” or shades of gray? how we think about emotion shapes our perception and neural representation of emotion. Psychological Science, 27:1428, 11 2016
2016
-
[38]
Sherburn, B
D. Sherburn, B. Chughtai, and O. Evans. Can language models explain their own classification behavior? arXiv, 5 2024
2024
-
[39]
TEW ARI, S
S. TEW ARI, S. Mehta, and N. Srinivasan. Iimi emotional face database, May 2023
2023
-
[40]
Toisoul, J
A. Toisoul, J. Kossaifi, A. Bulat, G. Tzimiropoulos, and M. Pantic. Estimation of continuous valence and arousal levels from faces in naturalistic conditions. Nature Machine Intelligence , 3(1):42–50, 2021
2021
-
[42]
Xenos, N
A. Xenos, N. M. Foteinopoulou, I. Ntinou, I. Patras, and G. Tzimiropou- los. Vllms provide better context for emotion understanding through common sense reasoning. arXiv preprint arXiv:2404.07078 , 2024
2024 arXiv
-
[43]
H. Xu, R. Lou, J. Du, V . Mahzoon, E. Talebianaraki, Z. Zhou, E. Gar- rison, S. Vucetic, and W. Yin. Llms’ classification performance is overclaimed. 6 2024
2024
-
[44]
Q. Yang, M. Ye, and B. Du. Emollm: Multimodal emotional under- standing meets large language models. arXiv preprint arXiv:2406.16442, 2024
2024 arXiv
-
[45]
Y . Yao, X. Mei, J. Xu, Z. Sun, C. Zeng, and Y . Chen. Vlm-emo: Context- aware emotion classification with clip. In 2024 5th International Sem- inar on Artificial Intelligence, Networking and Information Technology (AINIT), pages 1615–1620. IEEE, 2024
2024
-
[46]
Zhang, X
Y . Zhang, X. Yang, X. Xu, Z. Gao, Y . Huang, S. Mu, S. Feng, D. Wang, Y . Zhang, K. Song, and G. Yu. Affective computing in the era of large language models: A survey from the nlp perspective. 7 2024. 6
2024
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.