REVIEW 3 major objections 4 minor 38 references
Using Vision Language Models to Detect Students' Academic Emotion through Facial Expressions
T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Zero-shot vision-language models classify student faces with moderate accuracy; Qwen beats Llama, both miss 'distracted'.
desk verdict A genuinely zero-shot, zero-training baseline for VLM-based academic emotion classification, with honest limitations; the numbers support the descriptive claims, but the sampling and inference layer is thin enough that the Qwen-vs-Llama gap should be treated as provisional. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is zero-shot prompting of a vision-language model: a single prompt that combines an instruction to classify the student's expression, explicit definitions of the five target categories, and the input image, with the model forced to answer with one category word only. The task container is the OLSFED dataset, a corpus of 31,115 images of 82 Asian students watching instructional videos, from which the paper samples 1,000 images per expression. No weights are updated; the entire method rests on the model transferring its pretrained visual-linguistic knowledge to an unseen five-way classification.
What would settle it
Re-running the same prompt on the full OLSFED set with per-student stratified splits and a fixed seed, and separately having independent raters re-label a random subset of the 5,000 images, would settle the claim: if Qwen no longer beats Llama, or happy no longer outperforms the other classes, the reported F1 ranking and per-class conclusions would be falsified.
Extended reading notes
Core claim
The central finding is that a zero-shot prompt can push general-purpose vision-language models to recognize five academic facial expressions, but only at moderate accuracy. Qwen2.5-VL-7B-Instruct reaches an overall F1 of 0.5197 and accuracy of 0.5546, ahead of Llama-3.2-11B-Vision-Instruct's F1 of 0.4519 and accuracy of 0.5118. The per-class pattern is consistent: happy stands out (F1 0.92 for Llama and 0.83 for Qwen), neutral and tired sit in a middling band, and distracted collapses to F1 scores of 0.04 and 0.01 because the models almost never assign that label. Qwen2.5-VL-7B-Instruct also picks out confused expressions with an F1 of 0.65, which the authors connect to the practical goal of identifying lecture content that confuses students.
Load-bearing premise
The results assume the dataset's labels are correct and that the randomly chosen 1,000 images per expression stand in for the full set; the paper gives no evidence that labels are reliable, no detail on how the 82 students are distributed across the sample, and no sampling seed.
Editorial extensions
If this is right
- A zero-training emotion signal is available for online-learning video: the same prompt can be run on new images without a local labeling campaign, though at roughly 0.5 F1 rather than supervised accuracy.
- Happy expressions can be trusted as a positive-emotion indicator: both models score above 0.8 F1 on this class, so a simple happy-detector is immediately usable.
- Distracted behavior should not be measured with these models as prompted: F1 below 0.05 means the zero-shot approach is effectively blind to this class.
- Qwen2.5-VL-7B-Instruct's confusion F1 of 0.65 gives a plausible low-cost way to flag video moments that puzzle students, if the result repeats on the full dataset.
- Because the method needs no retraining, it can move across classrooms and camera setups faster than supervised classifiers, but the paper's own limitations paragraph warns that a broader dataset and more models are needed before generalizing.
Reading between the lines
- Beyond the paper: the near-zero recall for distracted suggests the label prior, not visual evidence, is the bottleneck; adding concrete negative examples to the prompt is a cheap next test.
- Beyond the paper: a per-student split of the 1,000-image sample could change the class-level F1s, since no stratification or seed is reported.
- Beyond the paper: allowing the models to abstain or to explain their choice would separate prompt-design effects from model capability on confusion and tiredness.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports a preliminary zero-shot evaluation of two open-source vision-language models, Llama-3.2-11B-Vision-Instruct and Qwen2.5-VL-7B-Instruct, on the task of classifying students' academic emotions from facial expressions. Using 5,000 randomly selected images from the OLSFED dataset (1,000 per class across confused, distracted, happy, neutral, and tired), the authors prompt each model with a fixed instruction plus class definitions and report accuracy, precision, recall, and F1. They find that both models achieve moderate overall performance, that Qwen2.5-VL outperforms Llama-3.2, that both models perform best on the happy class, and that both fail to detect the distracted class (near-zero recall), while Qwen2.5-VL performs relatively well on confused expressions. The paper concludes that VLMs offer a promising zero-training baseline for academic emotion recognition, with limitations acknowledged for future work.
Significance. If the reported results are robust, the study provides a useful zero-shot baseline for five-class academic emotion classification on OLSFED and demonstrates that off-the-shelf VLMs can be applied in educational settings without fine-tuning. The evaluation is externally grounded: labels come from the OLSFED dataset, model outputs come from fixed pre-trained checkpoints with a single prompt, and no parameters are fitted to the evaluation data, so circularity is not a concern. The authors are transparent about the exploratory nature of the work. The main value lies in establishing a reference point for future zero-shot or few-shot work, though the lack of statistical rigor currently limits the strength of the comparative and per-class conclusions.
major comments (3)
- [§3.1 and §4, Table 1 and Figures 3–5] The central comparative claims—Qwen outperforming Llama and the per-class patterns—rest on point estimates from a single random subsample of 5,000 images with no stated random seed, no stratification by the 82 students, and no confidence intervals or significance tests. Because OLSFED images are video frames from 82 students, the images are not independent; a random subsample may overrepresent particular students or sessions, and the effective sample size for model comparison could be far smaller than 5,000. The observed accuracy difference (0.5546 vs. 0.5118) and F1 difference (0.5197 vs. 0.4519) may be within sampling variability. Please provide the sampling seed, report results with bootstrap confidence intervals or student-level aggregation, and include a confusion matrix with raw counts so that the stability of the ranking and the per-class gaps can be assessed.
- [§3.2 and §4, 'distracted' category] The claim that both models 'fail to detect distracted behavior' is based on recall values of 0.02 and 0.01 despite precision of 1.00, meaning the models made very few positive predictions for this class (on the order of 10–20 out of 1,000). The 'distracted' category is defined behaviorally (looking away, turning away) rather than as a facial expression, so the near-zero recall may reflect an ambiguity or mismatch between the prompt's definition and the OLSFED annotation protocol, rather than a pure VLM capability failure. To support the conclusion, please report the full confusion matrix, the number of predicted 'distracted' instances, and, if feasible, perform a small re-annotation or label-reliability check on a subset of the 'distracted' images to establish that the ground-truth labels are consistent.
- [§1, §5, and Table 1] The characterization of the models' performance as 'moderate' is uncalibrated without a baseline. A random classifier on the five-way task would achieve 20% accuracy, and a majority-class classifier would also be near that range if classes are balanced; the reported 51.18% and 55.46% accuracies need context to be interpretable. The paper would be substantially strengthened by comparing the zero-shot VLMs with a simple supervised baseline (e.g., a CNN trained on OLSFED) or at least a majority-class baseline, so that the practical value of the zero-shot approach can be judged. Without such a comparison, the conclusion that VLMs 'offer a promising solution' to generalization issues is not fully supported.
minor comments (4)
- [Figure captions (Figures 3, 4, 5)] The captions state 'on four facial expressions' but the figures and the paper use five categories (confused, distracted, happy, neutral, tired); please correct the captions to say 'five facial expressions.'
- [§3.2] The paper does not specify the exact inference settings used with the two models, such as temperature, maximum number of new tokens, or the model checkpoint dates/commits. Please add these details to ensure the results are reproducible.
- [§3.1] The phrase 'randomly selected 1,000 images' should be accompanied by the random seed and a description of the sampling procedure (e.g., whether selection was done per student or from the full collection) to enable reproducibility and to allow assessment of potential student-level bias.
- [§4] The text states that both models achieve high precision scores of '1.00 and 1.00' for the distracted class; while the figures corroborate this, reporting actual numbers (e.g., 10/10) rather than only the rounded values would be more informative.
Circularity Check
No significant circularity: the evaluation uses off-the-shelf zero-shot VLMs against an external dataset, with no fitted parameters and no load-bearing self-citation.
full rationale
The paper's central claims are that two open-source VLMs achieve moderate zero-shot performance on the OLSFED academic facial expression dataset, that Qwen2.5-VL outperforms Llama-3.2, and that both models recognize 'happy' well but fail on 'distracted.' The derivation chain is: take ground-truth labels from OLSFED (an externally published dataset by Bian et al. [3]), randomly subsample 1,000 images per class, run two pre-trained checkpoints with a fixed hand-written prompt, and compute standard metrics. No parameter is fitted to the evaluation data, no model is fine-tuned on the target labels, and no claim is derived from a quantity that depends on the target result. The prompt definitions are simply the five category names with commonsense wording; they are not estimated from the labels and no prompt variant is selected based on the reported metrics. The dataset citation [3] is not authored by the present authors, and the model citations [19, 26] are external model cards. There is no self-citation chain and no uniqueness theorem imported from prior work. The only author-supplied input is the prompt wording, but nothing in the paper suggests that the prompt was tuned on the test subset; even if it were, that would be a methodology weakness rather than circularity because the labels remain external and the models are fixed. Concerns about random sampling without a seed, lack of student-level stratification, and label reliability are validity threats, not circularity: they question whether the reported numbers generalize, but they do not make the metrics equivalent to the inputs. Under the stated criteria, the analysis is self-contained against an external benchmark, so the appropriate circularity score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption The OLSFED dataset's ground-truth labels for confused, distracted, happy, tired, and neutral are correct and represent students' academic emotions.
- domain assumption Facial expressions alone are a valid indicator of the five academic emotion categories in an online learning context.
- domain assumption The random subset of 1,000 images per class is representative of the full dataset and independent of any student-specific bias.
Cite this review
Pith. "Pith review of Using Vision Language Models to Detect Students' Academic Emotion through Facial Expressions." pith.science (2026). https://pith.science/paper/IQI76YCZ
@misc{pith2026250610334,
author = {Pith},
title = {Pith review of: Using Vision Language Models to Detect Students' Academic Emotion through Facial Expressions},
year = {2026},
howpublished = {\url{https://pith.science/paper/IQI76YCZ}},
note = {Machine review of arXiv:2506.10334}
}
read the original abstract
Students' academic emotions significantly influence their social behavior and learning performance. Traditional approaches to automatically and accurately analyze these emotions have predominantly relied on supervised machine learning algorithms. However, these models often struggle to generalize across different contexts, necessitating repeated cycles of data collection, annotation, and training. The emergence of Vision-Language Models (VLMs) offers a promising alternative, enabling generalization across visual recognition tasks through zero-shot prompting without requiring fine-tuning. This study investigates the potential of VLMs to analyze students' academic emotions via facial expressions in an online learning environment. We employed two VLMs, Llama-3.2-11B-Vision-Instruct and Qwen2.5-VL-7B-Instruct, to analyze 5,000 images depicting confused, distracted, happy, neutral, and tired expressions using zero-shot prompting. Preliminary results indicate that both models demonstrate moderate performance in academic facial expression recognition, with Qwen2.5-VL-7B-Instruct outperforming Llama-3.2-11B-Vision-Instruct. Notably, both models excel in identifying students' happy emotions but fail to detect distracted behavior. Additionally, Qwen2.5-VL-7B-Instruct exhibits relatively high performance in recognizing students' confused expressions, highlighting its potential for practical applications in identifying content that causes student confusion.
Figures
Reference graph
Works this paper leans on
-
[1]
Using Vision Language Models to Detect Students' Academic Emotion through Facial Expressions
INTRODUCTION Academic emotions refer to the emotions and feelings ex- perienced by students in educational settings, such as en- joyment of learning, pride in success, and test anxiety [23]. These emotions have been found to significantly impact stu- dents’ learning outcomes [2, 9, 25]. For instance, a meta- analysis indicates a significantly positive cor...
work page Pith review arXiv 2025
-
[2]
The Role of Emotions in Stu- dents’ Learning and Achievement,
RELA TED WORK In 1998, the American Educational Research Association held an annual meeting on “The Role of Emotions in Stu- dents’ Learning and Achievement,” which marked the begin- ning of significant attention being paid to students’ emotions by researchers. Numerous studies have since found that stu- dents experience a variety of emotions in academic ...
work page 1998
-
[3]
METHOD 3.1 Dataset In this study, we analyzed students’ academic emotions through their facial expressions. Specifically, we utilized the Online Learning Spontaneous Facial Expression Database (OLS- FED) [3] as our academic emotion dataset. This dataset captures the facial expressions of 82 students while they watched instructional videos in an online lea...
work page 2024
-
[4]
RESULTS Table 1 presents the overall performance of Llama-3.2-11B- Vision-Instruct and Qwen2.5-VL-7B-Instruct in classifying five academic facial expressions. Specifically, Llama-3.2- 11B-Vision-Instruct achieves 0.4519 in F1 score, 0.5118 in accuracy, 0.6473 in precision, and 0.5118 in recall, respec- tively. In contrast, Qwen2.5-VL-7B-Instruct achieves ...
-
[5]
DISCUSSION AND CONCLUSION To automatically and accurately analyze students’ academic emotions, researchers have utilized supervised machine learn- ing algorithms to train various models. However, these mod- els face challenges in generalizing well across different con- Table 1: The overall performance of Llama-3.2-11B-Vision-Instruct and Qwen2.5-VL-7B-Ins...
-
[6]
L. Cen, F. Wu, Z. L. Yu, and F. Hu. A real-time speech emotion recognition system and its application in online learning. InEmotions, technology, design, and learning, pages 27–46. Elsevier, 2016
work page 2016
-
[7]
S. M. S. A. Abdullah, S. Y. A. Ameen, M. A. Sadeeq, and S. Zeebaree. Multimodal emotion recognition using deep learning.Journal of Applied Science and Technology Trends, 2(01):73–79, 2021
work page 2021
-
[8]
N. Alshareef, I. Fletcher, and S. Giga. The role of emotions in academic performance of undergraduate medical students: a narrative review.BMC Medical Education, 24(1):907, 2024
work page 2024
Show all 38 references
-
[9]
C. Bian, Y. Zhang, F. Yang, W. Bi, and W. Lu. Spontaneous facial expression database for academic emotion inference in online learning.IET Computer Vision, 13(3):329–337, 2019
2019
-
[10]
Bordes, R
F. Bordes, R. Y. Pang, A. Ajay, A. C. Li, A. Bardes, S. Petryk, O. Ma˜ nas, Z. Lin, A. Mahmoud, B. Jayaraman, et al. An introduction to vision-language modeling.arXiv preprint arXiv:2405.17247, 2024
2024 arXiv
-
[11]
Castellano, L
G. Castellano, L. Kessous, and G. Caridakis. Emotion recognition through multiple modalities: face, body gesture, speech.Affect and Emotion in Human-Computer Interaction: From Theory to Applications, pages 92–103, 2008
2008
-
[12]
L. Guo, W. Wang, T. Wang, W. Li, M. Gong, S. Zhang, W.-H. Zhang, and C. Lu. Association of emotional and behavioral problems with single and multiple suicide attempts among chinese adolescents: modulated by academic performance.Journal of affective disorders, 258:25–32, 2019
2019
-
[13]
Im- ages of students’ performance during learning can be used to recognize their academic emotions through facial expres- sions, gestures, and postures [5, 18]
and computer-supported collaborative learning [31]. Im- ages of students’ performance during learning can be used to recognize their academic emotions through facial expres- sions, gestures, and postures [5, 18]. Furthermore, videos of classroom teaching can be employed to ana...
-
[14]
Chiang and C.-J
W.-W. Chiang and C.-J. Liu. Scale of academic emotion in science education: Development and validation.International Journal of Science Education, 36(6):908–928, 2014
2014
-
[15]
I. Dubovi. Cognitive and emotional engagement while learning with vr: The perspective of multimodal methodology.Computers & Education, 183:104495, 2022
2022
-
[16]
S. Ekornes. The impact of perceived psychosocial environment and academic emotions on higher education students’ intentions to drop out.Higher Education Research & Development, 41(4):1044–1059, 2022
2022
-
[17]
X. Feng, Y. Wei, X. Pan, L. Qiu, and Y. Ma. Academic emotion classification and recognition method for large-scale online learning environment—based on a-cnn and lstm-att deep learning pipeline method.International journal of environmental research and public health, 17(6):1941, 2020
1941
-
[18]
Govaerts and J
S. Govaerts and J. Gr´ egoire. Development and construct validation of an academic emotions scale. International Journal of Testing, 8(1):34–54, 2008
2008
-
[19]
Huang, Z
C. Huang, Z. Han, M. Li, X. Wang, and W. Zhao. Sentiment evolution with interaction levels in blended learning environments: Using learning analytics and epistemic network analysis.Australasian Journal of Educational Technology, 37(2):81–95, 2021
2021
-
[20]
Huber and M
K. Huber and M. Bannert. What happens to your body during learning with computer-based environments? exploring negative academic emotions using psychophysiological measurements.Journal of Computers in Education, 10(1):189–215, 2023
2023
-
[21]
E. E. Ketonen, V. Salonen, K. Lonka, and K. Salmela-Aro. Can you feel the excitement? physiological correlates of students’ self-reported emotions.British Journal of Educational Psychology, 93:113–129, 2023
2023
-
[22]
Kubsch, D
M. Kubsch, D. Caballero, and P. Uribe. Once more with feeling: emotions in multimodal learning analytics. InThe multimodal learning analytics handbook, pages 261–285. Springer, 2022
2022
-
[23]
Lei and Y
H. Lei and Y. Cui. Effects of academic emotions on achievement among mainland chinese students: A meta-analysis.Social Behavior and Personality: an international journal, 44(9):1541–1553, 2016
2016
-
[24]
J. X.-Y. Lek and J. Teo. Academic emotion classification using fer: A systematic review.Human Behavior and Emerging Technologies, 2023(1):9790005, 2023
2023
-
[25]
Llama-3.2-11b-vision-instruct, September 2024
Meta. Llama-3.2-11b-vision-instruct, September 2024
2024
-
[26]
R. Pekrun. Academic emotions. InHandbook of motivation at school, pages 120–144. Routledge, 2016
2016
-
[27]
Pekrun and M
R. Pekrun and M. B ¨uhner. Self-report measures of academic emotions. InInternational handbook of emotions in education, pages 561–579. Routledge, 2014
2014
-
[28]
Pekrun, T
R. Pekrun, T. Goetz, A. C. Frenzel, P. Barchfeld, and R. P. Perry. Measuring emotions in students’ learning and performance: The achievement emotions questionnaire (aeq).Contemporary educational psychology, 36(1):36–48, 2011
2011
-
[29]
Pekrun, T
R. Pekrun, T. Goetz, W. Titz, and R. P. Perry. Academic emotions in students’ self-regulated learning and achievement: A program of qualitative and quantitative research.Educational psychologist, 37(2):91–105, 2002
2002
-
[30]
Rajan, P
S. Rajan, P. Chenniappan, S. Devaraj, and N. Madian. Facial expression recognition techniques: a comprehensive survey.IET Image Processing, 13(7):1031–1040, 2019
2019
-
[31]
J. Tan, J. Mao, Y. Jiang, and M. Gao. The influence of academic emotions on learning effects: A systematic review.International journal of environmental research and public health, 18(18):9678, 2021
2021
-
[32]
Q. Team. Qwen2.5-vl, January 2025
2025
-
[33]
Wei and J
X. Wei and J. Yin. The application of physiological feedback to the evaluation of academic emotion: A literature review. In2021 Tenth International Conference of Educational Innovation through Technology (EITT), pages 346–349. IEEE, 2021
2021
-
[34]
Xiang, Z
A. Xiang, Z. Qi, H. Wang, Q. Yang, and D. Ma. A multimodal fusion network for student emotion recognition based on transformer and tensor product. In2024 IEEE 2nd International Conference on Sensors, Electronics and Computer Engineering (ICSECE), pages 1–4. IEEE, 2024
2024
-
[35]
H. Zeng, X. Shu, Y. Wang, Y. Wang, L. Zhang, T.-C. Pong, and H. Qu. Emotioncues: Emotion-oriented visual summarization of classroom videos.IEEE transactions on visualization and computer graphics, 27(7):3168–3181, 2020
2020
-
[36]
Zhang, J
J. Zhang, J. Huang, S. Jin, and S. Lu. Vision-language models for vision tasks: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024
2024
-
[37]
Zheng, M
L. Zheng, M. Long, J. Niu, and L. Zhong. An automated group learning engagement analysis and feedback approach to promoting collaborative knowledge building, group performance, and socially shared regulation in cscl.International Journal of Computer-Supported Collaborative Lea...
2023
-
[38]
X. Zhu, C. Guo, H. Feng, Y. Huang, Y. Feng, X. Wang, and R. Wang. A review of key technologies for emotion analysis using multimodal information. Cognitive Computation, 16(4):1504–1530, 2024
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.