REVIEW 4 major objections 8 minor 43 references
AI-Based Feedback in Counselling Competence Training of Prospective Teachers
T0 review · 4 major / 8 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The paper argues that AI-derived speech and gaze features can predict expert-rated quality in simulated teacher-parent conversations, and that students find the resulting feedback useful.
desk verdict The qualitative findings are worth a look, but the abstract's 'significant correlations' claim is not backed by any reported statistics. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is a multimodal feedback loop built around an analysis-and-visualization tool that fuses automatic transcription with frame-level audio and facial features. The transcription splits each recording into speaking and listening segments; on spoken frames the pipeline computes paraverbal features such as segment duration, words per segment, word length, speaking rate, statement and question shares, sentiment, pitch, and loudness, and across all frames it computes gaze direction clustered into a 'main direction' versus other, mutual gaze, smiling, mutual smiling, happiness, sadness, and anger. These features are aggregated over the session, displayed in several complementary forms, and compared with expert ratings. The load-bearing piece is the gaze feature: the system assumes that the cluster of gaze angles called 'main direction' represents looking at the interlocutor, and gaze and mutual gaze enter both the selected feature set and the feedback advice, but the clustering hyperparameters were chosen on a private medical dataset and applied without re-tuning.
What would settle it
Have experts label eye contact in a sample of the recorded teacher-parent conversations, then compare those labels with the gaze-in-main-direction feature; if agreement is low, or if the feature's correlation with expert ratings disappears after the gaze clustering is re-tuned on teacher-parent data, the gaze-based part of the feedback claim is falsified. As a second check, run the classifier with and without the gaze features and see whether accuracy and selected features change.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that an AI-based feedback loop can work for teacher-parent counselling training. The authors designed and ran an iterative block seminar four times with 29 student teachers; recordings of the simulated conversations were processed into session-level statistics for paraverbal and nonverbal behaviour, then shown to students in tables, parallel-coordinate plots, radar charts, and synchronized video playback together with expert ratings. The quantitative results show predictive capability of these features for expert rating above chance (56.0% accuracy versus a 50% dummy baseline), and the qualitative interviews show that students perceived the AI feedback positively, valuing the combination of video excerpts and numeric comparison. The verbal component is treated as groundwork: inter-rater agreement for conversation phases was 66.9% and for communication techniques 72.4%, below the 80% level the authors say is needed before automatic verbal feedback can be attempted.
Load-bearing premise
The load-bearing premise is that clustering a person's gaze directions into a 'main direction' correctly captures looking at the conversation partner in teacher-parent video conversations; the clustering was tuned on a private medical dataset and transferred without re-tuning, so if that transfer fails, the gaze-based findings and feedback advice lose their grounding.
Editorial extensions
If this is right
- If the correlations hold, teacher training programmes can define concrete, machine-measurable communication targets such as pace, questioning, gaze, and smiling instead of relying only on holistic expert impressions.
- Students' preference for video-plus-table feedback gives a concrete design rule: combine synchronized playback of the recorded conversation with session-level numeric comparisons.
- The verbal-skill annotations, once agreement is raised from 66.9% and 72.4% toward 80%, would allow automatic detection of conversation phases and techniques, extending feedback from paraverbal and nonverbal signals to what is said.
- The system is intended to support experts rather than replace them: at 56% accuracy the AI feedback is presented alongside instructor ratings and expert interpretation.
Reading between the lines
- A direct validation of the gaze feature in teacher-parent conversations is missing: if the gaze clustering were re-tuned or validated on this setting, the feature selection and accuracy might change, so the reported correlation should be treated as conditional on that transfer.
- The small sample of 29 students over four seminars makes the 56% accuracy a proof-of-concept; a natural extension is to fix the feature set and classifier before the next seminar and test whether the same features re-emerge.
- Students' wish to see the parent's mood over time suggests a testable design: displaying reciprocal affect dynamics rather than session averages may make feedback more actionable and could show whether teacher techniques shift parent affect during the conversation.
- If such feedback works for teacher-parent counselling, the same pipeline could be adapted to other professional communication trainings, such as medical consultations or social work interviews, where the underlying paraverbal and nonverbal features are similar.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes a four-day block seminar for prospective teachers in which simulated teacher-parent conversations are recorded, analyzed with an AI-based multimodal toolchain (NOVA/DISCOVER, WhisperX, OpenFace, LibreFace, openSMILE), and used to generate feedback. Three research questions are posed: (I1) whether nonverbal and paraverbal features correlate with expert-rated conversation quality, (I2) how students perceive AI-based feedback, and (I3) whether AI models can identify conversation phases and communication techniques. The paper reports classifier accuracies around 55–56% for predicting a dichotomized expert rating, qualitative interview findings showing generally positive student reception, and inter-rater agreement for verbal annotations. The central claim is that AI-based feedback provides objective, actionable insights for teacher training.
Significance. If the quantitative claims were adequately supported, the paper would be a useful contribution to an emerging area: using multimodal AI feedback in teacher education. The qualitative part is a strength: the interview study reports concrete student preferences, including the combination of video playback and tabular data, and it gives authentic quotations, which lends credibility to the user-acceptance findings. The paper also describes the full pipeline in enough detail for replication, and it is transparent about limitations such as the small sample size and the transfer of gaze-model hyperparameters from a medical dataset. However, the load-bearing quantitative claim of 'significant correlations' is not backed by the reported statistics, so the paper's central assertion of objective, data-grounded feedback remains unsubstantiated.
major comments (4)
- [§4.4 and Abstract] The abstract and the discussion state that significant correlations exist between nonverbal and paraverbal features and conversation quality, but §4.4 reports no correlation coefficients, p-values, confidence intervals, or significance tests for any feature-rating relationship. The evidence offered is a five-fold stratified cross-validation with XGB accuracy of 56.0% ± 16.0% and LR/SVC accuracy of 55.3% ± 6.9% on n = 29 sessions. With this sample size, 56.0% is not shown to exceed the 50% majority baseline; the reported standard deviations overlap the baseline and no binomial test or cross-validation confidence interval is provided. The phrase 'significant correlations' in the abstract is therefore unsupported by the reported statistics, and the feedback recommendations in §5.1 that are derived from feature differences lack empirical grounding.
- [§4.4 and §6] The FeatureWiz feature set appears to be selected on the full dataset before the five-fold split, because §4.4 lists the selected features (question, statement, sentiment, gaze, smile, happiness, sadness, anger) and then describes cross-validation using that fixed set. This can leak label information into the feature-selection step and inflate accuracy estimates. Feature selection should be nested inside the cross-validation loop or otherwise restricted to training folds. In addition, the dichotomization of the five-point expert rating is not specified: the discussion mentions a 50% majority-class baseline for rating '4 = helpful', but the paper does not state how the five-point scale was converted to a binary target.
- [§4.3] The gaze-clustering hyperparameters were chosen on a private medical doctor-patient dataset so that gaze in the 'main direction' correlated with patient satisfaction, and were then applied without re-tuning to teacher-parent conversations. Since gaze and mutual gaze appear in the FeatureWiz-selected feature set, the validity of these features is load-bearing for the quantitative claim. The paper provides no evidence that this transfer is valid for the teacher-parent setting, and no internal validation or calibration of the gaze feature is reported. This is particularly concerning given that the gaze feature is among the few that contribute to the reported classifier results.
- [§4.4 and §1 (I3)] The third research question, 'Training AI models to identify conversation phases and techniques,' is not actually answered. §4.4 reports only inter-rater agreement for annotations (66.9% for phases, 72.4% for techniques) and coincidence matrices; no trained model, no model accuracy, and no comparison to baselines is presented. The discussion in §6 correctly states that the annotation guidelines need to be improved first, but the paper nevertheless lists I3 as a guiding investigation and the conclusion does not clearly state that this part of the study was not completed. The reader is left without any evidence about the trainability of models for verbal skill identification.
minor comments (8)
- [§4.4] The feature list contains the typo 'happines' for 'happiness'.
- [§5] The sentence beginning 'The students’ first received the collective feedback' should read 'The students first received'.
- [§5.2] There is a typo in 'the former to forms of feedback', which should be 'the former two forms of feedback'.
- [§5.2] The quotation 'Apparently I am beeing perceived as more positively than i thought' contains two spelling/capitalization errors ('beeing' and 'i'), which should be corrected even in translated quotations.
- [§4.3] The phrase 'three preceeding frames' should be 'three preceding frames'.
- [§5.1] The section header 'NOV A Play Back' contains an unwanted space and should be 'NOVA Playback'.
- [§3] The camera resolution '2160p25' is later contrasted with '360p25' and '540p25'; these terms are inconsistently formatted and should be clarified (e.g., as 4K, 360p, 540p).
- [§5.1] The relative-deviation formula is defined twice in slightly different forms; the presentation would be clearer if the formula were given once as relative = (absolute - mean) / mean.
Circularity Check
No significant circularity: the study is empirical and data-driven, self-cited tools rest on external validation, and the abstract's unsupported correlation claim is a statistical-evidence issue rather than a circular derivation.
full rationale
The paper is an empirical study rather than a derivation, and its central results (I1-I3) are obtained from data: classifier cross-validation, inter-rater agreement, and qualitative interviews. I found no step in which an output is defined in terms of its own input or in which a fitted parameter is relabeled as a prediction. The self-citations of NOVA, DISCOVER, and the custom smile/gaze models are references to implemented, previously published tools, and the paper notes external validation for these components: NOVA was used and validated in psychotherapy research (Terhürne et al.), the smile detector was trained on the NoXi dataset, and the gaze-direction hyperparameters were tuned on a private medical dataset against the SPSQ satisfaction questionnaire. These external anchors mean the pipeline does not reduce to the present paper's own conclusions. The abstract's phrase 'significant correlations' is not supported by Section 4.4, which reports no coefficients or significance tests and only 56.0% +/- 16.0% accuracy against a 50% majority baseline; however, unsupported strength of evidence is a correctness or statistical concern, not circularity. Similarly, if FeatureWiz feature selection were performed on the full dataset before cross-validation, that would be a leakage risk, but the paper does not specify the order, and no quoted reduction to the target labels is exhibited. Accordingly, no circular step is identified and the score is 0.
Assumptions & free parameters
free parameters (3)
- Gaze clustering hyperparameters =
Not specified (chosen on private medical dataset)
- FeatureWiz selected feature set =
question, statement, sentiment, gaze, smile, happiness, sadness, anger
- Classifier hyperparameters =
Not reported
assumptions (4)
- domain assumption Expert Likert ratings are a valid ground-truth measure of conversation quality
- domain assumption AI-extracted features accurately measure the intended communication constructs
- domain assumption The communication-theory categories (Benien phases, Gerich techniques) are exhaustive and disjunct for teacher-parent conversations
- standard math Standard cross-validation assumptions hold despite n=29 and feature selection on the same data
Cite this review
Pith. "Pith review of AI-Based Feedback in Counselling Competence Training of Prospective Teachers." pith.science (2026). https://pith.science/paper/45BKBECK
@misc{pith2026250503423,
author = {Pith},
title = {Pith review of: AI-Based Feedback in Counselling Competence Training of Prospective Teachers},
year = {2026},
howpublished = {\url{https://pith.science/paper/45BKBECK}},
note = {Machine review of arXiv:2505.03423}
}
read the original abstract
This study explores the use of AI-based feedback to enhance the counselling competence of prospective teachers. An iterative block seminar was designed, incorporating theoretical foundations, practical applications, and AI tools for analysing verbal, paraverbal, and nonverbal communication. The seminar included recorded simulated teacher-parent conversations, followed by AI-based feedback and qualitative interviews with students. The study investigated correlations between communication characteristics and conversation quality, student perceptions of AI-based feedback, and the training of AI models to identify conversation phases and techniques. Results indicated significant correlations between nonverbal and paraverbal features and conversation quality, and students positively perceived the AI feedback. The findings suggest that AI-based feedback can provide objective, actionable insights to improve teacher training programs. Future work will focus on refining verbal skill annotations, expanding the dataset, and exploring additional features to enhance the feedback system.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Gernot Aich and Michael Behr. 2019. Gesprächsführung mit Eltern. Beltz
work page 2019
-
[2]
Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. 2023. WhisperX: Time-Accurate Speech Transcription of Long-Form Audio. INTERSPEECH 2023 (2023)
2023
-
[3]
Tadas Baltrušaitis, Peter Robinson, and Louis-Philippe Morency. 2016. Openface: an open source facial behavior analysis toolkit. In 2016 IEEE winter conference on applications of computer vision (W ACV). IEEE, 1–10
work page 2016
-
[4]
Francesco Barbieri, Luis Espinosa Anke, and Jose Camacho-Collados. 2022. XLM- T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond. In Proceedings of the Thirteenth Language Resources and Evaluation Conference . European Language Resources Association, Marseille, France, 258–266. https: //aclanthology.org/2022.lrec-1.27
work page 2022
-
[5]
Tobias Baur, Sina Clausen, Alexander Heimerl, Florian Lingenfelser, Wolfgang Lutz, and Elisabeth André. 2020. NOVA: A Tool for Explanatory Multimodal Behavior Analysis and Its Application to Psychotherapy. InMultiMedia Modeling - 26th International Conference, MMM 2020, Daejeon, South Korea, January 5-8, 2020, Proceedings, Part II (Lecture Notes in Comput...
work page 2020
-
[6]
Valstar, Björn Schuller, and Elisabeth André
Tobias Baur, Alexander Heimerl, Florian Lingenfelser, Johannes Wagner, Michel F. Valstar, Björn Schuller, and Elisabeth André. 2020. eXplainable Cooperative Machine Learning with NOVA. KI - Künstliche Intelligenz (19 Jan 2020). https: //doi.org/10.1007/s13218-020-00632-3
-
[7]
Rolf H Bay. 2021. Erfolgreiche Gespräche durch aktives Zuhören . expert verlag
work page 2021
-
[8]
Valentin Bazarevsky, Yury Kartynnik, Andrey Vakunov, Karthik Raveendran, and Matthias Grundmann. 2019. Blazeface: Sub-millisecond neural face detection on mobile gpus. arXiv preprint arXiv:1907.05047 (2019)
arXiv 2019
Show all 43 references
-
[9]
Karl Benien. 2003. Schwierige Gespräche führen. Modelle für Beratungs-, Kritik- und Konfliktgespräche im Berufsalltag. Rowohlt, Reinbeck bei Hamburg (2003)
2003
-
[10]
Angelo Cafaro, Johannes Wagner, Tobias Baur, Soumia Dermouche, Mercedes Torres Torres, Catherine Pelachaud, Elisabeth André, and Michel Valstar. 2017. The NoXi database: multimodal recordings of mediated novice-expert interac- tions. In Proceedings of the 19th ACM Internationa...
2017
-
[11]
Di Chang, Yufeng Yin, Zongjian Li, Minh Tran, and Mohammad Soleymani
-
[12]
Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’16) . ACM, 785–794. https://doi. org/10.1145/2939672.2939785
2016
-
[13]
Alexander W Chessman, Amy V Blue, Gregory E Gilbert, Maura Carey, and Arch G Mainous. 2003. Assessing students’ communication and interpersonal skills across evaluation settings. FAMILY MEDICINE-KANSAS CITY- 35, 9 (2003), 643–648
2003
-
[14]
Anjali Choudhary and Vineeta Gupta. 2015. Teaching communications skills to medical students: Introducing the fine art of medical practice. International Journal of Applied and Basic Medical Research 5, Suppl 1 (2015), S41–S44
2015
-
[15]
David A Cook. 2014. How much evidence does it take? A cumulative meta- analysis of outcomes of simulation-based education. Medical education 48, 8 (2014), 750–760
2014
-
[16]
Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine learning 20, 3 (1995), 273–297
1995
-
[17]
Diane D Cox. 2005. Evidence-based interventions using home-school collabora- tion. School Psychology Quarterly 20, 4 (2005), 473
2005
-
[18]
David R Cox. 1958. The regression analysis of binary sequences. Journal of the Royal Statistical Society: Series B (Methodological) 20, 2 (1958), 215–232
1958
-
[19]
Thorsten Dresing and Thorsten Pehl. 2015. Praxisbuch interview, transkription & analyse: Anleitungen und Regelsysteme für qualitativ Forschende . dr dresing & pehl GmbH
2015
-
[20]
Paul Ekman and Wallace V Friesen. 1976. Measuring facial movement. Environ- mental psychology and nonverbal behavior 1 (1976), 56–75
1976
-
[21]
Joyce L Epstein and Frances L Van Voorhis. 2001. More than minutes: Teachers’ roles in designing homework. Educational psychologist 36, 3 (2001), 181–193
2001
-
[22]
Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010. Opensmile: the munich versatile and fast open-source audio feature extractor. In Proceedings of the 18th ACM international conference on Multimedia . 1459–1462
2010
-
[23]
Martin Gartmeier. 2018. Gespräche zwischen Lehrpersonen und Eltern . Springer
2018
-
[24]
Mara Gerich. 2016. Teachers ‘Counseling Competence in Parent-Teacher Talks: Modeling, Intervention, Behavior-Based Assessment . Springer
2016
-
[25]
Mara Gerich, Simone Bruder, Silke Hertel, Monika Trittel, and Bernhard Schmitz
-
[26]
Pam Grossman. 2021. Teaching core practices in teacher education . Harvard Education Press
2021
-
[27]
Cornelia Helfferich. 2022. Leitfaden-und experteninterviews. In Handbuch Methoden der empirischen Sozialforschung . Springer, 875–892
2022
-
[28]
Yury Kartynnik, Artsiom Ablavatski, Ivan Grishchenko, and Matthias Grund- mann. 2019. Real-time facial surface geometry from monocular video on mobile GPUs. arXiv preprint arXiv:1907.06724 (2019)
2019 arXiv
-
[29]
Jan-Christoph Klie, Michael Bugert, Beto Boullosa, Richard Eckart de Castilho, and Iryna Gurevych. 2018. The INCEpTION Platform: Machine-Assisted and Knowledge-Oriented Interactive Annotation. In Proceedings of the 27th Interna- tional Conference on Computational Linguistics: ...
2018
-
[30]
Mark L Knapp, Judith A Hall, and Terrence G Horgan. 1972. Nonverbal commu- nication in human interaction . Thomson Wadsworth
1972
-
[31]
Udo Kuckartz. 2012. Qualitative inhaltsanalyse: methoden, praxis, computerunter- stützung. Beltz Juventa
2012
-
[32]
Charles Landert, Martina Brägger, Dachverband Schweizer Lehrerinnen, and LCH Lehrer. 2009. LCH Arbeitszeiterhebung 2009 (AZE’09): Bericht zur Erhebung bei 5’000 Lehrpersonen im Zeitraum Oktober 2008-September 2009: im Auftrag des Dachverbandes Schweizer Lehrer LCH; Charles Lan...
2009
-
[33]
Claire Lane and Stephen Rollnick. 2007. The use of simulated patients and role- play in communication skills training: a review of the literature to August 2005. Patient education and counseling 67, 1-2 (2007), 13–20
2007
-
[34]
Kazuma Matoba and Daniel Scheible. 2007. Interkulturelle und transkulturelle Kommunikation. Workingpaper International Society For Diversity Management (2007)
2007
-
[35]
Andrea Rocci and Louis de Saussure. 2016. Verbal communication. Vol. 3. Walter de Gruyter GmbH & Co KG
2016
-
[36]
Werner Sacher. 2005. Erfolgreiche und misslingende Elternarbeit: Ursachen und Handlungsmöglichkeiten; erarbeitet auf der Grundlage der Repräsentativbefragung an bayerischen Schulen im Sommer 2004 . Lehrstuhl für Schulpädagogik, Friedrich- Alexander-Univ
2005
-
[37]
Daniela Sauer. 2015. Wie beraten Lehrkräfte Eltern?: Eine qualitativ-rekonstruktive Studie zur Beratungsaufgabe von Lehrkräften . Verlag Barbara Budrich
2015
-
[38]
Dominik Schiller, Tobias Hallmen, Daksitha Withanage Don, Elisabeth André, and Tobias Baur. 2024. DISCOVER: A Data-driven Interactive System for Com- prehensive Observation, Visualization, and ExploRation of Human Behaviour. arXiv preprint arXiv:2407.13408 (2024)
2024 arXiv
-
[39]
Ram Seshadri. 2020. GitHub - AutoViML/featurewiz: Use advanced feature engineering strategies and select the best features from your data set fast with a single line of code. https://github.com/AutoViML/featurewiz. Source code repository
2020
-
[40]
Eber- hardt, Elisabeth André, and Wolfgang Lutz
Patrick Terhürne, Brian Schwartz, Tobias Baur, Dominik Schiller, Steffen T. Eber- hardt, Elisabeth André, and Wolfgang Lutz. 2022. Validation and application of the Non-Verbal Behavior Analyzer: an automated tool to assess non-verbal emo- tional expressions in psychotherapy. F...
2022
-
[41]
Lars Wegner. 2016. Lehrkraft-Eltern-Interaktionen am Elternsprechtag: eine gesprächs-und gattungsanalytische Untersuchung. de Gruyter. Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009
2016
-
[2015]
What Skills and Abilities Are Essential for Counseling on Learning Dif- ficulties and Learning Strategies? Zeitschrift für Entwicklungspsychologie und Pädagogische Psychologie (2015)
2015
-
[2024]
In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (W ACV)
LibreFace: An Open-Source Toolkit for Deep Facial Expression Analysis. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (W ACV). To appear
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.