REVIEW 4 major objections 5 minor 40 references
Exploring LLMs for Predicting Tutor Strategy and Student Outcomes in Dialogues
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper finds that even state-of-the-art LLMs struggle to predict future tutor moves, while tutor strategy is highly indicative of student outcomes.
desk verdict First solid LLM benchmark for future tutor-move prediction, but the paper's second headline claim about tutor moves predicting outcomes is undercut by an internal regression inconsistency. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the tutor move, a categorical label capturing the pedagogical action of a tutor turn: MathDial uses four moves (probing, focus, telling, generic), AlgebraNation uses sixteen multi-label moves. The paper frames four prediction functions over dialogue history: future tutor move prediction, current move classification, dialogue success prediction, and next-turn student success prediction. The machinery includes a LoRA-fine-tuned Llama 3 model and a zero-shot prompted GPT-4o, together with three label-sequence baselines (a second-order Markov chain, logistic regression, and an LSTM) that isolate the contribution of textual context. Comparing text-only inputs with inputs that include previous move labels is what lets the paper separate the question of whether moves are predictable from the question of whether moves carry outcome signal.
What would settle it
A re-annotation study measuring inter-annotator agreement on the confusable move pairs—probing versus focus in MathDial and giving instruction versus giving explanation in AlgebraNation—would settle whether the low future-move F1 is a model limitation or a label-noise artifact, since poor agreement would mean the prediction ceiling is already near the reliability of the ground truth.
Extended reading notes
Core claim
The central discovery is an asymmetry: tutor moves are informative about student outcomes, but they are not predictable by current LLMs. On MathDial, Llama 3 with dialogue and previous move labels reaches 49.33 F1 for future tutor moves; on AlgebraNation, GPT-4o reaches 27.32 F1. The errors are systematic—the models most often confuse probing with focus in MathDial and giving instruction with giving explanation in AlgebraNation, the same pairs that annotators found hard to separate. Meanwhile, dialogue success prediction reaches a much higher level, and on the real-student AlgebraNation data the move labels improve Llama 3's F1 by more than five points. The authors take this as evidence that tutor strategy is highly indicative of student outcomes, but that forecasting the next pedagogical move remains beyond the reach of current generative LLMs.
Load-bearing premise
The conclusion depends on the tutor-move labels being a trustworthy operationalization of tutoring strategy, but the paper reports that MathDial annotators struggled to distinguish probing from focus and that the models' top misclassifications fall on exactly those confusable pairs.
Editorial extensions
If this is right
- Future tutor moves are not reliably forecastable from dialogue history alone with current LLMs, so automated tutors cannot yet anticipate their next best pedagogical action from text.
- Tutor move annotations are a usable intermediate signal: including them raises Llama 3's AlgebraNation dialogue-success F1 from 75.52 to 80.84.
- The error pattern is concentrated on semantically adjacent move pairs, so improving or merging ambiguous label definitions is a more promising lever than simply scaling models.
- Dialogue-level success prediction is substantially easier than turn-level success prediction, and on real student data the move-label gain is clearer than on the synthetic MathDial dialogues.
Reading between the lines
- The paper's 'tutor behavior is unpredictable' conclusion may overstate the model limitation: because annotators themselves confused probing with focus in MathDial, the measured F1 ceiling could partly reflect label noise rather than a fundamental unpredictability of tutor behavior.
- A direct testable extension would be to re-label both datasets with a merged or behaviorally precise taxonomy and measure whether future-move F1 rises sharply; if it does, annotation granularity, not prediction, is the bottleneck.
- The success-prediction gain from moves suggests a practical loop the authors only gesture toward: an LLM tutor could select among predicted future moves by estimating which move maximizes the student's predicted success probability.
- On AlgebraNation, Llama 3 collapses toward the majority label giving_instruction in future-move prediction, so any deployment of these predictions would need calibration or class-balanced training before the outputs are usable.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies four tasks—current and future tutor move prediction, dialogue-level success prediction, and next-turn success prediction—using Llama 3 fine-tuned with LoRA and GPT-4o zero-shot on MathDial and AlgebraNation. It reports that future move prediction is hard (best F1 ≈49% on MathDial, ≈27% on AlgebraNation), that move labels sometimes improve outcome prediction, and that logistic regression coefficients suggest certain moves correlate with success. The paper is first to evaluate generative LLMs for the joint task of predicting tutor strategy and student outcomes.
Significance. If the central claims are robust, the paper provides a useful benchmark for a task that has received little attention, with careful held-out splits, multiple baselines, and error analysis. The qualitative analysis of confusable move categories is informative. However, the internal contradiction in the regression results and the ambiguity in the prompt condition construction need to be resolved before the conclusions about the value of move information and the intrinsic difficulty of strategy prediction can be trusted.
major comments (4)
- [Section 3.4, Table 9] Section 3.4 states that 'for MathDial, generic and probing have the strongest positive impact on success, whereas telling has a negative impact on success,' but Table 9 reports a significant negative coefficient for probing (-0.1867, chi-squared 6.7967, p=0.0091), with generic positive (+1.0635) and telling negative (-1.0627). This is an internal contradiction in the evidence for RQ3 and for the abstract's claim that 'tutor strategy is highly indicative of student outcomes.' If Table 9 is correct, the prose misreports the finding; if the prose is correct, the regression output is misreported. Please correct the text or the analysis and re-state the conclusion accordingly.
- [Section 3.4, Table 3] The paper claims in Section 3.4 that 'previous move labels improve performance for Llama 3,' but Table 3 shows that on MathDial, adding move labels slightly decreases weighted F1 for dialogue success (70.11 vs 69.96) and for turn success (50.54 vs 49.27); the only consistent improvement is on AlgebraNation dialogue success (75.52 to 80.84). The claim should be qualified to the AlgebraNation dataset, and the authors should report variance (e.g., multiple seeds) or significance tests before concluding that move annotations generally help outcome prediction.
- [Section 2.2, Figures 7-8] The zero-shot GPT-4o prompts shown in Figures 7 and 8 embed the previous tutor move labels directly in the dialogue text (e.g., '(generic)', '(focus)' in Figure 7; '["giving_instruction"]' in Figure 8). If the same prompt template is used in the 'Dialogue' condition, then that condition is not actually a text-only input; it already contains the previous move labels, which would conflate the 'Dialogue' and 'Dialogue & Moves' conditions and invalidate the comparison for GPT-4o. Please clarify whether the dialogue-only prompt strips these labels, and if not, re-run the experiments with a properly controlled prompt.
- [Section 3.3, Abstract] The conclusion that 'even state-of-the-art LLMs struggle to predict future tutor strategy' rests on the assumption that the ground-truth move labels are reliable. The paper itself notes in Section 3.3 that 'MathDial authors also note that annotators had difficulty differentiating between probing and focus moves' (citing Macina et al., 2023), and the top misclassifications for both models are precisely these confusable pairs (Tables 4-7). Without reporting inter-annotator agreement or a noise-adjusted evaluation, the low future-move F1 (49% MathDial, 27% AlgebraNation) may reflect annotation noise rather than an intrinsic limit of LLMs. Please provide an estimate of label reliability and discuss this alternative interpretation.
minor comments (5)
- [Section 2.2, References] The LoRA method is attributed to Saari et al. (2018), which is about Long Range (LoRa) wireless; the Low-Rank Adaptation method should cite Hu et al. (2021).
- [Section 3.3, Table 2] The statement 'LLMs improve over baselines for future move prediction' is not true for Llama 3 on AlgebraNation, whose F1 (20.20) is below the LSTM baseline (23.17). Please qualify the claim.
- [Section 3.3] The phrase 'GPT-4o was not trained on AlgebraNation' is trivially true for a zero-shot model; presumably the authors mean it was not fine-tuned on that dataset.
- [Tables 8-9] The logistic regression analysis does not report confidence intervals or multiple-testing correction; adding these would strengthen the RQ3 claims.
- [General] The paper does not state whether code is available; if applicable, please add a reproducibility statement.
Circularity Check
No circularity: the paper's predictions are evaluated on held-out test splits with external labels and standard baselines; the cited prior work supplies data and related context, not the target result.
full rationale
No circular step is present. RQ1 future-move prediction and RQ2 outcome prediction are evaluated on held-out test splits (MathDial 1,947/537 and AlgebraNation 1,854/464), and the reported F1 values are direct measurements rather than fitted quantities renamed as predictions. The ground-truth tutor move labels come from externally published datasets (MathDial, Macina et al. 2023; AlgebraNation, Lyu et al. 2024), and turn-level student correctness labels are taken from Scarlatos et al. (2025a), a prior published paper by co-authors. Although this is a self-citation, it is used as a data source, not as a load-bearing argument for the paper's central claim, and the labels are externally generated and falsifiable outside the present fitted values. The logistic-regression coefficients in Tables 8 and 9 are descriptive associations, not predictions derived from the target claim, so RQ3 does not reduce to an input. The main conclusion that even state-of-the-art LLMs struggle to predict future tutor moves is an empirical finding independent of any prior claim by the same authors. A separate correctness concern is noted but is not circular: Section 3.4 states that in MathDial 'probing' has a strong positive impact on success, while Table 9 reports a negative coefficient for probing (-0.1867, p=0.0091). This inconsistency affects the reliability of RQ3's interpretation, but it does not make the derivation self-referential.
Assumptions & free parameters
free parameters (7)
- LoRA rank =
8
- LoRA alpha =
16
- Learning rate for Llama 3 =
1e-4
- LSTM hidden dimension =
128
- LSTM dropout =
0.3
- Multi-label classification threshold =
0.85
- Logistic regression coefficients (Tables 8-9) =
Various, e.g., generic +1.0635 in MathDial
assumptions (4)
- domain assumption MathDial's four move labels and AlgebraNation's 16 move labels are valid operationalizations of tutor strategy.
- domain assumption Turn-level student correctness labels in MathDial, taken from Scarlatos et al. (2025a), are accurate ground truth.
- domain assumption MathDial's LLM-simulated student responses approximate real student behavior well enough for the conclusions to transfer.
- domain assumption The random train/test splits are representative and reproducible, despite no seed being reported.
Cite this review
Pith. "Pith review of Exploring LLMs for Predicting Tutor Strategy and Student Outcomes in Dialogues." pith.science (2026). https://pith.science/paper/D4FJXV36
@misc{pith2026250706910,
author = {Pith},
title = {Pith review of: Exploring LLMs for Predicting Tutor Strategy and Student Outcomes in Dialogues},
year = {2026},
howpublished = {\url{https://pith.science/paper/D4FJXV36}},
note = {Machine review of arXiv:2507.06910}
}
read the original abstract
Tutoring dialogues have gained significant attention in recent years, given the prominence of online learning and the emerging tutoring abilities of artificial intelligence (AI) agents powered by large language models (LLMs). Recent studies have shown that the strategies used by tutors can have significant effects on student outcomes, necessitating methods to predict how tutors will behave and how their actions impact students. However, few works have studied predicting tutor strategy in dialogues. Therefore, in this work we investigate the ability of modern LLMs, particularly Llama 3 and GPT-4o, to predict both future tutor moves and student outcomes in dialogues, using two math tutoring dialogue datasets. We find that even state-of-the-art LLMs struggle to predict future tutor strategy while tutor strategy is highly indicative of student outcomes, outlining a need for more powerful methods to approach this task.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Mark Abdelshiheed, Jennifer K. Jacobs, and Sidney K. D'Mello. 2024. Aligning tutor discourse supporting rigorous thinking with tutee content mastery for predicting math achievement. In Artificial Intelligence in Education, pages 150--164, Cham. Springer Nature Switzerland
work page 2024
-
[2]
Inneke Berghmans, Lotte Michiels, Sara Salmon, Filip Dochy, and Katrien Struyven. 2014. Directive versus facilitative peer tutoring? a view on students’ appraisal, reported learning gains and experiences within two differently-tutored learning environments. Learning Environments Research, 17:437--459
work page 2014
-
[3]
Conrad Borchers, Kexin Yang, Jionghao Lin, Nikol Rummel, Kenneth R. Koedinger, and Vincent Aleven. 2024. https://doi.org/10.5281/zenodo.12729784 Combining dialog acts and skill modeling: What chat interactions enhance learning rates during ai-supported peer tutoring? In Proceedings of the 17th International Conference on Educational Data Mining, pages 117...
-
[4]
Kristy Elizabeth Boyer, Eun Young Ha, Michael D Wallis, Robert Phillips, Mladen A Vouk, and James C Lester. 2009. Discovering tutorial dialogue strategies with hidden markov models. In Artificial Intelligence in Education, pages 141--148. IOS Press
work page 2009
-
[5]
Carnegie Learning . 2024. Livehint overview. Online: https://support.carnegielearning.com/help-center/math/livehint/article/livehint-overview/
work page 2024
-
[6]
Jiahao Chen, Zitao Liu, Mingliang Hou, Xiangyu Zhao, and Weiqi Luo. 2024. https://doi.org/10.1145/3627673.3679108 Multi-turn classroom dialogue dataset: Assessing student performance from one-on-one conversations . In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, CIKM '24, page 5333–5337, New York, NY, USA. ...
arXiv 2024
-
[7]
Dorottya Demszky and Heather Hill. 2023. The ncte transcripts: A dataset of elementary math classroom transcripts. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 528--538
2023
-
[8]
Dorottya Demszky, Jing Liu, Zid Mancenido, Julie Cohen, Heather Hill, Dan Jurafsky, and Tatsunori B Hashimoto. 2021. Measuring conversational uptake: A case study on student-teacher interactions. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Proces...
2021
Show all 40 references
-
[9]
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783
2024 arXiv
-
[10]
Ananya Ganesh, Martha Palmer, and Katharina Kann. 2021. https://doi.org/10.18653/v1/2021.findings-acl.418 What would a teacher do? P redicting future talk moves . In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4739--4751, Online. Associati...
2021 doi
-
[11]
Thomas Huber, Christina Niklaus, and Siegfried Handschuh. 2023. https://doi.org/10.18653/v1/2023.bea-1.59 Enhancing educational dialogues: A reinforcement learning approach for generating AI teacher responses . In Proceedings of the 18th Workshop on Innovative Use of NLP for B...
2023 doi
-
[12]
Khan Academy . 2023. Supercharge your teaching experience with khanmigo. Online: https://www.khanmigo.ai/
2023
-
[13]
Seanie Lee, Jianpeng Cheng, Joris Driesen, Alexandru Coca, and Anders Johannsen. 2024. https://doi.org/10.18653/v1/2024.naacl-long.6 Effective and efficient conversation retrieval for dialogue state tracking with implicit text summaries . In Proceedings of the 2024 Conference ...
2024 doi
-
[14]
Hang Li, Tianlong Xu, Jiliang Tang, and Qingsong Wen. 2024. https://arxiv.org/abs/2406.13885 Knowledge tagging system on math questions via llms with flexible demonstration retriever . Preprint, arXiv:2406.13885
2024 arXiv
-
[15]
Jionghao Lin, Shaveen Singh, Lele Sha, Wei Tan, David Lang, Dragan Ga s evi \'c , and Guanliang Chen. 2022. Is it a good move? mining effective tutoring strategies from human--human tutorial dialogues. Future Generation Computer Systems, 127:194--207
2022
-
[16]
Baraniuk
Naiming Liu, Shashank Sonkar, and Richard G. Baraniuk. 2025. https://arxiv.org/abs/2502.15140 Do llms make mistakes like students? exploring natural alignment between language models and human error patterns . Preprint, arXiv:2502.15140
2025 arXiv
-
[17]
Bailing Lyu, Chenglu Li, Hai Li, Wangda Zhu, and Wanli Xing. 2024. Explaining technical, social, and discursive participation in online mathematical discussions. Distance Education, pages 1--24
2024
-
[18]
Jakub Macina, Nico Daheim, Sankalan Chowdhury, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. 2023. M ath D ial: A dialogue tutoring dataset with rich pedagogical properties grounded in math reasoning problems. In Findings of the Association for Computational L...
2023
-
[19]
Hunter McNichols and Andrew Lan. 2025. https://arxiv.org/abs/2503.07928 The studychat dataset: Student dialogues with chatgpt in an artificial intelligence course . Preprint, arXiv:2503.07928
2025
-
[20]
Baptiste Moreau-Pernet, Yu Tian, Sandra Sawaya, Peter Foltz, Jie Cao, Brent Milne, and Thomas Christie. 2024. https://doi.org/10.1145/3657604.3664664 Classifying tutor discursive moves at scale in mathematics classrooms with large language models . In Proceedings of the Eleven...
2024
-
[21]
Andre Nickow, Philip Oreopoulos, and Vincent Quan. 2020. The impressive effects of tutoring on prek-12 learning: A systematic review and meta-analysis of the experimental evidence. Working Paper 27476, National Bureau of Economic Research
2020
-
[22]
Benjamin D Nye, Arthur C Graesser, and Xiangen Hu. 2014. Autotutor and family: A review of 17 years of natural language tutoring. International Journal of Artificial Intelligence in Education, 24:427--469
2014
-
[23]
OpenAI. 2024 a . Gpt-4.1 prompting guide. https://cookbook.openai.com/examples/gpt4-1_prompting_guide. Accessed: 2025-04-21
2024
-
[24]
OpenAI. 2024 b . https://openai.com/index/hello-gpt-4o/ Hello gpt-4o . Accessed: 2025-02-19
2024
-
[25]
Mika Saari, A Muzaffar bin Baharudin, Pekka Sillberg, Sami Hyrynsalmi, and Wanglin Yan. 2018. Lora—a survey of recent research trends. In 2018 41st international convention on information and communication technology, electronics and microelectronics (MIPRO), pages 0872--0877. IEEE
2018
-
[26]
Baker, and Andrew Lan
Alexander Scarlatos, Ryan S. Baker, and Andrew Lan. 2025 a . Exploring knowledge tracing in tutor-student dialogues using llms. In Proceedings of the 15th Learning Analytics and Knowledge Conference, LAK 2025, Dublin, Ireland, March 3-7, 2025 . ACM
2025
-
[27]
Alexander Scarlatos, Naiming Liu, Jaewook Lee, Richard Baraniuk, and Andrew Lan. 2025 b . https://arxiv.org/abs/2503.06424 Training llm-based tutors to improve student learning outcomes in dialogues . Preprint, arXiv:2503.06424
2025 arXiv
-
[28]
Shashank Sonkar, Kangqi Ni, Sapana Chaudhary, and Richard Baraniuk. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.797 Pedagogical alignment of large language models . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13641--13650, Miami, F...
2024 doi
-
[29]
Martin, and Tamara Sumner
Abhijit Suresh, Jennifer Jacobs, Charis Harty, Margaret Perkoff, James H. Martin, and Tamara Sumner. 2022. The T alk M oves dataset: K-12 mathematics lesson transcripts annotated for teacher and student discursive moves. In Proceedings of the Thirteenth Language Resources and ...
2022
-
[30]
Ana \"i s Tack, Ekaterina Kochmar, Zheng Yuan, Serge Bibauw, and Chris Piech. 2023. https://doi.org/10.18653/v1/2023.bea-1.64 The BEA 2023 shared task on generating AI teacher responses in educational dialogues . In Proceedings of the 18th Workshop on Innovative Use of NLP for...
2023 doi
-
[31]
LearnLM Team, Abhinit Modi, Aditya Srikanth Veerubhotla, Aliya Rysbek, Andrea Huber, Brett Wiltshire, Brian Veprek, Daniel Gillick, Daniel Kasenberg, Derek Ahmed, Irina Jurenka, James Cohan, Jennifer She, Julia Wilkowski, Kaiz Alarakyia, Kevin R. McKee, Lisa Wang, Markus Kunes...
2024 arXiv
-
[32]
Justin Vasselli, Christopher Vasselli, Adam Nohejl, and Taro Watanabe. 2023. https://doi.org/10.18653/v1/2023.bea-1.63 NAIST eacher: A prompt and rerank approach to generating teacher utterances in educational dialogues . In Proceedings of the 18th Workshop on Innovative Use o...
2023 doi
-
[33]
Deliang Wang, Dapeng Shan, Yaqian Zheng, and Gaowei Chen. 2023. Teacher talk moves in k12 mathematics lessons: Automatic identification, prediction explanation, and characteristic exploration. In Artificial Intelligence in Education, pages 651--664, Cham. Springer Nature Switzerland
2023
-
[34]
Rose Wang and Dorottya Demszky. 2024. https://doi.org/10.18653/v1/2024.naacl-demo.6 Edu- C onvo K it: An open-source library for education conversation data . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: ...
2024 doi
-
[35]
Rose Wang, Qingyang Zhang, Carly Robinson, Susanna Loeb, and Dorottya Demszky. 2024 a . Bridging the novice-expert gap via models of decision-making: A case study on remediating math mistakes. In Proceedings of the 2024 Conference of the North American Chapter of the Associati...
2024
-
[36]
Rose E Wang, Ana T Ribeiro, Carly D Robinson, Susanna Loeb, and Dora Demszky. 2024 b . Tutor copilot: A human-ai approach for scaling real-time expertise. arXiv preprint arXiv:2410.03017
2024 arXiv
-
[37]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771
2019 arXiv
-
[38]
Stella Xin Yin, Zhengyuan Liu, Dion Hoe-Lian Goh, Choon Lang Quek, and Nancy F. Chen. 2025. https://doi.org/10.1145/3706468.3706474 Scaling up collaborative dialogue analysis: An ai-driven approach to understanding dialogue patterns in computational thinking education . In Pro...
2025
-
[39]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[40]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.