REVIEW 4 major objections 6 minor 40 references
Findings of MEGA: Maths Explanation with LLMs using the Socratic Method for Active Learning
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Students judge Socratic-style LLM tutoring better than step-by-step explanations on hard math problems.
desk verdict Useful empirical study of Socratic-style LLM tutoring with public artifacts, but the headline significance likely rests on a misapplied unpaired test and the novelty claim is contradicted by the authors' own reference [17]. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the MEGA system prompt, which instructs the LLM to act as a tutor, solve step by step, and at each step ask the user a two-choice question and wait for the answer, giving corrective feedback before continuing and offering to repeat with an easier explanation if the user says they do not understand. This prompt turns a monologue into a forced dialogue: the student must answer correctly before the model proceeds, and the model must respond to the student's choice. The evaluation machinery is a within-group pairwise comparison in which each student sees the CoT explanation and then interacts with MEGA for the same problem, with two students per question resolving by agreement, plus accuracy metrics (MEGA, Socratic, and CoT) computed over the model outputs.
What would settle it
Give two matched groups of students the same set of hard mathematics problems, let one group study with MEGA and the other with CoT explanations, and then test both groups on unseen problems; if the MEGA group does not score higher, the preference advantage reported here would not translate into a learning advantage.
Extended reading notes
Core claim
The paper's central claim is that MEGA—a prompt-driven tutoring format in which the LLM breaks a problem into Socratic sub-questions, offers two answer choices at each step, waits for the student's answer, and gives feedback before continuing—is experienced by students as better for learning than the traditional chain-of-thought explanation. The evidence is a within-group pairwise comparison: each student read a CoT explanation and then worked through a MEGA interaction for the same 15 questions, and the two students assigned to each question had to agree on which was better, with "None" recorded on disagreement. Averaged across GPT-4o and Claude 3.5 Sonnet, MEGA won 47.5% of the unanimous agreements on the MATH dataset versus 26.67% for CoT, a difference the authors report as statistically significant (p = 0.0201), while on GSM8K the difference was not significant (26.67% vs 24.17%). The claim is therefore specifically about hard mathematics: on routine word problems the interactive format does not help, and on difficult competition-style problems it does. The paper additionally claims that MEGA exposes hallucinations better than CoT, because errors in individual sub-questions become visible and students can catch them, and that model accuracies differ by method, with sub-question (Socratic) accuracy generally exceeding final MEGA accuracy.
Load-bearing premise
The study assumes that students' subjective judgment of which explanation was better to understand reflects which method actually helps them learn, since no test of learning was administered after the comparison.
Editorial extensions
If this is right
- On hard mathematics problems, an LLM tutor that asks guided questions with immediate feedback may be more useful to students than one that simply prints a worked solution; on easy problems the two formats are roughly equivalent.
- Because only three of six state-of-the-art LLMs tested could follow the MEGA prompt, prompt-following capability is currently a bottleneck for this teaching method.
- MEGA's sub-question format makes model hallucinations more visible: errors in individual steps appear during the dialogue and can be corrected by the student, even when the final answer is right or wrong.
- The sizable minority of students who prefer non-interactive learning (22.22%) suggests an interactive-only tutoring mode will not suit everyone.
- Accuracy results suggest the Socratic format can improve an LLM's own performance on some problems, but not uniformly: Socratic accuracy beats CoT accuracy in some settings and trails it in others.
Reading between the lines
- The paper measures perceived learning experience, not learning gain; an obvious next test is whether MEGA users actually solve unseen problems better after the interaction than CoT users.
- If the effect replicates, the same Socratic-tutoring prompt could be transferred to other quantitative subjects, such as signal processing, where the paper argues interactive hands-on pedagogy is underused.
- The "None" disagreement rate dropping on MATH hints that hard problems create clearer perceived differences; a follow-up could vary problem difficulty within one benchmark to map where the method's advantage begins.
- Students correcting model errors in the dialogue suggests a possible guardrail: pairing MEGA with an answer-checking step could reduce the impact of hallucinated sub-steps.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces MEGA, a prompt-based method that combines the Socratic method, chain-of-thought (CoT) reasoning, simplified gamification, and formative feedback for LLM-generated mathematics explanations. The study compares MEGA with a traditional CoT explanation condition through a within-group human evaluation: 34 master's students judged pairwise which explanation of two LLMs (GPT-4o and Claude 3.5 Sonnet) was better to understand, using 60 sampled GSM8K and 60 sampled MATH problems. The authors report that MEGA was preferred more often on both datasets, with a large apparent advantage on MATH (47.5% vs 26.67%, reported p = 0.0201), and they supplement this with LLM accuracy analyses across difficulty levels and problem types, plus qualitative student feedback and hallucination examples.
Significance. If the preference result were robust, MEGA would be a useful and timely contribution to AI-assisted mathematics education, particularly for harder problems. The paper has concrete strengths: the MEGA app, the system prompt, and the student annotations are publicly released; the evaluation uses a blind pairwise protocol with two state-of-the-art LLMs; the questions are drawn from widely used external benchmarks (GSM8K and MATH); and the hallucination examples are instructive. However, the central statistical claim rests on a test that appears mismatched to the paired design, and the outcome measure is perceived preference rather than measured learning. The per-category accuracy analyses are also too sparse to support the conclusions drawn. These issues are fixable by reanalysis and careful rewording, but they currently block acceptance of the headline claim.
major comments (4)
- [§IV-A, Fig. 2a] The headline MATH comparison (MEGA 47.5% vs CoT 26.67%, reported p = 0.0201) is analyzed as if the two proportions were independent, but the design is within-group: the same students judged both methods for the same LLM-generated explanations on the same question set. For paired binary preference data, McNemar's test is the standard check. Using the printed margins (28.5 MEGA vs 16 CoT preferences among 60 MATH questions, with 15.5 ties), McNemar's chi-square is about (12.5)^2/44.5 = 3.51, p ≈ 0.061, or p ≈ 0.085 with continuity correction, so the difference is no longer significant at α = 0.05. The same paired-structure issue applies to the per-LLM p-values reported for Figure 2b. The authors should report the exact test used, apply a paired test, and account for the clustering of multiple questions within student pairs before claiming a significant effect.
- [Abstract, §IV-A, §V] The conclusion that 'MEGA is better at explaining difficult Maths problems' exceeds what the study measures. The outcome is a subjective preference for which explanation was 'better to understand,' collected after the student read the CoT explanation and then interacted with MEGA in a fixed order; no test of learning or retention was administered, and the paper itself defers learning measurement to future work in §V. The claims should be limited to perceived understandability/preference, and the fixed presentation order should be acknowledged as a potential order-effect confound or addressed by counterbalancing.
- [§III-A, §III-D] The CoT baseline is not specified as a system prompt, so the comparison is not fully defined. Section III-A gives the full MEGA instruction, but for CoT the reader is only told that it is the 'traditional step-by-step explanation' and shown sample chats. Without the exact CoT prompt, model version/API details, temperature/top-p settings for both conditions, and whether the same model was queried with identical underlying question text, it is impossible to know whether any difference reflects the MEGA method or the authors' prompt engineering. Please provide the CoT prompt and the inference protocol for both conditions, and ideally an ablation that separates the Socratic, gamification, and feedback components.
- [§IV-C, Tables III and IV] The per-difficulty and per-type MATH accuracy analyses are based on very small cells (e.g., 2 level-1 problems, 4 counting-probability problems, 5 prealgebra problems) and the authors themselves label them 'merely indicative' in §IV-C. Despite this, the text draws directional conclusions such as 'accuracies generally fall as the difficulty rises after level 3' and 'Prealgebra questions appear to have the highest accuracies.' These statements should be explicitly labeled as exploratory, with no inferential language, or supported by a statistical model that accounts for problem-level clustering.
minor comments (6)
- [§III-C] The default hyperparameter line reads 'top p (or nucleus sampling) = 1 (or 7 for Claude 3.5 Sonnet)'; this should be '0.7' to match the subsequent explanation.
- [§II-B and Appendix] Several inline equations are garbled in the text version (e.g., the trigonometric ratios and the Laplace transform in the appendix); the final PDF should be proofread for missing fraction bars and operator spacing.
- [§IV-B] The wordclouds are described as content analysis, but the preprocessing (stopword removal, stemming, phrase extraction, thresholding) is not specified; add these details or present the wordclouds as illustrative rather than as a formal content analysis.
- [§I] The statement that the paper 'was accepted for the special issue' does not belong in a submitted manuscript; remove it before publication.
- [Table II] Student quotations contain typos such as 'managable', 'oponion', and 'Bot overall'; if kept as verbatim responses, note that they are unedited, otherwise copyedit them.
- [§II-D] The text contains a typo, 'Fianlly' for 'Finally', and a few other minor spelling errors elsewhere; a careful proofreading pass is recommended.
Circularity Check
No circularity: the study is an empirical human-preference comparison whose inputs (benchmark questions, LLM outputs, student pairwise choices) are not constructed from the claimed conclusion.
full rationale
This paper contains no mathematical derivation whose conclusion could be equivalent to its inputs. The central claim is an empirical comparison: students pairwise preferred MEGA explanations over CoT explanations for samples drawn from GSM8K and MATH. MEGA is defined by a system prompt authored by the researchers, and evaluating one's own prompt design is a study-design choice, not circularity under the defined criteria: the outcome (student preference) is not used to construct the prompt, and no parameter is fitted to the preference data and then renamed as a prediction. Model accuracies are computed directly from external benchmark questions using explicit formulas (Eqs. 1-3), so they are not derived from the evaluation outcome. Self-citations appear in the paper, but they are not load-bearing: [37] supports a general claim about energy consumption, [39] supports a demographic survey item, and [12] supplies a general definition of hallucination. None of these citations is invoked to forbid alternatives or to justify the MEGA design. The paper's acknowledged weaknesses, such as small per-category MATH sample sizes and the use of subjective preference as a proxy for learning, are validity or generalizability concerns, not circularity. The skeptical statistical concern about McNemar versus an unpaired test is likewise a correctness risk, not a case of the conclusion reducing to its inputs. Therefore, the derivation chain is self-contained and the appropriate circularity score is 0.
Assumptions & free parameters
assumptions (4)
- domain assumption Students' self-reported preference for an explanation is a valid proxy for actual learning.
- domain assumption The two-option multiple-choice interaction with LLMs represents the Socratic method and yields valid pedagogy.
- domain assumption The two selected LLMs, GPT-4o and Claude 3.5 Sonnet, are representative of state-of-the-art capable tutors.
- domain assumption GSM8K and MATH dataset answers are correct ground truth.
Cite this review
Pith. "Pith review of Findings of MEGA: Maths Explanation with LLMs using the Socratic Method for Active Learning." pith.science (2026). https://pith.science/paper/PTZKZGJS
@misc{pith2026250712079,
author = {Pith},
title = {Pith review of: Findings of MEGA: Maths Explanation with LLMs using the Socratic Method for Active Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/PTZKZGJS}},
note = {Machine review of arXiv:2507.12079}
}
read the original abstract
This paper presents an intervention study on the effects of the combined methods of (1) the Socratic method, (2) Chain of Thought (CoT) reasoning, (3) simplified gamification and (4) formative feedback on university students' Maths learning driven by large language models (LLMs). We call our approach Mathematics Explanations through Games by AI LLMs (MEGA). Some students struggle with Maths and as a result avoid Math-related discipline or subjects despite the importance of Maths across many fields, including signal processing. Oftentimes, students' Maths difficulties stem from suboptimal pedagogy. We compared the MEGA method to the traditional step-by-step (CoT) method to ascertain which is better by using a within-group design after randomly assigning questions for the participants, who are university students. Samples (n=60) were randomly drawn from each of the two test sets of the Grade School Math 8K (GSM8K) and Mathematics Aptitude Test of Heuristics (MATH) datasets, based on the error margin of 11%, the confidence level of 90%, and a manageable number of samples for the student evaluators. These samples were used to evaluate two capable LLMs at length (Generative Pretrained Transformer 4o (GPT4o) and Claude 3.5 Sonnet) out of the initial six that were tested for capability. The results showed that students agree in more instances that the MEGA method is experienced as better for learning for both datasets. It is even much better than the CoT (47.5% compared to 26.67%) in the more difficult MATH dataset, indicating that MEGA is better at explaining difficult Maths problems.
Reference graph
Works this paper leans on
-
[17]
Socratic as mathematics learning application for differential equations concept,
W. N. Yanuarto, A. Jaelani, J. Purwanto, and M. I. Zakaria, “Socratic as mathematics learning application for differential equations concept,” Hipotenusa: Journal of Mathematical Society, vol. 3, no. 1, pp. 97–110, 2021
work page 2021
-
[1]
H. J. Sherman, L. I. Richardson, and G. J. Yard, Teaching learners who struggle with mathematics: Responding with systematic intervention and remediation. Waveland Press, 2019
work page 2019
-
[2]
A. Karp and G. Schubring, Handbook on the history of mathematics education. Springer, 2014
work page 2014
-
[3]
M. Narwaria, “The transition from white box to black box: Challenges and opportunities in signal processing education,” IEEE Signal Processing Magazine, vol. 38, no. 3, pp. 163–173, 2021
work page 2021
-
[4]
A signal processor teaches generative artificial intelligence [sp education],
R. J. Radke, “A signal processor teaches generative artificial intelligence [sp education],” IEEE Signal Processing Magazine, vol. 41, no. 2, pp. 6–10, 2024
work page 2024
-
[5]
A pragmatic introduction to signal processing,
T. O’Haver, “A pragmatic introduction to signal processing,” University of Maryland at College Park, 1997
work page 1997
- [6]
-
[7]
J. Jia, T. Wang, Y. Zhang, and G. Wang, “The comparison of general tips for mathematical problem solving generated by generative ai with those generated by human teachers,” Asia Pacific Journal of Education, vol. 44, no. 1, pp. 8–28, 2024
work page 2024
Show all 40 references
-
[8]
Language models are few-shot learners,
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020
1901
-
[9]
Socratic method,
H. H. Benson, “Socratic method,” The Cambridge companion to socrates, pp. 179–200, 2011
2011
-
[10]
To describe, transmit or inquire: Ethics and technology in school,
V. Gardelli, “To describe, transmit or inquire: Ethics and technology in school,” Ph.D. dissertation, Lulea˚ tekniska universitet, 2016
2016
-
[11]
Ai4edu: An innovative conversational ai assistant for teaching and learning,
A. Vacalopoulou, V. Gardelli, T. Karafyllidis, F. Liwicki, H. Mokayed, M. Papaevripidou, G. Paraskevopoulos, S. Stamouli, A. Katsamanis, and V. Katsouros, “Ai4edu: An innovative conversational ai assistant for teaching and learning,” in INTED2024 Proceedings. IATED, 2024, pp. ...
2024
-
[12]
Digihealth-ai: Outcomes of the first blended intensive programme (bip) on ai for health –a cross-disciplinary multi-institutional short teaching course,
T. Adewumi, M. Gerdes, G. Chaltikyan, F. Fernandes, L. Lindsko¨ld, M. Liwicki, and M. Catta -Preta, “Digihealth-ai: Outcomes of the first blended intensive programme (bip) on ai for health –a cross-disciplinary multi-institutional short teaching course,” Journal of Applied Int...
2024
-
[13]
Training verifiers to solve math word problems,
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano et al., “Training verifiers to solve math word problems,” arXiv preprint arXiv:2110.14168, 2021
2021 arXiv
-
[14]
Measuring mathematical problem solving with the math dataset,
D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt, “Measuring mathematical problem solving with the math dataset,” in Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, J. Vanschoren and S. Yeu...
2021
-
[15]
Comparison of new mathematics teaching methods with traditional method,
A. Lessani, A. Yunus, and K. Bakar, “Comparison of new mathematics teaching methods with traditional method,” People: International Journal of Social Sciences, vol. 3, no. 2, pp. 1285–1297, 2017
2017
-
[16]
Personalized education in the artificial intelligence era: what to expect next,
S. Maghsudi, A. Lan, J. Xu, and M. van Der Schaar, “Personalized education in the artificial intelligence era: what to expect next,” IEEE Signal Processing Magazine, vol. 38, no. 3, pp. 37–50, 2021
2021
-
[18]
Experiments with a socratic intelligent tutoring system for source code understanding,
Z. Alshaikh, L. J. Tamang, and V. Rus, “Experiments with a socratic intelligent tutoring system for source code understanding,” in The Thirty-Third International Florida Artificial Intelligence Research Society Conference (FLAIRS-32), 2020
2020
-
[19]
https://ai4edu.eu/
(2024) AI4Edu. https://ai4edu.eu/
2024
-
[20]
Where’s the evidence that active learning works?
J. Michael, “Where’s the evidence that active learning works?” Advances in physiology education, 2006. 20 s s s
2006
-
[21]
Does active learning work? a review of the research,
M. Prince, “Does active learning work? a review of the research,” Journal of engineering education, vol. 93, no. 3, pp. 223–231, 2004
2004
-
[22]
Beyond active learning: A constructivist approach to learning,
S. E. Cooperstein and E. Kocevar-Weidinger, “Beyond active learning: A constructivist approach to learning,” Reference services review, vol. 32, no. 2, pp. 141–148, 2004
2004
-
[23]
Gamification and education: A literature review,
I. Caponetto, J. Earp, and M. Ott, “Gamification and education: A literature review,” in European conference on games based learning, vol. 1. Academic Conferences International Limited, 2014, p. 50
2014
-
[24]
Self-determination theory,
E. L. Deci and R. M. Ryan, “Self-determination theory,” Handbook of theories of social psychology, vol. 1, no. 20, pp. 416–436, 2012
2012
-
[25]
Student-centered learning in higher education
G. B. Wright, “Student-centered learning in higher education.” International journal of teaching and learning in higher education, vol. 23, no. 1, pp. 92– 97, 2011
2011
-
[26]
Experiential learning theory: Previous research and new directions,
D. A. Kolb, R. E. Boyatzis, and C. Mainemelis, “Experiential learning theory: Previous research and new directions,” in Perspectives on thinking, learning, and cognitive styles. Routledge, 2014, pp. 227–247
2014
-
[27]
Guiding students in using llms in supported learning environments: Effects on interaction dynamics, learner performance, confidence, and trust,
H. Kumar, I. Musabirov, M. Reza, J. Shi, X. Wang, J. J. Williams, A. Kuzminykh, and M. Liut, “Guiding students in using llms in supported learning environments: Effects on interaction dynamics, learner performance, confidence, and trust,” Proceedings of the ACM on Human -Compu...
2024
-
[28]
The 2 sigma problem: The search for methods of group instruction as effective as one -to-one tutoring,
B. S. Bloom, “The 2 sigma problem: The search for methods of group instruction as effective as one -to-one tutoring,” Educational researcher, vol. 13, no. 6, pp. 4–16, 1984
1984
-
[29]
Higgins and A
S. Higgins and A. Simpson, “Visible learning: A synthesis of over 800 meta -analyses relating to achievement. by john ac hattie: Pp 392. london: Routledge. 2008.£ 90 (hbk),£ 27.99 (pbk),£ 35.37 (e-book). isbn-13 978-0415476171 (hbk), isbn-13 978-0415476188 (pbk), asin: B001olr...
2008
-
[30]
Recent developments in using digital technology in mathematics education,
J. Engelbrecht and M. C. Borba, “Recent developments in using digital technology in mathematics education,” ZDM–Mathematics Education, vol. 56, no. 2, pp. 281–292, 2024
2024
-
[31]
13 chatgpt and active learning,
A. Adiyono, J. Jasiah, M. Ritonga, and A. S. Al-Matari, “13 chatgpt and active learning,” Empowering Digital Education with ChatGPT: From Theoretical to Practical Applications, p. 189, 2024
2024
-
[32]
Deepseek -v3 technical report,
A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan et al. , “Deepseek -v3 technical report,” arXiv preprint arXiv:2412.19437, 2024
2024 arXiv
-
[33]
(2022) Introducing ChatGPT
OpenAI. (2022) Introducing ChatGPT. https://openai.com/blog/chatgpt
2022
-
[34]
(2024) Claude
Anthropic. (2024) Claude. https://claude.ai/
2024
-
[35]
(2024) Gemini
Google. (2024) Gemini. https://gemini.google.com/
2024
-
[36]
Attention is all you need,
A. Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems, 2017
2017
-
[37]
Fairness and bias in multimodal ai: A survey,
T. Adewumi, L. Alkhaled, N. Gurung, G. van Boven, and I. Pagliai, “Fairness and bias in multimodal ai: A survey,” arXiv preprint arXiv:2406.19097, 2024
2024 arXiv
-
[38]
Applying criteria to examples or learning by comparison: Effects on students’ evaluative judgment and performance in writing,
R. Bouwer, M. Lesterhuis, P. Bonne, and S. De Maeyer, “Applying criteria to examples or learning by comparison: Effects on students’ evaluative judgment and performance in writing,” in Frontiers in Education, vol. 3. Frontiers Media SA, 2018, p. 86
2018
-
[39]
Generative ai and teachers–for us or against us? a case study,
J. Pettersson, E. Hult, T. Eriksson, and T. Adewumi, “Generative ai and teachers–for us or against us? a case study,” in 14th Scandinavian Conference on Artificial Intelligence SCAI, 2024
2024
-
[40]
Artificial general intelligence: concept, state of the art, and future prospects,
B. Goertzel, “Artificial general intelligence: concept, state of the art, and future prospects,” Journal of Artificial General Intelligence, vol. 5, no. 1, p. 1, 2014. APPENDIX MEGA’s correct solution of the Laplace transform of the Heaviside function supplied as an image moda...
2014
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.