Pith. sign in

REVIEW 3 major objections 3 minor 39 references

TutorLLM: Customizing Learning Recommendations with Knowledge Tracing and Retrieval-Augmented Generation

T0 review · 3 major / 3 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TutorLLM claims a 10% user-satisfaction gain and 5% higher quiz scores by feeding knowledge tracing and course content into GPT-4.

desk verdict A plausible KT+RAG tutoring pipeline whose own numbers undercut its headline 5% and 10% claims; useful as a systems note, not as evidence. read the letter →

arxiv 2502.15709 v2 pith:RY6J3P4O submitted 2025-01-20 cs.IR cs.AIcs.LG

Zhaoxing Li , Vahid Yazdanpanah , Jindi Wang , Wen Gu , Lei Shi , Alexandra I. Cristea , Sarah Kiden , Sebastian Stein This is my paper · ORCID
classification cs.IRcs.AIcs.LG
keywords LearningRecommenderSystemLargeLanguageModelsPersonalizedKnowledgeTracingRetrieval-AugmentedGenerationMLFBKUsabilityScaleChromeplugin
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TutorLLM sets out to close the gap between general-purpose LLMs, which answer fluently but ignore who is asking, and knowledge-tracing recommenders, which model the student but speak from a fixed database. The paper proposes a Chrome-plugin system that fuses a knowledge-tracing model (MLFBK), a scraper that captures course-specific text such as video captions, and a retrieval-augmented GPT-4 backend so that each answer and study recommendation is conditioned on the student's predicted mastery state and the current course content. The authors report that in a two-week crossover study with 30 undergraduates, TutorLLM users showed a 10% increase in user satisfaction and a 5% increase in quiz scores over a general LLM alone. If those gains hold, the practical point is that personalized intelligent tutoring can be built on top of existing LLM APIs without retraining, by wiring student state and course content into the prompt.

What carries the argument

The load-bearing mechanism is the prompt-level fusion of two signals: the KT-predicted student state (skill mastery, ability profile, problem difficulty, and predicted next action) and the Scraper-built course knowledge base, both fed into a retrieval-augmented GPT-4 generator. MLFBK is a BERT-based knowledge-tracing model with multi-feature embedding and latent-relation embedding that turns historical interactions into a correctness-sequence prediction; RAG lets the LLM retrieve relevant passages from the scraped course text before generating. This combination is what the paper claims is new: KT supplies personalization, RAG supplies relevance, and the LLM supplies fluency.

What would settle it

A pre-registered randomized trial with at least 50 students per arm, identical daily quizzes, and the same linear-algebra content would settle it: if the TutorLLM arm's mean quiz score is within a trivial margin of the general-LLM arm's mean, or the 95% confidence interval for the difference includes zero, the paper's 5% improvement claim fails to replicate.

Watch

Extended reading notes

Core claim

On the paper's own account, the central discovery is that putting a knowledge-tracing signal in front of a retrieval-augmented LLM changes the quality and personalization of educational recommendations enough to move both satisfaction and performance. TutorLLM's pipeline has three stages: the Scraper turns the current course page into a background knowledge base via the Jina Reader API; the MLFBK knowledge-tracing model takes the student's feature sequence and predicts mastery, ability, difficulty, and next action; and the RAG-enhanced GPT-4 model receives the retrieved course passages plus the KT state and produces answers and study recommendations. The paper reports that in a two-week linear algebra field study with 30 students allocated to general-LLM, hybrid, and TutorLLM conditions, the TutorLLM group had the highest mean quiz score (74.48 vs 72.81 and 71.97) and that users gave a mean System Usability Scale score of 76.35, with the abstract citing a 10% satisfaction gain and 5% score gain over general LLMs. The authors also note that the quiz-score difference was not statistically significant (ANOVA p = 0.462), and frame the result as a trend needing larger or longer studies.

Load-bearing premise

The quiz-score claim assumes the two-week, 30-student crossover study has enough statistical power to detect a real effect, and the paper's own ANOVA (p = 0.462) shows the observed 5% difference is also consistent with no true effect.

Editorial extensions

If this is right

  • If the effect holds, a university course can get a personalized tutor as a browser plugin, without retraining or fine-tuning any model.
  • The same three-component design can be pointed at any course whose materials exist as page text or captions, which suggests it is course-agnostic.
  • Students using TutorLLM spent 36% more time in the system, so the personalization appears to increase engagement even if score gains are uncertain.
  • Grounding answers in scraped course text gives a concrete path to reducing LLM hallucination in education.
  • The KT state also enables post-session recommendations, turning tutoring from answer-only to curriculum-aware.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A cleaner causal test of the paper's mechanism would hold the recommendation list fixed: comparing TutorLLM with and without the KT prompt would separate the contribution of knowledge tracing from that of retrieval alone, an ablation the paper does not run.
  • The 10% satisfaction gain may partly reflect novelty or the extra study recommendations; a second study that logs whether users actually followed the recommendations would show whether engagement or personalization drives the effect.
  • The architecture could generalize beyond text: the same prompt-fusion idea could feed the KT state and scraped math notation or diagrams into a multimodal model, a step the paper leaves implicit.
  • If the non-significant quiz gap is the true state of the world, the practical product claim rests on the satisfaction and engagement gains rather than on measured learning gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes TutorLLM, a personalized learning recommender that combines Knowledge Tracing (specifically the MLFBK BERT-based model), a web-scraping component, and Retrieval-Augmented Generation on top of GPT-4, delivered as a Chrome browser plugin. The authors report a two-week user study with 30 undergraduate students in a linear algebra course, comparing TutorLLM, a hybrid condition, and a general-LLM control. The Abstract and Introduction claim a 10% improvement in user satisfaction (SUS) and a 5% increase in quiz scores relative to using general LLMs alone. Section 5 reports the TutorLLM group's mean quiz score as 74.48, the hybrid group as 72.81, and the general-LLM group as 71.97, with an ANOVA F=0.795 and p=0.462. Section 6 explicitly acknowledges that no statistically significant improvements were found. SUS data are reported only for TutorLLM users (mean 76.35, SD 14.46), with no control-group SUS baseline.

Significance. If TutorLLM genuinely produced the claimed gains in satisfaction and quiz performance, it would be a useful empirical contribution to personalized educational AI. The paper also ships a concrete implementation (Chrome plugin) and a field-study design with SUS and UX instruments, which are appropriate evaluation tools. However, the paper's headline quantitative claims are not supported by its own reported statistics: the quiz-score difference is about 3.5% relative, not 5%, and the ANOVA does not reject the null hypothesis; the satisfaction improvement has no measured control baseline. The system description is plausible, and the combination of KT and RAG in an LLM tutor is a reasonable integration idea, but the evidence presented cannot support the Abstract's 'demonstrating' language.

major comments (3)
  1. [Abstract, Section 5, Section 6] The claimed '5% increase in quiz scores' is not consistent with the reported results. Section 5 reports overall mean scores of 74.48 (TutorLLM), 72.81 (hybrid), and 71.97 (general LLM), so the TutorLLM-versus-general difference is 2.51 points, or approximately 3.5% relative to 71.97, not 5%. The only inferential test reported, an ANOVA, yields F=0.795 and p=0.462, indicating no significant between-group differences. Section 6 even states that 'our study did not show statistically significant improvements in academic performance.' The Abstract's quantified benefit claim therefore overstates the paper's own evidence and should be corrected or removed.
  2. [Section 4, Section 5] The claimed '10% improvement in user satisfaction' is unsupported because no control-group SUS data are reported. Section 4 says the questionnaire was administered to students in Group 1 and Group 2, not to the general-LLM control group, and Section 5 reports only the TutorLLM SUS mean (76.35, SD 14.46) with no SUS scores for the control condition. Without a baseline SUS measurement, the paper cannot quantify a 10% improvement over general LLMs; this statement should be removed or replaced with a descriptive-only claim about the TutorLLM SUS score.
  3. [Section 5] The statistical reporting is ambiguous about which metric was analyzed. Section 5 first introduces 'overall mean score' (74.48, 72.81, 71.97) and then says an ANOVA was conducted on 'final exam scores' across groups. These are not the same metric, and the paper does not clarify which scores entered the ANOVA. The p-value of 0.462 is used to discuss the overall mean differences, but if the ANOVA was actually on final-exam scores only, the reported means are not the tested quantities. Please specify the exact data (daily quiz average, final exam, or a repeated-measures structure) used in each test, and consider a paired analysis that exploits the crossover design for Group 1.
minor comments (3)
  1. [Section 4] The paper says 30 students were 'randomly allocated to one of three groups' but does not explicitly state the per-group sample size; please report n per group and the allocation procedure.
  2. [Section 3] The knowledge tracing description says problem difficulty is derived from success rates on a 1-10 scale, but the exact formula or discretization is not given; a sentence clarifying this step would improve reproducibility.
  3. [References] The manuscript retains placeholder ACM DOI and ISBN fields; these need to be filled or removed for the posted version.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TutorLLM's satisfaction and quiz-score claims come from a user study, not from the KT/RAG components or from self-citations.

full rationale

The paper contains no derivation chain of the kind that can be circular: TutorLLM is an engineering integration of the MLFBK knowledge-tracing model, a Scraper, and RAG with GPT-4, and its headline claims are empirical outcomes of a two-week user study. The Abstract's '10% improvement in user satisfaction and a 5% increase in quiz scores' are not defined in terms of, or fitted from, the MLFBK parameters, the Scraper output, or the retrieval corpus; they are measured from SUS questionnaires and quiz scores. The paper's own Section 5 reports an ANOVA with F=0.795 and p=0.462, and Section 6 explicitly says 'our study did not show statistically significant improvements in academic performance,' and no control-group SUS baseline is reported; these are evidential weaknesses, not circularity. References [15]-[21] and [31]-[34] include prior work by the authors, and Section 3 says 'we use MLFBK[16]', so there is self-citation, but the load-bearing empirical claims are not reduced to those citations: the MLFBK model is used as a component rather than being invoked to prove the study's outcomes, and no uniqueness or derivation claim depends on the authors' prior results. The study is therefore self-contained as an empirical system paper; the correct criticism is about statistical support and internal consistency of the reported percentages, not about circular reasoning.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The system is composed of prior components (MLFBK, RAG, GPT-4) with no new fitted parameters introduced in this paper; the accountability burden falls on the unvalidated integration and the study design.

free parameters (1)
  • Problem difficulty scale (1-10) = derived from success rates on the course data
    Imported from MLFBK (ref [16]); it shapes the latent-relations embedding and the predicted knowledge state, but the paper provides no estimate for the linear algebra cohort.
assumptions (3)
  • domain assumption MLFBK accurately predicts student knowledge state.
    The KT component relies on ref [16]; no validation on the linear algebra cohort is reported.
  • domain assumption Scraped course text is sufficient for RAG.
    Section 3 assumes captions and subtitles from Jina Reader contain the knowledge needed for accurate answers.
  • domain assumption Quiz scores are valid outcome measures.
    Section 4 uses daily quizzes and a final exam as learning outcomes without a pre-test or psychometric validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TutorLLM: Customizing Learning Recommendations with Knowledge Tracing and Retrieval-Augmented Generation." pith.science (2026). https://pith.science/paper/RY6J3P4O

@misc{pith2026250215709,
  author       = {Pith},
  title        = {Pith review of: TutorLLM: Customizing Learning Recommendations with Knowledge Tracing and Retrieval-Augmented Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RY6J3P4O}},
  note         = {Machine review of arXiv:2502.15709}
}
read the original abstract

The integration of AI in education offers significant potential to enhance learning efficiency. Large Language Models (LLMs), such as ChatGPT, Gemini, and Llama, allow students to query a wide range of topics, providing unprecedented flexibility. However, LLMs face challenges, such as handling varying content relevance and lack of personalization. To address these challenges, we propose TutorLLM, a personalized learning recommender LLM system based on Knowledge Tracing (KT) and Retrieval-Augmented Generation (RAG). The novelty of TutorLLM lies in its unique combination of KT and RAG techniques with LLMs, which enables dynamic retrieval of context-specific knowledge and provides personalized learning recommendations based on the student's personal learning state. Specifically, this integration allows TutorLLM to tailor responses based on individual learning states predicted by the Multi-Features with Latent Relations BERT-based KT (MLFBK) model and to enhance response accuracy with a Scraper model. The evaluation includes user assessment questionnaires and performance metrics, demonstrating a 10% improvement in user satisfaction and a 5\% increase in quiz scores compared to using general LLMs alone.

Figures

Figures reproduced from arXiv: 2502.15709 by the authors.

Figure 1
Figure 1. Overall architecture of TutorLLM. students upon request or after each study session. Based on the above ideas, we built a Chrome browser plug-in. When students open an online course website, they can open our TutorLLM by clicking the button on the right-hand side of the address bar [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. User Interface of the TutorLLM. derived from success rates. Additionally, the overall model integra￾tion encompasses a combination of embeddings such as question, item, response, skill mastery, ability profile, and problem difficulty embeddings, culminating in creating a final input embedding. In the BERT-based architecture, encoder blocks leverage a pre-LN Transformer architecture [29], incorporating monotonic conv… view at source ↗
Figure 3
Figure 3. Daily Mean Scores of User Performance Across [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 29 canonical work pages

  1. [1]

    Razvan Azamfirei, Sapna R Kudchadkar, and James Fackler. 2023. Large language models and the perils of their hallucinations. Critical Care 27, 1 (2023), 120

    work page 2023

  2. [2]

    Aaron Bangor, Philip T Kortum, and James T Miller. 2008. An empirical evaluation of the system usability scale. Intl. Journal of Human–Computer Interaction 24, 6 (2008), 574–594

    2008

  3. [3]

    Susan Bergin and Ronan Reilly. 2005. The influence of motivation and comfort- level on learning to program. (2005)

    work page 2005

  4. [4]

    Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

    work page 2020

  5. [5]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al . 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology 15, 3 (2024), 1–45

    2024

  6. [6]

    Hsinchun Chen, Roger HL Chiang, and Veda C Storey. 2020. Big data analytics for educational recommender systems: A review and research agenda. MIS quarterly 44, 1 (2020), 135–169

    work page 2020

  7. [7]

    Jiahao Chen, Zitao Liu, Shuyan Huang, Qiongqiong Liu, and Weiqi Luo. 2023. Improving interpretability of deep sequential knowledge tracing models with question-centric cognitive representations. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 14196–14204

    work page 2023

  8. [8]

    Tsung-Hsien Chen, Minghao Li, Ying-Hsun Chien, et al. 2021. A survey of large language models. arXiv preprint arXiv:2109.11601 (2021)

    work page Pith review arXiv 2021

Show all 39 references
  1. [9]

    Albert T Corbett and John R Anderson. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction 4, 4 (1994), 253–278

    1994

  2. [10]

    Luciano Floridi and Massimo Chiriatti. 2020. GPT-3: Its nature, scope, limits, and consequences. Minds and Machines 30, 4 (2020), 681–694

    2020

  3. [11]

    Mark Freiermuth and Douglas Jarrell. 2006. Willingness to communicate: can online chat help? 1. International journal of applied linguistics 16, 2 (2006), 189– 212

    2006

  4. [12]

    Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al. 2023. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language...

    2023

  5. [13]

    Unggi Lee, Yonghyun Park, Yujin Kim, Seongyune Choi, and Hyeoncheol Kim

  6. [14]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing S...

    2020

  7. [15]

    Zhaoxing Li. 2023. Deep Reinforcement Learning Approaches for Technology Enhanced Learning. Ph. D. Dissertation. Durham University

    2023

  8. [16]

    Zhaoxing Li, Mark Jacobsen, Lei Shi, Yunzhan Zhou, and Jindi Wang. 2023. Broader and Deeper: A Multi-Features with Latent Relations BERT Knowl- edge Tracing Model. In European Conference on Technology Enhanced Learning . Springer, 183–197

    2023

  9. [17]

    Zhaoxing Li, Lei Shi, Alexandra I Cristea, and Yunzhan Zhou. 2021. A survey of collaborative reinforcement learning: interactive methods and design patterns. In Proceedings of the 2021 ACM Designing Interactive Systems Conference. 1579–1590

    2021

  10. [18]

    Zhaoxing Li, Lei Shi, Jindi Wang, Alexandra I Cristea, and Yunzhan Zhou. 2023. Sim-GAIL: A generative adversarial imitation learning approach of student mod- elling for intelligent tutoring systems. Neural Computing and Applications 35, 34 (2023), 24369–24388

    2023

  11. [19]

    Zhaoxing Li, Lei Shi, Yunzhan Zhou, and Jindi Wang. 2023. Towards student behaviour simulation: a decision transformer based approach. In International Conference on Intelligent Tutoring Systems . Springer, 553–562

    2023

  12. [20]

    Zhaoxing Li, Jujie Yang, Jindi Wang, Lei Shi, Jiayi Feng, and Sebastian Stein

  13. [21]

    Zhaoxing Li, Jujie Yang, Jindi Wang, Lei Shi, and Sebastian Stein. 2024. Integrating LSTM and BERT for Long-Sequence Data Analysis in Intelligent Tutoring Systems. arXiv preprint arXiv:2405.05136 (2024)

    2024 arXiv

  14. [22]

    Qi Liu, Shuanghong Shen, Zhenya Huang, Enhong Chen, and Yonghe Zheng

  15. [23]

    Himanshu Pandey and Rajesh Kumar Pandey. 2019. Machine learning based approaches for personalized recommender system: A survey. Procedia computer science 165 (2019), 551–559

    2019

  16. [24]

    Radek Pelánek. 2017. Bayesian knowledge tracing, logistic models, and beyond: an overview of learner modeling techniques. User modeling and user-adapted interaction 27 (2017), 313–350

    2017

  17. [25]

    Chris Piech, Joe Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas Guibas, and Jascha Sohl-Dickstein. 2015. Deep knowledge tracing. Advances in neural information processing systems 28 (2015)

    2015

  18. [26]

    Francesco Ricci, Lior Rokach, Bracha Shapira, et al. 2011. Introduction to recom- mender systems handbook. Recommender systems handbook (2011), 1–35

    2011

  19. [27]

    Douglas Steinley. 2006. K-means clustering: a half-century synthesis. Brit. J. Math. Statist. Psych. 59, 1 (2006), 1–34

    2006

  20. [28]

    Zejie Tiana, Guangcong Zhengc, Brendan Flanaganb, Jiazhi Mic, and Hiroaki Ogatab. 2021. BEKT: Deep Knowledge Tracing with Bidirectional Encoder Repre- sentations from Transformers. In Proceedings of the 29th International Conference on Computers in Education . RecSys 2024, Oct...

    2021

  21. [29]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)

    2017

  22. [30]

    Arnold POS Vermeeren, Effie Lai-Chong Law, Virpi Roto, Marianna Obrist, Jettie Hoonhout, and Kaisa Väänänen-Vainio-Mattila. 2010. User experience evaluation methods: current state and development needs. In Proceedings of the 6th Nordic conference on human-computer interaction:...

    2010

  23. [31]

    Jindi Wang, Ioannis Ivrissimtzis, Zhaoxing Li, and Lei Shi. 2024. Comparative Efficacy of 2D and 3D Virtual Reality Games in American Sign Language Learning. In The 31st IEEE Conference on Virtual Reality and 3D User Interfaces . Newcastle University

    2024

  24. [32]

    Jindi Wang, Ioannis Ivrissimtzis, Zhaoxing Li, and Lei Shi. 2024. Enhancing User Experience in Chinese Initial Text Conversations with Personalised AI- Powered Assistant. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (<conf-loc>, <city>Honol...

    2024

  25. [33]

    Jindi Wang, Ioannis Ivrissimtzis, Zhaoxing Li, and Lei Shi. 2024. Impact of Personalised AI Chat Assistant on Mediated Human-Human Textual Conversa- tions: Exploring Female-Male Differences. In Companion Proceedings of the 29th International Conference on Intelligent User Inte...

    2024

  26. [34]

    Jindi Wang, Ioannis Ivrissimtzis, Zhaoxing Li, Yunzhan Zhou, and Lei Shi. 2023. Exploring the Potential of Immersive Virtual Environments for Learning Ameri- can Sign Language. In European Conference on Technology Enhanced Learning . Springer, 459–474

    2023

  27. [35]

    Lixiang Yan, Lele Sha, Linxuan Zhao, Yuheng Li, Roberto Martinez-Maldonado, Guanliang Chen, Xinyu Li, Yueqiao Jin, and Dragan Gašević. 2024. Practical and ethical challenges of large language models in education: A systematic scoping review. British Journal of Educational Tech...

    2024

  28. [36]

    Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. 2019. Deep learning based recom- mender system: A survey and new perspectives. ACM computing surveys (CSUR) 52, 1 (2019), 1–38

    2019

  29. [2021]

    arXiv preprint arXiv:2105.15106 (2021)

    A survey of knowledge tracing. arXiv preprint arXiv:2105.15106 (2021)

    2021 arXiv

  30. [2022]

    arXiv preprint arXiv:2208.12615 (2022)

    MonaCoBERT: Monotonic attention based ConvBERT for Knowledge Tracing. arXiv preprint arXiv:2208.12615 (2022)

    2022 arXiv

  31. [2024]

    In International Conference on Intelligent Tutoring Systems

    Lbkt: a lstm bert-based knowledge tracing model for long-sequence data. In International Conference on Intelligent Tutoring Systems . Springer, 174–184

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.