Pith. sign in

REVIEW 3 major objections 4 minor 51 references

TutorGym: A Testbed for Evaluating AI Agents as Tutors and Students

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TutorGym embeds AI agents inside classroom-tested tutoring systems; initial results: LLMs label incorrect actions below chance and generate correct next steps only 52–70% of the time, yet as in-context learners they trace human-like…

desk verdict TutorGym is a genuinely useful testbed for ITS research, but the 'human-like learning curves' result is a qualitative artifact of the protocol, not an established finding. read the letter →

arxiv 2505.01563 v1 pith:RBRQS3LK submitted 2025-05-02 cs.AI

Daniel Weitekamp , Momin N. Siddiqui , Christopher J. MacLellan This is my paper · ORCID
classification cs.AI
keywords intelligenttutoringsystemsLLMevaluationsimulatedlearnersin-contextlearningstep-levelfeedbackcurvescognitivetutorsAItestbed
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces TutorGym, a standard interface that lets AI agents act as tutors or students inside existing intelligent tutoring systems that have been used in classrooms. The authors' point is that final-answer benchmarks hide what matters for tutoring: everything happens at the level of individual steps inside a scaffolded interface, and an AI tutor must recognize correct and incorrect actions in many intermediate states. On an initial benchmark across 223 tutor domains, the paper reports that four current LLMs are poor at this step-level task: none labels incorrect actions better than chance, and correct next-step demonstrations are right only about 52–70% of the time. As students, however, LLMs trained by in-context learning produce error-rate learning curves that are qualitatively similar to human students' curves. The contribution is a common evaluation platform that makes such comparisons routine across very different tutor systems.

What carries the argument

The central object is the TutorGym environment API: an agent implements act(state) and train(state, action, reward), while a Trainer mediates interaction with a tutor by checking each action against the tutor's own grading function, applying correct actions, and demonstrating bottom-out hints when the agent is stuck. Each action is an SAI triple (selection, action_type, input), and states are JSON representations of the live HTML interface. Two derived mechanisms carry the evaluations: completeness profiles (reachable states with all correct next actions, plus correct and incorrect student actions when data exists) for tutor grading, and in-context example accumulation for simulated learning, in which the agent's past state-action-reward experiences are packed into a rolling 50k-character prompt. The completeness profile is what turns a tutor's private expert model into a scored benchmark, and the Trainer is what lets the same environment serve tutor-evaluation, learner-evaluation, reinforcement-learning, and computational-model agents.

What would settle it

Use actual classroom transaction logs from the CTAT and OATutor domains to build completeness profiles from real student states and mistakes, then rerun the four-LLM grading evaluation; if any model labels incorrect actions above chance on those profiles, the below-chance tutorial result was an artifact of synthetic profiles, whereas if models still cannot, the paper's central negative finding is confirmed.

Watch

Extended reading notes

Core claim

TutorGym's central claim is that AI tutoring and simulated learning should be evaluated on the interactive, step-level behavior that real tutoring systems require, not on final answers. The paper builds an interface that wraps three tutor-building platforms—CTAT example-tracing tutors, Apprentice Tutors, and OATutors—into one API where an agent sees a JSON representation of the current tutor screen and returns actions as (selection, action type, input) triples. From this, TutorGym generates completeness profiles of reachable states with their correct next actions and, where student data exists, actual correct and incorrect student actions. On 223 domains, the paper finds that Sonnet-3.5, Haiku-3.5, GPT-4o, and DeepSeek-v2.5 are poor tutors by this measure: no model labels incorrect actions above 50%, and demo generation ranges from about 52% to 71% accuracy. The same setup used with in-context learning produces learning curves for Haiku-3.5 and GPT-4o that are qualitatively similar to average human error rates, which the paper presents as an initial demonstration that LLM-based simulated learners can now be compared with human data on a large scale.

Load-bearing premise

The tutoring evaluation assumes the completeness profiles—sampled solution paths plus incorrect actions generated by DeepSeek-v2.5—represent the real states and student mistakes that tutoring would encounter, so the reported accuracies stand or fall with how well those synthetic profiles approximate actual student behavior.

Editorial extensions

If this is right

  • Deployed LLM tutors should not be expected to serve as standalone expert models; step-level grading and demo generation need guardrails or integration with existing tutor models.
  • Step-level metrics—correct-action labeling, incorrect-action labeling, and next-step demo accuracy—become standard complements to final-answer benchmarks in AIED.
  • In-context learning gives a practical baseline for LLM simulated learners that can be directly compared with human learning curves on hundreds of tutor domains.
  • Because TutorGym uses one interface for LLMs, reinforcement-learning agents, and computational models, cross-paradigm comparisons on identical tutor content become routine.
  • Evaluation cost is now a first-order constraint: the reported runs exceeded $730 in API fees, so scaling to more domains will require cheaper or local models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The synthetic incorrect actions generated by DeepSeek-v2.5 may be more or less detectable than real student errors; the below-chance grading result should be re-checked on completeness profiles built from actual student transactions in CTAT and OATutor domains.
  • The human-like learning curves may come less from acquiring math knowledge than from the prompt accumulating where and in what order to enter answers; running the learner with shuffled problem order or shuffled in-context examples would test this.
  • The 50k-character rolling context window drops the oldest experiences, so the learning curves could partly reflect recency bias; varying window size or splitting training from test would separate genuine improvement from prompt recency.
  • TutorGym's design makes possible a new kind of audit: replaying full classroom transaction logs to probe whether any future LLM tutor matches the step-by-step support students actually received.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. TutorGym is a Python testbed that exposes three intelligent tutoring system (ITS) platforms—CTAT example-tracing tutors, Apprentice Tutors, and OATutors—through a common interface for AI agents. The paper describes the API, the generation of 'completeness profiles' for evaluating tutoring behavior, and an initial evaluation in which four LLMs (Sonnet-3.5, Haiku-3.5, GPT-4o, DeepSeek-v2.5) are asked to label correct/incorrect next actions and to produce next-step demos across 223 domains. A second evaluation runs Haiku-3.5 and GPT-4o as in-context-learning simulated students on Apprentice Tutors and compares their first-attempt error rates with human learning curves from a 192-learner dataset. The authors report that the LLMs are poor tutors (no model exceeded chance at labeling incorrect actions; demo accuracy 52–70%) but produce qualitatively human-like learning curves as simulated students.

Significance. If the infrastructure is released with the described functionality, TutorGym fills a genuine gap: it provides a standardized, extensible interface for evaluating LLMs, RL agents, and symbolic computational models of learning inside classroom-tested ITSs, and it is backed by a nontrivial suite of 223 domains across three paradigms. The design decisions (SAI action format, Trainer-mediated interaction, completeness profiles, prompt engineering details) are clearly described and appear reproducible from the public repository. The tutoring evaluation is a useful demonstration, though its empirical claims are preliminary. The learning-curve result is interesting but, as argued below, is not yet established; the authors' own Section 4.3 candidly notes that the in-context examples may only convey answer-entry location and order. Overall the testbed contribution is valuable and the paper is well positioned to become a community benchmark, provided the empirical claims are either strengthened or appropriately qualified.

major comments (3)
  1. [§4.1–4.3, Fig. 3] The claim that LLMs 'produce remarkably human-like learning curves' is not supported by the reported evidence. The comparison in Figure 3 is qualitative only: no error bars, confidence intervals, or per-domain variance are shown, and no null baseline or quantitative curve-distance metric is reported. More importantly, the training protocol itself is data-dependent and demo-assisted: agents are trained on the maximum number of problems solved by more than 4 of 192 learners (3–10 per domain), and the Trainer provides a correct demo whenever the agent produces no action. With the 50k-character truncation retaining roughly 20–30 recent examples and short training sequences, the model's 'memory' is effectively a small nearest-neighbor store of recent state-action pairs. Any retrieval-based agent, not specifically an LLM, would show declining first-attempt error under this protocol. The authors' own Section 4.3 states that the in-context examples may primarily convey location and order of answer entry. I therefore regard the second headline result as unestablished. To support it, the authors should add (a) an ablation without demos or with random retrieval, (b) a comparison against a null model such as a TF-IDF or nearest-neighbor lookup over the same stored examples, and (c) quantitative curve-distance measures with confidence intervals.
  2. [§3.1, Table 2] The tutoring evaluation's ground truth is only partially based on student data. For CTAT and OATutor, completeness profiles are generated by sampling solution paths (5 problems per OATutor domain) and augmenting them with incorrect actions produced by DeepSeek-v2.5; only Apprentice Tutors use replay from a real student dataset. The reported correct/incorrect accuracy numbers therefore measure performance on synthetic profiles, and the abstract's 'none did better than chance at labeling incorrect actions' is only as strong as the assumption that these profiles represent the state space and error distribution encountered by real students. The paper should report, at minimum, the number of states per domain, the fraction of states derived from student logs versus synthetic sampling, and a sensitivity analysis (e.g., how accuracies change with more than 5 sampled problems per OATutor domain). Without this, the cross-platform comparison in Table 2 is difficult to interpret.
  3. [§3.2, Table 2] No variance or uncertainty is reported for any of the accuracy values in Table 2, and the correct/incorrect accuracy trade-off (e.g., Haiku-3.5 at 81.06% correct but 25.05% incorrect on CTAT) suggests strong response bias. 'Chance' is asserted to be 50%, which is only meaningful if the profile classes are balanced; the paper does not report the proportion of correct versus incorrect actions in the evaluation sets. At minimum, the authors should report class balance, per-domain accuracy ranges, and binomial confidence intervals or bootstrap intervals, and should run a significance test against the chance baseline (or against a majority-class classifier) before stating that 'none did better than chance.'
minor comments (4)
  1. [Abstract] The abstract contains a spacing typo in 'T utorGym'; also, the phrase 'remarkably human-like learning curves' is too strong given the limitations acknowledged in Section 4.3.
  2. [Figure 3] Figure 3 aggregates first-attempt error over all skills across 30 tutors; the figure would be more informative with per-domain curves, pointwise confidence bands, and labels indicating the number of human learners and agent runs contributing to each point.
  3. [§2.2] The symbolic interface is described as implementing act(state) and train(state, action, reward), but the text also refers to a get_state() endpoint; it would be clearer to define the complete agent interface explicitly, including how get_state() is invoked by the Trainer.
  4. [§5] The paper would benefit from an explicit artifact availability statement (e.g., a stable version/commit of the GitHub repository, a data release for the profiles, and prompt/config files) so that the evaluations can be reproduced exactly as run.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: TutorGym is an infrastructure and empirical-evaluation contribution, and neither headline result reduces to its inputs by construction.

full rationale

The paper's central contribution is a software interface plus two empirical evaluations, not a derivation. The tutoring evaluation compares LLM outputs against tutor-model ground truth and student data; the conclusion that LLMs are poor tutors is an empirical finding, and the paper does not fit any parameter to the evaluation data before reporting accuracies. The use of DeepSeek-v2.5 to generate incorrect actions for CTAT and OATutor profiles (Section 3.1) means DeepSeek is evaluated partly on its own generations, but this is a benchmark-representativeness concern rather than circularity: the paper's negative result is not entailed by that design, and DeepSeek scores below chance on incorrect-action labeling even on those items. The simulated-learner evaluation (Section 4) sets the number of training problems per domain from human data (3-10 problems), but the LLM error rates are not fitted to human learning-curve values; the comparison is qualitative, and the paper itself notes the mechanism may only convey answer-entry location and order (Section 4.3). The authors cite their own prior simulated-learner work for the 'near-zero-parameter' evaluation norm and for the claim that RL agents need many examples, but these citations are background framing and are not load-bearing: no uniqueness theorem or ansatz is imported to force the reported LLM learning curves. Because the empirical results are self-contained against external tutors, external LLM APIs, and a human student dataset, the paper is not circular; the main risks are validity and generalizability, not circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

TutorGym introduces software infrastructure, not a new physical or conceptual entity. The evaluation choices listed as free parameters are hand-chosen protocol settings rather than numbers fitted to data. The axioms are domain assumptions about the validity of tutor ground truth, profile coverage, skill mapping, and reference data; none is a new postulated entity.

free parameters (3)
  • OATutor profile sample size = 5 problems per domain
    Chosen by the authors to generate completeness profiles; a small sample may underrepresent the state space and affect the Table 2 accuracies.
  • Prompt length cutoff = 50k characters (approx. 15k tokens)
    Used to truncate in-context examples in the simulated learner evaluation; this affects which experiences remain available and therefore the shape of the learning curves.
  • Minimum human sample filter = more than 4 of 192 learners
    Used to set the number of training problems for simulated learners; a different threshold would change the comparison window.
assumptions (4)
  • domain assumption Tutor check(state, action) and get_all_demos are correct ground truth for step-level correctness.
    Used throughout Section 3; if the tutor models are wrong or incomplete, LLM grading and demo accuracies do not measure true tutoring skill.
  • domain assumption Synthetic completeness profiles approximate the distribution of real student states.
    Section 3.1: CTAT and OATutor profiles are sampled from tutor paths plus DeepSeek-generated incorrect actions rather than from student logs.
  • domain assumption PSLC DataShop skill mapping and learning-curve conventions are valid for comparing LLM and human errors.
    Section 4.2: each unique interface element is treated as one skill; if this conflates skills, the human and LLM curves are not measuring the same thing.
  • domain assumption The 192 human learners are a representative reference population.
    Section 4.1: human curves come from one Apprentice Tutor dataset, with no demographic or selection details provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TutorGym: A Testbed for Evaluating AI Agents as Tutors and Students." pith.science (2026). https://pith.science/paper/RBRQS3LK

@misc{pith2026250501563,
  author       = {Pith},
  title        = {Pith review of: TutorGym: A Testbed for Evaluating AI Agents as Tutors and Students},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RBRQS3LK}},
  note         = {Machine review of arXiv:2505.01563}
}
read the original abstract

Recent improvements in large language model (LLM) performance on academic benchmarks, such as MATH and GSM8K, have emboldened their use as standalone tutors and as simulations of human learning. However, these new applications require more than evaluations of final solution generation. We introduce TutorGym to evaluate these applications more directly. TutorGym is a standard interface for testing artificial intelligence (AI) agents within existing intelligent tutoring systems (ITS) that have been tested and refined in classroom studies, including Cognitive Tutors (CTAT), Apprentice Tutors, and OATutors. TutorGym is more than a simple problem-solution benchmark, it situates AI agents within the interactive interfaces of existing ITSs. At each step of problem-solving, AI agents are asked what they would do as a tutor or as a learner. As tutors, AI agents are prompted to provide tutoring support -- such as generating examples, hints, and step-level correctness feedback -- which can be evaluated directly against the adaptive step-by-step support provided by existing ITSs. As students, agents directly learn from ITS instruction, and their mistakes and learning trajectories can be compared to student data. TutorGym establishes a common framework for training and evaluating diverse AI agents, including LLMs, computational models of learning, and reinforcement learning agents, within a growing suite of learning environments. Currently, TutorGym includes 223 different tutor domains. In an initial evaluation, we find that current LLMs are poor at tutoring -- none did better than chance at labeling incorrect actions, and next-step actions were correct only ~52-70% of the time -- but they could produce remarkably human-like learning curves when trained as students with in-context learning.

Figures

Figures reproduced from arXiv: 2505.01563 by the authors.

Figure 1
Figure 1. An Apprentice Tutor (left) and a CTAT tutor (right). instead of only at the end of full problem-solving attempts [43]. The tutor model tracks students’ step-by-step answers and aligns their problem-solving progress with a number of acceptable strategies. There are several varieties of tutor mod￾els. Constraint-based models characterize correct solutions by the satisfaction of several independent requirements, that w… view at source ↗
Figure 2
Figure 2. The Trainer class training an AI agent with a tutor [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Learning curves from humans as well as Haiku-3.5 and GPT-4o agents with in-context learning solving problems in Apprentice Tutors. the prompt. The previously described tutoring evaluation prompt is essentially a zero-shot version of the prompt employed here. As the LLM agent experiences more problems, it accumulates an increasing number of in-context examples and the prompt size grows. Given the high cost of large p… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 40 canonical work pages

  1. [1]

    In: International Conference on Intelligent Tutoring Systems

    Aleven, V., McLaren, B.M., Sewall, J., Koedinger, K.R.: The cognitive tutor au- thoring tools (CTAT): Preliminary evaluation of efficiency gains. In: International Conference on Intelligent Tutoring Systems. pp. 61–70. Springer (2006)

    work page 2006

  2. [2]

    International Journal of Artificial Intelligence in Education 26(1), 224–269 (2016)

    Aleven, V., McLaren, B.M., Sewall, J., Van Velsen, M., Popescu, O., Demi, S., Ringenberg, M., Koedinger, K.R.: Example-tracing tutors: Intelligent tutor de- velopment for non-programmers. International Journal of Artificial Intelligence in Education 26(1), 224–269 (2016)

    work page 2016

  3. [3]

    Artificial intelligence42(1), 7–49 (1990)

    Anderson, J.R., Boyle, C.F., Corbett, A.T., Lewis, M.W.: Cognitive modeling and intelligent tutoring. Artificial intelligence42(1), 7–49 (1990)

    work page 1990

  4. [4]

    In: Advances in instructional Psychology, Volume 5, pp

    Anderson, J.R., Schunn, C.D.: Implications of the ACT-R learning theory: No magic bullets. In: Advances in instructional Psychology, Volume 5, pp. 1–33. Rout- ledge (2013)

    work page 2013

  5. [5]

    arXiv preprint arXiv:1606.01540 (2016)

    Brockman, G.: OpenAI Gym. arXiv preprint arXiv:1606.01540 (2016)

    arXiv 2016

  6. [6]

    In: Proceedings of the Eleventh ACM Conference on Learning@ Scale

    Calo, T., Maclellan, C.: Towards educator-driven tutor authoring: generative AI approaches for creating intelligent tutor interfaces. In: Proceedings of the Eleventh ACM Conference on Learning@ Scale. pp. 305–309 (2024)

    work page 2024

  7. [7]

    URL https://arxiv

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al.: Training verifiers to solve math word problems, 2021. URL https://arxiv. org/abs/2110.14168 (2021)

    arXiv 2021

  8. [8]

    arXiv preprint arXiv:2301.00234 (2022)

    Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Liu, T., et al.: A survey on in-context learning. arXiv preprint arXiv:2301.00234 (2022)

    arXiv 2022

Show all 51 references
  1. [9]

    Behavior Research Methods, Instruments, & Computers36, 180–192 (2004)

    Graesser, A.C., Lu, S., Jackson, G.T., Mitchell, H.H., Ventura, M., Olney, A., Louwerse, M.M.: AutoTutor: A tutor with dialogue in natural language. Behavior Research Methods, Instruments, & Computers36, 180–192 (2004)

    2004

  2. [10]

    In: Proceedings of the 26th International Conference on Artificial Intelligence in Education (AIED) (2025), https://doi.org/10.48550/arXiv.2503.16460

    Gupta, A., Reddig, J., Caló, T., Weitekamp, D., MacLellan, C.J.: Beyond final answers: Evaluating large language models for math tutoring. In: Proceedings of the 26th International Conference on Artificial Intelligence in Education (AIED) (2025), https://doi.org/10.48550/arXiv...

    2025 doi arXiv

  3. [11]

    arXiv preprint arXiv:2103.03874 (2021)

    Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., Steinhardt, J.: Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 (2021)

    2021 arXiv

  4. [12]

    International Journal of Artificial Intelligence in Education34(2), 545–585 (2024)

    Käser, T., Alexandron, G.: Simulated learners in educational technology: A sys- tematic literature review and a Turing-like test. International Journal of Artificial Intelligence in Education34(2), 545–585 (2024)

    2024

  5. [13]

    Preprint, Research Square (2024)

    Kestin, G., Miller, K., Klales, A., et al.: AI tutoring outperforms active learning. Preprint, Research Square (2024). https://doi.org/10.21203/rs.3. rs-4243877/v1

    2024 doi

  6. [14]

    Handbook of educational data mining43, 43–56 (2010)

    Koedinger,K.R.,Baker,R.S.,Cunningham,K.,Skogsholm,A.,Leber,B.,Stamper, J.: A data repository for the EDM community: The PSLC datashop. Handbook of educational data mining43, 43–56 (2010)

    2010

  7. [15]

    Cognitive Science36(5), 757–798 (2012)

    Koedinger, K.R., Corbett, A.T., Perfetti, C.: The knowledge-learning-instruction framework: Bridging the science-practice chasm to enhance robust student learn- ing. Cognitive Science36(5), 757–798 (2012)

    2012

  8. [16]

    In: AIED Workshops (2015)

    Koedinger, K.R., Matsuda, N., MacLellan, C.J., McLaughlin, E.A.: Methods for evaluating simulated learners: Examples from simstudent. In: AIED Workshops (2015)

    2015

  9. [17]

    Review of Educational Research86(1), 42–78 (2016) 14 D

    Kulik, J.A., Fletcher, J.: Effectiveness of intelligent tutoring systems: a meta- analytic review. Review of Educational Research86(1), 42–78 (2016) 14 D. Weitekamp et al

    2016

  10. [18]

    Journal of Educational Data Mining 9(1), 25–41 (2017)

    Liu, R., Koedinger, K.R.: Closing the loop: Automated data-driven cognitive model discoveries lead to improved instruction and learning gains. Journal of Educational Data Mining 9(1), 25–41 (2017)

    2017

  11. [19]

    In: Proceedings of the 47th Annual Meeting of the Cognitive Science Society

    MacLellan, C.J.: Model human learners: Computational models to guide instruc- tional design. In: Proceedings of the 47th Annual Meeting of the Cognitive Science Society. Cognitive Science Society, San Francisco, CA (2025)

    2025

  12. [20]

    International Educational Data Mining Society (2021)

    MacLellan, C.J., Gupta, A.: Learning expert models for educationally relevant tasks using reinforcement learning. International Educational Data Mining Society (2021)

    2021

  13. [21]

    In- ternational Educational Data Mining Society (2016)

    Maclellan, C.J., Harpstead, E., Patel, R., Koedinger, K.R.: The Apprentice Learner Architecture: Closing the loop between learning theory and educational data. In- ternational Educational Data Mining Society (2016)

    2016

  14. [22]

    International Journal of Artificial Intelligence in Education pp

    MacLellan, C.J., Koedinger, K.R.: Domain-general tutor authoring with apprentice learner models. International Journal of Artificial Intelligence in Education pp. 1– 42 (2020)

    2020

  15. [23]

    Advances in Cognitive Systems 10, 1–18 (2023)

    MacLellan, C.J., Stowers, K., Brady, L.: Evaluating alternative training interven- tions using personalized computational models of learning. Advances in Cognitive Systems 10, 1–18 (2023)

    2023

  16. [24]

    International Journal of Artificial Intelligence in Education25(1), 1–34 (2015)

    Matsuda, N., Cohen, W.W., Koedinger, K.R.: Teaching the teacher: Tutoring Sim- Student leads to more effective cognitive tutor authoring. International Journal of Artificial Intelligence in Education25(1), 1–34 (2015)

    2015

  17. [25]

    arXiv preprint arXiv:2410.05229 (2024)

    Mirzadeh, I., Alizadeh, K., Shahrokhi, H., Tuzel, O., Bengio, S., Farajtabar, M.: GSM-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05229 (2024)

    2024 arXiv

  18. [26]

    In: Advances in intelligent tutoring systems, pp

    Mitrovic, A.: Modeling domains and students with constraint-based modeling. In: Advances in intelligent tutoring systems, pp. 63–80. Springer (2010)

    2010

  19. [27]

    Artificial Intelligence Re- view 4(4), 251–277 (1990)

    Nwana, H.S.: Intelligent tutoring systems: an overview. Artificial Intelligence Re- view 4(4), 251–277 (1990)

    1990

  20. [28]

    In: Proceedings of the Eleventh ACM Conference on Learning@ Scale

    Pal Chowdhury, S., Zouhar, V., Sachan, M.: Autotutor meets large language mod- els: A language model tutor with rich pedagogy and guardrails. In: Proceedings of the Eleventh ACM Conference on Learning@ Scale. pp. 5–15 (2024)

    2024

  21. [29]

    PLOS One19(5), e0304013 (2024)

    Pardos, Z.A., Bhandari, S.: ChatGPT-generated help produces learning gains equivalent to human tutor-authored help on mathematics skills. PLOS One19(5), e0304013 (2024)

    2024

  22. [30]

    In: Proceedings of the 2023 CHI conference on human factors in computing systems

    Pardos, Z.A., Tang, M., Anastasopoulos, I., Sheel, S.K., Zhang, E.: OATutor: An open-source adaptive tutoring system and curated content library for learning sci- ences research. In: Proceedings of the 2023 CHI conference on human factors in computing systems. pp. 1–17 (2023)

    2023

  23. [31]

    In: International Conference on Artificial Intelligence in Education

    Rachatasumrit, N., Carvalho, P.F., Li, S., Koedinger, K.R.: Content matters: A computational investigation into the effectiveness of retrieval practice and worked examples. In: International Conference on Artificial Intelligence in Education. pp. 54–65. Springer (2023)

    2023

  24. [32]

    arXiv preprint arXiv:2404.18978 (2024)

    Radmehr, B., Singla, A., Käser, T.: Towards generalizable agents in text-based educational environments: A study of integrating rl with LLMs. arXiv preprint arXiv:2404.18978 (2024)

    2024 arXiv

  25. [33]

    In: Proceedings of the International Conference of the Learning Sciences (ICLS)

    Roll, I., Briseño, A., Yee, N., Welsh, A.: Not a magic bullet: The effect of scaf- folding on knowledge and attitudes in online simulations. In: Proceedings of the International Conference of the Learning Sciences (ICLS). pp. 879–886. Interna- tional Society of the Learning Sc...

    2014

  26. [34]

    In: International Conference on Artificial Intelligence in Education

    Schmucker, R., Xia, M., Azaria, A., Mitchell, T.: Ruffle&Riley: Insights from de- signing and evaluating a large language model-based conversational tutoring sys- tem. In: International Conference on Artificial Intelligence in Education. pp. 75–90. Springer (2024)

    2024

  27. [35]

    Self, J.: Computational mathetics: towards a science of learning systems design (1995)

    1995

  28. [36]

    In: Pro- ceedings of the Eleventh ACM Conference on Learning @ Scale

    Siddiqui, M.N., Gupta, A., Reddig, J.M., MacLellan, C.J.: HTN-based tutors: A new intelligent tutoring framework based on hierarchical task networks. In: Pro- ceedings of the Eleventh ACM Conference on Learning @ Scale. p. 491–495 (2024)

    2024

  29. [37]

    arXiv preprint arXiv:2404.07883 (2024)

    Smith, G., Gupta, A., MacLellan, C.: Apprentice tutor builder: A platform for users to create and personalize intelligent tutors. arXiv preprint arXiv:2404.07883 (2024)

    2024 arXiv

  30. [38]

    In: International Conference on Artificial Intelligence in Education

    Stamper, J., Xiao, R., Hou, X.: Enhancing LLM-based feedback: Insights from intelligent tutoring systems and the learning sciences. In: International Conference on Artificial Intelligence in Education. pp. 32–43. Springer (2024)

    2024

  31. [39]

    Journal of Artificial Intelligence in Education6, 405–437 (1995)

    Ur, S., VanLehn, K.: STEPS: a simulated, tutorable physics student. Journal of Artificial Intelligence in Education6, 405–437 (1995)

    1995

  32. [40]

    Artificial Intelligence31(1), 1–40 (1987)

    VanLehn, K.: Learning one subprocedure per lesson. Artificial Intelligence31(1), 1–40 (1987)

    1987

  33. [41]

    MIT press (1990)

    VanLehn, K.: Mind bugs: The origins of procedural misconceptions. MIT press (1990)

    1990

  34. [42]

    The Journal of the Learning Sciences8(1), 71–125 (1999)

    VanLehn, K.: Rule-learning events in the acquisition of a complex skill: An evalu- ation of cascade. The Journal of the Learning Sciences8(1), 71–125 (1999)

    1999

  35. [43]

    International Journal of Artificial Intelligence in Education16(3), 227–265 (2006)

    VanLehn, K.: The behavior of tutoring systems. International Journal of Artificial Intelligence in Education16(3), 227–265 (2006)

    2006

  36. [44]

    Educational Psychologist46(4), 197–221 (2011)

    VanLehn, K.: The relative effectiveness of human tutoring, intelligent tutoring sys- tems, and other tutoring systems. Educational Psychologist46(4), 197–221 (2011)

    2011

  37. [45]

    Journal of Artificial Intelligence in Education5, 135–135 (1994)

    VanLehn, K., Ohlsson, S., Nason, R.: Applications of simulated students: An ex- ploration. Journal of Artificial Intelligence in Education5, 135–135 (1994)

    1994

  38. [46]

    CHI (2020)

    Weitekamp, D., Harpstead, E., Koedinger, K.: An interaction design for machine teaching to develop AI tutors. CHI (2020)

    2020

  39. [47]

    arXiv preprint arXiv:2411.17924 (2024)

    Weitekamp, D., Harpstead, E., Koedinger, K.: AI2T: Building trustable AI tutors by interactively teaching a self-aware learning agent. arXiv preprint arXiv:2411.17924 (2024)

    2024 arXiv

  40. [48]

    In: International Conference on Artificial In- telligence in Education

    Weitekamp, D., Koedinger, K.: Computational models of learning: Deepening care and carefulness in AI in education. In: International Conference on Artificial In- telligence in Education. pp. 13–25. Springer (2023)

    2023

  41. [49]

    In: Proceedings of the 47th Annual Meeting of the Cognitive Science Society

    Weitekamp, D., MacLellan, C.: Decomposed inductive procedure learning: Learn- ing academic tasks with human-like data efficiency. In: Proceedings of the 47th Annual Meeting of the Cognitive Science Society. Cognitive Science Society, San Francisco, CA (2025)

    2025

  42. [50]

    In: Inter- national Conference on Artificial Intelligence in Education

    Weitekamp, D., Ye, Z., Rachatasumrit, N., Harpstead, E., Koedinger, K.: Inves- tigating differential error types between human and simulated learners. In: Inter- national Conference on Artificial Intelligence in Education. pp. 586–597. Springer (2020)

    2020

  43. [51]

    International Educational Data Mining Society (2019)

    Weitekamp III, D., Harpstead, E., MacLellan, C.J., Rachatasumrit, N., Koedinger, K.R.: Toward near zero-parameter prediction using a computational model of stu- dent learning. International Educational Data Mining Society (2019)

    2019

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.