{"id":"441756a8-f4b7-4357-bf7e-1d682094a221","arxiv_id":"2505.01563","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"TutorGym connects LLMs, RL agents, and computational learning models to 223 existing tutoring domains, and initial evaluations show LLMs grade steps poorly but can imitate human learning curves with in-context learning.","lead":"TutorGym is a shared interface that lets AI agents act as tutors or students inside 223 existing intelligent tutoring systems that have been used in classrooms. In first tests, top LLMs were poor at grading student steps and generating next steps, but produced human-shaped learning curves when trained with in-context learning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The human-like learning-curve claim rests on a qualitative visual match to a data-dependent schedule; without a null baseline or quantitative curve-distance test, the second headline result is not established.","rationale":"TutorGym's infrastructure (223 domains, three paradigms, code links, trainer classes, and an RL wrapper) is a real contribution, and the paper is appropriately careful in several places: it discloses API costs, notes that CTAT/OATutor profiles are synthetic, distinguishes in-context 'learning' from genuine learning, and limits the learner evaluation to single-action generation. I therefore do not think rejection is warranted. The most load-bearing issue is the learner-evaluation headline, because the 'remarkably human-like learning curves' are offered as a substantive empirical result and are likely to be the most-cited claim. In Section 4.1, each domain is trained for only 3-10 problems, selected as the maximum number solved by more than 4 of 192 learners. This data-dependent choice could make any improving agent look aligned with the human curve. The LLM is handed a correct demo whenever it does not act, so a simple retrieval of recent state-action pairs could generate declining error rates. Section 4.3 itself hypothesizes that the in-context examples mainly teach location and order of answer entry rather than math skills, which raises the possibility that the similarity is mostly protocol-driven. No quantitative distance, null model, or error bars are reported. The tutoring evaluation also has a legitimate weakness: incorrect-action ground truth for CTAT and OATutor is DeepSeek-generated. However, the Apprentice results with real student data show the same qualitative pattern, so that weakness is less decisive than the learner-evaluation issue. I therefore recommend keeping the conditional verdict; the required test is a non-LLM memory baseline plus a quantitative curve-distance comparison.","tokens_in":11321,"tokens_out":7644,"duration_ms":87152,"concrete_test":"Run the Section 4.1 learner protocol with the LLM replaced by a non-parametric baseline that stores each (state, action) pair from previous correct demos and, at test, emits the stored action for the nearest matching state (e.g., exact interface-element match or Jaccard similarity over the state JSON). Also train the same LLM on a shuffled problem order and on a fixed 10-problem schedule independent of the human-data threshold. Then compute a quantitative distance (mean absolute error per problem or RMSE) between each agent's first-attempt error curve and the human curve, with bootstrap confidence intervals. If the baseline's curve is as close to the human curve as the LLM's, or if the LLM's distance is not significantly smaller than to a random-replay control, the 'human-like learning curves' claim should be withdrawn or substantially weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the simulated-learner evaluation in Section 4. The headline that LLMs produce 'remarkably human-like learning curves' rests on a visual comparison in Figure 3 and a deliberately data-dependent training schedule: per domain, agents train for the maximum number of problems solved by more than 4 of 192 human learners (3-10 problems), and the LLM receives a correct demo whenever it produces no action. With only 3-10 problems per domain, the 50k-character context cap is unlikely to evict earlier examples, so the in-context 'memory' can behave like a small nearest-neighbor store of recent state-action pairs. Any agent with such a store, not specifically an LLM, would show declining first-attempt error. No null model, no quantitative curve-distance metric, and no error bars are reported. The paper itself (Section 4.3) notes the mechanism may just be conveying location and order of answer entry. If the learning curves are an artifact of the demo-and-retrieve protocol, the second headline result fails, even though the testbed infrastructure would remain intact.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"TutorGym is a Python testbed that exposes three intelligent tutoring system (ITS) platforms—CTAT example-tracing tutors, Apprentice Tutors, and OATutors—through a common interface for AI agents. The paper describes the API, the generation of 'completeness profiles' for evaluating tutoring behavior, and an initial evaluation in which four LLMs (Sonnet-3.5, Haiku-3.5, GPT-4o, DeepSeek-v2.5) are asked to label correct/incorrect next actions and to produce next-step demos across 223 domains. A second evaluation runs Haiku-3.5 and GPT-4o as in-context-learning simulated students on Apprentice Tutors and compares their first-attempt error rates with human learning curves from a 192-learner dataset. The authors report that the LLMs are poor tutors (no model exceeded chance at labeling incorrect actions; demo accuracy 52–70%) but produce qualitatively human-like learning curves as simulated students.","tokens_in":11458,"tokens_out":5254,"duration_ms":49356,"significance":"If the infrastructure is released with the described functionality, TutorGym fills a genuine gap: it provides a standardized, extensible interface for evaluating LLMs, RL agents, and symbolic computational models of learning inside classroom-tested ITSs, and it is backed by a nontrivial suite of 223 domains across three paradigms. The design decisions (SAI action format, Trainer-mediated interaction, completeness profiles, prompt engineering details) are clearly described and appear reproducible from the public repository. The tutoring evaluation is a useful demonstration, though its empirical claims are preliminary. The learning-curve result is interesting but, as argued below, is not yet established; the authors' own Section 4.3 candidly notes that the in-context examples may only convey answer-entry location and order. Overall the testbed contribution is valuable and the paper is well positioned to become a community benchmark, provided the empirical claims are either strengthened or appropriately qualified.","major_comments":[{"comment":"The claim that LLMs 'produce remarkably human-like learning curves' is not supported by the reported evidence. The comparison in Figure 3 is qualitative only: no error bars, confidence intervals, or per-domain variance are shown, and no null baseline or quantitative curve-distance metric is reported. More importantly, the training protocol itself is data-dependent and demo-assisted: agents are trained on the maximum number of problems solved by more than 4 of 192 learners (3–10 per domain), and the Trainer provides a correct demo whenever the agent produces no action. With the 50k-character truncation retaining roughly 20–30 recent examples and short training sequences, the model's 'memory' is effectively a small nearest-neighbor store of recent state-action pairs. Any retrieval-based agent, not specifically an LLM, would show declining first-attempt error under this protocol. The authors' own Section 4.3 states that the in-context examples may primarily convey location and order of answer entry. I therefore regard the second headline result as unestablished. To support it, the authors should add (a) an ablation without demos or with random retrieval, (b) a comparison against a null model such as a TF-IDF or nearest-neighbor lookup over the same stored examples, and (c) quantitative curve-distance measures with confidence intervals.","section":"§4.1–4.3, Fig. 3"},{"comment":"The tutoring evaluation's ground truth is only partially based on student data. For CTAT and OATutor, completeness profiles are generated by sampling solution paths (5 problems per OATutor domain) and augmenting them with incorrect actions produced by DeepSeek-v2.5; only Apprentice Tutors use replay from a real student dataset. The reported correct/incorrect accuracy numbers therefore measure performance on synthetic profiles, and the abstract's 'none did better than chance at labeling incorrect actions' is only as strong as the assumption that these profiles represent the state space and error distribution encountered by real students. The paper should report, at minimum, the number of states per domain, the fraction of states derived from student logs versus synthetic sampling, and a sensitivity analysis (e.g., how accuracies change with more than 5 sampled problems per OATutor domain). Without this, the cross-platform comparison in Table 2 is difficult to interpret.","section":"§3.1, Table 2"},{"comment":"No variance or uncertainty is reported for any of the accuracy values in Table 2, and the correct/incorrect accuracy trade-off (e.g., Haiku-3.5 at 81.06% correct but 25.05% incorrect on CTAT) suggests strong response bias. 'Chance' is asserted to be 50%, which is only meaningful if the profile classes are balanced; the paper does not report the proportion of correct versus incorrect actions in the evaluation sets. At minimum, the authors should report class balance, per-domain accuracy ranges, and binomial confidence intervals or bootstrap intervals, and should run a significance test against the chance baseline (or against a majority-class classifier) before stating that 'none did better than chance.'","section":"§3.2, Table 2"}],"minor_comments":[{"comment":"The abstract contains a spacing typo in 'T utorGym'; also, the phrase 'remarkably human-like learning curves' is too strong given the limitations acknowledged in Section 4.3.","section":"Abstract"},{"comment":"Figure 3 aggregates first-attempt error over all skills across 30 tutors; the figure would be more informative with per-domain curves, pointwise confidence bands, and labels indicating the number of human learners and agent runs contributing to each point.","section":"Figure 3"},{"comment":"The symbolic interface is described as implementing act(state) and train(state, action, reward), but the text also refers to a get_state() endpoint; it would be clearer to define the complete agent interface explicitly, including how get_state() is invoked by the Trainer.","section":"§2.2"},{"comment":"The paper would benefit from an explicit artifact availability statement (e.g., a stable version/commit of the GitHub repository, a data release for the profiles, and prompt/config files) so that the evaluations can be reproduced exactly as run.","section":"§5"}],"recommendation":"major_revision","confidential_remarks":"The infrastructure contribution is strong and, I believe, publishable after revision. My main concern is that the in-context-learning learning-curve result is framed in the abstract as a headline finding despite the paper's own mechanisms discussion suggesting a retrieval artifact. Please weigh whether the authors should be asked to add the missing baselines or to substantively downgrade the claim in the abstract. The paper also sits at the boundary between a benchmark/tool paper and an empirical study; the requested revisions should make clear which parts are claims about the testbed and which are claims about LLM behavior."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the infrastructure is the real contribution: a unified Python interface over CTAT, Apprentice Tutors, and OATutors, with 223 domains and code on GitHub. That gives the field a common benchmark for evaluating LLM tutors and simulated learners, and it is genuinely new. Second, the more exciting headline—LLMs producing 'remarkably human-like learning curves'—does not hold up. The learning-curve comparison in Section 4 is visual, based on 3–10 problems per domain, and trained under the demo-and-retrieve protocol where the LLM sees the correct action whenever it produces none. With only 20–30 examples in context, the model can act as a small nearest-neighbor store. The authors even say the effect may just be conveying location and order of answer entry. No null baseline, no quantitative curve-distance metric, no error bars. The stress-test note lands.\n\nWhat the paper does well: the API design is sensible, the completeness-profile method for evaluating tutors is a workable way to handle open-ended states, and the tutoring results, while preliminary, are honestly reported—no LLM exceeds chance at labeling incorrect actions. The authors also deserve credit for flagging the mechanism behind the learning curves in Section 4.3, which is the kind of honesty you want in a benchmark paper.\n\nSoft spots, in proportion: the empirical claims are not as sharp as the paper's own language. Table 2 has no confidence intervals. CTAT and OATutor completeness profiles are synthetically generated (DeepSeek-produced incorrect actions), so the grading numbers should be treated as conditional on that synthetic coverage. The 50k-character prompt cap and the per-domain training schedule are reasonable choices, but they make the learning-curve result more of a sanity check than a finding.\n\nWho is this for? AIED researchers who need a shared testbed, and anyone building LLM tutors or simulated learners. It deserves a serious referee, but the revision should add error bars, a quantitative human-vs-LLM comparison, and a discussion of what the synthetic profiles do to the tutor numbers.","headline":"TutorGym is a genuinely useful testbed for ITS research, but the 'human-like learning curves' result is a qualitative artifact of the protocol, not an established finding.","tokens_in":12034,"tokens_out":1984,"would_cite":true,"duration_ms":19239,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TutorGym embeds AI agents inside classroom-tested tutoring systems; initial results: LLMs label incorrect actions below chance and generate correct next steps only 52–70% of the time, yet as in-context learners they trace human-like…","keywords":["intelligent tutoring systems","LLM evaluation","simulated learners","in-context learning","step-level feedback","learning curves","cognitive tutors","AI testbed"],"falsifier":"Use actual classroom transaction logs from the CTAT and OATutor domains to build completeness profiles from real student states and mistakes, then rerun the four-LLM grading evaluation; if any model labels incorrect actions above chance on those profiles, the below-chance tutorial result was an artifact of synthetic profiles, whereas if models still cannot, the paper's central negative finding is confirmed.","tokens_in":11074,"feed_emoji":"🎓","tokens_out":8686,"duration_ms":79544,"temperature":0.7,"pith_summary":"This paper introduces TutorGym, a standard interface that lets AI agents act as tutors or students inside existing intelligent tutoring systems that have been used in classrooms. The authors' point is that final-answer benchmarks hide what matters for tutoring: everything happens at the level of individual steps inside a scaffolded interface, and an AI tutor must recognize correct and incorrect actions in many intermediate states. On an initial benchmark across 223 tutor domains, the paper reports that four current LLMs are poor at this step-level task: none labels incorrect actions better than chance, and correct next-step demonstrations are right only about 52–70% of the time. As students, however, LLMs trained by in-context learning produce error-rate learning curves that are qualitatively similar to human students' curves. The contribution is a common evaluation platform that makes such comparisons routine across very different tutor systems.","feed_headline":"LLM tutors fail step grading, mimic humans as students","feed_subtitle":"TutorGym: LLMs grade wrong steps below chance, yet learn like humans with in-context examples","key_machinery":"The central object is the TutorGym environment API: an agent implements act(state) and train(state, action, reward), while a Trainer mediates interaction with a tutor by checking each action against the tutor's own grading function, applying correct actions, and demonstrating bottom-out hints when the agent is stuck. Each action is an SAI triple (selection, action_type, input), and states are JSON representations of the live HTML interface. Two derived mechanisms carry the evaluations: completeness profiles (reachable states with all correct next actions, plus correct and incorrect student actions when data exists) for tutor grading, and in-context example accumulation for simulated learning, in which the agent's past state-action-reward experiences are packed into a rolling 50k-character prompt. The completeness profile is what turns a tutor's private expert model into a scored benchmark, and the Trainer is what lets the same environment serve tutor-evaluation, learner-evaluation, reinforcement-learning, and computational-model agents.","core_discovery":"TutorGym's central claim is that AI tutoring and simulated learning should be evaluated on the interactive, step-level behavior that real tutoring systems require, not on final answers. The paper builds an interface that wraps three tutor-building platforms—CTAT example-tracing tutors, Apprentice Tutors, and OATutors—into one API where an agent sees a JSON representation of the current tutor screen and returns actions as (selection, action type, input) triples. From this, TutorGym generates completeness profiles of reachable states with their correct next actions and, where student data exists, actual correct and incorrect student actions. On 223 domains, the paper finds that Sonnet-3.5, Haiku-3.5, GPT-4o, and DeepSeek-v2.5 are poor tutors by this measure: no model labels incorrect actions above 50%, and demo generation ranges from about 52% to 71% accuracy. The same setup used with in-context learning produces learning curves for Haiku-3.5 and GPT-4o that are qualitatively similar to average human error rates, which the paper presents as an initial demonstration that LLM-based simulated learners can now be compared with human data on a large scale.","pith_inferences":["The synthetic incorrect actions generated by DeepSeek-v2.5 may be more or less detectable than real student errors; the below-chance grading result should be re-checked on completeness profiles built from actual student transactions in CTAT and OATutor domains.","The human-like learning curves may come less from acquiring math knowledge than from the prompt accumulating where and in what order to enter answers; running the learner with shuffled problem order or shuffled in-context examples would test this.","The 50k-character rolling context window drops the oldest experiences, so the learning curves could partly reflect recency bias; varying window size or splitting training from test would separate genuine improvement from prompt recency.","TutorGym's design makes possible a new kind of audit: replaying full classroom transaction logs to probe whether any future LLM tutor matches the step-by-step support students actually received."],"forward_implications":["Deployed LLM tutors should not be expected to serve as standalone expert models; step-level grading and demo generation need guardrails or integration with existing tutor models.","Step-level metrics—correct-action labeling, incorrect-action labeling, and next-step demo accuracy—become standard complements to final-answer benchmarks in AIED.","In-context learning gives a practical baseline for LLM simulated learners that can be directly compared with human learning curves on hundreds of tutor domains.","Because TutorGym uses one interface for LLMs, reinforcement-learning agents, and computational models, cross-paradigm comparisons on identical tutor content become routine.","Evaluation cost is now a first-order constraint: the reported runs exceeded $730 in API fees, so scaling to more domains will require cheaper or local models."],"supporting_citations":[{"why":"defines example-tracing CTAT tutors whose behavior graphs are the backbone of the CTAT domains in TutorGym","marker":"[2]"},{"why":"supplies the student transaction format and learning-curve conventions used to compare agents with human learners","marker":"[14]"},{"why":"represents the theory-first computational-model approach that LLM simulated learners are being compared against","marker":"[19]"},{"why":"documents the tens of thousands of examples reinforcement-learning agents need, the baseline for data efficiency in the learner evaluation","marker":"[20]"},{"why":"provides the Apprentice Learner architecture that learns inside tutor interfaces and motivates the simulated-learner design","marker":"[22]"},{"why":"shows final-answer benchmark scores drop under simple surface changes, motivating step-level evaluation rather than answer-only benchmarks","marker":"[25]"},{"why":"defines OATutors and their hint-and-scaffold problem pathways, one of the three tutor platforms integrated into TutorGym","marker":"[30]"},{"why":"introduces Apprentice Tutors, the hierarchical-rule platform used for both the tutoring evaluation and the simulated-learner study","marker":"[37]"},{"why":"supplies an interactive tutor-authoring agent that TutorGym could be used to evaluate as a future application","marker":"[47]"}],"fun_headline_variants":["LLMs flunk tutoring, ace imitation of student learning","TutorGym: LLMs grade wrong, learn right","LLMs can't tutor, but mimic human learning curves","Step-by-step: LLMs poor tutors, human-like students","LLMs fail as tutors, pass as simulated students"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The tutoring evaluation assumes the completeness profiles—sampled solution paths plus incorrect actions generated by DeepSeek-v2.5—represent the real states and student mistakes that tutoring would encounter, so the reported accuracies stand or fall with how well those synthetic profiles approximate actual student behavior.","fun_headline_variants_meta":{"raw":{"variants":["LLMs flunk tutoring, ace imitation of student learning","TutorGym: LLMs grade wrong, learn right","LLMs can't tutor, but mimic human learning curves","Step-by-step: LLMs poor tutors, human-like students","LLMs fail as tutors, pass as simulated students"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000214,"raw_usage":{"total_tokens":1506,"prompt_tokens":1106,"completion_tokens":400,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":722,"completion_tokens_details":{"reasoning_tokens":318}},"tokens_in":722,"tokens_out":400,"duration_ms":3843,"temperature":1.0,"reasoning_tokens":318,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:15:31.302590+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use actual classroom transaction logs from the CTAT and OATutor domains to build completeness profiles from real student states and mistakes, then rerun the four-LLM grading evaluation; if any model labels incorrect actions above chance on those profiles, the below-chance tutorial result was an artifact of synthetic profiles, whereas if models still cannot, the paper's central negative finding is confirmed.","supporting_citations":[{"cited_title":"International Journal of Artificial Intelligence in Education 26(1), 224–269 (2016)","cited_arxiv_id":null,"evidence_quote":"defines example-tracing CTAT tutors whose behavior graphs are the backbone of the CTAT domains in TutorGym"},{"cited_title":"Handbook of educational data mining43, 43–56 (2010)","cited_arxiv_id":null,"evidence_quote":"supplies the student transaction format and learning-curve conventions used to compare agents with human learners"},{"cited_title":"In: Proceedings of the 47th Annual Meeting of the Cognitive Science Society","cited_arxiv_id":null,"evidence_quote":"represents the theory-first computational-model approach that LLM simulated learners are being compared against"},{"cited_title":"International Educational Data Mining Society (2021)","cited_arxiv_id":null,"evidence_quote":"documents the tens of thousands of examples reinforcement-learning agents need, the baseline for data efficiency in the learner evaluation"},{"cited_title":"International Journal of Artificial Intelligence in Education pp","cited_arxiv_id":null,"evidence_quote":"provides the Apprentice Learner architecture that learns inside tutor interfaces and motivates the simulated-learner design"},{"cited_title":"In: Proceedings of the 2023 CHI conference on human factors in computing systems","cited_arxiv_id":null,"evidence_quote":"defines OATutors and their hint-and-scaffold problem pathways, one of the three tutor platforms integrated into TutorGym"}],"review_version":1}