{"id":"727ec4fc-2479-4493-89d9-42a8dfc0ba1a","arxiv_id":"2412.15226","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Students who taught a ChatGPT agent to solve the Eight Queens puzzle learned more and wrote clearer pseudocode than a video-only control group, but showed no extra gain in code correctness.","lead":"Teaching ChatGPT to solve a programming puzzle instead of just watching a video led to bigger knowledge gains and clearer pseudocode in a 41-student experiment. The same students did not get better at finding and fixing bugs, a result that points to how AI agents should sometimes make mistakes to support learning.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The teachable-agent effect is confounded with AI code generation and task type; gains cannot yet be attributed to learning-by-teaching.","rationale":"The reader's weakest assumption identified exactly this confound: the experimental and control conditions differed not only in the teachable ChatGPT agent but also in the coding task, exposure to model solutions, interactivity, and debugging demands. The reader's conditional verdict already flags this as the central threat to causal interpretation. My stress-test confirms the concern is load-bearing: the paper's own data (submission counts) show behavioral differences consistent with AI-generated code reducing debugging, and the authors acknowledge the absence of conversation analysis, undermining the 'teaching' mechanism. The proposed three-arm experiment directly separates the ChatGPT interaction from the teaching role, and the conversation coding would establish whether the presumed LBT mechanism actually occurred. Because this is an addressable design limitation rather than an internal inconsistency, the conditional verdict stands: the paper can be considered a promising exploratory study, but its central claim requires stronger evidence. I therefore recommend no change to the reader's conditional verdict, while emphasizing that the concrete test is necessary before the causal claim should be accepted.","tokens_in":14963,"tokens_out":4346,"duration_ms":41066,"concrete_test":"Run a three-arm randomized experiment using the same videos, materials, and online judging platform: (A) teachable ChatGPT as in the current EG, (B) ChatGPT used as a direct coding assistant with no teaching role (participants ask ChatGPT to generate the solution and then submit it), and (C) control as in the current CG. If arm B performs as well as arm A on knowledge, pseudocode clearness/readability, and self-efficacy, then the benefit is due to AI-assisted code generation rather than the teachable-agent design; if arm A significantly outperforms arm B, the teaching role has incremental value. Additionally, record and code all arm A conversation logs to verify the presence of explanatory teaching moves (explanations, questions, feedback) and to test whether teaching quality predicts learning gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central causal claim—that interacting with a ChatGPT 'teachable agent' improves knowledge, programming clarity/readability, and SRL—is not identifiable from the design. In Section 3.2 and 3.3, EG participants guided ChatGPT to produce code for the eight-queens puzzle, while CG participants wrote code from scratch in a local editor. The two conditions therefore differ in multiple ways: natural-language interaction with an LLM, exposure to ChatGPT-generated model solutions, amount of hands-on debugging (EG averaged 1.95 submissions vs. 2.90 for CG, as reported in Section 5.2), and the specific instruction to adopt a teaching role. Any of these differences, not just the teachable-agent framing, could explain the outcome differences. The ANCOVA presented in Section 4.1 uses pre-test scores as a covariate, but covariance adjustment cannot control for unmeasured task-related confounds. Furthermore, the authors state in the Limitations section that conversation data were not collected or analyzed, so there is no direct evidence that EG participants actually engaged in explanatory teaching (e.g., giving explanations, asking questions, providing feedback) rather than simply issuing code-generation instructions. Without a control condition that isolates the teaching role from mere AI assistance, the observed effects cannot be attributed to learning-by-teaching with a teachable agent.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a quasi-experimental comparison of two instructional conditions for learning to solve the eight-queens problem in C++. Forty-one university students were assigned to an experimental group (EG) that interacted with a ChatGPT-based teachable agent and guided it to generate solution code, or to a control group (CG) that learned from online videos and wrote solution code from scratch. Pre- and post-tests measured knowledge, pseudo-code quality (clearness, correctness, readability), and self-regulated learning (test anxiety, self-efficacy, cognitive strategies). ANCOVA results are reported as showing significantly larger knowledge gains, higher clearness/readability scores, and higher self-efficacy/cognitive-strategy scores for EG, with no significant difference in correctness. The paper interprets these outcomes as evidence that ChatGPT can serve as an effective teachable agent for learning-by-teaching.","tokens_in":15149,"tokens_out":3476,"duration_ms":39350,"significance":"If the causal interpretation were justified, this would be a valuable, low-cost extension of the teachable-agent literature to LLM-based natural-language interaction. The study uses several sensible measurement tools, including an automated judging platform, an experienced-lecturer knowledge test, and two scorers with reported inter-rater agreement for pseudo-code, and it acknowledges an important limitation about missing conversation data. However, the design does not isolate the 'teaching' role from the many other differences between conditions, and the paper's central causal claims go beyond what the data can support. The contribution is better characterized as an exploratory comparison of two instructional conditions, with useful hypotheses for future work.","major_comments":[{"comment":"The intervention is confounded with AI-assisted code generation and task type. EG participants guided ChatGPT to produce code, while CG participants wrote code from scratch in a local editor. The conditions therefore differ in at least four respects: natural-language interaction with an LLM, exposure to model solutions, amount of hands-on debugging (EG averaged 1.95 submissions versus 2.90 for CG, as reported in §5.2), and the instruction to adopt a teaching role. ANCOVA with pre-test scores as a covariate cannot control for these unmeasured task-related differences. Consequently, the conclusion that 'learners who learning AI knowledge with teaching the ChatGPT teachable agent have better knowledge gains' (§4.1) is not identifiable from this design. The manuscript should either reframe the claims as a comparison of two instructional conditions or add a control condition that isolates the teaching role from mere AI assistance.","section":"§3.2 and §3.3"},{"comment":"The authors state that the study 'lacks of collecting and analysing students' conversation data on the teaching process.' Without such data, there is no direct evidence that EG participants actually engaged in explanatory teaching (e.g., giving explanations, asking questions, providing feedback) rather than simply issuing code-generation instructions. This is load-bearing for the learning-by-teaching mechanism and for the fidelity of the manipulation. The reported outcome differences cannot be attributed to teaching behavior unless process evidence or a manipulation check is provided.","section":"Limitations section"},{"comment":"The abstract claims that interacting with ChatGPT improves programming abilities 'particularly in writing readable and logically sound code,' and the conclusion repeats 'writing readable and logically sound code.' This overstates the results: §4.2 and Table 3 report no significant difference in code correctness (F = 2.98, p > 0.05, η² = 0.19), and the adjusted means are nearly identical. The claims should be restricted to clearness and readability, with correctness reported as non-significant.","section":"Abstract and Conclusion"},{"comment":"The manuscript does not describe the assignment procedure (randomization, matching, or self-selection) and does not state whether the two pseudo-code scorers were blind to condition. Non-blind scoring can bias the clearness and readability ratings, which are the main programming-skill findings. The authors should report the assignment method and the blinding status of the scorers, or explicitly acknowledge the risk of scoring bias and interpret the corresponding results with that caveat.","section":"§3.1 and §3.4.2"}],"minor_comments":[{"comment":"The third dimension is labeled 'Clearness' again, but based on the text and the scoring standard in Table 1 it should be 'Readability.'","section":"Table 3"},{"comment":"The sentence 'the three groups were significantly different' should refer to two groups.","section":"§4.1"},{"comment":"The significance stars for the clearness row are inconsistent with the text: the text reports F = 7.39, p < 0.01, but the table marks *** (p < 0.001). Please align the notation.","section":"Table 3"},{"comment":"Levene's test checks equality of variances, not differences in pre-test means; the sentence 'Levene's test ... revealed no significant difference in the pre-test scores between the two groups (F = 2.07, p > 0.05)' is therefore misleading and should be reworded.","section":"§4.3"},{"comment":"There are numerous language issues that should be corrected, for example 'learners who learning AI knowledge,' 'can be ruled towards generating fully correct code,' and 'This finding was echoed previous studies.' A careful proofreading pass is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and important question, and the authors are transparent about some limitations. My main concern is methodological: the causal language in the abstract, results, and conclusion exceeds what the confounded design can support. A revision that reframes the claims as exploratory, adds process evidence or a manipulation check, and clarifies assignment and scoring procedures would substantially strengthen the paper. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a small, honestly-written pilot on using ChatGPT as a teachable agent in programming education. The new bit is an empirical comparison with SRL measures on the eight-queens task, using a GPT-4 agent prompted to act as a help-seeking student. It extends work like GPTeach and 'Teach AI How to Code' rather than breaking new conceptual ground. Credit where due: the instruments are standard (MSLQ), the pseudocode scoring has good inter-rater reliability, and the authors state their limitations plainly, including that they did not collect or analyze conversation data. That last admission is more damaging than they seem to realize.\n\nThe soft spots are load-bearing. The experimental group guided ChatGPT to produce code; the control group wrote code from scratch in a local editor. So 'learning-by-teaching' is bundled with AI code generation, exposure to model solutions, interactivity, and less hands-on debugging (EG averaged 1.95 submissions vs 2.90 for CG). ANCOVA on pre-test scores cannot fix a confound of this kind. Without a control condition that isolates the teaching role—or conversation data showing students actually explained, asked, and gave feedback—the effects cannot be attributed to learning-by-teaching. The abstract overclaims by saying 'logically sound code' when correctness was not significant. Table 3 mislabels the second 'Clearness' row (it should be Readability) and the reported p-values don't match the asterisks. No blinding of pseudocode scorers is described, and randomization is not mentioned. These are all fixable, but together they mean the central causal claim is not yet supported.\n\nThe paper is worth a serious referee, not a desk reject. It is a legitimate pilot with a plausible direction and useful instrumentation, and the authors are transparent about many of its limits. A reviewer should push for a cleaner design—e.g., a third condition with an LLM assistant but no teaching role—and for open data and materials. I'd want that before believing the effect is specifically about teaching.\n\nBring to reading group? Maybe, as a case study in confounding in LLM education studies. I would not cite it in my own work yet.","headline":"A confounded but readable pilot study of ChatGPT as a teachable agent: the results are directionally plausible, but the design cannot separate teaching from AI assistance.","tokens_in":15704,"tokens_out":2370,"would_cite":false,"duration_ms":101469,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a ChatGPT-based teachable agent—an AI configured as a help-seeking student—improves programming learners' knowledge gains, pseudocode clarity and readability, and self-efficacy, while leaving code correctness…","keywords":["ChatGPT","teachable agent","programming education","self-regulated learning","learning by teaching","large language models","eight queens puzzle"],"falsifier":"Run the same eight-queens task in three conditions—teachable ChatGPT, ChatGPT as a non-teachable code-writing assistant, and video-only control—with pseudocode scored by raters blind to condition and the analysis pre-registered. If the assistant condition reproduces the teachable condition's gains, the student role is not what matters; if the teachable group still wins, the teaching interaction carries the effect.","tokens_in":14746,"feed_emoji":"🤖","tokens_out":13553,"duration_ms":124892,"temperature":0.7,"pith_summary":"This paper tries to establish that a ChatGPT-based teachable agent—an AI prompted to act as a curious, help-seeking student—can support learning-by-teaching in programming education. In a small experiment on the eight-queens backtracking problem, students who taught the ChatGPT agent scored higher on a knowledge test, wrote pseudocode rated higher in clearness and readability, and reported stronger self-efficacy and cognitive strategies than students who learned only from instructional videos and wrote code themselves. Code correctness did not improve, which the authors attribute to the agent generating correct code and so removing chances to practice debugging. If the result holds, ChatGPT offers an inexpensive replacement for hand-built teachable agents and a natural-language route to socialized learning-by-teaching.","feed_headline":"Teaching a ChatGPT student lifts coding learners' scores","feed_subtitle":"Students who taught a ChatGPT student outperformed video-only peers on knowledge, code clarity, and self-efficacy.","key_machinery":"The central object is the 'teachable ChatGPT agent': a GPT-4 session prompted to behave as a naive learner who asks for help in stages, using a five-stage help-seeking process model drawn from the learning sciences. Learners must converse with it in natural language to produce a working C++ solution to the eight-queens puzzle, which an online judging platform verifies. The mechanism is the teaching pressure itself—learners have to articulate the backtracking algorithm, break the task into parts, answer the agent's questions, and take responsibility for the final code. The agent's tendency to answer with correct code is the same machinery used to explain why error-correction skills did not improve: fewer bugs appeared, so learners had fewer opportunities to fix them.","core_discovery":"The central claim is that a ChatGPT model prompted to act as a teachable student can carry the learning-by-teaching effect in programming education. In the paper's pre/post experiment, students who taught the agent about the eight-queens puzzle had higher adjusted post-test scores on backtracking knowledge than the video-and-self-coding control group; their pseudocode was scored higher on clearness and readability but not correctness; and their self-reported self-efficacy and cognitive strategies improved, while test anxiety did not. The authors interpret these patterns as evidence that natural-language teaching produces deeper cognitive processing, that generating correct code on demand robs learners of debugging practice, and that guiding the agent requires planning and task decomposition, which are self-regulated learning strategies. They also chose not to suppress ChatGPT's occasional hallucinations, arguing that beginner-like mistakes give learners something to correct.","pith_inferences":["A testable extension the paper leaves open is a three-arm study: teachable ChatGPT, ChatGPT as a plain code-writing assistant, and video-only control, which would separate the student-role effect from pure exposure to generated code.","The error-correction mechanism could be tested directly by injecting intentional bugs into the agent's generated code; the paper's explanation predicts that learners would then gain debugging practice and improve on code correctness.","The SRL finding suggests a follow-up that scaffolds metacognitive prompts inside the teaching dialogue—asking learners to set goals and reflect before each teaching turn—and measures whether self-regulated learning gains grow further."],"forward_implications":["Natural-language teaching with ChatGPT can serve as a scalable, low-cost version of learning-by-teaching that does not require hand-built tutoring systems.","Courses using such agents should expect gains in knowledge and code clarity but plan separate debugging exercises, because correct code from the agent suppresses error-correction practice.","Deliberately letting the agent produce occasional wrong code—or using a weaker model—may turn the same setup into a debugging trainer.","The self-efficacy and cognitive-strategy gains imply that ChatGPT-based teaching can act as a lightweight intervention for self-regulated learning, not just a content tutor."],"supporting_citations":[{"why":"Introduces the protégé effect, the effort-increasing mechanism that the paper extends from human teachable agents to ChatGPT.","marker":"[1]"},{"why":"Establishes that learning-by-teaching can work without a live audience but depends on interaction, motivating the use of natural-language dialogue.","marker":"[12]"},{"why":"Documents ChatGPT's hallucination behavior and human-AI collaboration, the basis for the deliberate decision to keep agent errors in the experiment.","marker":"[16]"},{"why":"Demonstrates LLM-simulated student role-play, the technical precedent for using ChatGPT as a teachable agent.","marker":"[18]"},{"why":"Supplies the teachable-agent environment that links teaching to self-regulated learning, the basis for the study's SRL measures.","marker":"[27]"},{"why":"Shows that self-explanation during teaching predicts learning gains, the mechanism the paper invokes for knowledge improvements.","marker":"[42]"},{"why":"Provides evidence that LLM-based code generators can support novice programmers, grounding the choice of the eight-queens programming task.","marker":"[58]"},{"why":"Directly supports the claim that LLM-generated correct code reduces debugging opportunities, explaining the null result on error-correction skills.","marker":"[76]"}],"fun_headline_variants":["Teaching ChatGPT agent boosts coding gains, not debugging","ChatGPT as student: learn by teaching, get better code","Teaching a ChatGPT student boosts coding scores, self-efficacy","ChatGPT teachable agent: better programming, little debugging gain","Learn programming by teaching ChatGPT, gains but few bug fixes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The study assumes the only meaningful difference between groups was the teachable agent, yet the experimental group guided ChatGPT to generate code while the control group wrote code alone from scratch, so the reported gains could be due to model-generated code or extra interactivity rather than to teaching.","fun_headline_variants_meta":{"raw":{"variants":["Teaching ChatGPT agent boosts coding gains, not debugging","ChatGPT as student: learn by teaching, get better code","Teaching a ChatGPT student boosts coding scores, self-efficacy","ChatGPT teachable agent: better programming, little debugging gain","Learn programming by teaching ChatGPT, gains but few bug fixes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000301,"raw_usage":{"total_tokens":1715,"prompt_tokens":903,"completion_tokens":812,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":732}},"tokens_in":519,"tokens_out":812,"duration_ms":8864,"temperature":1.0,"reasoning_tokens":732,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:59:36.818454+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same eight-queens task in three conditions—teachable ChatGPT, ChatGPT as a non-teachable code-writing assistant, and video-only control—with pseudocode scored by raters blind to condition and the analysis pre-registered. If the assistant condition reproduces the teachable condition's gains, the student role is not what matters; if the teachable group still wins, the teaching interaction carries the effect.","supporting_citations":[{"cited_title":"Learning-by-teaching without audience presence or interaction: When and why does it work? Educational Psychology Review, 34(2):575–607, 2022","cited_arxiv_id":null,"evidence_quote":"Establishes that learning-by-teaching can work without a live audience but depends on interaction, motivating the use of natural-language dialogue."},{"cited_title":"Gpteach: Interactive ta training with gpt-based students","cited_arxiv_id":null,"evidence_quote":"Demonstrates LLM-simulated student role-play, the technical precedent for using ChatGPT as a teachable agent."},{"cited_title":"Learning by teaching: A new agent paradigm for educational software","cited_arxiv_id":null,"evidence_quote":"Supplies the teachable-agent environment that links teaching to self-regulated learning, the basis for the study's SRL measures."},{"cited_title":"Cognitive anatomy of tutor learning: Lessons learned with simstudent","cited_arxiv_id":null,"evidence_quote":"Shows that self-explanation during teaching predicts learning gains, the mechanism the paper invokes for knowledge improvements."},{"cited_title":"How novices use llm-based code generators to solve cs1 coding tasks in a self-paced learning environment","cited_arxiv_id":null,"evidence_quote":"Provides evidence that LLM-based code generators can support novice programmers, grounding the choice of the eight-queens programming task."},{"cited_title":"How to teach programming in the ai era? using llms as a teachable agent for debugging","cited_arxiv_id":null,"evidence_quote":"Directly supports the claim that LLM-generated correct code reduces debugging opportunities, explaining the null result on error-correction skills."}],"review_version":1}