{"id":"c2b573bd-7bda-4cec-a768-47caa7a49415","arxiv_id":"2608.10492","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A simulator that generates a reconstructed internal dialogue before each code edit matches real student code more closely and reaches 57.9% on a reasoning-alignment metric.","lead":"This paper trains AI student simulators to first produce an internal 'think-aloud' dialogue that explains each code edit, then generate the code itself. The approach improves how closely simulated students match real student behavior and thinking, which matters for testing tutoring systems before they reach classrooms.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'reasoning alignment' metric validates consistency with observed code edits, not fidelity to students' latent cognition; the central 'thinks like them' claim rests on this unvalidated metric.","rationale":"The reader's weakest assumption identifies exactly the load-bearing gap: retrospective teacher-generated traces are used both as supervision and as the basis for the alignment metric, so the metric cannot distinguish student-like latent reasoning from teacher rationalizations. This is not a peripheral caveat: the paper explicitly motivates INSIDE by the claim that 'two students may submit identical submissions for entirely different reasons' and lists 'reconstruction of pedagogically grounded reasoning traces' as a contribution. If the alignment metric only measures edit-consistency, the central claim that INSIDE models the student's mind is unverified. The action-fidelity results on test_OP are a genuine, independently interpretable positive result, though the abstract's blanket statement overstates the improvement because INSIDE and SFT are comparable on test_NP. The paper is transparent about its limitations and includes useful qualitative failure-mode analysis, which supports a conditional rather than dismissive verdict. My read does not change the reader's verdict; it reinforces the condition: the latent-cognition claim should be re-scoped or validated against human verbalized reasoning before being accepted as established.","tokens_in":21957,"tokens_out":3839,"duration_ms":37119,"concrete_test":"Run a think-aloud study with roughly 30 students from the same course solving two unseen problems with the AI tutor, asking them to verbalize their thoughts immediately before each submission. Segment the transcripts into atomic claims; for each transition, generate INSIDE and BloomCoT internal dialogues conditioned on the same history and next code; compute alignment between generated dialogue claims and transcript claims using the paper's judge or manual annotation, replacing ground-truth code-diff support with human transcript support. If INSIDE does not significantly beat BloomCoT or a generic 'novice rationalization' control on this human-reasoning alignment, the central claim that INSIDE captures latent student reasoning is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central contribution is that INSIDE models latent student reasoning, but Section 4.2's alignment metric only tests whether the generated internal dialogue is consistent with the student's observed code diff. The judge (GPT-5-mini) is given the ground-truth transition c_{t-1} -> c_t and asked whether each claim in the generated think trace is reflected in that diff. High alignment therefore measures post-hoc explanatory fit to the action, not agreement with the student's actual unobserved cognition. Section 3.3's training traces are retrospective reconstructions by GPT-5 conditioned on the same ground-truth next submission, so training and evaluation share the same rationalization bias: the model is rewarded for producing teacher-style explanations of code edits. Section 6 concedes these traces may be 'more coherent or structured reasoning than what real students exhibit.' The n=209 judge validation (95.2% on teacher traces) and the n=25 human-agreement check (88%, kappa=0.754) validate only that the judge reliably detects whether claims are reflected in diffs; they do not establish that the traces match any real student's thinking. Appendix C sharpens the issue: prompting models achieve 86.9-99.0% self-consistency while having poor action fidelity, confirming that consistency with code changes is not evidence of student-like reasoning. Without human ground truth for internal reasoning, the reported 'reasoning alignment' (up to 57.9%) supports only the weaker claim that INSIDE generates coherent rationalizations aligned with observed edits.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces INTERNAL STUDENT DIALOGUE (INSIDE), a framework for LLM-based student simulation in which a base model is fine-tuned to first generate a first-person 'internal dialogue' (a think trace grounded in Bloom's Taxonomy) and then produce the next code submission. The training traces are produced by a teacher LLM (GPT-5) that performs retrospective inference conditioned on the student's prior submissions, tutor feedback, and the ground-truth next submission. The authors evaluate on two dimensions: action fidelity, measured by Wasserstein distances between model-generated and real student code on pass rate, LOC, AST depth/width, and PEP 8 violations; and reasoning alignment, measured by an LLM judge (GPT-5-mini) that checks whether each claim in the generated dialogue is reflected in the ground-truth code diff. The paper reports that INSIDE achieves the lowest Wasserstein distances on test_OP, comparable performance to SFT on test_NP, and the highest alignment scores (up to 57.9%).","tokens_in":22362,"tokens_out":6216,"duration_ms":53493,"significance":"The problem is timely and the dataset is a genuine asset: a semester-based train/test split, careful IRB handling, a released codebase, and a candid limitations section are all strengths. If the reasoning-alignment metric were validated against actual student cognition, the framework would be an important step toward student simulators that produce interpretable reasoning alongside behavior. As it stands, the contribution is better characterized as improving the explanatory coherence of simulated student behavior with observed code edits, rather than establishing fidelity to latent reasoning. The qualitative failure-mode analysis in Appendix H is a useful addition and partly offsets the lack of a stronger quantitative validation of the reasoning component.","major_comments":[{"comment":"The abstract's claim that INSIDE 'improves simulation fidelity in both action fidelity' is not supported on test_NP. In Table 1b, INSIDE and SFT are effectively tied across metrics; for example, Qwen2.5-7B-INSIDE has pass-rate distance 0.04 versus 0.01 for Qwen2.5-7B-SFT, and Qwen3-8B-SFT is better on AST width (0.28 vs 0.40) and PEP 8 (0.12 vs 0.18). Additionally, the Wasserstein distances are reported as point estimates from 500 bootstrap resamples without confidence intervals or significance tests, so even the test_OP improvements (e.g., pass rate 0.05 vs 0.14) may not be statistically reliable. Please report bootstrap CIs and pairwise significance tests, and qualify the action-fidelity claim by split.","section":"§5.1, Table 1; Abstract"},{"comment":"The 'reasoning alignment' metric cannot by itself support the central claim that INSIDE models students' latent cognition. Section 3.3 generates training traces by conditioning the teacher (GPT-5) on the ground-truth next submission, and Section 4.2 evaluates alignment with an LLM judge (GPT-5-mini) from the same model family against the same ground-truth diff; high alignment therefore measures post-hoc explanatory fit to observed code edits, not agreement with unobserved student reasoning. The n=25 human agreement check (88%, κ=0.754) validates only whether claims are reflected in diffs, and the n=209 teacher-trace check does the same at a larger scale. Appendix C's self-consistency results (86.9–99.0% for prompting models with poor action fidelity) further show that consistency with code changes is not evidence of student-like reasoning. The paper's own Section 6 concedes that traces 'may reflect more coherent or structured reasoning than what real students exhibit.' At minimum, the abstract and conclusion should be reframed, and ideally the alignment metric should be validated against think-aloud or retrospective verbalization data.","section":"§4.2, §3.3, Appendix C"},{"comment":"The reported alignment differences are often within sampling error. On test_NP, Qwen3-8B-INSIDE scores 57.9% (SE 2.0) versus 56.0% (SE 2.3) for the best BloomCoT baseline; on test_OP, 51.8% (0.8) versus 50.5% (0.8). Given the standard errors, these gaps are not clearly significant, and no significance tests or effect sizes are reported. The claim of 'the highest alignment across models' therefore needs paired statistical testing or a more modest statement.","section":"§5.2, Table 2"}],"minor_comments":[{"comment":"The abstract should be qualified: on test_NP, INSIDE is comparable to SFT, so 'improves simulation fidelity in both action fidelity' is too strong.","section":"Abstract"},{"comment":"The caption says 'Bolded and underlined values' but does not explain what underlining indicates; specify the role of underlining or remove it.","section":"Table 1"},{"comment":"The displayed formula for Alignment_t is typeset incorrectly (the summation is missing an index); please fix the notation.","section":"§4.2"},{"comment":"Please report the inference protocol: sampling temperature, number of generations per instance, seed, and how a single 'next-attempt' is drawn; this is needed to interpret the Wasserstein estimates and MAE values.","section":"§5 / §4.1"},{"comment":"The alignment procedure for submission steps (final submissions mapped to a shared terminal step) should be described in the caption or text; otherwise the comparison of trajectories of different lengths is not well defined.","section":"Figure 2"},{"comment":"State how many students were excluded for non-consent and whether the AI tutor was offered to all students; this affects the generalizability of the training distribution.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"This is a potentially interesting paper, but the central claims currently outrun the evidence. The action-fidelity result is split-dependent, and the reasoning-alignment metric is not validated as a measure of latent student cognition. If the authors cannot obtain think-aloud or retrospective-verbalization ground truth, they should substantially reframe the contribution as modeling explanatory rationalizations rather than 'thinking like' students. I would be willing to review a revised version that addresses the statistical and validity concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The main thing to know: this paper introduces INSIDE, a fine-tuning framework that conditions code generation on teacher-reconstructed internal dialogue, and a claim-level alignment metric. On seen problems it improves action fidelity over SFT; on unseen problems it is comparable. The reasoning-alignment metric measures whether generated think traces are consistent with the student's code diffs, not whether they match real cognition.\n\nWhat is genuinely new and good: the teacher-reconstruction procedure is a legitimate extension beyond prior student simulators. The two-axis evaluation is sensible, and the paper is unusually honest about limitations—Section 6 concedes the traces may be more coherent than real student thinking. The qualitative examples are informative and show the behavior differences between prompting and fine-tuning. The dataset is real, with consent and IRB approval.\n\nWhere the soft spots are: the abstract overstates action fidelity improvements, because on test_NP INSIDE and SFT are statistically indistinguishable; the reported Wasserstein distances lack confidence intervals. The alignment metric is partially circular: the teacher (GPT-5) writes traces after seeing the ground-truth next submission, and the judge (GPT-5-mini) comes from the same family. The judge validation (n=209, n=25) confirms it reliably detects whether claims appear in diffs, not that traces reflect student cognition. Appendix C sharpens the problem: prompting models achieve high self-consistency with their own code but poor action fidelity, so consistency with code changes is not evidence of student-like reasoning. These points are mostly acknowledged in the paper, but the framing still leans on the stronger claim.\n\nNone of this kills the contribution. A simulator that produces coherent rationalizations aligned with observed edits is useful for evaluating tutoring systems and for generating plausible misconception hypotheses, even if it is not a faithful cognitive model. The paper just needs to be more careful about what the alignment metric can and cannot say.\n\nThis deserves a serious referee. I would send it to peer review with a request to revise the abstract, add significance tests or intervals, and reframe the alignment metric as consistency with edits rather than fidelity to latent thought. The target audience is educational AI and LLM-based human simulation researchers; they will get real value from the method, the baselines, and the analysis. I would bring it to a reading group and would cite it if I were working on student modeling.","headline":"A useful simulator framework that generates rationalizations aligned with code edits, but the 'thinks like them' claim overreaches the evidence.","tokens_in":22772,"tokens_out":1793,"would_cite":true,"duration_ms":17323,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By conditioning code generation on reconstructed internal dialogue, an LLM student simulator can match real students' code distributions and explain up to 57.9 percent of their edits.","keywords":["student simulation","LLM fine-tuning","internal dialogue","chain-of-thought","Bloom's Taxonomy","action fidelity","reasoning alignment","programming education"],"falsifier":"Collect in-the-moment think-aloud protocols from students solving the same programming problems, align each verbalization to the submission that follows, and score INSIDE's generated dialogues against those recordings with the same claim-level judge; if alignment is no better than a BloomCoT prompt baseline or near chance, the retrospective reconstruction of latent reasoning is not faithful.","tokens_in":21739,"feed_emoji":"🧠","tokens_out":5121,"duration_ms":44830,"temperature":0.7,"pith_summary":"The paper tries to establish that an LLM student simulator produces more faithful behavior when it is explicitly trained to generate the student's internal reasoning before generating the student's next action. It introduces INSIDE, which fine-tunes models on paired first-person think traces and code edits, with the traces reconstructed by a teacher model and structured along cognitive, affective, and action dimensions from Bloom's Taxonomy. Evaluated on thousands of real student submissions from an introductory programming course, INSIDE yields code distributions closer to real students than prompting or SFT baselines on familiar problems, and its internal dialogue matches the observed code edits up to 57.9 percent of the time. If true, student simulators could expose latent reasoning a tutor must address, not just reproduce surface behavior.","feed_headline":"Student simulators that think first match real student code","feed_subtitle":"Training on reconstructed internal dialogue beats prompting on both code fidelity and reasoning alignment.","key_machinery":"The load-bearing object is the internal dialogue (think trace): a first-person chain-of-thought, written in the student's voice, that states what the student intends before the next submission. In training, a teacher model reconstructs these traces retrospectively from the observed next code edit, organizing them into cognitive, affective, and action states drawn from Bloom's Taxonomy; in inference, the fine-tuned model generates the trace first and then generates code conditioned on it. The dialogue is what carries the claim: it turns an unobservable latent state into a supervised training signal, so action fidelity and reasoning alignment can be jointly optimized and evaluated.","core_discovery":"INSIDE's central claim is that conditioning code generation on an explicit internal-dialogue layer makes simulated students behave more like real students and, at the same time, produces a readable proxy of the invisible reasoning behind each submission. On the action side, fine-tuned INSIDE models achieve the lowest Wasserstein distances on test_OP across pass rate, code length, AST depth, AST width, and PEP 8 violations; on test_NP they remain comparable to SFT. On the reasoning side, their think traces achieve the highest alignment with ground-truth code edits, up to 57.9% for Qwen3-8B-INSIDE on test_NP, while larger prompt-only models such as GPT-5 score lower, suggesting stronger general reasoning does not by itself produce student-like reasoning.","pith_inferences":["A direct test of the reconstruction premise would pair INSIDE with think-aloud recordings: students verbalize while solving the same homework problems, and the reconstructed dialogues are scored against these recordings instead of against code diffs.","The same structure should transfer to other domains where an observable action hides a latent state, such as medical triage or customer-support diagnosis, by replacing Bloom's Taxonomy with a domain-specific set of state dimensions.","The alignment metric could be converted into a reward for reinforcement learning: rather than imitating teacher-written traces, the model could search for dialogues that explain the student's actual next edit, potentially closing the gap between the teacher's near-perfect 95.2% and INSIDE's roughly 58%.","The finding that larger, more capable models score lower on reasoning alignment suggests that student-like reasoning is a distinct target from expert reasoning; a practical extension is to deliberately select training data and objectives that preserve novice errors rather than optimizing toward correctness."],"forward_implications":["AI tutors can be evaluated on whether they resolve the misconceptions revealed by a simulated student's internal dialogue, not only on whether the simulated code eventually passes tests.","Counterfactual tutoring interventions become testable: change the feedback given to the simulator and inspect how the reconstructed reasoning and the next edit both shift.","On familiar problems, INSIDE removes much of the over-competence bias that makes prompted simulators start at artificially high pass rates; on new problems it matches the behavior-only SFT baseline, so adding reasoning costs little action fidelity.","Because INSIDE's internal dialogue and code are evaluated against the same ground-truth edit, high self-consistency alone (prompting models score up to 99%) no longer suffices; a model must also match what real students actually did."],"supporting_citations":[{"why":"Supplies Bloom's Taxonomy, the cognitive, affective, and action dimensions used to structure the reconstructed internal dialogue.","marker":"Bloom et al., 1956"},{"why":"Defines chain-of-thought reasoning, which INSIDE extends from expert-style reasoning to novice, possibly erroneous reasoning.","marker":"Wei et al., 2022"},{"why":"Supplies the action-fidelity metrics (pass rate, LOC, AST, PEP 8) and the SFT-on-student-trajectories baseline that INSIDE is compared against.","marker":"Miroyan et al., 2025"},{"why":"Provides GPT-5, the teacher model that retrospectively infers internal dialogue traces from observed student code edits.","marker":"OpenAI, 2025"},{"why":"Provides the LLM-based AI tutor feedback and the course interaction data that form the training and evaluation context.","marker":"Zamfirescu-Pereira et al., 2025"},{"why":"Closest prior work that reconstructs student errors post-hoc; the contrast motivates INSIDE's emphasis on generating reasoning before the action.","marker":"Ross & Andreas, 2025"}],"fun_headline_variants":["Modeling hidden reasoning improves student simulators","Think-first student simulators match real code","Internal dialogue makes simulated students more realistic","Student LLMs with inner thoughts beat prompting baselines","Simulated students that reason align with real code edits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's central premise is that the internal monologue a large language model writes after seeing the student's real next submission is a faithful stand-in for what the student actually thought before submitting.","fun_headline_variants_meta":{"raw":{"variants":["Modeling hidden reasoning improves student simulators","Think-first student simulators match real code","Internal dialogue makes simulated students more realistic","Student LLMs with inner thoughts beat prompting baselines","Simulated students that reason align with real code edits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000473,"raw_usage":{"total_tokens":2308,"prompt_tokens":861,"completion_tokens":1447,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":477,"completion_tokens_details":{"reasoning_tokens":1378}},"tokens_in":477,"tokens_out":1447,"duration_ms":13990,"temperature":1.0,"reasoning_tokens":1378,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:19:15.395028+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect in-the-moment think-aloud protocols from students solving the same programming problems, align each verbalization to the submission that follows, and score INSIDE's generated dialogues against those recordings with the same claim-level judge; if alignment is no better than a BloomCoT prompt baseline or near chance, the retrospective reconstruction of latent reasoning is not faithful.","supporting_citations":[{"cited_title":"voice in their head","cited_arxiv_id":null,"evidence_quote":"Supplies Bloom's Taxonomy, the cognitive, affective, and action dimensions used to structure the reconstructed internal dialogue."}],"review_version":1}