{"id":"463c2756-4992-4c0f-b1d3-5980ba08ee13","arxiv_id":"2605.23933","paper_version":2,"verdict":"CONDITIONAL","confidence":"UNKNOWN","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":5,"one_line_summary":"KT4EQG combines a knowledge tracing model for concept selection with an alignment-trained LLM for question generation, achieving higher simulated exam scores than baseline generators on two educational datasets.","lead":"The paper proposes KT4EQG, a framework that uses a knowledge tracing model to select which knowledge concept a student should practice next, then trains an LLM to generate questions aligned with that concept. The approach is evaluated in simulation, showing improved exam scores over baseline question generators. A smart generalist might read this to understand how AI tutoring systems could personalize practice questions for individual learners.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"KT4EQG's exam score metric is nearly the same quantity its KC selection optimizes, both computed by KT2; under independent evaluators the advantage largely vanishes.","rationale":"The reader identified the correct load-bearing concern: the evaluation pipeline is circular because KT2 is used for KC selection, practice state updates, and exam scoring. I agree this is the central weakness.\n\nMy contribution is to make the concern more precise: it is not just that the same model is used throughout, but that the optimization objective (education value, Eq. 4) and the evaluation metric (exam score, §4.2) are both functions of KT2's posterior mastery probabilities, related through Eq. 11. The practice simulation updates these probabilities by assuming correct answers (A_t*=1), which directly increases the quantities that the education value was designed to maximize. This makes the improvement under KT2 evaluation close to tautological.\n\nThe Appendix D results already provide strong evidence that the concern lands: under BKT evaluation, KT4EQG wins 3/6 modules; under DKT, also 3/6 — far from the 6/6 sweep under KT2. The paper acknowledges reduced consistency but frames it as expected model differences. However, the fact that KT4EQG loses to simple baselines (e.g., Qwen3-8B without oracle) in several modules under independent evaluators suggests the KT2-evaluated results are substantially inflated by the circularity.\n\nThe proposed concrete test (fully independent pipeline for both practice and evaluation) would settle whether the concern fully lands. If KT4EQG still wins a majority of modules under a fully independent pipeline, the framework's value would be better supported; if not, the headline claim needs qualification.\n\nThe alignment training component (SFT + RL with alignment reward) is independently sound — the case studies in Table 6 and the ablation in Table 2 show genuine improvement in question-concept alignment. This is a real contribution independent of the circularity concern. But the headline claim about 'more effective questions' as measured by learning gains rests on the circular evaluation.\n\nThe reader's CONDITIONAL verdict is appropriate. The concern is real and load-bearing, but the paper has partial independent evidence (Appendix D) and the alignment component has standalone value. I recommend UNCHANGED because the reader already identified this as the key weakness and arrived at the right verdict.","tokens_in":16639,"tokens_out":4010,"duration_ms":159459,"concrete_test":"Run the full multi-round evaluation pipeline with BKT (or DKT) replacing KT2 for BOTH practice simulation state updates AND exam scoring — not just exam scoring as in Appendix D. Specifically: after each practice round, update the student's BKT/DKT state assuming correct answers, and compute the final exam score using the same BKT/DKT model. If KT4EQG's win rate drops below 4/6 modules under this fully independent pipeline, the circularity concern is confirmed and the headline 'consistently generates more effective questions' does not hold as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader correctly identifies the circularity, but the concern is more precise than 'KT2 is used everywhere.' The education value (Eq. 4) that KT4EQG maximizes is the sum of KT2 posterior mastery probabilities after a correct answer: Σ_c p(K_c=1 | H, ψ(X_t*)=c_t*, A_t*=1). The exam score (§4.2) is the average predicted correctness on exam questions, which from Eq. 11 equals p(K_c=0|H')·ε + p(K_c=1|H')·r_d — a direct function of the same posterior mastery probabilities. During multi-round practice, the state is updated by assuming A_t*=1 (correct answers), which increases exactly the mastery probabilities that the education value was designed to maximize. So KT4EQG selects KCs that maximally increase KT2's mastery posteriors, then is evaluated on KT2's mastery posteriors after those increases. Under KT2 evaluation, the improvement is close to tautological.\n\nAppendix D already provides partial evidence that this concern lands: when BKT or DKT (instead of KT2) compute the final exam score, KT4EQG wins only ~3/6 modules under each independent evaluator, compared to 6/6 under KT2. However, even Appendix D still uses KT2 for the practice simulation (state updates), so the student state after practice remains KT2-internal. A fully independent pipeline has never been tested.\n\nThe A_t*=1 assumption amplifies this: it gives maximum mastery boost to KCs selected for high education value (which are defined as KCs where a correct answer most increases mastery), differentially favoring KT4EQG over baselines that select KCs by other criteria.","agreement_with_reader":"agree"},"referee_report":{"model":"glm-5.2","summary":"The paper proposes KT4EQG, a framework for personalized educational question generation that integrates a knowledge tracing model (KT2) for concept selection with an alignment-trained LLM generator. The method decomposes the problem into (1) selecting the knowledge concept (KC) that maximizes a student's predicted mastery improvement (education value), and (2) generating a question faithfully aligned with that KC via supervised fine-tuning followed by RL with a contrastive alignment reward. Experiments on XES3G5M and MOOCRADAR compare KT4EQG against several LLM-based generators under a simulated multi-round practice evaluation, reporting exam scores computed by the KT model.","tokens_in":16928,"tokens_out":1393,"duration_ms":242010,"significance":"The integration of structured knowledge tracing with open-ended question generation is a reasonable and underexplored direction. The decomposition into education value and alignment value is principled, and the RL-based alignment training with a contrastive verifier is a concrete, reproducible design. The authors provide code, and the ablation study (Table 2) and case study (Table 6) provide useful qualitative evidence for alignment quality. However, the significance of the main empirical claim is substantially tempered by the evaluation methodology: the primary exam score metric is computed by the same KT2 model used for KC selection and state simulation, creating a self-referential evaluation loop that the paper does not fully resolve.","major_comments":[{"comment":"§4.2 and Eq. (4): The main evaluation metric (exam score) is computed by KT2, the same model used to select the optimal KC (Step 1, maximizing education value) and to update the student state during multi-round practice. The education value in Eq. (4) is defined as the sum of KT2 posterior mastery probabilities after a correct answer, and the exam score (Eq. 11) is a direct function of those same posteriors. This means KT4EQG selects the KC that maximally increases KT2's mastery posteriors, then is evaluated on KT2's mastery posteriors after those increases. Under this pipeline, the improvement is close to tautological. The paper acknowledges this partially in Appendix D by testing BKT and DKT as alternative evaluators, but Table 5 shows KT4EQG wins only approximately 3/6 modules under each independent evaluator (compared to 6/6 under KT2). Moreover, even in Appendix D, the practice-time","section":null},{"comment":"§3.1, Eq. (2): The assumption that A_t*=1 (students always answer correctly) is a strong idealization that amplifies the circularity concern. Under this assumption, every practice round deterministically increases mastery for the selected KC, which differentially favors KCs selected for high education value (defined as KCs where a correct answer most increases mastery). The paper justifies this as modeling exercises as learning opportunities, but no sensitivity analysis is provided for the case where A_t* is sampled from the model's predicted correctness probability. This would test whether the gains persist when students sometimes answer incorrectly, which is the realistic scenario.","section":null},{"comment":"§4.2, Table 1: The ORACLE baseline (which selects the KC maximizing education value but does not generate real questions) achieves exam scores very close to KT4EQG in several modules (e.g., Application: 0.8274 vs. 0.8036; Counting: 0.8164 vs. 0.8128). Since ORACLE does not involve actual question generation, its performance reflects only the KT2 selection mechanism evaluated within KT2. The small gap between ORACLE and KT4EQG under KT2 evaluation, combined with the larger gap under independent evaluators (Appendix D), suggests that much of KT4EQG's advantage in the main table may stem from the selection mechanism operating within KT2's internal dynamics rather than from the quality of generated questions.","section":null}],"minor_comments":[{"comment":"§4.1: The number of exam questions n is not specified in the main text. Please state this in the experimental setup.","section":null},{"comment":"§4.2: The number of practice rounds k used for the main results in Table 1 is not clearly stated. Figure 2 shows k=10,20,30, but Table 1 does not specify which k was used.","section":null},{"comment":"Table 5 (Appendix D): The text states KT4EQG 'remains competitive,' but in several modules KT4EQG is not the best method (e.g., Computation under DKT: 0.8077 vs. ScaleQuest 0.8685; Circuit Design under BKT: 0.6479 vs. PromptCoT 0.6818). The framing should be more precise about where the advantage holds and where it does not.","section":null},{"comment":"§4.5, Figure 3: The y-axis label 'Within-Student KC Mastery Rank' is somewhat ambiguous. Clarify whether rank 1 corresponds to the best-mastered or worst-mastered KC (the text says rank 1 = highest mastery, but this is counterintuitive for a figure showing adaptive difficulty).","section":null},{"comment":"Table 3: The Wine Knowledge module has only 4 candidate KCs and Education Theory has 3. Results on modules with so few KCs may not be very informative about the selection mechanism. Consider discussing this limitation.","section":null},{"comment":"§3.5, Eq. (7): The temperature τ is listed as 0.07 in Appendix C.1 but appears in the axiom ledger as a free parameter. Clarify whether τ was tuned or fixed a priori.","section":null}],"recommendation":"major_revision","confidential_remarks":"The circularity concern is the central issue. The paper is well-written and the method design is reasonable, but the main empirical claim ('consistently generates more effective questions') rests on an evaluation where the selector, simulator, and evaluator are the same model. Appendix D provides partial mitigation but the results there actually weaken the paper's claim rather than support it. The authors should either (a) run a fully independent pipeline (different KT model for practice simulation and evaluation), or (b) substantially reframe the claims to reflect what was actually demonstrated (alignment quality + KT2-internal selection optimality) versus what was not (generalization to real learning outcomes). Option (b) alone may suffice if the claims are appropriately scoped."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and insightful review. The central concern—self-referential evaluation—is valid and important. We agree to add a sensitivity analysis for the A_t*=1 assumption and to reframe the main claims to properly reflect the independent-evaluator results. We respectfully disagree that the framework lacks significance, and explain why below.","responses":[{"response":"We agree that the self-referential concern is legitimate and that the manuscript does not adequately address it in the main text. We will make the following revisions: (1) We will move the independent-evaluator results (BKT, DKT) from Appendix D into the main paper and discuss the circularity limitation explicitly. (2) We will reframe the main claims to state that KT4EQG achieves consistent improvements under the KT2 evaluator and remains competitive—though not uniformly best—under independent evaluators, rather than claiming universal superiority. (3) We will add a discussion of why the independent-evaluator results are noisier: BKT assumes independent binary states with no concept hierarchy, and DKT uses a flat sequential model, so neither captures the inter-KC dependencies that KT2's tree structure models and that our selection mechanism exploits. This is an architectural mismatch, not a defense of circularity, but it explains why the transfer is imperfect. We acknowledge that the current evidence does not fully resolve the circularity concern, and we will state this as a limitation.","revision_made":"yes","referee_comment":"§4.2 and Eq. (4): The main evaluation metric (exam score) is computed by KT2, the same model used for KC selection and state simulation, creating a self-referential evaluation loop. KT4EQG selects KCs that maximally increase KT2 posteriors, then is evaluated on those same posteriors. Appendix D shows KT4EQG wins only ~3/6 modules under BKT and DKT."},{"response":"This is a fair point. The A_t*=1 assumption was motivated by the pedagogical view that exercises are learning opportunities (students retry until mastery), but we agree that a sensitivity analysis is needed. We will add an experiment where A_t* is sampled from the KT model's predicted correctness probability p(A_t*=1|H, psi(X_t*)=c_t*) at each practice round, rather than being fixed to 1. This will test whether the gains persist when students sometimes answer incorrectly. We expect the absolute exam scores to decrease for all methods (since incorrect answers produce smaller mastery gains in KT2), but the relative ranking should be informative about whether our selection mechanism's advantage is robust to realistic correctness rates. We will report these results in the revised manuscript.","revision_made":"yes","referee_comment":"§3.1, Eq. (2): The assumption that A_t*=1 (students always answer correctly) is a strong idealization that amplifies circularity. No sensitivity analysis is provided for sampling A_t* from the model's predicted correctness probability."},{"response":"We partially agree with the referee's observation but note an important distinction. The ORACLE baseline assumes perfect concept alignment—it updates the student state using the intended KC without any question generation. The gap between ORACLE and KT4EQG therefore measures the cost of imperfect alignment: when a generated question does not faithfully test the intended KC, the verifier redirects the state update to a different KC, producing a suboptimal practice round. The fact that KT4EQG approaches ORACLE (e.g., 0.8036 vs. 0.8274 on Application) is actually evidence that our alignment-trained generator achieves high concept fidelity. The referee is correct that under independent evaluators (Appendix D), the gap pattern changes, which reinforces the need to present those results prominently. However, we respectfully note that the comparison between KT4EQG and the +ORACLE variants of baselines (e.g., QWEN3-8B+ORACLE: 0.7378 vs. KT4EQG: 0.8036) isolates the question generation quality component, since both receive the same KC selection. This gap persists because baseline generators produce questions that the verifier frequently maps to different KCs (35.3% concentration for QWEN3-8B+ORACLE vs. 5.2% for KT4EQG, as reported in Section 4.6). We will add a clearer discussion of what the ORACLE gap does and does not measure, and will present the +ORACLE comparisons more prominently as evidence for the alignment training contribution.","revision_made":"partial","referee_comment":"§4.2, Table 1: ORACLE achieves exam scores very close to KT4EQG, and since ORACLE does not involve question generation, its performance reflects only the KT2 selection mechanism. The small gap suggests much of KT4EQG's advantage stems from the selection mechanism within KT2's dynamics rather than question generation quality."}],"tokens_in":16536,"tokens_out":1111,"duration_ms":128830,"standing_objections":["The fundamental circularity concern cannot be fully resolved within the current framework: any KT model used for both selection and evaluation will share some internal dynamics. A truly independent evaluation would require real student studies or a ground-truth learning outcome measure external to all KT models, which is beyond the scope of this paper. We will acknowledge this as a limitation but cannot eliminate it in revision."]},"desk_editor":{"model":"glm-5.2","letter":"Here's my read on KT4EQG. The core idea — using a knowledge tracer to select which concept a student should practice next, then training an LLM to generate questions aligned to that concept — is a reasonable and genuinely new combination. The decomposition in Eq. 4 (education value × alignment) is clean. The alignment training pipeline (SFT + GRPO with a contrastive verifier) is the strongest part of the paper: the ablation in Table 2 shows clear progressive improvement, the case studies in Table 6 demonstrate real qualitative gains over baselines, and the diversity metric (most frequent verified KC at 5.2% vs 35.3% for the baseline) is convincing evidence that alignment training prevents mode collapse. Code is public. This component is independently sound and model-agnostic — it doesn't depend on KT2 at all. The adaptive KC selection analysis (Figure 3) also shows sensible behavior: weaker students get easier concepts, stronger students get harder ones. That's a nice result. Now the problem. The stress-test note is correct and the concern is worse than the reader's summary suggests. KT2 selects the KC by maximizing the sum of posterior mastery probabilities after a correct answer (Eq. 4). KT2 then updates the student state during multi-round practice assuming the student always answers correctly (A_t*=1). KT2 then computes the exam score, which from Eq. 11 is a direct function of those same posterior mastery probabilities. So the method selects KCs that maximally increase KT2's posteriors, assumes correct answers that maximally increase those posteriors, and is evaluated on those posteriors. Under KT2 evaluation, the 6/6 win rate is close to what you'd expect by construction. Appendix D provides partial evidence this lands: under BKT or DKT as evaluators, KT4EQG wins roughly 3/6 modules each — but even there, the practice simulation still uses KT2 for state updates, so the student state after practice remains KT2-internal. A fully independent pipeline has never been tested. The A_t*=1 assumption amplifies the circularity: it gives maximum mastery boost to the KCs selected for high education value, which by definition are the KCs where correct answers most increase mastery. This differentially favors KT4EQG over baselines that select KCs by other criteria. The alignment training and the clean formulation are real contributions. But the headline claim — 'consistently generates more effective questions' — rests on a self-referential loop. The paper needs either a real student study or an evaluation where KC selection, state updates, and exam scoring use independent models. This is for researchers working on personalized education and question generation. The alignment work alone is worth reading. It deserves a serious referee — the framework is interesting and the alignment component is sound — but the evaluation must be addressed before publication.","headline":"KT4EQG combines KT-guided concept selection with RL-trained question generation. The alignment training is solid and independently useful, but the headline evaluation is circular: KT2 selects KCs, updates student states, and computes exam scores, making the main result close to tautological.","tokens_in":17461,"tokens_out":1188,"would_cite":false,"duration_ms":46789,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Knowledge tracing guides LLM question generation for personalized learning","keywords":[],"falsifier":"Deploy KT4EQG in a real classroom with human students: if students who practice KT4EQG-generated questions do not show measurably greater learning gains on held-out assessments than students using questions from baseline generators, the framework's educational value claim would not hold.","tokens_in":16869,"feed_emoji":"🎓","tokens_out":915,"duration_ms":84671,"temperature":0.7,"pith_summary":"The paper proposes KT4EQG, a framework that couples a knowledge tracing model with an LLM-based question generator to produce personalized exercise questions. The knowledge tracer identifies which knowledge concept a student should practice next by maximizing predicted improvement in overall mastery across all concepts, accounting for correlations between them. An LLM generator is then trained via supervised fine-tuning followed by reinforcement learning to produce questions faithfully aligned with the selected concept. The central decomposition splits the problem into two parts: an education value term computed by the tracer, which measures how much a student's total mastery would improve if they practiced a given concept, and an alignment term, which verifies that the generated question actually tests that concept. Experiments on two datasets show the method outperforms baselines including LLMs given oracle concept selection, suggesting that alignment training, not just concept selection, drives question quality.","feed_headline":"Knowledge tracing guides LLM question generation for personalized learning","feed_subtitle":"A framework called KT4EQG pairs a student-state model with an alignment-trained generator, outperforming baselines in simulated exams.","key_machinery":"The framework rests on three components. First, a tree-structured knowledge tracer (KT2) models student mastery as hidden binary variables over a hierarchy of knowledge concepts, using transition probabilities between parent and child concepts and emission probabilities linking concept mastery to question correctness. The tracer computes an education value for each candidate concept by simulating the student's updated mastery state after successful practice, summed across all concepts. Second, an alignment verifier trained with contrastive learning maps questions and concepts into a shared embedding space to score how well a question matches its intended concept. Third, an LLM question发电机 is","core_discovery":"The paper's central claim is that decomposing personalized question generation into KT-guided concept selection plus alignment-trained question generation yields questions that produce greater predicted learning gains than existing methods, even when those methods are given the correct concept to target. The alignment training phase, using a contrastive verifier as reward signal, prevents the question generator from collapsing to generic questions regardless of the intended concept, a failure mode observed in baselines where one KC accounted for over 35 percent of generated questions.","pith_inferences":["The entire evaluation is simulated within the KT2 model: it selects concepts, updates student states during practice, and computes exam scores. If KT2's internal dynamics do not reflect real learning trajectories, the reported improvements could be artifacts of the model rather than genuine educational gains. A real-world test with human students would be needed to confirm the framework's educatio","The assumption that students always answer correctly (At*=1) idealizes practice as guaranteed mastery, which may overstate the education value of difficult concepts. In reality, students who fail on a challenging concept might benefit less than the model predicts.","The adaptive difficulty pattern shown in Figure 3, where weaker students are guided toward better-mastered concepts and stronger students toward challenging ones, emerges naturally from the objective rather than from an explicit difficulty heuristic, which is a non-trivial property of the formulation that could inform other adaptive learning system designs."],"forward_implications":["If the framework holds in real classrooms, it could enable automated tutoring systems that generate unlimited practice questions tailored to each student's current knowledge gaps, rather than relying on fixed question banks.","The decomposition into concept selection and question alignment is architecture-agnostic on both sides: any knowledge tracer could drive selection, and any text generator could serve as the question producer, making the approach adaptable to different domains and model scales.","The finding that alignment training prevents question collapse suggests that RL with a learned verifier reward may be broadly useful for controlling LLM output toward specific semantic targets beyond education."],"fun_headline_variants":["KT4EQG traces student knowledge to generate personalized practice questions","Contrastive alignment stops LLMs from generating generic exam questions","Knowledge tracing guides concept selection for personalized question generation","Decomposing question generation into tracing and alignment improves outcomes","KT4EQG pairs student-state models with aligned generators for targeted practice"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The entire evaluation pipeline runs inside one knowledge tracing model (KT2), which both selects concepts and judges whether learning improved. No real student data validates that the generated questions actually help learners, so the reported gains could partly reflect the model's internal preferences rather than genuine educational benefit.","fun_headline_variants_meta":{"raw":{"variants":["KT4EQG traces student knowledge to generate personalized practice questions","Contrastive alignment stops LLMs from generating generic exam questions","Knowledge tracing guides concept selection for personalized question generation","Decomposing question generation into tracing and alignment improves outcomes","KT4EQG pairs student-state models with aligned generators for targeted practice"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":993,"prompt_tokens":470,"completion_tokens":523,"prompt_tokens_details":null},"tokens_in":470,"tokens_out":523,"duration_ms":25520,"temperature":1.0,"reasoning_tokens":572,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-04T19:38:50.139434+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"Deploy KT4EQG in a real classroom with human students: if students who practice KT4EQG-generated questions do not show measurably greater learning gains on held-out assessments than students using questions from baseline generators, the framework's educational value claim would not hold.","supporting_citations":[],"review_version":1}