{"id":"e333e9f5-c343-4988-bc5a-7b98a6aed081","arxiv_id":"2501.05220","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A contrastive mixed-context fine-tuning method lets a 60M-parameter T5 model generate topic-controlled educational questions, with best topical alignment from augmented data and a Jaccard Wikipedia metric.","lead":"Researchers fine-tuned a small T5 model on a contrastive dataset derived from SQuAD to generate questions about a specified topic from paragraph context. They evaluate topic alignment with Wikipedia-based semantic relatedness and show that data augmentation and quantization retain quality while shrinking the model.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Topic-control evaluation may be circular: topic definition and WikiSemRel metric both rely on the same WAT Wikification system, so surface mention of the topic entity could inflate the headline results.","rationale":"The reader's weakest assumption identified the small human validation of WikiSemRel and the sufficiency of the topic string as a control signal. I agree that the metric's validity is the crux, but I sharpen the concern: the topic is defined by the same WAT Wikification system used to compute the evaluation metric, creating a potential circularity. A model can inflate its WikiSemRel score by copying the topic entity into the question, and the human validation does not test whether the metric distinguishes shallow topic mention from educationally meaningful topic focus. This is a more specific mechanism than the reader's small-sample critique, hence 'partial' agreement. The paper is otherwise methodologically coherent: the contrastive dataset construction, the fine-tuning setup, and the quantisation experiments are clearly described. However, the central claim of superior topic alignment rests on a metric whose validity is not independently established for the actual generated outputs. This reinforces the reader's CONDITIONAL verdict rather than changing it; the concern is testable, and the paper should provide human evaluation of the generated questions or otherwise break the circular link between topic definition and evaluation. I therefore recommend UNCHANGED (still CONDITIONAL), with the concrete test above as a potential path to ACCEPT if the metric's validity is demonstrated on the real outputs.","tokens_in":15888,"tokens_out":8737,"duration_ms":85708,"concrete_test":"Run a human evaluation on the actual generated outputs (e.g., 100 questions per model from MixKhanQ), asking teachers or qualified annotators to rate (a) whether the question addresses the prescribed topic and (b) the educational quality/usefulness of the question. Then correlate the human topic-alignment ratings with the WikiSemRel (Jaccard) scores. If the correlation is weak, or if TopicQG2X's advantage over Baseline disappears under human topic ratings, the central claim is not supported. As an additional computational check, recompute WikiSemRel after masking the topic entity words in the generated questions; if the model ranking in Table 4 changes substantially, the metric is dominated by surface topic mention.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central evidence that TopicQG2X 'surpasses all other models' (Section 4.3) is the WikiSemRel (Jaccard) difference in Table 4. However, the target topic t is defined as the highest-PageRank Wikipedia concept in the reference question, extracted via WAT (Section 3.2.1). The evaluation metric WikiSemRel (Jaccard) also uses WAT to extract concepts from the generated and reference questions, then computes overlap of outward links (Section 3.6.2). Thus the control signal and the evaluation instrument share the same entity linker. A model that simply includes the topic entity string in its output will have that concept linked by WAT, matching the reference's top concept and raising the Jaccard score, regardless of whether the question is educationally meaningful. The human validation in Section 3.4 only asked four annotators to choose between two generated questions on 30 pairs; it did not test whether the metric rewards superficial topic mention over deeper topical understanding. With MAE 0.23 for Jaccard, the per-item metric error is substantial, and the 0.135 difference between TopicQG2X and Baseline could partly reflect increased surface mention rather than genuine topic control. The paper's own limitation statement (Section 6) acknowledges the small human sample, but the deeper problem is that the metric may not measure what the abstract claims: 'high-quality, topic-focused questions.' Additionally, no significance testing is reported for the Table 4 differences, so the improvement is not yet separated from metric noise.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes Topic-Controlled Question Generation (T-CQG) by fine-tuning T5-small. The authors create SQuAD+, MixSQuAD, and MixSQuAD2X from SQuAD by wikifying contexts and questions, filtering on concept overlap, and concatenating context pairs to create contrastive examples. They evaluate a Baseline, TopicQG, TopicQGedu, quantised variants, and TopicQG2X on MixKhanQ, using lexical and embedding metrics plus Wikipedia-based semantic relatedness (WikiSemRel, w2v and Jaccard) to measure topical alignment. A small human annotation study (30 pairs, 4 annotators) is used to compare BERTScore and WikiSemRel. The headline finding is that TopicQG2X achieves the largest topic-vs-alternative difference on WikiSemRel (Jaccard), which is presented as evidence of superior topic control with a small open-source model. The paper also reports memory savings from 8-bit and 4-bit quantisation and concludes that the approach can reduce reliance on proprietary LLMs in educational question generation.","tokens_in":16165,"tokens_out":5806,"duration_ms":53627,"significance":"If the headline result holds, the paper offers a useful, low-cost alternative to proprietary LLMs for topic-focused educational question generation, and the dataset-construction and evaluation methodology could inform future work. The authors are transparent about limitations, and the approach is in principle reproducible using public datasets, an open model, and a linked code repository. However, the evidence for the central claim is currently weakened by the lack of inferential statistics for the key comparisons and by the dependence of both the control signal and the main evaluation metric on the same Wikipedia annotation pipeline. The paper also provides a useful human-evaluation baseline for semantic relatedness metrics, although the sample is small.","major_comments":[{"comment":"The central claim that TopicQG2X 'surpasses all other models' and the RQ2/RQ5 comparisons rest on differences in WikiSemRel (Jaccard) that are reported without confidence intervals or significance tests. The only inferential test in the paper (paired t-test, p=0.083, in Section 4.2) is applied to a comparison that is not central to the headline result. Given that Table 2 reports MAE 0.23 for the Jaccard metric on the 30-pair human validation, the observed differences between models (e.g., 0.595 vs 0.545 for TopicQG vs Baseline, 0.680 vs 0.595 for TopicQG2X vs TopicQG) are not obviously outside the metric's noise level. Please report paired significance tests or bootstrap confidence intervals for all model comparisons in Tables 3 and 4.","section":"Section 4.3, Table 4"},{"comment":"The target topic t is defined as the most authoritative WAT Wikipedia concept in the reference question, and the WikiSemRel (Jaccard) evaluation score is computed from WAT concepts and their outward links in the generated and reference questions. The control signal and the evaluation instrument therefore share the same entity linker, creating a partial circularity: a model that echoes the topic entity string can raise the Jaccard score without producing a question with genuine topical depth. The human validation does not break this circularity because the four annotators were only asked to choose which of two generated questions was closer to a reference, not to judge whether the metric rewards superficial topic mention. Please add a validation study or a quantitative analysis (e.g., comparing scores of outputs with and without the surface topic string) that directly tests what the metric rewards.","section":"Sections 3.2.1 and 3.6.2, with Section 3.4 and Table 2"},{"comment":"The RQ5 conclusion that data augmentation improves topical relevance is confounded with dataset size. MixSQuAD2X is twice as large as MixSQuAD because two orderings of each context pair are included, so TopicQG2X is trained on strictly more examples than TopicQG. Setting aside the qualitative difference between the two orderings, the gain in Table 4 could be due to more training data rather than to the augmentation strategy itself. An ablation that trains TopicQG on the same number of examples, or that otherwise controls for training steps or data volume, is needed to support the claim that the reversal augmentation is the cause of the improvement.","section":"Sections 3.2.3 and 4.3"}],"minor_comments":[{"comment":"The sentence 'all newly proposed models except the 4bit quantised TopicQG model outperforms the baseline' is contradicted by Table 4, where TopicQGedu has a difference of 0.536, below the baseline's 0.545; please correct this factual error and adjust the surrounding RQ3 discussion.","section":"Section 4.2"},{"comment":"The metric name is inconsistent: 'WikiSemRel' appears in Section 3.6.2 and the abstract, while Tables 2-4 and Section 4.1 use 'WikiSimRel'; please standardise the terminology throughout.","section":"Section 3.6.2 and Tables 2-4"},{"comment":"The paired t-test is described only as comparing 'performance scores' of TopicQGedu and TopicQG; please specify which metric was tested, the sample size, and whether correction for multiple comparisons was considered, since the paper reports many metrics.","section":"Section 4.2"},{"comment":"There are several typos and minor wording issues, including 'BERTSCore' in Section 3.6.1, 'MATEOR' in Section 4.3, and the unexplained notation in equation (2); a careful copyedit would improve readability.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a learning-analytics or AI-in-education venue, and the empirical direction is promising. The main risk is that the headline numeric results may not survive significance testing or a more independent validation of the metric. I would encourage a revision that addresses the shared-annotation circularity and adds inferential statistics rather than a rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is a competent applied NLP effort: contrastive data enrichment for T5-small, a Wikipedia-based relatedness metric, and a clean comparison of quantization and pre-training variants. It is not a breakthrough, and it does not pretend to be—the authors openly say output quality is below large LLMs. What is new is the MixSQuAD/MixSQuAD2X recipe and the demonstration that a 60M-parameter model can be fine-tuned to follow a topic string reasonably well. That is worth a serious look if you work on educational question generation or resource-constrained deployment.\n\nWhat the paper does well: the experimental design is straightforward, the baseline is fair, and the authors acknowledge the main limitations (small human evaluation, sLM quality gap). The quantized models retaining most performance is a genuinely useful result for low-resource settings. The related-work coverage is adequate, including prior topic-controlled QG, so they do not oversell novelty.\n\nWhere the soft spots are, in proportion: the biggest issue is the potential circularity the stress-tester flagged. Topics are extracted with WAT, and the WikiSemRel metric also uses WAT to extract concepts and compute Jaccard overlap of Wikipedia links. A model that simply echoes the topic entity string in its output will get a higher Jaccard score regardless of whether the question is educationally meaningful. The human validation (30 pairs, 4 annotators) is too small to rule this out—they only asked which of two generated questions is closer to the reference, not whether the metric rewards superficial topic mention over deeper understanding. So the 0.135 difference between TopicQG2X and baseline in Table 4 could partly be a surface-mention artifact. Second, no significance tests are reported on the main comparisons; the one paired t-test they ran (TopicQGedu vs TopicQG) gave p=0.083, which should make you cautious about claims like “significantly outperform.” The abstract's “rigorous” evaluation is an overstatement. Third, datasets are not released (the GitHub link is mentioned but no clear data artifact), which hurts reproducibility.\n\nNone of these are fatal. The central idea is plausible, and the limitations are acknowledged in Section 6. The paper deserves a serious referee—with requests for significance testing on Tables 3 and 4, a larger human evaluation or error analysis of the Jaccard metric, and dataset release. I would not desk-reject it, but I would send it back for revision before accepting. If you are in the educational QG area, cite it for the contrastive dataset method; otherwise it is a useful read but not essential.","headline":"Solid, modest contribution to small-model topic-controlled question generation, but the headline metric may partly reward superficial topic mention because it shares its entity linker with the topic definition.","tokens_in":16758,"tokens_out":1490,"would_cite":true,"duration_ms":16572,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A small, open question generator can produce education questions that stick to a prescribed topic.","keywords":["topic-controlled question generation","educational question generation","small language models","T5 fine-tuning","contrastive data augmentation","Wikipedia semantic relatedness","model quantisation","scalable AI in education"],"falsifier":"Take a fresh set of, say, 200 Khan Academy question contexts, generate questions with the prescribed and an alternative topic using TopicQG2X, and have teachers judge which set is more on topic; if teacher judgments do not reproduce the reported WikiSemRel (Jaccard) gap, the central claim fails.","tokens_in":15624,"feed_emoji":"🎓","tokens_out":6577,"duration_ms":57885,"temperature":0.7,"pith_summary":"This paper tries to show that a small, open language model can be fine-tuned to generate educational questions that track a prescribed topic, without the cost or privacy concerns of calling a large proprietary model. The authors build contrastive training datasets from SQuAD and KhanQ, linking each context-question pair to a Wikipedia concept that serves as the topic, and train T5-small variants on them. They also propose a Wikipedia-based semantic relatedness metric, WikiSemRel (Jaccard), and validate it against human judgments. On the MixKhanQ evaluation set, their best model, TopicQG2X, shows the largest gap between questions generated on the prescribed topic and questions generated on an alternative topic, which they interpret as superior topic control. If the result holds, teachers and tutoring systems could generate topic-specific questions with a model small enough to run on a phone.","feed_headline":"Fine-tuned T5-small generates education questions that stay on topic","feed_subtitle":"A 60M-parameter open model stays on topic while cutting cost and dependence on proprietary AI.","key_machinery":"The load-bearing mechanism is the contrastive data-construction method. Starting from SQuAD and KhanQ pairs, the authors wikify each context and question, keep the top five Wikipedia concepts by PageRank, and select the most authoritative concept in the question as the target topic; they then concatenate two randomly chosen contexts into one shared context and ask the model to generate a question for a given topic from the combined paragraph, creating MixSQuAD and, with both orderings, MixSQuAD2X. This forces the model to learn which concepts in a long context the topic points to. The other central object is the evaluation metric, WikiSemRel with the Jaccard measure, which computes relatedness from outward Wikipedia links and, per the paper's human annotation study, agrees with human judgments better than BERTScore or the w2v variant.","core_discovery":"The central discovery is that fine-tuning a T5-small model (about 60M parameters) on contrastively mixed contexts makes its questions follow a specified topic more closely. On the MixKhanQ evaluation set, the best model, TopicQG2X, achieves a WikiSemRel (Jaccard) difference of 0.680 between questions generated with the prescribed topic and questions generated with an alternative topic, against 0.545 for the baseline, and it surpasses all other models on this indicator. The paper also reports that the TopicQG model improves over the baseline on most linguistic-quality metrics, that 8-bit and 4-bit quantisation cut memory from about 230MB to around 110MB and 94MB with only small metric drops, and that further pre-training on scientific text (TopicQGedu) does not yield a statistically significant improvement. These results are offered as evidence that topic-controlled educational question generation is feasible with a small, locally deployable model rather than requiring proprietary large language models.","pith_inferences":["An implication the paper leaves implicit is that the same contrastive pairing recipe could be extended to control other generation properties, such as question difficulty, length, or cognitive load, by replacing the topical-pairing criterion with those attributes.","Because the topic signal is whatever Wikipedia concept has the highest PageRank in the reference question, the model's 'topic' is an annotation artifact; whether it matches the curriculum topic a teacher intends is a question the paper does not yet test.","The metric validation rests on 30 question pairs, so a larger human study is the natural next check: if WikiSemRel (Jaccard) fails to track teacher judgments at scale, the headline topic-control gains would need re-interpretation.","One could test the approach on other educational domains and languages; the paper's evidence is confined to English STEM content from Khan Academy and SQuAD-derived Wikipedia articles."],"forward_implications":["Teachers and tutoring systems could generate topic-specific questions from paragraph-level materials using an open model small enough to deploy on a phone.","Data augmentation by reversing context order is a cheap way to strengthen topical control without new human annotation.","Quantised versions of the model cut memory by roughly half while keeping most of the quality gains, lowering infrastructure and energy costs.","The WikiSemRel (Jaccard) metric can serve as an automated substitute for human ratings of topical relatedness in educational question generation.","Educational institutions can avoid depending on externally hosted proprietary models for this task, addressing privacy and governance concerns."],"supporting_citations":[{"why":"SQuAD provides the underlying context-question pairs from which SQuAD+, MixSQuAD, and MixSQuAD2X are derived for fine-tuning.","marker":"[50]"},{"why":"T5 is the pre-trained text-to-text transformer that is fine-tuned in all the proposed models.","marker":"[49]"},{"why":"KhanQ supplies the evaluation examples that are transformed into MixKhanQ to measure educational question quality.","marker":"[31]"},{"why":"WAT is the entity annotator used to wikify contexts and questions and to compute the WikiSemRel relatedness scores.","marker":"[45]"},{"why":"Wikipedia entity relatedness defines the outward-link Jaccard measure used by WikiSemRel, the metric that best matches human judgments.","marker":"[47]"},{"why":"Prior educational question generation with pre-trained language models provides the pre-training strategy and the comparison point for scalable question generation.","marker":"[7]"},{"why":"Controllable text simplification supplies the fine-tuning recipe the baseline model follows for conditioning generation.","marker":"[42]"},{"why":"LLM.int8() is the 8-bit quantisation algorithm used to create the TopicQG8bit model.","marker":"[18]"},{"why":"QLoRa is the 4-bit quantisation method used to create the TopicQG4bit model.","marker":"[19]"},{"why":"BERTScore is the embedding-based metric the paper compares against WikiSemRel in the human-alignment study.","marker":"[60]"}],"fun_headline_variants":["T5-small fine-tuned for sharper topic control in question generation","60M-parameter model improves topic focus in educational questions","Quantized T5-small cuts memory 60% for on-topic question generation","Open 60M model delivers topic-specific questions without big LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire topic-control result rests on a semantic-similarity metric validated with only 30 human-rated question pairs, so if that metric misses what teachers actually mean by 'on topic,' the headline improvement may be an artifact.","fun_headline_variants_meta":{"raw":{"variants":["T5-small fine-tuned for sharper topic control in question generation","60M-parameter model improves topic focus in educational questions","Quantized T5-small cuts memory 60% for on-topic question generation","Open 60M model delivers topic-specific questions without big LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000238,"raw_usage":{"total_tokens":1531,"prompt_tokens":987,"completion_tokens":544,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":468}},"tokens_in":603,"tokens_out":544,"duration_ms":6444,"temperature":1.0,"reasoning_tokens":468,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:14:53.243651+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh set of, say, 200 Khan Academy question contexts, generate questions with the prescribed and an alternative topic using TopicQG2X, and have teachers judge which set is more on topic; if teacher judgments do not reproduce the reported WikiSemRel (Jaccard) gap, the central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"T5 is the pre-trained text-to-text transformer that is fine-tuned in all the proposed models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"KhanQ supplies the evaluation examples that are transformed into MixKhanQ to measure educational question quality."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Wikipedia entity relatedness defines the outward-link Jaccard measure used by WikiSemRel, the metric that best matches human judgments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior educational question generation with pre-trained language models provides the pre-training strategy and the comparison point for scalable question generation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Controllable text simplification supplies the fine-tuning recipe the baseline model follows for conditioning generation."}],"review_version":1}