{"id":"e4762cc4-a2c1-4e5d-a8e5-dbb36ba29f97","arxiv_id":"2506.06812","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A fine-tuned Flan-T5-large can jointly control narrative element and IRT-estimated difficulty of generated QA pairs, with consistent control at easy, hard and extreme levels but unreliable intermediate levels.","lead":"This paper tests whether a single question-generation model can control both the narrative focus and the difficulty of reading-comprehension questions it produces. Using IRT-based difficulty labels estimated from simulated learners, it finds the joint control is feasible but uneven, especially at intermediate difficulty levels.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The five-level difficulty scale rests on IRT estimates from only five simulated learners; the medium/moderate inconsistency is likely label noise, not a genuine property of joint control.","rationale":"The reader's weakest assumption is correct: simulated-learner QA systems may not reflect human ability, and the paper itself acknowledges this. My stress test identifies a more immediate and internal problem: with only five simulated learners, the IRT difficulty labels are statistically unreliable, and the paper's five-level scale is essentially a coarse discretization of a five-response sum score. This undermines the interpretation of the headline result — inconsistent control at medium and moderate levels — because that inconsistency is exactly what label noise would produce. It also amplifies the circularity the reader noted: the same five systems define the labels and evaluate the output, so systematic biases are propagated. I do not see this as fatal; the paper is a feasibility study, and its own three-level experiment (Figure 6) is a sensible response. But the main claim about granular difficulty control should be conditioned on demonstrating label stability, e.g., via leave-one-learner-out or a larger ensemble. The reader's conditional verdict is appropriate; no verdict change is needed. I credit the paper for clearly stating the simulated-learner limitation and for reporting error analysis, which shows awareness of the proxy's limits.","tokens_in":14309,"tokens_out":6363,"duration_ms":71401,"concrete_test":"Re-estimate the five difficulty labels with leave-one-learner-out: for each of the five QA systems, drop that system, re-run the IRT estimation (Section 4.4) on the remaining four systems, and map to the five labels. Compute the fraction of items whose label changes across folds and report the standard errors of the difficulty parameters from mirt. If more than about 20% of items change level, the five-level scale is not stable and the medium/moderate comparisons in Figures 4-5 cannot be interpreted as a property of the method. A complementary check is to repeat label creation with an independent set of simulated learners (e.g., 10-20 models or multiple checkpoints) and see whether the Figure 4 trends persist.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the joint model controls difficulty at five levels depends on the reliability of the IRT-based difficulty labels (Section 4.4). These labels are estimated from a binary response matrix with exactly five respondents: the simulated-learner QA systems of Section 4.2. Each question's difficulty parameter is therefore fit from five binary observations. With so little data, the Rasch model's item parameters are essentially determined by the sum score (how many of the five systems answered correctly), and the five normalized levels (0, 0.28, 0.50, 0.72, 1) are arbitrary bins on that count. No standard errors, model fit, or label-stability analysis is reported. Consequently, the observed confusion between medium and moderate levels in the joint model (Figures 4-5) may be an artifact of noisy, unreliable labels rather than a limitation of combining narrative and difficulty control. The same five systems are also used as the evaluation instrument, so any systematic bias in these systems (e.g., poor performance on 'why' questions) is shared between label creation and evaluation; this shared bias can make the difficulty ordering look more consistent than it would be for human learners.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a joint narrative and difficulty control strategy for question-answer pair generation from narrative texts. It augments FairyTaleQA with IRT-based difficulty labels estimated from binary correctness of five simulated-learner QA systems (DeBERTaV3, RoBERTa, BERT, DistilBERT, GPT-2), then fine-tunes Flan-T5-large on four data setups (text-only, narrative+text, difficulty+text, narrative+difficulty+text). Narrative control is evaluated by ROUGE-L F1 and BLEURT similarity to human-authored ground-truth questions; difficulty control is evaluated by the correctness of the same five simulated learners on generated QA pairs across difficulty levels. The paper reports that narrative control is consistently effective, difficulty control alone shows a monotonic decrease in correctness, joint control is generally effective at easy, hard, and extreme levels but inconsistent at medium and moderate levels, and that reducing difficulty granularity improves consistency. The authors frame the results as preliminary evidence of feasibility.","tokens_in":14538,"tokens_out":7084,"duration_ms":61349,"significance":"The topic addresses a genuine gap in controllable question generation by combining two control dimensions. The paper has several strengths: it uses an established dataset with expert narrative annotations, adopts a principled IRT framework, includes an error analysis and a linguistic-feature analysis (PINC), and ablate difficulty granularity. If the difficulty-control evidence were externally validated, the contribution would be useful for educational question generation. However, the significance of the difficulty-control claims is currently tempered by the circularity between label construction and evaluation, the very small number of simulated learners used for IRT estimation, and the absence of statistical inference. The narrative-control results, while not subject to circularity, also lack significance testing.","major_comments":[{"comment":"The difficulty labels used for training and the evaluation instrument for difficulty control are the same five simulated-learner QA systems. Section 4.4 constructs the IRT difficulty labels from the binary response matrix of these systems; Section 5.2 then measures difficulty control by the correctness of the same systems on the generated QA pairs (Figures 3-6). Under this protocol, the monotonic ordering of correctness across difficulty levels is partly by construction: the systems' systematic biases (e.g., specific failure modes on 'why' questions) enter both the label definition and the outcome measure. This shared-bias problem is acknowledged in the Limitations section only with respect to the use of simulated learners generally, not with respect to the circularity. The difficulty-control claim would be substantially strengthened by evaluating on held-out QA systems not used in label creation, or on human student responses.","section":"Section 4.4 and Section 5.2"},{"comment":"The Rasch model is fit to a binary response matrix with only five simulated learners as respondents. With five respondents per question, the item difficulty parameter has very little information; the five normalized difficulty values (0, 0.28, 0.50, 0.72, 1) correspond roughly to sum scores of the five systems, and the labels are effectively bins on a count from 0 to 5. The paper reports no standard errors, item fit statistics, or test-retest/stability analysis of the estimated difficulties. Consequently, the observed confusion between medium and moderate levels in the joint model (Figures 4-5) may be an artifact of noisy labels rather than a genuine property of joint control. Please report the response matrix size, the distribution of sum scores, and label-stability checks (e.g., bootstrap resampling of questions or leave-one-learner-out estimation).","section":"Section 4.4"},{"comment":"The claims that narrative control improves similarity to ground truth (Table 4 and Table 6) and that difficulty control is 'consistent' at easy, hard, and extreme but 'inconsistent' at medium and moderate (Figures 3-6) are based on descriptive statistics without significance tests, confidence intervals, or effect sizes. For example, in Figure 4 the difference between medium and moderate for DeBERTaV3 is 44.6 vs 44.6, and for RoBERTa 38.7 vs 39.3, an inversion; these differences are within the range of sampling noise for the number of generated QA pairs. The paper should provide bootstrap or paired statistical tests across the generated QA pairs for the key comparisons, or at least report the number of items per cell and confidence intervals.","section":"Section 5.2"},{"comment":"The evaluation protocol in Section 4.6 states that five QA pairs are generated for each test section (one per difficulty level) and that the approach 'ensures that the generated QA pairs are balanced across distinct difficulty levels and narrative elements,' but it does not specify how the narrative label is chosen for each generated pair. Since the joint model is conditioned on a narrative label, the per-narrative difficulty-control analysis in Figure 5 depends on a defined mapping from sections to narrative labels. Without this, the number of generated QA pairs per (narrative, difficulty) cell is unclear, and the lower consistency reported for 'prediction' and 'feeling' may simply reflect fewer examples. Please clarify the inference protocol and report cell sizes.","section":"Section 4.6"}],"minor_comments":[{"comment":"The labels 'Dagumented' and 'D1-agumented' are typos; they should read 'Augmented' and 'D1-augmented'.","section":"Figure 2 and surrounding text"},{"comment":"The metric name is written inconsistently as 'ROUGEL-F1' (e.g., Table 1, Table 4) and 'ROUGE L-F1' (Section 5.1); please standardize to 'ROUGE-L F1'.","section":"Throughout"},{"comment":"The dataset name is inconsistently capitalized as 'FairyTaleQA' and 'FairytaleQA'; please choose one form and use it consistently.","section":"Throughout"},{"comment":"The sentence 'Appendix A shows further support by reporting semantic proximity results' could be clearer as 'reporting BLEURT semantic similarity results'.","section":"Section 5.2"},{"comment":"The paper does not mention whether code or models will be released; for reproducibility, please include an availability statement.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The paper's results are appropriately framed as preliminary, and the narrative-control evidence is reasonably solid. The main risk is the difficulty-control evaluation circularity; if the authors can validate on held-out QA systems or real students and add statistical tests, the contribution would be acceptable. The paper is borderline for a journal in its current form; it may be better suited as a workshop paper unless the evaluation is hardened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the paper. It's a decent feasibility study for combining narrative and difficulty control in educational question generation, built on FairyTaleQA and Flan-T5. The genuinely new bit is pairing IRT-estimated difficulty labels (from simulated learners) with narrative-element control; the PINC finding—that higher difficulty levels produce more lexically novel QA pairs—is a nice, new empirical observation, and the error analysis of hallucinated and nonsensical pairs is honest and useful.\n\nThe paper does several things right. Narrative control is consistently supported: adding narrative labels improves ROUGE-L and BLEURT similarity to human-authored questions across all seven labels, and this holds when difficulty labels are also added. The authors are also candid about limitations—simulated learners, unbalanced categories, hallucinations—which makes the work easy to place.\n\nThe soft spot is the difficulty-control evaluation. The same five SQuAD-trained QA systems that generate the difficulty labels are used to verify that the labels work. That is a real circularity, and the paper should have said more about it. Also, the stress-test point is fair: with only five respondents per item, the Rasch difficulty parameters are essentially a sum score, and the five normalized levels are arbitrary bins on that count. So the medium/moderate confusion the authors observe in the joint model may be label noise rather than a property of joint control. No standard errors or label-stability analysis are reported. Still, the pattern across easy/hard/extreme is clear, and the authors explicitly frame the result as preliminary, so I wouldn't call this fatal. It does mean the difficulty-control claims are weaker than the narrative-control claims.\n\nOne more minor thing: no significance tests or confidence intervals anywhere; the round numbers in Figures 3-5 would be easier to trust with error bars, especially given the small test set (394 sections).\n\nWho is this for? People working on controllable QG for education, and anyone building on IRT-style difficulty labels. It's a workshop-grade paper, not a major new framework, but it's honest and has a couple of findings worth building on. I'd send it to review.","headline":"Honest, modest feasibility study for joint narrative and difficulty control; narrative evidence is solid, difficulty evidence is weakened by a circular simulated-learner setup.","tokens_in":15033,"tokens_out":1916,"would_cite":false,"duration_ms":21173,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that a single fine-tuned Flan-T5-large can simultaneously control the narrative element and difficulty tier of generated reading-comprehension QA pairs, with reliable difficulty separation at easy, hard, and extreme…","keywords":["question generation","difficulty control","narrative control","item response theory","reading comprehension","educational NLP","Flan-T5","FairyTaleQA"],"falsifier":"Have a group of students with known reading ability answer the 1,970 generated QA pairs (or a stratified subset) and compute the proportion of correct answers per requested difficulty tier and narrative attribute. If the easy-to-extreme ordering does not appear in students' responses, or if students' error patterns differ substantially from those of the simulated-learner QA systems, then the difficulty labels and the reported difficulty control lack external validity for real learners.","tokens_in":14116,"feed_emoji":"🎯","tokens_out":9725,"duration_ms":88015,"temperature":0.7,"pith_summary":"This paper asks whether one modest-scale model—Flan-T5-large—can generate reading-comprehension question–answer pairs that obey two user-supplied attributes at once: a narrative element (such as causal relationship, character, or prediction) and a difficulty level (easy through extreme). To get difficulty labels without human annotators, the authors answer all FairyTaleQA questions with five SQuAD-trained QA systems of different abilities, then fit a Rasch Item Response Theory model to the resulting binary responses and bucket the estimated difficulties into five tiers. The answer is a qualified yes: narrative control is consistently achieved, difficulty control works well when conditioned alone, and the joint strategy separates easy from hard and extreme clearly but confuses the middle two tiers. The result matters for educational question generation, where tailoring both what a question asks and how hard it is would let a single system produce personalized practice items.","feed_headline":"Fine-tuned model steers question difficulty and narrative in one pass","feed_subtitle":"Simulated-learner difficulty labels plus narrative tags let one model steer both attributes of generated QA pairs.","key_machinery":"The load-bearing machinery is the Rasch model of Item Response Theory, which converts the binary correctness matrix produced by five simulated-learner QA systems (DeBERTaV3, RoBERTa, BERT, DistilBERT, GPT-2; ability estimates from −1.60 to 0.43) into a per-question difficulty parameter $b$, via $P(X=1 \\mid \\theta, b)=e^{\\theta-b}/(1+e^{\\theta-b})$, normalized to five labels (easy, medium, moderate, hard, extreme). Those labels, together with FairyTaleQA's seven expert narrative annotations, are injected into a prompt template that conditions Flan-T5-large; the model is fine-tuned on the augmented dataset and decodes with top-$k$ sampling ($k=50$, $p=0.9$, temperature 1.2). The argument that control works is carried by two comparisons: narrative control is measured by ROUGE-L F1 and BLEURT similarity between generated and human-authored questions, and difficulty control by the percentage of correct answers from the simulated learners across requested difficulty tiers. A secondary mechanism is the PINC metric (paraphrase in n-gram changes), which quantifies lexical novelty and is shown to increase with difficulty.","core_discovery":"On its own terms, the paper's central claim is that joint narrative and difficulty control is feasible in a single fine-tuned encoder–decoder model: given a prompt of the form 'Generate a ⟨d⟩ question-answer pair about narrative label ⟨n⟩ considering the following text: ⟨t⟩', Flan-T5-large produces questions whose narrative content resembles human-authored ground truth (ROUGE-L F1 and BLEURT both rise over a text-only baseline) and whose difficulty, measured by the percentage of simulated learners answering correctly, falls monotonically as the requested difficulty rises from easy to hard to extreme. The qualification the authors emphasize is that the middle tiers (medium and moderate) are not reliably separated under joint control, and that the effect varies by narrative attribute: causal and outcome questions show the most consistent difficulty ordering, while prediction and feeling questions—the rarest in FairyTaleQA—show the least. The paper also reports that reducing the difficulty granularity from five levels to three restores consistency, and that lexical novelty of generated pairs relative to the source text rises with difficulty, with 14% of a sampled hundred pairs containing hallucinated content and 10% being nonsensical.","pith_inferences":["Because the same QA systems that produce the IRT difficulty labels also measure whether difficulty control worked, the two evaluation stages are coupled; independent learner data would be a stronger check on the difficulty claims.","The PINC-vs-difficulty trend suggests a cheap, model-free proxy for difficulty control: lexical novelty relative to the source text could be monitored during decoding without invoking QA systems, though the paper does not itself claim this.","The flattening of the difficulty slope under joint control is likely a general phenomenon: when one control attribute constrains the content of the question, the space of possible questions shrinks, so the other attribute has less room to vary; similar joint-control systems should expect the same trade-off."],"forward_implications":["A single fine-tuned model can offer both content control and difficulty control at once, so an educational QG system can generate personalized practice items without separate pipelines for narrative and difficulty.","Difficulty labels can be produced automatically for any QA dataset via the simulated-learner plus IRT pipeline, removing the need for human difficulty annotation at scale.","Joint control is reliable at the extremes of the difficulty scale; systems aiming for fine-grained intermediate difficulty should expect confusion, and coarser three-level difficulty labels restore consistency.","Hallucinated content (14% of sampled outputs) probably inflates measured difficulty, since QA systems fail on questions about invented entities; evaluating controllability must therefore be paired with a hallucination check."],"supporting_citations":[{"why":"Supplies the IRT-based difficulty-control method and simulated-learner setup that the joint strategy builds on.","marker":"Uto et al., 2023"},{"why":"Provides the DCQG evaluation protocol and the adaptive QA generation approach using IRT with pretrained transformers.","marker":"Tomikawa et al., 2024"},{"why":"Contributes FairyTaleQA, the dataset whose expert narrative annotations are the narrative control attribute.","marker":"Xu et al., 2022"},{"why":"Establishes narrative-controllable QG via question-type distribution and event-centric summarization, the basis for NCQG evaluation.","marker":"Zhao et al., 2022"},{"why":"Shows that narrative control improves similarity to ground-truth questions and supplies the evaluation protocol for Hypothesis 1.","marker":"Leite and Lopes Cardoso, 2024"},{"why":"Provides SQuAD v1.1, the training data for the five simulated-learner QA systems.","marker":"Rajpurkar et al., 2016"},{"why":"Defines the Item Response Theory framework and the Rasch model used to estimate difficulty.","marker":"Lord, 2012"},{"why":"Introduces Flan-T5, the instruction-tuned encoder-decoder model that is fine-tuned for joint control.","marker":"Chung et al., 2024"}],"fun_headline_variants":["Joint difficulty and narrative control in QG: feasible but uneven","One fine-tuned model, two controls: QG difficulty and narrative","Steering both difficulty and narrative in one QG pass","Joint control of QG difficulty and narrative works, with conditions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that five SQuAD-trained QA systems (DeBERTaV3, RoBERTa, BERT, DistilBERT, GPT-2) approximate how real students answer narrative questions, so that IRT difficulty estimates computed from their correctness and the difficulty-control evaluation both rest on this proxy; the authors acknowledge that real student data are still needed.","fun_headline_variants_meta":{"raw":{"variants":["Joint difficulty and narrative control in QG: feasible but uneven","One fine-tuned model, two controls: QG difficulty and narrative","Steering both difficulty and narrative in one QG pass","Joint control of QG difficulty and narrative works, with conditions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000485,"raw_usage":{"total_tokens":2383,"prompt_tokens":926,"completion_tokens":1457,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":1385}},"tokens_in":542,"tokens_out":1457,"duration_ms":10496,"temperature":1.0,"reasoning_tokens":1385,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:48:23.738843+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a group of students with known reading ability answer the 1,970 generated QA pairs (or a stratified subset) and compute the proportion of correct answers per requested difficulty tier and narrative attribute. If the easy-to-extreme ordering does not appear in students' responses, or if students' error patterns differ substantially from those of the simulated-learner QA systems, then the difficulty labels and the reported difficulty control lack external validity for real learners.","supporting_citations":[],"review_version":1}