From Model to Classroom: Evaluating Generated MCQs for Portuguese with Narrative and Difficulty Concerns

Abel Ferreira; Bernardo Leite; Henrique Lopes Cardoso; Isabel Rangel; Lu\'is Abreu; Pedro Pinto; Sandra Monteiro

REVIEW 4 major objections 6 minor 65 references

From Model to Classroom: Evaluating Generated MCQs for Portuguese with Narrative and Difficulty Concerns

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GPT-4o and Gemma-2 can generate Portuguese reading-comprehension MCQs that expert reviewers and student-response analysis find comparable in quality to human-authored items.

desk verdict A solid empirical study of LLM-generated Portuguese MCQs with real classroom data; the headline claim needs scoping to one-step models and the psychometric comparison carries a selection-bias caveat. read the letter →

arxiv 2506.15598 v1 pith:63OGGOOC submitted 2025-06-18 cs.CL cs.AI

Bernardo Leite , Henrique Lopes Cardoso , Pedro Pinto , Abel Ferreira , Luís Abreu , Isabel Rangel , Sandra Monteiro This is my paper · ORCID

classification cs.CLcs.AI

keywords multiple-choicequestiongenerationreadingcomprehensionEuropeanPortugueselargelanguagemodelspsychometricevaluationdifficultycontrolnarrativeelementsclassroomassessment

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether current large language models can produce multiple-choice reading-comprehension questions for Portuguese elementary students that are good enough for real classrooms, and answers yes for one-step generation. Expert reviewers found that MCQs generated zero-shot by GPT-4o and Gemma-2 were as acceptable as human-authored items, with both achieving the same 73.3% overall acceptance rate on the strictest quality gate, answerability. A two-step pipeline that first generates the question with a fine-tuned small model and then generates options with Gemini-2 did not: only 55.6% of its items were accepted. The paper also shows that models can control narrative focus (character, feeling, setting, action, causality) and produce difficulty ratings that line up with expert judgment, particularly when difficulty is assigned after the full question is written. The attraction of the result is that it moves the question from 'can models generate text' to 'can models generate usable assessment items' in a language and age group where manual item writing is costly.

What carries the argument

The carrying mechanism is a dual evaluation pipeline. First, an expert review protocol with majority voting scores each MCQ on well-formedness, narrative alignment, option clarity, answerability, distractor plausibility, and difficulty; answerability—whether the text contains the answer and whether any option matches it—is the strictest gate. Second, a psychometric analysis based on Classical Test Theory uses student responses to compute item difficulty (1 − P), discrimination (D, the difference between top- and bottom-27% performers), distractor engagement, and a three-rule option-quality test. The generation side pairs zero-shot prompting (GPT-4o, Gemma-2) with a two-step modular pipeline (Ptt5-v2 question generator plus Gemma-2 options), and difficulty is annotated either during generation or after the full MCQ exists.

What would settle it

Run the same Classical Test Theory analysis on the full unfiltered output of each generator—or on a proportional sample that includes rejected items—and check whether the Kruskal-Wallis tests for difficulty and discrimination remain non-significant; if significant differences appear, the comparability finding is an artifact of expert pre-selection.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that zero-shot prompting of current LLMs yields reading-comprehension MCQs in European Portuguese whose quality is indistinguishable from human-authored items by both expert review and psychometrics. Concretely, 33 of 45 (73.3%) LLM-generated items and 33 of 45 (73.3%) human items were accepted as answerable, well-formed, and clearly written; student-response analysis over 284 participants found no statistically significant differences in difficulty (1 − P) or discrimination (D) between human and one-step LLM MCQs. Human-authored items retained an edge in distractor engagement—students selected all three distractors in 57.6% of human items versus 45.5% for GPT-4o and 51.5% for Gemma-2—and in adherence to option-selection rules, so the paper concludes models are approaching, not yet surpassing, human benchmarks. The two-step method, by contrast, produced items that were significantly less answerable (67.6%) and less discriminative, and the authors attribute this to a bottleneck in the fine-tuned question-generation module.

Load-bearing premise

The psychometric comparison is run only on the 124 MCQs that experts had already cleared as well-formed, clear, and answerable, which disproportionately removes the lowest-quality generated items before difficulty and discrimination are measured.

Editorial extensions

If this is right

Teachers and educational platforms can treat one-step LLM output as a draft pool that still requires expert review, since roughly a quarter of items fail answerability.
Two-step generation, at least with a small fine-tuned first-stage model, should be avoided for MCQ production.
Difficulty annotations requested after the full MCQ is generated align better with expert perception than annotations made during generation, giving a practical recipe for calibration.
Narrative control works well enough that generated items can be targeted at specific curriculum elements such as character, feeling, or causality.
Human distractors remain more engaging, so research on distractor design—not just correctness—is where the next quality gain lies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the equality of acceptance rates (73.3%) holds on a small sample of 45 items per provenance, the 'comparable quality' conclusion is sensitive to sample size; a larger head-to-head could reveal differences the current study lacks power to detect.
The paper's psychometric comparison ignores expert-rejected items, and since rejection was most frequent for the two-step method, the conclusion that the two-step pipeline is psychometrically inferior is actually stated more cautiously in the paper than the data would allow.
If model-assigned difficulty correlates better with experts than with students, then using LLM difficulty scores to construct homogeneous tests could yield tests that feel calibrated to teachers but not to actual student performance; a test-construction experiment would settle this.
The same evaluation scaffold—expert review gates plus Classical Test Theory indices—could be reused to benchmark MCQ generators in other morphologically rich languages.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. First, this is one of the few studies that actually takes LLM-generated MCQs into a classroom: 284 students, 124 expert-validated items, CTT difficulty and discrimination, plus a 3-rule distractor analysis. That is real evidence, and it is rare in this literature. Second, the headline "current models can generate MCQs of comparable quality" is too broad. The two-step Ptt5-v2+Gemma-2 pipeline clearly underperforms (55.6% vs 73.3% answerable), and the paper itself says so; the comparable-quality claim only holds for the one-step GPT-4o and Gemma-2 pipelines.

What is actually new: first evaluation of LLM-generated MCQs in European Portuguese with narrative-element and difficulty control, validated by expert review and student responses. The design is careful: three provenances, balanced forms, majority-vote expert ratings, clear RQs. The psychometric analysis using CTT on real responses is a genuine step up from the usual n-gram or expert-only evaluations.

Where the soft spots are. The abstract overclaim is the main one; it is internally contradicted by the paper's own Table 5. That should be fixed by scoping to GPT-4o and Gemma-2. Second, the psychometric comparison in Section 5 is run only on the 124 MCQs that experts deemed well-formed, clear, and answerable. The paper is transparent about this, but it means the 'comparable psychometric properties' conclusion applies only to items that survive expert screening, not to the unfiltered output of the models. Since the two-step model is filtered harder, the comparison is not apples-to-apples across provenances. This is a real limitation, but it does not sink the paper's core accept/reject claim, which is based on the full 45-item sample. Third, the non-significant Kruskal-Wallis/ANOVA results are used to support 'no difference,' but no effect sizes or confidence intervals are reported; with 33-45 items per group, a null result is weak evidence for equivalence. Fourth, no inter-rater reliability is reported for the expert majority votes.

The citation pattern is fine; the related work is current and relevant. The paper ships no code or data, but the prompt is in an appendix and the evaluation protocol is detailed enough to replicate. If the authors release the item pools and student responses, that would substantially increase the value.

Who this is for: anyone working on multilingual question generation, automated assessment, or practical deployment of LLMs in education for lower-resourced languages. It deserves a serious referee, not a desk reject. For peer review, I'd ask for the abstract rewrite, effect sizes/CIs, a discussion of the filtering bias (and ideally an analysis on unfiltered samples or a correction), and IRR. With those, it would be a solid publication.

Referee Report

4 major / 6 minor

Summary. The paper presents an empirical evaluation of automatic multiple-choice question generation for European Portuguese reading comprehension, targeting elementary-school students. Three generation pipelines are compared with human-authored items: zero-shot one-step generation with GPT-4o, zero-shot one-step generation with Gemma-2-27B, and a two-step pipeline combining fine-tuned Ptt5-v2 (question generation) with Gemma-2 (options and difficulty). Quality is assessed through expert review (18 experts, 180 items) on well-formedness, narrative alignment, option clarity, answerability, distractor plausibility, and difficulty, and through classical test theory applied to 284 students' responses on 124 pre-filtered items, yielding difficulty and discrimination indices. The paper also analyzes model-assigned difficulty against expert and student judgments. The main finding is that one-step LLMs can generate MCQs of comparable quality to human-authored ones, while the two-step pipeline performs substantially worse on answerability.

Significance. If its claims hold, this is a valuable contribution to MCQ generation for low-resource languages, providing the first evaluation of narrative- and difficulty-controlled generation in European Portuguese with real classroom data. The study is carefully designed: expert evaluation uses majority voting, provenance is blinded, student data was collected under exam conditions, and the authors include model prompts and a detailed limitation section. The negative result for the two-step pipeline and the multi-perspective difficulty analysis are useful for practitioners. However, the headline claim of 'comparable quality' needs to be scoped to the specific one-step models tested and supported with effect-size or equivalence evidence, since the psychometric comparisons rely on small, pre-filtered samples and non-significant test results.

major comments (4)

[Abstract, Section 4.6, Section 7] The claim that 'current models can generate MCQs of comparable quality to human-authored ones' is not supported for the two-step pipeline and is internally inconsistent. Section 4.5.4/Table 5 shows Ptt5-v2+Gemma-2 achieves 25/45 (55.6%) answerable items versus 33/45 (73.3%) for human-authored and both one-step LLMs, which the paper itself calls 'substantially lower.' Aggregated across all three pipelines, the LLM acceptance rate is 91/135 (67.4%), below the human 73.3%. Please revise the abstract, summary of findings, and conclusions to scope the comparability claim to the GPT-4o and Gemma-2 one-step pipelines.
[Section 5.2] The psychometric comparison is conducted only on the 124 MCQs that survived expert filtering (well-formed, clear, answerable), so the conclusion in Section 5.7 that generated MCQs are 'generally comparable' applies only to items that already passed a quality gate. Because the two-step pipeline produced many more unanswerable items (67.6% answerable vs. 82.5% for human), the filter disproportionately removes low-quality generated items, potentially biasing the comparison toward comparability. The authors should either add an analysis of the unfiltered output (e.g., treating unanswerable items as incorrect) or explicitly state in the conclusions that the psychometric comparability holds only for expert-validated items.
[Section 5.5] The equivalence claim for difficulty and discrimination rests on non-significant Kruskal-Wallis tests (H=1.249, p=0.741 for difficulty; H=0.854, p>0.05 for discrimination) with small group sizes (33 to 25 items per provenance). Absence of a statistically significant difference is not evidence of equivalence. Please report effect sizes (e.g., epsilon-squared), confidence intervals for group means, and ideally an equivalence test with pre-specified bounds, so readers can judge whether 'comparable' is actually supported.
[Section 6.2, Table 9] The claim in Section 6.4 that models can 'effectively assign difficulty values' is only supported by statistically significant differences for GPT-4o and Ptt5-v2+Gemma-2; for Gemma-2 the differences are not significant for either expert (p=0.2502) or student (p=0.2475) perspectives. This mixed result should be reported as such rather than being subsumed into a general statement about all models.

minor comments (6)

[Throughout] The typo 'plausability' appears in Section 4.4, Figure 4, and elsewhere; it should be 'plausibility.'
[Section 5.5] The text says 'as indicated by the one-way ANOVA test (H=0.854, p>0.05)'; H is the Kruskal-Wallis statistic, so the test name should be corrected for consistency with the difficulty test.
[Table 6] The 'Form ID' labels in Table 6 appear to be the same as those used for the expert review forms in Table 1; please clarify whether these are the same forms converted to paper sheets or a separate set of forms.
[Figure 8] The provenance codes in the panel labels (G7, GE12, P25) are not defined; explain the naming convention so readers can map them to the provenances in Table 8.
[Section 6.3.1] The semantic-similarity features mention the Serafim encoder (reference [63]) but do not specify the similarity measure (presumably cosine) or how the averages are computed over option pairs; please add these details for reproducibility.
[References] References [4] and [16] are the same Alhazmi et al. paper and should be consolidated to avoid duplicate entries.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical evaluation whose predictions are compared against external expert and student data.

full rationale

The paper makes no claimed first-principles derivation; it is an empirical evaluation of generated MCQs. Model-assigned difficulty values are produced by the generators (Section 3) and then compared, rather than fitted, to expert ratings and student-based psychometric estimates (Sections 4.5.6, 5.5, 6.2). The psychometric outcomes and expert judgments are external to the generation process and are not used as training or fitting signals for the evaluated models. Semantic-similarity features in Section 6.3.1 come from an external pre-trained encoder (Serafim), not from the target outcomes. The equal 33/45 acceptance rates in Section 4.6 are observed counts from Table 5 and are not derived by an equation that defines the claim in terms of its inputs. The only self-citation of note is Ref. [58], which supplies the translated FairytaleQA dataset used to fine-tune the ptt5-v2 question generator in Section 3.3; this is a resource, not a load-bearing theorem or a fitted parameter, and the paper's conclusions do not rest on an unverified result imported from that citation. Some reporting choices (e.g., dividing answerable counts by 45 when some items were not evaluated, and restricting the psychometric comparison to expert-cleaned items) raise external-validity concerns, but they are not circularity: no quantity is defined in terms of the quantity it purports to predict, and no fitted value is renamed as a prediction.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

This is an empirical evaluation paper, so the ledger contains no fitted constants and no invented theoretical entities. The load-bearing assumptions are methodological: expert ratings are treated as ground truth, CTT estimates are assumed stable, and the filtered sample is assumed representative. These assumptions are stated in the methods but not independently validated.

assumptions (3)

domain assumption Expert majority rating (at least 2 of 3 raters) is a valid ground truth for MCQ quality dimensions.
The expert review in Section 4.3 uses majority voting with no inter-rater reliability statistics; all subsequent filtering and acceptance conclusions depend on this.
domain assumption Classical Test Theory item statistics (P and D) estimated from a single classroom administration are stable and comparable across provenance groups.
Section 5.4 applies CTT with top and bottom 27% groups; no standard errors or IRT modeling are reported, so sampling variability may affect the group comparisons.
domain assumption The 124 MCQs that passed expert filtering are representative of each provenance's output for psychometric comparison.
Section 5.2 excludes all flawed items, disproportionately affecting Ptt5-v2+Gemma-2; the psychometric results therefore describe only pre-validated items, not full generation output.

how reviews work

0 comments

Cite this review

Pith. "Pith review of From Model to Classroom: Evaluating Generated MCQs for Portuguese with Narrative and Difficulty Concerns." pith.science (2026). https://pith.science/paper/63OGGOOC

@misc{pith2026250615598,
  author       = {Pith},
  title        = {Pith review of: From Model to Classroom: Evaluating Generated MCQs for Portuguese with Narrative and Difficulty Concerns},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/63OGGOOC}},
  note         = {Machine review of arXiv:2506.15598}
}

read the original abstract

While MCQs are valuable for learning and evaluation, manually creating them with varying difficulty levels and targeted reading skills remains a time-consuming and costly task. Recent advances in generative AI provide an opportunity to automate MCQ generation efficiently. However, assessing the actual quality and reliability of generated MCQs has received limited attention -- particularly regarding cases where generation fails. This aspect becomes particularly important when the generated MCQs are meant to be applied in real-world settings. Additionally, most MCQ generation studies focus on English, leaving other languages underexplored. This paper investigates the capabilities of current generative models in producing MCQs for reading comprehension in Portuguese, a morphologically rich language. Our study focuses on generating MCQs that align with curriculum-relevant narrative elements and span different difficulty levels. We evaluate these MCQs through expert review and by analyzing the psychometric properties extracted from student responses to assess their suitability for elementary school students. Our results show that current models can generate MCQs of comparable quality to human-authored ones. However, we identify issues related to semantic clarity and answerability. Also, challenges remain in generating distractors that engage students and meet established criteria for high-quality MCQ option design.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 49 canonical work pages

[1]

Research and Practice in Technology Enhanced Learning16(1), 5 (2021)

Das, B., Majumder, M., Phadikar, S., Sekh, A.A.: Automatic question generation and answer assessment: a survey. Research and Practice in Technology Enhanced Learning16(1), 5 (2021)

work page 2021
[2]

International Journal of Artificial Intelligence in Education30(1), 121–204 (2020)

Kurdi, G., Leo, J., Parsia, B., Sattler, U., Al-Emari, S.: A systematic review of automatic question generation for educational purposes. International Journal of Artificial Intelligence in Education30(1), 121–204 (2020)

work page 2020
[3]

IEEE Transactions on Learning Technologies13(1), 14–25 (2020) https: //doi.org/10.1109/TLT.2018.2889100

CH, D.R., Saha, S.K.: Automatic multiple choice question generation from text: A survey. IEEE Transactions on Learning Technologies13(1), 14–25 (2020) https: //doi.org/10.1109/TLT.2018.2889100

work page arXiv 2020
[5]

Education and Information Technologies29(9), 11483–11515 (2024)

Lee, U., Jung, H., Jeon, Y., Sohn, Y., Hwang, W., Moon, J., Kim, H.: Few-shot is enough: exploring chatgpt prompt engineering method for automatic question generation in english education. Education and Information Technologies29(9), 11483–11515 (2024)

work page 2024
[6]

In: Carpuat, M., Marneffe, M.-C., Meza Ruiz, I.V

Wang, X., Fan, S., Houghton, J., Wang, L.: Towards process-oriented, modu- lar, and versatile question generation that meets educational needs. In: Carpuat, M., Marneffe, M.-C., Meza Ruiz, I.V. (eds.) Proceedings of the 2022 Confer- ence of the North American Chapter of the Association for Computational 19https://mime.dgeec.mec.pt/InqueritoConsultar.aspx?...

work page 2022
[7]

AI Magazine43(2), 239–248 (2022) https://doi.org/10.1002/aaai.12058

Holstein, K., Aleven, V.: Designing for human–ai complementarity in k-12 edu- cation. AI Magazine43(2), 239–248 (2022) https://doi.org/10.1002/aaai.12058

doi:10.1002/aaai.12058 2022
[8]

KI-K¨ unstliche Intelligenz30, 183–188 (2016)

Alsubait, T., Parsia, B., Sattler, U.: Ontology-based multiple choice question generation. KI-K¨ unstliche Intelligenz30, 183–188 (2016)

work page 2016
[9]

European journal of clinical pharmacology80(5), 729–735 (2024)

Kıyak, Y.S., Co¸ skun, ¨O., Budako˘ glu, I.˙I., Uluo˘ glu, C.: Chatgpt for generating multiple-choice questions: evidence on the use of artificial intelligence in auto- matic item generation for a rational pharmacotherapy exam. European journal of clinical pharmacology80(5), 729–735 (2024)

work page 2024

Show all 65 references

[10]

In: Proceedings of the 16th International Conference on Computer Supported Education - Volume 1: AIG, pp

Malec, W.: Investigating the Quality of AI-Generated Distractors for a Multiple- Choice Vocabulary Test. In: Proceedings of the 16th International Conference on Computer Supported Education - Volume 1: AIG, pp. 836–843. SciTePress, ??? (2024). https://doi.org/10.5220/001276240...

2024 doi
[11]

In: Kochmar, E., Burstein, J., Horbach, A., Laarmann-Quante, R., Madnani, N., Tack, A., Yaneva, V., Yuan, Z., Zesch, T

Ashok Kumar, N., Fernandez, N., Wang, Z., Lan, A.: Improving reading comprehension question generation with data augmentation and overgenerate- and-rank. In: Kochmar, E., Burstein, J., Horbach, A., Laarmann-Quante, R., Madnani, N., Tack, A., Yaneva, V., Yuan, Z., Zesch, T. (ed...

2023 doi
[12]

Australasian Journal of Early Childhood39(1), 50–55 (2014)

Sim, S., Berthelsen, D.: Shared book reading by parents with young children: Evidence-based practice. Australasian Journal of Early Childhood39(1), 50–55 (2014)

2014
[13]

Reading Psychology29(4), 327–365 (2008)

Lynch, J.S., Van Den Broek, P., Kremer, K.E., Kendeou, P., White, M.J., Lorch, E.P.: The development of narrative comprehension and its relation to other early reading skills. Reading Psychology29(4), 327–365 (2008)

2008
[14]

In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp

Xu, Y., Wang, D., Yu, M., Ritchie, D., Yao, B., Wu, T., Zhang, Z., Li, T., Brad- ford, N., Sun, B., Hoang, T., Sang, Y., Hou, Y., Ma, X., Yang, D., Peng, N., Yu, Z., Warschauer, M.: Fantastic questions and where to find them: FairytaleQA – an authentic dataset for narrative co...

2022
[15]

In: Rogers, A., Boyd-Graber, J., Okazaki, N

Eo, S., Moon, H., Kim, J., Hur, Y., Kim, J., Lee, S., Chun, C., Park, S., Lim, H.: Towards diverse and effective question-answer pair generation from children 34 storybooks. In: Rogers, A., Boyd-Graber, J., Okazaki, N. (eds.) Findings of the Association for Computational Lingu...

2023
[16]

In: Al-Onaizan, Y., Bansal, M., Chen, Y.-N

Alhazmi, E., Sheng, Q.Z., Zhang, W.E., Zaib, M., Alhazmi, A.: Distractor gen- eration in multiple-choice tasks: A survey of methods, datasets, and evaluation. In: Al-Onaizan, Y., Bansal, M., Chen, Y.-N. (eds.) Proceedings of the 2024 Con- ference on Empirical Methods in Natura...

2024 doi
[17]

In: Tetreault, J., Burstein, J., Leacock, C., Yannakoudakis, H

Jiang, S., Lee, J.: Distractor generation for Chinese fill-in-the-blank items. In: Tetreault, J., Burstein, J., Leacock, C., Yannakoudakis, H. (eds.) Pro- ceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pp. 143–148. Association for...

2017 doi
[18]

In: Proceedings of the 7th International Conference on Computer Sup- ported Education - CSEDU, pp

Susanti, Y., Iida, R., Tokunaga, T.: Automatic Generation of English Vocabulary Tests. In: Proceedings of the 7th International Conference on Computer Sup- ported Education - CSEDU, pp. 77–87. SciTePress, ??? (2015). https://doi.org/ 10.5220/0005437200770087 . INSTICC

2015 doi
[19]

In: SLaTE, pp

Pino, J., Eskenazi, M.: Semi-automatic generation of cloze question distractors effect of students’ l1. In: SLaTE, pp. 65–68 (2009)

2009
[20]

In: Palmer, M., Hwa, R., Riedel, S

Lai, G., Xie, Q., Liu, H., Yang, Y., Hovy, E.: RACE: Large-scale ReAding comprehension dataset from examinations. In: Palmer, M., Hwa, R., Riedel, S. (eds.) Proceedings of the 2017 Conference on Empirical Methods in Nat- ural Language Processing, pp. 785–794. Association for C...

2017 doi
[21]

Proceedings of the AAAI Conference on Artificial Intelligence33(01), 6423–6430 (2019) https://doi.org/ 10.1609/aaai.v33i01.33016423

Gao, Y., Bing, L., Li, P., King, I., Lyu, M.R.: Generating distractors for read- ing comprehension questions from real examinations. Proceedings of the AAAI Conference on Artificial Intelligence33(01), 6423–6430 (2019) https://doi.org/ 10.1609/aaai.v33i01.33016423

2019 doi
[22]

Proceedings of the AAAI Conference on Artificial Intelligence34(05), 9725–9732 (2020) https://doi.org/10.1609/aaai

Zhou, X., Luo, S., Wu, Y.: Co-attention hierarchical network: Generating coherent long distractors for reading comprehension. Proceedings of the AAAI Conference on Artificial Intelligence34(05), 9725–9732 (2020) https://doi.org/10.1609/aaai. v34i05.6522

2020 doi
[23]

Journal of Machine Learning Research21(140), 1–67 (2020)

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text 35 transformer. Journal of Machine Learning Research21(140), 1–67 (2020)

2020
[24]

In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J

Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoy- anov, V., Zettlemoyer, L.: BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J. ...

2020 doi
[25]

In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L.u., Polosukhin, I.: Attention is all you need. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processi...

2017
[26]

Expert Systems with Applications208, 118258 (2022) https://doi.org/10.1016/ j.eswa.2022.118258

Rodriguez-Torrealba, R., Garcia-Lopez, E., Garcia-Cabot, A.: End-to-end gener- ation of multiple-choice questions using text-to-text transfer transformer models. Expert Systems with Applications208, 118258 (2022) https://doi.org/10.1016/ j.eswa.2022.118258

2022
[27]

In: Rogers, A., Boyd- Graber, J., Okazaki, N

Wang, H.-J., Hsieh, K.-Y., Yu, H.-C., Tsou, J.-C., Shih, Y.A., Huang, C.- H., Fan, Y.-C.: Distractor generation based on Text2Text language models with pseudo Kullback-Leibler divergence regulation. In: Rogers, A., Boyd- Graber, J., Okazaki, N. (eds.) Findings of the Associati...

2023 doi
[28]

In: Cal- zolari, N., Kan, M.-Y., Hoste, V., Lenci, A., Sakti, S., Xue, N

Taslimipoor, S., Benedetto, L., Felice, M., Buttery, P.: Distractor generation using generative and discriminative capabilities of transformer-based models. In: Cal- zolari, N., Kan, M.-Y., Hoste, V., Lenci, A., Sakti, S., Xue, N. (eds.) Proceedings of the 2024 Joint Internati...

2024
[29]

In: Meo, R., Silvestri, F

Bitew, S.K., Deleu, J., Develder, C., Demeester, T.: Distractor generation for multiple-choice questions with predictive prompting and large language models. In: Meo, R., Silvestri, F. (eds.) Machine Learning and Principles and Practice of Knowledge Discovery in Databases, pp....

2025
[30]

System123, 103344 (2024) https://doi.org/ 10.1016/j.system.2024.103344

Lin, Z., Chen, H.: Investigating the capability of chatgpt for generating multiple- choice reading comprehension items. System123, 103344 (2024) https://doi.org/ 10.1016/j.system.2024.103344

2024
[31]

In: Proceedings of the 40th Annual Meeting of 36 the Association for Computational Linguistics, pp

Papineni, K., Roukos, S., Ward, T., Zhu, W.-J.: Bleu: a Method for Automatic Evaluation of Machine Translation. In: Proceedings of the 40th Annual Meeting of 36 the Association for Computational Linguistics, pp. 311–318. ACL, Philadelphia, Pennsylvania, USA (2002). https://doi...

2002
[32]

In: Text Summarization Branches Out, pp

Lin, C.-Y.: ROUGE: A Package for Automatic Evaluation of Summaries. In: Text Summarization Branches Out, pp. 74–81. ACL, Barcelona, Spain (2004). https://www.aclweb.org/anthology/W04-1013

2004
[33]

Master’s thesis, Insti- tuto Superior T´ ecnico (2010)

Curto, S.: Automatic generation of multiple-choice tests. Master’s thesis, Insti- tuto Superior T´ ecnico (2010). Dissertation for obtaining the Master Degree in Information Systems and Computer Engineering. https://fenix.tecnico.ulisboa. pt/departamentos/dei/dissertacao/2353642299631

2010
[34]

In: Second Language Studies: Acquisition, Learning, Education and Technology (L2WS 2010), pp

Santos Correia, R.P., Baptista, J., Mamede, N., Trancoso, I., Eskenazi, M.: Auto- matic generation of cloze question distractors. In: Second Language Studies: Acquisition, Learning, Education and Technology (L2WS 2010), pp. 2–11 (2010)

2010
[35]

Master’s thesis, Faculdade de Engenharia da Universidade do Porto (2020)

Leite, B.: Automatic question generation for the portuguese language. Master’s thesis, Faculdade de Engenharia da Universidade do Porto (2020). Disserta- tion for obtaining the Integrated Master Degree in Informatics and Computer Engineering. https://repositorio-aberto.up.pt/h...

2020
[36]

In: Sim˜ oes, A., Ber´ on, M.M., Portela, F

Gon¸ calo Oliveira, H., Caetano, I., Matos, R., Amaro, H.: Generating and Ranking Distractors for Multiple-Choice Questions in Portuguese. In: Sim˜ oes, A., Ber´ on, M.M., Portela, F. (eds.) 12th Symposium on Languages, Appli- cations and Technologies (SLATE 2023). Open Access...

2023 doi
[37]

In: Findings of the Association for Computational Lin- guistics: ACL 2022, pp

Ghanem, B., Lutz Coleman, L., Rivard Dexter, J., Ohe, S., Fyshe, A.: Question generation for reading comprehension assessment by modeling how and what to ask. In: Findings of the Association for Computational Lin- guistics: ACL 2022, pp. 2131–2146. Association for Computationa...

2022 doi
[38]

(eds.) Artificial Intelligence in Edu- cation

Elkins, S., Kochmar, E., Serban, I., Cheung, J.C.K.: How useful are educational questions generated by large language models? In: Wang, N., Rebolledo-Mendez, G., Dimitrova, V., Matsuda, N., Santos, O.C. (eds.) Artificial Intelligence in Edu- cation. Posters and Late Breaking R...

2023
[39]

Theory Into Practice41(4), 212–218 (2002) https://doi.org/10.1207/s15430421tip4104 2 37

Krathwohl, D.R.: A revision of bloom’s taxonomy: An overview. Theory Into Practice41(4), 212–218 (2002) https://doi.org/10.1207/s15430421tip4104 2 37

2002 doi
[40]

In: Proceedings of the 60th Annual Meeting of the Asso- ciation for Computational Linguistics (Volume 1: Long Papers), pp

Zhao, Z., Hou, Y., Wang, D., Yu, M., Liu, C., Ma, X.: Educational question gen- eration of children storybooks via question type distribution learning and event- centric summarization. In: Proceedings of the 60th Annual Meeting of the Asso- ciation for Computational Linguistic...

2022 doi
[41]

In: Wang, N., Rebolledo-Mendez, G., Matsuda, N., San- tos, O.C., Dimitrova, V

Leite, B., Lopes Cardoso, H.: Towards enriched controllability for educational question generation. In: Wang, N., Rebolledo-Mendez, G., Matsuda, N., San- tos, O.C., Dimitrova, V. (eds.) Artificial Intelligence in Education, pp. 786–791. Springer, Cham (2023)

2023
[42]

In: Proceedings of the 16th International Conference on Computer Supported Education - Volume 2: CSEDU, pp

Leite, B., Lopes Cardoso, H.: On Few-Shot Prompting for Controllable Question- Answer Generation in Narrative Comprehension. In: Proceedings of the 16th International Conference on Computer Supported Education - Volume 2: CSEDU, pp. 63–74. SciTePress, ??? (2024). https://doi.o...

2024 doi
[43]

In: Ku, L.-W., Martins, A., Sriku- mar, V

Li, K., Zhang, Y.: Planning first, question second: An LLM-guided method for controllable question generation. In: Ku, L.-W., Martins, A., Sriku- mar, V. (eds.) Findings of the Association for Computational Linguis- tics: ACL 2024, pp. 4715–4729. Association for Computational ...

2024 doi
[44]

In: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pp

Gao, Y., Bing, L., Chen, W., Lyu, M., King, I.: Difficulty control- lable generation of reading comprehension questions. In: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pp. 4968–4974. International Joint Conferences on ...

2019 doi
[45]

In: The Semantic Web – ISWC 2019: 18th International Semantic Web Conference, Auck- land, New Zealand, October 26–30, 2019, Proceedings, Part I, pp

Kumar, V., Hua, Y., Ramakrishnan, G., Qi, G., Gao, L., Li, Y.-F.: Difficulty- controllable multi-hop question generation from knowledge graphs. In: The Semantic Web – ISWC 2019: 18th International Semantic Web Conference, Auck- land, New Zealand, October 26–30, 2019, Proceedin...

2019 doi
[46]

arXiv preprint arXiv:2110.06560 (2021)

Bi, S., Cheng, X., Li, Y.-F., Qu, L., Shen, S., Qi, G., Pan, L., Jiang, Y.: Simple or complex? complexity-controllable question generation with soft templates and deep mixture of experts model. arXiv preprint arXiv:2110.06560 (2021)

2021 arXiv
[47]

Cheng, Y., Li, S., Liu, B., Zhao, R., Li, S., Lin, C., Zheng, Y.: Guiding the growth: Difficulty-controllable question generation through step-by-step rewriting. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internatio...

2021 doi
[48]

In: Kochmar, E., Burstein, J., Horbach, A., Laarmann-Quante, R., Madnani, N., Tack, A., Yaneva, V., Yuan, Z., Zesch, T

Uto, M., Tomikawa, Y., Suzuki, A.: Difficulty-controllable neural question gen- eration for reading comprehension using item response theory. In: Kochmar, E., Burstein, J., Horbach, A., Laarmann-Quante, R., Madnani, N., Tack, A., Yaneva, V., Yuan, Z., Zesch, T. (eds.) Proceedi...

2023 doi
[49]

IEEE Transactions on Learning Technologies17, 2240–2252 (2024) https://doi

Tomikawa, Y., Suzuki, A., Uto, M.: Adaptive question–answer generation with difficulty control using item response theory and pretrained transformer models. IEEE Transactions on Learning Technologies17, 2240–2252 (2024) https://doi. org/10.1109/TLT.2024.3491801

2024
[50]

In: Proceedings of the 8th International Conference on Knowledge Capture

Lin, C., Liu, D., Pang, W., Apeh, E.: Automatically predicting quiz difficulty level using similarity measures. In: Proceedings of the 8th International Conference on Knowledge Capture. K-CAP 2015. Association for Computing Machin- ery, New York, NY, USA (2015). https://doi.or...

2015
[51]

In: OWL: Experiences And Directions–reasoner Evaluation, pp

Kurdi, G., Parsia, B., Sattler, U.: An experimental evaluation of automatically generated multiple choice questions from ontologies. In: OWL: Experiences And Directions–reasoner Evaluation, pp. 24–39. Springer, ??? (2016)

2016
[52]

Research and practice in technology enhanced learning12(1), 1–16 (2017)

Susanti, Y., Tokunaga, T., Nishikawa, H., Obari, H.: Controlling item difficulty for automatic vocabulary question generation. Research and practice in technology enhanced learning12(1), 1–16 (2017)

2017
[53]

In: Olney, A.M., Chounta, I.-A., Liu, Z., Santos, O.C., Bittencourt, I.I

Tomikawa, Y., Uto, M.: Difficulty-controllable multiple-choice question generation for reading comprehension using item response theory. In: Olney, A.M., Chounta, I.-A., Liu, Z., Santos, O.C., Bittencourt, I.I. (eds.) Artificial Intelligence in Edu- cation. Posters and Late Br...

2024
[54]

Internet of Things and Cyber-Physical Systems3, 121–154 (2023) https://doi.org/10.1016/j.iotcps.2023

Ray, P.P.: Chatgpt: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems3, 121–154 (2023) https://doi.org/10.1016/j.iotcps.2023. 04.003

2023 doi
[55]

International Medical Education2(3), 198–205 (2023) https://doi.org/10.3390/ime2030019

Heston, T.F., Khun, C.: Prompt engineering in medical education. International Medical Education2(3), 198–205 (2023) https://doi.org/10.3390/ime2030019

2023 doi
[56]

In: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

Wu, T., Terry, M., Cai, C.J.: Ai chains: Transparent and controllable human-ai 39 interaction by chaining large language model prompts. In: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. CHI ’22. Association for Computing Machinery, New York, NY,...

2022
[57]

In: Brazilian Conference on Intelligent Systems, pp

Piau, M., Lotufo, R., Nogueira, R.: ptt5-v2: A closer look at continued pretraining of t5 models for the portuguese language. In: Brazilian Conference on Intelligent Systems, pp. 324–338 (2024). Springer

2024
[58]

In: Fer- reira Mello, R., Rummel, N., Jivet, I., Pishtari, G., Ruip´ erez Valiente, J.A

Leite, B., Os´ orio, T.F., Cardoso, H.L.: Fairytaleqa translated: Enabling edu- cational question and answer generation in less-resourced languages. In: Fer- reira Mello, R., Rummel, N., Jivet, I., Pishtari, G., Ruip´ erez Valiente, J.A. (eds.) Technology Enhanced Learning for...

2024
[59]

Rivista di Grammatica Generativa, vol

Haegeman, L.: Subject omission in present-day written english. Rivista di Grammatica Generativa, vol. 32 (2007), p. 91-124 (2007)

2007
[60]

Rout- ledge, ??? (2004)

Haladyna, T.M.: Developing and Validating Multiple-choice Test Items. Rout- ledge, ??? (2004)

2004
[61]

ERIC, ??? (1986)

Crocker, L., Algina, J.: Introduction to Classical and Modern Test Theory. ERIC, ??? (1986)

1986
[62]

R Journal10(2) (2018)

Martinkov´ a, P., Drabinov´ a, A.: Shinyitemanalysis for teaching psychometrics and to enforce routine analysis of educational tests. R Journal10(2) (2018)

2018
[63]

In: EPIA Conference on Artificial Intelligence, pp

Rodrigues, J., Gomes, L., Silva, J., Branco, A., Santos, R., Cardoso, H.L., Os´ orio, T.: Advancing neural encoding of portuguese with transformer albertina pt. In: EPIA Conference on Artificial Intelligence, pp. 441–453 (2023). Springer

2023
[64]

In: Olney, A.M., Chounta, I.-A., Liu, Z., Santos, O.C., Bitten- court, I.I

Scaria, N., Dharani Chenna, S., Subramani, D.: Automated educational question generation at different bloom’s skill levels using large language models: Strategies and evaluation. In: Olney, A.M., Chounta, I.-A., Liu, Z., Santos, O.C., Bitten- court, I.I. (eds.) Artificial Inte...

2024
[65]

Reading Research Quarterly38(1), 36–76 (2003)

Paris, A.H., Paris, S.G.: Assessing narrative comprehension in young children. Reading Research Quarterly38(1), 36–76 (2003)

2003
[66]

Who...?” ); • Setting: Focusing on the time and place of events, often starting with “Where...?

Alonzo, J., Basaraba, D., Tindal, G., Carriveau, R.S.: They read, but how well do they understand? an empirical look at the nuances of measuring reading comprehension. Assessment for Effective Intervention35(1), 34–44 (2009) Appendix A Narrative Elements The narrative elements...

2009

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Research and Practice in Technology Enhanced Learning16(1), 5 (2021)

Das, B., Majumder, M., Phadikar, S., Sekh, A.A.: Automatic question generation and answer assessment: a survey. Research and Practice in Technology Enhanced Learning16(1), 5 (2021)

work page 2021

[2] [2]

International Journal of Artificial Intelligence in Education30(1), 121–204 (2020)

Kurdi, G., Leo, J., Parsia, B., Sattler, U., Al-Emari, S.: A systematic review of automatic question generation for educational purposes. International Journal of Artificial Intelligence in Education30(1), 121–204 (2020)

work page 2020

[3] [3]

IEEE Transactions on Learning Technologies13(1), 14–25 (2020) https: //doi.org/10.1109/TLT.2018.2889100

CH, D.R., Saha, S.K.: Automatic multiple choice question generation from text: A survey. IEEE Transactions on Learning Technologies13(1), 14–25 (2020) https: //doi.org/10.1109/TLT.2018.2889100

work page arXiv 2020

[4] [5]

Education and Information Technologies29(9), 11483–11515 (2024)

Lee, U., Jung, H., Jeon, Y., Sohn, Y., Hwang, W., Moon, J., Kim, H.: Few-shot is enough: exploring chatgpt prompt engineering method for automatic question generation in english education. Education and Information Technologies29(9), 11483–11515 (2024)

work page 2024

[5] [6]

In: Carpuat, M., Marneffe, M.-C., Meza Ruiz, I.V

Wang, X., Fan, S., Houghton, J., Wang, L.: Towards process-oriented, modu- lar, and versatile question generation that meets educational needs. In: Carpuat, M., Marneffe, M.-C., Meza Ruiz, I.V. (eds.) Proceedings of the 2022 Confer- ence of the North American Chapter of the Association for Computational 19https://mime.dgeec.mec.pt/InqueritoConsultar.aspx?...

work page 2022

[6] [7]

AI Magazine43(2), 239–248 (2022) https://doi.org/10.1002/aaai.12058

Holstein, K., Aleven, V.: Designing for human–ai complementarity in k-12 edu- cation. AI Magazine43(2), 239–248 (2022) https://doi.org/10.1002/aaai.12058

doi:10.1002/aaai.12058 2022

[7] [8]

KI-K¨ unstliche Intelligenz30, 183–188 (2016)

Alsubait, T., Parsia, B., Sattler, U.: Ontology-based multiple choice question generation. KI-K¨ unstliche Intelligenz30, 183–188 (2016)

work page 2016

[8] [9]

European journal of clinical pharmacology80(5), 729–735 (2024)

Kıyak, Y.S., Co¸ skun, ¨O., Budako˘ glu, I.˙I., Uluo˘ glu, C.: Chatgpt for generating multiple-choice questions: evidence on the use of artificial intelligence in auto- matic item generation for a rational pharmacotherapy exam. European journal of clinical pharmacology80(5), 729–735 (2024)

work page 2024