Generating AI Literacy MCQs: A Multi-Agent LLM Approach

Jiayi Wang; Ruiwei Xiao; Ying-Jui Tseng

REVIEW 4 major objections 5 minor 1 cited by

Generating AI Literacy MCQs: A Multi-Agent LLM Approach

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a multi-agent LLM workflow can generate high-quality AI literacy MCQs, supported by expert ratings averaging 84.2% usability.

desk verdict Feasibility demo of multi-agent LLM MCQ generation for K-12 AI literacy with a clearly described system and honest limitations—but the headline 84.2% usability statistic conflates as-is and rephrased items, so the central claim overreaches. read the letter →

arxiv 2412.00970 v1 pith:QUBJXB66 submitted 2024-12-01 cs.HC

Jiayi Wang , Ruiwei Xiao , Ying-Jui Tseng This is my paper · ORCID

classification cs.HC

keywords AIliteracymultiple-choicequestiongenerationlargelanguagemodelsmulti-agentsystemsK-12educationBloom'staxonomyqualityevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that an LLM-powered multi-agent workflow can generate multiple-choice questions for K-12 AI literacy that teachers would actually use. It proposes a pipeline in which one agent drafts a question and two critique agents review it for readability, grade alignment, and common item-writing flaws, with a supervisor approving or sending it back for revision. To test this, the authors generated 40 questions for grades 7-9 and had three experienced K-12 AI literacy teachers rate them on a ten-item rubric. The ratings were largely positive: clarity and answerability exceeded 93% agreement, and on average 84.2% of the questions were judged usable in class. The paper treats this as preliminary evidence that the approach can help fill the shortage of scalable AI literacy assessment materials.

What carries the argument

The load-bearing mechanism is the iterative multi-agent critique loop. A Generator Agent produces an initial question with a stem, a correct key, and distractors; a Language Critique Agent checks readability and grade-level fit; an IWF (Item-Writing Flaw) Critique Agent applies rule-based checks such as implausible distractors and absolute terms; and a Supervisor Agent either approves the question or sends it back for revision. This loop is what turns raw LLM output into questions that meet pedagogical standards.

What would settle it

A classroom trial in which grades 7-9 students take the generated MCQs and their results are compared with a validated AI literacy measure; if the questions fail to separate students who understand AI concepts from those who do not, or if teachers reject them in practice, the central claim would be undercut.

Watch

Extended reading notes

Core claim

The central claim is that LLM-powered multi-agent workflows can effectively generate high-quality MCQs for AI literacy. The system operates by taking user-provided learning objectives, grade levels, Bloom's Taxonomy levels, and optional scenarios, then running an iterative generate-critique-revise loop until a question passes quality checks. In the authors' evaluation, three experts with K-12 AI literacy teaching experience rated all 40 questions; the system's correct answers matched expert judgment for 85-97.5% of questions, criteria like Understandable and Answerable drew yes ratings 93.3-99.2% of the time, and willingness to use the question in class averaged 84.2%. The authors conclude that the multi-agent system demonstrates strong potential to generate pedagogically sound and scalable high-quality assessment questions for AI literacy, while noting that the system has not been tested in real-world classrooms.

Load-bearing premise

The expert rubric ratings are a reliable proxy for whether the questions will actually help students learn in a real classroom.

Editorial extensions

If this is right

If the workflow works, teachers can type a learning objective and grade level and receive draft MCQs aligned with Bloom's Taxonomy levels, reducing the time needed to build AI literacy assessments.
The two-critic design automatically catches language and item-writing flaws, so common MCQ pitfalls are addressed before a human sees the item.
Because the pipeline is model-driven, the same workflow could be extended to other grade bands, subjects, and question formats without redesigning the agent structure.
Expert disagreement on rephrasing and grade level suggests that end-user customization or adjustable strictness would be needed for broad adoption.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A likely next step, implied but not tested here, is that the same generate-critique loop would work for other under-resourced assessment domains, such as digital citizenship or data literacy, where item banks are thin.
The strongest check on the central claim would be a classroom study measuring whether students who learn from these questions show measurable gains in AI literacy; expert rubric ratings alone cannot establish learning impact.
Because the evaluation relies on three experts, the 84.2% usability figure has wide uncertainty; a crowd-sourced or student-performance-based validation would either strengthen or weaken the conclusion.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short workshop paper, so calibrate expectations. The system is a multi-agent LangGraph pipeline: one generator, two critique agents (language and item-writing flaws), and a supervisor that iterates until clean. The application to K-12 AI literacy is new, and the sample question is concrete. The authors adapt an existing rubric, report three experts on 40 questions, and are transparent in the Limitations section that there is no classroom testing yet. That honesty is real credit.

The main soft spot is exactly what the stress-test note says: the WouldYouUseIt rubric item asks the expert to pick this/rephrased/both/neither, and the paper reports the 84.2% average without the per-category breakdown. Given Expert 2 flagged 97.5% as needing rephrasing, a large share of those positive responses are likely 'both' or 'rephrased' rather than 'this.' The abstract's claim that 'experts expressed strong interest in using the LLM-generated MCQs' is therefore not supported by the reported data. That is an internal measurement issue, not an external generalizability issue, and it should be fixed from existing data before the paper's central claim is taken at face value.

Two smaller issues: no baseline (single-agent or no-critique condition), so the specific value of the multi-agent loop is untested; and no released prompts, code, or data, which limits reproducibility. Neither is fatal for a two-page exploratory paper, but a baseline would make the contribution much clearer.

What does work: expert agreement on answer correctness (85-97.5%) and on clarity/correctness (93-99%) is a genuinely useful signal. The system is sensible, the writing is clear, and the limitations are stated rather than hidden.

I would send this to peer review. The research question is real, the system is concrete, and the main flaw is a reporting gap that can be addressed from data the authors already have. A referee should ask for the WouldYouUseIt breakdown and a single-agent comparison. After that, this could be a solid workshop paper.

Referee Report

4 major / 5 minor

Summary. The paper proposes a multi-agent LLM system that generates K-7 to K-9 AI-literacy multiple-choice questions from user-specified learning objectives, grade level, and Bloom's Taxonomy level. The system uses a generator agent, two critique agents (language and item-writing-flaw), and a supervisor agent in an iterative LangGraph workflow with gpt-4o-mini. Three experts rated the 40 generated questions on a ten-item rubric. The paper reports high aggregate agreement on clarity, grammaticality, learning-objective relevance, and centrality, and a 84.2% average WouldYouUseIt score, while documenting disagreements on rephrasing, Bloom's level, and grade level. It concludes that the multi-agent approach shows strong potential for generating scalable AI-literacy assessment items.

Significance. If the central claim is established, the pipeline would be a practical contribution to K-12 AI literacy assessment, where scalable materials are lacking. The paper is a useful pilot: it describes a concrete workflow, provides a sample question, adapts a rubric, and includes an honest discussion of limitations in Section 4, including the absence of classroom testing and the subjectivity of expert judgments. The main value is demonstrating that LLM-based generation with critique agents can produce questions that experienced teachers find mostly usable, often after rephrasing. However, the evaluation is preliminary: the headline usability number conflates as-is use with post-hoc rephrasing, there is no baseline condition, and agreement statistics are missing. These issues must be resolved before the term high-quality is fully supported.

major comments (4)

[§3, Table 1] The WouldYouUseIt rubric item has response options this/rephrased/both/neither, yet the reported 84.2% average pools these categories. Given that Expert 2 judged 97.5% of the 40 questions as needing rephrasing, the pooled average does not establish that the generated MCQs are usable as produced; it may largely reflect expert willingness to repair them. Please report the per-option breakdown, or per-expert distributions, and adjust the abstract's claim of strong interest in using the LLM-generated MCQs accordingly.
[§2 (Figure 1), §3] The research question asks whether multi-agent workflows can effectively generate high-quality MCQs, but the evaluation does not compare the full workflow against any baseline, such as single-agent generation without critique or a non-iterative prompt. The results therefore do not show that the multi-agent architecture or the critique loop contributes to the observed quality. An ablation or baseline comparison is needed to support the central claim.
[§3] The expert evaluation lacks chance-corrected inter-rater reliability statistics despite substantial disagreement: Bloom's Level was rated appropriate by Expert 1 for only 35% of questions while Experts 2 and 3 rated 100%, and Rephrase rates ranged from 7.5% to 97.5%. Reporting only average percentages over three raters is not sufficient for a rubric-based evaluation; provide per-rater item-level data and a metric such as Fleiss' kappa.
[§4] The paper explicitly acknowledges that the system has not been tested in real-world classrooms and that expert judgments are subjective. Since the research question uses the term high-quality, the manuscript should either validate the expert rubric against an external outcome such as classroom use or student performance, or restrict the conclusion to expert-rated quality. As written, the conclusion that the system demonstrates strong potential is acceptable, but the research-question phrasing overshoots the evidence.

minor comments (5)

[§1] The AI4K12 Five Big Ideas are misstated: the list repeats Natural Interaction and omits Societal Impact.
[§2] The acceptance criterion that a question with 0 or 1 flaw is considered acceptable should specify whether the flaws are summed across the two critique agents and how conflicting critique feedback is resolved by the Supervisor Agent.
[§3, Table 1] The Rephrase rubric item is phrased as Could you rephrase the question with yes/no options, but the results interpret a yes response as the question needing rephrasing; the wording should be clarified to avoid ambiguity.
[§2] Reporting the distribution of revision counts, such as how many questions were approved after 0, 1, 2, or more iterations and how many reached the maximum revision count, would help readers assess the workflow's efficiency and yield.
[§3] Making the 40 generated questions and the full per-expert item-level rubric ratings available in supplementary material would substantially improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is supported by external expert rubric judgments, and self-citations are contextual only.

full rationale

The paper's derivation chain is an empirical system-building and evaluation study, not a mathematical derivation. The multi-agent workflow generates MCQs from user inputs, and the central claim that the questions are high-quality is supported by ratings from three external experts using a ten-item rubric adapted from Scaria et al. (reference [7]), which is not authored by the present paper's authors. The IWF critique agent's rule-based checks are modified from Moore et al. (reference [6]), also not a self-citation. The only self-citations, references [8] and [10], support background assertions about LLM-based feedback and the scalability/reliability gap in AI literacy resources; neither is used to justify the evaluation outcome or to force the system's design choices. No parameter is fitted to the evaluation data and then renamed as a prediction; no uniqueness theorem is imported; no ansatz is smuggled in via citation. The reader-identified concern that the 84.2% WouldYouUseIt average may conflate as-is and rephrased items is a measurement-validity issue, not circularity, because the expert ratings are independent of the system's outputs and the paper explicitly concedes that the system 'has not been tested in real-world classrooms' and that expert judgment 'introduces potential subjectivity.' The claim stands or falls on external evaluation rather than on the paper's own definitions or prior self-citations, so the appropriate circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper has no fitted parameters in the physics sense, but the hand-set system thresholds, undeclared revision limit, and model choice are effectively free parameters. The load-bearing assumptions are that expert rubric ratings measure quality and that the acceptance threshold is valid. No prompts or thresholds are released, which hampers reproducibility.

free parameters (3)

Acceptance threshold for IWF flaws = 0 or 1
A question with 0 or 1 rule-based flaw is accepted (Section 2). This cutoff is hand-chosen and directly determines which questions survive.
Maximum revision count = not disclosed
The workflow 'continues until the question meets quality standards or reaches the maximum number of revisions' (Section 2), but the number is not given, so the stopping rule is underspecified.
LLM model selection = gpt-4o-mini-2024-07-18
The generator and critique agents all use this model, chosen without ablation or comparison, though output quality is expected to depend on it.

assumptions (3)

domain assumption Expert rubric judgments are a valid measure of MCQ quality.
The entire evaluation in Section 3 relies on three experts' ratings; the paper notes in Section 4 that expert judgment 'introduces potential subjectivity'.
ad hoc to paper A question with 0 or 1 item-writing flaws is pedagogically acceptable.
Adopted in Section 2 from Moore et al. [6], the cutoff is not independently validated against student outcomes.
domain assumption AI4K12's Five Big Ideas adequately scope AI literacy assessment content.
The learning objectives are aligned to this framework (Section 1); no content-coverage validation is reported.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Generating AI Literacy MCQs: A Multi-Agent LLM Approach." pith.science (2026). https://pith.science/paper/QUBJXB66

@misc{pith2026241200970,
  author       = {Pith},
  title        = {Pith review of: Generating AI Literacy MCQs: A Multi-Agent LLM Approach},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QUBJXB66}},
  note         = {Machine review of arXiv:2412.00970}
}

read the original abstract

Artificial intelligence (AI) is transforming society, making it crucial to prepare the next generation through AI literacy in K-12 education. However, scalable and reliable AI literacy materials and assessment resources are lacking. To address this gap, our study presents a novel approach to generating multiple-choice questions (MCQs) for AI literacy assessments. Our method utilizes large language models (LLMs) to automatically generate scalable, high-quality assessment questions. These questions align with user-provided learning objectives, grade levels, and Bloom's Taxonomy levels. We introduce an iterative workflow incorporating LLM-powered critique agents to ensure the generated questions meet pedagogical standards. In the preliminary evaluation, experts expressed strong interest in using the LLM-generated MCQs, indicating that this system could enrich existing AI literacy materials and provide a valuable addition to the toolkit of K-12 educators.

Figures

Figures reproduced from arXiv: 2412.00970 by the authors.

**Figure 1.** Multi-Agent MCQ Generation System. The Multi-Agent MCQ Generation System developed for this study generates and refines MCQs to assess AI literacy, using a workflow built on the LangGraph framework2 , and OpenAI’s gpt-4o-mini-2024-07-18 model. User inputs, including learning objectives, Bloom’s taxonomy levels [1], grade level, and specific scenarios, guide the Generator Agent in producing an initial question with f… view at source ↗

**Figure 2.** Evaluation Result were seen as suitable for classroom use, demonstrating that the multi-agent system can produce questions that are not only wellformed but also useful in practical educational settings. However, the diverse responses suggest that individual preferences or teaching styles influence the willingness to use certain questions. Overall, while some refinement is necessary to accommodate diverse educatio… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

CODE-GEN: A Human-in-the-Loop RAG-Based Agentic AI System for Multiple-Choice Question Generation
cs.AI 2026-04 conditional novelty 5.0 of 10

A tool-augmented dual-agent RAG system generates and validates coding MCQs, reaching 79.9–98.6% human-validated success and showing AI handles computational checks while humans remain essential for pedagogical depth.

Reference graph

Works this paper leans on

14 extracted references · 11 canonical work pages · cited by 1 Pith paper

[1]

Anderson and David R

Lorin W. Anderson and David R. Krathwohl (Eds.). 2001.A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives (complete ed ed.). Longman, New York

work page 2001
[2]

Andrew Caines, Luca Benedetto, Shiva Taslimipoor, Christopher Davis, Yuan Gao, Oeistein Andersen, Zheng Yuan, Mark Elliott, Russell Moore, Christopher Bryant, et al. 2023. On the application of large language models for language teaching and assessment technology. arXiv preprint arXiv:2307.08393 (2023)

arXiv 2023
[3]

Jacob Doughty, Zipiao Wan, Anishka Bompelli, Jubahed Qayum, Taozhi Wang, Juran Zhang, Yujia Zheng, Aidan Doyle, Pragnya Sridhar, Arav Agarwal, et al
[4]

Christina Gardner-McCune, David Touretzky, Fred Martin, and Deborah Seehorn

work page
[5]

Inthrani Raja Indran, Priya Paranthaman, Neelima Gupta, and Nurulhuda Mustafa

work page
[6]

Steven Moore, Huy A Nguyen, Tianying Chen, and John Stamper. 2023. Assessing the quality of multiple-choice questions using gpt-4 and rule-based methods. In European Conference on Technology Enhanced Learning . Springer, 229–245

work page 2023
[7]

Nicy Scaria, Suma Dharani Chenna, and Deepak Subramani. 2024. Automated Educational Question Generation at Different Bloom’s Skill Levels Using Large Language Models: Strategies and Evaluation. In International Conference on Arti- ficial Intelligence in Education . Springer, 165–179

work page 2024
[8]

Medical Teacher (2024), 1–6

Twelve tips to leverage AI for efficient and effective medical question generation: a guide for educators using Chat GPT. Medical Teacher (2024), 1–6

work page 2024

Show all 14 references

[9]

Touretzky, Christina Gardner-Mccune, and Deborah W

David S. Touretzky, Christina Gardner-Mccune, and Deborah W. Seehorn. 2022. Machine Learning and the Five Big Ideas in AI. International Journal of Artificial Intelligence in Education 33 (2022), 233–266

2022
[10]

Ying-Jui Tseng, Ruiwei Xiao, Christopher Bogart, Jaromir Savelka, and Majd Sakr
[11]

John Stamper, Ruiwei Xiao, and Xinying Hou. 2024. Enhancing llm-based feed- back: Insights from intelligent tutoring systems and the learning sciences. In International Conference on Artificial Intelligence in Education . Springer, 32–43

2024
[14]

In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V

Assessing the Efficacy of Goal-Based Scenarios in Scaling AI Literacy for Non-Technical Learners. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 2. 1842–1843
[2019]

In Proceedings of the 50th ACM Technical Symposium on Computer Science Education

AI for K-12: Making room for AI in K-12 CS curricula. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education . 1244–1244
[2024]

In Proceedings of the 26th Australasian Computing Education Conference

A comparative study of AI-generated (GPT-4) and human-crafted MCQs in programming education. In Proceedings of the 26th Australasian Computing Education Conference. 114–123

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Anderson and David R

Lorin W. Anderson and David R. Krathwohl (Eds.). 2001.A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives (complete ed ed.). Longman, New York

work page 2001

[2] [2]

Andrew Caines, Luca Benedetto, Shiva Taslimipoor, Christopher Davis, Yuan Gao, Oeistein Andersen, Zheng Yuan, Mark Elliott, Russell Moore, Christopher Bryant, et al. 2023. On the application of large language models for language teaching and assessment technology. arXiv preprint arXiv:2307.08393 (2023)

arXiv 2023

[3] [3]

Jacob Doughty, Zipiao Wan, Anishka Bompelli, Jubahed Qayum, Taozhi Wang, Juran Zhang, Yujia Zheng, Aidan Doyle, Pragnya Sridhar, Arav Agarwal, et al

[4] [4]

Christina Gardner-McCune, David Touretzky, Fred Martin, and Deborah Seehorn

work page

[5] [5]

Inthrani Raja Indran, Priya Paranthaman, Neelima Gupta, and Nurulhuda Mustafa

work page

[6] [6]

Steven Moore, Huy A Nguyen, Tianying Chen, and John Stamper. 2023. Assessing the quality of multiple-choice questions using gpt-4 and rule-based methods. In European Conference on Technology Enhanced Learning . Springer, 229–245

work page 2023

[7] [7]

Nicy Scaria, Suma Dharani Chenna, and Deepak Subramani. 2024. Automated Educational Question Generation at Different Bloom’s Skill Levels Using Large Language Models: Strategies and Evaluation. In International Conference on Arti- ficial Intelligence in Education . Springer, 165–179

work page 2024

[8] [8]

Medical Teacher (2024), 1–6

Twelve tips to leverage AI for efficient and effective medical question generation: a guide for educators using Chat GPT. Medical Teacher (2024), 1–6

work page 2024