Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

Do LLMs Give Psychometrically Plausible Responses in Educational Assessments?

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Zero-shot instruction-tuned LLMs are not psychometrically plausible simulated test takers: their correct-answer probabilities correlate weakly with human item difficulty, and in some history items the correlation is significantly negative.

desk verdict Careful, transparent benchmark showing weak psychometric plausibility for LLMs, but the result rests on a first-token proxy rather than actual generated answers. read the letter →

arxiv 2506.09796 v1 pith:R3QQITW5 submitted 2025-06-11 cs.CL

Andreas Säuberli , Diego Frassinelli , Barbara Plank This is my paper · ORCID
classification cs.CL
keywords psychometricplausibilitylargelanguagemodelsclassicaltesttheoryitemresponseeducationalassessmentmultiple-choiceitemstemperaturescalingsimulatedtakers
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a zero-shot instruction-tuned large language model can stand in for a human test taker in pilot testing of multiple-choice educational items, so that item quality could be checked without recruiting hundreds of students. To answer it, the authors evaluate 18 open-weight instruction-tuned models on items from two publicly available datasets of multiple-choice items, one U.S. national assessment covering reading, history, and economics and one English-as-a-second-language reading dataset, using two standard psychometric frameworks: classical test theory (item facility) and item response theory (three-parameter logistic item characteristic curves). Their evidence is that LLM correct-answer probabilities correlate only weakly with human item difficulty—mostly below 0.5, often not significantly, and sometimes significantly negatively in history items under the IRT analysis—and models are poor at predicting which wrong options attract humans. The conclusion is that zero-shot instruction-tuned LLMs are not psychometrically plausible simulated test takers, so they should not be used to pilot educational assessments in that setting.

What carries the argument

The machinery is the pair of psychometric lenses applied to LLM response distributions. In classical test theory (CTT), the item facility—the proportion of humans who answer correctly—is the anchor statistic: a plausible model should assign higher probability to the correct answer on easier items, measured by Pearson correlation across items. In item response theory (IRT), the three-parameter logistic (3PL) model $P(X=1)=c+(1-c)/(1+e^{-a(\theta-b)})$ defines the item characteristic curve; the paper plugs in $\theta=0$ (an average-ability test taker, the mean of the ability scale) and compares the predicted human correct probability for each item with the LLM's temperature-scaled correct probability, again by correlation. The LLM response distribution itself is produced by a softmax over first-token logits for the four answer letters, averaged over four cyclic permutations of the options to remove order effects, then sharpness-adjusted by a temperature parameter fitted to minimize KL divergence to the human distribution.

What would settle it

Re-run the full evaluation scoring the models' actual generated text answers—greedy decoding or sampling the same prompt and reading the stated letter—and correlate those responses with human item facilities and IRT expected probabilities. If the text-based correlations are substantially higher than the first-token correlations, the paper's conclusion applies to the logit proxy rather than to the LLMs' test-taking behavior.

Watch

Extended reading notes

Core claim

The central claim is that human-like response behavior in educational assessments has not emerged from the process of training instruction-tuned LLMs. The authors define psychometric plausibility as agreement with human item characteristics derivable from classical test theory and item response theory, and they operationalize it with two correlation analyses: between a model's probability of the correct answer and the human item facility across items (CTT), and between that probability and the probability expected for an average-ability test taker under the 3PL IRT model (IRT). Across 18 models, they find the correlations are not strong overall—the best are in reading items (up to 0.56 in one subset), most cells lack significant correlations, and some 4th-grade history IRT scales show significant negative correlations, meaning models are more confident on items humans find harder. They also show that temperature-calibrated distributions approach human KL divergence for large models but rarely beat a simple oracle baseline that ignores distractor identity, indicating LLMs are good at the correct answer but not at predicting distractors. The paper states its conclusion directly: LLMs should not be used for piloting educational assessments in a zero-shot setting.

Load-bearing premise

The load-bearing assumption is that the first-token logit distribution over the four answer letters (after cyclic-permutation averaging) faithfully represents how the LLM would answer the item; the paper cites evidence that in instruction-tuned models first-token probabilities can diverge from text answers, yet all of its correlations and divergences are computed on that proxy.

Editorial extensions

If this is right

  • Test developers should not rely on zero-shot instruction-tuned LLM responses to pilot multiple-choice items: item difficulty rankings, discrimination indices, and distractor analyses would be systematically off.
  • LLM distributions cannot support evaluation of the wrong answer options, because models assign far less probability than humans do to attractive distractors.
  • Psychometric plausibility is subject-dependent: reading comprehension responses are more human-like than history or economics responses, so any simulated test taker needs per-subject validation.
  • Model family, model size, and temperature scaling all fail to close the gap with human response patterns, pointing to fine-tuning on human response distributions as the necessary next step.
  • The CTT/IRT correlation framework presented here is a reusable benchmark for judging whether any LLM-based simulated test taker is psychometrically plausible.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the first-token proxy understates how instruction-tuned models actually answer, the paper's negative conclusion may be too pessimistic for text-based responses; a replication with greedily decoded or sampled answers could show higher facility correlations, especially in reading.
  • The significantly negative IRT correlations on some history items suggest the models' internal ordering of fact difficulty is inverted relative to human difficulty for knowledge-retrieval questions; a testable extension is whether adding image access (the history items include pictures) or retrieved context flips the correlations positive.
  • A practical diagnostic implied by the 'not easily distracted' finding is to measure KL divergence restricted to distractor probability mass; that metric isolates guessing behavior from accuracy and could serve as a standalone human-likeness benchmark.
  • If fine-tuning on human response distributions is the path forward, the same CTT and IRT correlations could be used as the training objective and the acceptance test; one would predict that fine-tuned models beat the OracleBaseline on distractor distributions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper investigates whether zero-shot instruction-tuned LLMs produce psychometrically plausible responses to multiple-choice test items, i.e., response distributions and item-level statistics that resemble those of human test takers. Using two public datasets (NAEP and CMCQRD) across three subjects, the authors evaluate 18 open-weight instruction-tuned LLMs under three analyses: average KL divergence between model and human response distributions, CTT-based correlation between LLM correct-response probabilities and human item facilities, and IRT-based correlation between LLM probabilities and expected human probabilities at average ability. They apply temperature scaling calibrated per model and item subset, explicitly labeling the results as an upper bound. They find that larger models are overconfident without calibration, that temperature scaling improves KL divergence but that LLMs generally do not outperform a simple oracle baseline in distractor modeling, and that correlations with human item statistics are weak overall, occasionally negative (e.g., some 4th-grade history scales). The paper concludes that zero-shot instruction-tuned LLMs are not sufficiently psychometrically plausible for piloting educational assessments.

Significance. If the conclusion holds, this is a useful negative result: test developers should be cautious about using current open-weight instruction-tuned LLMs as substitutes for human pilots. The paper's methodology is transparent and reproducible: the code is released, baselines (uniform and oracle) are included, bootstrap confidence intervals are provided, and the in-sample temperature optimization is explicitly disclosed and framed as an upper bound. The breadth of models (18) and datasets (two, three subjects, multiple grades/proficiency levels) strengthens the empirical grounding. The main contribution is methodological: a concrete CTT/IRT-based evaluation protocol for assessing psychometric plausibility of LLM responses, which future work can reuse.

major comments (2)
  1. [Section 4.3 and Sections 5.1-5.3] The entire evaluation uses the softmax of first-token logits for the four answer letters as the LLM response distribution. The paper cites Wang et al. (2024) for option-order sensitivity, but that paper's central finding is that first-token probabilities do not match the distribution of text answers in instruction-tuned LLMs. This mismatch is not discussed or controlled for. All reported KL divergences, CTT correlations, and IRT correlations therefore characterize a proxy rather than the distribution of answers a test developer would observe when eliciting a response from the model. If first-token and final-answer distributions diverge, the central conclusion that 'LLMs should not be used for piloting educational assessments in a zero-shot setting' is not directly supported for actual LLM responses. The authors should either validate the proxy on a subset (e.g., by sampling complete answers with the same cyclic permutation setup and recomputing the main metrics) or explicitly limit the conclusion to first-token probabilities and justify why that quantity is the relevant one for psychometric piloting.
  2. [Section 5.2 and Section 7] The conclusion that correlations are 'not very strong overall' and hence that LLMs should not be used for piloting lacks an explicit threshold or decision-theoretic criterion. The paper does not specify what level of correlation (or KL divergence) would be needed for a piloting application, such as preserving item ordering or identifying low-quality items. Without such a criterion, 'not very strong' is a qualitative judgment, and the strength of the negative conclusion is difficult to calibrate. Since the measurements are also in-sample after temperature optimization, the authors should either define an a priori acceptability cut-off or soften the practical recommendation to reflect that no evidence of sufficient psychometric plausibility was found under an upper-bound evaluation.
minor comments (5)
  1. [Abstract] There are missing spaces in 'human-likeness orpsychometric plausibility' and 'classical test theoryanditem response theory' in the abstract.
  2. [Section 6] The first paragraph refers to the 'CCT analysis'; this should be 'CTT analysis'.
  3. [Section 5.1] The description of OracleBaseline says it is 'optimized using the same temperature scaling approach as the other models,' but as an oracle it has no free temperature parameter; consider clarifying that the temperature is irrelevant or fixed for this baseline.
  4. [Figures 3 and 4] Significance is assessed with two-tailed tests at p < 0.05 without correction for multiple comparisons; given the large number of tests across models and subsets, some of the marked correlations are expected to be false positives. A note on this, or a correction such as Benjamini-Hochberg, would help interpret the significant entries.
  5. [Section 4.4] The in-sample temperature optimization is disclosed clearly, but the discussion section could more explicitly state that the cross-subject and cross-model comparisons (e.g., 'reading correlates better than history') are also based on in-sample fits, so the relative differences may be optimistic as well.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the weak-psychometric-plausibility result is an empirical benchmark, not a consequence of the paper's fitting choices.

full rationale

The paper makes no claim that is forced by its own construction. The central negative claim—that zero-shot instruction-tuned LLMs show weak psychometric plausibility—is an empirical benchmark result obtained by correlating LLM response probabilities with human item facilities and IRT-based expected probabilities; these correlations are not defined in terms of the quantities they support. The temperature-scaling step (Section 4.4) is an in-sample calibration: the optimizer minimizes KL divergence on the same item subsets used for evaluation, so the reported KL values are best-case fits rather than out-of-sample predictions. However, the paper explicitly discloses this and labels the results an upper bound, and the main conclusion does not rest on the temperature fit succeeding; the CTT and IRT correlations are low even under this favorable calibration, so the negative finding is conservative rather than manufactured. The first-token logit proxy (Section 4.3) is a validity limitation—the paper's own cited work (Wang et al., 2024) suggests first-token probabilities may diverge from generated text answers—but it is not a circular step because the paper defines the evaluated distribution as the first-token distribution and does not identify it with final answers. Self-citations to prior work by the authors (Plank 2022; Baan et al. 2022; Wang et al. 2024; Chen et al. 2024) are background for calibration and option-order sensitivity; none is invoked as a uniqueness theorem or as the sole support for the central claim, so none is load-bearing. The human IRT 'upper bound' row is a sanity check, not an input to the LLM evaluations. The derivation chain is self-contained against external human data and simple baselines.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central empirical contribution rests on imported IRT parameters and aggregated human response data, one fitted calibration parameter (temperature, fit in-sample), and one unvalidated measurement proxy (first-token probabilities). No new entities are postulated.

free parameters (1)
  • Temperature scaling parameter T = optimized per model and item subset, values in Figure 6 (roughly 1 to 30)
    T minimizes KL divergence between LLM and human response distributions on the same item subsets used for evaluation; this in-sample fit makes all calibrated results best-case upper bounds (Section 4.4).
assumptions (5)
  • standard math The 3PL item response theory model (Eq. 1) with NAEP's published item parameters correctly describes human response probabilities.
    Used in Section 4.5 to compute expected correct-response probabilities for an average test taker; parameters are imported from NAEP's scaling rather than re-estimated.
  • domain assumption NAEP ability parameters are scaled to mean zero, so theta=0 corresponds to the average test taker.
    Section 4.5 states this and uses it to specialize Eq. 1 to Eq. 2.
  • domain assumption Softmax over first-token letter logits, averaged over four cyclic option permutations, faithfully represents the model's response distribution.
    Section 4.3 defines this extraction; the paper does not validate it against generated text answers.
  • domain assumption Public NAEP and CMCQRD aggregated response distributions are valid human reference data.
    Section 4.1 selects these datasets; the entire comparison treats these aggregates as ground truth.
  • domain assumption Items with images are answerable from alternative text with manual exclusion of unanswerable items.
    Section 4.1 describes alt-text substitution and manual filtering; the authors note this may affect history and economics results (Discussion).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do LLMs Give Psychometrically Plausible Responses in Educational Assessments?." pith.science (2026). https://pith.science/paper/R3QQITW5

@misc{pith2026250609796,
  author       = {Pith},
  title        = {Pith review of: Do LLMs Give Psychometrically Plausible Responses in Educational Assessments?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R3QQITW5}},
  note         = {Machine review of arXiv:2506.09796}
}
read the original abstract

Knowing how test takers answer items in educational assessments is essential for test development, to evaluate item quality, and to improve test validity. However, this process usually requires extensive pilot studies with human participants. If large language models (LLMs) exhibit human-like response behavior to test items, this could open up the possibility of using them as pilot participants to accelerate test development. In this paper, we evaluate the human-likeness or psychometric plausibility of responses from 18 instruction-tuned LLMs with two publicly available datasets of multiple-choice test items across three subjects: reading, U.S. history, and economics. Our methodology builds on two theoretical frameworks from psychometrics which are commonly used in educational assessment, classical test theory and item response theory. The results show that while larger models are excessively confident, their response distributions can be more human-like when calibrated with temperature scaling. In addition, we find that LLMs tend to correlate better with humans in reading comprehension items compared to other subjects. However, the correlations are not very strong overall, indicating that LLMs should not be used for piloting educational assessments in a zero-shot setting.

Figures

Figures reproduced from arXiv: 2506.09796 by the authors.

Figure 1
Figure 1. Example item from the NAEP dataset and illustration of our psychometric analyses of LLM re￾sponses. We use the first-token probabilities produced by LLMs and analyze how well they correspond to hu￾man test taker responses. Specifically, we look at (a) the similarity between LLM and human response distribu￾tions, (b) whether items that are difficult for humans are also difficult for LLMs, and (c) how well response pr… view at source ↗
Figure 2
Figure 2. Mean KL divergence between temperature-scaled LLM response probability distributions and human [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Pearson correlation between LLM correct re [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Mode accuracy across item subsets, models, baselines, and humans. Error bars are bootstrapped 95% [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Optimized temperature value for each model and item subset. [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Mean KL divergence between LLM response probability distributions [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Pearson correlation between all LLM correct response probabilities and human item facilities. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Defense Against LLM Backdoors using Critical Neuron Isolation Pruning

    cs.CR 2026-07 conditional novelty 5.0 of 10

    A trigger-inversion plus activation-difference pruning pipeline removes LLM backdoors with ~0.1% neuron intervention and >95% relative ASR reduction.

Reference graph

Works this paper leans on

39 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, and 110 others. 2024 a . https://doi.org/10.48550/ARXIV.2404.14219 Phi-3 technical ...

    arXiv doi:10.48550/arxiv.2404.14219 2024

  2. [2]

    Hewett, Mojan Javaheripi, Piero Kauffmann, James R

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, and 8 others. 2024 b . https://doi.org/10.48550/ARXIV.2412.08905 P...

    arXiv doi:10.48550/arxiv.2412.08905 2024

  3. [3]

    LaFlair, Kevin Yancey, Sarah Goodwin, Yena Park, and Alina A

    Yigal Attali, Andrew Runge, Geoffrey T. LaFlair, Kevin Yancey, Sarah Goodwin, Yena Park, and Alina A. von Davier. 2022. https://doi.org/10.3389/frai.2022.903077 The interactive reading task: Transformer-based automatic item generation . Frontiers in Artificial Intelligence, 5

    work page arXiv 2022

  4. [4]

    Joris Baan, Wilker Aziz, Barbara Plank, and Raquel Fernandez. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.124 Stop measuring calibration when humans disagree . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1892--1915. Association for Computational Linguistics

    doi:10.18653/v1/2022.emnlp-main.124 2022

  5. [5]

    Matthew Byrd and Shashank Srivastava. 2022. https://doi.org/10.18653/v1/2022.acl-short.15 Predicting difficulty and discrimination of natural language questions . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 119--130. Association for Computational Linguistics

    doi:10.18653/v1/2022.acl-short.15 2022

  6. [6]

    Hua-Hua Chang, Chun Wang, and Susu Zhang. 2021. https://doi.org/10.1146/annurev-statistics-042720-104044 Statistical applications in educational measurement . Annual Review of Statistics and Its Application, 8(1):439--461

    work page doi:10.1146/annurev-statistics-042720-104044 2021

  7. [7]

    Beiduo Chen, Xinpeng Wang, Siyao Peng, Robert Litschko, Anna Korhonen, and Barbara Plank. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.842 S eeing the big through the small : Can LLM s approximate human judgment distributions on NLI from a few explanations? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 14396--144...

    doi:10.18653/v1/2024.findings-emnlp.842 2024

  8. [8]

    Xitao Fan. 1998. https://doi.org/10.1177/0013164498058003001 Item response theory and classical test theory: An empirical comparison of their item/person statistics . Educational and Psychological Measurement, 58(3):357--381

    work page doi:10.1177/0013164498058003001 1998

Show all 39 references
  1. [9]

    Guher Gorgun and Okan Bulut. 2024. https://doi.org/10.1111/emip.12663 Instruction-tuned large-language models for quality control in automatic item generation: A feasibility study . Educational Measurement: Issues and Practice

    2024 doi

  2. [10]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 doi arXiv

  3. [11]

    Rita Green. 2020. Pilot testing: Why and how we trial. In The Routledge Handbook of Second Language Acquisition and Language Testing, chapter 11, pages 115--124. Routledge

    2020

  4. [12]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. https://proceedings.mlr.press/v70/guo17a.html On calibration of modern neural networks . In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Resea...

    2017

  5. [13]

    Haladyna

    Thomas M. Haladyna. 2013. Automatic item generation: A historical perspective. In Mark J. Gierl and Thomas M. Haladyna, editors, Automatic Item Generation: Theory and Practice, chapter 2, pages 13--25. Routledge, New York

    2013

  6. [14]

    Hambleton and Russell W

    Ronald K. Hambleton and Russell W. Jones. 1993. https://doi.org/10.1111/j.1745-3992.1993.tb00543.x An NCME instructional module on: Comparison of classical test theory and item response theory and their applications to test development . Educational Measurement: Issues and Pra...

    1993

  7. [15]

    Akio Hayakawa and Horacio Saggion. 2024. https://openreview.net/forum?id=Gu47GKK85N Can LLM s solve reading comprehension tests as second language learners? In Fourth Workshop on Knowledge-infused Learning

    2024

  8. [16]

    Ivan Hernandez and Weiwen Nie. 2022. https://doi.org/10.1111/peps.12543 The AI‐IP : Minimizing the guesswork of personality scale item development through artificial intelligence . Personnel Psychology, 76(4):1011--1035

    2022 doi

  9. [17]

    Ghader Kurdi, Jared Leo, Bijan Parsia, Uli Sattler, and Salam Al-Emari. 2019. https://doi.org/10.1007/s40593-019-00186-y A systematic review of automatic question generation for educational purposes . International Journal of Artificial Intelligence in Education, 30(1):121--204

    2019 doi

  10. [18]

    Lalor, Hao Wu, and Hong Yu

    John P. Lalor, Hao Wu, and Hong Yu. 2019. https://doi.org/10.18653/v1/D19-1434 Learning latent parameters without human response patterns: Item response theory with artificial crowds . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing an...

    2019 doi

  11. [19]

    Antonio Laverghetta Jr, Animesh Nighojkar, Jamshidbek Mirzakhalov, and John Licato. 2022. https://doi.org/10.1007/978-3-031-04572-1_12 Predicting human psychometric properties using computational language models . In Quantitative Psychology, pages 151--169, Cham. Springer Inte...

    2022 doi

  12. [20]

    Adian Liusie, Vatsal Raina, Andrew Mullooly, Kate Knill, and Mark J. F. Gales. 2023. https://doi.org/10.48550/ARXIV.2306.13047 Analysis of the Cambridge Multiple-Choice Questions Reading Dataset with a focus on candidate response distribution . arXiv

    2023 doi arXiv

  13. [21]

    Livingston

    Samuel A. Livingston. 2011. Item analysis. In Steven M. Downing and Thomas M. Haladyna, editors, Handbook of Test Development, pages 421--441. Taylor & Francis Group

    2011

  14. [22]

    Xinyi Lu and Xu Wang. 2024. https://doi.org/10.1145/3657604.3662031 Generative students: Using LLM -simulated student profiles to support question item evaluation . In Proceedings of the Eleventh ACM Conference on Learning @ Scale, L@S ’24, pages 16--27. ACM

    2024

  15. [23]

    May, Yiyun Kate Fan, Gregory E

    Toni A. May, Yiyun Kate Fan, Gregory E. Stone, Kristin L. K. Koskey, Connor J. Sondergeld, Timothy D. Folger, James N. Archer, Kathleen Provinzano, and Carla C. Johnson. 2025. https://doi.org/10.3390/educsci15020144 An effectiveness study of generative artificial intelligence ...

    2025 doi

  16. [24]

    Andrew Mullooly, Øistein Andersen, Luca Benedetto, Paula Buttery, Andrew Caines, Mark J. F. Gales, Yasin Karatay, Kate Knill, Adian Liusie, Vatsal Raina, and Shiva Taslimipoor. 2023. https://doi.org/10.17863/CAM.102185 The Cambridge Multiple-Choice Questions Reading Dataset . ...

    2023 doi

  17. [25]

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, an...

    2025 doi arXiv

  18. [26]

    Valentine Joseph Owan, Kinsgley Bekom Abang, Delight Omoji Idika, Eugene Onor Etta, and Bassey Asuquo Bassey. 2023. https://doi.org/10.29333/ejmste/13428 Exploring the potential of artificial intelligence tools in educational measurement and assessment . Eurasia Journal of Mat...

    2023 doi

  19. [27]

    Norris, Pablo Garcia Gomez, Venessa F

    Spiros Papageorgiou, Larry Davis, John M. Norris, Pablo Garcia Gomez, Venessa F. Manna, and Lora Monfils. 2021. https://www.ets.org/Media/Research/pdf/RM-21-03.pdf Design Framework for the TOEFL Essentials Test 2021 . Educational Testing Service

    2021

  20. [28]

    Jae-Woo Park, Seong-Jin Park, Hyun-Sik Won, and Kang-Min Kim. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.477 Large language models are students at various levels: Zero-shot question difficulty estimation . In Findings of the Association for Computational Linguistics...

    2024 doi

  21. [29]

    Pouya Pezeshkpour and Estevam Hruschka. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.130 Large language models sensitivity to the order of options in multiple-choice questions . In Findings of the Association for Computational Linguistics: NAACL 2024. Association for ...

    2024 doi

  22. [30]

    Barbara Plank. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.731 The “problem” of human label variation: On ground truth in data, modeling and evaluation . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computatio...

    2022 doi

  23. [31]

    Team Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 others. 2024. https://doi.org/10.48...

    2024 doi arXiv

  24. [32]

    Vatsal Raina and Mark Gales. 2022. https://doi.org/10.48550/ARXIV.2209.11830 Multiple-choice question generation: Towards an automated assessment framework . arXiv

    2022 doi arXiv

  25. [33]

    Vatsal Raina, Adian Liusie, and Mark Gales. 2023. https://doi.org/10.21437/slate.2023-1 Analyzing multiple-choice reading and listening comprehension tests . In 9th Workshop on Speech and Language Technology in Education (SLaTE), pages 1--5. ISCA

    2023 doi

  26. [34]

    Andreas S \"a uberli and Simon Clematide. 2024. https://aclanthology.org/2024.readi-1.3/ Automatic generation and evaluation of reading comprehension test items with large language models . In Proceedings of the 3rd Workshop on Tools and Resources for People with REAding DIffi...

    2024

  27. [35]

    Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul R \"o ttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. 2024. https://doi.org/10.18653/v1/2024.findings-acl.441 `` M y answer is C '' : First-token probabilities do not match text answers in instruction-tuned lang...

    2024 doi

  28. [36]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and ...

    2020 doi

  29. [37]

    Victoria Yaneva, Kai North, Peter Baldwin, Le An Ha, Saed Rezayi, Yiyun Zhou, Sagnik Ray Choudhury, Polina Harik, and Brian Clauser. 2024. https://aclanthology.org/2024.bea-1.39 Findings from the first shared task on automated prediction of difficulty and response time for mul...

    2024

  30. [38]

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024. https://openreview.net/forum?id=shr9PXz7T0 Large language models are not robust multiple choice selectors . In The Twelfth International Conference on Learning Representations

    2024

  31. [39]

    Leonidas Zotos, Hedderik van Rijn, and Malvina Nissim. 2025. https://aclanthology.org/2025.coling-main.749/ Can model uncertainty function as a proxy for multiple-choice question item difficulty? In Proceedings of the 31st International Conference on Computational Linguistics,...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.