pith. sign in

arxiv: 2606.28186 · v1 · pith:Q3CROSNMnew · submitted 2026-06-26 · 💻 cs.CL · cs.AI· cs.CY· cs.LG

Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction

Chenguang Wang , Ming Li , Xinyue Zeng , Zhuochun Li , Hong Jiao , Tianyi Zhou , Dawei Zhou This is my paper

Pith reviewed 2026-06-29 03:54 UTC · model grok-4.3

classification 💻 cs.CL cs.AIcs.CYcs.LG
keywords cognitive episodesitem difficulty predictionlarge reasoning modelsreasoning traceseducational assessmentprocess modeling
0
0 comments X

The pith

Cognitive episodes from reasoning model traces predict human item difficulty better than text features alone.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that item difficulty for humans stems from the cognitive burden of solving it, which large reasoning models reveal through their step-by-step traces. It introduces a method to break these traces into sequences of functional episodes that represent states such as planning and execution. These episode features, when added to item semantics, improve prediction accuracy on four human difficulty datasets and allow interpretation of why items are hard. The work shows that harder items trigger more iterative and implementation-focused episode patterns rather than simply longer outputs. This shifts difficulty estimation from static text analysis to dynamic process modeling.

Core claim

Epi2Diff converts LRM reasoning traces into cognitively grounded episode sequences, from which it derives features of reasoning scale, effort allocation, and state transitions to predict human item difficulty, outperforming baselines by up to 8.1 percent relative gain on SAT classification tasks.

What carries the argument

Epi2Diff framework that segments reasoning traces into episode sequences representing functional problem-solving states.

If this is right

  • Harder items produce more effortful, iterative, and implementation-centered episode dynamics.
  • Difficulty prediction benefits from combining episode-dynamic features with semantic representations.
  • Process evidence from models provides interpretable insights into cognitive demands beyond item text length.
  • Scalable prediction of human difficulty is possible without extensive human calibration for each item.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • This approach could enable automated generation of test items calibrated to specific difficulty levels by simulating episode patterns.
  • Episode analysis might identify which problem-solving stages create the greatest burden for learners.
  • Similar episode extraction could apply to other human performance predictions where reasoning traces are available.

Load-bearing premise

The problem-solving states identified in model traces align with the cognitive processes that make items difficult for humans.

What would settle it

Finding a collection of items where adding episode features from model traces does not improve difficulty prediction accuracy compared to using only the item text or response length.

read the original abstract

Predicting human item difficulty is central to educational assessment, where reliable estimates support fairness and effective test construction. Existing methods often depend on costly human calibration or item-level textual representations, providing limited evidence about the cognitive processes that make items difficult. We argue that difficulty should be viewed not only as a property of item text, but also as an observable consequence of the problem-solving burden an item induces. Large Reasoning Models (LRMs) offer scalable process evidence through reasoning traces, but such evidence must be structured to support interpretable modeling. To this end, we introduce Epi2Diff (Episode to Difficulty), a framework that maps LRM reasoning traces into cognitively grounded episode sequences. These episodes group trace segments into functional problem-solving states, enabling difficulty to be modeled through reasoning scale, effort allocation, and state transitions. Epi2Diff extracts compact episode-dynamic features and combines them with semantic item representations for human difficulty prediction. Experiments on four real-world human difficulty datasets show that Epi2Diff consistently outperforms strong baselines, including fine-tuned small language models, LLM in-context learning, and supervised LLM adaptation. On SAT-derived classification benchmarks, Epi2Diff achieves an 8.1% average relative gain over supervised LLM fine-tuning baselines. Further analyses show that harder items induce more effortful, iterative, and implementation-centered episode dynamics, rather than merely longer responses. These results demonstrate that cognitive episodes in LRM reasoning traces provide a predictive and interpretable process representation for human item difficulty, offering a new lens for educational measurement with reasoning models.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper introduces Epi2Diff, a framework that maps LRM reasoning traces into cognitively grounded episode sequences representing functional problem-solving states. These episodes enable modeling of difficulty via reasoning scale, effort allocation, and state transitions; compact episode-dynamic features are combined with semantic item representations for prediction. Experiments on four real-world human difficulty datasets show consistent outperformance over baselines (fine-tuned SLMs, LLM ICL, supervised LLM adaptation), with an 8.1% average relative gain on SAT-derived classification benchmarks; further analyses indicate harder items induce more effortful, iterative, and implementation-centered dynamics.

Significance. If the central claim holds, the work supplies a scalable process-based lens for item difficulty that moves beyond text-only or human-calibrated methods, potentially improving fairness and test construction in educational measurement by linking observable reasoning burden to difficulty labels.

major comments (2)
  1. [Abstract] Abstract: the claim that extracted episode states correspond to the cognitive processes determining human item difficulty is load-bearing for both the predictive gains and the 'interpretable process representation' argument, yet no independent validation (human protocol analysis, eye-tracking, or expert cognitive coding) is supplied to distinguish LRM-specific artifacts from genuine human-process alignment.
  2. [Abstract] Abstract: without details on episode extraction rules, feature definitions, statistical controls, or dataset characteristics, it is impossible to assess whether episode-dynamic features reduce to fitted quantities that tautologically predict the difficulty labels or whether the 8.1% gain is robust.
minor comments (1)
  1. [Abstract] Abstract: naming the four real-world datasets would strengthen the generalizability claim.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We respond point-by-point to the major comments below, indicating revisions where appropriate.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the claim that extracted episode states correspond to the cognitive processes determining human item difficulty is load-bearing for both the predictive gains and the 'interpretable process representation' argument, yet no independent validation (human protocol analysis, eye-tracking, or expert cognitive coding) is supplied to distinguish LRM-specific artifacts from genuine human-process alignment.

    Authors: We agree that the absence of direct validation against human cognitive data (such as protocol analysis or eye-tracking) is a substantive limitation for the stronger claims of process alignment. Our evidence for the episodes' relevance rests on (i) consistent predictive gains over semantic-only baselines across four datasets and (ii) post-hoc analyses showing that episode dynamics (e.g., iteration count, implementation focus) scale with human difficulty labels in theoretically expected directions. These results are indirect. We have added an explicit Limitations subsection that states the lack of independent human validation, qualifies the interpretability argument accordingly, and identifies direct cognitive validation as a priority for future work. This revision does not change the reported experiments but improves transparency. revision: partial

  2. Referee: [Abstract] Abstract: without details on episode extraction rules, feature definitions, statistical controls, or dataset characteristics, it is impossible to assess whether episode-dynamic features reduce to fitted quantities that tautologically predict the difficulty labels or whether the 8.1% gain is robust.

    Authors: The full manuscript supplies these details in the main text and appendices. Episode extraction rules and segmentation criteria appear in Section 3.2 (with pseudocode). Feature definitions (reasoning scale, effort allocation, transition probabilities) are formalized in Section 3.3. Statistical controls, including cross-validation, multiple random seeds, and significance testing, are described in Section 4.2. Dataset characteristics (size, source, label distributions, and preprocessing) are reported in Section 4.1 and Appendix A. We have inserted a reproducibility checklist and expanded the experimental setup paragraph to make these elements easier to locate. The 8.1% relative gain is accompanied by standard deviations across runs and datasets; the gains remain after controlling for response length, indicating the features are not merely length proxies. revision: no

Circularity Check

0 steps flagged

No significant circularity in Epi2Diff derivation chain

full rationale

The paper defines Epi2Diff as a framework extracting episode-dynamic features from LRM reasoning traces (grouping segments into functional states) and combines them with semantic representations to predict human difficulty on external datasets. Experiments report empirical gains (e.g., 8.1% relative improvement) over baselines on four real-world human difficulty datasets. No equations, self-citations, or method descriptions in the abstract reduce the claimed predictive or interpretive result to its inputs by construction; the cognitive correspondence is presented as an interpretive lens supported by performance, not a definitional or fitted tautology. The derivation remains self-contained against the reported external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Abstract-only review yields no explicit free parameters or invented entities; the core domain assumption is the cognitive correspondence between LRM episodes and human difficulty.

axioms (1)
  • domain assumption LRM reasoning traces contain observable functional states that reflect the problem-solving burden experienced by humans
    Invoked when the abstract states that difficulty should be viewed as a consequence of the problem-solving burden an item induces and that episodes enable modeling through reasoning scale and state transitions.

pith-pipeline@v0.9.1-grok · 5833 in / 1156 out tokens · 68933 ms · 2026-06-29T03:54:27.773850+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 1 canonical work pages

  1. [1]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matt...

    Pith/arXiv arXiv

  2. [2]

    Reasoning language models: A blueprint.arXiv preprint arXiv:2501.11223,

    Maciej Besta, Julia Barth, Eric Schreiber, Ales Kubicek, Afonso Catarino, Robert Gerstenberger, Piotr Nyczyk, Patrick Iff, Yueling Li, Sam Houliston, et al. Reasoning language models: A blueprint.arXiv preprint arXiv:2501.11223,

    arXiv

  3. [3]

    Thought anchors: Which llm reasoning steps matter? arXiv preprint arXiv:2506.19143,

    Paul C Bogdan, Uzay Macar, Neel Nanda, and Arthur Conmy. Thought anchors: Which llm reasoning steps matter? arXiv preprint arXiv:2506.19143,

    arXiv

  4. [4]

    Han Chen, Ming Li, Chenguang Wang, Yijun Liang, Dawei Zhou, Hong Jiao, and Tianyi Zhou. LLMs struggle to measure what distinguishes students of different proficiency levels: A study of item discrimination in reading comprehension assessment.arXiv preprint arXiv:2606.18709,

    Pith/arXiv arXiv

  5. [5]

    Do not think that much for 2+ 3=? on the overthinking of o1-like llms.arXiv preprint arXiv:2412.21187,

    Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of o1-like llms.arXiv preprint arXiv:2412.21187,

    Pith/arXiv arXiv

  6. [6]

    Le, and Christopher D

    Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. Electra: Pre-training text encoders as discriminators rather than generators, 2020.https://arxiv.org/abs/2003.10555. Ricardo Conejo, Eduardo Guzmán, Jose-Luis Perez-De-La-Cruz, and Beatriz Barros. An empirical study on the quantitative notion of task difficulty.Expert Systems with Appl...

    Pith/arXiv arXiv 2020

  7. [7]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    11 Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186,

    2019

  8. [8]

    Longreasonarena: A long reasoning benchmark for large language models.arXiv preprint arXiv:2508.19363,

    Jiayu Ding, Shuming Ma, Lei Cui, Nanning Zheng, and Furu Wei. Longreasonarena: A long reasoning benchmark for large language models.arXiv preprint arXiv:2508.19363,

    arXiv

  9. [9]

    Upn-icc at bea 2024 shared task: Leveraging llms for multiple-choice questions difficulty prediction

    George Dueñas, Sergio Jimenez, and Geral Mateus Ferro. Upn-icc at bea 2024 shared task: Leveraging llms for multiple-choice questions difficulty prediction. InProceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), pages 542–550,

    2024

  10. [10]

    Missing premise exacerbates overthinking: Are reasoning models losing critical thinking skill?arXiv preprint arXiv:2504.06514,

    Chenrui Fan, Ming Li, Lichao Sun, and Tianyi Zhou. Missing premise exacerbates overthinking: Are reasoning models losing critical thinking skill?arXiv preprint arXiv:2504.06514,

    arXiv

  11. [11]

    Reasoning and sampling-augmented mcq difficulty prediction via llms

    Wanyong Feng, Peter Tran, Stephen Sireci, and Andrew S Lan. Reasoning and sampling-augmented mcq difficulty prediction via llms. InInternational Conference on Artificial Intelligence in Education, pages 31–45. Springer, 2025a. Yunzhen Feng, Julia Kempe, Cheng Zhang, Parag Jain, and Anthony Hartshorn. What characterizes effective reasoning? revisiting leng...

    arXiv 2024

  12. [12]

    Omni-math: A universal olympiad level mathematic benchmark for large language models.arXiv preprint arXiv:2410.07985,

    Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-math: A universal olympiad level mathematic benchmark for large language models.arXiv preprint arXiv:2410.07985,

    Pith/arXiv arXiv

  13. [13]

    The llama 3 herd of models, 2024.https://arxiv.org/abs/2407.21783

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava S...

    Pith/arXiv arXiv 2024

  14. [14]

    Openai o1 system card.arXiv preprint arXiv:2412.16720,

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720,

    Pith/arXiv arXiv

  15. [15]

    Can llms estimate student struggles? human- ai difficulty alignment with proficiency simulation for item difficulty prediction.arXiv preprint arXiv:2512.18880, 2025a

    Ming Li, Han Chen, Yunze Xiao, Jian Chen, Hong Jiao, and Tianyi Zhou. Can llms estimate student struggles? human- ai difficulty alignment with proficiency simulation for item difficulty prediction.arXiv preprint arXiv:2512.18880, 2025a. Ming Li, Chenrui Fan, Yize Cheng, Soheil Feizi, and Tianyi Zhou. Schoenfeld’s anatomy of mathematical reasoning by langu...

    Pith/arXiv arXiv 2025

  16. [16]

    Textual complexity as a predictor of difficulty of listening items in language proficiency tests

    Anastassia Loukina, Su-Youn Yoon, Jennifer Sakano, Youhua Wei, and Kathy Sheehan. Textual complexity as a predictor of difficulty of listening items in language proficiency tests. InProceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3245–3253,

    2016

  17. [17]

    Jump-starting item parameters for adaptive language tests

    Arya D McCarthy, Kevin P Yancey, Geoffrey T LaFlair, Jesse Egbert, Manqian Liao, and Burr Settles. Jump-starting item parameters for adaptive language tests. InProceedings of the 2021 conference on empirical methods in natural language processing, pages 883–899,

    2021

  18. [18]

    s1: Simple test-time scaling

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori B Hashimoto. s1: Simple test-time scaling. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 20286–20332,

    2025

  19. [19]

    Shadi Noroozi and Hossein Karami

    doi: 10.17863/CAM.102185.https://www.repository.cam.ac.uk/handle/1810/358683. Shadi Noroozi and Hossein Karami. A scrutiny of the relationship between cognitive load and difficulty estimates of language test items.Language Testing in Asia, 12(1):13,

    work page doi:10.17863/cam.102185.https://www.repository.cam.ac.uk/handle/1810/358683

  20. [20]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner...

    Pith/arXiv arXiv 2024

  21. [21]

    Text-based approaches to item difficulty modeling in large-scale assessments: A systematic review.arXiv preprint arXiv:2509.23486,

    Sydney Peters, Nan Zhang, Hong Jiao, Ming Li, Tianyi Zhou, and Robert Lissitz. Text-based approaches to item difficulty modeling in large-scale assessments: A systematic review.arXiv preprint arXiv:2509.23486,

    arXiv

  22. [22]

    Estimating item difficulty using large language models and tree-based machine learning algorithms.arXiv preprint arXiv:2504.08804,

    Pooya Razavi and Sonya J Powers. Estimating item difficulty using large language models and tree-based machine learning algorithms.arXiv preprint arXiv:2504.08804,

    arXiv

  23. [23]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 3982–3992,

    2019

  24. [24]

    Unibucllm: Harnessing llms for automated prediction of item difficulty and response time for multiple-choice questions

    Ana-Cristina Rogoz and Radu Tudor Ionescu. Unibucllm: Harnessing llms for automated prediction of item difficulty and response time for multiple-choice questions. InProceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), pages 493–502,

    2024

  25. [25]

    Openai gpt-5 system card, 2025.https://arxiv.org/abs/2601.03267

    Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, Akshay Nathan, Alan Luo, Alec Helyar, Aleksander Madry, Aleksandr Efremov, Aleksandra Spyra, Alex Baker-Whitcomb, Alex Beutel, Alex Karpenko, Alex Makelov, Alex Neitz, Alex Wei, Alexandra Barr, Alexandre Kirchmeyer, Ale...

    Pith/arXiv arXiv 2025

  26. [26]

    Itec at bea 2024 shared task: Predicting difficulty and response time of medical exam questions with statistical, machine learning, and language models

    Anaïs Tack, Siem Buseyne, Changsheng Chen, Robbe D’hondt, Michiel De Vrindt, Alireza Gharahighehi, Sameh Metwaly, Felipe Kenji Nakano, and Ann-Sophie Noreillie. Itec at bea 2024 shared task: Predicting difficulty and response time of medical exam questions with statistical, machine learning, and language models. InProceedings of the 19th Workshop on Innov...

    2024

  27. [27]

    Qwq-32b: Embracing the power of reinforcement learning, March 2025.https://qwenlm.github.io/blog/ qwq-32b/

    Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025.https://qwenlm.github.io/blog/ qwq-32b/. Hariram Veeramani, Surendrabikram Thapa, Natarajan Balaji Shankar, and Abeer Alwan. Large language model-based pipeline for item difficulty and response time estimation for educational assessments. InProceedings of the 19th Workshop on In...

    2025

  28. [28]

    Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference, 2024.https://arxiv.org/abs/2412.13663

    15 Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference,...

    Pith/arXiv arXiv 2024

  29. [29]

    Self-rewarding correction for mathematical reasoning.arXiv preprint arXiv:2502.19613,

    Wei Xiong, Hanning Zhang, Chenlu Ye, Lichang Chen, Nan Jiang, and Tong Zhang. Self-rewarding correction for mathematical reasoning.arXiv preprint arXiv:2502.19613,

    arXiv

  30. [30]

    Findings from the first shared task on automated prediction of difficulty and response time for multiple-choice questions

    Victoria Yaneva, Kai North, Peter Baldwin, Le An Ha, Saed Rezayi, Yiyun Zhou, Sagnik Ray Choudhury, Polina Harik, and Brian Clauser. Findings from the first shared task on automated prediction of difficulty and response time for multiple-choice questions. InProceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BE...

    2024

  31. [31]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115,

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi T...

    Pith/arXiv arXiv

  32. [32]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  33. [33]

    Towards valid student simulation with large language models.arXiv preprint arXiv:2601.05473,

    Zhihao Yuan, Yunze Xiao, Ming Li, Weihao Xuan, Richard Tong, Mona Diab, and Tom Mitchell. Towards valid student simulation with large language models.arXiv preprint arXiv:2601.05473,

    arXiv

  34. [34]

    Are you doubtful? oh, it might be difficult then! exploring the use of model uncertainty for question difficulty estimation.arXiv preprint arXiv:2412.11831,

    Leonidas Zotos, Hedderik van Rijn, and Malvina Nissim. Are you doubtful? oh, it might be difficult then! exploring the use of model uncertainty for question difficulty estimation.arXiv preprint arXiv:2412.11831,

    arXiv

  35. [35]

    16 Appendix A Related Work A.1 Item Difficulty Prediction Estimating item difficulty has traditionally relied on response-data calibration under CTT or IRT, which remains central for calibrated assessment but is costly for newly authored items because it typically requires field testing before items can be operationalized (DeMars, 2010; Hsu et al., 2018; ...

    2010

  36. [36]

    These generated traces serve as intermediate supervision signals and are subsequently used as the foundation for downstream difficulty prediction

    andQwen3- 32B(Yang et al., 2025), in reasoning mode, to produce explicit reasoning traces for each item. These generated traces serve as intermediate supervision signals and are subsequently used as the foundation for downstream difficulty prediction. To provide a fair and comprehensive comparison against mainstream item difficulty prediction methods, we ...

    2025

  37. [37]

    Overall, the results show that Epi2Diff remains competitive against these alternatives

    These comparisons include using only LLM- extracted item-text features, combining item semantic representations with LLM-extracted features, and using only embeddings derived from the reasoning trace. Overall, the results show that Epi2Diff remains competitive against these alternatives. The LLM-extracted features provide useful item-level signals, but re...

    2025