Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Benchmarking the Pedagogical Knowledge of Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new benchmark measures whether AI knows how to teach, not just what it knows.

desk verdict A solid, reproducible pedagogy MCQ benchmark with a real gap to fill; the post-hoc filtering step needs more transparency but is not disqualifying. read the letter →

arxiv 2506.18710 v3 pith:J6KJHU3E submitted 2025-06-23 cs.CL cs.AI

Maxime Lelièvre , Amy Waldock , Meng Liu , Natalia Valdés Aspillaga , Alasdair Mackintosh , María José Ogando Portela , Jared Lee , Paul Atherton
show 2 more authors
Robin A. A. Ince Oliver G. B. Garrod
This is my paper
classification cs.CLcs.AI
keywords pedagogybenchmarkpedagogicalknowledgelargelanguagemodelsLLMevaluationmultiple-choicequestionsteachercertificationexamsspecialeducationneedsanddisabilitycost-accuracyfrontier
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Existing LLM benchmarks such as MMLU test what a model knows about a subject, but none test whether it understands how to teach. This paper argues that pedagogical knowledge — the methods and strategies of teaching — is a distinct, measurable capability, and introduces The Pedagogy Benchmark: 1,143 multiple-choice questions drawn from Chilean teacher certification exams, split into a 920-question cross-domain pedagogy set and a 223-question special-education set. Tested on 97 models, accuracy ranges from 28% for the smallest model to 89% for the current leader, and the authors pair each score with inference cost to show what each dollar of compute buys. If the benchmark is valid, it gives educators and developers a cheap, openly reproducible way to choose models for teaching tools, particularly small on-device models for low-resource classrooms.

What carries the argument

The load-bearing object is the dataset itself: 1,143 multiple-choice questions from teacher professional-development exams, converted from Spanish to English, de-duplicated with fuzzy string matching, trimmed by a pedagogy expert for clarity, relevance, and country-specific content, and then filtered once more by dropping roughly 10% of questions that 31 of 36 LLMs answered incorrectly. The evaluation protocol pairs the dataset with a single fixed three-shot prompt and a lenient answer parser so results are comparable across models, and a four-position answer-swap experiment checks for token- and position-based answer bias. The cost analysis rests on the Pareto value frontier, the convex upper hull of models plotted by inference price against accuracy, which identifies the best-performing model at each price point.

What would settle it

Take the roughly 120 excluded questions and give them to a panel of experienced teachers who have not seen the models' answers: if teachers answer a substantial share correctly, or independent experts judge the items valid, the exclusion was unjustified and the reported scores are too high. A second check is to compute the correlation between CDPK scores and a generic content-knowledge benchmark such as MMLU across the same 97 models; a near-perfect correlation would mean the test measures general multiple-choice ability rather than pedagogy, a possibility the paper leaves open.

Watch

Extended reading notes

Core claim

The paper's central claim is that pedagogical knowledge can be isolated from content knowledge and measured with genuine professional teacher-exam questions, and that current LLMs differ dramatically on this measure. The authors extracted, translated, de-duplicated, and expert-annotated multiple-choice questions from the ECEP teacher evaluation exams administered in Chile from 2017 to 2023, keeping 920 questions for the Cross-Domain Pedagogical Knowledge (CDPK) benchmark and 223 for the Special Educational Needs and Disability (SEND) benchmark, and scored every model with one fixed three-shot prompt. Across 97 models, CDPK accuracy spans 28% to 89% and SEND accuracy 29% to 86%, with the highest scores going to reasoning-focused models and the estimated mean score of human trainee teachers around 50%. The paper further claims that plotting accuracy against inference cost and model size exposes a rapidly improving value frontier: at $0.10 per million input tokens, benchmark accuracy rose from 50% in April 2024 to 82% in June 2025.

Load-bearing premise

The load-bearing premise is that the roughly 10% of questions removed because most of the 36 filtering models answered them incorrectly were genuinely flawed; if those questions were merely hard but valid, the reported accuracies are inflated and the final dataset is biased toward models that resemble the filters.

Editorial extensions

If this is right

  • Education developers can choose models by budget: open-weight models around 8B parameters score roughly 73% on CDPK, and the paper notes that the April 2024 leader's accuracy is nearly matched at over 400 times lower inference cost.
  • Pedagogical knowledge is improving rapidly at every price level; at $0.10 per million input tokens, accuracy rose from 50% in April 2024 to 70% in November 2024 to 82% in June 2025.
  • Reasoning-focused models dominate the top of the leaderboard, indicating that inference-time reasoning improves performance even on a knowledge-retrieval task.
  • SEND and CDPK scores are highly correlated ($r=0.94$), yet SEND is harder for the best models, so special-education pedagogy is a distinct capability worth evaluating separately.
  • Small open-weight models that can run on consumer hardware already exceed the estimated 50% human-trainee average, making offline teaching aids feasible in low-connectivity settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not report how CDPK correlates with general knowledge benchmarks such as MMLU or GPQA, and it flags this itself; a direct correlation study across the same 97 models would settle whether pedagogical knowledge is being measured distinctly or just general multiple-choice ability.
  • The benchmark measures pedagogical knowledge, not classroom practice, and the paper cites teacher-domain evidence linking the two but does not test that link for models; a scenario-based evaluation comparing high- and low-scoring models on generated lesson plans would test the connection.
  • Because all items come from one country's exams in translation, the benchmark probably reflects that system's pedagogical values; parallel benchmarks built from other countries' teacher exams would show whether the measured 'pedagogical knowledge' is universal or culturally specific.
  • Re-adding the excluded questions and recomputing the leaderboard is a cheap stability check that would quantify how much the rankings depend on the 10% of items dropped after the model-based filter.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper introduces The Pedagogy Benchmark, a multiple-choice question dataset for evaluating large language models on pedagogical knowledge, sourced from the Chilean Ministry of Education's ECEP teacher exams. The dataset consists of 920 Cross-Domain Pedagogical Knowledge (CDPK) questions and 223 Special Educational Needs and Disabilities (SEND) questions, processed through OCR, translation, de-duplication, expert curation, and a post-hoc filtering step. The authors evaluate 97 LLMs, reporting accuracy scores with bootstrap confidence intervals, subject-level breakdowns, SEND-specific results, and Pareto frontiers for accuracy versus inference cost and model size. They argue that the benchmark fills a gap in education-focused LLM evaluation and can guide model selection for educational applications.

Significance. If the dataset construction is valid, this is a useful and timely contribution: the benchmark is open, the evaluation pipeline is reproducible, and the authors provide a stability test (20 runs, maximum standard deviation 0.43%), bootstrap confidence intervals, and an answer-position bias check. The broad coverage of 97 models and the cost/accuracy analysis are practically valuable. However, the central validity claim depends critically on the post-hoc filtering in Section 3.2.4. If the excluded questions are merely hard rather than flawed, all reported accuracies are inflated and the model rankings are biased toward models similar to the 36 LLMs used for filtering. The manuscript also contains an internal inconsistency about whether the filtering step actually changed the dataset size. These issues must be resolved before the benchmark can be accepted as a valid measure of LLM pedagogical knowledge.

major comments (3)
  1. [Section 3.2.4] The post-hoc exclusion of approximately 10% of MCQs based on 31 of 36 LLMs answering them incorrectly is not adequately justified. The premise that questions most LLMs answer incorrectly are flawed is an assumption, not a demonstrated fact. If these questions are valid but hard, removing them inflates reported accuracies and biases the benchmark toward models that share the failure modes of the filtering LLMs. The paper does not report how many excluded questions were judged flawed versus simply hard, provides no examples of excluded questions, and includes no independent expert verification or inter-rater reliability. Please provide expert evidence for each excluded question, release the excluded set with justifications, or re-run the benchmark without exclusion and report both versions.
  2. [Section 3.2.4 vs Section 3.3] The reported dataset counts are internally inconsistent. The text says approximately 10% of MCQs were excluded, yet the final dataset is again reported as 1143 MCQs (920 CDPK + 223 SEND), and CDPK accuracy is computed over 899 of 920 questions, implying that only the 21 few-shot examples were removed. If 10% of the 920 CDPK questions had actually been excluded, the denominator would be roughly 828–830, not 920. The manuscript must clarify whether the filtering step changed the dataset, how many questions were removed, and which counts in the abstract and results are correct.
  3. [Section 4.1.2] The value-frontier analysis uses only input token costs, despite the text noting that reasoning models charge for hidden thinking traces as output tokens. Since the frontier includes Gemini 2.5 Flash/Pro and Deepseek R1, excluding output costs may overstate the value of reasoning models relative to non-reasoning models. Please report total (input+output) costs for at least a subset of models, or provide a sensitivity analysis showing that the frontier is robust to this choice.
minor comments (5)
  1. [Section 3.2.1 and Appendix B] The use of GPT-4o-mini for OCR and translation while GPT-4o Mini is later evaluated is a potential same-family bias; please clarify how many questions were manually checked per PDF and whether the pedagogy expert's review covered every translated question.
  2. [Section 5.2] The claim that models 'in many cases exceed human performance' should be tempered because the human baseline in Section 4.1.1 is only an aggregate estimate and does not correspond to actual performance on the benchmark questions.
  3. [Section 5.1] The correlation with other knowledge benchmarks (MMLU, GPQA) is not tested; acknowledging this is a good step, but a small empirical comparison would strengthen the claim that the benchmark isolates pedagogical knowledge.
  4. [Section 3.3] Please report the number of models excluded due to the 5% badly-formatted-response threshold, as this affects the generalizability of the model coverage statement.
  5. [Figure 5 and Table 2] Table 2 lists accuracies without confidence intervals; adding CIs or referencing Figure 5 would help readers assess the significance of rank differences.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the benchmark is grounded in external exam questions with official answer keys, and the reported pipeline steps are quality-control measures rather than self-referential derivations.

full rationale

The paper's claimed derivation chain is an empirical benchmark-construction pipeline, not a mathematical derivation whose outputs are forced by its inputs. Questions come from Chilean Ministry of Education ECEP exams with separate official answer keys; extraction, translation, de-duplication, and expert annotation are all external to the models being scored. The post-hoc filtering step in Section 3.2.4 uses LLM performance only as a screening signal, followed by manual inspection that identified concrete flaws, and the final accuracy scores are computed against official answer keys on the retained questions; even if the filtering were biased, that would be a dataset-validity threat rather than a circular reduction of the score to the filter. The use of GPT-4o-mini for OCR and translation, while a contamination-adjacent concern because that model family is later evaluated, does not make the benchmark's results equivalent to the construction inputs by definition. Self-references such as the AI-for-Education benchmark mapping note and the fabdata-llm interface are cited as context or infrastructure, not as load-bearing support for the benchmark's validity. The human baseline is explicitly acknowledged as an aggregate estimate, not a derived benchmark quantity. Overall, no step in the paper exhibits the equation-level or definitional equivalence required for a circularity finding.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The benchmark's central claims rest on domain assumptions about what the questions measure and how translation and filtering affect them. There are no free parameters in the statistical sense; the human baseline is an estimate, not a fitted parameter. No new theoretical entities are introduced.

assumptions (3)
  • domain assumption Pedagogical knowledge is measurable through multiple-choice questions sourced from the Chilean ECEP teacher exams.
    The entire benchmark rests on treating exam MCQs as valid indicators of pedagogical knowledge; the paper cites expert curation but no independent validation of this construct.
  • domain assumption English translations of original Spanish exam questions preserve the intended meaning and difficulty.
    Translation was performed by GPT-4o-mini with only a small random sample manually checked; the paper acknowledges translation may lose nuance.
  • ad hoc to paper Questions answered incorrectly by most LLMs are flawed and can be excluded.
    Section 3.2.4 excludes about 10% of questions based on model performance; this premise is what makes the post-hoc filter valid.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking the Pedagogical Knowledge of Large Language Models." pith.science (2026). https://pith.science/paper/J6KJHU3E

@misc{pith2026250618710,
  author       = {Pith},
  title        = {Pith review of: Benchmarking the Pedagogical Knowledge of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J6KJHU3E}},
  note         = {Machine review of arXiv:2506.18710}
}
read the original abstract

Benchmarks like Massive Multitask Language Understanding (MMLU) have played a pivotal role in evaluating AI's knowledge and abilities across diverse domains. However, existing benchmarks predominantly focus on content knowledge, leaving a critical gap in assessing models' understanding of pedagogy - the method and practice of teaching. This paper introduces The Pedagogy Benchmark, a novel dataset designed to evaluate large language models on their Cross-Domain Pedagogical Knowledge (CDPK) and Special Education Needs and Disability (SEND) pedagogical knowledge. These benchmarks are built on a carefully curated set of questions sourced from professional development exams for teachers, which cover a range of pedagogical subdomains such as teaching strategies and assessment methods. Here we outline the methodology and development of these benchmarks. We report results for 97 models, with accuracies spanning a range from 28% to 89% on the pedagogical knowledge questions. We consider the relationship between cost and accuracy and chart the progression of the Pareto value frontier over time. We provide online leaderboards at https://rebrand.ly/pedagogy which are updated with new models and allow interactive exploration and filtering based on various model properties, such as cost per token and open-vs-closed weights, as well as looking at performance in different subjects. LLMs and generative AI have tremendous potential to influence education and help to address the global learning crisis. Education-focused benchmarks are crucial to measure models' capacities to understand pedagogical concepts, respond appropriately to learners' needs, and support effective teaching practices across diverse contexts. They are needed for informing the responsible and evidence-based deployment of LLMs and LLM-based tools in educational settings, and for guiding both development and policy decisions.

Figures

Figures reproduced from arXiv: 2506.18710 by the authors.

Figure 1
Figure 1. Distribution of the MCQs by Category (left), Education Level (center) and Education [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Dataset Construction Pipeline 6 [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Percentage of duplicate questions (questions in each year that appeared in another year). [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Test of a subset of models for a selection bias. In particular, some models exhibit a selection [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Accuracy on the Pedagogy Benchmark (CDPK) for a subset of models. Error bars show [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Accuracy on the Pedagogy Benchmark (CDPK) by Weights Availability. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Accuracy on the Pedagogy Benchmark (CDPK) by Reasoning Capability [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Left: Trade-off between performance and cost, the best models being on the [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Left: Trade-off between performance and model size, the best models being on the [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Pedagogy Benchmark (CDPK) accuracy by subject area. Left: accuracy in individual [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Accuracy on the SEND Benchmark for a subset of models. Error bars show bootstrap 95% [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: Pedagogy (CDPK) vs SEND Benchmark Performance [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Calibrating Trustworthiness: Co-Designing Metrics and Visualizations for Evaluating LLMs in Education

    cs.HC 2026-08 conditional novelty 6.0 of 10

    A co-design study shows that making LLM trustworthiness metrics visible to learning engineers modestly increases agreement when choosing between AI tutor responses.

Reference graph

Works this paper leans on

34 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    R.K.Arora,J.Wei,R.S.Hicks,P.Bowman,J.Quinonero-Candela,F.Tsimpourlas,M.Sharman, M. Shah, A. Vallone, A. Beutel, J. Heidecke, and K. Singhal. HealthBench: Evaluating Large Language Models Towards Improved Human Health, 2025

    work page 2025

  2. [2]

    Biderman, H

    S. Biderman, H. Schoelkopf, L. Sutawika, L. Gao, J. Tow, B. Abbasi, A. F. Aji, P. S. Am- manamanchi, S. Black, J. Clive, et al. Lessons from the trenches on reproducible evaluation of language models.arXiv preprint arXiv:2405.14782, 2024

    arXiv 2024

  3. [3]

    S. K. Bitew, J. Deleu, C. Develder, and T. Demeester. Distractor generation for multiple- choice questions with predictive prompting and large language models, 2023. URL https: //arxiv.org/abs/2307.16338. 17

    arXiv 2023

  4. [4]

    Sastry, A

    T.Brown,B.Mann,N.Ryder,M.Subbiah,J.D.Kaplan,P.Dhariwal,A.Neelakantan,P.Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    work page 1901

  5. [5]

    Chang, X

    Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang, et al. A survey on evaluation of large language models.ACM Transactions on Intelligent Systems and Technology, 15(3):1–45, 2024

    work page 2024

  6. [6]

    Chiang, L

    W.-L. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference.arXiv preprint arXiv:2403.04132, 2024

    arXiv 2024

  7. [7]

    H.-B.Clark,M.Dowland,L.Benton,R.Budai,I.K.Keskin,E.Searle,M.Gregory,M.Hodierne, and J. Roberts. Auto-Evaluation: A Critical Measure in Driving Improvements in Quality and Safety of AI-Generated Lesson Resources.The AI + Open Education Initiative, Jan. 2025. URL https://aiopeneducation.pubpub.org/pub/i36sncz8/release/3

    work page 2025

  8. [8]

    Z. Guo, R. Jin, C. Liu, Y. Huang, D. Shi, L. Yu, Y. Liu, J. Li, B. Xiong, D. Xiong, et al. Evaluating large language models: A comprehensive survey.arXiv preprint arXiv:2310.19736, 2023

    arXiv 2023

Show all 34 references
  1. [9]

    Gupta, D

    V. Gupta, D. Pantoja, C. Ross, A. Williams, and M. Ung. Changing answer order can decrease mmlu accuracy.arXiv preprint arXiv:2406.19470, 2024

    2024 arXiv

  2. [10]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  3. [11]

    Kasenberg, A

    I.Jurenka,M.Kunesch,K.R.McKee,D.Gillick,S.Zhu,S.Wiltberger,S.M.Phal,K.Hermann, D. Kasenberg, A. Bhoopchand, et al. Towards responsible development of generative ai for education: An evaluation-driven approach.arXiv preprint arXiv:2407.12687, 2024

    2024

  4. [12]

    S. Khoo, G. Chua, and R. Shong. Minorbench: A hand-built benchmark for content-based risks for children, 2025. URL https://arxiv.org/abs/2503.10242

    2025 arXiv

  5. [13]

    M. T. R. Laskar, S. Alqahtani, M. S. Bari, M. Rahman, M. A. M. Khan, H. Khan, I. Jahan, A.Bhuiyan,C.W.Tan,M.R.Parvez,etal. Asystematicsurveyandcriticalreviewonevaluating large language models: Challenges, limitations, and recommendations. InProceedings of the 2024 Conference o...

    2024

  6. [14]

    Macina, N

    J. Macina, N. Daheim, I. Hakimi, M. Kapur, I. Gurevych, and M. Sachan. Mathtutorbench: A benchmark for measuring open-ended pedagogical capabilities of llm tutors, 2025. URL https://arxiv.org/abs/2502.18940

    2025

  7. [15]

    T. R. McIntosh, T. Susnjak, N. Arachchilage, T. Liu, P. Watters, and M. N. Halgamuge. Inadequacies of large language model benchmarks in the era of generative artificial intelligence. arXiv preprint arXiv:2402.09880, 2024

    2024 arXiv

  8. [16]

    Miller and K

    P. Miller and K. DiCerbo. LLM Based Math Tutoring: Challenges and Dataset, July 2024. URL https://osf.io/5zwv3_v1

    2024

  9. [17]

    Nancy, D

    O. Nancy, D. Stefania, and L. Andrew. A benchmark for math misconceptions: Bridging gaps in middle school algebra with ai-supported instruction, 2024. URL https://arxiv.org/abs/2412. 03765

    2024

  10. [18]

    Pezeshkpour and E

    P. Pezeshkpour and E. Hruschka. Large language models sensitivity to the order of options in multiple-choice questions.arXiv preprint arXiv:2308.11483, 2023

    2023 arXiv

  11. [19]

    L. Phan, A. Gatti, Z. Han, N. Li, J. Hu, H. Zhang, S. Shi, M. Choi, A. Agrawal, A. Chopra, A.Khoja,R.Kim,J.Hausenloy,O.Zhang,M.Mazeika,D.Anderson,T.Nguyen,M.Mahmood, F. Feng, S. Y. Feng, H. Zhao, M. Yu, V. Gangal, C. Zou, Z. Wang, J. P. Wang, P. Kumar, O. Pokutnyi, R. Gerbicz,...

    2025

  12. [20]

    Gpqa: A graduate-level google-proof q&a benchmark.arXiv preprint arXiv:2311.12022, 2023

    D.Rein,B.L.Hou,A.C.Stickland,J.Petty,R.Y.Pang,J.Dirani,J.Michael,andS.R.Bowman. Gpqa: A graduate-level google-proof q&a benchmark.arXiv preprint arXiv:2311.12022, 2023

    2023 arXiv

  13. [21]

    W. M. Reynolds and S. Park. Examining the relationship between the Educative Teacher Performance Assessment and preservice teachers’ pedagogical content knowledge.Jour- nal of Research in Science Teaching, 58(5):721–748, 2021. ISSN 1098-2736. doi: 10.1002/tea.21676. URL https:...

    2021 doi

  14. [22]

    L. Team, A. Modi, A. S. Veerubhotla, A. Rysbek, A. Huber, B. Wiltshire, B. Veprek, D. Gillick, D. Kasenberg, D. Ahmed, et al. LearnLM: Improving gemini for learning.arXiv preprint arXiv:2412.16429, 2024

    2024 arXiv

  15. [23]

    L. Team, A. Modi, A. S. Veerubhotla, A. Rysbek, A. Huber, A. Anand, A. Bhoopchand, B. Wiltshire, D. Gillick, D. Kasenberg, et al. Evaluating gemini in an arena for learning.arXiv preprint arXiv:2505.24477, 2025

    2025 arXiv

  16. [24]

    P. Wang, L. Li, L. Chen, Z. Cai, D. Zhu, B. Lin, Y. Cao, Q. Liu, T. Liu, and Z. Sui. Large language models are not fair evaluators.arXiv preprint arXiv:2305.17926, 2023

    2023 arXiv

  17. [25]

    Largelanguagemodels for education: A survey and outlook.arXiv preprint arXiv:2403.18105, 2024

    S.Wang,T.Xu,H.Li,C.Zhang,J.Liang,J.Tang,P.S.Yu,andQ.Wen. Largelanguagemodels for education: A survey and outlook.arXiv preprint arXiv:2403.18105, 2024

    2024 arXiv

  18. [26]

    my answer is c

    X. Wang, B. Ma, C. Hu, L. Weber-Genzel, P. Röttger, F. Kreuter, D. Hovy, and B. Plank. " my answer is c": First-token probabilities do not match text answers in instruction-tuned language models.arXiv preprint arXiv:2402.14499, 2024

    2024 arXiv

  19. [27]

    Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574, 2024

    2024 arXiv

  20. [28]

    W.X.Zhao,K.Zhou,J.Li,T.Tang,X.Wang,Y.Hou,Y.Min,B.Zhang,J.Zhang,Z.Dong,Y.Du, C. Yang, Y. Chen, Z. Chen, J. Jiang, R. Ren, Y. Li, X. Tang, Z. Liu, P. Liu, J.-Y. Nie, and J.-R. Wen. A Survey of Large Language Models.arXiv, Nov. 2023. doi: 10.48550/arXiv.2303.18223. URL http://arx...

    2023 doi arXiv

  21. [29]

    Zheng, H

    C. Zheng, H. Zhou, F. Meng, J. Zhou, and M. Huang. Large language models are not robust multiplechoiceselectors. InTheTwelfthInternationalConferenceonLearningRepresentations, 2023

    2023

  22. [30]

    Wheredoyouthinkthereismoreplasticine?

    L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023. 20 A Example Questions Maths (Pre-primary) In th...

    2023

  23. [31]

    Question

    **Segment** each paragraph into "Question" and "Answers": - "Question" should contain the main body of the paragraph that describes the situation and the instruction. Basically, the part before the list of choices. If there is a paragraph of text before the actual question, in...

  24. [32]

    The text is as follows: —– paragraph —–

    **Translate** the segmented text into English. The text is as follows: —– paragraph —–

  25. [33]

    question

    Please provide the segmented output in the following JSON format: "question": "<Segmented Question>", "A": "<Segmented Choice A>", "B": "<Segmented Choice B>", "C": "<Segmented Choice C>", "D": "<Segmented Choice D>"

  26. [34]

    question

    Please provide the translated output in the following JSON format: "question": "<Translated Question>", "A": "<Translated Choice A>", "B": "<Translated Choice B>", "C": "<Translated Choice C>", "D": "<Translated Choice D>" Prompt OCR You are a specialized OCR system trained to...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.