Pith. sign in

REVIEW 5 major objections 5 minor 54 references

GAOKAO-Eval: Does high scores truly reflect strong capabilities in LLMs?

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read High Gaokao scores do not prove LLMs reason like humans

desk verdict The 2024 Gaokao dataset is the real contribution; the Rasch analysis anchoring the headline claim does not survive contact with the data. read the letter →

arxiv 2412.10056 v1 pith:3G274RG4 submitted 2024-12-13 cs.CL cs.AI

Zhikai Lei , Tianyi Liang , Hanglei Hu , Jin Zhang , Yunhua Zhou , Yunfan Shao , Linyang Li , Chenchui Li
show 3 more authors
Changbo Wang Hang Yan Qipeng Guo
This is my paper · ORCID
classification cs.CLcs.AI
keywords LLMevaluationGaokaobenchmarkRaschmodelquestiondifficultyhuman-alignedcapabilitydataleakagereasoningtokensclosed-book
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper builds a closed-book benchmark from China's Gaokao exams, uses human teachers to grade the responses, and evaluates only models released before the exam date. It argues that even when data leakage and benchmark coverage are controlled, high scores do not imply human-aligned capability. LLM scoring rates stay nearly flat as question difficulty rises, and they scatter widely for questions of similar difficulty. Using the Rasch model from psychometrics as a human-performance yardstick, the paper shows LLM scoring deviates strongly from the theoretical human curve, with a negative R-squared for the fit. A follow-on experiment with o1 finds that re-defining difficulty by reasoning tokens instead of human difficulty improves the fit, suggesting human-derived difficulty may be the wrong axis for LLMs.

What carries the argument

The Rasch model, a one-parameter item response theory model, $P(X=1|\theta,b)=e^{\theta-b}/(1+e^{\theta-b})$, supplies the human-performance reference curve. The paper pairs it with an Elo rating system, seeded by human expert annotations and LLM judgments, to assign difficulty values to Gaokao items; the Elo difficulties correlate with human expert ratings at up to 0.94. Two derived statistics carry the argument: the Pearson correlation between difficulty and scoring rate (near zero, giving 'semi difficulty-invariant' behavior) and the variance of scoring rates within small difficulty bins (high, violating the expected monotone decline). The o1 experiment replaces difficulty with reasoning-token counts, which yields a positive R-squared and is offered as a more LLM-aligned difficulty axis.

What would settle it

Run the same 2024 Gaokao questions on a large sample of human examinees and fit the Rasch model to their scores; if the human data also show near-zero or negative R-squared, the reported mismatch is a property of the items or the difficulty scale, not evidence about LLM capabilities.

Watch

Extended reading notes

Core claim

The central claim is that high scores on human-crafted benchmarks do not necessarily reflect human-aligned capabilities in LLMs. The authors establish this by evaluating models on GAOKAO-Eval, a comprehensive, annually updated Gaokao-based benchmark with closed-book conditions and teacher-based grading, then comparing LLM scoring rates against the Rasch model's theoretical human performance curve. They find two systematic deviations: a semi difficulty-invariant scoring distribution, where the correlation between item difficulty and scoring rate is near zero, and high variance in scoring rates for items of similar difficulty. They also document grading inconsistencies among human raters for LLM responses, with an inconsistent score rate above 32% in some subjects, and recurring error patterns such as hallucinated poems or copying instead of summarizing. They further report that using o1's reasoning tokens as an alternative difficulty axis raises the Rasch fit from negative to positive, suggesting the mismatch reflects the human-aligned difficulty axis rather than only the models' deficiencies.

Load-bearing premise

The paper assumes that human examinee performance on these Gaokao items follows the Rasch curve, so that any LLM deviation from that curve is evidence of a capability mismatch; it relies on a theoretical human curve rather than collecting human scores on the same 2024 items.

Editorial extensions

If this is right

  • Leaderboard scores on human-crafted knowledge benchmarks should not be read as evidence of human-like reasoning, because a high aggregate score can coexist with insensitivity to item difficulty.
  • Benchmark designers should consider adding difficulty-response diagnostics, such as the Rasch fit and the difficulty-scoring correlation, alongside total scores.
  • LLM-aligned difficulty, for example based on reasoning effort, may be a more informative axis than human-perceived difficulty for evaluating and perhaps training models.
  • The high variance on similar-difficulty items implies reliability concerns for deploying LLMs in settings where consistent performance matters, such as tutoring or advice-giving systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test would be to collect human examinee scores on the same 2024 Gaokao items; if humans also deviate from the Rasch curve, the mismatch is a property of the items rather than a uniquely LLM failure.
  • The reasoning-token result suggests a testable hypothesis: models that spend more inference compute on harder-for-them questions will show a steeper scoring-rate curve; this could be validated across models with and without chain-of-thought.
  • The semi-invariance finding might partly reflect the granularity of the Elo difficulty scale or the mixture of question types; recomputing the correlations per question type could reveal whether the pattern is universal or concentrated in certain formats.
  • The high teacher disagreement rate raises a benchmark-design question: for subjective items, LLM answers may need a different rubric than human answers, and averaging three teachers' scores may obscure systematic oddities.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces GAOKAO-Eval, a 2024 Chinese Gaokao-based benchmark designed to be non-leaky and comprehensive, evaluates several LLMs released before the exam date, and reports that their scores do not reflect human-aligned capabilities. The main quantitative evidence is a Rasch-model comparison (Eq. 1, Fig. 6) showing a poor fit with R²=-0.23, alongside Pearson correlations between difficulty and scoring rates, variance computations within difficulty bins, teacher-grading inconsistency rates (Eq. 4), and qualitative examples of model errors. The authors also propose reasoning tokens as an LLM-aligned difficulty proxy, reporting an improved fit for o1 models in Fig. 11. The benchmark resource itself has strengths—temporal isolation, 54-teacher grading, and released code—but, as detailed below, the paper's central claim rests on a Rasch comparison made without human response data on the 2024 items and using an aggregate fitting procedure that is not a valid Rasch analysis.

Significance. If the Rasch mismatch were established against a genuine human baseline, the paper would make an important contribution to benchmark-validity research, showing that high benchmark scores need not imply human-like reasoning. The authors are explicit about their falsifiable prediction: LLM scoring rates should deviate from a human item-response curve. The benchmark construction and the release of code, model responses, and teacher-graded scores are concrete strengths that the community could build on. However, the stress-test concern raised in the review process is borne out by the manuscript: the human curve is assumed, not measured, and the pooled fit in Fig. 6 is not a Rasch fit in any standard sense. The paper's own limitation statement in Section 3.4 ('scores in GAOKAO-Eval should be interpreted with caution') further weakens the strength of the claims. For these reasons, the central conclusion is currently unsupported despite the value of the benchmark artifact.

major comments (5)
  1. [Section 3.1, Figure 6] The central Rasch comparison has no human response baseline. The text in Section 3.1 says 'we directly use this equation as the basis for evaluation' and the caption of Fig. 6 reports R²=-0.23, but no human examinee responses to the 2024 Gaokao items are collected or cited. The 'theoretical human performance curve' is imported from cognitive psychology (Rasch, 1993), while the x-axis difficulties come from the Elo system described in Section 3.1, which is only correlated with expert annotations (Fig. 5). Without human response data on the same items, a negative R² for LLMs does not establish a deviation from human-aligned capability; it only shows that the pooled LLM scoring-rate curve is not logistic. A concrete fix would be to estimate item difficulties and a human scoring-rate curve from pretesting data (which the Gaokao process nominally collects) and recompute the comparison.
  2. [Section 3.1, Eq. (1) and Figure 6] The fitted curve is not a valid Rasch fit. Eq. (1) defines the probability of a correct response for a single examinee with fixed ability θ, but the plotted points in Fig. 6 pool multiple models, subjects, and question types with different abilities. The mixture of Rasch-consistent examinees is not necessarily logistic, so fitting one logistic curve to the aggregate and reporting R²=-0.23 is not evidence against Rasch or IRT. The authors should instead fit a hierarchical or multilevel IRT model (e.g., random effects for models and subjects, or a 2PL model with a discrimination parameter) and report item-level fit statistics; alternatively, they should explicitly justify why the aggregate should preserve the logistic form.
  3. [Section 3.4, Eq. (4)] The Inconsistent Score Rate definition produces values near 32% for approximately normal score distributions, because P(|X−μ|>σ)≈0.317 for any normal distribution. Reporting 'over 32%' as 'too high' is therefore partly definitional without a human-grading baseline or a null model. The paper gives no ISR for human essays or human short answers graded under the same rubric, so the claim that LLM responses cause unusually high grading inconsistency is not established. This matters because the ISR discussion is used to support the 'high variance' finding in Section 3.2. A comparison against human responses under identical scoring conditions, or a simulation with calibrated teacher noise, is needed.
  4. [Section 3.1, difficulty estimation and Figure 5] The difficulty axis is not independent of the models being analyzed. Section 3.1 states that the hybrid difficulty ratings combine 'manual annotations with an Elo rating system' and that the system 'adjusts LLM scores based on pairwise comparisons,' and Fig. 5 shows Elo ratings derived partly from GPT-4o and GPT-4o-mini judgments. Because the same or similar LLM outputs are used to estimate item difficulties and to compute scoring rates, the Rasch comparison is partially circular. The reported 0.94 correlation with human expert annotations is a sanity check but does not replace human response data. A concrete test would be to re-estimate difficulties from human responses alone and recompute the Rasch fit with those difficulties.
  5. [Section 3.1, Tables 4 and 5] The paper's title and abstract claim that 'high scores' fail to reflect capability, but the evaluated models do not achieve high scores in an absolute sense. The top science total in Table 4 is 468.5/750, which is 62.5%, and most models are far below that. What the data actually show is that moderate scores are accompanied by a flat difficulty curve, not that near-ceiling scores fail to align with human difficulty. To support the stated claim, the paper would need to include a genuinely high-scoring model (with appropriate data-leakage controls) or substantially rephrase the title, abstract, and Section 3.1 to say that moderate scores do not imply human-aligned difficulty sensitivity.
minor comments (5)
  1. [Title and Abstract] The title contains a typo ('GAOKAO-E VAL') and the abstract has several grammatical and style issues, including 'phenomenons' and inconsistent capitalization of 'We' and 'we'; a careful proofreading pass is needed.
  2. [Appendix A.1] The text cites 'Query of CC technique (Fei et al., 2024)' but no Fei et al. entry appears in the reference list; either add the reference or remove the citation.
  3. [Section 3.2, Figure 8] The 'semi difficulty-invariant' conclusion is based on Pearson correlations in Eq. (2), but no confidence intervals, significance levels, or per-cell sample sizes are reported, making it difficult to judge whether the near-zero correlations are stable estimates or noise.
  4. [Section 4, Figure 11] The o1 experiment is presented as evidence that reasoning tokens 'mitigate the mismatch,' but it involves a single model family, the R² values are still low (0.1019), and o1 models were released after the June 6, 2024 cutoff used elsewhere in the paper; the claim should be scoped accordingly.
  5. [Sections 2.2 and 3.4] Section 2.2 says 54 teachers graded responses, while Section 3.4 says each question was reviewed by at least three teachers and the average was taken; the relationship between these statements is unclear and should be reconciled, including how the averaging affects the ISR calculation in Eq. (4).

Circularity Check

1 steps flagged · score 4.0 of 10

Difficulty ratings validated against their own human-annotation input; Rasch misfit mislabeled as prediction, but independent evidence remains

  1. fitted input called prediction [Section 3.1, 'Difficulty of Questions' and Figure 5]
    "To assess LLMs' alignment with this principle, we designed a hybrid approach combining manual annotations with an Elo rating system, which incorporates both human expertise and LLM-based judgments. This system adjusts LLM scores based on pairwise comparisons, allowing us to evaluate question difficulty and model performance consistency. The refined difficulty ratings closely align with human expert judgments, with an internal correlation of up to 0.94 (Figure 5)."

    The refined difficulty ratings are the output of a hybrid whose inputs include the manual human annotations. Correlating that output with the same human judgments (0.94) is partly an autocorrelation: the validation target is an input to the estimator. This is load-bearing because the refined difficulty is the x-axis of the Rasch comparison (Figure 6) and is used to claim the ratings 'accurately reflect the human-aligned capabilities of LLMs' (Section 3.2). The central high-score/mismatch conclusion therefore leans on a difficulty measure whose human alignment is asserted by construction, not independently verified.

full rationale

The paper's central claim is not itself definitional: high total scores and Rasch-consistent scoring are measured separately, and the claimed mismatch is an empirical conjunction. The Rasch comparison in Figure 6 is better characterized as a statistical-validity problem than a circular reduction: the red line is a fitted curve (not an out-of-sample prediction), and pooling many models/subjects under a single ability theta makes a poor fit unsurprising. Those are correctness concerns, not circularity. The one genuine circular step is the validation of the difficulty ratings: the 0.94 correlation with human expert judgments is partly automatic because manual annotations are an input to the hybrid Elo system. The paper also has independent evidence (54-teacher grading inconsistency, error patterns, qualitative examples) that does not reduce to the Rasch fit, so the partial circularity does not force the entire conclusion. There is no load-bearing self-citation chain; citations to Rasch, Bond & Fox, and psychometrics literature are external, and WQX training details are not used to justify the central mismatch claim.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The analysis depends on several fitted or chosen parameters: model abilities, item difficulties, Elo hyperparameters, variance bin widths, and reasoning-token bins. The main axioms are the assumed Rasch human curve, the validity of the Elo difficulty measure, and the non-leakage assumption. No genuinely new entities are introduced; 'semi difficulty-invariant scoring distribution' is a descriptive label, not an entity. The free-parameter count is moderate and mostly standard for IRT, but the unspecified Elo and binning choices add unaccounted flexibility.

free parameters (5)
  • Model ability parameter theta per model = not reported
    Used as the sole shift parameter in the Rasch model fit (Eq 1, Figure 6); estimated from each model's scoring data.
  • Item difficulty b_j per question = not reported
    Estimated via an Elo system combining human annotations and GPT-4o/model pairwise comparisons (Section 3.1, Figure 5).
  • Elo hyperparameters (initial rating, K-factor) = not specified
    Needed to compute the difficulty ratings but never stated; affects all derived b values (Section 3.1).
  • Variance bin width Delta b = not specified
    Chosen to define 'similar difficulty' in Eq 3 for the high-variance analysis; no justification given.
  • Reasoning-token difficulty proxy bins = not specified
    Used to fit the o1 curves in Figure 11; binning of the 0-to-9000 token range is not described.
assumptions (4)
  • domain assumption Human performance on Gaokao items follows the one-parameter Rasch model.
    Invoked in Section 3.1, Eq 1 and Figure 6, but no human examinee data are collected to verify this on the 2024 items.
  • domain assumption The hybrid Elo difficulty ratings (human annotations plus GPT-4o and model judgments) accurately represent human item difficulty.
    Stated in Section 3.1 with an internal correlation of 0.94 to human annotations, but the human annotation scale and collection are not detailed.
  • domain assumption The 2024 Gaokao questions are unseen by all evaluated models because the models were released before June 6, 2024.
    Core to the 'non-leaky' claim in Section 2.2; cannot be verified for proprietary models like GPT-4o beyond release dates.
  • standard math Standard definitions of Pearson correlation and R-squared apply to the reported statistics.
    Used in Eq 2 and Figure 6; the negative R² values suggest the calculation may differ from conventional linear-regression R², but the assumption is hidden.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GAOKAO-Eval: Does high scores truly reflect strong capabilities in LLMs?." pith.science (2026). https://pith.science/paper/3G274RG4

@misc{pith2026241210056,
  author       = {Pith},
  title        = {Pith review of: GAOKAO-Eval: Does high scores truly reflect strong capabilities in LLMs?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3G274RG4}},
  note         = {Machine review of arXiv:2412.10056}
}
read the original abstract

Large Language Models (LLMs) are commonly evaluated using human-crafted benchmarks, under the premise that higher scores implicitly reflect stronger human-like performance. However, there is growing concern that LLMs may ``game" these benchmarks due to data leakage, achieving high scores while struggling with tasks simple for humans. To substantively address the problem, we create GAOKAO-Eval, a comprehensive benchmark based on China's National College Entrance Examination (Gaokao), and conduct ``closed-book" evaluations for representative models released prior to Gaokao. Contrary to prevailing consensus, even after addressing data leakage and comprehensiveness, GAOKAO-Eval reveals that high scores still fail to truly reflect human-aligned capabilities. To better understand this mismatch, We introduce the Rasch model from cognitive psychology to analyze LLM scoring patterns and identify two key discrepancies: 1) anomalous consistent performance across various question difficulties, and 2) high variance in performance on questions of similar difficulty. In addition, We identified inconsistent grading of LLM-generated answers among teachers and recurring mistake patterns. we find that the phenomenons are well-grounded in the motivations behind OpenAI o1, and o1's reasoning-as-difficulties can mitigate the mismatch. These results show that GAOKAO-Eval can reveal limitations in LLM capabilities not captured by current benchmarks and highlight the need for more LLM-aligned difficulty analysis.

Figures

Figures reproduced from arXiv: 2412.10056 by the authors.

Figure 1
Figure 1. Comparison of LLM scores on the first and last questions of Gaokao Pa￾per. Despite the latter being more difficult, LLMs achieve similar scores, revealing po￾tential inconsistencies. To substantively address the problem, we introduce GAOKAO￾Eval, a comprehensive and annually updated benchmark based on Gaokao. The comprehensiveness and security of Gaokao is at￾tributed to the process where Gaokao experts spend two mo… view at source ↗
Figure 2
Figure 2. The GAOKAO-Eval pipeline. Built on the Gaokao benchmark, which ensures balanced [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Comprehensiveness of GAOKAO-Eval. LLM-Aligned Capabilities Improvement via GAOKAO Question. Training on a specialized Gaokao dataset contributes to a broader enhancement of its capabilities, specifically reflected in its performance on other sophisticated knowledge benchmarks. In this motivation, we propose a spe￾cific model named WQX based on InternLM2-20b-base (Cai et al., 2024). The training detail about WQX is i… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Total performance of LLMs in New Curriculum Standard Paper and National Type A [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Consistency distribution of Elo ratings across different models and methods, demonstrat [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: The fitted IRT curve for evaluating LLM performance. The x-axis represents the difficulty level of questions, while the y-axis represents the scoring rate S . The red line represents the fitted curve, indicating how well the Rasch model fits the observed data. P(X = 1|…
Figure 7
Figure 7. Figure 7: Analysis of question difficulty in relation to question types and order. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Correlation heatmaps between difficulty and scoring rate for all questions, multiple-choice [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Heatmap of subjects and model performance showing the In￾consistent Score Rate (ISR) across dif￾ferent subjects. Q: As shown in the figure, in the quadrilateral pyramid 𝑃 − 𝐴𝐵𝐶𝐷 ... (1) If 𝐴𝐷 ⊥ 𝑃𝐵, prove that 𝐴𝐷ԡ plane 𝑃𝐵𝐶... LLM (a) Model deduces a vertical relationsh…
Figure 11
Figure 11. Figure 11: Transforming O1’s reasoning tokens into human-aligned [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]
Figure 12
Figure 12. Figure 12: Examples of MM Geo Questions and Answers. [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Examples of GPT-4o Predictions [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 20 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and...

    2024

  3. [3]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  4. [4]

    a is b" fail to learn

    Lukas Berglund, Meg Tong, Maximilian Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Llms trained on "a is b" fail to learn "b is a". In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id...

    work page 2024

  5. [5]

    Applying the rasch model: fundamental measurement in the human sciences, 2007

    Trevor Bond and Christine Fox. Applying the rasch model: fundamental measurement in the human sciences, 2007

    work page 2007

  6. [6]

    Boone and Amity Noltemeyer

    William J. Boone and Amity Noltemeyer. Rasch analysis: A primer for school psychology researchers and practitioners. Cogent Education, 4 0 (1): 0 1416898, 2017. doi:10.1080/2331186X.2017.1416898. URL https://www.tandfonline.com/doi/abs/10.1080/2331186X.2017.1416898

    work page arXiv 2017

  7. [7]

    Internlm2 technical report, 2024

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, S...

    2024

  8. [8]

    M ^3 cot: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought, 2024 a

    Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. M ^3 cot: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought, 2024 a . URL https://arxiv.org/abs/2405.16473

    arXiv 2024

Show all 54 references
  1. [9]

    See what llms cannot answer: A self-challenge framework for uncovering llm weaknesses

    Yulong Chen, Yang Liu, Jianhao Yan, Xuefeng Bai, Ming Zhong, Yinghao Yang, Ziyi Yang, Chenguang Zhu, and Yue Zhang. See what llms cannot answer: A self-challenge framework for uncovering llm weaknesses. arXiv preprint arXiv:2408.08978, 2024 b

    2024 arXiv

  2. [10]

    Flash A ttention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flash A ttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024

    2024

  3. [11]

    Tuan Dinh, Jinman Zhao, Samson Tan, Renato M. P. Negrinho, Leonard Lausen, Sheng Zha, and George Karypis. Large language models of code fail at completing code with potential bugs. ArXiv, abs/2306.03438, 2023. URL https://api.semanticscholar.org/CorpusID:259089334

    2023 arXiv

  4. [12]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...

    2024 arXiv

  5. [13]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools, 2024

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie H...

    2024

  6. [14]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview...

    2021

  7. [15]

    Measuring massive multitask language understanding, January 2021 b

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, January 2021 b

    2021

  8. [16]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021 c

    2021

  9. [17]

    C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. In Advances in Neural ...

    2023

  10. [18]

    New Ontology and Knowledge Graph for University Curriculum Recommendation

    Nicolas Hubert, Armelle Brun, and Davy Monticolo. New Ontology and Knowledge Graph for University Curriculum Recommendation . In ISWC 2022 - The 21st International Semantic Web Conference , Hangzhou / Virtual, China, October 2022. URL https://hal.archives-ouvertes.fr/hal-03768154

    2022

  11. [19]

    Large language models and simple, stupid bugs

    Kevin Jesse, Toufique Ahmed, Prem Devanbu, and Emily Morgan. Large language models and simple, stupid bugs. 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR), pp.\ 563--575, 2023. URL https://api.semanticscholar.org/CorpusID:257636802

    2023

  12. [20]

    FigureQA : An annotated figure dataset for visual reasoning, February 2018

    Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Akos Kadar, Adam Trischler, and Yoshua Bengio. FigureQA : An annotated figure dataset for visual reasoning, February 2018

    2018

  13. [21]

    A diagram is worth a dozen images, March 2016

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images, March 2016

    2016

  14. [22]

    Rasch Measurement: Applications in Quantitative Educational Research, volume 1 of Education

    Myint Swe Khine. Rasch Measurement: Applications in Quantitative Educational Research, volume 1 of Education. Springer Singapore, 1 edition, 2020. ISBN 9789811517900. doi:10.1007/978-981-15-1800-3

    2020 doi

  15. [23]

    Cmmlu: Measuring massive multitask language understanding in chinese, 2023

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese, 2023

    2023

  16. [24]

    CMMLU: measuring massive multitask language understanding in chinese

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. CMMLU: measuring massive multitask language understanding in chinese. In Lun - Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational ...

    2024

  17. [25]

    Truthfulqa: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa...

    2022 doi

  18. [26]

    MMBench : Is your multi-modal model an all-around player?, August 2024

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. MMBench : Is your multi-modal model an all-around player?, August 2024

    2024

  19. [27]

    Learn to explain: Multimodal reasoning via thought chains for science question answering, October 2022 a

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, October 2022 a

    2022

  20. [28]

    IconQA : A new benchmark for abstract diagram understanding and visual language reasoning, July 2022 b

    Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. IconQA : A new benchmark for abstract diagram understanding and visual language reasoning, July 2022 b

    2022

  21. [29]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai - Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In The Twelfth International Conference on Learni...

    2024

  22. [30]

    OK-VQA : A visual question answering benchmark requiring external knowledge, September 2019

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK-VQA : A visual question answering benchmark requiring external knowledge, September 2019

    2019

  23. [31]

    Mistral Large 2: Designed for Single-Node Inference with Long-Context

    Mistral AI . Mistral Large 2: Designed for Single-Node Inference with Long-Context . https://mistral.ai/news/mistral-large-2407/#: :text=Mistral July 2024. [Online; accessed 2024-09-30]

    2024

  24. [32]

    Training on the benchmark is not all you need

    Shiwen Ni, Xiangtao Kong, Chengming Li, Xiping Hu, Ruifeng Xu, Jia Zhu, and Min Yang. Training on the benchmark is not all you need. arXiv preprint arXiv:2409.01790, 2024

    2024 arXiv

  25. [33]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  26. [34]

    We-math: Does your large multimodal model achieve human-like mathematical reasoning? arXiv preprint arXiv:2407.01284, 2024

    Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, et al. We-math: Does your large multimodal model achieve human-like mathematical reasoning? arXiv preprint arXiv:2407.01284, 2024

    2024 arXiv

  27. [35]

    Probabilistic models for some intelligence and attainment tests

    Georg Rasch. Probabilistic models for some intelligence and attainment tests. ERIC, 1993

    1993

  28. [36]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligenc...

    2020 doi

  29. [37]

    Math-llava: Bootstrapping mathematical reasoning for multimodal large language models

    Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong Bing, and Roy Ka-Wei Lee. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. arXiv preprint arXiv:2406.17294, 2024

    2024 arXiv

  30. [38]

    Assessing programming task difficulty for efficient evaluation of large language models

    Florian Tambon, Amin Nikanjam, Foutse Khomh, and Giuliano Antoniol. Assessing programming task difficulty for efficient evaluation of large language models. arXiv preprint arXiv:2407.21227, 2024

    2024

  31. [39]

    Internlm: A multilingual language model with progressively enhanced capabilities

    InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities. https://github.com/InternLM/InternLM, 2023

    2023

  32. [40]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language unders...

    2024

  33. [41]

    Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks

    Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Aky \"u rek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. In North American Chapter of the Associ...

    2023

  34. [42]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng...

    2024 arXiv

  35. [43]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2024

  36. [44]

    Hellaswag: Can a machine really finish your sentence? In Anna Korhonen, David R

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Anna Korhonen, David R. Traum, and Llu \' s M \` a rquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics...

    2019 doi

  37. [45]

    Evaluating the performance of large language models on gaokao benchmark

    Xiaotian Zhang, Chun yan Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. Evaluating the performance of large language models on gaokao benchmark. ArXiv, abs/2305.12474, 2023. URL https://api.semanticscholar.org/CorpusID:258833562

    2023 arXiv

  38. [46]

    Can llm replace stack overflow? a study on robustness and reliability of large language model code generation

    Li Zhong and Zilong Wang. Can llm replace stack overflow? a study on robustness and reliability of large language model code generation. In AAAI Conference on Artificial Intelligence, 2023. URL https://api.semanticscholar.org/CorpusID:261048682

    2023

  39. [47]

    Don't make your llm an evaluation benchmark cheater

    Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. Don't make your llm an evaluation benchmark cheater. arXiv preprint arXiv:2311.01964, 2023

    2023 arXiv

  40. [48]

    Larger and more instructable language models become less reliable

    Lexin Zhou, Wout Schellaert, Fernando Martínez-Plumed, Yael Moros-Daval, Cèsar Ferri, and José Hernández-Orallo. Larger and more instructable language models become less reliable. Nature, September 2024. ISSN 1476-4687. doi:10.1038/s41586-024-07930-y. URL https://doi.org/10.10...

    2024 doi

  41. [49]

    Gaokao-mm: A chinese human-level benchmark for multimodal models evaluation, 2024

    Yi Zong and Xipeng Qiu. Gaokao-mm: A chinese human-level benchmark for multimodal models evaluation, 2024

    2024

  42. [50]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  43. [51]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  44. [52]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

  45. [53]

    , " * write output.state after.block = add.period write

    ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION in...

  46. [54]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.