Pith. sign in

REVIEW 4 major objections 5 minor 83 references

BenchHub: A Unified Benchmark Suite for Holistic and Customizable LLM Evaluation

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Model rankings in LLM benchmarks shift significantly with the subject composition of the test set, and BenchHub's sample-level categorization makes that dependence visible and usable for customized evaluation.

desk verdict A solid infrastructure paper with a real categorizer-robustness gap; the benchmark is worth using, but the culture-specific claims need human validation before I'd trust them. read the letter →

arxiv 2506.00482 v1 pith:V6EP7UZZ submitted 2025-05-31 cs.LG cs.AIcs.CL

Eunsu Kim , Haneul Yoo , Guijin Son , Hitesh Patel , Amit Agarwal , Alice Oh This is my paper
classification cs.LGcs.AIcs.CL
keywords LLMevaluationbenchmarksuitedomain-awarecompositionsample-wisecategorizationculture-specificmodelrankingtaxonomy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a model's rank on an LLM leaderboard is not a stable property of the model but depends on what mix of subjects the test set happens to contain. To make that dependence visible and controllable, the authors build BenchHub, a repository that merges 303K questions from 38 English and Korean benchmarks and labels every question by skill (knowledge, reasoning, value/alignment), subject (64 fine-grained categories under six coarse ones), and cultural target (General or Local). Across seven model families they show rankings fluctuate sharply by category — Llama-3.3-70B, for instance, sits sixth in Science and Technology yet first in Culture and Social Intelligence — and that the sampling strategy used to compose a test set changes rankings significantly (Friedman test, $p < 0.01$). If the paper is right, single-number benchmark scores are composition-dependent, and fair model comparison requires domain-aware, customizable evaluation of the kind BenchHub supports.

What carries the argument

The load-bearing object is the sample-wise taxonomy together with the automated categorizer that assigns its labels. Each question receives a skill label (knowledge, reasoning, or value/alignment), one or more subject labels drawn from 64 fine-grained categories organized under six coarse-grained subjects (Science, Technology, Humanities and Social Science, Arts and Sports, Culture, Social Intelligence), and a target label (General, or Local with a specific culture such as Korea or the United States). The categorizer, BenchHub-Cat-7B, is a Qwen-2.5-7B model fine-tuned on GPT-4o-generated synthetic rationales, and it is what lets the authors reclassify whole existing benchmarks sample by sample, quantify how subject distribution shifts model rankings, and construct customized evaluation sets. The distribution experiments in Section 4.2 then use these labels to resample 5K-question test sets under four strategies and test ranking shifts with the Friedman and Wilcoxon tests.

What would settle it

Take a sample of the BenchHub questions, have human annotators assign subject and target labels, rebuild the STEM-only English set and the Korean math-and-culture teaching-agent set from those gold labels, and re-rank the same models: if the customized rankings collapse toward the stratified rankings, or if the Friedman test across the four sampling strategies in Section 4.2 loses significance under the gold labels, the paper's composition claim would be shown to be an artifact of classifier noise rather than a property of benchmarks.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that benchmark composition — the distribution of subject categories in a test set — substantially changes which LLM appears to be best. The authors show that existing holistic benchmarks already encode very different compositions: MMLU and MixEval are more than half Humanities and Social Sciences, Chatbot Arena contains only 25.5% HASS, and the Korean KMMLU is 76.1% STEM. They then demonstrate experimentally that the same set of models is ranked differently depending on whether the test set is assembled by random sampling, stratified sampling, or sampling to match the category distributions of Chatbot Arena or MixEval, with the differences statistically significant at $p < 0.01$. BenchHub is presented as the infrastructure that surfaces this composition effect: because every question carries skill, subject, and target labels, a user can assemble a purpose-built test set — a STEM-only English set or a Korean math-and-culture teaching-agent set — and obtain rankings that differ from, and better match the intended use than, a generic stratified evaluation.

Load-bearing premise

The whole argument rests on the automated labels — subject categories and especially the cultural-target label, which is correct only about half the time — being accurate enough that the reported ranking shifts and culture-specific scores reflect what the questions actually measure, rather than noise in the classification.

Editorial extensions

If this is right

  • Single-number leaderboard scores should no longer be read as model quality: the same models rank differently when the test set's subject mix changes, so a reported score needs a stated composition to be interpretable.
  • New datasets can be fed through the automated pipeline (reformatting, metadata assignment, sample-level classification, merging) and become part of BenchHub without manual curation, keeping evaluations current as models and benchmarks evolve.
  • Application-specific evaluation becomes practical: a team building a Korean math teaching agent can assemble a weighted test set of math, education, and Korea-culture questions and obtain a ranking that generic stratified sampling does not provide.
  • The categorized distribution statistics give benchmark designers a direct view of which subjects are over- and under-represented in current evaluation suites, identifying gaps worth filling with new datasets.
  • Small categorization noise is tolerable: the robustness study finds that corruption rates up to about 1.5% leave model rankings essentially undisturbed, so the framework does not require perfect labels to be useful.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the composition effect holds, it carries beyond English and Korean: any aggregated leaderboard in any language or modality silently encodes a subject-distribution choice, so multilingual and multimodal suites deserve the same kind of composition audit rather than a single averaged rank.
  • The paper's own Table 7 points to a stress point the robustness section does not cover: the target-culture label is accurate only about half the time, while the corruption analysis runs only to 10% misclassification. A fair reading suggests the culture-specific customized evaluations should be re-run with human-verified target labels to confirm they measure cultural knowledge rather than classific
  • One concrete extension an independent team could run: take the released BenchHub data, relabel a sample with human annotators, and check whether the Section 4.2 sampling-strategy differences and the Section 4.3 customized rankings survive; positive results would strengthen the composition thesis considerably.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces BenchHub, a benchmark infrastructure that aggregates 303K questions from 38 English and Korean datasets and classifies them at the sample level into skills, subjects, and target types using a fine-tuned Qwen-2.5-7B categorizer (BenchHub-Cat-7B). The authors evaluate seven or fourteen LLMs from several families and report that (i) model rankings fluctuate across coarse-grained subject categories, (ii) sampling strategies that mimic different benchmark compositions significantly change rankings, and (iii) customized subsets (STEM-focused English evaluation and a Korean math-teaching-agent scenario) yield different leaderboards. Section 5.2 attempts to show robustness of rankings to categorization errors via a corruption ablation up to 10% corruption.

Significance. If the central findings hold, BenchHub would be a practically useful and reproducible infrastructure for domain-aware and customizable LLM evaluation, and the demonstration that benchmark composition can alter rankings is an important caution for leaderboard interpretation. The paper's strengths include a public dataset and model release, a clearly described pipeline, and an unusually extensive set of evaluations across model families and languages. However, the validity of the empirical claims depends heavily on the accuracy of the automatic categorizer, and the paper's own robustness evidence does not cover the categorizer's actual operating point, especially for target labels. With additional validation at realistic error rates, structured noise, and per-subset statistical measures, the contribution could be solid; in its current form the load-bearing robustness claim is not established.

major comments (4)
  1. [Section 5.2 and Table 7] The corruption ablation only tests misclassification rates from 0% to 10%, but the categorizer's reported subject accuracy is 0.871 (12.9% error) and target accuracy is 0.494 (50.6% error). Consequently, the subject-label operating point lies above the tested range, and the target-label operating point lies roughly five times beyond the maximum tested corruption. Because the culture-specific 'Local' subsets and the customized Korean math-teaching evaluation in Section 4.3(b) rely directly on target labels, the paper does not currently demonstrate that those rankings are robust to real categorizer errors. The authors should repeat the ablation at the actual error rates and, ideally, with structured corruption that mimics realistic confusions between adjacent categories rather than purely random label corruption.
  2. [Section 2 and Section 4.2] There is a potential circularity in the composition experiments: the same BenchHub-Cat-7B categorizer is used to label MMLU, MixEval, and Chatbot Arena, and those labels define the category distributions that are then used to construct the sampling setups in Section 4.2. If the categorizer has systematic biases (for example, over- or under-labeling particular subject classes), those biases are baked into the very compositions that are shown to change rankings. The paper should validate the distributional labels against human annotation or an independent classifier, or at minimum analyze the sensitivity of the Section 4.2 conclusions to plausible bias patterns.
  3. [Appendix F, Tables 9 and 10] Fine-grained category accuracies are reported without confidence intervals or sample sizes, and several rows appear to be based on extremely small numbers of items (for example, the Korean 'Psychology' row shows values of 0.000 and 1.000 across models, and Section 2 mentions KorNAT EN has only 24 samples). Without per-category sample counts and uncertainty estimates, the fine-grained rank fluctuations in Figure 6 and the customized-evaluation conclusions in Table 1 may not be statistically distinguishable from noise. The authors should provide sample sizes and confidence intervals, and should consider restricting claims for categories with very few items.
  4. [Section 4.2, final paragraph] The text states that pairwise Wilcoxon signed-rank tests confirm all pairs of sampling setups significantly differ in average, 'except for random sampling versus sampling according to MixEval distribution (p < 0.01).' This is internally contradictory: if the exception has p < 0.01, it is not an exception. The intended result is presumably that random versus MixEval does not differ significantly, or that all pairs differ at p < 0.01; the sentence must be corrected and the associated p-values and multiple-testing correction should be reported explicitly.
minor comments (5)
  1. [Figures 2 and 3] The figure labels contain typos such as 'Humanities and Social Sciencce' and 'T ech'; these should be corrected for readability.
  2. [Section 6] The word 'interpertable' should be 'interpretable'.
  3. [Appendix F and Table 2] The notation is inconsistent between 'Marin eng.' in Table 9 and 'Marine eng.' in Table 10 and the taxonomy in Table 6; unify the spelling.
  4. [Section 4.1] The sentence 'We select 6,644 and 6,485 examples for English and Korean, respectively' would benefit from a brief explanation of the selection criterion relative to the total 303K samples, since the numbers are much smaller than the full corpus.
  5. [Section 3.2] The description of the synthetic rationale generation in Appendix D.1 is important but appears only in the appendix; a short summary in the main text of how the classifier was trained and validated would help readers assess the accuracy numbers without opening the appendix.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's ranking-variation claims are measured from actual model evaluations, and the categorizer's low accuracy is a robustness concern rather than a by-construction reduction.

full rationale

The central empirical chain is: BenchHub-Cat-7B assigns subject/skill/target labels to questions; the authors then evaluate LLMs on subsets defined by those labels and compare rankings. The ranking differences in Figures 6-7 and the Friedman/Wilcoxon results in Section 4.2 are computed from observed model accuracies on those subsets, not derived from the categorizer's labels or from any parameter fitted to the reported outcome. The 'sampling according to category distribution' strategy adopts distributions 'classified by our fine-tuned model' (Sec. 4.2), but the subsequent claim that rankings change across sampling strategies is an empirical measurement, not a consequence of the classifier's definitions. The low target accuracy (0.494, Table 7) and the 12.9% subject error exceeding the 0-10% corruption range tested in Sec. 5.2 are genuine validity/robustness limitations, but they do not make the derivation circular: a noisy measuring instrument does not make the measurement equivalent to its input. The paper also cites several works with overlapping authors (e.g., [23], [39], [44], [60]), but these are used as dataset sources and background surveys, not as load-bearing uniqueness theorems or as replacements for the reported experiments. I therefore find no step in which a claimed 'prediction' reduces by construction to a fit, a definition, or a self-citation.

Assumptions & free parameters 2 free parameters · 3 assumptions · 2 invented entities

The main burden is the automatic categorizer, which drives all downstream evaluation and custom set construction. Its target label accuracy is near chance, and the robustness study does not test the actual operating point. The hand-picked weights and sampling caps also shape results.

free parameters (2)
  • Customized Korean math agent weights = 0.6 (math), 0.1 (education), 0.3 (Korea-local)
    Weights in the weighted accuracy in Section 4.3(b) are hand-chosen for the use case; changing them changes the ranking.
  • Category sampling cap = 150 examples per category; merge categories below 80
    Section 4.1 sampling protocol is ad hoc and affects the balance of the evaluation set, and no sensitivity analysis is provided.
assumptions (3)
  • domain assumption The BenchHub-Cat-7B classifier's labels are correct enough to support the evaluation claims.
    Validation accuracy is 0.871 for subject, 0.967 for skill, but only 0.494 for target; Section 5.2 robustness only covers corruption up to 10%.
  • ad hoc to paper Synthetic rationales generated by GPT-4o teach the categorizer the taxonomy reliably.
    Section D.1 uses GPT-4o to produce training rationales; no human validation of the rationales is reported.
  • domain assumption The selected 38 benchmarks adequately represent general-purpose and culture-specific evaluation.
    Section 3.1.2 and the Limitations section acknowledge incomplete English coverage; dataset selection relies on references [75, 41, 44].
invented entities (2)
  • BenchHub-Cat-7B categorizer
    purpose: Automatically assign subject, skill, and target labels to each question
    Released on HuggingFace; no external benchmark confirms its labels, and target accuracy is 0.494.
  • Six coarse and 64 fine subject taxonomy
    purpose: Classify questions into subjects for filtering and customized evaluation
    Built by authors from DDC and Korean classification systems; no inter-annotator agreement or external validation reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BenchHub: A Unified Benchmark Suite for Holistic and Customizable LLM Evaluation." pith.science (2026). https://pith.science/paper/V6EP7UZZ

@misc{pith2026250600482,
  author       = {Pith},
  title        = {Pith review of: BenchHub: A Unified Benchmark Suite for Holistic and Customizable LLM Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V6EP7UZZ}},
  note         = {Machine review of arXiv:2506.00482}
}
read the original abstract

As large language models (LLMs) continue to advance, the need for up-to-date and well-organized benchmarks becomes increasingly critical. However, many existing datasets are scattered, difficult to manage, and make it challenging to perform evaluations tailored to specific needs or domains, despite the growing importance of domain-specific models in areas such as math or code. In this paper, we introduce BenchHub, a dynamic benchmark repository that empowers researchers and developers to evaluate LLMs more effectively. BenchHub aggregates and automatically classifies benchmark datasets from diverse domains, integrating 303K questions across 38 benchmarks. It is designed to support continuous updates and scalable data management, enabling flexible and customizable evaluation tailored to various domains or use cases. Through extensive experiments with various LLM families, we demonstrate that model performance varies significantly across domain-specific subsets, emphasizing the importance of domain-aware benchmarking. We believe BenchHub can encourage better dataset reuse, more transparent model comparisons, and easier identification of underrepresented areas in existing benchmarks, offering a critical infrastructure for advancing LLM evaluation research.

Figures

Figures reproduced from arXiv: 2506.00482 by the authors.

Figure 1
Figure 1. The concept of BENCHHUB. BENCHHUB automatically classifies and merges questions from existing benchmark datasets on a sample-wise basis. Through BENCHHUB, users can select test sets that align with their objectives and efficiently evaluate the models. ∗Equal contribution. Preprint. arXiv:2506.00482v1 [cs.LG] 31 May 2025 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Data distribution of existing evaluation benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Data distribution of MMLU series in En￾glish, Korean, Japanese, Indonesian, and Chinese, respectively Moreover, these biases are not limited to cross￾benchmark comparisons but can also manifest within multilingual contexts [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Data distribution of all datasets used in this paper by coarse-grained subjects, targets, and [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Fine-grained data distribution of all datasets used in this paper in terms of subjects [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: LLM evaluation ranking under BENCHHUB in terms of coarse-grained subjects In this section, we evaluate seven LLMs across diverse subjects using BENCHHUB. We select 6,644 and 6,485 examples for English and Korean, respectively. To manage the large number of fine-grained…
Figure 7
Figure 7. Figure 7: LLM ranking according to four sampling methods [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: User Interface of BENCHHUB Web Demo 21 [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: Detailed data distribution of MMLU series in English, Korean, Japanese, Indonesian, and [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

83 extracted references · 46 canonical work pages

  1. [1]

    Rahul K. Arora, Jason Wei, Hicks Rebecca Soskin, Preston Bowman, Joaquin Quiñonero- Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, and Johannes Heidecke. HealthBench: Evaluating large language models towards improved human health, 2025

    2025

  2. [2]

    CaLMQA: Exploring culturally specific long-form question answering across 23 languages.arXiv preprint arXiv:2406.17761, 2024

    Shane Arora, Marzena Karpinska, Hung-Ting Chen, Ipsita Bhattacharjee, Mohit Iyyer, and Eunsol Choi. CaLMQA: Exploring culturally specific long-form question answering across 23 languages.arXiv preprint arXiv:2406.17761, 2024

    arXiv 2024

  3. [3]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  4. [4]

    Axolotl: Scalable fine-tuning framework for llms

    Axolotl AI. Axolotl: Scalable fine-tuning framework for llms. https://axolotl-ai-cloud. github.io/axolotl/, 2025. Github

    2025

  5. [5]

    PIQA: Reasoning about physical commonsense in natural language.Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7432–7439, Apr

    Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about physical commonsense in natural language.Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7432–7439, Apr. 2020

    2020

  6. [6]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  7. [7]

    Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios N. Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: an open platform for evaluating LLMs by human preference. InProceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024

    work page 2024

  8. [8]

    CulturalBench: a robust, diverse and challenging benchmark on measuring the (lack of) cultural knowledge of LLMs

    Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, et al. CulturalBench: a robust, diverse and challenging benchmark on measuring the (lack of) cultural knowledge of LLMs. arXiv preprint arXiv:2410.02677, 2024

    arXiv

Show all 83 references
  1. [9]

    Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  2. [10]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  3. [11]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025 arXiv

  4. [12]

    SciEx: Benchmarking large language models on scientific exams with human expert grading and automatic grading

    Tu Anh Dinh, Carlos Mullov, Leonard Bärmann, Zhaolin Li, Danni Liu, Simon Reiß, Jueun Lee, Nathan Lerzer, Jianfeng Gao, Fabian Peller-Konrad, Tobias Röddiger, Alexander Waibel, Tamim Asfour, Michael Beigl, Rainer Stiefelhagen, Carsten Dachsbacher, Klemens Böhm, and Jan Niehues...

    2024

  5. [13]

    MME: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2024

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. MME: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2024

    2024 arXiv

  6. [14]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ah- mad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, A...

    2024 arXiv

  7. [15]

    NativQA: Multilingual culturally-aligned natural query for LLMs.arXiv preprint arXiv:2407.09823, 2024

    Md Arid Hasan, Maram Hasanain, Fatema Ahmad, Sahinur Rahman Laskar, Sunaya Upadhyay, Vrunda N Sukhadia, Mucahid Kutlu, Shammur Absar Chowdhury, and Firoj Alam. NativQA: Multilingual culturally-aligned natural query for LLMs.arXiv preprint arXiv:2407.09823, 2024

    2024 arXiv

  8. [16]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InInternational Conference on Learning Representations, 2021

    2021

  9. [17]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In J. Vanschoren and S. Yeung, editors,Proceedings of the Neural Information Processing Systems Tr...

    2021

  10. [18]

    MedQA-SWE - a clinical question & answer dataset for Swedish

    Niclas Hertzberg and Anna Lokrantz. MedQA-SWE - a clinical question & answer dataset for Swedish. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, editors,Proceedings of the 2024 Joint International Conference on Computat...

    2024

  11. [19]

    Liger kernel: Efficient triton kernels for llm training.arXiv preprint arXiv:2410.10989, 2024

    Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, and Yanning Chen. Liger kernel: Efficient triton kernels for llm training.arXiv preprint arXiv:2410.10989, 2024

    arXiv

  12. [20]

    MoralBench: Moral evaluation of LLMs.arXiv preprint arXiv:2406.04428, 2024

    Jianchao Ji, Yutong Chen, Mingyu Jin, Wujiang Xu, Wenyue Hua, and Yongfeng Zhang. MoralBench: Moral evaluation of LLMs.arXiv preprint arXiv:2406.04428, 2024

    2024 arXiv

  13. [21]

    KoBBQ: Korean bias benchmark for question answering.Transactions of the Association for Computational Linguistics, 12:507–524, 2024

    Jiho Jin, Jiseon Kim, Nayeon Lee, Haneul Yoo, Alice Oh, and Hwaran Lee. KoBBQ: Korean bias benchmark for question answering.Transactions of the Association for Computational Linguistics, 12:507–524, 2024

    2024

  14. [22]

    Dynabench: Rethinking benchmarking in NLP

    Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adi...

    2021

  15. [23]

    CLIcK: A benchmark dataset of cultural and linguistic intelligence in Korean

    Eunsu Kim, Juyoung Suk, Philhoon Oh, Haneul Yoo, James Thorne, and Alice Oh. CLIcK: A benchmark dataset of cultural and linguistic intelligence in Korean. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, editors, Proceedi...

    2024

  16. [24]

    Developing a pragmatic benchmark for assessing Korean legal language understanding in large language models

    Yeeun Kim, Youngrok Choi, Eunkyung Choi, JinHwan Choi, Hai Jin Park, and Wonseok Hwang. Developing a pragmatic benchmark for assessing Korean legal language understanding in large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Findings 13 of the...

    2024

  17. [25]

    Understand, solve and translate: Bridging the multilingual mathematical reasoning gap.arXiv preprint arXiv:2501.02448, 2025

    Hyunwoo Ko, Guijin Son, and Dasol Choi. Understand, solve and translate: Bridging the multilingual mathematical reasoning gap.arXiv preprint arXiv:2501.02448, 2025

    2025 arXiv

  18. [26]

    The NarrativeQA reading comprehension challenge

    Tomáš Koˇciský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gá- bor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317–328, 2018

    2018

  19. [27]

    KorMedMCQA: multi-choice question answering benchmark for korean healthcare professional licensing examinations.arXiv preprint arXiv:2403.01469, 2024

    Sunjun Kweon, Byungjin Choi, Gyouk Chu, Junyeong Song, Daeun Hyeon, Sujin Gan, Jueon Kim, Minkyu Kim, Rae Woong Park, and Edward Choi. KorMedMCQA: multi-choice question answering benchmark for korean healthcare professional licensing examinations.arXiv preprint arXiv:2403.01469, 2024

    2024 arXiv

  20. [28]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019

  21. [29]

    KoSBI: A dataset for mitigating social bias risks towards safer large language model applications

    Hwaran Lee, Seokhee Hong, Joonsuk Park, Takyoung Kim, Gunhee Kim, and Jung-woo Ha. KoSBI: A dataset for mitigating social bias risks towards safer large language model applications. In Sunayana Sitaram, Beata Beigman Klebanov, and Jason D Williams, editors,Proceedings of the 6...

    2023

  22. [30]

    KorNAT: LLM alignment benchmark for Korean social values and common knowledge

    Jiyoung Lee, Minwoo Kim, Seungho Kim, Junghwan Kim, Seunghyun Won, Hwaran Lee, and Edward Choi. KorNAT: LLM alignment benchmark for Korean social values and common knowledge. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Findings of the Association for Computationa...

    2024

  23. [31]

    LegalAgentBench: Evaluating LLM agents in legal domain

    Haitao Li, Junjie Chen, Jingli Yang, Qingyao Ai, Wei Jia, Youfeng Liu, Kai Lin, Yueyue Wu, Guozhi Yuan, Yiran Hu, et al. LegalAgentBench: Evaluating LLM agents in legal domain. arXiv preprint arXiv:2412.17259, 2024

    2024 arXiv

  24. [32]

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D Manning, Christopher Re, Diana Acosta-Navas,...

    2023

  25. [33]

    TruthfulQA: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors,Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long...

    2022

  26. [34]

    Benchmark data repositories for better benchmarking

    Rachel Longjohn, Markelle Kelly, Sameer Singh, and Padhraic Smyth. Benchmark data repositories for better benchmarking. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors,Advances in Neural Information Processing Systems, volume 37, p...

    2024

  27. [35]

    João Matos, Shan Chen, Siena Kathleen V . Placino, Yingya Li, Juan Carlos Climent Pardo, Daphna Idan, Takeshi Tohyama, David Restrepo, Luis Filipe Nakayama, José María Millet 14 Pascual-Leone, Guergana K Savova, Hugo Aerts, Leo Anthony Celi, An-Kwok Ian Wong, Danielle Bitterma...

    2025

  28. [36]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors,Proceedings of the 2018 Conference on Empiric...

    2018

  29. [37]

    FActScore: Fine-grained atomic evaluation of factual precision in long form text generation

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,...

    2023

  30. [38]

    Right answer, wrong score: Uncovering the inconsistencies of llm evaluation in multiple-choice question answering.arXiv preprint arXiv:2503.14996, 2025

    Francesco Maria Molfese, Luca Moroni, Luca Gioffrè, Alessandro Scirè, Simone Conia, and Roberto Navigli. Right answer, wrong score: Uncovering the inconsistencies of llm evaluation in multiple-choice question answering.arXiv preprint arXiv:2503.14996, 2025

    2025 arXiv

  31. [39]

    BLEnD: A benchmark for llms on everyday knowledge in diverse cultures and languages

    Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsu- vas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, Víctor Gutiérrez- Basulto, Yazmín Ibáñez García, Hwaran Lee, Shamsuddeen Hassan Muhammad, Kiwoong Park, Anar Sabuhi Rzaye...

    2024

  32. [40]

    Extracting cultural commonsense knowledge at scale

    Tuan-Phong Nguyen, Simon Razniewski, Aparna Varde, and Gerhard Weikum. Extracting cultural commonsense knowledge at scale. InProceedings of the ACM Web Conference 2023, WWW ’23, page 1907–1917, New York, NY , USA, 2023. Association for Computing Machinery

    2023

  33. [41]

    MixEval: Deriving wisdom of the crowd from LLM benchmark mixtures

    Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain, Graham Neubig, and Yang You. MixEval: Deriving wisdom of the crowd from LLM benchmark mixtures. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Ne...

    2024

  34. [42]

    Chatterji, Faisal Ladhak, and Tatsunori Hashimoto

    Yonatan Oren, Nicole Meister, Niladri S. Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. Proving test set contamination in black-box language models. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  35. [43]

    BBQ: A hand-built bias benchmark for question answering

    Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thomp- son, Phu Mon Htut, and Samuel Bowman. BBQ: A hand-built bias benchmark for question answering. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors,Findings of the Associ...

    2022

  36. [44]

    Survey of cultural awareness in language models: Text and beyond.arXiv preprint arXiv:2411.00860, 2024

    Siddhesh Pawar, Junyeong Park, Jiho Jin, Arnav Arora, Junho Myung, Srishti Yadav, Faiz Ghifari Haznitrama, Inhwa Song, Alice Oh, and Isabelle Augenstein. Survey of cultural awareness in language models: Text and beyond.arXiv preprint arXiv:2411.00860, 2024

    2024 arXiv

  37. [45]

    Zero: Memory optimiza- tions toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimiza- tions toward training trillion parameter models. InSC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020. 15

    2020

  38. [46]

    NormAd: A framework for measuring the cultural adaptability of large language models

    Abhinav Sukumar Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, and Maarten Sap. NormAd: A framework for measuring the cultural adaptability of large language models. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors,Proceedings of the 2025 Conference of the Nations o...

    2025

  39. [47]

    DiversityMedQA: A benchmark for assessing demographic biases in medical diagnosis using large language models

    Rajat Rawat, Hudson McBride, Rajarshi Ghosh, Dhiyaan Nirmal, Jong Moon, Dhruv Alamuri, Sean O’Brien, and Kevin Zhu. DiversityMedQA: A benchmark for assessing demographic biases in medical diagnosis using large language models. In Daryna Dementieva, Oana Ignat, Zhijing Jin, Rad...

    2024

  40. [48]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof Q&A benchmark. InFirst Conference on Language Modeling, 2024

    2024

  41. [49]

    Kochenderfer

    Anka Reuel, Amelia Hardy, Chandler Smith, Max Lamparth, Malcolm Hardy, and Mykel J. Kochenderfer. BetterBench: Assessing AI benchmarks, uncovering issues, and establishing best practices. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, edi...

    2024

  42. [50]

    WinoGrande: an adversarial winograd schema challenge at scale.Commun

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: an adversarial winograd schema challenge at scale.Commun. ACM, 64(9):99–106, August 2021

    2021

  43. [51]

    Social IQa: Commonsense reasoning about social interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors,Proceedings of the 2019 Conference on Empirical Methods in Natural Language Pr...

    2019

  44. [52]

    Benchmarks as microscopes: A call for model metrology

    Michael Saxon, Ari Holtzman, Peter West, William Yang Wang, and Naomi Saphra. Benchmarks as microscopes: A call for model metrology. InFirst Conference on Language Modeling, 2024

    2024

  45. [53]

    Multi-fact: Assessing factuality of multilingual llms using factscore, 2024

    Sheikh Shafayat, Eunsu Kim, Juhyun Oh, and Alice Oh. Multi-fact: Assessing factuality of multilingual llms using factscore, 2024

    2024

  46. [54]

    Yourbench: Easy custom evaluation sets for everyone.arXiv preprint arXiv:2504.01833, 2025

    Sumuk Shashidhar, Clémentine Fourrier, Alina Lozovskia, Thomas Wolf, Gokhan Tur, and Dilek Hakkani-Tür. Yourbench: Easy custom evaluation sets for everyone.arXiv preprint arXiv:2504.01833, 2025

    2025 arXiv

  47. [55]

    CultureBank: An online community-driven knowledge base towards culturally aware language technologies

    Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Sunny Yu, Raya Horesh, Rogério Abreu De Paula, and Diyi Yang. CultureBank: An online community-driven knowledge base towards culturally aware language technologies. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Fi...

    2024

  48. [56]

    Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David I. Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Wei-Yin Ko, Sebastian Ruder, Madeline Smith, Antoine Bosse- lut, Al...

    2024 arXiv

  49. [57]

    KRX bench: Automating financial benchmark creation via large language models

    Guijin Son, Hyunjun Jeon, Chami Hwang, and Hanearl Jung. KRX bench: Automating financial benchmark creation via large language models. In Chung-Chi Chen, Xiaomo Liu, Udo Hahn, Armineh Nourbakhsh, Zhiqiang Ma, Charese Smiley, Veronique Hoste, Sanjiv Ranjan Das, Manling Li, Moha...

    2024

  50. [58]

    Beyond classification: Financial reasoning in state-of-the-art language models.arXiv preprint arXiv:2305.01505, 2023

    Guijin Son, Hanearl Jung, Moonjeong Hahm, Keonju Na, and Sol Jin. Beyond classification: Financial reasoning in state-of-the-art language models.arXiv preprint arXiv:2305.01505, 2023

    2023 arXiv

  51. [59]

    Multi-step reasoning in Korean and the emergent mirage

    Guijin Son, Hyunwoo Ko, and Dasol Choi. Multi-step reasoning in Korean and the emergent mirage. In Vinodkumar Prabhakaran, Sunipa Dev, Luciana Benotti, Daniel Hershcovich, Yong Cao, Li Zhou, Laura Cabello, and Ife Adebara, editors,Proceedings of the 3rd Workshop on Cross-Cultu...

    2025

  52. [60]

    KMMLU: Measuring massive multitask language understanding in Korean

    Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. KMMLU: Measuring massive multitask language understanding in Korean. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of th...

    2025

  53. [61]

    HAE-RAE bench: Evaluation of Korean knowledge in language models

    Guijin Son, Hanwool Lee, Suwan Kim, Huiseo Kim, Jae cheol Lee, Je Won Yeom, Jihyu Jung, Jung woo Kim, and Songseong Kim. HAE-RAE bench: Evaluation of Korean knowledge in language models. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, an...

    2024

  54. [62]

    Challenging BIG-bench tasks and whether chain-of-thought can solve them

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In Anna Rogers, Jordan Boyd- Graber, and Naoak...

    2023

  55. [63]

    MMLU-Pro+: Evaluating higher-order reasoning and shortcut learning in llms.arXiv preprint arXiv:2409.02257, 2024

    Saeid Asgari Taghanaki, Aliasgahr Khani, and Amir Khasahmadi. MMLU-Pro+: Evaluating higher-order reasoning and shortcut learning in llms.arXiv preprint arXiv:2409.02257, 2024

    2024 arXiv

  56. [64]

    CommonsenseQA: A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio, editors,Proceedings of the 2019 Conference of the North American Chapter of t...

    2019

  57. [65]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouil- lard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon...

    2025 arXiv

  58. [66]

    Benchmark suites instead of leaderboards for evaluating AI fairness.Patterns, 5(11):101080, 2024

    Angelina Wang, Aaron Hertzmann, and Olga Russakovsky. Benchmark suites instead of leaderboards for evaluating AI fairness.Patterns, 5(11):101080, 2024

    2024

  59. [67]

    SeaEval for multilingual foundation models: From cross-lingual alignment to cultural reasoning

    Bin Wang, Zhengyuan Liu, Xin Huang, Fangkai Jiao, Yang Ding, AiTi Aw, and Nancy Chen. SeaEval for multilingual foundation models: From cross-lingual alignment to cultural reasoning. In Kevin Duh, Helena Gomez, and Steven Bethard, editors,Proceedings of the 2024 Conference of t...

    2024

  60. [68]

    KULTURE Bench: A bench- mark for assessing language model in Korean cultural context.arXiv preprint arXiv:2412.07251, 2024

    Xiaonan Wang, Jinyoung Yeo, Joon-Ho Lim, and Hansaem Kim. KULTURE Bench: A bench- mark for assessing language model in Korean cultural context.arXiv preprint arXiv:2412.07251, 2024

    2024 arXiv

  61. [69]

    Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks

    Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuzni...

    2022

  62. [70]

    MMLU-Pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro: A more robust and challenging multi-task language unders...

    2024

  63. [71]

    Toward an evaluation science for generative AI systems.arXiv preprint arXiv:2503.05336, 2025

    Laura Weidinger, Inioluwa Deborah Raji, Hanna Wallach, Margaret Mitchell, Angelina Wang, Olawale Salaudeen, Rishi Bommasani, Deep Ganguli, Sanmi Koyejo, and William Isaac. Toward an evaluation science for generative AI systems.arXiv preprint arXiv:2503.05336, 2025

    2025 arXiv

  64. [72]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, 18 Jianxin Yang, ...

    2025 arXiv

  65. [73]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  66. [74]

    Rethinking benchmark and contamination for language models with rephrased samples.arXiv preprint arXiv:2311.04850, 2023

    Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E Gonzalez, and Ion Stoica. Rethinking benchmark and contamination for language models with rephrased samples.arXiv preprint arXiv:2311.04850, 2023

    2023 arXiv

  67. [75]

    FLASK: Fine-grained language model eval- uation based on alignment skill sets

    Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. FLASK: Fine-grained language model eval- uation based on alignment skill sets. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  68. [76]

    GeoM- LAMA: Geo-diverse commonsense probing on multilingual pre-trained language models

    Da Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li, and Kai-Wei Chang. GeoM- LAMA: Geo-diverse commonsense probing on multilingual pre-trained language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors,Proceedings of the 2022 Conference on Empirica...

    2022

  69. [77]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Màrquez, edi- tors,Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pa...

    2019

  70. [78]

    Maruf Hossain, Guang- Jie Ren, Kate Soule, Yifan Mai, and Yada Zhu

    Bing Zhang, Mikio Takeuchi, Ryo Kawahara, Shubhi Asthana, Md. Maruf Hossain, Guang- Jie Ren, Kate Soule, Yifan Mai, and Yada Zhu. Evaluating large language models with enterprise benchmarks. In Weizhu Chen, Yi Yang, Mohammad Kachuee, and Xue-Yong Fu, editors,Proceedings of the...

    2025

  71. [79]

    Task me anything

    Jieyu Zhang, Weikai Huang, Zixian Ma, Oscar Michel, Dong He, Tanmay Gupta, Wei-Chiu Ma, Ali Farhadi, Aniruddha Kembhavi, and Ranjay Krishna. Task me anything. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors,Advances in Neural Infor...

    2024

  72. [82]

    Users can interactively explore the overall data distribution they are interested in

    BenchHub Distribution(Figure 8a) This feature offers comprehensive statistics of all datasets we have. Users can interactively explore the overall data distribution they are interested in. Additionally, it provides researchers with insights into which datasets are currently la...

  73. [83]

    By reviewing samples, users can verify whether the dataset matches their needs and explore datasets suitable for their purposes

    Customizing BenchHub(Figure 8b) This allows users to access sample lists and statistics for selected categories. By reviewing samples, users can verify whether the dataset matches their needs and explore datasets suitable for their purposes. Users can also download the entire ...

  74. [84]

    Is the Earth flat?

    Submitting New Dataset(Figure 8c) To facilitate the addition of new datasets, We provide a submission section to input the Dataset Name, Huggingface URL, and Metadata/Descriptions. Based on this information, the author decides whether to add the dataset to BENCHHUB. 8Our inter...

  75. [2024]

    Association for Computational Linguistics

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.