pith. machine review for the scientific record. sign in

arxiv: 2601.08584 · v1 · submitted 2026-01-13 · 💻 cs.CL

Recognition: 2 theorem links

· Lean Theorem

Ministral 3

Abhinav Rastogi, Adrien Sad\'e, Alan Jeffares, Albert Jiang, Alexander H. Liu, Alexandre Cahill, Alexandre Gavaudan, Alexandre Sablayrolles, Am\'elie H\'eliou, Amos You, Andy Ehrenberg, Andy Lo, Anton Eliseev, Antonia Calvi, Avinash Sooriyarachchi, Baptiste Bout, Baptiste Rozi\`ere, Baudouin De Monicault, Cl\'emence Lanfranchi, Corentin Barreau, Cyprien Courtot, Daniele Grattarola, Darius Dabert, Diego de las Casas, Elliot Chane-Sane, Faruk Ahmed, Gabrielle Berrada, Ga\"etan Ecrepont, Gauthier Guinet, Georgii Novikov, Guillaume Kunsch, Guillaume Lample, Guillaume Martin, Gunshi Gupta, Jan Ludziejewski, Jason Rute, Joachim Studnia, Jonas Amar, Jos\'ephine Delas, Josselin Somerville Roberts, Karmesh Yadav, Kartik Khandelwal, Khyathi Chandu, Kush Jain, Laurence Aitchison, Laurent Fainsin, L\'eonard Blier, Lingxiao Zhao, Louis Martin, Lucile Saulnier, Luyu Gao, Maarten Buyl, Margaret Jennings, Marie Pellat, Mark Prins, Mathieu Poir\'ee, Mathilde Guillaumin, Matthieu Dinot, Matthieu Futeral, Maxime Darrin, Maximilian Augustin, Mia Chiquier, Michel Schimpf, Nathan Grinsztajn, Neha Gupta, Nikhil Raghuraman, Olivier Bousquet, Olivier Duchenne, Patricia Wang, Patrick von Platen, Paula Kurylowicz, Paul Jacob, Paul Wambergue, Pavankumar Reddy Muddireddy, Philom\`ene Chagniot, Pierre Stock, Pravesh Agrawal, Quentin Torroba, Romain Sauvestre, Roman Soletskyi, Rupert Menneer, Sagar Vaze, Samuel Barry, Sanchit Gandhi, Sandeep Subramanian, Siddhant Waghjale, Siddharth Gandhi, Soham Ghosh, Srijan Mishra, Sumukh Aithal, Szymon Antoniak, Teven Le Scao, Th\'eo Cachet, Theo Simon Sorg, Thibaut Lavril, Thiziri Nait Saada, Thomas Chabal, Thomas Foubert, Thomas Robert, Thomas Wang, Tim Lawson, Tom Bewley, Tom Edwards, Umar Jamil, Umberto Tomasini, Valeriia Nemychnikova, Van Phung, Victor Jouault, Vincent Maladi\`ere, Virgile Richard, Wassim Bouaziz, Wen-Ding Li, William Marshall, Xinghui Li, Xinyu Yang, Yassine El Ouahidi, Yihan Wang, Yunhao Tang, Zaccharie Ramzi

Pith reviewed 2026-05-14 19:08 UTC · model grok-4.3

classification 💻 cs.CL
keywords dense language modelscascade distillationmodel pruningparameter efficientinstruction tuningreasoning modelsmultimodal capabilitiesmodel compression
0
0 comments X

The pith

Ministral 3 derives 3B, 8B, and 14B dense models through iterative pruning and distillation for constrained hardware.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces the Ministral 3 family of dense language models sized at 3B, 8B, and 14B parameters, built specifically for applications with limited compute and memory. It details a derivation recipe called Cascade Distillation that repeatedly prunes the model and continues training with distillation to shrink size while aiming to keep performance. Each size offers three variants: a base pretrained model, an instruction-finetuned version, and a reasoning model, all equipped with image understanding. The work centers on releasing these under an open license so they can run where larger models cannot.

Core claim

Ministral 3 is a series of parameter-efficient dense language models at 3B, 8B, and 14B parameters obtained by Cascade Distillation, an iterative process of pruning followed by continued training with distillation, yielding base, instruction-tuned, and reasoning variants that each support image understanding.

What carries the argument

Cascade Distillation: the iterative pruning and continued training with distillation technique that shrinks model size while transferring capabilities from larger teachers.

If this is right

  • The three model sizes enable deployment on hardware that cannot host larger dense models.
  • Instruction-tuned variants directly support user command following without further adaptation.
  • Reasoning variants target complex multi-step problem solving at reduced cost.
  • Built-in image understanding extends the models to multimodal tasks without separate vision components.
  • Apache 2.0 release permits commercial and research reuse without licensing restrictions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same cascade process could be tested on even smaller targets such as 1B parameters to map the size-performance curve.
  • Combining cascade distillation with post-training quantization might produce further efficiency gains for edge devices.
  • The approach suggests a repeatable path for converting existing large models into families of progressively smaller siblings.

Load-bearing premise

Iterative pruning plus distillation training preserves strong instruction following, reasoning, and image understanding at the reduced parameter counts.

What would settle it

Benchmark results showing the 3B Ministral 3 model scores more than 20 points below a comparable 7B model on standard instruction-following and multimodal reasoning tests.

read the original abstract

We introduce the Ministral 3 series, a family of parameter-efficient dense language models designed for compute and memory constrained applications, available in three model sizes: 3B, 8B, and 14B parameters. For each model size, we release three variants: a pretrained base model for general-purpose use, an instruction finetuned, and a reasoning model for complex problem-solving. In addition, we present our recipe to derive the Ministral 3 models through Cascade Distillation, an iterative pruning and continued training with distillation technique. Each model comes with image understanding capabilities, all under the Apache 2.0 license.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 1 invented entities

No mathematical derivations, free parameters, or background axioms are invoked. The only potential new element is the named 'Cascade Distillation' process, presented without external references or validation.

invented entities (1)
  • Cascade Distillation no independent evidence
    purpose: Iterative pruning combined with continued training via distillation to create smaller capable models
    Described as the authors' recipe in the abstract with no prior citations or independent evidence provided.

pith-pipeline@v0.9.0 · 5965 in / 1198 out tokens · 51648 ms · 2026-05-14T19:08:31.206940+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 21 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. K12-KGraph: A Curriculum-Aligned Knowledge Graph for Benchmarking and Training Educational LLMs

    cs.CL 2026-05 conditional novelty 7.0

    K12-KGraph is a textbook-derived knowledge graph that powers a new benchmark revealing LLMs' poor curriculum cognition and a small training corpus that outperforms general instruction data on educational tasks.

  2. Stateful Agent Backdoor

    cs.CR 2026-05 unverdicted novelty 7.0

    A stateful backdoor for LLM agents, modeled as a Mealy machine with a decomposition framework, enables incremental malicious actions across sessions and achieves 80-95% attack success rate on four models.

  3. RESP: Reference-guided Sequential Prompting for Visual Glitch Detection in Video Games

    cs.CV 2026-04 unverdicted novelty 7.0

    RESP uses reference-guided sequential prompting with VLMs to improve frame-level and video-level visual glitch detection in games by establishing per-video baselines.

  4. BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidence

    cs.CL 2026-04 unverdicted novelty 7.0

    BAS aggregates utility from an answer-or-abstain model across risk thresholds and is uniquely maximized by truthful confidence estimates.

  5. PanoWorld: Towards Spatial Supersensing in 360$^\circ$ Panorama World

    cs.CV 2026-05 unverdicted novelty 6.0

    PanoWorld adds spherical geometry to MLLMs via cross-attention and pano-specific instruction data, yielding better performance on panoramic spatial reasoning benchmarks than standard perspective-based pipelines.

  6. On-Policy Self-Evolution via Failure Trajectories for Agentic Safety Alignment

    cs.AI 2026-05 unverdicted novelty 6.0

    FATE lets LLM agents self-evolve safer behaviors by generating and filtering repairs from their own failure trajectories using verifiers and Pareto optimization.

  7. PRISM: A Geometric Risk Bound that Decomposes Drift into Scale, Shape, and Head

    cs.CL 2026-05 unverdicted novelty 6.0

    PRISM supplies a geometric upper bound on LLM variant risk that splits drift into scale, shape, and head axes and doubles as a differentiable regularizer against forgetting.

  8. Causal Bias Detection in Generative Artifical Intelligence

    cs.AI 2026-05 unverdicted novelty 6.0

    A causal framework unifies fairness analysis across generative AI and standard ML by deriving decompositions that separate biases along causal pathways and differences between real-world and model mechanisms.

  9. Mirror, Mirror on the Wall: Can VLM Agents Tell Who They Are at All?

    cs.AI 2026-05 unverdicted novelty 6.0

    Stronger VLM agents use mirror reflections for self-identification in controlled 3D tests, while weaker ones inspect but fail to extract or correctly attribute self-relevant information.

  10. LLMs are not (consistently) Bayesian: Quantifying internal (in)consistencies of LLMs' probabilistic beliefs

    cs.LG 2026-05 unverdicted novelty 6.0

    LLMs do not consistently perform Bayesian updates on probabilistic beliefs; heuristic approaches often outperform exact Bayesian computation on downstream tasks, indicating misspecified internal models of the world.

  11. Skill Neologisms: Towards Skill-based Continual Learning

    cs.LG 2026-05 unverdicted novelty 6.0

    Skill neologisms are optimized soft tokens that improve LLM performance on targeted skills without weight updates and allow zero-shot composition for continual learning.

  12. When LLMs Stop Following Steps: A Diagnostic Study of Procedural Execution in Language Models

    cs.CL 2026-05 unverdicted novelty 6.0

    LLM accuracy on controlled procedural arithmetic drops from 61% at 5 steps to 20% at 95 steps, with failures including skipped steps, premature answers, and hallucinated operations.

  13. AVISE: Framework for Evaluating the Security of AI Systems

    cs.CR 2026-04 unverdicted novelty 6.0

    AVISE provides a new framework and automated SET that identifies jailbreak vulnerabilities in language models with 92% accuracy, finding all nine tested models vulnerable to an augmented Red Queen attack.

  14. Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale

    cs.CV 2026-04 unverdicted novelty 6.0

    Evidence for cross-modal representational convergence weakens substantially at scale and in realistic many-to-many settings, indicating models learn rich but distinct representations.

  15. The Master Key Hypothesis: Unlocking Cross-Model Capability Transfer via Linear Subspace Alignment

    cs.LG 2026-04 unverdicted novelty 6.0

    The Master Key Hypothesis states that capabilities are low-dimensional directions transferable across models through linear subspace alignment, with UNLOCK demonstrating gains such as 12.1% accuracy improvement on MAT...

  16. eOptShrinkQ: Near-Lossless KV Cache Compression Through Optimal Spectral Denoising and Quantization

    cs.LG 2026-04 unverdicted novelty 6.0

    eOptShrinkQ compresses KV caches to ~2.2 bits per entry via optimal spectral shrinkage and quantization, outperforming prior methods on LongBench while matching FP16 on multi-needle retrieval.

  17. Agentic Insight Generation in VSM Simulations

    cs.CL 2026-04 unverdicted novelty 5.0

    A two-step agentic system for extracting insights from VSM simulations achieves up to 86% accuracy with top LLMs by using progressive data discovery and slim context.

  18. Granite Embedding Multilingual R2 Models

    cs.IR 2026-05 unverdicted novelty 4.0

    Granite Embedding Multilingual R2 releases 311M and 97M parameter bi-encoder models that achieve state-of-the-art retrieval performance on multilingual text, code, long-document, and reasoning datasets.

  19. Optimizing Korean-Centric LLMs via Token Pruning

    cs.CL 2026-04 unverdicted novelty 4.0

    Token pruning of non-Korean vocabulary in LLMs improves generation stability and often boosts machine translation on Korean tasks while cutting vocabulary size substantially.

  20. Phoenix-VL 1.5 Medium Technical Report

    cs.CL 2026-05 unverdicted novelty 3.0

    Phoenix-VL 1.5 Medium is a 123B-parameter natively multimodal model that reaches state-of-the-art results on Singapore multimodal, legal, and policy benchmarks after localized training on 1T+ tokens while staying comp...

  21. Benchmarking EngGPT2-16B-A3B against Comparable Italian and International Open-source LLMs

    cs.CL 2026-05 unverdicted novelty 3.0

    EngGPT2MoE-16B-A3B matches or beats other Italian models on most international benchmarks but trails top international models such as GPT-5 nano and Qwen3-8B.

Reference graph

Works this paper leans on

29 extracted references · 29 canonical work pages · cited by 21 Pith papers · 15 internal anchors

  1. [1]

    Pixtral 12b.arXiv preprint arXiv:2410.07073,

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pixtral 12b.arXiv preprint arXiv:2410.07073,

    work page arXiv

  2. [2]

    GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    Program Synthesis with Large Language Models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    Qwen3-VL Technical Report

    URLhttps://arxiv.org/abs/2511.21631. Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russ Webb. Distillation scaling laws.arXiv preprint arXiv:2502.08606,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    URLhttps://arxiv.org/abs/2501.12948. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pages arXiv–2407,

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, Johan Ferret, and Mathieu Blondel

    URLhttps://arxiv.org/abs/2509.01649. Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, Johan Ferret, and Mathieu Blondel. Direct language model alignment from online ai feedback,

    work page arXiv

  8. [8]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt

    URLhttps://arxiv.org/abs/2402.04792. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300,

    work page arXiv 2009

  9. [9]

    Measuring Mathematical Problem Solving With the MATH Dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974,

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.arXiv preprint arXiv:1705.03551,

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    Gemma 3 Technical Report

    URL https://arxiv.org/abs/2503.19786. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research.Transactions of the Association for Computational Linguistics, 7:453–466,

    work page internal anchor Pith review Pith/arXiv arXiv

  13. [13]

    Race: Large-scale reading comprehension dataset from examinations

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785–794,

    work page 2017

  14. [14]

    From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline.arXiv preprint arXiv:2406.11939,

    12 Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline.arXiv preprint arXiv:2406.11939,

    work page arXiv

  15. [15]

    Wildbench: Benchmarking llms with challenging tasks from real users in the wild.arXiv preprint arXiv:2406.04770,

    Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking llms with challenging tasks from real users in the wild.arXiv preprint arXiv:2406.04770,

    work page arXiv

  16. [16]

    Phybench: Holistic evaluation of physical perception and reasoning in large language models

    Zihan Liu, Zijian Wang, Yue Zhang, Jianing Wang, Jian Tang, Xiang He, and Xiangyu Zhang. Phybench: Holistic evaluation of physical perception and reasoning in large language models. arXiv preprint arXiv:2504.16074,

    work page arXiv

  17. [17]

    Compact language models via pruning and knowledge distillation.arXiv preprint arXiv:2407.14679,

    Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation.arXiv preprint arXiv:2407.14679,

    work page arXiv

  18. [18]

    Scalable-softmax is superior for attention

    Ken M Nakanishi. Scalable-softmax is superior for attention.arXiv preprint arXiv:2501.19399,

    work page arXiv

  19. [19]

    YaRN: Efficient Context Window Extension of Large Language Models

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models.arXiv preprint arXiv:2309.00071,

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [20]

    Are we done with mmlu?arXiv preprint arXiv:2406.04127,

    Aryo Perez, Tomasz Stanislawek, Andrzej Pohl, Kamil Dwojak, Dawid Jurkiewicz, Piotr Kobus, and Tomasz Trzci´nski. Are we done with mmlu?arXiv preprint arXiv:2406.04127,

    work page arXiv

  21. [21]

    Direct Preference Optimization: Your Language Model is Secretly a Reward Model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model.arXiv preprint arXiv:2305.18290,

    work page internal anchor Pith review Pith/arXiv arXiv

  22. [22]

    Jiang, Andy Lo, Gabrielle Berrada, Guillaume Lample, et al

    Abhinav Rastogi, Albert Q. Jiang, Andy Lo, Gabrielle Berrada, Guillaume Lample, et al. Magistral. arXiv preprint arXiv:2506.10910,

    work page arXiv

  23. [23]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

    URLhttps://arxiv.org/abs/2402.03300. Noam Shazeer. Glu variants improve transformer.arXiv preprint arXiv:2002.05202,

    work page internal anchor Pith review Pith/arXiv arXiv 2002

  24. [24]

    Llm pruning and distillation in practice: The minitron approach.arXiv preprint arXiv:2408.11796,

    13 Sharath Turuvekere Sreenivas, Saurav Muralidharan, Raviraj Joshi, Marcin Chochowski, Ameya Sunil Mahabaleshwarkar, Gerald Shen, Jiaqi Zeng, Zijia Chen, Yoshi Suhara, Shizhe Diao, Chenhan Yu, Wei-Chun Chen, Hayley Ross, Oluwatobi Olabiyi, Ashwath Aithal, Oleksii Kuchaiev, Daniel Korzekwa, Pavlo Molchanov, Mostofa Patwary, Mohammad Shoeybi, Jan Kautz, an...

    work page arXiv

  25. [25]

    RoFormer: Enhanced Transformer with Rotary Position Embedding

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.arXiv preprint arXiv:2104.09864,

    work page internal anchor Pith review Pith/arXiv arXiv

  26. [26]

    arXiv preprint arXiv:2306.11695 , year=

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models.arXiv preprint arXiv:2306.11695,

    work page arXiv

  27. [27]

    URLhttps://arxiv.org/abs/2505.09388. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understa...

    work page internal anchor Pith review Pith/arXiv arXiv

  28. [28]

    HellaSwag: Can a Machine Really Finish Your Sentence?

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830,

    work page internal anchor Pith review Pith/arXiv arXiv 1905

  29. [29]

    Agieval: A human-centric benchmark for evaluating foundation models

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364,

    work page arXiv