Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

EnigmaEval: A Benchmark of Long Multimodal Reasoning Challenges

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces EnigmaEval, a 1,184-puzzle benchmark on which the strongest evaluated AI models score 7.0% overall and 0% on the hard split, evidence that frontier models still lack unstructured, lateral reasoning.

desk verdict EnigmaEval is a genuinely useful new benchmark resource, but the 0% hard-split headline needs a human-solvability check on the exact evaluated inputs before the 'models lack unstructured reasoning' claim is fully supported. read the letter →

arxiv 2502.08859 v2 pith:5NELFENS submitted 2025-02-13 cs.AI cs.CL

Clinton J. Wang , Dean Lee , Cristina Menghini , Johannes Mols , Jack Doughty , Adam Khoja , Jayson Lynch , Sean Hendryx
show 2 more authors
Summer Yue Dan Hendrycks
This is my paper · ORCID
classification cs.AIcs.CL
keywords multimodalreasoningpuzzlehuntsbenchmarkdesignvision-languagemodelslateralknowledgesynthesislong-contextevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces EnigmaEval, a benchmark of 1,184 puzzles drawn from puzzle-hunt competitions, each with an unambiguous answer and presented both as original page images and as human transcriptions. The benchmark is designed to test a kind of reasoning that standard exams do not capture: finding hidden connections between unrelated clues and carrying out many unscripted deductive steps. Evaluating current vision-language models on it, the authors find that the strongest model reaches only 7.0% accuracy on the normal split and 0% on the hard split, with all others below 1.5%. They read this as evidence that frontier models still lack the unstructured, lateral reasoning that experienced puzzle solvers routinely apply.

What carries the argument

The load-bearing instrument is the EnigmaEval dataset itself: 1,184 puzzles from eight puzzle-source collections, stratified into a normal split (949 puzzles) and a hard split (235 puzzles), each with an official solution and a human transcription. The evaluation protocol formats every puzzle as text plus images and requires the model to answer on a final 'Answer: <answer>' line, which is then graded by exact string matching; a separate variant supplies the component answers for 77 meta-puzzles (puzzles that fuse the answers of component puzzles). This design is what lets a low score be attributed to reasoning: the transcription removes parsing difficulty, the strict answer format makes grading objective, and the meta-puzzle variant isolates whether models can synthesize given information.

What would settle it

Run the same 1,184 puzzles with lenient grading that accepts the correct answer appearing anywhere in the response, or with a one-solved-example demonstration of the answer format; if accuracy on the normal split rises substantially above 7.0%, the benchmark is measuring answer extraction and format compliance at least as much as reasoning.

Watch

Extended reading notes

Core claim

The paper's central discovery is that state-of-the-art multimodal language models are nearly incapable of solving puzzle-hunt problems: the best evaluated model scores 7.0% on the normal split and 0% on the hard split of EnigmaEval, while all other tested models score below 1.5%. Because the puzzles have human-verified, unambiguous solutions and were not adversarially constructed against models, the paper takes these near-zero scores as an organic measure of a missing capability rather than an artifact of benchmark design. The inclusion of human transcriptions alongside raw documents allows the authors to separate failures of reasoning from failures of OCR and document parsing, and giving models the component answers to meta-puzzles (best score 8.7%) shows that integration of provided information is itself a bottleneck.

Load-bearing premise

The load-bearing assumption is that the fixed prompt and exact-string matching on the 'Answer:' line are a fair way to collect and grade puzzle solutions, so that near-zero accuracy means models cannot reason rather than merely cannot guess the expected output format.

Editorial extensions

If this is right

  • EnigmaEval has enough headroom to measure future progress: with the best model at 7.0% and the rest near zero, improvements in unstructured and lateral reasoning will be visible for a long time.
  • Because accuracy on raw document images is lower than on transcriptions for several models, part of the current gap is document parsing and OCR rather than pure reasoning.
  • Giving models the correct answers to a meta-puzzle's component puzzles still leaves the best model at 8.7%, so combining given information is a distinct, largely unsolved step.
  • The uniformly low accuracy despite publicly available solutions indicates that memorization or web-search leakage is not currently inflating scores.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension would measure human solvers on the same items under the same single-attempt, no-interaction protocol; the reported 'hours to days' team experience does not quantify the human baseline the benchmark implies.
  • If the difficulty is real, puzzle hunts are a renewable well of benchmark items, since new events produce new puzzles with verifiable answers each year; this could slow the saturation that fixed academic question sets have shown.
  • The paper's private, continually updated release means the 7.0% number cannot be independently recomputed by outside researchers; a public sample with a hidden held-out subset would make the benchmark's difficulty auditable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces EnigmaEval, a benchmark of 1,184 puzzles drawn from puzzle hunts, with two input formats: human-transcribed text+image puzzles and raw PDF/page screenshots. The benchmark is private; evaluation is via fixed prompt templates and exact string matching on a final 'Answer:' line. Across nine frontier vision-language models, the reported accuracy is 7.0% at best on the normal split and 0% on the hard split (Table 2), with meta-puzzle solve rates also very low (Table C.1). The authors argue that these results expose a fundamental gap in models' unstructured and lateral reasoning abilities, and that data contamination is unlikely despite the public availability of puzzle solutions.

Significance. If the central claim is correct, EnigmaEval is a valuable and much-needed stress test for long-horizon multimodal reasoning, complementing knowledge-heavy benchmarks like Humanity's Last Exam. The dataset is large, draws from multiple puzzle traditions, includes both original and transcribed formats, and provides a clean separation between meta-puzzle reasoning and puzzle-solving. The use of human transcription to reduce parsing confounds is a thoughtful design choice, and the meta-puzzle protocol that supplies component answers is a useful methodological contribution. However, the absence of a human-solver calibration on the exact benchmark inputs, combined with the private release, leaves the benchmark's core difficulty claim only partially verified. As a result, the significance of the paper is real but conditional on additional validation.

major comments (4)
  1. [Section 3.2 and Table 2] The central claim that state-of-the-art models score 7.0% on normal and 0% on hard puzzles requires that the benchmark items are actually solvable in the exact form presented to the models. The paper reports no human-solver baseline on the same text+image or raw-PDF inputs; the stated comparison to 'experienced human puzzle hunters' capabilities' is not backed by any human data in the paper. Because original puzzle events involved team interaction, original documents, and often web access, the static prompt-and-extraction setup used here is uncalibrated. I would ask the authors to either run a human-solvability study on a representative sample of both splits using the same input presentation and answer protocol, or release a public development sample with human-verified solutions so that independent verification is possible. Without this, the 0% hard result is consistent with both genuine reasoning failure and unsolvable or ambiguous items as presented.
  2. [Appendix A.2] The transcription protocol gives annotators substantial discretion: they decide which layouts are 'relevant' and may replace layouts with screenshots, and they are told to 'preserve any text formatting' but also to exclude boilerplate. This creates a risk that spatial or typographic clues essential to the puzzle are lost or altered in the transcribed version. Since the paper interprets low scores on the transcribed format as evidence about reasoning ability, it needs to establish that the transcriptions are informationally sufficient. A human-solve check on the transcribed puzzles would close this gap; at minimum, the paper should report a systematic audit of transcription completeness across sources, not only the few examples shown in Figure 2 and Appendix A.3.
  3. [Section 3.2, 'No evidence of data leakage' paragraph] The claim that 'the low pass rate across all benchmarks suggests that data contamination is not prominent' is circular: low performance cannot distinguish between absence of contamination and contamination that fails to transfer to the benchmark's prompt format. The paper provides no direct contamination test (e.g., perplexity analysis, membership inference, or performance on modified puzzle variants with the same solution mechanism). Given that the dataset is private and the solutions are publicly hosted, the contamination claim is currently unsupported and should be either substantiated with explicit evidence or removed/softened to state that contamination cannot currently be ruled out.
  4. [Appendices B.2 and B.4] The evaluation uses a fixed prompt template and exact string matching on the 'Answer:' line. This protocol can produce false negatives when a model solves the puzzle but formats the answer differently, for example by adding a period, using quotation marks, or placing the answer before a trailing explanation. The reported accuracy numbers therefore conflate reasoning failure with answer-extraction brittleness. To make the comparison fair and cross-model consistent, I recommend either reporting a secondary metric based on flexible matching or human/LLM judging of final answers, or presenting evidence that exact matching does not materially change the results.
minor comments (5)
  1. [Section 1] There is a duplicated word in 'specific specific puzzle types'; please fix this typo.
  2. [Appendix A.2] In Step 2, 'Reformat the "Problem Web Wage"' appears to be a typo for 'Problem Web Page'.
  3. [Table 2] The raw-format column reports accuracy only for the normal split; since the hard split is 0% in text+image format, it would be informative to also state whether the raw-format hard split was measured and whether it was also 0%.
  4. [Section 3.2] The statement that 'the difficulties these models face emerge organically from puzzles requiring sophisticated reasoning' is an interpretation rather than a result; consider phrasing it as a hypothesis supported by the data.
  5. [References] Reference [4] is listed as 'SOME PUZZLES by Mark Halpin' but the text refers to both Cryptic Crosswords and Labor Day Extravaganza as separate sources in Table 1; please clarify the relationship between these sources.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: EnigmaEval's difficulty claim is an external empirical finding, not a consequence of its own definitions or fitted parameters.

full rationale

EnigmaEval is a benchmark paper rather than a derivational one, and its central claim—that frontier vision-language models score 7.0% on the normal split and 0% on the hard split—is an externally measured empirical result. The puzzle answers are sourced from published puzzle events and official solution documents, not derived from model outputs or from the benchmark's own construction. There are no fitted parameters, no equations whose outputs equal their inputs, and no prediction that is statistically forced by a calibration subset. The comparison to Humanity's Last Exam involves overlapping authors (e.g., Dan Hendrycks, Adam Khoja, Summer Yue), but that comparison is not load-bearing: the EnigmaEval scores stand independently in Table 2, and the HLE discussion is contextual related work rather than evidence for the EnigmaEval numbers. The self-audit procedure in Appendix B.3 uses O1 to check O1's own correct answers, but this affects a plagiarism check only and does not define or inflate the reported accuracy. Concerns about the lack of a human-solvability calibration on the exact benchmark inputs are validity or correctness risks, not circularity: they do not show that the benchmark's conclusion reduces to its inputs by construction. Accordingly, the appropriate circularity finding is none.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted. The central claim relies on domain assumptions about transcription fidelity, answer verifiability, and the fairness of the fixed evaluation protocol, and on the ad hoc assumption that exact string matching captures solving ability.

assumptions (4)
  • domain assumption Puzzle answers are unambiguous and verifiable.
    The abstract and Section 1 assert that the benchmark has unambiguous, verifiable solutions; this is required for exact string-matching scoring to be meaningful.
  • domain assumption Human transcriptions preserve semantic relationships and visual elements, and reviewer checks ensure accuracy.
    Section 2.1 says annotations were reviewed by separate human reviewers; the claim that raw-format and transcribed-format comparisons isolate reasoning from parsing depends on this.
  • ad hoc to paper The fixed prompt template and exact 'Answer:' extraction are a fair protocol for measuring model ability.
    Appendices B.2 and B.4 define the protocol, but the paper provides no evidence that this format does not disadvantage models; this is a load-bearing evaluation assumption.
  • ad hoc to paper A model's failure to output the exact expected string means it did not solve the puzzle.
    Section 3.1 uses string matching as the only metric; partial or paraphrased solutions are counted as failures.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EnigmaEval: A Benchmark of Long Multimodal Reasoning Challenges." pith.science (2026). https://pith.science/paper/5NELFENS

@misc{pith2026250208859,
  author       = {Pith},
  title        = {Pith review of: EnigmaEval: A Benchmark of Long Multimodal Reasoning Challenges},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5NELFENS}},
  note         = {Machine review of arXiv:2502.08859}
}
read the original abstract

As language models master existing reasoning benchmarks, we need new challenges to evaluate their cognitive frontiers. Puzzle-solving events are rich repositories of challenging multimodal problems that test a wide range of advanced reasoning and knowledge capabilities, making them a unique testbed for evaluating frontier language models. We introduce EnigmaEval, a dataset of problems and solutions derived from puzzle competitions and events that probes models' ability to perform implicit knowledge synthesis and multi-step deductive reasoning. Unlike existing reasoning and knowledge benchmarks, puzzle solving challenges models to discover hidden connections between seemingly unrelated pieces of information to uncover solution paths. The benchmark comprises 1184 puzzles of varying complexity -- each typically requiring teams of skilled solvers hours to days to complete -- with unambiguous, verifiable solutions that enable efficient evaluation. State-of-the-art language models achieve extremely low accuracy on these puzzles, even lower than other difficult benchmarks such as Humanity's Last Exam, unveiling models' shortcomings when challenged with problems requiring unstructured and lateral reasoning.

Figures

Figures reproduced from arXiv: 2502.08859 by the authors.

Figure 1
Figure 1. While many existing reasoning and/or multimodal benchmarks are largely solved by frontier models, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. A sample of puzzles and solutions from (a) PuzzledPint (link, CC BY-NC-SA Intl. 4.0), (b) Labor Day [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Sudoku-Bench: Evaluating creative reasoning with Sudoku variants

    cs.AI 2025-05 conditional novelty 6.0 of 10

    A new 100-puzzle Sudoku-variant benchmark is hard for frontier LLMs, which solve under 15 percent unaided.

  2. lmgame-Bench: How Good are LLMs at Playing Games?

    cs.AI 2025-05 conditional novelty 6.0 of 10

    lmgame-Bench turns six classic games into a scaffolded LLM evaluation suite, ranks 13 models, detects contamination, and reports RL transfer from Sokoban or Tetris to unseen games and planning tasks.

Reference graph

Works this paper leans on

39 extracted references · 29 canonical work pages · cited by 2 Pith papers

  1. [1]

    https://puzzledpint.org/

    Puzzled Pint. https://puzzledpint.org/. CC BY-NC-SA Intl. 4.0

    work page

  2. [2]

    David J. Malan. CS50x Puzzle Day. https://cs50.harvard.edu/x/2025/puzzles/

    work page 2025

  3. [3]

    Puzzle Potluck

    Bradley Wu, Curtis Liu, Darren Yin, Julz Huang, Lindsey Shi, Rajeev Nayak, and Stephanie Chang. Puzzle Potluck. https://puzzlepotluck.com/

    work page

  4. [4]

    SOME PUZZLES by Mark Halpin

    Mark Halpin. SOME PUZZLES by Mark Halpin. https://www.markhalpin.com/puzzles/puzzles.html

    work page

  5. [5]

    Zach Barnett, Alex Walker, and Sara Walker. CRUMS. https://crumspuzzlehunt.com/. CC BY-NC 3.0

    work page

  6. [6]

    https://puzzles.mit.edu/

    MIT Mystery Hunt. https://puzzles.mit.edu/

    work page

  7. [7]

    Grandmaster Puzzles

    Thomas Snyder. Grandmaster Puzzles. https://www.gmpuzzles.com/about/

    work page

  8. [8]

    Measuring mathematical problem solving with the math dataset, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021

    2021

Show all 39 references
  1. [9]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024

    2024

  2. [10]

    Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai, 2024

    Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, Olli Järviniemi, Matthew Barnett, Robert Sandler, Matej Vrzala, Jaime Sevilla, Qiuyu Ren, Elizabeth Pratt, L...

    2024

  3. [11]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, 2024

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scienti...

    2024

  4. [12]

    Humanity’s Last Exam, 2025

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, Summer Yue, Alexandr Wang, and Dan Hend...

    2025

  5. [13]

    Measuring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021

    2021

  6. [14]

    Mmmu: A massive multi- discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2024

  7. [15]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representat...

    2024

  8. [16]

    Vista: A rubric- based visual task assessment

    Cristina Menghini, Diego Mares, Ernesto Hernandez, Dean Lee, Mike Lunati, and Summer Yue. Vista: A rubric- based visual task assessment. https: // scale. com/ leaderboard/ visual_ language_ understanding, 2024

    2024

  9. [17]

    On the measure of intelligence, 2019

    François Chollet. On the measure of intelligence, 2019

    2019

  10. [18]

    Lanzendörfer, Yannick Niedermayr, and Roger Wattenhofer

    Benjamin Estermann, Luca A. Lanzendörfer, Yannick Niedermayr, and Roger Wattenhofer. PUZZLES: A Benchmark for Neural Algorithmic Reasoning, 2024

    2024

  11. [19]

    PuzzlePlex: A Benchmark to Evaluate the Reasoning and Planning of Large Language Models on Puzzles, 2025

    Anonymous. PuzzlePlex: A Benchmark to Evaluate the Reasoning and Planning of Large Language Models on Puzzles, 2025. 8

    2025

  12. [20]

    PuzzleBench: Can LLMs Solve Challenging First-Order Combinatorial Reasoning Problems? arXiv preprint arXiv:2402.02611, 2024

    Chinmay Mittal, Krishna Kartik, Parag Singla, et al. PuzzleBench: Can LLMs Solve Challenging First-Order Combinatorial Reasoning Problems? arXiv preprint arXiv:2402.02611, 2024

    2024 arXiv

  13. [21]

    Step-by-Step Reasoning to Solve Grid Puzzles: Where do LLMs Falter? arXiv preprint arXiv:2407.14790, 2024

    Nemika Tyagi, Mihir Parmar, Mohith Kulkarni, Aswin Rrv, Nisarg Patel, Mutsumi Nakamura, Arindam Mitra, and Chitta Baral. Step-by-Step Reasoning to Solve Grid Puzzles: Where do LLMs Falter? arXiv preprint arXiv:2407.14790, 2024

    2024 arXiv

  14. [22]

    Riddlesense: Answering riddle questions as commonsense reasoning

    Bill Yuchen Lin, Ziyi Wu, Yichi Yang, Dong-Ho Lee, and Xiang Ren. Riddlesense: Answering riddle questions as commonsense reasoning. CoRR, abs/2101.00376, 2021

    2021 arXiv

  15. [23]

    https://www.melbunimathsstats.org/puzzlehunt

    Melburne Uni Maths and Stats Society. https://www.melbunimathsstats.org/puzzlehunt

  16. [24]

    https://web.archive.org/web/20210725192741/https://www

    Sydney University Mathematics Society. https://web.archive.org/web/20210725192741/https://www. maths.usyd.edu.au/ub/sums/puzzlehunt/2018/index

    2018

  17. [25]

    https://harvardpuzzles.github.io/

    Harvard Mystery Hunt. https://harvardpuzzles.github.io/

  18. [26]

    https://www.mezzacotta.net/puzzle/cisra/

    CISRA. https://www.mezzacotta.net/puzzle/cisra/

  19. [27]

    https://www.janestreet.com/puzzles/archive/index.html

    Jane Street Puzzles. https://www.janestreet.com/puzzles/archive/index.html

  20. [28]

    https://gooooogol.theburninators.org/puzzles/

    Googol Conglomerate. https://gooooogol.theburninators.org/puzzles/

  21. [29]

    https://playdash.org/

    Different Area, Same Hunt. https://playdash.org/

  22. [30]

    https://www.baphl.org/

    Boston Area Puzzle Hunt League. https://www.baphl.org/

  23. [31]

    Forbidden actions

    Galactic Puzzle Hunt. https://2024.galacticpuzzlehunt.com/. 9 Appendix A. Dataset Details A.1 Number of Images per Puzzle Text-only puzzles comprise a small minority of the dataset. While most puzzles have a single key visual component, there is a significant number of puzzles...

    2024

  24. [32]

    Listen to audio or watch a video

  25. [33]

    Be physically present in a location

  26. [34]

    Creatively build, write, or perform something 10

  27. [35]

    Interact with “HQ” or a puzzlemaster

  28. [36]

    Problem Web Wage

    Interact with the web page with the mouse or keyboard Step 2 Reformat the "Problem Web Wage" into a text/image-based format that can be used as input for an automated solver. You do not need to consult the solution or hints for this part. • Leverage web or AI tools to parse te...

    2024

  29. [37]

    Single answer: For puzzles requiring one word or short phrase, Format: Answer: <answer>

  30. [38]

    Answer pairs: For solutions with exactly two elements, Format: Answer: <answer1>, <answer2>

  31. [39]

    This structured approach to answer formats allows us to extract answers consistently and reduces ambiguity when comparing model outputs to ground-truth solutions

    Comma-separated lists: For puzzles requiring multiple answers, Format: Answer: <answer1>, <answer2>, ... This structured approach to answer formats allows us to extract answers consistently and reduces ambiguity when comparing model outputs to ground-truth solutions. C. Additi...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.