Pith. sign in

REVIEW 4 major objections 76 references

Closed-source frontier models beat open-weight peers by about 10% on human-easy tasks that still stump modern AI, even when scores match on standard benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 09:29 UTC pith:RZ72U5OO

load-bearing objection Useful public multimodal stress test with real leaderboard work; the ~10% closed–open gap is measured carefully but sits on a student-adversarial sample that may overfit 2025 chatbot quirks. the 4 major comments →

arxiv 2607.08317 v1 pith:RZ72U5OO submitted 2026-07-09 cs.AI

Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models

Matteo Santelmo , Xiuying Wei , Israa Fakih , Felix Bauer , Juan Garcia Giraldo , Chengkun Li , Etienne Bamas , Emmanuel Abb\'e This is my paper
classification cs.AI
keywords multimodal benchmarksblind spotsvision-language modelsimage generationmodel evaluationtask taxonomyopen-weight vs closed modelsperceptual counting
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Modern AI systems look strong on established benchmarks yet still fail at problems humans find almost trivial—exact string lengths, counting objects in a photo, or drawing a dog with five legs. This paper builds Blind-Spots-Bench: 235 such open-ended questions collected from students, cleaned, given structured reference solutions, and labeled with a three-category taxonomy covering object-centric skills, abstract reasoning, and language-and-knowledge. An automated grading pipeline, checked against humans, scores dozens of language, vision-language, and image-generation models. The main result is that closed-source frontier systems substantially outperform open-weight models—roughly a 10% accuracy gap on text-only items—even among models that look comparable on broad intelligence indices. No single model leads every subcategory, and some skills, especially fine-grained visual counting and pattern recognition, stay hard for all systems. The benchmark is meant as a diagnostic stress test that surfaces concrete weaknesses aggregate leaderboards under-measure.

Core claim

On Blind-Spots-Bench, closed-source frontier models substantially outperform open-weight models—about a 10% accuracy gap on text-only problems—even when those models attain comparable scores on established benchmarks such as the Artificial Analysis Intelligence Index. Fine-grained taxonomy analysis shows no single model dominates all task types, and some subtasks, notably perceptual counting and attribute/pattern recognition, remain difficult for every evaluated system.

What carries the argument

Blind-Spots-Bench: a 235-sample multimodal set of human-easy, model-hard open-ended tasks, equipped with structured reference solutions, a taxonomy of three high-level categories and twelve subcategories, and an AI grading pipeline with human-validated agreement. That package turns student-elicited failures into comparable scores that separate models which look equal on aggregate public benchmarks.

Load-bearing premise

The load-bearing premise is that student-invented questions meant to stump late-2025 frontier chatbots, after cleaning and difficulty filtering, form a fair map of persistent blind spots rather than a catalog of those particular models’ quirks.

What would settle it

If an open-weight model with Artificial Analysis Intelligence Index scores comparable to a top closed model matches or exceeds that closed model’s mean accuracy on Blind-Spots-Bench text-only and multi-to-text splits—especially perceptual counting and character-level manipulation—the claimed closed–open robustness gap would not hold.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Aggregate public benchmarks can overstate robustness on underrepresented skills that humans find trivial.
  • Open-weight models can deliver better accuracy per unit inference cost on these tasks even when absolute accuracy lags.
  • Tool use such as code execution is not uniformly helpful and can lower accuracy when models mishandle tool inputs.
  • Scaling size within a model family does not consistently improve every subtask; larger variants sometimes regress on specific categories.
  • Taxonomy-structured stress tests can reveal complementary model strengths that a single overall score hides.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Training and evaluation optimized for widely used public suites may systematically under-weight character-level control, counting, and spatial binding.
  • The closed–open gap on this style of constraint-heavy prompt is a natural target for open post-training experiments that would test whether the gap is architectural or data-driven.
  • Exact-count and inverted-spatial failures in image generation may share roots with VLM counting errors, favoring joint multimodal diagnostics over separate image and language suites.
  • A measured human baseline on the same 235 items would turn the “easy for humans” claim into a quantified model–human gap useful for product and safety risk assessment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 0 minor

Summary. The paper introduces Blind-Spots-Bench, a 235-item multimodal benchmark of open-ended tasks that are intended to be easy for humans but hard for current AI systems. Items were collected from graduate students asked (around October 2025) to propose failures of frontier chatbots, then cleaned, annotated with structured reference solutions, question formats, and a three-category / 12-subtask taxonomy (object-centric, abstract reasoning, language-and-knowledge). The authors build an Inspect-AI grading pipeline (Gemini-3-flash grader with code execution), validate grader–human agreement (96.6% text, 90.9% image) and same-provider bias, and evaluate 32 LLMs/VLMs plus 6 image-generation models with mean@k / pass@k, cost, and token reporting. Main empirical claims are: (i) closed-source frontier models outperform open-weight models by roughly 10% on text-only items even at comparable Artificial Analysis Intelligence Index scores; (ii) open models can be more cost-effective; (iii) tool use is not uniformly helpful; (iv) no single model dominates all subtasks, and fine-grained visual perception (e.g., perceptual counting, attribute/pattern recognition) remains hard for all systems.

Significance. If the sampling and evaluation hold up, the work is a useful diagnostic complement to saturated aggregate benchmarks: it ships a public dataset with structured solutions, a reproducible grading harness, multi-format coverage (text-only, multi-to-text, image-gen), cost–accuracy trade-offs, tool-use ablations, and taxonomy-level breakdowns that show complementary model strengths rather than a single ranking. The grader validation and same-provider bias check are stronger than typical LLM-as-judge practice. The closed–open gap at matched AAII and the shared weakness on counting/attribute tasks are concrete, actionable findings for robustness research. Significance is tempered by modest size, subtask imbalance, and student-adversarial construction, so the primary value is as a stress test and analysis framework rather than a definitive measure of general capability.

major comments (4)
  1. §3.1 and Limitations: the central diagnostic claim—that the ~10% closed–open gap (Abstract; Fig. 1; Table 1) and the ranking of “hard for all” subtasks reflect persistent blind spots in modern models—depends on treating student-proposed failures of October-2025 frontier chatbots, after cleaning and difficulty thresholding, as a representative stress set. That process can over-weight quirks of the specific systems students probed (e.g., character-length tricks, particular counting/spatial prompts). The paper flags this but still presents headline comparative results as general. Please either (a) report which models students primarily failed against and analyze item difficulty stratified by that provenance, or (b) reframe claims as results on this adversarial construction and add a small held-out or independently authored item set to test whether the closed–open gap and subtask hardness or
  2. §4.3, Table 8 / Table 4: fine-grained conclusions (e.g., “even the strongest models obtain only 41.67% and 57.14%” on attribute/pattern recognition and perceptual counting; “no single model remains top-1 across all tasks”) rest on very small and uneven subtask counts (attribute recognition n=6; constraint reasoning n=9; several image-gen abstract cells n=1–2). With mean@4 and stderr, these percentages are unstable and can flip rankings. Either pool rare subtasks into coarser categories for primary claims, report bootstrap CIs / exact counts in the main text, or clearly mark which subtask comparisons are exploratory only.
  3. §3.1 Review and Quality Control and Limitations: the premise that tasks are “almost trivial” / “easy for humans” is not quantified. Difficulty thresholding removed items “easily solved by models or overly difficult for humans,” but no human accuracy, time, or agreement study is reported. Without a human baseline (even on a stratified subset), the human–model gap that motivates the benchmark remains asserted. A modest human study on a representative sample would substantially strengthen the central framing.
  4. §4.1 / Table 1 vs Fig. 1a: the claim that closed models outperform open models “even when they attain comparable performance on existing benchmarks” is important and only partially supported. Fig. 1a shows a positive AAII correlation with a visual open/closed separation, but there is no matched-pair or regression analysis controlling for AAII (or cost). Please quantify the residual closed–open gap at fixed AAII (e.g., regression with family fixed effects or nearest-neighbor matching) so the “even at comparable AAII” claim is statistical rather than visual.

Circularity Check

0 steps flagged

Empirical leaderboard paper: accuracies are measured against independently curated reference solutions, not derived from a theory that redefines its targets.

full rationale

Blind-Spots-Bench is a diagnostic evaluation paper, not a first-principles derivation. The load-bearing chain is: (1) collect student-proposed failures, clean/annotate with structured reference solutions, (2) grade model outputs against those solutions via an automated grader, (3) report comparative accuracies and correlate with external AAII scores. None of these steps reduces a claimed prediction to its inputs by construction. Reference solutions and correctness criteria are written independently of the models under test; mean@k / pass@k are empirical measurements, not fitted identities. The AAII comparison is correlational (Fig. 1), not a circular re-expression of Blind-Spots-Bench scores. The taxonomy is data-driven organization of the collected items, not a uniqueness theorem or ansatz that forces the reported rankings. Using gemini-3-flash as grader while ranking Gemini models is a potential bias concern, but the paper reports human–grader agreement and disaggregated FPR (Tables 5–6) and does not treat grader agreement as a derived theoretical result. Difficulty thresholding selects hard items by design (standard for stress tests) but does not force the closed–open gap, cost–accuracy trade-offs, or cross-family complementary strengths. No self-definitional equations, fitted-input-as-prediction, load-bearing self-citation uniqueness claims, or renamed known results appear in the derivation chain. Sampling bias (Limitations) is a representativeness/correctness issue, not circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

As an empirical benchmark paper, load-bearing structure is methodological rather than axiomatic physics. The central comparisons rest on: (i) the student-sourced item pool after cleaning, (ii) human-written reference solutions and failure modes, (iii) the AI grader plus binary C/I rule, and (iv) fixed generation settings (thinking medium, k=4 text, no tools by default). No free parameters are fitted to produce a theoretical constant; configuration choices still affect absolute scores and should be treated as free experimental knobs.

free parameters (4)
  • grader_model_choice
    gemini-3-flash is chosen as the sole automatic grader; absolute accuracies and relative rankings can shift under a different judge.
  • k_attempts_text
    mean@4 / pass@4 for text and multi-to-text; image-gen uses mean@1. Changes k changes reported success rates.
  • thinking_effort_and_max_tokens
    Thinking mode on, medium effort when available, max 32,768 output tokens; these knobs affect both accuracy and cost.
  • difficulty_thresholding_rules
    Manual removal of items that are too easy for models or too hard for humans is a human filter that shapes the final 235-item distribution.
axioms (4)
  • domain assumption Tasks that are easy for humans but failed by frontier models of ~Oct 2025 are informative persistent blind spots rather than transient product bugs.
    Stated motivation in abstract/§1 and collection protocol in §3.1; underpins diagnostic value of the benchmark.
  • domain assumption Structured reference solutions plus binary AI grading (with code tools) are a sufficiently faithful proxy for human correctness judgments.
    Evaluation pipeline §4.1; partially validated by human agreement tables but still assumed for the full leaderboard.
  • ad hoc to paper The three-category / 12-subtask taxonomy captures the main skill dimensions needed to interpret failures on this dataset.
    Taxonomy is tailored to the collected set (§3.1, Table 4); multi-label items use response-dependent failure modes.
  • domain assumption Comparable AAII (or similar public index) scores imply roughly matched general capability for interpreting residual Blind-Spots-Bench gaps.
    Used in Fig. 1 and §4.2 alignment discussion to argue the closed–open gap is not just overall capability.
invented entities (2)
  • blind-spots-bench dataset (235 items) independent evidence
    purpose: Provide a fixed set of human-easy, model-hard multimodal items with reference solutions for diagnostic evaluation.
    Core artifact of the paper; independent evidence is the public HF release and evaluation logs, not an external physical measurement.
  • Blind-spots task taxonomy (object-centric / abstract reasoning / language-and-knowledge + 12 subtasks) no independent evidence
    purpose: Decompose accuracy into skill-level failure modes for model comparison.
    Author-defined labeling scheme tailored to this collection; useful but not independently validated against an external skill ontology.

pith-pipeline@v1.1.0-grok45 · 29785 in / 3348 out tokens · 34783 ms · 2026-07-10T09:29:37.143201+00:00 · methodology

0 comments
read the original abstract

Modern AI models achieve strong performance on many established benchmarks, yet they still fail on tasks that humans find almost trivial, such as manipulating a string or drawing a dog with five legs. These examples suggest that existing benchmarks may under-measure persistent blind spots in current systems. We introduce $\texttt{blind-spots-bench}$, a benchmark designed to expose such blind spots through tasks that appear simple for humans but remain challenging for modern AI. We collect raw questions from students in an AI course, clean and annotate them with structured reference solutions, and propose a task taxonomy tailored to the resulting dataset of 235 samples. We further develop an automated grading pipeline to evaluate a wide range of models, including open-weight and closed-source language, vision-language, and image-generation models. Our analysis on $\texttt{blind-spots-bench}$ reveals that closed-source frontier models can substantially outperform open-weight models with even $\approx10\%$ gap, even when they attain comparable performance on existing benchmarks. A more fine-grained analysis shows that no single model dominates across all task types, and that some tasks remain challenging for all evaluated models. These results highlight the value of $\texttt{blind-spots-bench}$ as a diagnostic stress test for identifying concrete weaknesses in current modern models.

Figures

Figures reproduced from arXiv: 2607.08317 by Chengkun Li, Emmanuel Abb\'e, Etienne Bamas, Felix Bauer, Israa Fakih, Juan Garcia Giraldo, Matteo Santelmo, Xiuying Wei.

Figure 1
Figure 1. Figure 1: Left: Accuracy on blind-spots-bench vs. Artificial Analysis Intelligence Index score for text-only problems. Right: Performance of four VLM models on several sub-tasks. ∗Equal contribution. Correspondence to <matteo.santelmo@epfl.ch>, <xiuying.wei@epfl.ch>. Matteo led the evaluation and Xiuying led dataset cleaning and labeling. Dagger† denotes advising roles. Preprint. arXiv:2607.08317v1 [cs.AI] 9 Jul 202… view at source ↗
Figure 2
Figure 2. Figure 2: Representative examples from the dataset. While these tasks are generally easy for human, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Left: Question format composition. Right: Task category composition. Some ques￾tions (about 15) involve multiple subtask categories; for these cases, we count one occurrence for each applicable subtask. The bar chart reports the total number of occurrences for each fine-grained task category, grouped into three major categories. ing 53 times, consistent with prior observations that vision-language models s… view at source ↗
Figure 4
Figure 4. Figure 4: Accuracy on blind-spots-bench vs. average cost for 100 samples. Colors distinguish model families; models of the same version but different sizes are connected [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Performance comparison of leading image generation and VLM models on the largest [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Average accuracy for each model on each task, on [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Average accuracy for each model on each task, on [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Average accuracy for each model on each task, on [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

76 extracted references · 76 canonical work pages · 36 internal anchors

  1. [1]

    DeepSeek-V4: Technical report.https://huggingface.co/deepseek-ai/ DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf, 2026

    DeepSeek-AI. DeepSeek-V4: Technical report.https://huggingface.co/deepseek-ai/ DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf, 2026

    work page 2026

  2. [2]

    Introducing GPT-5.5.https://openai.com/index/introducing-gpt-5-5/, 2026

    OpenAI. Introducing GPT-5.5.https://openai.com/index/introducing-gpt-5-5/, 2026

    work page 2026

  3. [3]

    Gemini 3.1 Pro Model Card.https://deepmind.google/models/ model-cards/gemini-3-1-pro/, 2026

    Google DeepMind. Gemini 3.1 Pro Model Card.https://deepmind.google/models/ model-cards/gemini-3-1-pro/, 2026

    work page 2026

  4. [5]

    GPQA: A Graduate-Level Google-Proof Q&A Benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark.arXiv preprint arXiv:2311.12022, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  5. [6]

    Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan.τ2-bench: Eval- uating conversational agents in a dual-control environment.arXiv preprint arXiv:2506.07982, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  6. [7]

    Swe-bench: Can language models resolve real-world github issues? InThe twelfth international conference on learning representations, 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? InThe twelfth international conference on learning representations, 2023

    work page 2023

  7. [8]

    Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs

    Jasper Dekoninck, Nikola Jovanovi ´c, Tim Gehrunger, K ´ari R ¨ognvalddson, Ivo Petrov, Chen- hao Sun, and Martin Vechev. Beyond benchmarks: Matharena as an evaluation platform for mathematics with llms, 2026. URLhttps://arxiv.org/abs/2605.00674

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  8. [9]

    MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathe- matical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  9. [10]

    Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233. Springer, 2024

    work page 2024

  10. [11]

    Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9556–9567, 2024. 10

    work page 2024

  11. [12]

    Instruction-Following Evaluation for Large Language Models

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  12. [13]

    Sudoku-Bench: Evaluating creative reasoning with Sudoku variants

    Jeffrey Seely, Yuki Imajuku, Tianyu Zhao, Edoardo Cetin, and Llion Jones. Sudoku-bench: Evaluating creative reasoning with sudoku variants.arXiv preprint arXiv:2505.16135, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  13. [14]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024

    work page 2024

  14. [15]

    Why Do Large Language Models (LLMs) Struggle to Count Letters?

    Tairan Fu, Raquel Ferrando, Javier Conde, Carlos Arriaga, and Pedro Reviriego. Why do large language models (llms) struggle to count letters?arXiv preprint arXiv:2412.18626, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  15. [16]

    BabyVision: Visual Reasoning Beyond Language

    Liang Chen, Weichu Xie, Yiyan Liang, Hongfeng He, Hans Zhao, Zhibo Yang, Zhiqi Huang, Haoning Wu, Haoyu Lu, Yiping Bao, et al. Babyvision: Visual reasoning beyond language. arXiv preprint arXiv:2601.06521, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  16. [17]

    Enhancing LLM Character-Level Manipulation via Divide and Conquer

    Zhen Xiong, Yujun Cai, Bryan Hooi, Nanyun Peng, Zhecheng Li, and Yiwei Wang. Enhancing llm character-level manipulation via divide and conquer.arXiv preprint arXiv:2502.08180, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  17. [18]

    Com- positional generalization from first principles.Advances in Neural Information Processing Systems, 36:6941–6960, 2023

    Thadd ¨aus Wiedemer, Prasanna Mayilvahanan, Matthias Bethge, and Wieland Brendel. Com- positional generalization from first principles.Advances in Neural Information Processing Systems, 36:6941–6960, 2023

    work page 2023

  18. [19]

    Make it count: Text-to-image generation with an accurate number of objects

    Lital Binyamin, Yoad Tewel, Hilit Segev, Eran Hirsch, Royi Rassin, and Gal Chechik. Make it count: Text-to-image generation with an accurate number of objects. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 13242–13251, 2025

    work page 2025

  19. [20]

    Lost in Time: Clock and Calendar Understanding Challenges in Multimodal LLMs

    Rohit Saxena, Aryo Pradipta Gema, and Pasquale Minervini. Lost in time: Clock and calendar understanding challenges in multimodal llms.arXiv preprint arXiv:2502.05092, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  20. [21]

    Constraintbench: Bench- marking llm constraint reasoning on direct optimization.arXiv preprint arXiv:2602.22465, 2026

    Joseph Tso, Preston Schmittou, Quan Huynh, and Jibran Hutchins. Constraintbench: Bench- marking llm constraint reasoning on direct optimization.arXiv preprint arXiv:2602.22465, 2026

    work page arXiv 2026

  21. [22]

    SpatiaLab: Can Vision-Language Models Perform Spatial Reasoning in the Wild?

    Azmine Toushik Wasi, Wahid Faisal, Abdur Rahman, Mahfuz Ahmed Anik, Munem Shahriar, Mohsin Mahmud Topu, Sadia Tasnim Meem, Rahatun Nesa Priti, Sabrina Afroz Mitu, Md Iqramul Hoque, et al. Spatialab: Can vision-language models perform spatial reasoning in the wild?arXiv preprint arXiv:2602.03916, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  22. [23]

    Evaluating the Logical Reasoning Abilities of Large Reasoning Models

    Hanmeng Liu, Yiran Ding, Zhizhang Fu, Chaoli Zhang, Xiaozhang Liu, and Yue Zhang. Evaluating the logical reasoning abilities of large reasoning models.arXiv preprint arXiv:2505.11854, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  23. [24]

    R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation

    Meng-Hao Guo, Jiajun Xu, Yi Zhang, Jiaxi Song, Haoyang Peng, Yi-Xuan Deng, Xinzhi Dong, Kiyohiro Nakayama, Zhengyang Geng, Chen Wang, et al. R-bench: Graduate-level multi-disciplinary benchmarks for llm & mllm complex reasoning evaluation.arXiv preprint arXiv:2505.02018, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  24. [25]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri `a Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on machine learning research, 2023

    work page 2023

  25. [26]

    Beyond accuracy: Behavioral testing of nlp models with checklist

    Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of nlp models with checklist. InProceedings of the 58th annual meeting of the association for computational linguistics, pages 4902–4912, 2020

    work page 2020

  26. [27]

    Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference

    R Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. InProceedings of the 57th annual meeting of the association for computational linguistics, pages 3428–3448, 2019. 11

    work page 2019

  27. [28]

    Dynabench: Rethink- ing benchmarking in nlp

    Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. Dynabench: Rethink- ing benchmarking in nlp. InProceedings of the 2021 conference of the North American chapter of the Association for Computational Linguistics: human language technologies, pages 4110...

    work page 2021

  28. [29]

    Evaluating models’ local deci- sion boundaries via contrast sets

    Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, et al. Evaluating models’ local deci- sion boundaries via contrast sets. InFindings of the Association for Computational Linguistics: EMNLP 2020, pages 1307–1323, 2020

    work page 2020

  29. [30]

    MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, Rongrong Ji, Caifeng Shan, and Ran He. Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv: 2306.13394, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  30. [31]

    MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMs

    Jiakang Yuan, Tianshuo Peng, Yilei Jiang, Yiting Lu, Renrui Zhang, Kaituo Feng, Chaoyou Fu, Tao Chen, Lei Bai, Bo Zhang, et al. Mme-reasoning: A comprehensive benchmark for logical reasoning in mllms.arXiv preprint arXiv:2505.21327, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  31. [32]

    MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabil- ities.arXiv preprint arXiv:2308.02490, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  32. [33]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP, pages 353–355, 2018

    work page 2018

  33. [34]

    SIV-Bench: A Video Benchmark for Social Interaction Understanding and Reasoning

    Fanqi Kong, Weiqin Zu, Xinyu Chen, Yaodong Yang, Song-Chun Zhu, and Xue Feng. Siv- bench: A video benchmark for social interaction understanding and reasoning.arXiv preprint arXiv:2506.05425, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  34. [35]

    A survey on large language model reasoning failures

    Peiyang Song, Pengrui Han, and Noah Goodman. A survey on large language model reasoning failures. In2nd AI for Math Workshop@ ICML 2025, 2025

    work page 2025

  35. [36]

    GSM-plus: A compre- hensive benchmark for evaluating the robustness of LLMs as mathematical problem solvers

    Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. GSM-plus: A compre- hensive benchmark for evaluating the robustness of LLMs as mathematical problem solvers. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguis- tics (V olume 1: Long Papers). Association for Computational Linguistics, aug 2024. URL https:/...

    work page 2024

  36. [37]

    MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations

    Anonymous. Math-perturb: Benchmarking llms’ math reasoning abilities against hard pertur- bations.arXiv, 2025. doi: 10.48550/arXiv.2502.06453

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2502.06453 2025

  37. [38]

    GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models

    Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reason- ing in large language models.arXiv, 2025. doi: 10.48550/arxiv.2410.05229

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2410.05229 2025

  38. [39]

    Varbench: Robust language model benchmarking through dynamic variable perturbation

    Kun Qian, Shunji Wan, Claudia Tang, Youzhi Wang, Xuanming Zhang, Maximillian Chen, and Zhou Yu. Varbench: Robust language model benchmarking through dynamic variable perturbation. InFindings of the Association for Computational Linguistics: EMNLP 2024. As- sociation for Computational Linguistics, 2024. URLhttps://aclanthology.org/2024. findings-emnlp.950/

    work page 2024

  39. [40]

    Longbench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. InProceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), pages 3119–3137, 2024

    work page 2024

  40. [41]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. InFindings of the association for computational linguistics: ACL 2022, pages 2263–2279, 2022. 12

    work page 2022

  41. [42]

    Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models

    Ilias Stogiannidis, Steven McDonagh, and Sotirios A Tsaftaris. Mind the gap: Benchmarking spatial reasoning in vision-language models.arXiv preprint arXiv:2503.19707, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  42. [43]

    Do vision language models rotate in mind? evalu- ating spatial transformation reasoning, 2026

    Yiwei Zhang, Yixuan Li, and Song Gao. Do vision language models rotate in mind? evalu- ating spatial transformation reasoning, 2026. URLhttps://openreview.net/forum?id= up2LD7vVdW

    work page 2026

  43. [44]

    Lvlm- count: Enhancing the counting ability of large vision-language models.arXiv preprint arXiv:2412.00686, 2024

    Muhammad Fetrat Qharabagh, Mohammadreza Ghofrani, and Kimon Fountoulakis. Lvlm- count: Enhancing the counting ability of large vision-language models.arXiv preprint arXiv:2412.00686, 2024

    work page arXiv 2024

  44. [45]

    Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought

    Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought.arXiv preprint arXiv:2210.01240, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  45. [46]

    Counting Ability of Large Language Models and Impact of Tokenization

    Xiang Zhang, Juntai Cao, and Chenyu You. Counting ability of large language models and impact of tokenization.arXiv preprint arXiv:2410.19730, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  46. [47]

    Inspect AI: Framework for Large Language Model Evaluations, May

    UK AI Security Institute. Inspect AI: Framework for Large Language Model Evaluations, May

    work page

  47. [48]

    MIT License

    URLhttps://github.com/UKGovernmentBEIS/inspect_ai. MIT License

    work page

  48. [49]

    Qwen3 Technical Report

    Qwen Team. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025. URLhttps: //arxiv.org/abs/2505.09388

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  49. [50]

    Qwen3.5-Omni Technical Report

    Qwen Team. Qwen3.5-omni technical report.arXiv preprint arXiv:2604.15804, 2026. URL https://arxiv.org/abs/2604.15804

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  50. [51]

    GLM-5: from Vibe Coding to Agentic Engineering

    Aohan Zeng et al. GLM-5: From vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763, 2026. URLhttps://arxiv.org/abs/2602.15763

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  51. [52]

    Kimi K2.5: Visual Agentic Intelligence

    Kimi Team. Kimi K2.5: Visual agentic intelligence.arXiv preprint arXiv:2602.02276, 2026. URLhttps://arxiv.org/abs/2602.02276

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  52. [53]

    Gemma 4 model card.https://ai.google.dev/gemma/docs/core/model_ card_4, 2026

    Google. Gemma 4 model card.https://ai.google.dev/gemma/docs/core/model_ card_4, 2026

    work page 2026

  53. [54]

    gpt-oss-120b & gpt-oss-20b Model Card

    Sandhini Agarwal et al. gpt-oss-120b and gpt-oss-20b model card.arXiv preprint arXiv:2508.10925, 2025. URLhttps://arxiv.org/abs/2508.10925

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  54. [56]

    URLhttps://arxiv.org/abs/2507.06261

    work page internal anchor Pith review Pith/arXiv arXiv

  55. [57]

    Gemini 3 Pro Model Card.https://storage.googleapis.com/ deepmind-media/Model-Cards/Gemini-3-Pro-Model-Card.pdf, 2025

    Google DeepMind. Gemini 3 Pro Model Card.https://storage.googleapis.com/ deepmind-media/Model-Cards/Gemini-3-Pro-Model-Card.pdf, 2025

    work page 2025

  56. [58]

    OpenAI GPT-5 System Card

    OpenAI. OpenAI GPT-5 System Card.arXiv preprint arXiv:2601.03267, 2025. URLhttps: //arxiv.org/abs/2601.03267

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  57. [59]

    Introducing GPT-5.4.https://openai.com/index/introducing-gpt-5-4/, 2026

    OpenAI. Introducing GPT-5.4.https://openai.com/index/introducing-gpt-5-4/, 2026

    work page 2026

  58. [60]

    Efficient memory management for large lan- guage model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. InProceedings of the 29th symposium on operating systems principles, pages 611–626, 2023

    work page 2023

  59. [61]

    Artificial Analysis Intelligence Benchmarking Methodology: Artificial Analysis Intelligence Index v4.0.4.https://artificialanalysis.ai/methodology/ intelligence-benchmarking, 2026

    Artificial Analysis. Artificial Analysis Intelligence Benchmarking Methodology: Artificial Analysis Intelligence Index v4.0.4.https://artificialanalysis.ai/methodology/ intelligence-benchmarking, 2026

    work page 2026

  60. [62]

    Humanity's Last Exam

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, et al. Humanity’s last exam.arXiv preprint arXiv:2501.14249, 2025. doi: 10.48550/arXiv.2501.14249...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2501.14249 2025

  61. [63]

    Generalizing Verifiable Instruction Following

    Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hannaneh Hajishirzi. Generalizing verifiable instruction fol- lowing.Advances in Neural Information Processing Systems, 38, 2025. URLhttps: //arxiv.org/abs/2507.02833

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  62. [64]

    Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces

    Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu, Or- feas Menis Mastromichalakis, Zhiwei Xu, et al. Terminal-bench: Benchmarking agents on hard, rea...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2601.11868 2026

  63. [65]

    Vision Language Models are Biased

    An V o, Khai-Nguyen Nguyen, Mohammad Reza Taesiri, Vy Tuong Dang, Anh Totti Nguyen, and Daeyoung Kim. Vision language models are biased.arXiv preprint arXiv:2505.23941, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  64. [66]

    Inksight: Offline-to-online handwriting conversion by teaching vision-language models to read and write.Transactions on Machine Learning Research, 2025

    Blagoj Mitrevski, Arina Rak, Julian Schnitzler, Chengkun Li, Andrii Maksai, Jesse Berent, and Claudiu Cristian Musat. Inksight: Offline-to-online handwriting conversion by teaching vision-language models to read and write.Transactions on Machine Learning Research, 2025

    work page 2025

  65. [67]

    Inkslop: Vibe-coded benchmark for spatial reasoning with digital ink, 2026

    Andrii Maksai. Inkslop: Vibe-coded benchmark for spatial reasoning with digital ink, 2026. URLhttps://inkslop.github.io. Benchmark for evaluating VLMs on digital ink tasks

    work page 2026

  66. [68]

    Measuring Compositional Generalization: A Comprehensive Method on Realistic Data

    Daniel Keysers, Nathanael Sch ¨arli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, et al. Mea- suring compositional generalization: A comprehensive method on realistic data.arXiv preprint arXiv:1912.09713, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1912

  67. [69]

    Scaling can lead to compositional gener- alization.arXiv preprint arXiv:2507.07207, 2025

    Florian Redhardt, Yassir Akram, and Simon Schug. Scaling can lead to compositional gener- alization.arXiv preprint arXiv:2507.07207, 2025

    work page arXiv 2025

  68. [70]

    Learning to Count Objects in Natural Images for Visual Question Answering

    Yan Zhang, Jonathon Hare, and Adam Pr ¨ugel-Bennett. Learning to count objects in natural images for visual question answering.arXiv preprint arXiv:1802.05766, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  69. [71]

    Tallyqa: Answering complex counting questions

    Manoj Acharya, Kushal Kafle, and Christopher Kanan. Tallyqa: Answering complex counting questions. 33(01):8076–8084, 2019. doi: 10.1609/aaai.v33i01.33018076. URLhttps:// ojs.aaai.org/index.php/AAAI/article/view/4815

    work page doi:10.1609/aaai.v33i01.33018076 2019

  70. [72]

    Spatial reasoning in multimodal large language models: A survey of tasks, benchmarks and methods

    Weichen Liu, Qiyao Xue, Haoming Wang, Xiangyu Yin, Boyuan Yang, and Wei Gao. Spatial reasoning in multimodal large language models: A survey of tasks, benchmarks and methods. arXiv preprint arXiv:2511.15722, 2025

    work page arXiv 2025

  71. [73]

    Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change.Advances in Neural Information Processing Systems, 36:38975–38987, 2023

    work page 2023

  72. [74]

    Counterfactual Invariance to Spurious Correlations: Why and How to Pass Stress Tests

    Victor Veitch, Alexander D’Amour, Steve Yadlowsky, and Jacob Eisenstein. Counterfac- tual invariance to spurious correlations: Why and how to pass stress tests.arXiv preprint arXiv:2106.00545, 2021. 14 A Examples of problems To illustrate a practical example for the annotation and task taxonomy, we provide the following prompt from the dataset: Example 1 ...

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  73. [75]

    **Content Accuracy**: Does the image contain the required elements/objects/scenes?

    work page

  74. [76]

    **Instruction Following**: Does the image follow the specific instructions in the prompt?

    work page

  75. [77]

    **Quality**: Is the generated image of reasonable quality (not corrupted, incomplete, or nonsensical)?

    work page

  76. [78]

    C" for CORRECT: The generated image satisfies the task requirements -

    **Relevance**: Is the generated image relevant to the task? If needed, you can execute Python code snippets to help verify aspects of the generated image (e.g., counting objects, checking colors, analyzing dimensions, etc.). After your analysis, provide your final grade. Reply with ’GRADE:$LETTER’ (without quotes) where LETTER is either: - "C" for CORRECT...

    work page 2025