Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

EasyARC: Evaluating Vision Language Models on True Visual Reasoning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read EasyARC, a procedurally generated benchmark of simple grid puzzles, claims that state-of-the-art vision-language models mostly fail, with all but one scoring below 20 percent, a sign that true visual reasoning is unsolved.

desk verdict EasyARC is a useful, honest benchmark evaluation with a clear design, but its central claim that it measures true visual reasoning needs a text-only control before it can stand. read the letter →

arxiv 2506.11595 v1 pith:HI5DXARV submitted 2025-06-13 cs.CV cs.LG

Mert Unsal , Aylin Akkus This is my paper
classification cs.CVcs.LG
keywords EasyARCvisualreasoningbenchmarkabstractvision-languagemodelsgridtransformationsproceduralgenerationtest-timescalingmultimodal
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces EasyARC, a procedurally generated benchmark of grid-transformation puzzles modeled on the ARC challenge, and argues that it measures true visual reasoning rather than visual information extraction. On tasks designed to be extremely easy, every tested vision-language model except one scored below 20 percent, a result the authors present as evidence that current models lack genuine visual cognition. The benchmark is designed to be verifiable, since outputs are exact grids, and scalable, since generators can produce unlimited examples with controlled difficulty levels. For a sympathetic reader, the payoff would be a practical testbed for studying test-time scaling and reinforcement learning in visual reasoning, with a clear quantitative target for improvement.

What carries the argument

The central object is the EasyARC generator suite: procedural generators that produce input-output grid pairs for five task classes, each with easy, medium, and hard variants defined by the skills they require. The mechanism that carries the argument is exact-output verification, because a model's answer must match the ground-truth grid exactly, so evaluation is unambiguous and does not depend on free-form answer scoring. A second mechanism is the multi-image task format, which presents several input-output demonstration pairs followed by a single test input and thereby forces the model to form, check, and refine a transformation hypothesis before applying it to the test grid.

What would settle it

Run the EasyARC test set with inputs supplied as textual row-by-row grids to a strong language-only model, and compare its accuracy with the vision-language results; if the text-only model matches or exceeds the vision-language models, then the tasks do not require true visual reasoning and EasyARC measures extraction-plus-text instead.

Watch

Extended reading notes

Core claim

The paper's central claim is that EasyARC is the first benchmark aimed squarely at true visual reasoning in vision-language models, and that state-of-the-art models fail on it even though the tasks are simple by design. The claim rests on an evaluation of five task families (cross/star pattern completion, counting connected components, grid duplication with flips, dominant-side detection, and color-drop mappings) in which all tested models except the best-performing one scored below 20 percent. The paper further argues that this failure is diagnostic: models can often state the transformation rule in words but cannot execute it on the grid, and the one reasoning model examined at length solves tasks by extracting images into text rather than by reasoning visually. In the paper's view, these patterns show that fine-grained spatial reasoning over images is a missing capability in current vision-language models.

Load-bearing premise

The load-bearing premise is that EasyARC tasks genuinely require reasoning over images and cannot be solved by converting the example grids into text and then applying text-based reasoning, yet the paper provides no text-only baseline or perceptual control to demonstrate that this is the case.

Editorial extensions

If this is right

  • EasyARC supplies a verifiable reward signal for reinforcement-learning pipelines, with difficulty levels that can be tuned to shape curriculum learning.
  • Test-time scaling on images becomes measurable: because outputs are exact grids, researchers can ask whether additional inference compute improves true visual reasoning, and the paper's failure analysis suggests it does not yet.
  • The observed failure modes imply that fine-grained perception, such as counting connected components, is a binding constraint even when a model can verbalize the rule.
  • Because new generators can be added without re-annotation, the benchmark can grow beyond its initial five task families while maintaining a clean separation of generated train and test samples.
  • If the benchmark's difficulty calibration is correct, it provides a sharper instrument than existing multimodal benchmarks for isolating where visual reasoning breaks down.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper never runs a text-only baseline, so its own observation of extract-then-reason behavior implies a direct control: if a strong language model solves the puzzles from textual grid transcriptions, EasyARC would be measuring multimodal extraction plus text reasoning rather than visual reasoning alone.
  • The near-zero image-only performance on the original ARC noted in the paper suggests EasyARC's usefulness may be as a calibrated probe: its difficulty levels could isolate individual visual subskills, one generator at a time, in future studies.
  • A natural extension the paper does not explore is to give the model intermediate visual scaffolding, such as highlighted regions or partially constructed outputs, to test whether step-by-step visual interaction, rather than a single pass, is what unlocks grid reasoning.
  • Given the human baseline on the much harder ARC, EasyARC tasks are likely near-ceiling for humans, which means the benchmark measures a model capability gap rather than task ambiguity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces EasyARC, a procedurally generated vision-language benchmark of grid-transformation puzzles inspired by the ARC challenge. It contains five task families (Cross/Star, Counting Cells, Double Grid, Dominant Side, Drop One Color) with three difficulty levels, 5000 training and 500 evaluation tasks, and exact-match evaluation against ground-truth grids. The authors evaluate five state-of-the-art VLMs at temperature 0.5 with one sample per model, reporting that all models except Claude 3.7 Sonnet score below 20% on tasks they describe as extremely easy. They analyze failure modes, including QVQ-72B-Preview's strategy of extracting grids into text before reasoning, and argue that EasyARC provides a scalable, verifiable testbed for true visual reasoning and test-time scaling in RL pipelines.

Significance. If validated, EasyARC would be a useful contribution: it is fully verifiable through exact-match scoring, procedurally expandable through configurable generators, and openly released with evaluation code. The task families and difficulty progression are sensible, and the initial finding that strong VLMs struggle on simple grid transformations is interesting and worth investigating. The caveat is that the paper's central construct---'true visual reasoning'---is not yet established, because the benchmark does not include the text-only or perceptual controls needed to distinguish reasoning over images from textual extraction followed by text-based reasoning.

major comments (4)
  1. [Section 3.5 and Appendix B] The claim that EasyARC measures 'true visual reasoning' rests on the assumption that the tasks cannot be solved by transcribing the image contents into text and then reasoning textually. The paper itself provides evidence that this assumption is insecure: Section 3.5 shows QVQ-72B-Preview explicitly follows an 'extract then reason' strategy, and Appendix B shows the prompt already includes the test input grid in text form. There is no text-only baseline in which the same demonstrations and test input are provided without images. I request such a control: run a strong text-only LLM (or a VLM with images replaced by faithful textual grids) on the same prompts and report accuracy. If a text-only model scores high, the benchmark primarily measures multimodal extraction plus text reasoning, and the paper's stronger claims about a new standard for visual reasoning would need to be substantially revised.
  2. [Section 3] The headline result---that all models except Claude 3.7 Sonnet score below 20%---is based on a single sample per model at temperature 0.5, with no error bars or repeated runs. This is load-bearing for the central empirical claim, because low temperature does not eliminate sampling variance, especially for reasoning models with long generations. I ask the authors to report results over multiple runs (or use deterministic decoding) and provide per-model success rates with confidence intervals or at least raw trial counts, so that the comparison across models is not driven by a single lucky or unlucky sample.
  3. [Section 7.3] The paper characterizes EasyARC as 'extremely easy' and expects human performance to be 'probably significantly above' the H-ARC range of 73.3--77.2%, but no human baseline is collected on EasyARC itself. H-ARC is a harder benchmark, so extrapolating its human scores to EasyARC is not a substitute for actual measurement, especially because EasyARC's counting and fine-grained grid tasks may be nontrivial for humans too. I recommend either running a small human evaluation on a sample of EasyARC tasks or softening the 'extremely easy' and human-expectation claims.
  4. [Section 2.2] The benchmark's verifiability depends on each task having a unique intended transformation rule, but the paper does not describe any automated or formal check that generated samples have unambiguous solutions. Appendix B states that 'to guarantee uniqueness of the solution we make sure to include diverse demonstrations,' yet no procedure or validation is reported. Since exact-match evaluation treats any alternative valid rule as a failure, I ask the authors to either add a uniqueness verification step in the generator or explicitly discuss how demonstration diversity is checked.
minor comments (5)
  1. [Section 7] The opening sentence of Section 7 is incomplete: 'In the following section we give an example for a successful and unsuccessful reasoning trace to give some insight into the' ends mid-phrase and should be finished or removed.
  2. [Section 7.2] There is a typo in the heading: 'unsucessful' should be 'unsuccessful'.
  3. [Appendix B] The prompt text says 'columns are separated by column'; this should read 'separated by spaces' or another concrete delimiter, since the current phrasing is not clear.
  4. [Figure 3] The success-rate bar chart would be easier to interpret if the number of samples per model and per category were stated in the caption or in the text; currently the reader cannot assess the statistical weight of each bar.
  5. [Section 3.2] The phrase 'the model perceives a blurry representation of images' is an interpretive claim; the evidence presented (better performance on easy tasks, worse on noise tasks) supports a coarse-feature hypothesis but not specifically 'blurry' perception. Recasting this as one possible hypothesis would avoid overstatement.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: EasyARC's benchmark results are independently measured against procedurally generated ground truth, and the paper's claims, while contestable, do not reduce to their inputs.

full rationale

The paper contains no fitted parameters, no equations whose outputs reproduce their inputs, and no load-bearing self-citation chain. EasyARC tasks are procedurally generated with known ground-truth outputs, and model performance is scored by exact-match against those outputs, so the reported failure rates are independent measurements. The central claim that EasyARC measures 'true visual reasoning' is a construct-validity claim rather than a circular derivation: the benchmark is designed to require visual reasoning, but whether it actually does is an empirical question. The absence of a text-only control (noted in Section 3.5, where QVQ-72B-Preview is observed to extract grids into text and then reason textually) is a legitimate threat to the interpretation of the results, but it is a missing baseline, not a circular step. The paper also does not rely on self-citation to justify its premise; it cites external sources such as Chollet's ARC and Li et al. for motivation. Therefore, while the paper's stronger claims may be under-supported, they are not circular in the sense of deriving a conclusion from its own definition or from a fitted parameter renamed as a prediction.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The benchmark's validity rests on three domain assumptions inherited from ARC and two ad hoc generation choices; no new theoretical entities are introduced. The central claim depends on the assumption that the tasks require visual reasoning, which is not controlled for.

free parameters (3)
  • difficulty sampling probabilities = easy 0.5, medium 0.35, hard 0.15
    Chosen by hand in Section 2.2 to compose the 5000-train/500-test split; not fitted to any target, but affects the empirical performance distribution.
  • grid size range = uniform between 3 and 10 rows/columns
    Chosen by hand; increases task variety but changes perceptual difficulty.
  • number of demonstrations per task = variable, not specified
    Appendix B says the number of examples is flexibly adjusted to ensure uniqueness, but the algorithm for choosing it is not described, making the dataset generation underspecified.
assumptions (3)
  • domain assumption Exact-match output prediction on procedurally generated grids is a valid measure of abstract visual reasoning.
    Inherited from the ARC challenge (Chollet 2019, cited in Section 2.1) and used to claim EasyARC evaluates true reasoning; not independently validated for this benchmark.
  • domain assumption Rendering example grids as a single stacked image preserves all information needed to solve the task.
    The paper asserts the resolution was set appropriately (Appendix B, Figure 6 caption) but does not test whether VLMs can reliably read every cell from the rendered image.
  • ad hoc to paper Each EasyARC task has a unique intended transformation rule.
    The paper states "To guarantee uniqueness of the solution we make sure to include diverse demonstrations" (Appendix B), but provides no formal uniqueness check, human annotation, or automated verification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EasyARC: Evaluating Vision Language Models on True Visual Reasoning." pith.science (2026). https://pith.science/paper/HI5DXARV

@misc{pith2026250611595,
  author       = {Pith},
  title        = {Pith review of: EasyARC: Evaluating Vision Language Models on True Visual Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HI5DXARV}},
  note         = {Machine review of arXiv:2506.11595}
}
read the original abstract

Building on recent advances in language-based reasoning models, we explore multimodal reasoning that integrates vision and text. Existing multimodal benchmarks primarily test visual extraction combined with text-based reasoning, lacking true visual reasoning with more complex interactions between vision and language. Inspired by the ARC challenge, we introduce EasyARC, a vision-language benchmark requiring multi-image, multi-step reasoning, and self-correction. EasyARC is procedurally generated, fully verifiable, and scalable, making it ideal for reinforcement learning (RL) pipelines. The generators incorporate progressive difficulty levels, enabling structured evaluation across task types and complexities. We benchmark state-of-the-art vision-language models and analyze their failure modes. We argue that EasyARC sets a new standard for evaluating true reasoning and test-time scaling capabilities in vision-language models. We open-source our benchmark dataset and evaluation code.

Figures

Figures reproduced from arXiv: 2506.11595 by the authors.

Figure 1
Figure 1. Example Task from EasyARC: The transformation is to [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Example ARC task from the public evaluation set: Vi [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Success Rate of VLMs on EasyARC 3.2. What Can Claude See? Claude 3.7 Sonnet is the only model that demonstrates a degree of true visual reasoning. To analyze its limitations, we break down its success rate by problem category [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Claude 3.7 success rate across problem types. [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]
Figure 5
Figure 5. Figure 5: Example Input-Output for Counting Cells Task [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: The goal is to predict the color of the lower side of the [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Cross/Star Task: The complexity increases as more lines and colors are introduced. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Counting Cells: The difficulty increases from simple counting to detecting and analyzing contiguous regions. [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Double Grid Task: Complexity grows from simple copying to flipping and repeating with transformations. [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Dominant Side Task: Increasing difficulty as noise complicates detection. [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]
Figure 11
Figure 11. Figure 11: Drop One Color Task: Increasing complexity in color transformation and background dynamics. [PITH_FULL_IMAGE:figures/full_fig_p010_11.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Can VLMs Reason Robustly? A Neuro-Symbolic Investigation

    cs.LG 2026-03 conditional novelty 6.0 of 10

    End-to-end fine-tuned VLMs fail to induce reasoning functions under object-count covariate shifts; VLC (VLM concepts + circuits) yields consistently higher OOD accuracy.

  2. VL-Cogito: Progressive Curriculum Reinforcement Learning for Advanced Multimodal Reasoning

    cs.CV 2025-07 conditional novelty 6.0 of 10

    VL-Cogito, trained with progressive curriculum RL, online difficulty weighting, and dynamic length rewards, matches or beats prior reasoning MLLMs on ten multimodal benchmarks.

Reference graph

Works this paper leans on

22 extracted references · 14 canonical work pages · cited by 2 Pith papers

  1. [1]

    The surprising effectiveness of test-time training for abstract reasoning, 2024

    Ekin Aky ¨urek, Mehul Damani, Linlu Qiu, Han Guo, Yoon Kim, and Jacob Andreas. The surprising effectiveness of test-time training for abstract reasoning, 2024. Preprint at https://arxiv. org/abs/2411.07279, 2024. 2

    arXiv 2024

  2. [2]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision , pages 2425– 2433, 2015. 1

    2015

  3. [3]

    Geoqa: A geometric question answering benchmark towards multimodal numeri- cal reasoning

    Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P Xing, and Liang Lin. Geoqa: A geometric question answering benchmark towards multimodal numeri- cal reasoning. arXiv preprint arXiv:2105.14517, 2021. 1

    arXiv 2021

  4. [4]

    On the measure of intelligence

    Franc ¸ois Chollet. On the measure of intelligence. arXiv preprint arXiv:1911.01547, 2019. 2

    arXiv 1911

  5. [5]

    New 50 arc result and current winners inter- viewed

    Jack Cole, Mohamed Osman, Michael Hodel, Keith Duggar, and Tim Scarfe. New 50 arc result and current winners inter- viewed. Podcast, 2024. Available on Spotify. 1

    work page 2024

  6. [6]

    re-arc: Reverse-engineering the abstrac- tion and reasoning corpus

    Michael Hodel. re-arc: Reverse-engineering the abstrac- tion and reasoning corpus. https://github.com/ michaelhodel/re-arc, 2024. 2

    work page 2024

  7. [7]

    Clevr: A diagnostic dataset for compositional language and elementary visual reasoning

    Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901–2910, 2017. 1

    work page 2017

  8. [8]

    H-arc: A robust estimate of human perfor- mance on the abstraction and reasoning corpus benchmark

    Solim LeGris, Wai Keen V ong, Brenden M Lake, and Todd M Gureckis. H-arc: A robust estimate of human perfor- mance on the abstraction and reasoning corpus benchmark. arXiv preprint arXiv:2409.01374, 2024. 3

    arXiv 2024

Show all 22 references
  1. [9]

    Is programming by example solved by llms? arXiv preprint arXiv:2406.08316, 2024

    Wen-Ding Li and Kevin Ellis. Is programming by example solved by llms? arXiv preprint arXiv:2406.08316, 2024. 2

    2024 arXiv

  2. [10]

    Combining induc- tion and transduction for abstract reasoning

    Wen-Ding Li, Keya Hu, Carter Larsen, Yuqing Wu, Si- mon Alford, Caleb Woo, Spencer M Dunn, Hao Tang, Michelangelo Naim, Dat Nguyen, et al. Combining induc- tion and transduction for abstract reasoning. arXiv preprint arXiv:2411.02272, 2024. 2

    2024 arXiv

  3. [11]

    Inter-gps: Interpretable geometry problem solving with formal language and sym- bolic reasoning

    Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and sym- bolic reasoning. arXiv preprint arXiv:2105.04165, 2021. 1, 4 EasyARC: Evaluating Vision Language Models o...

    2021 arXiv

  4. [12]

    Cross/Star The Cross/Star task involves generating structured patterns based on highlighted cells

    Appendix A: Details of the tasks 5.1. Cross/Star The Cross/Star task involves generating structured patterns based on highlighted cells. The complexity increases as fol- lows: • Easy: Given a uniform background with a single distinct cell, the model must generate a row, column...

  5. [13]

    Prompt We use the following prompt for evaluating VLMs

    Appendix B: Evaluation 6.1. Prompt We use the following prompt for evaluating VLMs. Here is an image of multiple input-output grids. The cells of the grids can be of color black, blue, red, green, yellow, grey, pink, orange, teal, maroon. Your goal is to find the common rule t...

  6. [14]

    Successful Reasoning trace This reasoning trace belongs to a problem of the category counting cells with difficulty easy

    Example Reasoning traces In the following section we give an example for a successful and unsuccessful reasoning trace to give some insight into the 7.1. Successful Reasoning trace This reasoning trace belongs to a problem of the category counting cells with difficulty easy. T...

  7. [15]

    First example: A red grid (6x6) with one yellow cell produces a yellow output of size 1x2

  8. [16]

    Second example: A green grid (6x6) with one teal cell produces a teal output of size 1x1

  9. [17]

    Third example: A green grid (6x6) with two gray cells produces a gray output of size 1x2. The transformation rule appears to be: - The output grid has the same color as the non-dominant color in the input grid - The size of the output depends on the number of non-dominant colo...

  10. [18]

    Identify the position of the unique colored cell in the input grid

  11. [19]

    Create a diagonal line of that same color starting from the unique cell and extending to the bottom-right corner

  12. [20]

    The diagonal moves one cell right and one cell down with each step Let me apply this rule to the test input grid:

  13. [21]

    The unique colored cell is yellow at position (7,4) (row 8, column 5)

  14. [22]

    I need to create a diagonal line of yellow cells from this position to the bottom-right corner Starting from the yellow cell at (7,4), I’ll create a diagonal line moving down-right: - Position (7,4): yellow (already exists) - Position (8,5): should be yellow The final grid wou...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.