Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

VisualSphinx: Large-Scale Synthetic Vision Logic Puzzles for RL

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VisualSphinx shows that RL training on over 660,000 automatically generated visual logic puzzles, produced for under $1,000, sharply improves a vision-language model's logical reasoning and transfers to algebraic, arithmetic, and geometry…

desk verdict A genuinely useful synthetic dataset for visual reasoning; treat the RL experiment as a demo, not evidence for the causal claim. read the letter →

arxiv 2505.23977 v1 pith:JAXERQ6O submitted 2025-05-29 cs.CV cs.AIcs.LG

Yichen Feng , Zhangchen Xu , Fengqing Jiang , Yuetai Li , Bhaskar Ramasubramanian , Luyao Niu , Bill Yuchen Lin , Radha Poovendran This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords visualreasoningsyntheticdatasetvision-languagemodelsreinforcementlearningGRPOlogicpuzzlesrule-to-imagesynthesisMathVista
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that large-scale synthetic visual logic puzzles are a viable and cheap way to train vision-language models to reason. The authors build VisualSphinx, a dataset of over 660K visual puzzles, each grounded in an explicit rule and assembled from program-generated images, and show that a 7-billion-parameter VLM trained with GRPO on a 10K-puzzle sample improves from 29.3% to 55.9% on a manually checked test set, beating several much larger closed models. It also reports transfer gains on MathVista, raising average accuracy from 59.4% to 64.0%, with improvements in algebraic, arithmetic, geometry, and logical reasoning. If this is right, it would make high-quality reasoning data for multimodal RL accessible at negligible cost and would suggest that structured synthetic puzzles can sharpen general visual reasoning.

What carries the argument

The load-bearing mechanism is the rule-to-image synthesis pipeline: each puzzle is generated from an explicit five-bullet visual rule, a rule-level genetic algorithm diversifies the rules, and an LLM writes Python scripts (Matplotlib, PIL, or NumPy) that render five images obeying the rule and three images breaking it. The fifth compliant image is the designated answer and the three violators are distractors, so every puzzle carries a verifiable ground truth without human labeling. GRPO then turns that ground truth into a binary reward for RL training.

What would settle it

Sample a few hundred puzzles from the 660K training set and have independent human solvers check that the designated answer is uniquely consistent with the stated rule; if a nontrivial share (say, more than a few percent) are ambiguous or wrong, the RL signal would be teaching artifact-matching rather than logic.

Watch

Extended reading notes

Core claim

The central discovery is that rule-grounded synthetic puzzles, not human-annotated data, can supply the verified reward signal that RL needs for multimodal reasoning. VisualSphinx starts from 2.4K seed rules abstracted from Chinese civil-service exam questions, expands them to 40K rules with a rule-level genetic algorithm, renders each rule into five compliant and three violating images via LLM-written Python scripts, and assembles the images into 660K four-option and ten-option puzzles. Training Qwen2.5-VL-7B with GRPO on 10K of these puzzles raises its accuracy on the VisualSphinx test set from 29.3% to 55.9% and produces transferable gains on MathVista, including logical, algebraic, arithmetic, and geometric reasoning. The authors interpret this as evidence that logical coherence and readability of the synthetic data are what drive the improvement.

Load-bearing premise

The generated labels are correct: the LLM-written Python scripts really render five images that follow the intended rule and three that break it, even though only the 930-puzzle test set was checked by hand.

Editorial extensions

If this is right

  • A 7B open VLM trained on VisualSphinx can outperform much larger closed models on visual logic puzzles, suggesting model scale is not the only route to reasoning ability.
  • The improvements transfer to MathVista, where the trained model gains on logical, algebraic, arithmetic, and geometric reasoning, implying visual logic training can strengthen broader multimodal math reasoning.
  • Since the dataset costs under $1,000 to generate, this recipe makes multimodal RL training data affordable for groups without large annotation budgets.
  • The three rendering styles and answer-shuffling variants reduce position and style overfitting, so the model must attend to relational patterns rather than visual surface features.
  • The pipeline can scale to more rules and styles without additional human effort, so the 660K puzzle count is not an upper bound on the dataset.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the gains come from rule-grounded logical coherence rather than style, then adding harder rule types such as temporal sequences or interactive puzzles should extend the same transfer; this is a testable prediction.
  • Editorial inference: a direct way to check whether the model learned general logic or generator artifacts is to hold out entire rendering styles or genetic-rule lineages; if accuracy drops sharply on held-out styles, part of the gain is artifact-specific.
  • Editorial inference: the same rule-to-image pipeline could be adapted to other domains, such as physics diagrams, circuit diagrams, or spatial navigation, wherever executable code can render rule-compliant and rule-violating instances, providing unlimited reward-grounded training data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces VisualSphinx, a synthetic dataset of over 660K visual logic puzzles spanning inductive, deductive, spatial, and structural reasoning categories. A four-stage pipeline—seed-question collection and rule abstraction, a rule-level genetic algorithm for rule expansion, LLM-generated Python scripts that render five rule-compliant and three rule-violating images per rule, and three puzzle assembly strategies—produces the dataset at a total cost under $1000. The authors then train Qwen2.5-VL-7B with GRPO on a 10K-example subset and report accuracy gains on a manually checked 930-puzzle test set (29.30% to 55.94%) and on MathVista-testmini (59.4% to 64.0% overall), with subcategory improvements in algebraic, arithmetic, geometric, and logical reasoning.

Significance. The dataset construction is a genuine engineering contribution: the pipeline is fully automated, low-cost, and yields a large, diverse set of visual logic puzzles with a manually verified test set and an accompanying human quality study (78% of 1,200 sampled puzzles rated comparable or better than seed puzzles). If the empirical claims are confirmed with proper controls, VisualSphinx would be a valuable, reproducible resource for VLM post-training. However, the current evidence for the causal role of 'logical coherence and readability' is incomplete: the experiments lack control conditions, multiple seeds, and uncertainty quantification, and the training-label verification is limited to the small test set.

major comments (4)
  1. [Section 5.2, Table 1, Figure 7] The central causal claim that GRPO on VisualSphinx improves logical reasoning and transfers to algebra, arithmetic, and geometry is not isolated by the reported experiments: the only comparison is between one post-RL run of Qwen2.5-VL-7B and its untrained checkpoint. There is no control run with GRPO on matched data that lacks VisualSphinx's logical structure (e.g., the same puzzles with shuffled answers, incoherent image sequences, or seed-only questions), no second seed, and no error bars. Under these conditions the +26.64-point gain on VisualSphinx-TEST is consistent with in-distribution overfitting or a generic GRPO/format effect, and the +4.6-point MathVista gain (Figure 8) could arise from longer chain-of-thought or better answer formatting rather than from 'logical coherence and readability' of the dataset. Please add at least one control GRPO condition and report multiple seeds or bootstrap confidence intervals.
  2. [Section 3, Step 3; Section 4] Label correctness is load-bearing and is verified only on the 930-puzzle VisualSphinx-TEST. The 660K training puzzles (and the 10K RL subset selected in Section 5.1) are generated by LLM-written Python scripts and filtered by pHash/SSIM and VLM-based readability/coherence scores; a script that renders images not actually satisfying the extracted rule would teach the model to match generator artifacts. The paper should report a human audit of a random sample of the 10K RL examples (or an automated check that the rendered images provably satisfy the rule constraints), with per-category agreement, and should state the distribution of VLM coherence scores for the RL subset.
  3. [Section 4.1, Pass Rate; Section 5.1] The difficulty filter used to select the RL training set is partly circular: pass rates come from a Qwen2.5-VL-7B model trained on the same seed-question distribution, and the readability/coherence filter uses GPT-4.1-mini, a VLM similar in capability to the model that generated the images. Selecting examples with pass rate 0.375-0.875 and coherence >= 8 could preferentially retain puzzles where the generator's artifacts are recognized by the scorer rather than puzzles with genuinely sound logic. Please show that the main results are robust to the selection thresholds, for example by also training on a random or differently filtered subset of the 110K groups.
  4. [Figure 8] The transfer claim rests on aggregate and subcategory accuracy on MathVista-testmini with no measures of uncertainty or per-category sample sizes; some subcategories (e.g., LOG, NUM) are small, so the reported gains may be within noise. In addition, without a control RL run the transfer gain could be a general effect of RL training on any visually grounded multiple-choice data. Please provide per-category counts, confidence intervals, and a matched control.
minor comments (5)
  1. [Section 5.1 and Table 4] The text says training uses a batch size of 128, whereas Table 4 reports Rollout Batch Size 64 and PPO Mini Batch Size 64; please reconcile these numbers.
  2. [Section 3, Step 4] Please clarify whether the six additional distractors in the 10-option variant are chosen to be rule-violating with respect to the target rule, or merely drawn from other image groups; the current wording is ambiguous.
  3. [Section 4, Figure 5] The text does not specify which LLM produced the readability and coherence scores in Figure 5; please state the model and, ideally, provide a small human agreement study on a subsample.
  4. [Appendix D] The human quality study reports that 935/1200 puzzles were comparable or superior to seed puzzles, but no inter-annotator agreement or detailed annotation instructions are provided; please include them or cite an annotation protocol.
  5. [Section 5.1] The statement that VisualSphinx-TEST is 'fully synthetic, ensuring no overlap with existing benchmarks' does not by itself rule out overlap with the training subset; please report the exact deduplication method (e.g., hash-based) used to separate the 930 test puzzles from the 10K RL examples.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the VisualSphinx pipeline, RL training, and MathVista evaluation are not defined in terms of the quantities they predict.

full rationale

VisualSphinx's derivation chain is not circular under the criteria defined here. The dataset is produced by an automated pipeline: seed rules are abstracted from seed questions, expanded by a rule-level genetic algorithm, rendered by LLM-written Python scripts into image groups, and assembled into puzzles. The RL model is then trained with GRPO on a filtered 10K-puzzle sample and evaluated on a manually checked 930-puzzle held-out test split and on the external MathVista benchmark. The in-domain test set is a held-out sample of the same generator, which limits the strength of external-generalization claims, but it is not a reduction of the measured accuracy to the training input: the test puzzles are distinct from the training puzzles, and no fitted parameter used during training is equal by construction to the reported accuracy. The pass-rate-based difficulty labels come from a separate annotation model trained on seed questions and are used only for stratification and data filtering, not as the performance metric; the readability and logical-coherence scores are dataset-quality filters, not the output of the RL evaluation. The MathVista transfer result is grounded in an external benchmark and therefore provides independent evidence. The only author-overlapping citation (Magpie, reference [43]) appears in related work and is not load-bearing in the derivation. The absence of a control condition, single-seed runs, and unverified synthetic labels for the full 660K corpus are experimental-design and correctness risks, but they do not constitute a circular argument in the paper's own derivation chain.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a chain of LLM-generated components, with multiple hand-chosen thresholds and a self-referential quality loop. I list the explicit thresholds and the assumptions that LLM outputs are faithful and that VLM-based quality measures are valid.

free parameters (6)
  • pHash Hamming distance threshold = 10
    Images with Hamming distance below 10 are removed as duplicates in Section 3, Step 3.
  • SSIM blank threshold = 0.1
    Images with SSIM below 0.1 against a white reference are flagged as blank (Section 3, Step 3).
  • Rule retention score thresholds = total score > 12, feasibility >= 3
    Rules are kept only if they pass these LLM-scored thresholds in Section 3, Step 2.
  • Pass rate selection window = 0.375 to 0.875
    Training puzzles are sampled with pass rates in this window (Section 5.1).
  • Grayscale gradient energy threshold = calibrated, not stated
    The paper says images below a calibrated threshold are discarded, without giving the value (Section 3, Step 3).
  • Genetic algorithm parameters = 10 generations, 10% migration every 3 generations
    Evolution hyperparameters chosen to balance diversity and coherence (Section 3, Step 2).
assumptions (5)
  • domain assumption Seed puzzles from the Chinese Civil Service Examination have unique, logically valid answers.
    The pipeline starts from 4K seed questions assumed to be well-posed; LLM verification, not formal proof, is used (Section 3, Step 1).
  • domain assumption LLM-generated Python scripts faithfully implement the intended rules for correct images and genuinely violate them for distractors.
    Step 3 assumes rule-to-code translation works; there is no per-image human check for the 660K training examples.
  • domain assumption VLM readability and logical coherence scores are accurate proxies for human quality judgments.
    Section 4 uses GPT-4.1-mini scores, and Appendix D reports a 1,200-puzzle human sample without calibrating the automated scores to it.
  • domain assumption Pass rate measured by a model trained on the same seed distribution is a valid difficulty measure.
    Section 4.1 uses a Qwen2.5-VL-7B annotation model trained on filtered seed questions to define difficulty bins.
  • domain assumption GRPO training on VisualSphinx, rather than the extra computation or RL itself, causes the observed MathVista gains.
    There is no control RL run on a different dataset under identical conditions (Section 5).

how reviews work

0 comments
Cite this review

Pith. "Pith review of VisualSphinx: Large-Scale Synthetic Vision Logic Puzzles for RL." pith.science (2026). https://pith.science/paper/JAXERQ6O

@misc{pith2026250523977,
  author       = {Pith},
  title        = {Pith review of: VisualSphinx: Large-Scale Synthetic Vision Logic Puzzles for RL},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JAXERQ6O}},
  note         = {Machine review of arXiv:2505.23977}
}
read the original abstract

Vision language models (VLMs) are expected to perform effective multimodal reasoning and make logically coherent decisions, which is critical to tasks such as diagram understanding and spatial problem solving. However, current VLM reasoning lacks large-scale and well-structured training datasets. To bridge this gap, we propose VisualSphinx, a first-of-its-kind large-scale synthetic visual logical reasoning training data. To tackle the challenge of image synthesis with grounding answers, we propose a rule-to-image synthesis pipeline, which extracts and expands puzzle rules from seed questions and generates the code of grounding synthesis image synthesis for puzzle sample assembly. Experiments demonstrate that VLM trained using GRPO on VisualSphinx benefit from logical coherence and readability of our dataset and exhibit improved performance on logical reasoning tasks. The enhanced reasoning capabilities developed from VisualSphinx also benefit other reasoning tasks such as algebraic reasoning, arithmetic reasoning and geometry reasoning.

Figures

Figures reproduced from arXiv: 2505.23977 by the authors.

Figure 1
Figure 1. Example VISUALSPHINX instances within each reasoning category. Each visual logic puzzle comprises a text prompt, a graphical question stem with four images and a question mark, and four candidate choices of graphical answers. *These authors contributed equally to this work. Preprint. arXiv:2505.23977v1 [cs.CV] 29 May 2025 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. This figure illustrates the four-stage pipeline for generating VISUALSPHINX. In Step 1, we collect 4K seed puzzles with explanations and abstract them into structured rule descriptions using LLMs. In Step 2, we apply a rule-level genetic algorithm to cross over, mutate and diversify the seed rules, scaling them to 40K high-quality rules. In Step 3, each rule is paired with a rendering style and used to generate five… view at source ↗
Figure 3
Figure 3. Comparison of t-SNE visualization be￾tween VISUALSPHINX’s synthetic rules and seed rules with 2,000 sampled rules per class. Details about 8 classes are provided in Appendix B.1. 1-2 3 4 5 Score Value 0 20 40 60 80 100 Percentage (%) 0.0% 0.8% 9.7% 89.5% 0.3% 7.8% 35.4% 56.5% 15.2% 35.1% 42.6% 7.1% Format Score Content Quality Score Feasibility Score [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (24 more)
Figure 5
Figure 5. Figure 5: This figure demonstrates the statistics of readability and logical co￾herence. We analyze the quality and difficulty of the 660K syn￾thetic puzzles in VISUALSPHINX using three evaluation attributes: readability, logical coherence, and pass rate. Attribute: Readability.…
Figure 6
Figure 6. Figure 6: This figure demonstrates the statistics of pass rates. Attribute: Pass Rate. We measure the pass rate us￾ing a VLM. We note that current open-source VLMs not trained on these tasks struggle to achieve useful pass rates on visual logic puzzles and like random guess. To …
Figure 7
Figure 7. Figure 7: This figure illustrates the performance of the model trained on V [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: This figure shows model per￾formance before and after RL training on VISUALSPHINX, evaluated on MathVista￾testmini across subcategories. Limitations. While VISUALSPHINX demonstrates significant empirical success in enhancing the logi￾cal reasoning capabilities of VLMs …
Figure 9
Figure 9. Figure 9: Example 1 14 [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Example 2 [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Example 3 [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Example 4 15 [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: Example 5 A.2 Breakdown of Statistics of VISUALSPHINX [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: Five different visual patterns of seeds We note that in principle, these two axes yield 15 possible visual pattern–reasoning style combina￾tions. However, we collapse all combinations involving an Others tag on either axis into a unified Others category for simplicity…
Figure 15
Figure 15. Figure 15: Min Rules Embedding Distance. We quantify the similarity among Step 2 rule-level genetic algorithm to remove repet￾itive rules. We measure the similarity using minimum neighbor distance in the embed￾ding space. Specifically, we first represent all instructions in the …
Figure 16
Figure 16. Figure 16: Two puzzles that are visually similar but semantically different. [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]
Figure 17
Figure 17. Figure 17: Example comparison of five models on the same rule prompt for image generation. [PITH_FULL_IMAGE:figures/full_fig_p019_17.png]
Figure 18
Figure 18. Figure 18: Prompt for rewriting and translating 21 [PITH_FULL_IMAGE:figures/full_fig_p021_18.png]
Figure 19
Figure 19. Figure 19: Prompt for verifying the question 22 [PITH_FULL_IMAGE:figures/full_fig_p022_19.png]
Figure 20
Figure 20. Figure 20: Prompt for abstracting the puzzle into pullet points [PITH_FULL_IMAGE:figures/full_fig_p023_20.png]
Figure 21
Figure 21. Figure 21: Prompt for categorization 24 [PITH_FULL_IMAGE:figures/full_fig_p024_21.png]
Figure 22
Figure 22. Figure 22: Prompt for crossover 25 [PITH_FULL_IMAGE:figures/full_fig_p025_22.png]
Figure 23
Figure 23. Figure 23: Prompt for mutation 26 [PITH_FULL_IMAGE:figures/full_fig_p026_23.png]
Figure 24
Figure 24. Figure 24: Prompt for scoring synthetic rules 27 [PITH_FULL_IMAGE:figures/full_fig_p027_24.png]
Figure 25
Figure 25. Figure 25: Prompt for first rule-to-image rendering style [PITH_FULL_IMAGE:figures/full_fig_p028_25.png]
Figure 26
Figure 26. Figure 26: Prompt for second rule-to-image rendering style [PITH_FULL_IMAGE:figures/full_fig_p029_26.png]
Figure 27
Figure 27. Figure 27: Prompt for third rule-to-image rendering style [PITH_FULL_IMAGE:figures/full_fig_p030_27.png]
Figure 28
Figure 28. Figure 28: Prompt for scoring assembled puzzles 31 [PITH_FULL_IMAGE:figures/full_fig_p031_28.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Trace: A Taxonomy-Guided Environment for Multidomain Visual Reasoning

    cs.CV 2026-07 conditional novelty 6.0 of 10

    RLVR training on 64,000 procedurally generated Trace instances improves Qwen2.5-VL macro-average on 24 external visual reasoning benchmarks by 3.51 points at 3B and 4.06 points at 7B.

Reference graph

Works this paper leans on

92 extracted references · 52 canonical work pages · cited by 1 Pith paper

  1. [1]

    Introducing the next generation of claude

    Anthropic. Introducing the next generation of claude. https://www.anthropic.com/news/ claude-3-family, 2024

    2024

  2. [2]

    Textocr-gpt4v

    Jimmy Carter. Textocr-gpt4v. https://huggingface.co/datasets/jimmycarter/ textocr-gpt4v, 2024

    2024

  3. [3]

    Allava: Harnessing gpt4v- synthesized data for lite vision-language models

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v- synthesized data for lite vision-language models. arXiv preprint arXiv:2402.11684, 2024

    arXiv 2024

  4. [4]

    Towards end-to-end embodied decision making via multi- modal large language model: Explorations with gpt4-vision and beyond

    Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Tianyu Liu, and Baobao Chang. Towards end-to-end embodied decision making via multi- modal large language model: Explorations with gpt4-vision and beyond. arXiv preprint arXiv:2310.02071, 2023

    arXiv

  5. [5]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In Euro- pean Conference on Computer Vision, pages 370–387. Springer, 2024

    2024

  6. [6]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  7. [7]

    Towards reasoning era: A survey of long chain- of-thought for reasoning large language models

    Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: A survey of long chain- of-thought for reasoning large language models. arXiv preprint arXiv:2503.09567, 2025

    arXiv 2025

  8. [8]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

Show all 92 references
  1. [9]

    The faiss library

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazar´e, Maria Lomeli, Lucas Hosseini, and Herv´e J´egou. The faiss library. 2024

    2024

  2. [10]

    Vila 2: Vila augmented vila

    Yunhao Fang, Ligeng Zhu, Yao Lu, Yan Wang, Pavlo Molchanov, Jan Kautz, Jang Hyun Cho, Marco Pavone, Song Han, and Hongxu Yin. Vila 2: Vila augmented vila. arXiv preprint arXiv:2407.17453, 2024

    2024 arXiv

  3. [11]

    ChessGPT: Bridging policy learning and lan- guage modeling

    Xidong Feng, Yicheng Luo, Ziyan Wang, Hongrui Tang, Mengyue Yang, Kun Shao, David Henry Mguni, Yali Du, and Jun Wang. ChessGPT: Bridging policy learning and lan- guage modeling. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023

    2023

  4. [12]

    Gemini Pro

    Google DeepMind. Gemini Pro. https://deepmind.google/technologies/gemini/ pro/, March 2025. Accessed: 2025-04-03

    2025

  5. [13]

    Perceptual image hashing

    Azhar Hadmi 1, William Puech1, and Brahim Ait Es Said. Perceptual image hashing. Water- marking: Volume 2, page 17, 2012

    2012

  6. [14]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  7. [15]

    Gunawi, Cody Hammock, Joe Mambretti, Alexander Barnes, Franc ¸ois Halbah, Alex Rocha, and Joe Stubbs

    Kate Keahey, Jason Anderson, Zhuo Zhen, Pierre Riteau, Paul Ruth, Dan Stanzione, Mert Ce- vik, Jacob Colleran, Haryadi S. Gunawi, Cody Hammock, Joe Mambretti, Alexander Barnes, Franc ¸ois Halbah, Alex Rocha, and Joe Stubbs. Lessons learned from the chameleon testbed. In 2020 U...

    2020

  8. [16]

    Chain-of-region: Visual language models need details for diagram analysis

    Xue Li, Yiyou Sun, Wei Cheng, Yinglun Zhu, and Haifeng Chen. Chain-of-region: Visual language models need details for diagram analysis. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  9. [17]

    RiddleSense: Rea- soning about riddle questions featuring linguistic creativity and commonsense knowledge

    Bill Yuchen Lin, Ziyi Wu, Yichi Yang, Dong-Ho Lee, and Xiang Ren. RiddleSense: Rea- soning about riddle questions featuring linguistic creativity and commonsense knowledge. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors,Findings of the Association for Comp...

    2021

  10. [18]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  11. [19]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 34892–34916. Curran Associates, Inc., 2023

    2023

  12. [20]

    Logiqa: A challenge dataset for machine reading comprehension with logical reasoning

    Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08124, 2020

    2007 arXiv

  13. [21]

    Mathvista: Evaluating mathe- matical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathe- matical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023

    arXiv

  14. [22]

    Mathvista: Evaluating mathemati- cal reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathemati- cal reasoning of foundation models in visual contexts. InThe Twelfth International Conference on Learnin...

    2024

  15. [23]

    Mmevol: Empowering multimodal large language models with evol-instruct

    Run Luo, Haonan Zhang, Longze Chen, Ting-En Lin, Xiong Liu, Yuchuan Wu, Min Yang, Minzheng Wang, Pengpeng Zeng, Lianli Gao, et al. Mmevol: Empowering multimodal large language models with evol-instruct. arXiv preprint arXiv:2409.05840, 2024

    2024 arXiv

  16. [24]

    Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning

    Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun He, et al. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2503.07365, 2025

    2025 arXiv

  17. [25]

    The Llama 4 herd: The beginning of a new era of natively multimodal AI innova- tion

    Meta AI. The Llama 4 herd: The beginning of a new era of natively multimodal AI innova- tion. https://ai.meta.com/blog/llama-4-multimodal-intelligence/ , April 2025. Accessed: 2025-04-05

    2025

  18. [26]

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Ak- shita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, ...

    2025

  19. [27]

    Introducing 4o image generation

    OpenAI. Introducing 4o image generation. OpenAI Website, March 2025. Accessed: 2025- 05-16

    2025

  20. [28]

    Openai o3 and o4-mini system card

    OpenAI. Openai o3 and o4-mini system card. OpenAI Website, April 2025. Accessed: 2025- 05-16

    2025

  21. [29]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, ...

    2021

  22. [30]

    Vgrp-bench: Visual grid reasoning puzzle benchmark for large vision- language models

    Yufan Ren, Konstantinos Tertikas, Shalini Maiti, Junlin Han, Tong Zhang, Sabine S ¨usstrunk, and Filippos Kokkinos. Vgrp-bench: Visual grid reasoning puzzle benchmark for large vision- language models. arXiv preprint arXiv:2503.23064, 2025

    2025 arXiv

  23. [31]

    Decrypting cryptic crosswords: Se- mantically complex wordplay puzzles as a target for NLP

    Joshua Rozner, Christopher Potts, and Kyle Mahowald. Decrypting cryptic crosswords: Se- mantically complex wordplay puzzles as a target for NLP. In A. Beygelzimer, Y . Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Sys- tems, 2021

    2021

  24. [32]

    Programming puzzles

    Tal Schuster, Ashwin Kalyan, Alex Polozov, and Adam Tauman Kalai. Programming puzzles. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Bench- marks Track (Round 1), 2021. 11

    2021

  25. [33]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  26. [34]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys ’25, page 1279–1297. ACM, March 2025

    2025

  27. [35]

    Visualpuz- zles: Decoupling multimodal reasoning evaluation from domain knowledge

    Yueqi Song, Tianyue Ou, Yibo Kong, Zecheng Li, Graham Neubig, and Xiang Yue. Visualpuz- zles: Decoupling multimodal reasoning evaluation from domain knowledge. arXiv preprint arXiv:2504.10342, 2025

    2025 arXiv

  28. [36]

    Qwen2.5-vl, January 2025

    Qwen Team. Qwen2.5-vl, January 2025

    2025

  29. [37]

    Visualizing data using t-sne.Journal of machine learning research, 9(11), 2008

    Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne.Journal of machine learning research, 9(11), 2008

    2008

  30. [38]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instruc- tions. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guis...

    2023

  31. [39]

    Image quality assess- ment: from error visibility to structural similarity

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assess- ment: from error visibility to structural similarity. IEEE transactions on image processing , 13(4):600–612, 2004

    2004

  32. [40]

    Grok 3 beta — the age of reasoning agents

    xAI. Grok 3 beta — the age of reasoning agents. https://x.ai/blog/grok-3, 2025. Accessed: 2025-02-21

    2025

  33. [41]

    Wizardlm: Empowering large language models to follow complex instructions

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023

    2023 arXiv

  34. [42]

    Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models

    Weiye Xu, Jiahao Wang, Weiyun Wang, Zhe Chen, Wengang Zhou, Aijun Yang, Lewei Lu, Houqiang Li, Xiaohua Wang, Xizhou Zhu, et al. Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models. arXiv preprint arXiv:2504.15279, 2025

    2025 arXiv

  35. [43]

    Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. arXiv preprint arXiv:2406.08464, 2024

    2024 arXiv

  36. [44]

    R1-onevision: Advancing generalized multi- modal reasoning through cross-modal formalization

    Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. R1-onevision: Advancing generalized multi- modal reasoning through cross-modal formalization. arXiv preprint arXiv:2503.10615, 2025

    2025 arXiv

  37. [45]

    Griffiths, Yuan Cao, and Karthik R Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  38. [46]

    UReader: Universal OCR-free visually-situated language understanding with multimodal large language model

    Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, Qin Jin, Liang He, Xin Lin, and Fei Huang. UReader: Universal OCR-free visually-situated language understanding with multimodal large language model. In Houda Bouamo...

    2023

  39. [47]

    Sigmoid loss for lan- guage image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for lan- guage image pre-training. In Proceedings of the IEEE/CVF international conference on com- puter vision, pages 11975–11986, 2023

    2023

  40. [48]

    Multimodal self-instruct: Syn- thetic abstract image and visual reasoning instruction using language model

    Wenqi Zhang, Zhenglin Cheng, Yuanyu He, Mengna Wang, Yongliang Shen, Zeqi Tan, Guiyang Hou, Mingqian He, Yanna Ma, Weiming Lu, et al. Multimodal self-instruct: Syn- thetic abstract image and visual reasoning instruction using language model. arXiv preprint arXiv:2407.07053, 2024

    2024 arXiv

  41. [49]

    Mm-rlhf: The next step forward in multimodal llm alignment

    Yi-Fan Zhang, Tao Yu, Haochen Tian, Chaoyou Fu, Peiyan Li, Jianshu Zeng, Wulin Xie, Yang Shi, Huanyu Zhang, Junkang Wu, et al. Mm-rlhf: The next step forward in multimodal llm alignment. arXiv preprint arXiv:2502.10391, 2025. 12

    2025 arXiv

  42. [50]

    Puzzlebench: A fully dynamic evaluation framework for large multimodal models on puzzle solving

    Zeyu Zhang, Zijian Chen, Zicheng Zhang, Yuze Sun, Yuan Tian, Ziheng Jia, Chunyi Li, Xi- aohong Liu, Xiongkuo Min, and Guangtao Zhai. Puzzlebench: A fully dynamic evaluation framework for large multimodal models on puzzle solving. arXiv preprint arXiv:2504.10885, 2025

    2025 arXiv

  43. [51]

    Easyr1: An efficient, scalable, multi-modality rl training framework, 2025

    Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, and Yuwen Xiong. Easyr1: An efficient, scalable, multi-modality rl training framework, 2025

    2025

  44. [52]

    Let’s think step by step and output the final answer within \\boxed{}

    Guanghao Zhou, Panjia Qiu, Cen Chen, Jie Wang, Zheming Yang, Jian Xu, and Minghui Qiu. Reinforced mllm: A survey on rl-based reasoning in multimodal large language models. arXiv preprint arXiv:2504.21277, 2025. 13 A More Examples and V ISUAL SPHINX Statistics A.1 Examples In t...

    2025 arXiv

  45. [53]

    Carefully read and understand the entire Chinese explanation

  46. [54]

    Translate the explanation from Chinese to English, ensuring that you capture the full meaning and logical relationships in the original text

  47. [55]

    As you translate, remove any statements that indicate the answer

  48. [56]

    如下图所示，有三个圆圈

    Remove phrases like "如下图所示" (as shown in the following figure) or any similar references to figures or images. When you remove these phrases, make sure to maintain the original meaning of the statement. For example, if the original text says "如下图所示，有三个圆圈" (as shown in the foll...

  49. [57]

    The goal is to provide a clear, coherent explanation in English that preserves the puzzle's logic without giving away the answer or referring to non-existent figures

    Ensure that you fully understand the logical flow and reasoning of the explanation before finalizing your translation. The goal is to provide a clear, coherent explanation in English that preserves the puzzle's logic without giving away the answer or referring to non-existent figures

  50. [58]

    white + black

    Write your final translated and modified explanation in English inside <translated_explanation> tags. Remember, the key is to provide a faithful translation that maintains the puzzle's logic and difficulty while removing direct answer indicators and references to figures. For ...

  51. [59]

    Carefully examine the image, paying attention to all visual elements, patterns, and relationships between objects

  52. [60]

    Consider the question (and hint) to guide your analysis

  53. [61]

    Identify the underlying logic or rule that governs the puzzle

  54. [62]

    Apply this logic to determine the correct answer

  55. [63]

    Explain your reasoning step by step, referencing specific elements in the image

  56. [64]

    **Answer in English.** ## Output <reasoning> Your reasoning. </reasoning> <answer> Your answer (select only from: A, B, C, D, or Problematic) </answer> Figure 19: Prompt for verifying the question 22 Prompt for abstracting the puzzle into pullet points ## Task Description You ...

  57. [65]

    **Analyze** the puzzle and its explanation thoroughly

  58. [66]

    **Create** a detailed breakdown of the puzzle inside `<detailed_analysis>` tags

  59. [67]

    **Present** your analysis within `<puzzle_breakdown>` tags

  60. [68]

    **Create** a list of key regularities within `<key_points>` tags

  61. [69]

    A": "A",

    **Review & refine** your list ... ### Example Examine the image of the puzzle: <image><!--EXAMPLE_SPLIT--></image> Read the question: <puzzle_question> From the given four options, select the most suitable one to fill in the question mark to present a certain regularity: </puz...

  62. [70]

    Examine every puzzle component

  63. [71]

    Focus on structure and relationships

  64. [72]

    Weigh arguments **for and against** each tag

  65. [73]

    Provide detailed reasoning inside `<puzzle_breakdown>` before giving tags. ## Input Here is the puzzle you need to analyze: <puzzle_image> <!--PUZZLE_SPLIT--> </puzzle_image> <puzzle_question> {{ prompt }} </puzzle_question> <puzzle_options> {{ options_block }} </puzzle_option...

  66. [74]

    Analyze both rule sets: - List out key elements from each rule set - Identify core principles, compatibility, and potential synergies - Look for opportunities to create emergent patterns through combination - Identify potential conflicts and propose resolutions

  67. [75]

    Wrap your detailed analysis in <comparative_analysis> tags: - Compare and contrast the rule sets - Identify complementary regularities and explore potential synergies - Consider application to puzzle generation - Analyze and propose resolutions for potential conflicts

  68. [76]

    Wrap your cross-over synthesis in <synthesis> tags: - Explain how you've integrated elements from both rule sets - Describe how the new combined regularities work together - Illustrate potential puzzle scenarios - Explain the coherence and unity of your cross-over - Discuss ho...

  69. [77]

    Figure 22: Prompt for crossover 25 Prompt for mutation ## Task Description You are an expert in analyzing and evolving complex vision logic puzzles

    Wrap your final cross-over rule set in <crossover_rules> tags: - Each rule should be less than 30 words and incorporate elements from both source rule sets - Aim for 4-6 cohesive items that flow logically and create a complete puzzle system - Ensure the rules could plausibly g...

  70. [78]

    Analyze the existing rule set: - Identify the core principles and key elements - Understand the underlying logic and structure

  71. [79]

    modified

    Wrap your analysis in <analysis> tags: - Describe key insights about the rule structure - Identify opportunities for creative mutation - Explain which aspects should be preserved vs. modified

  72. [80]

    Create a mutated version with controlled changes: - Introduce meaningful variations to one point or two points of the rules - Ensure the mutations maintain logical coherence - Preserve the core puzzle structure and logic - Aim for a mutation that could generate novel but solva...

  73. [81]

    Wrap your final mutated rule set in <mutated_rules> tags: - Each rule should be less than 30 words - Include 4-6 rules total (similar to the input set) - Ensure the mutated rules maintain logical consistency - The mutation should be meaningful but not completely transform the ...

  74. [82]

    Format (1-5 points) 1: Incomplete, missing multiple points, contains unrelated content 2: Incomplete, missing 1-2 points, may contain some unrelated content 3: Complete set of points, but may have some unrelated content 4: Complete set of 5-6 points, minimal unrelated content ...

  75. [83]

    Content Quality (1-5 points) 1: Inconsistent, contradictory, does not describe core concept 2: Some inconsistencies or contradictions, poorly describes core concept 3: Mostly consistent, few contradictions, adequately describes core concept 4: Consistent, no major contradictio...

  76. [84]

    Infer a reasonable implementation according to rules (e.g., based on patterns, shapes, or mathematical properties) and document your reasoning

    Feasibility (1-5 points) 1: Vague, unclear instructions, not suitable for code generation 2: Some clear instructions, but many gaps for code generation 3: Mostly clear instructions, some gaps for code generation 4: Clear instructions, minor gaps for code generation 5: Perfectl...

  77. [86]

    For each script: - Include all imports

    **Incorrect Script**: Generates 3 images, each breaking a different rule. For each script: - Include all imports. - Your code should well-documented and add concise comments explaining rule compliance or violation. - Use relative paths ./output_correct and ./output_incorrect. ...

  78. [88]

    For each script: - Include all imports

    **Incorrect Script**: Generates 3 images, each breaking a different rule. For each script: - Include all imports. - Your code should well-documented and add concise comments explaining rule compliance or violation. - Use relative paths ./output_correct and ./output_incorrect. ...

  79. [89]

    **Correct Script**: Generates 5 continuous images complying the rules

  80. [90]

    For each script: - Include all imports

    **Incorrect Script**: Generates 3 images, each breaking a different rule. For each script: - Include all imports. - Your code should well-documented and add concise comments explaining rule compliance or violation. - Use relative paths ./output_correct and ./output_incorrect. ...

  81. [91]

    **Question Reasonableness Evaluation:** Evaluate whether the provided question and answer align with the rules and are unambiguous

  82. [92]

    ## Input <image> <!--SPLIT--> </image> <question> {{question}} </question> <answer> {{answer}} </answer> <rules> {{rules}} </rules> ## Instructions

    **Overall Visual Readability Evaluation:** Assess the clarity and readability of the entire puzzle image. ## Input <image> <!--SPLIT--> </image> <question> {{question}} </question> <answer> {{answer}} </answer> <rules> {{rules}} </rules> ## Instructions

  83. [93]

    Score this on a scale of 1{5, where 5 is the most reasonable

    **Question Reasonableness Evaluation** Use the provided rules and answer to verify the reasonableness of the question. Score this on a scale of 1{5, where 5 is the most reasonable. Consider these factors: - Does the answer align perfectly with the rule described? - Is there an...

  84. [94]

    Score this on a scale of 1{5

    **Overall Visual Readability Evaluation** Assess the overall visual readability of the puzzle image. Score this on a scale of 1{5. **Score Criteria:** - **5**: Perfectly readable with no issues. - **4**: Minor readability issues, but still easy to understand. - **3**: Moderate...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.