Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Training a VLM to explicitly convert images into text before reasoning transfers text-side simple-to-hard generalization to images and closes most of the modality gap.

desk verdict A genuinely useful controlled benchmark and a plausible training recipe, but the headline S2H claim is overdrawn: the flagship gains are HARD-text transfer, not simple-to-hard generalization, and the motivating gap depends on a post-hoc CoT-trace metric. read the letter →

arxiv 2501.02669 v2 pith:OLYMTOX3 submitted 2025-01-05 cs.CV cs.CLcs.LG

Simon Park , Abhishek Panigrahi , Yun Cheng , Dingli Yu , Anirudh Goyal , Sanjeev Arora This is my paper · ORCID
classification cs.CVcs.CLcs.LG
keywords simple-to-hardgeneralizationmodalityimbalancevision-languagemodelsimage-to-textconversionchain-of-thoughtreasoninggradientalignmentlengthsyntheticvisual
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-language models solve easy visual versions of a task much worse than they solve the same task as text, and they fail to generalize from easy to hard visual instances. This paper argues that a missing step, the model never turning the image into the symbolic content the reasoning operates on, is what blocks the transfer, and shows that teaching explicit image-to-text conversion transfers text-side reasoning to images. On three synthetic tasks with matched image and text versions, adding image-to-text conversion to the training mixture raises HARD-image accuracy from below 25% to 64–92%, and to 76–96% with an added alignment phase. The conversion skill is internalized: at test time the model solves directly from the image, so the transfer costs no extra inference tokens. A gradient alignment score over training data predicts which mixtures generalize, and chain-of-thought traces are indispensable.

What carries the argument

The load-bearing object is the Image-via-Text supervision type: on an image input the model is trained to emit the converted text (LaTeX of the table, text version of the grid, or text description of the puzzle), then the chain-of-thought trace, then the final answer. It appears inside three mixtures: Mix (Text + Image + Image-via-Text on SIMPLE examples), Mix+ (Mix plus HARD-text examples, for tasks where text itself does not S2H-generalize), and Align-Mix+ (a first phase with SIMPLE Text and SIMPLE Image-via-Text, then Mix+). The mechanistic account is a gradient alignment score, $\langle g_{\text{SIMPLE}}, g_{\text{HARD}}\rangle / \langle g_{\text{HARD}}, g_{\text{HARD}}\rangle$, which measures how much an averaged SIMPLE-image gradient update would reduce the HARD-image loss relative to a HARD-image update; higher scores at large gradient norms distinguish Mix from Text+Image and Align-Mix+ from Mix+. The third essential component is the verbose chain-of-thought: ablations that remove CoT or progressively internalize it collapse HARD-image accuracy to roughly zero.

What would settle it

Re-score the HARD split of Consecutive Table Readout using only the final answer block (the reported list of numbers or their sum), ignoring the chain-of-thought trace, for Text- and Image-trained models. If the text/image accuracy gap shrinks to near zero, the paper's motivating modality imbalance is an artifact of its CoT-trace metric. A second test: at inference, replace the internal image-to-text conversion of a Mix+ model with a deliberately corrupted conversion and check whether HARD-image accuracy drops; if it does not, the reasoning transfer is not actually carried by the conversion.

Watch

Extended reading notes

Core claim

The paper's central claim is that explicit image-to-text conversion is the mechanism through which reasoning transfers from the text modality to the image modality. In its cleanest demonstration, supervised training on SIMPLE text examples (tables given as LaTeX) yields 80% accuracy on HARD-text examples, while the same training on SIMPLE images yields only 20% on HARD-image examples; that 60-point gap is the modality imbalance under study. Supervision that trains the model to first convert the image into text (Image-via-Text) and a mixture that combines it with direct image and text training (Mix) close most of the gap, and Mix keeps inference cost low because the conversion is not reproduced at test time. For the three harder tasks where neither modality simple-to-hard generalizes, adding HARD-text examples (Mix+) yields 64%, 92%, and 35% accuracy on HARD images, and a two-phase version with a SIMPLE image-to-text alignment phase (Align-Mix+) reaches 76%, 96%, and 56%. The paper reports that the transfer is carried by training gradients that align SIMPLE-image updates with HARD-image loss, and that chain-of-thought is necessary. It also discloses that for Consecutive Table Readout the reported S2H generalization appears in the chain-of-thought trace and not in the final answer the model gives (Section C.6.1), which is the basis for its headline modality gap.

Load-bearing premise

The motivating 80% versus 20% modality gap for Consecutive Table Readout is computed on the chain-of-thought trace, not the final answer; the paper discloses (Section C.6.1) that only the trace shows simple-to-hard generalization. If final-answer accuracy is the correct metric, the motivating example of modality imbalance weakens.

Editorial extensions

If this is right

  • On tasks where the base LLM already shows simple-to-hard generalization in text, mixing Image-via-Text supervision transfers that generalization to images while keeping generation length low, because the conversion skill is internalized and not reproduced at test time.
  • On tasks where S2H generalization fails in text as well, injecting HARD-text examples (Mix+) is needed; adding a short SIMPLE image-to-text alignment phase first (Align-Mix+) improves the transfer further, so stronger future LLM backbones should amplify the effect.
  • Chain-of-thought is load-bearing: completely removing it or progressively internalizing it during the Mix+ phase drops HARD-image accuracy to near zero across supervision types.
  • Including the synthetic datasets in visual instruction tuning improves real-world benchmarks (at least 3%p on MMMU, 5.1%p on chart descriptive questions), indicating the trained conversion-and-reasoning skill is not confined to the synthetic tasks.
  • The gradient alignment score identifies which training mixture will reduce HARD-image loss before evaluation is needed, which makes it a usable selection signal for supervision design during model development.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's cleanest illustration, the 80% versus 20% text/image gap on Consecutive Table Readout, is scored on the chain-of-thought trace, and the authors disclose that the final answer does not S2H-generalize; re-scoring on final answers is the direct test of whether the motivating gap is as large as claimed.
  • The internalization result suggests a production recipe the paper only sketches: fine-tune with explicit conversion (possibly on lossy text descriptions, since the paper's Visual Analogy text is not image-reconstructing), then rely on the model's internalized shortcut at inference; Mix is already this recipe in one stage.
  • The authors note that minor CoT template changes substantially change transferred accuracy; extrapolating, the method's success is coupled to the trace format, so a robustness-oriented follow-up should randomize trace verbosity during the Mix/Mix+ phase.
  • Because the gradient alignment score predicts which mixture transfers without waiting for HARD evaluation, a natural extension is to use it online to adjust the Text/Image/Image-via-Text ratio during training, rather than measuring it post hoc as done here.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper introduces a synthetic benchmark for studying modality imbalance in vision-language models (VLMs) using three algorithmic visual reasoning tasks, each with SIMPLE and HARD versions and matched text/image inputs. It proposes supervision strategies centered on explicit image-to-text conversion and reports that they improve performance on HARD image inputs. The main claims are: (i) when a VLM already shows S2H generalization on text, Mix supervision transfers part of that behavior to images; (ii) on tasks where neither modality S2H-generalizes, including HARD Text examples in training via Mix+ or Align-Mix+ yields large gains on HARD images; and (iii) gradient-alignment scores correlate with these gains. The paper also reports ablations on chain-of-thought, text warm-up, and multi-task training, and a partial replication on Qwen2.5-VL.

Significance. The controlled task suite and the image-to-text conversion intervention are potentially useful for the VLM community, and the paper is commendable for releasing code, comparing two model families, and reporting a large number of ablations. If the central claim were established for genuinely SIMPLE-only training, this would be an important step. As written, however, the headline 'simple-to-hard' generalization on the three main tasks is not what is tested: the successful protocols train on HARD text examples. The strongest defensible contribution is a more modest one about cross-modal transfer of already-learned HARD text reasoning, plus the observation that image-to-text conversion and chain-of-thought are important for such transfer. The paper should be revised to align its framing with its protocol.

major comments (3)
  1. [Sections 2.3–2.4, 4, Algorithm 1] The paper's formal framework defines S2H generalization as training on X_SIMPLE and evaluating on X_HARD: Section 2.3 splits X into X_SIMPLE and X_HARD, and Section 2.4 describes supervisions applied to SIMPLE examples. Section 4 then says 'We then adapt Mix supervision to include HARD Text in training,' and Algorithm 1 sets N_HARD = N/2 for Image-via-Text+ and Mix+, adding {x(t), CoT(x), f(x)} for x ~ X_HARD. The headline results in Figure 5, including 64/92/35 for Mix+ and 76/96/56 for Align-Mix+, are therefore not demonstrations of simple-to-hard generalization by the paper's own definition; the SIMPLE-only Text/Image/Mix baselines remain below 25%. The abstract's phrase 'training on the SIMPLE version' is accordingly misleading. Please either rename this phenomenon, for example as cross-modal transfer of HARD-text reasoning, or re-derive the claim so that the definition of S2H is the one actually tested. The genuinely S2H results, such as Consecutive Table Readout and Pattern-Heldout Visual Analogy, should be clearly separated from the HARD-text-injection results.
  2. [Section C.6.1 and Section E.1] The motivating modality-gap result for Consecutive Table Readout, Text 80% vs Image 20% on HARD in Figure 3, is measured on the chain-of-thought trace rather than the final answer. Section C.6.1 states: 'we find that the verbose CoT trace shows S2H generalization, and not the final solution that the model reports. Hence, we report our evaluation performance for Consecutive Table Readout on the CoT trace.' Section E.1 says the paper reports the best performance of the two extraction sources. If exact match on the final answer is used instead, the text-side S2H advantage and the modality gap can change substantially. Please report final-answer accuracy separately, state evaluation-set sizes, and justify the CoT-trace choice as the primary metric; otherwise the opening illustration of modality imbalance is not supported by a standard answer-correctness metric.
  3. [Figures 3–9 and Tables 5–8] All accuracy and loss curves are reported without error bars or multiple random seeds, and the evaluation-set sizes are not stated for the main figures (some appendix tables mention 100 or 500 examples). Without variance information, conclusions such as 'Align-Mix+ further enhances generalization' in Figure 5, or the gradient-alignment comparisons in Figures 7–9, cannot be reliably separated from training noise. Please add at least two or three seeds with standard deviations, or state the evaluation-set sizes and, where feasible, bootstrap confidence intervals.
minor comments (4)
  1. [Section 7 / Table 9] The statement that including the synthetic datasets in pretraining 'seems useful for improving the performance of VLMs in real-world settings' should be qualified: Table 9 shows MME falling from 1529 to 1242 with the 30k synthetic mixture and TextVQA declining with the 240k mixture, so the utility is task-dependent.
  2. [Section I.5] There is a typographical error ('puxxle' for 'puzzle') in Section I.5; the appendix should be proofread for similar typos.
  3. [Section 2.4 / Appendix D] The conversion prompt P_convert is described verbally but never quoted; please include the exact prompt template used for reproducibility.
  4. [Section F.3] For the Qwen2.5-VL replication of Consecutive Table Readout, the paper relaxes HARD to MEDIUM; this should be stated in the main-text discussion of cross-family consistency, not only in Appendix F.3.

Circularity Check

0 steps flagged · score 1.0 of 10

No load-bearing circularity: the core results are empirical cross-modal transfers, not derivations from fitted parameters or from the authors' own prior theorems. Two caveats (Mix+ trains on HARD Text; Consecutive Table Readout is scored on the CoT trace) weaken the S2H framing but do not make the claims circular by construction.

full rationale

The paper's central results are measured training outcomes on held-out HARD-image examples; no parameter is fitted to the target and then renamed a prediction, and no uniqueness theorem is imported from the authors' own work. The gradient-alignment score in Eq. (3) is connected to expected loss reduction by Theorem H.1, which is a self-contained Taylor-expansion statement with explicit Lipschitz assumptions, and it is used diagnostically rather than to fit the models. Self-citations (e.g., Park et al. 2023 for gradient projection, and Yu et al. 2024 / Zhao et al. 2024 for compositionality) are background context and are not load-bearing. Two non-circular caveats should be weighed in the verdict. First, for the three non-S2H tasks, the headline Mix+ and Align-Mix+ results in Figure 5 are obtained after adding HARD Text supervision, as disclosed in Section 4 and Algorithm 1; these are therefore cross-modal transfers of already-trained HARD-text reasoning rather than SIMPLE-only S2H generalization. This is a labeling and protocol mismatch, not a reduction by construction: the image-side accuracy could have failed to improve, and the paper reports ablations where it does fail. Second, Section C.6.1 states that 'the verbose CoT trace shows S2H generalization, and not the final solution that the model reports,' so the Consecutive Table Readout metric is the CoT trace; this is an evaluation-choice caveat, not a circular derivation. Because the results are externally benchmarked (Table 9) and the image-side improvements are contingent empirical findings, there is no significant circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central empirical results rest on a few hand-chosen experimental numbers (notably the epoch counts) and on domain assumptions about synthetic-task validity and CoT fidelity. No new physical or ontological entities are introduced; the gradient alignment score is a diagnostic measure, not an entity.

free parameters (1)
  • Number of training epochs for Text and Image supervision on Consecutive Table Readout = Text: 2 epochs; Image: 3 epochs at Nu=40k
    Section G.1 and Figure 13: epoch counts are chosen because Text S2H peaks at 2 epochs and Image improves slightly up to 3; this choice maximizes the reported text-image gap.
assumptions (4)
  • standard math The loss l(I;S) is Lipschitz and has bounded gradient norm on X around model parameters theta (Theorem H.1)
    Used to prove the gradient alignment score quantifies expected loss drop; standard regularity assumptions.
  • domain assumption Synthetic tasks with text-equivalent representations (LaTeX or attribute lists) capture the modality imbalance of real VLMs
    The entire study uses synthetic tasks as a proxy; Section G.6 provides partial external validation on real benchmarks but with mixed results.
  • domain assumption The chain-of-thought traces used for supervision are correct and sufficient to convey the algorithmic procedure
    Training is supervised on gold CoT traces; Section C.6 describes how CoT design was iterated until models could learn, and Section I.7 shows removing CoT destroys transfer.
  • domain assumption The text representation is aligned enough with the image that learning image-to-text conversion transfers reasoning
    Section 2.4 assumes LaTeX/attribute text is a faithful conversion; Section I.5 tests a lossy variant and finds methods still work, partially supporting this assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?." pith.science (2026). https://pith.science/paper/OLYMTOX3

@misc{pith2026250102669,
  author       = {Pith},
  title        = {Pith review of: Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OLYMTOX3}},
  note         = {Machine review of arXiv:2501.02669}
}
read the original abstract

Vision Language Models (VLMs) are impressive at visual question answering and image captioning. But they underperform on multi-step visual reasoning -- even compared to LLMs on the same tasks presented in text form -- giving rise to perceptions of modality imbalance or brittleness. Towards a systematic study of such issues, we introduce a synthetic framework for assessing the ability of VLMs to perform algorithmic visual reasoning, comprising three tasks: Table Readout, Grid Navigation, and Visual Analogy. Each has two levels of difficulty, SIMPLE and HARD, and even the SIMPLE versions are difficult for frontier VLMs. We propose strategies for training on the SIMPLE version of tasks that improve performance on the corresponding HARD task, i.e., simple-to-hard (S2H) generalization. This controlled setup, where each task also has an equivalent text-only version, allows a quantification of the modality imbalance and how it is impacted by training strategy. We show that 1) explicit image-to-text conversion is important in promoting S2H generalization on images, by transferring reasoning from text; 2) conversion can be internalized at test time. We also report results of mechanistic study of this phenomenon. We identify measures of gradient alignment that can identify training strategies that promote better S2H generalization. Ablations highlight the importance of chain-of-thought.

Figures

Figures reproduced from arXiv: 2501.02669 by the authors.

Figure 1
Figure 1. (Left) Example Data Point for Consecutive Table Readout. Input table can be provided as an image or LaTeX code. The task is to sequentially read numbers from a start cell to an end cell in row major order. (Right) Illustration of Key Concepts using examples from Consecutive Table Readout. We observe that current models can S2H generalize on text – when trained to read short sequences from small LaTeX-formatted table… view at source ↗
Figure 2
Figure 2. Illustration of our synthetic tasks: Table Readout involves reading numbers along a specified path in a table. Grid Navigation involves navigating a grid to collect objects while avoiding obstacles. Visual Analogy involves solving analogical reasoning queries using two in-context examples. More details on Visual Analogy: For the example of Visual Analogy above, we only include one in-context example for simplicity a… view at source ↗
Figure 3
Figure 3. S2H Generalization of different supervisions for Con￾secutive Table Readout to MEDIUM (left) and HARD (right) ex￾amples. S2H generalization on text of Text (⋆) outperforms S2H generalization on image of Image (▲), highlighting modality im￾balance. Mix (•) mitigates this imbalance. adapted supervision strategies will have a + sign appended to represent this additional component (e.g., Mix+ adapted from Mix supervisio… view at source ↗
Figures from the paper (34 more)
Figure 4
Figure 4. Figure 4: Effect of Image-via-Text on Consecutive Table Read￾out: S2H Generalization (left) and Generation Length (right) for HARD task. Number of training data is 16 × 104 . Text+Image un￾derperforms Mix and Image-via-Text supervision. Image-via-Text supervision improves perfor…
Figure 5
Figure 5. Figure 5: Results on non S2H-generalizing tasks: We report the S2H generalization on image on Table Readout (left), Grid Navigation (middle), and Visual Analogy (right). S2H generalization on text from Text supervision serves as a reference (in gray dashed line). Text, Image, an…
Figure 6
Figure 6. Figure 6: Image-via-Text+ on Table Readout and Visual Analogy: S2H Generalization on image (left) and Generation Length (right) with 12 × 104 training examples. Image-via-Text+ achieves good performance but with higher inference cost. Mix+ matches the performance of Image-via-Te…
Figure 7
Figure 7. Figure 7: Analysis of gradients on Consecutive Table Read￾out: (Left) Average Gradient Norm on SIMPLE Image examples (Ex∈XSIMPLE ∥∇l(I;S)(x)∥2) vs. Gradient Alignment Score (Equa￾tion (3)) for different training checkpoints; (Right) Average Loss on solution given HARD image (l (…
Figure 8
Figure 8. Figure 8: Analysis of evaluation losses on HARD examples on Table Readout: (Left) HARD image-to-text conversion loss (l (H) (I#;T ) (Eq.4)); (Middle) loss on solution given HARD image and text (l (H) (I,#T ;S) (Eq.5)); (Right) loss on solution given HARD image (l (H) (I;S) (Eq.2…
Figure 9
Figure 9. Figure 9: Analysis of gradients on Table Readout: Average Gra￾dient Norm on SIMPLE Image examples(Ex∈XSIMPLE ∥∇l(I;S)(x)∥2) vs. Gradient Alignment Score for different training checkpoints. Larger gradients for Align-Mix+ have higher gradient alignment scores. Mix+ has better gra…
Figure 10
Figure 10. Figure 10: Pseudo-code for generating data mixture: For ablation studies, the algorithm might be slightly modified. C.2. Consecutive Table Readout Given a table with nr rows and nc columns, a start cell (rs, cs) and an end cell (re, ce), the model is tasked to read all numbers b…
Figure 11
Figure 11. Figure 11: Pseudo-code for generating spiral and sinusoidal paths on Table Readout: For simplicity, we present a single variant of each pattern. By permuting the Direction-Change map, the presented variants can be modified to include other direction patterns. C.4. Grid Navigatio…
Figure 12
Figure 12. Figure 12: Details on Grid Navigation: Unicode characters used for specifying each object [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13 [PITH_FULL_IMAGE:figures/full_fig_p028_13.png]
Figure 14
Figure 14. Figure 14: Results on Consecutive Table Readout based on the number of unique samples N u SIMPLE: Our observations from Section 3 hold true even when different types of supervision are compared at the same value N u SIMPLE, instead of NSIMPLE [PITH_FULL_IMAGE:figures/full_fig_p…
Figure 15
Figure 15. Figure 15: Results on non S2H-generalizing tasks based on the total number of training data ( [PITH_FULL_IMAGE:figures/full_fig_p029_15.png]
Figure 16
Figure 16. Figure 16: Effect of text warm-up pretraining: We report the S2H generalization on image with/without text warm-up before Mix+ (Consecutive Table Readout) or Align-Mix+ (Table Readout and Visual Analogy). S2H generalization on text from Text supervision serves as a reference (in…
Figure 17
Figure 17. Figure 17: Analysis of gradients on Table Readout (additional plots for [PITH_FULL_IMAGE:figures/full_fig_p033_17.png]
Figure 18
Figure 18. Figure 18: Analysis of gradients on Table Readout (replacing gradient alignment score from Figures 9 and 17 with gradient cosine similarity): (Left) Average Gradient Norm on SIMPLE Image examples (Ex∈XSIMPLE ∥∇l(I;S)(x)∥2) vs. Gradient Cosine Similarity (Equation (7)) for differ…
Figure 19
Figure 19. Figure 19: Analysis of gradients on Consecutive Table Readout (replacing gradient alignment score from [PITH_FULL_IMAGE:figures/full_fig_p035_19.png]
Figure 20
Figure 20. Figure 20: Analysis of gradients on Table Readout (replacing gradient alignment score from Figures 9 and 17 with Adam update alignment score): (Left) Average Gradient Norm on SIMPLE Image examples (Ex∈XSIMPLE ∥∇l(I;S)(x)∥2) vs. Adam Update Alignment Score (Equation (8)) for diff…
Figure 21
Figure 21. Figure 21: Analysis of evaluation losses (repeating [PITH_FULL_IMAGE:figures/full_fig_p035_21.png]
Figure 22
Figure 22. Figure 22 [PITH_FULL_IMAGE:figures/full_fig_p037_22.png]
Figure 23
Figure 23. Figure 23: Ablation on the amount of data for the alignment phase of Align - Mix+ on Table Readout: Second phase (Mix+) uses 12 × 104 training data. We don’t observe a monotonic improvement in generaliza￾tion performance with increasing number of training samples in the first ph…
Figure 24
Figure 24. Figure 24: Ablation on the amount of data for the text warm-up phase on Visual Analogy: Second phase (Mix+) uses 12 × 104 training data. Using more data for the warm-up stage results in a stronger LLM backbone with better HARD-text performance (gray dashed line), but does not ne…
Figure 25
Figure 25. Figure 25: Ablation on lossy vs. lossless Visual Analogy: We measure the image S2H generalization of different types of supervision for two different versions of text representation for Visual Analogy. Models can perform better on Lossless Visual Analogy with Mix+. However, the …
Figure 26
Figure 26. Figure 26: Ablation on jointly training on all three non S2H-generalizing tasks: (Left) Average S2H Generalization on image; (Middle, Right) Comparison of Trained Jointly vs. Individually. Similar to training on each task individually, Mix+ and Image-via-Text+ outperform Image, …
Figure 27
Figure 27. Figure 27: Ablation on the number of repetitions of unique HARD examples, while maintaining the total amount of HARD training data, on Table Readout and Visual Analogy: Image S2H generalization degrades with more repetitions of HARD Text examples, with the effect on Mix+ being m…
Figure 28
Figure 28. Figure 28: Visualization of pixel attribute scores on Table Readout: (Top) Mix+; (Bottom) Align-Mix+. Models are trained with 24×104 training data. Pixel attribute scores are averaged across CoT tokens that belong to the first 5 pixels roughly in the 10th column (left), the next…
Figure 29
Figure 29. Figure 29: Visualization of pixel attribute scores on Visual Analogy: The model is trained with 12 × 104 training data of (TW) Align-Mix+. Pixel attribute scores are averaged across CoT tokens that belong to Example 1 (left), Example 2 (middle), and the query (right) respectivel…
Figure 30
Figure 30. Figure 30: Analysis of failure modes on Table Readout: (Left) Precision and Recall; (Right) Example of a common mistake. Models are trained on 24 × 104 examples of Text, Image and Mix+ supervision and evaluated on corresponding inputs from HARD. Models often hallucinate a “short…
Figure 31
Figure 31. Figure 31: Analysis of failure modes on Grid Navigation: (Left) Whether model generates a sequence of actions that leads to the destination; (Middle) Average fraction of objects collected; (Right) Average number of obstacles passed through. Models trained with Text and Image fai…
Figure 32
Figure 32. Figure 32: A SIMPLE example from Table Readout. 48 [PITH_FULL_IMAGE:figures/full_fig_p048_32.png]
Figure 33
Figure 33. Figure 33: A HARD example from Table Readout. 49 [PITH_FULL_IMAGE:figures/full_fig_p049_33.png]
Figure 34
Figure 34. Figure 34: A SIMPLE example from Grid Navigation. 50 [PITH_FULL_IMAGE:figures/full_fig_p050_34.png]
Figure 35
Figure 35. Figure 35: A HARD example from Grid Navigation. 51 [PITH_FULL_IMAGE:figures/full_fig_p051_35.png]
Figure 36
Figure 36. Figure 36: A SIMPLE example from Visual Analogy: The common relation is r = AND and the domains are d1 = d2 = dquery = shape quantity, and the combinations (d, r) are not in the held-out set S = {(line type, XOR), (line color, OR), (shape type, AND), (shape size, XOR), (shape co…
Figure 37
Figure 37. Figure 37: A HARD example from Visual Analogy: The common relation is r = AND and the domains are distinct: d1 = line color, d2 = shape position, dquery = line color, and the combinations (d, r) are in the held-out set S = {(line type, XOR), (line color, OR), (shape type, AND), …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-Language Models

    cs.AI 2025-08 unverdicted novelty 6.0 of 10

    SEAM measures VLM reasoning consistency across modalities using paired semantically equivalent textual and visual notations, and finds systematic vision-language imbalance.

Reference graph

Works this paper leans on

32 extracted references · 25 canonical work pages · cited by 1 Pith paper

  1. [1]

    shape type), the model first needs to correctly enumerate the attribute values (e.g

    given an attribute (e.g. shape type), the model first needs to correctly enumerate the attribute values (e.g. circle) for each image in the examples

    work page

  2. [2]

    knowing how to retrieve the current location (row, col index) and the destination location

    work page

  3. [3]

    backtracking

    knowing which action is preferred (the one that minimizes the distance towards destination) but the model never picked up on why we sometimes backtrack or sometimes take an action that is not the most preferred. At generation, it would ignore all obstacles and try to take the most preferred action. On the other hand, we observed that the model could still...

    work page 2019

  4. [4]

    • To reason about the query: the model needs to correctly enumerate the attribute values for each image in the query similarly

    finally, the model needs to identify the target relationr 1 =r 2 =r query from the examples. • To reason about the query: the model needs to correctly enumerate the attribute values for each image in the query similarly. • To reason about the options:

    work page

  5. [6]

    Singh, A., Natarajan, V ., Shah, M., Jiang, Y ., Chen, X., Batra, D., Parikh, D., and Rohrbach, M

    URL https://openreview.net/forum? id=Y2RW9EVwhT. Singh, A., Natarajan, V ., Shah, M., Jiang, Y ., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2019. Socher, R., Ganjoo, M., Manning, C. D., and Ng, A. Zero- shot learning through ...

    work page 2019

  6. [7]

    Sun, Z., Yu, L., Shen, Y ., Liu, W., Yang, Y ., Welleck, S., and Gan, C

    URL https://openreview.net/forum? id=bmbRCRiNDu. Sun, Z., Yu, L., Shen, Y ., Liu, W., Yang, Y ., Welleck, S., and Gan, C. Easy-to-hard generalization: Scalable alignment beyond human supervision. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems,

    work page

  7. [8]

    URL https://openreview.net/forum? id=qwgfh2fTtN. Tan, H. and Bansal, M. V okenization: Improving language understanding with contextualized, visual-grounded su- pervision. In Webber, B., Cohn, T., He, Y ., and Liu, Y . (eds.),Proceedings of the 2020 Conference on Empiri- cal Methods in Natural Language Processing (EMNLP), pp. 2066–2080, Online, November 2...

    arXiv 2020

  8. [9]

    Yue, X., Ni, Y ., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y ., et al

    URL https://openreview.net/forum? id=Jf5gplvglq. Yue, X., Ni, Y ., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y ., et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556–9567, ...

    arXiv 2024

Show all 32 references
  1. [10]

    single-hop

    proposed new benchmarks that aim to evaluate the spatial understanding and reasoning of VLM, most evaluation tasks are in the form of VQA questions that only require “single-hop” reasoning or relatively fewer reasoning steps. To create a controlled setting with well-definedSIM...

    2022

  2. [15]

    the model then needs to decide whether the values in all three images of that example are consistent with a logical relation (e.g.XOR)

  3. [16]

    after repeating the process for both in-context examples, the model summarizes the two relational patterns (d1, r1) and(d 2, r2)for the examples

  4. [18]

    (line type , XOR), the model needs to identify the correct values of the attribute domain d for each option image and the correct relationr

    assuming the query when combined with each option follows a relational pattern (domain d, relation r) (e.g. (line type , XOR), the model needs to identify the correct values of the attribute domain d for each option image and the correct relationr

  5. [19]

    46 Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs? Table 14

    the model also needs to reason whether the identified relationris the desired target relationr query. 46 Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs? Table 14. Analysis of failure modes onVisual Analogy:Models are trained on 12...

  6. [20]

    CoT We enumerate the relevant row indices, column indices, row names, column names, and their corresponding values. Row Index, Column Index, Row Name, Column Name, Value 1 3 RadiateTool January 2 2 3 OmnicronInstrument January 4 3 3 ShimmerApparatus January 5 4 3 XiTool Decemb...

  7. [21]

    Answer: 95 November December Figure 32.ASIMPLEexample fromTable Readout

    Prediction The total is 2 + 4 + 5 + 9 + 6 + 0 + 6 + 2 + 2 + 6 + 3 + 8 + 8 + 9 + 9 + 4 + 8 + 1 + 3 = 95. Answer: 95 November December Figure 32.ASIMPLEexample fromTable Readout. 48 Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs? Th...

  8. [22]

    CoT We enumerate the relevant row indices, column indices, row names, column names, and their corresponding values. Row Index, Column Index, Row Name, Column Name, Value 10 1 PsiDevice September 4 9 1 EmberGadget September 0 8 1 ApexWidget September 5 7 1 DeltaTool September 8...

  9. [23]

    Answer: 233 Figure 33.AHARDexample fromTable Readout

    Prediction The total is 4 + 0 + 5 + 8 + 8 + 9 + 2 + 3 + 7 + 1 + 4 + 1 + 7 + 4 + 5 + 0 + 6 + 2 + 9 + 3 + 1 + 4 + 2 + 4 + 4 + 2 + 7 + 4 + 9 + 1 + 0 + 9 + 1 + 8 + 4 + 0 + 6 + 3 + 7 + 5 + 1 + 0 + 8 + 4 + 2 + 2 + 1 + 6 + 9 + 2 + 3 + 7 + 0 + 5 + 3 + 9 + 1 + 1 = 233. Answer: 233 Figu...

  10. [24]

    The grid is filled up with objects, which you will be asked to recognize and collect, and obstacles, which you should avoid

    Prediction Answer: up up left up up left down right down right down down left left up left up left up left The image shows a 2 dimensional grid. The grid is filled up with objects, which you will be asked to recognize and collect, and obstacles, which you should avoid. Possibl...

  11. [25]

    Text Conversion up: d, f okay d, f: Collect flag left: d, e okay d, e: left: d, d okay d, d: left: d, c closed( square) down: e, d closed( square) up: c, d closed( square) right: d, e visited No available actions. retrace 1 step d, e: left: d, d visited down: e, e okay e, e: l...

  12. [26]

    The grid is filled up with objects, which you will be asked to recognize and collect, and obstacles, which you should avoid

    CoT i, c: right: i, d okay i, d: right: i, e okay i, e: Collect gear right: i, f okay i, f: right: i, g okay i, g: right: i, h okay i, h: Collect spark up: h, h okay h, h: up: g, h closed( cross) left: h, g okay h, g: up: g, g okay g, g: up: f, g okay f, g: up: e, g okay e, g:...

  13. [27]

    Prediction Answer: left up left down down down right right right right right right up left up up up up up up up left left down down left left down left right up right right right right up right up right right down down right

  14. [28]

    51 Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs? The image shows a a puzzle in a 3 by 3 grid followed by 4 options

    Text Conversion eda b c f g h i kj i ✚ ♛ ✖ ✚ ✖ ✖ ✚ ✖‗ ▲ ✖♖ ✖ ✖✖ ▲ ✖▲ ✚ ✖ ▲ ✖ ✖✚ ✖ ⚙ ✦ a b c d e f g h right: c, f okay c, f: right: c, g okay c, g: right: c, h closed( plus) up: b, g okay b, g: right: b, h okay b, h: right: b, i closed( cross) down: c, h closed( plus) up: a, h...

  15. [29]

    … position: Image 1: (1, 0), (0, 2) Image 2: (0, 2), (1, 1) Image 3: (0, 2) This suggests the AND relation

    Text Conversion We first analyze the examples by considering the applicable attributes and relations: Example 1: type: Image 1: circle, triangle Image 2: triangle, rectangle Image 3: circle No pattern. … position: Image 1: (1, 0), (0, 2) Image 2: (0, 2), (1, 1) Image 3: (0, 2)...

  16. [30]

    Prediction Example 1: (shape position, AND) Example 2: (shape position, AND) Option 1: (shape color, OR), not consistent Option 2: (shape color, XOR), not consistent Option 3: (shape type, XOR), not consistent Option 4: (shape position, AND), consistent Answer: 4 Figure 36. AS...

  17. [31]

    color: Image 1: 189, 135 Image 2: 189 Image 3: 189 This suggests the AND relation

    Text Conversion We first analyze the examples by considering the applicable attributes and relations: Example 1: type: Image 1: diamond lines, horizontal line Image 2: vertical line Image 3: falling diagonal line No pattern. color: Image 1: 189, 135 Image 2: 189 Image 3: 189 T...

  18. [32]

    Prediction Example 1: (line color, AND) Example 2: (shape color, AND) Option 1: (line color, XOR), not consistent Option 2: (line color, OR), not consistent Option 3: (line type, XOR), not consistent Option 4: (line color, AND), consistent Answer: 4 Figure 37. AHARDexample fro...

  19. [62]

    findings-naacl.62/

    URL https://aclanthology.org/2024. findings-naacl.62/. Chen, X., Fang, H., Lin, T.-Y ., Vedantam, R., Gupta, S., Doll´ar, P., and Zitnick, C. L. Microsoft coco captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325, 2015. Cherian, A., Peng, K.-C., Lohi...

    2024 arXiv

  20. [576]

    emnlp-main.576/

    URL https://aclanthology.org/2020. emnlp-main.576/. Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y ., Ecoffet, A., Joglekar, M., Leike, J., et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision.arXiv...

    2020 arXiv

  21. [2015]

    Convert,

    during our experiments. Brief definition of Adam:The Adam optimizer maintains two additional states, each representing the running average of the gradients and their squares during training. If mt and vt denote the two states, then the update rule at training step t 33 General...

  22. [2023]

    doi: 10.18653/v1/2023.acl-short.43

    Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-short.43. Nesterov, Y . Lectures on convex optimization.Springer Optimization and Its Applications, 137, 2018. Nguyen, C.-V . T., Le, T.-S., Mai, A.-T., and Le, D.-T. Ada2i: Enhancing modality balance for mul...

    2023

  23. [2024]

    Fan, W.-C., Chen, Y .-C., Liu, M., Yuan, L., and Sigal, L

    URL https://openreview.net/forum? id=GC8HkKeH8s. Fan, W.-C., Chen, Y .-C., Liu, M., Yuan, L., and Sigal, L. On pre-training of multimodal language models customized for chart understanding.arXiv preprint arXiv:2407.14506, 2024. Fan, Y ., Xu, W., Wang, H., Wang, J., and Guo, S....

    2024 arXiv

  24. [2025]

    Fu, C., Chen, P., Shen, Y ., Qin, Y ., Zhang, M., Lin, X., Qiu, Z., Lin, W., Yang, J., Zheng, X., Li, K., Sun, X., and Ji, R

    URL https://openreview.net/forum? id=2edigk8yoU. Fu, C., Chen, P., Shen, Y ., Qin, Y ., Zhang, M., Lin, X., Qiu, Z., Lin, W., Yang, J., Zheng, X., Li, K., Sun, X., and Ji, R. Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:23...

    2023 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.