Pith. sign in

REVIEW 9 cited by

ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2501.05452 v1 pith:MGYOWXK3 submitted 2025-01-09 cs.CV cs.CL

Xingyu Fu , Minqian Liu , Zhengyuan Yang , John Corring , Yijuan Lu , Jianwei Yang , Dan Roth , Dinei Florencio
show 1 more author
Cha Zhang
This is my paper
classification cs.CVcs.CL
keywords visualimagerefocustaskseditingllmsmultimodalstructured
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Structured image understanding, such as interpreting tables and charts, requires strategically refocusing across various structures and texts within an image, forming a reasoning sequence to arrive at the final answer. However, current multimodal large language models (LLMs) lack this multihop selective attention capability. In this work, we introduce ReFocus, a simple yet effective framework that equips multimodal LLMs with the ability to generate "visual thoughts" by performing visual editing on the input image through code, shifting and refining their visual focuses. Specifically, ReFocus enables multimodal LLMs to generate Python codes to call tools and modify the input image, sequentially drawing boxes, highlighting sections, and masking out areas, thereby enhancing the visual reasoning process. We experiment upon a wide range of structured image understanding tasks involving tables and charts. ReFocus largely improves performance on all tasks over GPT-4o without visual editing, yielding an average gain of 11.0% on table tasks and 6.8% on chart tasks. We present an in-depth analysis of the effects of different visual edits, and reasons why ReFocus can improve the performance without introducing additional information. Further, we collect a 14k training set using ReFocus, and prove that such visual chain-of-thought with intermediate information offers a better supervision than standard VQA data, reaching a 8.0% average gain over the same model trained with QA pairs and 2.6% over CoT.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Counterfactual present/removed teacher views attribute visually supported corrections and reconstruct student-anchored distillation targets that beat source-mixed multimodal OPD.

  2. FaithEyes: Towards Faithful Tool Use via Multi-Agent Process-Image Verification

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A self-judging multi-agent VLM injects process-image helpfulness verdicts into tool observations and scales tool rewards by the helpful-call ratio, improving accuracy and tool faithfulness.

  3. Fully Spiking Neural Networks with Target Awareness for Energy-Efficient UAV Tracking

    cs.CV 2026-03 conditional novelty 6.0 of 10

    A two-stage pure RL method with an information-gap global view and hierarchical grounding loss makes MLLMs truly rely on precise crops and sets SOTA on high-res VQA under tight token budgets.

  4. Learning Only with Images: Visual Reinforcement Learning with Reasoning, Rendering, and Visual Feedback

    cs.CV 2025-07 conditional novelty 6.0 of 10

    RRVF trains an image-to-code MLLM using reinforcement learning with a render-and-compare visual feedback loop, and it outperforms supervised fine-tuning on chart and web benchmarks.

  5. TableMoE: Neuro-Symbolic Routing for Structured Expert Reasoning in Multimodal Table Understanding

    cs.AI 2025-06 conditional novelty 6.0 of 10

    TableMoE is a multimodal table model whose role-aware router sends table tokens to HTML, JSON, and code experts and reports state-of-the-art results on its own WildStruct benchmarks and MMMU-Table.

  6. ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A verifiable RL proxy task that asks VLMs to locate a single injected hallucination in a 200-word caption improves visual perception and transfers to math and abstract reasoning benchmarks.

  7. MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering

    cs.CV 2026-04 unverdicted novelty 5.0 of 10

    MedLVR interleaves latent visual reasoning segments in autoregressive decoding and uses two-stage training to raise average medical VQA accuracy from 48.3% to 53.4% over a Qwen2.5-VL-7B backbone on OmniMedVQA and five...

  8. Empowering Nanoscale Connectivity through Molecular Communication: A Case Study of Virus Infection

    cs.NI 2025-08 unverdicted novelty 4.0 of 10

    A position paper proposing molecular communication as the link layer for epidemic-control bio-nano networks, with an ORF3a-based mutation identification simulation; the provided manuscript body does not match this abstract.

  9. Empowering Multimodal LLMs with External Tools: A Comprehensive Survey

    cs.CV 2025-08 unverdicted novelty 2.0 of 10

    A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.

Pith tools