Pith. sign in

REVIEW 2 major objections 5 minor 30 references

Training multimodal models on hierarchical scene graphs yields large gains in fine-grained perception and spatial reasoning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 16:12 UTC pith:LZNX5N5M

load-bearing objection Solid systems paper: hierarchical scene-graph data + node-articulated CoT + dual node-as-proxy GRPO rewards give large, ablated gains on fine-grained and spatial MLLM benchmarks. the 2 major comments →

arxiv 2607.05716 v3 pith:LZNX5N5M submitted 2026-07-07 cs.CV

Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models

Zhiwei Yang , Yuanchen Wu , Nan Zhang , Yucong Meng , Ke Yan , Shouhong Ding This is my paper
classification cs.CV
keywords scene graphmultimodal large language modelsfine-grained visual perceptionstructured visual reasoningnode-as-proxy rewardschain-of-thoughtspatial understandingGRPO
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Current multimodal large language models still treat scenes as collections of isolated objects and rely on heuristic crops or zooms, which makes target navigation inefficient and hurts fine-grained tasks. This paper shows that converting ordinary image–text pairs into hierarchical scene graphs—nodes holding entities, attributes, boxes and depth, edges holding spatial and semantic relations—supplies the missing structure. From those graphs the authors sample 120K node- and edge-centric reasoning traces, then run a two-stage post-training pipeline: supervised fine-tuning internalizes the graph-aligned chain-of-thought, and reinforcement learning with node-as-proxy rewards consolidates efficient, query-relevant traversal. The resulting models (SaGe-3B/7B) deliver consistent lifts across eight visual benchmarks, often surpassing larger open models and matching or beating proprietary systems on high-resolution and spatial tests.

Core claim

Explicit scene-graph representations, when used both to generate node-articulated chain-of-thought data and to define node-as-proxy rewards for reinforcement learning, equip multimodal models with fine-grained, relation-aware visual reasoning that isolated-object or pure zoom-in methods lack.

What carries the argument

Node-as-proxy graph rewards (node-grounded visual alignment plus node-relevance to the query) inside GRPO, together with the automated hierarchical scene-graph construction and the 120K sampled reasoning traces that train the model to follow those graphs.

Load-bearing premise

That the automatically constructed scene graphs are accurate and complete enough for the sampled traces and node rewards to teach genuine structured reasoning rather than merely imitate residual annotation noise.

What would settle it

Train an otherwise identical model on the same volume of data but with the graph structure ablated (flat captions or random crops only); if the large gains on VStarBench, HRBench and CVBench disappear, the claim that the graphs themselves are doing the work is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • High-resolution visual search becomes a guided walk over relational pathways instead of repeated heuristic zooms.
  • Explicit depth ranges on nodes give smaller models competitive 3D spatial reasoning without specialized 3D architectures.
  • Node-articulated intermediate steps supply verifiable evidence that can be audited or used for further self-correction.
  • The same data engine can be reapplied to new image corpora, turning any flat multimodal collection into structured training fuel.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Residual 5–7 % node/edge error rates imply that stronger foundation detectors would further amplify the observed gains.
  • The same node-proxy idea could regularize video or multi-view reasoning by treating temporal or cross-view links as additional edges.
  • Forcing every reasoning step to name a grounded node may reduce visual hallucinations even on tasks that do not explicitly require localization.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes Scene Graph Thinking (SaGe), a paradigm that equips MLLMs with structured visual reasoning via explicit hierarchical scene graphs. An automated engine converts flat image–text pairs into graphs with compositional nodes (attributes, bboxes, depth ranges from Depth-Anything + SAM) and multi-type edges built from node priors. From these graphs the authors sample 120K node-/edge-centric QA pairs and multi-hop traces with node-articulated CoT (<entity>, <bbox>, <depth>). Two-stage post-training follows: SFT cold-start on the 120K corpus, then GRPO with complementary node-as-proxy rewards (node-grounded visual–text alignment + node-relevance to query anchors) plus accuracy and format terms. On Qwen2.5-VL-3B/7B backbones, SaGe reports large gains on VStarBench (75.4→89.0 / 76.4→89.0), HRBench-4K/8K, CVBench-2D/3D, and modest lifts on MMStar, RefCOCO, ChartQA, often matching or exceeding larger open models and some proprietary systems. Ablations isolate SFT, GRPO, CoT tags, and each reward; a Gemini audit of 1k graphs reports 94.5% node / 92.9% edge precision; code is released.

Significance. If the empirical gains hold under independent re-implementation, the work supplies a practical, scalable recipe for injecting hierarchical relational structure into MLLMs without requiring online tool use or multi-turn search at inference. The combination of an automated hierarchical graph engine, node-articulated CoT, and node-as-proxy GRPO is a concrete advance over pure crop/zoom “think-with-image” methods. Strengths include consistent multi-benchmark lifts, teacher-beating results on several axes, complementary ablations (Tables 3–5), data-scale curves, and public code. Residual graph noise is acknowledged and partially quantified; the central claim remains empirical and falsifiable.

major comments (2)
  1. §3.2–3.3 and Appendix B.1: residual graph noise (94.5% node / 92.9% edge precision on a 1k Gemini audit; incomplete recall; attribute/relation error modes) is real. While two-round filtering and human spot-check (<1%) mitigate transfer, the paper should quantify how residual errors affect final student accuracy—e.g., by injecting controlled noise into a subset of the 120K traces or reporting performance stratified by graph-quality bins. Without this, the load-bearing assumption that node-articulated CoT and node-as-proxy rewards transfer robustly remains only partially stress-tested.
  2. §3.4 Eq. (5)–(7) and Appendix D: both node-grounded and node-relevance rewards rely on external LLM/MLLM judges (Qwen3-VL-30B / Qwen3-30B). The manuscript should report inter-judge agreement or a small human validation of the reward labels, and ideally an ablation that replaces the learned judges with cheaper heuristics, to show that the reported GRPO gains are not artifacts of judge–student co-training.
minor comments (5)
  1. Table 1 vs. Table 2: GPT-4o numbers appear inconsistent across tables (e.g., CVBench-2D overall); please verify and unify.
  2. Figure 1 and §1: the “heuristic search” caricature of prior work is slightly overstated; several cited methods already use multi-step visual search. Soften the contrast.
  3. §4.1 Implementation: list the exact KL coefficient β, clip ε, and rollout temperature used for GRPO so that the RL stage is fully reproducible from the main text.
  4. Appendix C data mixture: the 25K multi-hop + 20K caption split is useful; a short table summarizing exact counts per subtype would help readers re-create the mixture.
  5. Typos / notation: “SaGe” vs. “SaGe” capitalization is consistent, but “hieratically” (§3.2) should be “hierarchically”; “proxy-as-node” in Fig. 4 caption should match “node-as-proxy” used elsewhere.

Circularity Check

0 steps flagged

No significant circularity: empirical benchmark gains from external-graph data and node-proxy rewards are self-contained and externally falsifiable.

full rationale

The paper's load-bearing claims are purely empirical: after constructing hierarchical scene graphs via an automated hybrid pipeline (Qwen2.5-VL-72B detection + compositional mining + Depth-Anything + SAM + LLM edge priors), sampling 120K node-articulated CoT traces, performing SFT cold-start, then GRPO with node-grounded and node-relevance rewards (Eqs. 5–8), SaGe-3B/7B produce large lifts on eight independent external benchmarks (VStarBench 75.4→89.0, HRBench, CVBench-2D/3D, MMStar, RefCOCO, ChartQA; Tables 1–2). Graph quality is audited externally (Gemini-3-Pro: 94.5% node / 92.9% edge precision on 1k graphs) and data filtered (GME + MLLM two-round verification, <1% human failure). Ablations (Tables 3–5) isolate additive contributions of CoT tags and dual NPR rewards; SaGe-7B exceeds its 72B teacher on multiple axes (Table 6). No equation, definition, or fitted parameter reduces any reported accuracy to an input by construction; no uniqueness theorem or ansatz is imported via self-citation as a forcing premise; no known empirical pattern is merely renamed. LLM-as-judge rewards are training signals only; final metrics use held-out ground truth. The derivation chain is therefore non-circular and externally testable.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 3 invented entities

The central empirical claim rests on standard ML training assumptions, the fidelity of the automated graph construction pipeline, and the design of two node-level rewards. No new physical constants or free parameters are fitted to the evaluation benchmarks; hyper-parameters are ordinary training knobs. The main invented constructs are the hierarchical scene-graph representation used as training prior and the node-as-proxy reward pair.

free parameters (4)
  • SFT learning rate / batch size / epochs
    1e-5, batch 256, 1 epoch on 120K; ordinary but chosen by authors and affect final numbers.
  • RL learning rate / batch / rollouts / KL coefficient
    1e-6, batch 128, 8 rollouts, β for KL; standard GRPO knobs that control exploration vs. reference.
  • Node-relevance reward tiers (1.0 / 0.8 / 0.5 / 0.2 / 0.0)
    Hand-designed discrete scores for semantic match levels; directly shape the policy.
  • Data mixture sizes (25K attr, 30K spatial, 25K multi-hop, 20K caption, 10K ground, 10K count)
    Composition of the 120K corpus is chosen by authors and influences which capabilities are internalized.
axioms (4)
  • domain assumption Hierarchical scene graphs with attribute/bbox/depth nodes and multi-type edges are a sufficient structured prior for fine-grained visual reasoning.
    Stated in §1–3.1 as the core modeling choice; if false, the sampled CoT and rewards lose their grounding.
  • ad hoc to paper Node-level visual-textual consistency and query relevance are adequate proxies for valid graph trajectories (edge-level supervision is too ambiguous).
    Explicit design decision in §3.4; the paper argues edges are multi-valid, so nodes become the rewardable units.
  • domain assumption GRPO with accuracy + format + node-proxy rewards improves structured exploration without collapse.
    Standard RL assumption imported from DeepSeek-R1 / recent visual RL papers; used in §3.5.
  • domain assumption Two-round automatic verification (GME similarity + proprietary MLLM re-check) yields <1% residual error.
    Claimed after human spot-check; underpins data quality for both SFT and RL.
invented entities (3)
  • Node-as-proxy graph rewards (node-grounded + node-relevance) no independent evidence
    purpose: Provide dense, verifiable supervision for graph-aligned reasoning traces without requiring unique edge sequences.
    Core technical contribution of the RL stage; defined in §3.4 and ablated in Table 5.
  • Node-articulated CoT format (<entity><bbox><depth>) no independent evidence
    purpose: Force the model to externalize hierarchical evidence during generation.
    Used throughout data sampling and training; ablated in Table 4.
  • Automated hierarchical scene-graph data engine (compositional mining + depth-aware nodes + prior-based edges) no independent evidence
    purpose: Convert flat image-text corpora into 120K structured training traces.
    Described in §3.2–3.3; quality audited in appendix.

pith-pipeline@v1.1.0-grok45 · 31829 in / 3161 out tokens · 34992 ms · 2026-07-14T16:12:01.345835+00:00 · methodology

0 comments
read the original abstract

Multimodal Large Language Models (MLLMs) have demonstrated strong perception and reasoning capabilities. However, most existing models focus on isolated objects and neglect structured relationships for efficient target navigation, limiting their performance on visually intensive tasks. To address this challenge, we introduce Scene Graph Thinking (SaGe), a novel paradigm that enables fine-grained and structured visual reasoning through explicit scene-graph representations. Specifically, we first introduce an automated data engine that converts flat image-text corpora into structured scene graphs, where hierarchical entities constitute the nodes and diverse visual relations define the edges. Building upon this, we construct 120K high-quality training data by sampling reasoning traces from scene graphs. Then, two-stage graph-aligned post-training paradigms are introduced, where supervised fine-tuning internalizes MLLMs with structured reasoning, and subsequent reinforcement fine-tuning proposes node-as-proxy graph rewards to consolidate efficient graph exploration. With curated data and graph-aligned training, our approach achieves significant improvements across eight multimodal benchmarks, demonstrating strong effectiveness on fine-grained perception and reasoning tasks. Code is available at https://github.com/zwyang6/SaGe.

Figures

Figures reproduced from arXiv: 2607.05716 by Ke Yan, Nan Zhang, Shouhong Ding, Yuanchen Wu, Yucong Meng, Zhiwei Yang.

Figure 1
Figure 1. Figure 1: Our motivation. (a) Previous methods overlook struc￾tured relationships within the scene, leading to inefficient target navigation and suboptimal performance. (b) The proposed SaGe internalizes structured scene graphs to enable fine-grained visual reasoning, achieving more efficient and reliable performance. tence in multimodal perception (Lai et al., 2024; Liu et al., 2025) understanding (Wang et al., 202… view at source ↗
Figure 2
Figure 2. Figure 2: Scene Graph Construction. We convert each flat image–text pair into a hierarchical scene graph. Given an image Ii, (a) we first perform Compositional Node Mining, modeling all objects and their sub-components as nodes, each equipped with bounding boxes and attribute annotations. (b) We then build Depth-aware Node Modeling by assigning depth cues to each node. (c) Finally, edges are constructed based on nod… view at source ↗
Figure 3
Figure 3. Figure 3: Training Data Sampling and Verification. We adopt (a) node- and edge-centric query sampling with a (b) two-round data verification strategy, (c) yielding 120K CoT-augmented training samples. we use it to prompt SAM for precise segmentation. After boundary erosion, we extract depth values within the mask to estimate a depth range for each entity, providing each node with reliable depth cues for 3D-aware rea… view at source ↗
Figure 4
Figure 4. Figure 4: Two-stage Training Pipeline. We first adopt the (a) supervised fine-tuning stage to internalize the MLLMs with graph-aligned reasoning. Then (b) a reinforcement learning stage is conducted with proxy-as-node graph rewards, consolidating the structured reasoning. GRPO-based reinforcement learning. We contend that defin￾ing a verifiable edge-level supervision signal is inherently ambiguous, as a single pair … view at source ↗
Figure 5
Figure 5. Figure 5: Case study on various QA scenarios. Compared to baseline, our method generates correct answers with node-articulated CoT [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Data Scaling and Composition Analysis. Base: ground￾ing and counting. Cap.: caption. Mult.: multi-hop reasoning data. level supervision without trace-level guidance may lead to reward hacking, where the model attends to visually salient but target-irrelevant nodes. When combined, NRR and NGR exhibit strong complementarity: NRR guides which nodes to traverse, while NGR ensures the correctness of the visited… view at source ↗
Figure 7
Figure 7. Figure 7: Node Error Breakdown. (a) Distribution of node errors. (b) Number distribution of different error types. The nodes constructed in our scene graph achieve an accuracy of 94.5%, demonstrating the high quality of our scene graph. 55.6% （1273） 14.0% （320） 15.2% （347） 15.3% （350） Wrong direction Wrong desciption Wrong relation type Invalid relation 1273 350 347 320 0 200 400 600 800 1000 1200 1400 Edge Error Br… view at source ↗
Figure 8
Figure 8. Figure 8: Edge Error Breakdown. (a) Distribution of edeg errors. (b) Number distribution of different error types. The edges in our constructed scene graph attain an accuracy of 92.9%, which validates the reliability and quality of the scene graph construction. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Case study on various QA scenarios. Our method demonstrates strong spatial understanding and fine-grained perception through node-articulated CoT, highlighting the effectiveness of our scene-graph–based structured reasoning. B.2. More Visualized QA Cases As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Illustration of attribute-centric samples. We generate 25K attribute-centric samples, comprising 10K color-related, 5K state-related, 5K shape-related, and 5K material-related instances. Each sample is accompanied by a node-articulated CoT, providing explicit evidence to support structured reasoning. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Illustration of spatial reasoning, grounding, and counting samples. We include 30K spatial reasoning samples, covering both 2D and 3D spatial reasoning tasks. The traceable evidence within the <bbox> and <depth> tags provides spatial clues to support reasoning. To further enhance data diversity, we additionally generate 10K grounding samples and 10K counting samples. 17 [PITH_FULL_IMAGE:figures/full_fig_… view at source ↗
Figure 12
Figure 12. Figure 12: Illustration of multi-hop navigation and local caption samples. We generate 25K samples with multi-hop reasoning traces, together with 10K local captions. The local caption data focus on detailed descriptions of specific regions, such as a bounding box or half of the image. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_12.png] view at source ↗
Figure 13
Figure 13. Figure 13 [PITH_FULL_IMAGE:figures/full_fig_p019_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Multi-criteria scoring scheme for captioning. We employ a multi-criteria scoring scheme that emphasizes node coverage and edge accuracy, retaining only high-consistency caption samples. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Judge prompt of the node-grounded reward. More detailed examples used to instruct the MLLM are omitted for brevity. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Judge prompt of the node-relevance reward. More detailed examples used to instruct the LLM are omitted for brevity. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_16.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 21 linked inside Pith

  1. [1]

    K., Bai, Y ., Baker, B., Bao, H., et al

    Agarwal, S., Ahmad, L., Ai, J., Altman, S., Applebaum, A., Arbus, E., Arora, R. K., Bai, Y ., Baker, B., Bao, H., et al. gpt-oss-120b & gpt-oss-20b model card.arXiv preprint arXiv:2508.10925,

    Pith/arXiv arXiv

  2. [2]

    Qwen3-vl technical report

    Bai, S., Cai, Y ., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., Ge, W., Guo, Z., Huang, Q., Huang, J., Huang, F., Hui, B., Jiang, S., Li, Z., Li, M., Li, M., Li, K., Lin, Z., Lin, J., Liu, X., Liu, J., Liu, C., Liu, Y ., Liu, D., Liu, S., Lu, D., Luo, R., Lv, C., Men, R., Meng, L., Ren, X., Ren, X., Song, S., Sun, Y ., Tan...

    Pith/arXiv arXiv

  3. [3]

    Unifying vision-and- language tasks via text generation

    Cho, J., Lei, J., Tan, H., and Bansal, M. Unifying vision-and- language tasks via text generation. InInternational Con- ference on Machine Learning, pp. 1931–1942. PMLR,

    1931

  4. [4]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261,

    Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261,

    Pith/arXiv arXiv

  5. [5]

    J., Guan, X., and Wang, X

    Fan, Y ., He, X., Yang, D., Zheng, K., Kuo, C.-C., Zheng, Y ., Narayanaraju, S. J., Guan, X., and Wang, X. E. Grit: Teaching mllms to think with images.arXiv preprint arXiv:2505.15879,

    Pith/arXiv arXiv

  6. [6]

    com/deepmind-media/Model-Cards/ Gemini-3-Pro-Model-Card.pdf

    URL https://storage.googleapis. com/deepmind-media/Model-Cards/ Gemini-3-Pro-Model-Card.pdf. Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Pith/arXiv arXiv

  7. [7]

    Deepeyesv2: Toward agentic multimodal model.arXiv preprint arXiv:2511.05271,

    Hong, J., Zhao, C., Zhu, C., Lu, W., Xu, G., and Yu, X. Deepeyesv2: Toward agentic multimodal model.arXiv preprint arXiv:2511.05271,

    Pith/arXiv arXiv

  8. [8]

    Vision-r1: Incentivizing reasoning capability in multimodal large language models.arXiv preprint arXiv:2503.06749,

    Huang, W., Jia, B., Zhai, Z., Cao, S., Ye, Z., Zhao, F., Xu, Z., Hu, Y ., and Lin, S. Vision-r1: Incentivizing reasoning capability in multimodal large language models.arXiv preprint arXiv:2503.06749,

    Pith/arXiv arXiv

  9. [9]

    P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al

    Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276,

    Pith/arXiv arXiv

  10. [10]

    Openai o1 system card.arXiv preprint arXiv:2412.16720,

    Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Car- ney, A., et al. Openai o1 system card.arXiv preprint arXiv:2412.16720,

    Pith/arXiv arXiv

  11. [11]

    Referitgame: Referring to objects in photographs of natu- ral scenes

    Kazemzadeh, S., Ordonez, V ., Matten, M., and Berg, T. Referitgame: Referring to objects in photographs of natu- ral scenes. InProceedings of the 2014 conference on em- pirical methods in natural language processing (EMNLP), pp. 787–798,

    2014

  12. [12]

    B., Liu, O., Guo, P., Neiswanger, W., Huang, F., et al

    Li, A., Wang, C., Fu, D., Yue, K., Cai, Z., Zhu, W. B., Liu, O., Guo, P., Neiswanger, W., Huang, F., et al. Zebra- cot: A dataset for interleaved vision language reasoning. arXiv preprint arXiv:2507.16746, 2025a. Li, B., Zhang, Y ., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y ., Liu, Z., et al. Llava- onevision: Easy visual task tra...

    arXiv

  13. [13]

    Latent visual reasoning.arXiv preprint arXiv:2509.24251, 2025b

    Li, B., Sun, X., Liu, J., Wang, Z., Wu, J., Yu, X., Chen, H., Barsoum, E., Chen, M., and Liu, Z. Latent visual reasoning.arXiv preprint arXiv:2509.24251, 2025b. Li, G., Xu, J., Zhao, Y ., and Peng, Y . Dyfo: A training- free dynamic focus visual search for enhancing lmms in fine-grained visual understanding. InProceedings of the Computer Vision and Patter...

    Pith/arXiv arXiv

  14. [14]

    Lin, T.-Y ., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Doll´ar, P., and Zitnick, C. L. Microsoft coco: Common objects in context. InComputer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740–

    2014

  15. [15]

    Visual-rft: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785,

    Liu, Z., Sun, Z., Zang, Y ., Dong, X., Cao, Y ., Duan, H., Lin, D., and Wang, J. Visual-rft: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785,

    Pith/arXiv arXiv

  16. [16]

    Omnicaptioner: One cap- tioner to rule them all.arXiv preprint arXiv:2504.07089,

    Lu, Y ., Yuan, J., Li, Z., Zhao, S., Qin, Q., Li, X., Zhuo, L., Wen, L., Liu, D., Cao, Y ., et al. Omnicaptioner: One cap- tioner to rule them all.arXiv preprint arXiv:2504.07089,

    Pith/arXiv arXiv

  17. [17]

    L., Tan, J

    Masry, A., Do, X. L., Tan, J. Q., Joty, S., and Hoque, E. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. InFindings of the association for computational linguistics: ACL 2022, pp. 2263–2279,

    2022

  18. [18]

    Vlm-r1: A stable and generalizable r1-style large vision-language model

    Shen, H., Liu, P., Li, J., Fang, C., Ma, Y ., Liao, J., Shen, Q., Zhang, Z., Zhao, K., Zhang, Q., et al. Vlm-r1: A stable and generalizable r1-style large vision-language model. arXiv preprint arXiv:2504.07615,

    Pith/arXiv arXiv

  19. [19]

    Thinking with images for multimodal reasoning: Foundations, methods, and future frontiers.arXiv preprint arXiv:2506.23918,

    Su, Z., Xia, P., Guo, H., Liu, Z., Ma, Y ., Qu, X., Liu, J., Li, Y ., Zeng, K., Yang, Z., et al. Thinking with images for multimodal reasoning: Foundations, methods, and future frontiers.arXiv preprint arXiv:2506.23918,

    Pith/arXiv arXiv

  20. [20]

    Vl-rethinker: Incentivizing self-reflection of vision- language models with reinforcement learning.arXiv preprint arXiv:2504.08837, 2025a

    Wang, H., Qu, C., Huang, Z., Chu, W., Lin, F., and Chen, W. Vl-rethinker: Incentivizing self-reflection of vision- language models with reinforcement learning.arXiv preprint arXiv:2504.08837, 2025a. Wang, H., Su, A., Ren, W., Lin, F., and Chen, W. Pixel reasoner: Incentivizing pixel-space reasoning with curiosity-driven reinforcement learning.arXiv prepri...

    Pith/arXiv arXiv

  21. [21]

    Leveraging large language and vision models for knowledge extrac- tion from large-scale image–text colonoscopy records

    Wang, S., Zhu, Y ., Yang, Z., Luo, X., Zhang, Y ., Fu, P., Wang, H., Wang, M., Song, Z., Li, Q., et al. Leveraging large language and vision models for knowledge extrac- tion from large-scale image–text colonoscopy records. Nature Biomedical Engineering, pp. 1–12, 2025d. Wang, T., Zhang, J., Fei, J., Zheng, H., Tang, Y ., Li, Z., Gao, M., and Zhao, S. Cap...

    Pith/arXiv arXiv

  22. [22]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025a

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025a. Yang, L., Kang, B., Huang, Z., Zhao, Z., Xu, X., Feng, J., and Zhao, H. Depth anything v2.Advances in Neural Information Processing Systems, 37:21875–21911,

    Pith/arXiv arXiv

  23. [23]

    Look-back: Implicit visual re-focusing in mllm reasoning

    Yang, S., Niu, Y ., Liu, Y ., Ye, Y ., Lin, B., and Yuan, L. Look-back: Implicit visual re-focusing in mllm reasoning. arXiv preprint arXiv:2507.03019, 2025b. Yang, Z., Meng, Y ., Fu, K., Tang, F., Wang, S., and Song, Z. Exploring clip’s dense knowledge for weakly supervised semantic segmentation. InProceedings of the IEEE/CVF Conference on Computer Visio...

    Pith/arXiv arXiv

  24. [24]

    Mllms know where to look: Training-free perception of small visual details with multimodal llms.arXiv preprint arXiv:2502.17422, 2025a

    Zhang, J., Khayatkhoei, M., Chhikara, P., and Ilievski, F. Mllms know where to look: Training-free perception of small visual details with multimodal llms.arXiv preprint arXiv:2502.17422, 2025a. Zhang, X., Zhang, Y ., Xie, W., Li, M., Dai, Z., Long, D., Xie, P., Zhang, M., Li, W., and Zhang, M. Gme: Improving universal multimodal retrieval by multimodal l...

    Pith/arXiv arXiv

  25. [25]

    Thyme: Think beyond images.arXiv preprint arXiv:2508.11630, 2025b

    Zhang, Y .-F., Lu, X., Yin, S., Fu, C., Chen, W., Hu, X., Wen, B., Jiang, K., Liu, C., Zhang, T., et al. Thyme: Think beyond images.arXiv preprint arXiv:2508.11630, 2025b. Zheng, Z., Yang, M., Hong, J., Zhao, C., Xu, G., Yang, L., Shen, C., and Yu, X. Deepeyes: Incentivizing” thinking with images” via reinforcement learning.arXiv preprint arXiv:2505.14362,

    Pith/arXiv arXiv

  26. [26]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025a

    Zhu, J., Wang, W., Chen, Z., Liu, Z., Ye, S., Gu, L., Tian, H., Duan, Y ., Su, W., Shao, J., et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025a. Zhu, L., Ji, D., Chen, T., Xu, P., Ye, J., and Liu, J. Ibd: Alleviating hallucinations in large vision-language mod- els v...

    Pith/arXiv arXiv 2023

  27. [27]

    In the RL post-training stage, we use a total of 26K samples, including 2K grounding samples from (Lin et al., 2014), 4K chart-related samples from (Masry et al., 2022), and 20K samples from V* following (Zheng et al., 2025). We adopt Qwen3- VL-30B-A3B-Instruct (Bai et al., 2025a) as the judge for node-grounded rewards, while Qwen3-30B-A3B-Instruct (Yang ...

    2014

  28. [28]

    More Results and Discussions B.1

    B. More Results and Discussions B.1. Quality of Scene Graphs To assess scene graph quality, we evaluate 1,000 sampled raw graphs using Gemini-3-Pro (Google DeepMind, 2025), measuring the correctness of 22029 nodes and 26283 edges. A node is considered correct if its bounding box region matches its entity tag and attributes. It reports that the nodes and e...

    2025

  29. [29]

    bike on left\

    is lower on image than the sconce’s y‑ range (67‑ 95), indicating that the lamp appears below the sconce.</think><answer>Below</answer>. Output: Below. 2D Understanding: CVBench-3D Question: What is the color of the cyclist's box? SaGe (Ours): Thinking: <think>I first observe a <entity>cyclist riding a bike</entity><bbox>[[338,752,381,819]]</bbox><depth>[...

    2020

  30. [30]

    Together, they effectively promote graph-aligned visual reasoning

    It instructs the LLM judge to first determine the number of anchored nodes and then evaluate the relevance of the visited nodes with respect to them, encouraging the reasoning process to efficiently navigate toward target nodes.When combined, the node-relevance reward (NRR) and node-grounded reward (NGR) exhibit strong complementarity: NRR guides which no...

    1949