Pith. sign in

REVIEW 3 major objections 5 minor 62 references

A two-stage pure RL method forces MLLMs to actually use high-resolution crops instead of ignoring them.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 16:54 UTC pith:J67H3ZBE

load-bearing objection The arXiv ID points to an SNN UAV tracker, but the full manuscript is the MLLM cropping paper; that paper’s real contribution is a clean diagnosis of crop under-use plus a workable two-stage pure-RL fix that delivers SOTA under tight token budgets. the 3 major comments →

arxiv 2603.27493 v2 pith:J67H3ZBE submitted 2026-03-29 cs.CV

Fully Spiking Neural Networks with Target Awareness for Energy-Efficient UAV Tracking

Pengzhi Zhong , Jiwei Mo , Dan Zeng , Feixiang He , Shuiwang Li This is my paper
classification cs.CV
keywords multimodal large language modelsreinforcement learningvisual croppinginformation gaphigh-resolution VQAgrounding rewardagentic workflows
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Existing agentic multimodal large language models that crop image regions often answer from the full image and treat the crop as a formality. This paper shows that creating an explicit information gap—feeding a carefully downsampled global view while still cropping from the original high-resolution image—trains the model to rely on the crop content. A second stage then uses a small set of hierarchical bounding-box annotations plus an IoU-plus-distance grounding reward to make those crops more precise and compact. The resulting models reach state-of-the-art accuracy on high-resolution visual-question-answering benchmarks under both tight and generous visual-token budgets, and remain faster because they process fewer tokens overall.

Core claim

When the global image is deliberately degraded just enough that the model’s own answers diverge from its high-resolution answers, and crops are still taken from the original full-resolution source, pure reinforcement learning produces agents that genuinely attend to and depend on the cropped regions rather than treating the crop tool as optional confirmation.

What carries the argument

The Information Gap mechanism (offline resolution selection that stops at the first downsampling level where the model’s answer changes) together with the later hierarchical grounding reward that combines conditional IoU and L1 corner distance.

Load-bearing premise

The offline rule that keeps downsampling the global image until the model’s own answer flips will reliably teach a lasting habit of using crop content, rather than an artifact of the particular training images or the selection heuristic itself.

What would settle it

Retrain the identical two-stage pipeline but always use the original high-resolution global image (no information gap) and measure whether crop-dependence (accuracy drop when crops are replaced by noise, accuracy gain when replaced by ground-truth boxes) and the reported SOTA gains disappear.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Agentic MLLMs can be trained without expensive teacher trajectories and still outperform hybrid SFT+RL baselines on fine-grained high-resolution VQA.
  • Under a 1 024-token global budget the method already beats prior agents that are allowed 16 384 tokens, showing that crop reliance yields better token efficiency.
  • A few hundred hierarchical box annotations are enough to shrink crop area and token count while further lifting accuracy.
  • The same information-gap idea can be applied at test time by simply downsampling the global view, giving an immediate accuracy boost even for already-trained agents.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same resolution-selection trick could be used as a diagnostic probe for any tool-using MLLM: if replacing the tool output with noise barely hurts accuracy, the model is not really using the tool.
  • Hierarchical boxes may transfer to other open-ended visual tools (segmentation, depth) where a single tight box is too brittle.
  • Because Stage-I already produces usable crops, synthetic box data generated by pasting annotated low-resolution objects onto high-resolution backgrounds could remove the remaining manual annotation bottleneck.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper identifies that existing agentic MLLMs (SFT+RL or pure RL) for high-resolution VQA often invoke cropping tools perfunctorily, relying primarily on the global image rather than crop content (validated via GT-crop vs. random-noise replacements in Table 1 under both 16 384- and 1 024-token budgets). It proposes a two-stage pure-RL framework without trajectory supervision: Stage I creates an Information Gap by offline iterative downsampling of the global image until the model’s answer diverges from its high-resolution answer, while still extracting crops from the original high-resolution source, so that accuracy reward forces crop utilization; Stage II adds hierarchical nested bbox annotations on 256 VisualProbe samples plus a composite grounding reward (conditional IoU only when coverage > 0.9, plus L1 corner distance). Trained with GRPO on Qwen2.5-VL-7B, the method reports SOTA accuracy on HR-Bench 8K/4K and V* under both token budgets, improved crop-dependence deltas (Table 5), higher IoU, and lower inference time when using only 1 024 global tokens.

Significance. If the central claim holds—that the offline answer-divergence rule induces a transferable test-time habit of attending to high-resolution crop content rather than an artifact of the selection heuristic or training mix—the work supplies a practical, teacher-free pure-RL recipe that simultaneously raises accuracy and reduces visual-token cost for fine-grained agentic VQA. Strengths include clean controlled replacements (Tables 1, 5, 7), systematic ablations isolating Stage I, Stage II, data source, L1 term and IoU threshold (Tables 3–4, 8–10), public benchmarks, released code, and explicit efficiency comparison (Table 6). These elements make the contribution falsifiable and reusable for energy- or latency-constrained multimodal agents.

major comments (3)
  1. [§4.1 Offline Resolution Selection] §4.1 Offline Resolution Selection: the stopping criterion is defined by answer divergence of the very model being trained. This creates a risk that the selected resolutions are tuned to that model’s particular failure modes rather than inducing a general crop-attention habit. A load-bearing control is missing: re-run Stage I with resolutions chosen by a frozen independent model (or by fixed hard/random schedules already partially ablated in Table 9) and measure whether the GT/noise deltas of Table 5 and the SOTA numbers of Table 2 still appear at test time.
  2. [§5.1 Training Datasets; Table 4] §5.1 and Table 4: Stage-II gains rest on only 256 manually annotated hierarchical boxes from VisualProbe. The paper shows that reusing Stage-I data yields no gain, yet provides no scaling curve or cross-validation on annotation quantity/quality. Because the hierarchical IoU+L1 reward is the sole precision signal, it remains unclear whether the reported IoU lifts (Table 3) and final SOTA would survive a larger or differently sourced labeled set; this directly underpins the “precise cropping” half of the central claim.
  3. [Tables 1b, 5] Tables 1b and 5: even after Stage I the accuracy delta under GT versus noise replacement remains modest (≈10–18 points). The paper interprets this as successful crop dependence, yet the residual performance under pure noise (and the continued utility of the global image under the 16 384-token regime) suggests that the Information Gap only partially overrides the global-image shortcut. A quantitative analysis of how much of the final answer still depends on global tokens after training would strengthen or qualify the “genuine reliance” claim.
minor comments (5)
  1. [Tables 1–6, §5] Throughout: the V* benchmark is rendered as “� �” or “V Star”; replace with consistent “V*” and cite the original paper properly.
  2. [§4 Eq. (1)] Eq. (1) and surrounding text contain garbled symbols (�� �� �����); clean the action formulation so the crop coordinate and image variables are unambiguous.
  3. [Figure 3; §4.2] Figure 3 caption and §4.2: the hierarchical bbox notation (B1 \subset o Bn) is never formally defined; a short recursive definition would help.
  4. [Table 2] Table 2 footnotes mix evaluation protocols (temperature averaging, option shuffling); state a single consistent protocol for all compared methods.
  5. [§6 Efficiency Analysis] §6 and Table 6: report wall-clock time on identical hardware and batch size; the current numbers mix different token budgets without error bars.

Circularity Check

0 steps flagged

No load-bearing circularity; empirical RL training method with external-benchmark evaluation and controlled ablations.

full rationale

This is a pure empirical ML paper proposing a two-stage RL procedure (offline resolution selection to create an information gap, plus a small set of hierarchical bbox annotations for a grounding reward). The central claims are SOTA accuracy numbers and increased crop-dependence on held-out public benchmarks (HR-Bench 8K/4K, V*), measured by accuracy and by controlled GT-crop / random-noise replacements. The offline selection rule uses the same model family to choose training resolutions, which is a mild self-reference in data preparation, but it does not algebraically force the reported leaderboard metrics or the ablation deltas; those are obtained by training and then evaluating against external ground-truth answers and boxes. Rewards (accuracy, format, tool-use bonus, conditional IoU + L1) are ordinary RL signals, not tautological rewrites of the evaluation metrics. There are no uniqueness theorems, no fitted parameters renamed as predictions, no self-citation chains that substitute for evidence, and no equations that reduce by construction to their inputs. The derivation chain is therefore self-contained against external benchmarks; the only residual is the ordinary (non-circular) risk that the selection heuristic may not generalize, already noted by the reader.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

Load-bearing content is mostly empirical training design rather than formal derivation. The claim rests on standard RL/MLLM machinery plus a few paper-specific constructions (information-gap data prep, hierarchical boxes, composite grounding reward) and several hand-chosen hyperparameters. No new physical entities; invented entities are methodological mechanisms whose only evidence is the paper’s own ablations.

free parameters (5)
  • overlap threshold τ for conditional IoU reward = 0.9
    Set to 0.9 so IoU reward is granted only when the smallest GT box is almost fully covered; directly shapes Stage-II cropping behavior.
  • reward component weights (acc / format / tool-call) = 0.8 / 0.2 / 1.2
    Supplementary hyperparameters set accuracy 0.8, format 0.2, tool-call bonus 1.2; these scale the GRPO advantage signal in Stage-I.
  • Stage-I and Stage-II learning rates = 1e-6, 5e-7
    Chosen as 1e-6 then 5e-7; control optimization of the 7B policy and thus final accuracy/IoU.
  • offline resolution-selection stopping rule = first divergent downsampled resolution
    Downsample until the model’s answer diverges from the high-res answer; this discrete selection defines the entire Stage-I training distribution.
  • Stage-II labeled set size and source = 256 annotated samples
    256 VisualProbe instances with manual hierarchical boxes; small hand-curated set that drives the precision gains attributed to Stage-II.
axioms (5)
  • domain assumption GRPO-style group-relative policy optimization with the stated clipped objective is a valid way to optimize multi-component trajectory rewards for MLLM tool use.
    Invoked in §4.1 Eq. (3) without re-deriving RL convergence properties.
  • ad hoc to paper If the global image is informative enough to localize but not to answer, maximizing answer accuracy will force the policy to extract answer-critical evidence from the high-resolution crop.
    Core justification of the information-gap mechanism in §4.1; supported by ablations but not independently proven.
  • domain assumption Hierarchical nested ground-truth boxes (minimal necessary region through broader context) are valid supervision targets for a single crop action.
    §4.2 Hierarchical Bbox Annotation, motivated by compositional ROI structure citations.
  • ad hoc to paper Accuracy, format, and tool-use bonuses plus IoU/L1 grounding are sufficient proxies for desired agent behavior (no trajectory supervision required).
    Reward design in §4.1–4.2; alternative process supervisions are not compared exhaustively.
  • domain assumption Standard high-res VQA benchmarks (HR-Bench 8K/4K, V*) and accuracy/IoU metrics measure the intended fine-grained perception capability.
    Evaluation protocol §5.1–5.2.
invented entities (2)
  • Information Gap Mechanism (offline resolution selection + asymmetric crop source) no independent evidence
    purpose: Force the policy to treat the crop as indispensable by training on a deliberately under-informative global view while cropping from full resolution.
    Central Stage-I construction; evidence is internal ablations and crop-replacement deltas, not an external independent phenomenon.
  • Hierarchical bbox annotation + composite grounding reward (conditional IoU + L1) no independent evidence
    purpose: Counteract oversized crops induced by accuracy reward under resolution asymmetry and improve localization precision with few labels.
    Stage-II invention; validated only within this paper’s 256-box setup and IoU/accuracy tables.

pith-pipeline@v1.1.0-grok45 · 22692 in / 3837 out tokens · 51705 ms · 2026-07-13T16:54:11.249468+00:00 · methodology

0 comments
read the original abstract

Spiking Neural Networks (SNNs), characterized by their event-driven computation and low power consumption, have shown great potential for energy-efficient visual tracking on unmanned aerial vehicles (UAVs). However, existing SNN-based trackers often rely on costly event cameras, which limits their deployment on standard RGB-camera UAV platforms. To address this limitation, we propose STATrack, a fully spiking neural network framework for UAV visual tracking using only RGB inputs. To the best of our knowledge, this is the first study to explore fully spiking neural networks for RGB-based UAV visual tracking. To alleviate target semantic degradation caused by spike discretization and reduce background interference in UAV scenes, we introduce an Adaptive Mutual Information Maximization (AMIM) mechanism. AMIM maximizes the mutual information between template inputs and their deep target-aware features, encouraging the spiking backbone to preserve discriminative target semantics. In addition, a sample-difficulty-aware dynamic weighting strategy is designed to adaptively adjust the mutual information constraint during training. Extensive experiments on four widely used UAV tracking benchmarks demonstrate that STATrack achieves state-of-the-art tracking performance with low theoretical energy consumption, highlighting its potential for energy-constrained UAV applications.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

62 extracted references · 28 linked inside Pith

  1. [1]

    Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 1(2):3,

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 1(2):3,

    Pith/arXiv arXiv

  2. [2]

    Qwen2.5-vl technical report, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhao- hai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. 2

    2025

  3. [3]

    Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling.arXiv preprint arXiv:2412.05271, 2024. 2

    Pith/arXiv arXiv 2024

  4. [4]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.Science China Information Sciences, 67(12):220101, 2024

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.Science China Information Sciences, 67(12):220101, 2024

    2024

  5. [5]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. InPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24185–24198, 2024. 2

    2024

  6. [6]

    Scaling instruction- finetuned language models.Journal of Machine Learning Research, 25(70):1–53, 2024

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction- finetuned language models.Journal of Machine Learning Research, 25(70):1–53, 2024. 2

    2024

  7. [7]

    Blink: Multimodal large language models can see but not perceive

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. InEuropean Conference on Com- puter Vision, pages 148–166. Springer, 2024. 1

    2024

  8. [8]

    Refocus: Visual editing as a chain of thought for structured image understanding.arXiv preprint arXiv:2501.05452, 2025

    Xingyu Fu, Minqian Liu, Zhengyuan Yang, John Cor- ring, Yijuan Lu, Jianwei Yang, Dan Roth, Dinei Floren- cio, and Cha Zhang. Refocus: Visual editing as a chain of thought for structured image understanding.arXiv preprint arXiv:2501.05452, 2025. 2

    Pith/arXiv arXiv 2025

  9. [9]

    Deepseek-r1: Incentivizing reasoning ca- pability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning ca- pability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025. 5

    Pith/arXiv arXiv 2025

  10. [10]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 6700–6709, 2019. 1, 4

    2019

  11. [11]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 3

    Pith/arXiv arXiv 2024

  12. [12]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. InProceedings of the IEEE/CVF international confer- ence on computer vision, pages 4015–4026, 2023. 6

    2023

  13. [13]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9579–9589, 2024. 4

    2024

  14. [14]

    Mini-o3: Scaling up reasoning pat- terns and interaction turns for visual search.arXiv preprint arXiv:2509.07969, 2025

    Xin Lai, Junyi Li, Wei Li, Tao Liu, Tianjian Li, and Hengshuang Zhao. Mini-o3: Scaling up reasoning pat- terns and interaction turns for visual search.arXiv preprint arXiv:2509.07969, 2025. 1, 3, 6, 7

    Pith/arXiv arXiv 2025

  15. [15]

    Proxyclip: Proxy attention improves clip for open-vocabulary segmentation

    Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Proxyclip: Proxy attention improves clip for open-vocabulary segmentation. InEuropean Conference on Computer Vision, pages 70–88. Springer, 2024. 4

    2024

  16. [16]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 2

    Pith/arXiv arXiv 2024

  17. [17]

    Imag- ine while reasoning in space: Multimodal visualization-of- thought.arXiv preprint arXiv:2501.07542, 2025

    Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vuli´c, and Furu Wei. Imag- ine while reasoning in space: Multimodal visualization-of- thought.arXiv preprint arXiv:2501.07542, 2025. 2

    Pith/arXiv arXiv 2025

  18. [18]

    Dyfo: A training-free dynamic focus visual search for enhancing lmms in fine-grained visual understanding

    Geng Li, Jinglin Xu, Yunzhen Zhao, and Yuxin Peng. Dyfo: A training-free dynamic focus visual search for enhancing lmms in fine-grained visual understanding. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 9098–9108, 2025. 3

    2025

  19. [19]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InIn- ternational conference on machine learning, pages 19730– 19742. PMLR, 2023. 2

    2023

  20. [20]

    Star-r1: Spatial transformation reasoning by rein- forcing multimodal llms.arXiv preprint arXiv:2505.15804,

    Zongzhao Li, Zongyang Ma, Mingze Li, Songyou Li, Yu Rong, Tingyang Xu, Ziqi Zhang, Deli Zhao, and Wenbing Huang. Star-r1: Spatial transformation reasoning by rein- forcing multimodal llms.arXiv preprint arXiv:2505.15804,

    Pith/arXiv arXiv

  21. [21]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023. 2

    2023

  22. [22]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26296–26306, 2024. 2

    2024

  23. [23]

    Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vi- sion, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vi- sion, pages 216–233. Springer, 2024. 1

    2024

  24. [24]

    Seg-zero: Reasoning-chain guided segmentation via cognitive reinforcement.arXiv preprint arXiv:2503.06520, 2025

    Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg-zero: Reasoning-chain guided segmentation via cognitive reinforcement.arXiv preprint arXiv:2503.06520, 2025. 3

    Pith/arXiv arXiv 2025

  25. [25]

    Chain-of-spot: Interactive reasoning improves large vision-language models.arXiv preprint arXiv:2403.12966,

    Zuyan Liu, Yuhao Dong, Yongming Rao, Jie Zhou, and Ji- wen Lu. Chain-of-spot: Interactive reasoning improves large vision-language models.arXiv preprint arXiv:2403.12966,

    Pith/arXiv arXiv

  26. [26]

    Through the magnifying glass: Adaptive perception magnifica- tion for hallucination-free vlm decoding.arXiv preprint arXiv:2503.10183, 2025

    Shunqi Mao, Chaoyi Zhang, and Weidong Cai. Through the magnifying glass: Adaptive perception magnifica- tion for hallucination-free vlm decoding.arXiv preprint arXiv:2503.10183, 2025. 3

    Pith/arXiv arXiv 2025

  27. [27]

    Kam-cot: Knowledge augmented multimodal chain-of-thoughts reasoning

    Debjyoti Mondal, Suraj Modi, Subhadarshi Panda, Rituraj Singh, and Godawari Sudhakar Rao. Kam-cot: Knowledge augmented multimodal chain-of-thoughts reasoning. InPro- ceedings of the AAAI conference on artificial intelligence, pages 18798–18806, 2024. 2

    2024

  28. [28]

    Spacer: Rein- forcing mllms in video spatial reasoning.arXiv preprint arXiv:2504.01805, 2025

    Kun Ouyang, Yuanxin Liu, Haoning Wu, Yi Liu, Hao Zhou, Jie Zhou, Fandong Meng, and Xu Sun. Spacer: Rein- forcing mllms in video spatial reasoning.arXiv preprint arXiv:2504.01805, 2025. 3

    Pith/arXiv arXiv 2025

  29. [29]

    Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl

    Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.07536, 2025. 3

    Pith/arXiv arXiv 2025

  30. [30]

    Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024. 6

    Pith/arXiv arXiv 2024

  31. [31]

    Deepseekmath: Pushing the limits of math- ematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of math- ematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024. 5

    Pith/arXiv arXiv 2024

  32. [32]

    Vlm-r1: A stable and general- izable r1-style large vision-language model, 2025.URL https://arxiv

    Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-r1: A stable and general- izable r1-style large vision-language model, 2025.URL https://arxiv. org/abs/2504.07615, 3(6):11, 2025. 3

    Pith/arXiv arXiv 2025

  33. [33]

    Towards more unified in-context visual un- derstanding

    Dianmo Sheng, Dongdong Chen, Zhentao Tan, Qiankun Liu, Qi Chu, Jianmin Bao, Tao Gong, Bin Liu, Shengwei Xu, and Nenghai Yu. Towards more unified in-context visual un- derstanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13362– 13372, 2024. 4

    2024

  34. [34]

    Unicl-sam: Uncertainty-driven in-context segmen- tation with part prototype discovery

    Dianmo Sheng, Dongdong Chen, Zhentao Tan, Qiankun Liu, Qi Chu, Tao Gong, Bin Liu, Jing Han, Wenbin Tu, Shengwei Xu, et al. Unicl-sam: Uncertainty-driven in-context segmen- tation with part prototype discovery. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 20201–20211, 2025. 4

    2025

  35. [35]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 1

    2019

  36. [36]

    Visual agents as fast and slow thinkers.arXiv preprint arXiv:2408.08862, 2024

    Guangyan Sun, Mingyu Jin, Zhenting Wang, Cheng-Long Wang, Siqi Ma, Qifan Wang, Tong Geng, Ying Nian Wu, Yongfeng Zhang, and Dongfang Liu. Visual agents as fast and slow thinkers.arXiv preprint arXiv:2408.08862, 2024. 2

    Pith/arXiv arXiv 2024

  37. [37]

    Ufo: A unified approach to fine-grained visual perception via open- ended language interface.arXiv preprint arXiv:2503.01342,

    Hao Tang, Chenwei Xie, Haiyang Wang, Xiaoyi Bao, Tingyu Weng, Pandeng Li, Yun Zheng, and Liwei Wang. Ufo: A unified approach to fine-grained visual perception via open- ended language interface.arXiv preprint arXiv:2503.01342,

    arXiv

  38. [38]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 2

    Pith/arXiv arXiv 2023

  39. [39]

    Pixel reasoner: Incentivizing pixel-space rea- soning with curiosity-driven reinforcement learning.arXiv preprint arXiv:2505.15966, 2025

    Haozhe Wang, Alex Su, Weiming Ren, Fangzhen Lin, and Wenhu Chen. Pixel reasoner: Incentivizing pixel-space rea- soning with curiosity-driven reinforcement learning.arXiv preprint arXiv:2505.15966, 2025. 1, 3, 7

    Pith/arXiv arXiv 2025

  40. [40]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024. 2

    Pith/arXiv arXiv 2024

  41. [41]

    Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models

    Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, Wei Yu, and Dacheng Tao. Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 7907–7915, 2025. 1, 3

    2025

  42. [42]

    Seggpt: Segmenting ev- erything in context.arXiv preprint arXiv:2304.03284, 2023

    Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting ev- erything in context.arXiv preprint arXiv:2304.03284, 2023. 4

    Pith/arXiv arXiv 2023

  43. [43]

    Perception in reflection.arXiv preprint arXiv:2504.07165, 2025

    Yana Wei, Liang Zhao, Kangheng Lin, En Yu, Yuang Peng, Runpei Dong, Jianjian Sun, Haoran Wei, Zheng Ge, Xi- angyu Zhang, et al. Perception in reflection.arXiv preprint arXiv:2504.07165, 2025. 2

    Pith/arXiv arXiv 2025

  44. [44]

    Reinforcing spatial reasoning in vision-language models with interwoven think- ing and visual drawing.arXiv preprint arXiv:2506.09965,

    Junfei Wu, Jian Guan, Kaituo Feng, Qiang Liu, Shu Wu, Liang Wang, Wei Wu, and Tieniu Tan. Reinforcing spatial reasoning in vision-language models with interwoven think- ing and visual drawing.arXiv preprint arXiv:2506.09965,

    Pith/arXiv arXiv

  45. [45]

    V?: Guided visual search as a core mechanism in multimodal llms

    Penghao Wu and Saining Xie. V?: Guided visual search as a core mechanism in multimodal llms. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13084–13094, 2024. 1, 3

    2024

  46. [46]

    Datasetdm: Synthesizing data with perception annota- tions using diffusion models.Advances in Neural Informa- tion Processing Systems, 36:54683–54695, 2023

    Weijia Wu, Yuzhong Zhao, Hao Chen, Yuchao Gu, Rui Zhao, Yefei He, Hong Zhou, Mike Zheng Shou, and Chunhua Shen. Datasetdm: Synthesizing data with perception annota- tions using diffusion models.Advances in Neural Informa- tion Processing Systems, 36:54683–54695, 2023. 4

    2023

  47. [47]

    Side adapter network for open-vocabulary semantic segmentation

    Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xi- ang Bai. Side adapter network for open-vocabulary semantic segmentation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2945– 2954, 2023. 4

    2023

  48. [48]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InThe eleventh international conference on learning representations, 2022. 3

    2022

  49. [49]

    Mllms know where to look: Training-free per- ception of small visual details with multimodal llms.arXiv preprint arXiv:2502.17422, 2025

    Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Filip Ilievski. Mllms know where to look: Training-free per- ception of small visual details with multimodal llms.arXiv preprint arXiv:2502.17422, 2025. 3

    Pith/arXiv arXiv 2025

  50. [50]

    Person- alize segment anything model with one shot.arXiv preprint arXiv:2305.03048, 2023

    Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junt- ing Pan, Hao Dong, Peng Gao, and Hongsheng Li. Person- alize segment anything model with one shot.arXiv preprint arXiv:2305.03048, 2023. 6

    Pith/arXiv arXiv 2023

  51. [51]

    Chain-of-focus: Adaptive visual search and zooming for multimodal reasoning via rl.arXiv e-prints, pages arXiv–2505, 2025

    Xintong Zhang, Zhi Gao, Bofei Zhang, Pengxiang Li, Xi- aowen Zhang, Yang Liu, Tao Yuan, Yuwei Wu, Yunde Jia, Song-Chun Zhu, et al. Chain-of-focus: Adaptive visual search and zooming for multimodal reasoning via rl.arXiv e-prints, pages arXiv–2505, 2025. 1, 3, 7

    2025

  52. [52]

    X-paste: Revisiting scalable copy- paste for instance segmentation using clip and stablediffu- sion

    Hanqing Zhao, Dianmo Sheng, Jianmin Bao, Dongdong Chen, Dong Chen, Fang Wen, Lu Yuan, Ce Liu, Wenbo Zhou, Qi Chu, et al. X-paste: Revisiting scalable copy- paste for instance segmentation using clip and stablediffu- sion. InInternational Conference on Machine Learning, pages 42098–42109. PMLR, 2023. 4

    2023

  53. [53]

    Training- free open-vocabulary semantic segmentation via diverse pro- totype construction and sub-region matching

    Xuanpu Zhao, Dianmo Sheng, Zhentao Tan, Zhiwei Zhao, Tao Gong, Qi Chu, Bin Liu, and Nenghai Yu. Training- free open-vocabulary semantic segmentation via diverse pro- totype construction and sub-region matching. InProceed- ings of the AAAI Conference on Artificial Intelligence, pages 10474–10482, 2025. 4

    2025

  54. [54]

    Deep- eyes: Incentivizing” thinking with images” via reinforce- ment learning.arXiv preprint arXiv:2505.14362, 2025

    Ziwei Zheng, Michael Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang, Chao Shen, and Xing Yu. Deep- eyes: Incentivizing” thinking with images” via reinforce- ment learning.arXiv preprint arXiv:2505.14362, 2025. 1, 3, 7

    Pith/arXiv arXiv 2025

  55. [55]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shen- glong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025. 2 Learning to Focus and Precise Cropping: A Reinforcement Learning Framework with Information Gap...

    Pith/arXiv arXiv 2025

  56. [56]

    Training Dynamics Analysis In this section, we visualize and analyze evolution of several key metrics during the first and second stages of training. 8.1. Stage-I To analyze the model’s behavior during the first stage of training, we present the evolution of four key metrics for both the BaseLine and our Stage-I Model with Info Gap in Figure 5. These metr...

  57. [57]

    All experiments in this section are conducted with the maximum number of visual tokens set to 1,024

    More Preliminary Analysis To more rigorously demonstrate DeepEyes’s insufficient at- tention to cropped regions, we also conducted the following experiments. All experiments in this section are conducted with the maximum number of visual tokens set to 1,024. GT test.We isolate samples from benchmarks where re- gions predicted by DeepEyes poorly cover GT (...

  58. [58]

    The first strategy, ‘Mix Data’, involves training the model in a single stage by mix- ing our collected data with the Visual Probe data at a 1:1 ratio

    More Ablation Studies Data Utilization Strategy.As shown in the Table 8, we compare two data usage strategies. The first strategy, ‘Mix Data’, involves training the model in a single stage by mix- ing our collected data with the Visual Probe data at a 1:1 ratio. The second strategy, which we term ‘Stage-II’, is the two-stage approach adopted. In the first...

  59. [59]

    Benchmarks and Metrics Details Our method is evaluated on three benchmarks. The first, HR-Bench 8Kwith an average resolution of 7680, which consists of two sub-tasks: Fine-grained Single-instance Per- ception (FSP) and Fine-grained Cross-instance Perception (FCP). The 8K images are cropped around the objects in question to produceHR-Bench 4K. The third,� ...

  60. [60]

    We also compare our training time with that of DeepEyes as shown in Table 11

    Training Details We show the related hyper-parameters we use in Table 12. We also compare our training time with that of DeepEyes as shown in Table 11. Our total training time is shorter than that of DeepEyes. This is because we reduce the number Parameter Value train batch size 256 rollout num per sample 16 ppo mini batch size 32 ppo micro batch size per...

    2048

  61. [61]

    In the first example, while both mod- els correctly crop the flag, DeepEyes provides an incorrect answer, whereas Stage-I arrives at the correct one

    Visualization Analysis In Figure 7, we visually analyze the inference processes of DeepEyes and Stage-I. In the first example, while both mod- els correctly crop the flag, DeepEyes provides an incorrect answer, whereas Stage-I arrives at the correct one. In the second example, both models initially fail to crop the jack’s sleeve. However, DeepEyes proceed...

  62. [62]

    We address this by propos- ing an information gap mechanism

    Limitations and Future Works In this work, we first identify a critical issue in existing agent-based workflows for complex image understanding: the sub-optimal tool invocation that stems from a rigid for- malization of the cropping tool. We address this by propos- ing an information gap mechanism. Building upon this, we further enhance model’s cropping p...