Pith. sign in

REVIEW 3 major objections 8 minor 62 references

RefBench-PRO claims that referring-expression comprehension is not one skill but six, and that current multimodal models—best at 71.4% on the five non-rejection tasks—fail most visibly when the task demands relational, commonsense, or rejec

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 18:10 UTC pith:VTN4TRS4

load-bearing objection Useful six-task decomposition for REC, but Ref-R1's gains may be inflated by train/test overlap; the paper needs a clear holdout statement. the 3 major comments →

arxiv 2512.06276 v3 pith:VTN4TRS4 submitted 2025-12-06 cs.CV cs.AI

RefBench-PRO: Perceptual and Reasoning Oriented Benchmark for Referring Expression Comprehension

Tianyi Gao , Hao Li , Han Fang , Xin Wei , Xiaodong Dong , Hongbo Sun , Ye Yuan , Zhongjiang He
show 3 more authors
Jinglin Xu Jingmin Xin Hao Sun
This is my paper
classification cs.CV cs.AI
keywords referring expression comprehensionvisual groundingbenchmarkmultimodal large language modelsperception vs reasoningreinforcement learningGRPOobject localization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Existing referring-expression benchmarks are nearly saturated: models score in the high 80s on RefCOCO-style tests, making it hard to see what they actually cannot do. This paper argues that the task decomposes into two capabilities, visual-cue perception and compositional reasoning, and further into six sub-tasks: attribute, position, interaction, relation, commonsense, and rejection. It builds a 6,000-pair benchmark that stresses small, cluttered targets and requires implicit or comparative reasoning, plus a 200k-pair training set and a reinforcement-learning recipe called Ref-R1. If the paper is right, a single aggregate grounding score is misleading; models need separate, per-capability diagnosis, and dynamic-IoU reinforcement learning is a workable way to improve both perception and reasoning.

Core claim

The paper's central claim is that current REC evaluation confounds two distinct abilities—seeing a visual cue and reasoning over it—and that separating them reveals a large, previously hidden gap. RefBench-PRO organizes 6,000 human-corrected pairs into six 1,000-pair tasks; images are high-resolution and objects are small, with an average target area of about 10%, and expressions are engineered to be unambiguous only through the targeted cue. Across 24 models, none exceeds 72% accuracy on the five positive tasks, and the average drop from the three perception tasks to the two reasoning tasks is 6.02 points. The rejection task is the clearest failure: most models cannot say 'absent' and inste

What carries the argument

The load-bearing device is the six-way task taxonomy paired with rule-based selection: an object is assigned to Attribute when it has a unique property among same-category peers, to Position when unambiguous only through a spatial anchor, to Interaction when identified by ordinal placement in a homogeneous group, to Relation when linked to an anchor by similarity or contrast, to Commonsense when named by function, and to Reject when the expression describes a plausible but absent object. That taxonomy turns an aggregate accuracy number into a per-capability diagnostic. The second device is the DyIoU-GRPO reward: the IoU acceptance threshold rises during training while smaller boxes get a low

Load-bearing premise

The benchmark's usefulness as a generalization test rests on the 6,000 benchmark pairs being excluded from the 203,985-pair RefObjects-200k training mixture used to train Ref-R1, but the paper nowhere states that this exclusion was made.

What would settle it

Compute the overlap between RefBench-PRO's 6,000 expression–image pairs and RefObjects-200k's training pairs by matching image IDs, bounding boxes, and expression text; if a substantial share of benchmark pairs are near-duplicates of training pairs, the reported difficulty and Ref-R1's gains are inflated by memorization.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • High scores on RefCOCO/+/g and Ref-L4 overstate real grounding ability; the same models fall below 72% when small targets and compositional expressions are required.
  • Reporting per-task accuracy for attribute, position, interaction, relation, commonsense, and rejection makes failure modes legible instead of hiding them behind one average.
  • Dynamic-IoU reinforcement learning is a transferable training recipe: it lifts a 7B model's five-task accuracy by roughly 12 points and rejection accuracy from 3.1% to 58.2%.
  • Explicit thinking traces consistently improve relation and commonsense accuracy across three models that support a thinking mode, so reasoning chains are a practical lever for grounding.
  • Near-chance performance on the reject task even as binary classification implies localization hallucination is not fixed by instruction wording; models need training that does not reward unconditional box outputs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the benchmark and the training set are generated by the same pipeline, the six-way taxonomy may be partly measuring the generator's biases; an independently human-written test subset would be a sterner check that the paper does not run.
  • The 6.02-point perception-to-reasoning gap suggests steering post-training toward relation, commonsense, and reject samples; the paper's own ablation table indicates those are where the largest headroom sits, but it does not perform that curriculum experiment.
  • If the dynamic IoU threshold is the true source of small-object gains, then ablating the threshold schedule—for instance, training with a fixed 0.5 IoU threshold but the same data—would isolate whether the improvement comes from the schedule or from the RL data itself.
  • The rejection result implies models need an explicit absent-object decision, not just a better box regressor; a concrete extension is adding negative-only images to the RL mixture and measuring whether rejection accuracy moves off chance.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. RefBench-PRO proposes a new REC benchmark that separates referring expression comprehension into two dimensions—visual-cue perception (attribute, position, interaction) and compositional reasoning (relation, commonsense, reject)—with 1,000 test pairs per task. The benchmark is built from FineHARD images through an automated pipeline using Qwen2.5-VL-72B for parsing and expression generation and Grounding DINO for boxes, producing a 6,000-pair test set and a 203,985-pair training set called RefObjects-200k. The paper also presents Ref-R1, a two-stage training method (CoT SFT followed by DyIoU-GRPO with a dynamic IoU threshold and a group-quality reward) on Qwen2.5-VL-7B, and evaluates 24 models. The main empirical claims are that no model exceeds 72% Accp on RefBench-PRO, that there is a systematic perception-to-reasoning drop, that most models fail at rejection, and that Ref-R1 raises Qwen2.5-VL-7B to 69.35 Accp / 67.49 Acco.

Significance. If the benchmark independence issue is resolved, RefBench-PRO has clear value: the six-task decomposition gives more interpretable diagnostics than a single aggregate REC score, the reject task exposes grounding hallucination in a controlled way, and the difficulty controls (distractor count, object area, hop count) provide useful stress-test axes. The automated pipeline is a practical contribution, and the dynamic-IoU reward is a sensible extension of GRPO for localization. The evaluation of 24 models and the analysis of thinking mode and rejection settings are useful strengths. However, the paper's central Ref-R1 baseline claim is currently not secure because the benchmark and training set may overlap, and the per-category ablation claims are overstated in the supplementary.

major comments (3)
  1. [§3.2 vs §4.1/§5.1] Benchmark independence from the training set is not established. §3.2 says 'A subset of pairs across six dimensions is then sampled... to establish RefBench-PRO' from RefObjects-200k, while §4.1 trains Ref-R1 on 180K samples from RefObjects-200k and §5.1 selects 80k samples from the same pool for RL. No sentence states that the 6,000 benchmark pairs or their source images are excluded from training. If any overlap exists, the reported +11.77 Accp gain over Qwen2.5-VL-7B may partly reflect memorization rather than generalization; the 'further corrected by ten annotators' step does not remove the exposure. Please provide an explicit disjoint split (ideally disjoint images), re-run Ref-R1 on that split, and report the held-out numbers, or provide overlap statistics demonstrating no contamination.
  2. [Supplementary §9, Table 8] The supplementary states that Ref-R1 'demonstrates comprehensive performance gains across all sub-categories,' but Table 8 contradicts this: Ref-R1's Reject accuracy (58.20) is lower than SFT-CoT (61.50), and its Commonsense accuracy (65.40) is lower than GRPO+RGroupQuality+RIoU (66.87). The aggregate Accp/Acco gains remain positive, but the per-category claim is not supported. Please correct the claim and discuss the specific trade-offs, especially the decrease in rejection accuracy relative to the CoT SFT stage.
  3. [§3.2 / §3.1] Annotation quality is asserted but not measured. The 6,000 benchmark pairs are machine-generated and then 'corrected by ten annotators,' yet no inter-annotator agreement, correction rate, or residual error analysis is reported. Given that the generator is Qwen2.5-VL-72B—the same model family as several evaluated models and the backbone of Ref-R1—independent verification is essential for the benchmark's credibility. Please add the annotation protocol, agreement statistics (e.g., pairwise agreement or Fleiss' kappa), and an analysis of what fraction of labels changed during correction.
minor comments (8)
  1. [§3.2] Typo: 'corrrected' should be 'corrected'; also the name 'REFbench-PRO' appears with inconsistent capitalization at the end of §3.2.
  2. [§5.2] Typo in 'a comprehensive benchmark fir evaluating' should be 'for evaluating.'
  3. [Table 2] The Qwen2-VL row appears as '46.528.543.0' without separators; it should read '46.5 28.5 43.0'.
  4. [§2 Related Work] Citation mismatches: 'Deepeyes [35]' and 'Chain-of-Focus [23]' appear to cite references [35] (VLM-R1) and [23] (Visual-Rft), which are not the intended methods. Please correct these references.
  5. [References] References [46] and [47] both point to the same paper (Yu et al., Modeling Context in Referring Expressions). Please deduplicate.
  6. [Figure 2 caption] The caption uses 'Vision-language Interleaved Reasoning' while the text consistently says 'Compositional Reasoning'; align the terminology.
  7. [Table 5] The column header layout is confusing: it reads 'Model Acc RC / Rejection setting / Grounding Classification.' Clarify which metric corresponds to which column, and define the reported Acc_RC value.
  8. [Figure 4 caption] The caption mentions 'semantic density' without defining it; please give a formal definition or remove the term.

Circularity Check

1 steps flagged

RefBench-PRO test pairs are sampled from the same RefObjects-200k pool used to train Ref-R1, with no stated holdout; Ref-R1's headline gains on RefBench-PRO are not an independent evaluation.

specific steps
  1. fitted input called prediction [§3.2 (Data statistics), §4.1 (Chain-of-Thought Cold Start), §5.1 (Experimental Settings); Table 6]
    "We construct RefObjects-200k via our annotation pipeline. A subset of pairs across six dimensions is then sampled, further corrrected by ten annotators to establish RefBench-PRO. ... Using the RefObjects-200k dataset, we leverage Qwen2.5-VL-72B to generate chain-of-thought reasoning for each sample. ... In the first stage, we train on 180K samples generated from RefObjects-200k for one epoch"

    RefBench-PRO is defined as a sampled subset of RefObjects-200k, and Ref-R1 is trained on samples generated from RefObjects-200k. The paper never states that the 6,000 benchmark pairs, or their source images, are excluded from the 180K SFT / 80K RL training mixtures. Therefore the reported Ref-R1 gains on RefBench-PRO are partly measured on training-distribution data. The ten-annotator 'correction' changes wording or boxes but not the underlying image-target pair, so an image-level or pair-level memorization channel remains. The benchmark accuracy is thus presented as an evaluation of generalization when, by construction, it is also a fit to the training pool.

full rationale

The benchmark diagnostics for frozen models are not circular: the six-task decomposition, the scoring rules, and the observation that current MLLMs decline on harder perceptual/reasoning splits are defined independently of any fitted parameters and are supported by external comparisons (RefCOCO/+/g, Ref-L4). The circular content is confined to the Ref-R1 evaluation on RefBench-PRO. §3.2 makes RefBench-PRO a sampled subset of RefObjects-200k; §4.1 and §5.1 train Ref-R1 on samples generated from that same pool, without any stated holdout of the benchmark pairs or their source images. Hence the headline +11.77 Accp / +18.99 Acco over Qwen2.5-VL-7B partly reflects overlap with the training distribution. This does not invalidate the benchmark's interpretability claim for frozen baselines, nor the independent external support for Ref-R1 on RefCOCO/+/g (+2.14) and Ref-L4 (+4.43). The circularity is therefore partial, not total.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central benchmark rests on trusting an MLLM to generate properties and verify them, Grounding DINO to supply boxes, and ten human annotators to fix only the 6,000 test pairs. The method part rests on hand-chosen RL reward hyperparameters. No new physical entities are introduced.

free parameters (5)
  • α (start IoU threshold) = 0.5
    Hand-set in Eq. (5), controls initial reward threshold; no validation procedure specified.
  • β (end IoU threshold) = 0.8
    Hand-set in Eq. (5), raises localization difficulty as training progresses.
  • d_max (small-target penalty) = 0.15
    Hand-set in Eq. (5); lowers the threshold for smaller bounding boxes.
  • p (group quality reward weight) = 0.5
    Hand-set in Eq. (6); controls the ±k/n adjustment for hard groups.
  • τ_q dynamic group threshold schedule = unspecified
    Defined as gradually increasing during training; exact schedule and value are not given.
axioms (5)
  • domain assumption Qwen2.5-VL-72B property dictionaries, consistency checks, and expression generation are accurate enough to serve as ground-truth generation.
    The fully automated pipeline hinges on one MLLM for labels; errors propagate into RefObjects-200k and, before human correction, into the benchmark.
  • domain assumption Grounding DINO bounding boxes correctly localize the described objects.
    Target boxes come from Grounding DINO; if detection fails, generated boxes and paired expressions are wrong.
  • domain assumption Ten human annotators provide reliable correction for the 6,000 benchmark pairs.
    No inter-annotator agreement, adjudication protocol, or annotation-instruction details are reported.
  • standard math The standard GRPO objective is applicable and stable for bounding-box rewards.
    Eq. (4) assumes the KL-penalized, group-relative importance-sampling objective used in GRPO.
  • ad hoc to paper The dynamic IoU threshold (Eq. 5) improves coarse-to-fine localization learning.
    Motivated by human cognition but not derived; the threshold formula and its hand-chosen hyperparameters are a design hypothesis.

pith-pipeline@v1.3.0-alltime-deepseek · 21117 in / 12912 out tokens · 121916 ms · 2026-08-03T18:10:44.409805+00:00 · methodology

0 comments
read the original abstract

Referring Expression Comprehension (REC) is a vision-language task that localizes a specific image region based on a textual description. Existing REC benchmarks primarily evaluate perceptual capabilities and lack interpretable scoring mechanisms, which cannot reveal the grounding capability of Multi-modal Large Language Model (MLLM) across different cognitive abilities. To address this limitation, we introduce RefBench-PRO, a comprehensive REC benchmark, which decomposes referring expressions into two core dimensions, i.e., perception and reasoning, and further subdivides them into six progressively challenging tasks, such as attribute, position, interaction, commonsense, relation and reject. We also develop a fully automated data-generation pipeline that produces diverse referring expressions across these six sub-dimensions. Furthermore, We propose Ref-R1, an RL-based learning scheme, which incorporates Dynamic IoU-based GRPO to improve localization accuracy under increasingly complex reasoning conditions, establishing a stronger baseline for REC. Extensive experiments demonstrate that our RefBench-PRO enables interpretable evaluation of MLLM on referring expression comprehension, presenting greater challenges in both perception and reasoning.

Figures

Figures reproduced from arXiv: 2512.06276 by Han Fang, Hao Li, Hao Sun, Hongbo Sun, Jinglin Xu, Jingmin Xin, Tianyi Gao, Xiaodong Dong, Xin Wei, Ye Yuan, Zhongjiang He.

Figure 1
Figure 1. Figure 1: Comparison of the proposed RefBench-PRO and the Re [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Visualizations of the six tasks in the RefBench-PRO. The three tasks on the left (Attribute, Position, Interaction) belong to [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The fine-grained referential annotation pipeline to construct RefObjects-200k and RefBench-PRO. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: A comparative analysis of RefBench-PRO against es [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: A comprehensive evaluation of model performance across three key aspects on the perception dimension: (a) the number of [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Visual comparisons of Ref-R1 and state-of-the [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Prompt for Image Parsing [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Prompts for Referring Expression Generation. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Visual case studies for the Visual-cue Perception subcategories (Attribute, Position, and Interaction). Each row presents a [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Visual case studies for the Compositional Reasoning subcategories (Relation, Commonsense, and Reject). Each row presents a [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

62 extracted references · 29 linked inside Pith

  1. [1]

    Flamingo: a visual language model for few-shot learning.NeurIPS, 35: 23716–23736, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.NeurIPS, 35: 23716–23736, 2022. 3

    2022

  2. [2]

    Llava-onevision-1.5: Fully open framework for democratized multimodal training

    Xiang An, Yin Xie, Kaicheng Yang, Wenkang Zhang, Xiuwei Zhao, Zheng Cheng, Yirui Wang, Songcen Xu, Changrui Chen, Chunsheng Wu, et al. Llava-onevision-1.5: Fully open framework for democratized multimodal training. arXiv preprint arXiv:2509.23661, 2025. 6

    Pith/arXiv arXiv 2025

  3. [3]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025. 6, 8

    Pith/arXiv arXiv 2025

  4. [4]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 2, 3, 5, 6

    Pith/arXiv arXiv 2025

  5. [5]

    Univg-r1: Reasoning guided universal visual grounding with reinforce- ment learning.arXiv preprint arXiv:2505.14231, 2025

    Sule Bai, Mingxing Li, Yong Liu, Jing Tang, Haoji Zhang, Lei Sun, Xiangxiang Chu, and Yansong Tang. Univg-r1: Reasoning guided universal visual grounding with reinforce- ment learning.arXiv preprint arXiv:2505.14231, 2025. 2, 3, 6

    Pith/arXiv arXiv 2025

  6. [6]

    Revisiting referring expression comprehension eval- uation in the era of large multimodal models

    Jierun Chen, Fangyun Wei, Jinjing Zhao, Sizhe Song, Bo- huai Wu, Zhuoxuan Peng, S-H Gary Chan, and Hongyang Zhang. Revisiting referring expression comprehension eval- uation in the era of large multimodal models. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 513–524, 2025. 2, 4, 5, 7, 8

    2025

  7. [7]

    Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling.arXiv preprint arXiv:2412.05271, 2024. 2, 3

    Pith/arXiv arXiv 2024

  8. [8]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blis- tein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025. 6

    Pith/arXiv arXiv 2025

  9. [9]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models.arXiv e-prints, pages arXiv–2409, 2024

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tri- pathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models.arXiv e-prints, pages arXiv–2409, 2024. 2, 3

    2024

  10. [10]

    Grec: Generalized referring expression comprehension

    Shuting He, Henghui Ding, Chang Liu, and Xudong Jiang. Grec: Generalized referring expression comprehension. arXiv preprint arXiv:2308.16182, 2023. 2

    Pith/arXiv arXiv 2023

  11. [11]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 6

    Pith/arXiv arXiv 2024

  12. [12]

    Chatrex: Tam- ing multimodal llm for joint perception and understanding

    Qing Jiang, Gen Luo, Yuqin Yang, Yuda Xiong, Yihao Chen, Zhaoyang Zeng, Tianhe Ren, and Lei Zhang. Chatrex: Tam- ing multimodal llm for joint perception and understanding. arXiv preprint arXiv:2411.18363, 2024. 2, 3, 6

    Pith/arXiv arXiv 2024

  13. [13]

    Rex-thinker: Grounded object re- ferring via chain-of-thought reasoning.arXiv preprint arXiv:2506.04034, 2025

    Qing Jiang, Xingyu Chen, Zhaoyang Zeng, Junzhi Yu, and Lei Zhang. Rex-thinker: Grounded object re- ferring via chain-of-thought reasoning.arXiv preprint arXiv:2506.04034, 2025. 6

    Pith/arXiv arXiv 2025

  14. [14]

    Referring to any person.arXiv preprint arXiv:2503.08507, 2025

    Qing Jiang, Lin Wu, Zhaoyang Zeng, Tianhe Ren, Yuda Xiong, Yihao Chen, Qin Liu, and Lei Zhang. Referring to any person.arXiv preprint arXiv:2503.08507, 2025. 2, 3

    Pith/arXiv arXiv 2025

  15. [15]

    Referitgame: Referring to objects in pho- tographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in pho- tographs of natural scenes. InProceedings of the 2014 con- ference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014. 2

    2014

  16. [16]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models.arXiv preprint arXiv:2407.07895, 2024

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models.arXiv preprint arXiv:2407.07895, 2024. 2, 3, 6

    Pith/arXiv arXiv 2024

  17. [17]

    Migician: Revealing the magic of free-form multi-image grounding in multimodal large language mod- els.arXiv preprint arXiv:2501.05767, 2025

    You Li, Heyu Huang, Chi Chen, Kaiyu Huang, Chao Huang, Zonghao Guo, Zhiyuan Liu, Jinan Xu, Yuhua Li, Ruix- uan Li, et al. Migician: Revealing the magic of free-form multi-image grounding in multimodal large language mod- els.arXiv preprint arXiv:2501.05767, 2025. 2, 6

    Pith/arXiv arXiv 2025

  18. [18]

    Eagle 2: Building post- training data strategies from scratch for frontier vision- language models.arXiv preprint arXiv:2501.14818, 2025

    Zhiqi Li, Guo Chen, Shilong Liu, Shihao Wang, Vibashan VS, Yishen Ji, Shiyi Lan, Hao Zhang, Yilin Zhao, Sub- hashree Radhakrishnan, et al. Eagle 2: Building post- training data strategies from scratch for frontier vision- language models.arXiv preprint arXiv:2501.14818, 2025. 2, 3

    Pith/arXiv arXiv 2025

  19. [19]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014. 2

    2014

  20. [20]

    Perceive anything: Recognize, explain, caption, and segment anything in images and videos.arXiv preprint arXiv:2506.05302, 2025

    Weifeng Lin, Xinyu Wei, Ruichuan An, Tianhe Ren, Tingwei Chen, Renrui Zhang, Ziyu Guo, Wentao Zhang, Lei Zhang, and Hongsheng Li. Perceive anything: Recognize, explain, caption, and segment anything in images and videos.arXiv preprint arXiv:2506.05302, 2025. 3

    Pith/arXiv arXiv 2025

  21. [21]

    Finecops-ref: A new dataset and task for fine-grained compositional referring expression comprehension.arXiv preprint arXiv:2409.14750, 2024

    Junzhuo Liu, Xuzheng Yang, Weiwei Li, and Peng Wang. Finecops-ref: A new dataset and task for fine-grained compositional referring expression comprehension.arXiv preprint arXiv:2409.14750, 2024. 2

    Pith/arXiv arXiv 2024

  22. [22]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. InEuro- pean conference on computer vision, pages 38–55. Springer,

  23. [23]

    Visual- rft: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785, 2025

    Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual- rft: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785, 2025. 3

    Pith/arXiv arXiv 2025

  24. [24]

    Ovis: Structural embed- ding alignment for multimodal large language model.arXiv preprint arXiv:2405.20797, 2024

    Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural embed- ding alignment for multimodal large language model.arXiv preprint arXiv:2405.20797, 2024. 2, 3, 8

    Pith/arXiv arXiv 2024

  25. [25]

    Shiyin Lu, Yang Li, Yu Xia, Yuwei Hu, Shanshan Zhao, Yanqing Ma, Zhichao Wei, Yinglun Li, Lunhao Duan, Jian- shan Zhao, et al. Ovis2. 5 technical report.arXiv preprint arXiv:2508.11737, 2025. 6

    Pith/arXiv arXiv 2025

  26. [26]

    Multi-task collabora- tive network for joint referring expression comprehension and segmentation

    Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Liujuan Cao, Chenglin Wu, Cheng Deng, and Rongrong Ji. Multi-task collabora- tive network for joint referring expression comprehension and segmentation. InProceedings of the IEEE/CVF Con- ference on computer vision and pattern recognition, pages 10034–10043, 2020. 2

    2020

  27. [27]

    Groma: Localized visual tokenization for grounding multimodal large language models

    Chuofan Ma, Yi Jiang, Jiannan Wu, Zehuan Yuan, and Xiao- juan Qi. Groma: Localized visual tokenization for grounding multimodal large language models. InEuropean Conference on Computer Vision, pages 417–435. Springer, 2024. 2, 3

    2024

  28. [28]

    Generation and comprehension of unambiguous object descriptions

    Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11–20, 2016. 2, 4

    2016

  29. [29]

    Generation and comprehension of unambiguous object descriptions

    Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11–20, 2016. 7, 8, 9

    2016

  30. [30]

    Gpt-4v(ision) system card.https://cdn

    OpenAI. Gpt-4v(ision) system card.https://cdn. openai.com/papers/GPTV_System_Card.pdf,

  31. [31]

    Gpt-5 system card.https://cdn.openai

    OpenAI. Gpt-5 system card.https://cdn.openai. com/gpt-5-system-card.pdf, 2025. 6

    2025

  32. [32]

    Referring ex- pression comprehension: A survey of methods and datasets

    Yanyuan Qiao, Chaorui Deng, and Qi Wu. Referring ex- pression comprehension: A survey of methods and datasets. IEEE Transactions on Multimedia, 23:4426–4440, 2020. 2

    2020

  33. [33]

    Visual cot: Unleashing chain-of-thought reasoning in multi-modal language models.CoRR, 2024

    Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuo- fan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Unleashing chain-of-thought reasoning in multi-modal language models.CoRR, 2024. 3

    2024

  34. [34]

    Deepseekmath: Pushing the limits of math- ematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of math- ematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024. 3

    Pith/arXiv arXiv 2024

  35. [35]

    Vlm-r1: A stable and generaliz- able r1-style large vision-language model.arXiv preprint arXiv:2504.07615, 2025

    Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-r1: A stable and generaliz- able r1-style large vision-language model.arXiv preprint arXiv:2504.07615, 2025. 2, 3, 6, 8

    Pith/arXiv arXiv 2025

  36. [36]

    Patch-as-decodable-token: Towards uni- fied multi-modal vision tasks in mllms.arXiv preprint arXiv:2510.01954, 2025

    Yongyi Su, Haojie Zhang, Shijie Li, Nanqing Liu, Jingyi Liao, Junyi Pan, Yuan Liu, Xiaofen Xing, Chong Sun, Chen Li, et al. Patch-as-decodable-token: Towards uni- fied multi-modal vision tasks in mllms.arXiv preprint arXiv:2510.01954, 2025. 6

    arXiv 2025

  37. [37]

    Glm-4.5v and glm-4.1v-thinking: Towards versatile multi- modal reasoning with scalable reinforcement learning, 2025

    V Team, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang, Yan Wang, Yean Cheng, Zehai He, Zhe Su, Zhen Yang, Ziyang Pan, Aohan Zeng, Baoxu Wang, Bin Chen, Boyan Shi, Changyu Pang, Chenhui Zhang, Da Yin, Fan Yang, Guoqing Chen, Jiazheng Xu, Jiale Zhu, Jiali Chen, J...

    2025

  38. [38]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024. 2, 3, 6

    Pith/arXiv arXiv 2024

  39. [39]

    Cogvlm: Visual expert for pretrained language models.Advances in Neural Information Processing Sys- tems, 37:121475–121499, 2024

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiX- uan, et al. Cogvlm: Visual expert for pretrained language models.Advances in Neural Information Processing Sys- tems, 37:121475–121499, 2024. 6

    2024

  40. [40]

    Internvl3

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025. 6

    Pith/arXiv arXiv 2025

  41. [41]

    General object foundation model for images and videos at scale

    Junfeng Wu, Yi Jiang, Qihao Liu, Zehuan Yuan, Xiang Bai, and Song Bai. General object foundation model for images and videos at scale. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 3783–3795, 2024. 6

    2024

  42. [42]

    Deepseek-vl2: Mixture-of- experts vision-language models for advanced multimodal understanding.arXiv preprint arXiv:2412.10302, 2024

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-vl2: Mixture-of- experts vision-language models for advanced multimodal understanding.arXiv preprint arXiv:2412.10302, 2024. 3

    Pith/arXiv arXiv 2024

  43. [43]

    Mimo: Unlocking the reasoning potential of language model–from pretraining to posttraining.arXiv preprint arXiv:2505.07608, 2025

    LLM Xiaomi, Bingquan Xia, Bowen Shen, Dawei Zhu, Di Zhang, Gang Wang, Hailin Zhang, Huaqiu Liu, Jiebao Xiao, Jinhao Dong, et al. Mimo: Unlocking the reasoning potential of language model–from pretraining to posttraining.arXiv preprint arXiv:2505.07608, 2025. 6

    Pith/arXiv arXiv 2025

  44. [44]

    FG-CLIP: Fine-grained visual and textual alignment

    Chunyu Xie, Bin Wang, Fanjing Kong, Jincheng Li, Dawei Liang, Gengshen Zhang, Dawei Leng, and Yuhui Yin. FG-CLIP: Fine-grained visual and textual alignment. In Forty-second International Conference on Machine Learn- ing, 2025. 2, 3

    2025

  45. [45]

    Perception-r1: Pioneering perception policy with reinforcement learning.arXiv preprint arXiv:2504.07954,

    En Yu, Kangheng Lin, Liang Zhao, Jisheng Yin, Yana Wei, Yuang Peng, Haoran Wei, Jianjian Sun, Chunrui Han, Zheng Ge, et al. Perception-r1: Pioneering perception policy with reinforcement learning.arXiv preprint arXiv:2504.07954,

    Pith/arXiv arXiv

  46. [46]

    Modeling context in referring expres- sions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expres- sions. InEuropean conference on computer vision, pages 69–85. Springer, 2016. 7, 8, 9

    2016

  47. [47]

    Modeling context in referring expres- sions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expres- sions. InComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69–85. Springer, 2016. 2, 4

    2016

  48. [48]

    Revisiting counterfactual prob- lems in referring expression comprehension

    Zhihan Yu and Ruifan Li. Revisiting counterfactual prob- lems in referring expression comprehension. InProceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 13438–13448, 2024. 2

    2024

  49. [49]

    Referring expression comprehension with semantic visual relationship and word mapping

    Chao Zhang, Weiming Li, Wanli Ouyang, Qiang Wang, Woo-Shik Kim, and Sunghoon Hong. Referring expression comprehension with semantic visual relationship and word mapping. InProceedings of the 27th ACM International Conference on Multimedia, pages 1258–1266, 2019. 2

    2019

  50. [50]

    Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, et al. Mm1. 5: Methods, analysis & insights from multimodal llm fine-tuning.arXiv preprint arXiv:2409.20566, 2024. 2, 3

    Pith/arXiv arXiv 2024

  51. [51]

    Towards unifying reference expression generation and comprehension.arXiv preprint arXiv:2210.13076, 2022

    Duo Zheng, Tao Kong, Ya Jing, Jiaan Wang, and Xiaojie Wang. Towards unifying reference expression generation and comprehension.arXiv preprint arXiv:2210.13076, 2022. 2

    Pith/arXiv arXiv 2022

  52. [52]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shen- glong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025. 6 RefBench-PRO: Perceptual and Reasoning Oriented Benchmark for Referring Expression Comprehensio...

    Pith/arXiv arXiv 2025

  53. [53]

    Detailed Definition for Each Subcategory of RefBench-PRO Attribute. The Attribute category focuses on the intrinsic and directly observable visual properties of objects, which includes characteristics such as an object’s color, texture, material, shape, and state. Position. The Position sub-category captures the spatial re- lationships between different o...

  54. [54]

    We filtered the images based on their resolution, retaining only those within the range of 1024x1024 to 2048x2048 pixels

    Details of RefBench-PRO Construction Data ProcessingWe source images from the FineHARD dataset, a large-scale, high-quality dataset comprising 12 million images based on the GRiT dataset. We filtered the images based on their resolution, retaining only those within the range of 1024x1024 to 2048x2048 pixels. This criterion was established to ensure a cons...

  55. [55]

    The improve- ments are most pronounced in Compositional Reasoning, especially within the Relation sub-category

    More Results of Ref-R1 As shown in Table 8, Ref-R1 demonstrates comprehensive performance gains across all sub-categories. The improve- ments are most pronounced in Compositional Reasoning, especially within the Relation sub-category. Furthermore, the model achieves a significant uplift in the Reject task, showcasing a more robust ability to handle invali...

  56. [56]

    These fig- ures provide a visual comparison of several representative models across challenging examples from all six categories

    Case Study More cases are shown in Figure 9 and Figure 10. These fig- ures provide a visual comparison of several representative models across challenging examples from all six categories. Table 8. Detailed results of ablation experiments of Ref-R1 on RefBench-PRO. Models Overall Visual-cue Perception Compositional Reasoning Accp Acco Attribute Position I...

  57. [57]

    Assign a clear and concise class label to each object (e.g., 'person', 'laptop', 'coffee mug', 'dog')

    Comprehensive Object Identification: Your first step is to perform a thorough scan of the entire image to identify every distinct, tangible object. Assign a clear and concise class label to each object (e.g., 'person', 'laptop', 'coffee mug', 'dog'). Do not omit objects, even if they are small or in the background. Focus strictly on concrete items and ign...

  58. [58]

    Each instance requires its own property dictionary

    Handling Multiple Instances: If you identify multiple objects of the same class (e.g., two chairs, three books), you MUST treat each one as a separate and unique entry in the output list. Each instance requires its own property dictionary

  59. [59]

    Populate this object with all observable visual properties

    Detailed Inherent Attribute Extraction: For each identified object, you must create a nested attributes object. Populate this object with all observable visual properties. Be as descriptive as possible. Key attributes to consider include, but are not limited to: color: (e.g., 'blue', 'pale yellow', 'multi-colored'). material: (e.g., 'polished wood', 'brus...

  60. [60]

    to the left of the monitor

    Interactive Relationship Mapping: After describing an object's inherent attributes, identify its spatial connections to other identified objects. If such relationships exist, add an interactive_relationships field. This field must be a list of strings. Each string must describe a relationship from the perspective of the current object. Examples: ["to the ...

  61. [61]

    a white ceramic mug filled with dark coffee

    Brief Descriptive Phrase Generation: For every object, you must provide a description phrase. This should be a concise, natural-language summary that captures the essence of the object within its specific context in the image. Examples: For a cup, "a white ceramic mug filled with dark coffee". For a person, "a woman with brown hair smiling at the camera"

  62. [62]

    scene_properties

    Strict JSON Output Format: Your final and only output MUST be a single, valid JSON object. The root of the object must have one key: "scene_properties". The value of "scene_properties" must be a list (JSON array) containing the property dictionaries for every visual element you detected. Each dictionary in the list must contain: "class": The object's cate...