Pith. sign in

REVIEW 4 major objections 6 minor 76 references

NAVER: A Neuro-Symbolic Compositional Automaton for Visual Grounding with Explicit Logic Reasoning

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read NAVER claims that composing perception, explicit probabilistic logic, and a self-correcting verification loop outperforms both monolithic and compositional visual grounding pipelines.

desk verdict A solid compositional visual grounding system whose headline SoTA gain is partly inherited from the detector, and which needs one missing ablation before the logic contribution is proven. read the letter →

arxiv 2502.00372 v3 pith:FYTZ3TEJ submitted 2025-02-01 cs.CV

Zhixi Cai , Fucai Ke , Simindokht Jahangard , Maria Garcia de la Banda , Reza Haffari , Peter J. Stuckey , Hamid Rezatofighi This is my paper · ORCID
classification cs.CV
keywords visualgroundingreferringexpressiondetectionsegmentationneuro-symbolicreasoningprobabilisticlogicProbLogfinite-stateautomatonself-correction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

NAVER's claim is that the hard part of visual grounding is not perception alone but the composition of attributes and relations, and that this composition is best done in explicit probabilistic logic rather than in the implicit weights of a monolithic model. The paper builds a five-state deterministic finite-state automaton that moves from perceiving entities, to generating ProbLog code from the query, to running probabilistic inference over candidate targets, to verifying the top candidate with a vision-language model, with transitions that loop back to retry a failed stage. The payoff is interpretability and robustness: each stage can be inspected and corrected, and the full chain is validated before a target is returned. On four public benchmarks the paper reports top scores, including 96.2% on RefCOCO test A and 91.6% on RefCOCOg test, with a 0.3% runtime failure rate.

What carries the argument

The central object is a deterministic finite-state automaton (DFA) with five states—Perception, Logic Generation, Logic Reasoning, Answering, and Return Target—whose transition function is driven by intermediate results such as "no entity found", "code generation error", "logic returned zero candidates", and "answerer said no". On top of it sits ProbLog, a probabilistic logic language in which each detected entity, attribute, and relation becomes a fact with a probability, and the query becomes a rule; the generated code is executed by a probabilistic logic engine to rank candidates. The automaton is what makes the pipeline self-correcting: a failed code generation retries in place, an empty logic result sends the system back to logic generation, and a rejected top candidate triggers either the next candidate or a return to perception, with retries capped to avoid infinite loops.

What would settle it

Replace NAVER's probabilistic logic stage with a learned ranker over the same detector, attribute, and relation scores, holding the automaton and answerer fixed; if the ranker matches or beats NAVER's accuracy on RefCOCOg and Ref-Adv, then the explicit logic is not the source of the reported gains.

Watch

Extended reading notes

Core claim

The paper's central claim is that a modular neuro-symbolic pipeline with explicit probabilistic reasoning can beat both monolithic end-to-end models and earlier compositional code-generation pipelines on referring-expression detection and segmentation. The mechanism is the automaton: NAVER renders the query as a ProbLog rule over perceived entities, attributes, and relations, executes the rule with a probabilistic logic engine to get a ranked list of target candidates, and then asks a vision-language answerer to confirm or reject the top candidate, falling back to the next candidate or back to perception when verification fails. The paper reports accuracy of 96.2% on RefCOCO test A, 92.8% on RefCOCO+ test A, 91.6% on RefCOCOg test, and 75.4% on Ref-Adv for detection, and IoU of 76.0 on RefCOCOg test for segmentation, surpassing the compared baselines under matched foundation models. The self-correction loop is what the authors credit for much of the gain: with logic and answerer alone but no automaton feedback, RefCOCO accuracy drops from 96.2% to 78.1%.

Load-bearing premise

The pipeline assumes the upstream object detector will propose the actual object named by the query among its candidates with a decent confidence score; if it misses that object or ranks it too low, the logic and verification stages cannot recover it.

Editorial extensions

If this is right

  • If NAVER's numbers hold, compositional pipelines can beat specialized end-to-end grounding models without training on grounding data, by wrapping an off-the-shelf detector, an LLM, and a VLM in a verification loop.
  • Explicit probabilistic logic turns the system's reasoning into inspectable facts and rules, so failures can be traced to a specific stage and corrected there rather than propagating to the final answer.
  • The full-chain validation makes the pipeline dramatically more reliable in practice: the paper reports a 0.3% runtime failure rate, versus 10.3% for the nearest compositional baseline and 70.2% for a code-generation baseline under the same foundation models.
  • Because the automaton is modular, swapping in a stronger grounding model directly lifts accuracy (from 70.1% to 96.2% on RefCOCO test A in the paper's tables), so the design can absorb future improvements to foundation models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same automaton pattern should transfer to other query-driven multimodal tasks, such as spatial question answering, video grounding, or embodied instruction following, provided a perception stage can emit candidate entities and relations as probabilistic facts; the paper only demonstrates it on referring expressions.
  • The authors' remark that LLMs are better at verifying an answer than producing one suggests a testable design principle: a cheap verifier wrapped around a weaker generator may beat a stronger generator alone, and this could be measured as accuracy per inference dollar on the same benchmarks.
  • Because the paper's failure rate is dominated by an external content-policy refusal on innocuous queries, a practical extension would be to add a fallback prompt or a second answerer model when the primary VLM refuses to answer, which the automaton's retry structure already accommodates.
  • A fair comparison that holds the candidate set fixed across methods, rather than letting each method choose its own detector, would reveal how much of NAVER's gain comes from logic and verification versus raw detector quality; the paper's own ablation table suggests the logic and answerer stages contribute large independent gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes NAVER, a compositional visual grounding method that combines a foundation-model entity detector, an LLM that generates explicit ProbLog rules from the query, probabilistic logic inference (implemented via Scallop) over detected entities and their attributes/relations, and a VLM-based answerer that verifies the top candidate. These components are orchestrated by a deterministic finite-state automaton with a self-correction mechanism that can revisit earlier states up to a fixed number of retries. The method is evaluated on referring expression detection (RefCOCO, RefCOCO+, RefCOCOg, Ref-Adv) and referring expression segmentation (RefCOCOg, RefCLEF), reporting state-of-the-art accuracy/IoU numbers as well as ablations over DFA, logic, and answerer components, detector choice, LLM choice, and VLM choice.

Significance. If the reported results are taken at face value, the paper makes a useful contribution to modular neuro-symbolic visual grounding: it provides an interpretable pipeline with explicit probabilistic logic, a self-correction mechanism, a low runtime failure rate, and consistent same-configuration improvements over the HYDRA and ViperGPT compositional baselines (e.g., Table 2: NAVER† 70.1 vs. HYDRA† 60.9 and ViperGPT† 62.6 on RefCOCO). The paper also ships code, reports detailed per-component ablations, and includes a useful self-correction distribution analysis in the supplementary material. However, the headline SoTA claim is currently conditional: the largest numbers are obtained by wrapping benchmark-trained detectors (Florence2-L, GLaMM), the marginal gains over those detectors are small, and the specific contribution of explicit logic reasoning in the full system is not isolated by the ablation table. The central empirical claim is therefore defensible but incomplete until these issues are addressed.

major comments (4)
  1. [Section 4.4, Table 6] The ablation matrix omits the configuration with DFA and Answerer enabled but Logic disabled (✔, ✘, ✔). This is exactly the configuration needed to isolate the contribution of explicit logic reasoning in the full system. Since the Answerer is a VLM that receives a highlighted candidate and the query, and since Table 7 shows that NAVER's accuracy closely tracks the wrapped detector, the gap between rows 6 and 7 (76.2→96.2 on RefCOCO) currently conflates the effect of logic with the effect of the Answerer. Please add this row (and the analogous rows for RefCOCO+, RefCOCOg, and Ref-Adv) and discuss how much of the final gain remains when logic is removed.
  2. [Section 4.3, Tables 2 and 7] The headline SoTA numbers (NAVER*) rely on Florence2-L for detection and GLaMM for segmentation, both of which are trained on the evaluation datasets, as the Table 2 note acknowledges. The residual gain over Florence2-L is small (e.g., +1.1 on RefCOCO test A, +0.3 on RefCOCO+ test A, +0.7 on RefCOCOg test), and Table 7 reveals that NAVER's final accuracy largely tracks the detector it wraps (e.g., GLIP-L 55.0→70.1, Florence2-L 95.1→96.2). As written, the claim that 'NAVER achieves SoTA performance' is misleading because it is dominated by the choice of a benchmark-trained detector rather than by the proposed logic automaton. Please reframe the central claim to emphasize the same-configuration comparisons, and consider reporting a primary configuration whose detector is not trained on the evaluation datasets, or explicitly separating the detector's contribution in the headline numbers.
  3. [All experimental tables] All reported metrics are single point estimates with no error bars, despite the pipeline containing stochastic LLM and VLM calls (GPT-4o Mini, InternVL2, and the various recognizers). Several headline margins over Florence2-L are only 0.1–1.1 accuracy points, which is within typical run-to-run variability for such stochastic pipelines. Please report means and standard deviations over at least three independent runs (or bootstrap confidence intervals) for the main tables and for the ablation study in Table 6.
  4. [Supplementary Section 9] The maximum retry count max_retries=6 is selected from the self-correction behavior observed on the RefCOCO test A set, where the paper states that 'this threshold effectively addresses the majority of the errors.' Tuning a termination threshold on the test set is a form of test-set adaptation and can inflate reported accuracy. Please select this hyperparameter on a validation split and report sensitivity to its value (e.g., 3, 6, 9) in the main text or supplementary material.
minor comments (6)
  1. [Throughout] The name 'NA VER' appears with a space in many places (including the title, figures, and section headings); please use the consistent spelling 'NAVER' throughout.
  2. [Table 2 footnote] The note 'The methods† and methods* use the same VFMs (GLIP-L, BLIP) and same VFMs (Florence2-L, InternVL2), respectively' is confusing; please rephrase to something like 'Methods with † use GLIP-L and BLIP; methods with * use Florence2-L and InternVL2; both groups use GPT-4o Mini as the LLM.'
  3. [Section 3.2 and Table 1] The text says that when |E|=0 a fallback result from the entity detector is used, but Transition 2 in Table 1 sends |E| < 2 to the Answering state; please clarify how the fallback path coexists with the automaton transition for both |E|=0 and |E|=1.
  4. [Section 3.3, Code 3.1] The ProbLog facts assign probabilities to attributes and relations (e.g., 0.8711::attribute(person_0,'wearing_blue_shirt')), but the paper does not specify how these probabilities are computed from the VLM or geometric outputs. Please add a paragraph describing the probability calibration procedure for attribute recognizers and the symbolic geometry/depth-based relation recognizers.
  5. [Section 4.3, Table 3] The implementation details state that GLaMM is used as the entity detector for segmentation, but the table and text do not explicitly say that the NAVER* segmentation row wraps GLaMM; please make this explicit so that the reader can interpret the +1.1 gain over GLaMM's own 74.9 test IoU correctly.
  6. [Section 3.4] The deterministic finite-state automaton is described as deterministic because the transition conditions are mutually exclusive, yet the actual transition outcomes depend on stochastic LLM and VLM outputs. Please clarify that the automaton is deterministic given the observed outputs of the perception and reasoning modules, not that the modules themselves are deterministic.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: NAVER's SoTA claim is an empirical benchmark result, and no prediction reduces by construction to its inputs.

full rationale

The central claim of the paper is an empirical state-of-the-art benchmark result, not a derivation, and the system is evaluated against external datasets (RefCOCO, RefCOCO+, RefCOCOg, Ref-Adv) and a wide set of independent baselines. No equation in the paper defines a prediction in terms of the measured outcome: the ProbLog facts, query rules, and VLM verifier are separate modules, and accuracy is computed against human ground-truth annotations. The only test-set-derived choice, max_retries=6, is a termination cap for the automaton's self-correction loop and is not used to construct the reported accuracies; it is an implementation/robustness parameter, not a fitted output passed off as prediction. The paper's citation of its own HYDRA baseline is not load-bearing: HYDRA is an externally published ECCV 2024 method and the comparison also includes many independent end-to-end and compositional systems, so the SoTA claim does not reduce to a self-citation. The concern that NAVER's best configuration inherits much of its performance from Florence2-L, which was trained on the same benchmark, is an attribution/fairness limitation rather than circularity, and the absence of a DFA+Answerer-without-Logic ablation is an experimental gap, not a reduction by construction. Accordingly, there are no circular steps to report.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The system is not trained end-to-end; it composes off-the-shelf models, so the ledger consists mainly of modeling assumptions about the reliability and independence of the perception components, plus one test-set-derived hyperparameter. No new entities or physical quantities are introduced.

free parameters (1)
  • max_retries = 6
    Maximum number of self-correction retries before forced termination; chosen from the distribution of retries on RefCOCO test A (Supplementary Section 9), so tuned on the test set.
assumptions (4)
  • domain assumption Detected entity set E from the grounding model contains the true target (or an acceptable fallback) with high enough probability.
    The Perception state forwards only detected entities to logic; if the target is missed, logic cannot recover. Section 3.2 and Table 7 show NAVER inherits detector recall.
  • domain assumption Probabilistic facts (entity, attribute, relation) are independent so their conjunction in a ProbLog rule gives a valid target probability.
    The ProbLog/Scallop query multiplies probabilities; the paper does not justify independence of detector confidence, VLM attribute probabilities, and geometry relations. Section 3.3 Code 3.1.
  • domain assumption LLM-generated ProbLog/Scallop code faithfully represents the natural-language query semantics.
    Logic Generation relies on an LLM to translate Q into a rule; parsing errors trigger retries but correctness is not formally verified. Section 3.3.
  • domain assumption The answerer VLM's next-token Yes/No probability is a valid verification signal.
    Answering state uses the higher of P(Yes), P(No) to accept or reject the top candidate; no calibration is reported. Section 3.5.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NAVER: A Neuro-Symbolic Compositional Automaton for Visual Grounding with Explicit Logic Reasoning." pith.science (2026). https://pith.science/paper/FYTZ3TEJ

@misc{pith2026250200372,
  author       = {Pith},
  title        = {Pith review of: NAVER: A Neuro-Symbolic Compositional Automaton for Visual Grounding with Explicit Logic Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FYTZ3TEJ}},
  note         = {Machine review of arXiv:2502.00372}
}
read the original abstract

Visual Grounding (VG) tasks, such as referring expression detection and segmentation tasks are important for linking visual entities to context, especially in complex reasoning tasks that require detailed query interpretation. This paper explores VG beyond basic perception, highlighting challenges for methods that require reasoning like human cognition. Recent advances in large language methods (LLMs) and Vision-Language methods (VLMs) have improved abilities for visual comprehension, contextual understanding, and reasoning. These methods are mainly split into end-to-end and compositional methods, with the latter offering more flexibility. Compositional approaches that integrate LLMs and foundation models show promising performance but still struggle with complex reasoning with language-based logical representations. To address these limitations, we propose NAVER, a compositional visual grounding method that integrates explicit probabilistic logic reasoning within a finite-state automaton, equipped with a self-correcting mechanism. This design improves robustness and interpretability in inference through explicit logic reasoning. Our results show that NAVER achieves SoTA performance comparing to recent end-to-end and compositional baselines. The code is available at https://github.com/ControlNet/NAVER .

Figures

Figures reproduced from arXiv: 2502.00372 by the authors.

Figure 1
Figure 1. Overview of NAVER. Given the query “A person on the left wearing a blue shirt in front of a car”, NAVER first identifies the entities of interest, their relations (e.g. “Person 1 is left of Person 2”, “Person 1 is in front of Car 1”), and attributes (e.g. “Person 1 is wearing blue shirt”). It then transforms these statements into ProbLog [15] logic expressions, and applies probabilistic reasoning to find the correct… view at source ↗
Figure 2
Figure 2. Pipeline of NAVER. NAVER is organized as a deterministic finite-state automaton (DFA) with five states: (1) Perception, which is the initial state where relevant visual information is extracted by identifying entity categories and localizing entities; (2) Logic Generation, where an LLM uses the caption and query to generate logic expressions in ProbLog, incorporating entities, and ProbLog query. The relations and at… view at source ↗
Figure 3
Figure 3. Distribution of self-correction times in NAVER. The x-axis represents the number of retries for self-correction, while the y-axis shows the proportion of samples requiring self-correction within each group of self-correction times. The annotated values indicate the cumulative proportion of samples resolved by that number of retries. Time # LLM Tokens LLM Cost Method (Seconds) Input Output (USD) ViperGPT [45] 2.02 41… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Accuracy performance of different text query length in NAVER. The x-axis represents the length of text query, while the y-axis shows the accuracy of NAVER and baselines within each group of query length. Captioner ECE RefCOCO RefCOCO+ RefCOCOg RefAdv ✗ VLM 91.6 87.5 80…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 51 canonical work pages

  1. [1]

    Words Aren’t Enough, Their Or- der Matters: On the Robustness of Grounding Visual Refer- ring Expressions

    Arjun Akula, Spandana Gella, Yaser Al-Onaizan, Song- Chun Zhu, and Siva Reddy. Words Aren’t Enough, Their Or- der Matters: On the Robustness of Grounding Visual Refer- ring Expressions. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics , pages 6555–6565, Online, 2020. Association for Computational Linguistics. 6, 1, 2

    work page 2020

  2. [2]

    Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sa- hand Sharifzadeh, Mikołaj Bi ´nkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sa- hand Sharifzadeh, Mikołaj B...

    work page 2022

  3. [3]

    Neuro-Symbolic Visual Reasoning: Disentangling

    Saeed Amizadeh, Hamid Palangi, Alex Polozov, Yichen Huang, and Kazuhito Koishida. Neuro-Symbolic Visual Reasoning: Disentangling. In Proceedings of the 37th Inter- national Conference on Machine Learning , pages 279–290. PMLR, 2020. ISSN: 2640-3498. 1

    work page 2020

  4. [4]

    Qwen2.5-VL Technical Report, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhao- hai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VL Technical Report,...

    arXiv 2025

  5. [5]

    Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Her- nandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran- Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse...

    arXiv 2022

  6. [6]

    Language Models are Few-Shot Learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, S...

    work page 1901

  7. [7]

    SpatialVLM: Endow- ing Vision-Language Models with Spatial Reasoning Capa- bilities

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. SpatialVLM: Endow- ing Vision-Language Models with Spatial Reasoning Capa- bilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14455– 14465, 2024. 2

    work page 2024

  8. [8]

    UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathemat- ical Expression

    Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, and Xiaodan Liang. UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathemat- ical Expression. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 3313–3323, Abu Dhabi, United Arab Emirates, 2022. Asso- ciation for Computational Li...

    work page 2022

Show all 76 references
  1. [9]

    MiniGPT-v2: large language model as a uni- fied interface for vision-language multi-task learning, 2023

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elho- seiny. MiniGPT-v2: large language model as a uni- fied interface for vision-language multi-task learning, 2023. arXiv:2310.0947...

    2023 arXiv

  2. [10]

    InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks. In Proceed- ...

    2024

  3. [11]

    YOLO-World: Real-Time Open-V ocabulary Object Detection

    Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xing- gang Wang, and Ying Shan. YOLO-World: Real-Time Open-V ocabulary Object Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16901–16911, 2024. 3, 6, 2

    2024

  4. [12]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vi...

    2023

  5. [13]

    SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion

    Ming Dai, Lingfeng Yang, Yihao Xu, Zhenhua Feng, and Wankou Yang. SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion. In Ad- vances in Neural Information Processing Systems , pages 121670–121698, 2024. 2, 6, 8

    2024

  6. [14]

    InstructBLIP: Towards General- purpose Vision-Language Models with Instruction Tuning,

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards General- purpose Vision-Language Models with Instruction Tuning,

  7. [15]

    ProbLog: a probabilistic prolog and its application in link discovery

    Luc De Raedt, Angelika Kimmig, and Hannu Toivonen. ProbLog: a probabilistic prolog and its application in link discovery. In Proceedings of the 20th international joint conference on Artifical intelligence , pages 2468–2473, San Francisco, CA, USA, 2007. Morgan Kaufmann Publis...

    2007

  8. [16]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025

    DeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025. arXiv:2501.12948 [cs]. 8

    2025 arXiv

  9. [17]

    Vision-Language Transformer and Query Generation for Re- ferring Segmentation

    Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vision-Language Transformer and Query Generation for Re- ferring Segmentation. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision , pages 16321– 16330, 2021. 7

    2021

  10. [18]

    The Llama 3 Herd of Models, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, and et al. The Llama 3 Herd of Models, 2024. 3, 8

    2024

  11. [19]

    Visual Program- ming: Compositional Visual Reasoning Without Training

    Tanmay Gupta and Aniruddha Kembhavi. Visual Program- ming: Compositional Visual Reasoning Without Training. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 14953–14962, 2023. 2, 3

    2023

  12. [20]

    Video OWL-ViT: Temporally-consistent Open-world Localization in Video

    Georg Heigold, Matthias Minderer, Alexey Gritsenko, Alex Bewley, Daniel Keysers, Mario Lu ˇci´c, Fisher Yu, and Thomas Kipf. Video OWL-ViT: Temporally-consistent Open-world Localization in Video. In Proceedings of the IEEE/CVF International Conference on Computer Vision , page...

    2023

  13. [21]

    ReferItGame: Referring to Objects in Pho- tographs of Natural Scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. ReferItGame: Referring to Objects in Pho- tographs of Natural Scenes. In Proceedings of the 2014 Con- ference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 787–798, Doha, Qatar, 2014. Assoc...

    2014

  14. [22]

    HYDRA: A Hyper Agent for Dynamic Compositional Visual Reasoning

    Fucai Ke, Zhixi Cai, Simindokht Jahangard, Weiqing Wang, Pari Delir Haghighi, and Hamid Rezatofighi. HYDRA: A Hyper Agent for Dynamic Compositional Visual Reasoning. In Computer Vision – ECCV 2024 , pages 132–149, Cham,

    2024

  15. [23]

    Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment Anything, 2023. arXiv:2304.02643 [cs]. 6, 7, 8

    2023 arXiv

  16. [24]

    LISA: Reasoning Seg- mentation via Large Language Model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. LISA: Reasoning Seg- mentation via Large Language Model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579–9589, 2024. 3, 6, 7

    2024

  17. [25]

    Otter: A Multi-Modal Model with In-Context Instruction Tuning, 2023

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A Multi-Modal Model with In-Context Instruction Tuning, 2023. arXiv:2305.03726 [cs]. 2, 3

    2023 arXiv

  18. [26]

    BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. In In- ternational Conference on Machine Learning, 2023. Version Number: 3. 1, 2

    2023

  19. [27]

    Grounded Language-Image Pre-Training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jian- wei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded Language-Image Pre-Training. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion a...

    2022

  20. [28]

    Scallop: A Lan- guage for Neurosymbolic Programming

    Ziyang Li, Jiani Huang, and Mayur Naik. Scallop: A Lan- guage for Neurosymbolic Programming. Proceedings of the ACM on Programming Languages , 7(PLDI):166:1463– 166:1487, 2023. 6

    2023

  21. [29]

    GRES: Gen- eralized Referring Expression Segmentation

    Chang Liu, Henghui Ding, and Xudong Jiang. GRES: Gen- eralized Referring Expression Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 23592–23601, 2023. 2, 7

    2023

  22. [30]

    Improved Baselines with Visual Instruction Tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved Baselines with Visual Instruction Tuning, 2023. arXiv:2310.03744 [cs]. 1, 2

    2023 arXiv

  23. [31]

    Visual Instruction Tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning, 2023. arXiv:2304.08485 [cs]. 3

    2023 arXiv

  24. [32]

    Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection,

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection,

  25. [33]

    Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models

    Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models. In Advances in Neural Information Processing Systems, 2023. 2, 3

    2023

  26. [34]

    Multi-Task Collabora- tive Network for Joint Referring Expression Comprehension and Segmentation

    Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Liujuan Cao, Chenglin Wu, Cheng Deng, and Rongrong Ji. Multi-Task Collabora- tive Network for Joint Referring Expression Comprehension and Segmentation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, p...

    2020

  27. [35]

    2, 3, 6, 7, 8

    arXiv:2303.05499 [cs]. 2, 3, 6, 7, 8

    arXiv

  28. [36]

    GPT-4o System Card, 2024

    OpenAI. GPT-4o System Card, 2024. 3, 6, 7, 8

    2024

  29. [37]

    Christiano, Jan Leike, and Ryan Lowe

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike...

    2022

  30. [38]

    Scaling Open-V ocabulary Object Detection

    Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling Open-V ocabulary Object Detection. In Advances in Neural Information Processing Systems , pages 72983– 73007. Curran Associates, Inc., 2023. 3

    2023

  31. [39]

    Kosmos-2: Ground- ing Multimodal Large Language Models to the World, 2023

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Ground- ing Multimodal Large Language Models to the World, 2023. arXiv:2306.14824 [cs]. 2, 3, 6

    2023 arXiv

  32. [40]

    PerceptionGPT: Effectively Fusing Visual Percep- tion into LLM

    Renjie Pi, Lewei Yao, Jiahui Gao, Jipeng Zhang, and Tong Zhang. PerceptionGPT: Effectively Fusing Visual Percep- tion into LLM. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27124– 27133, 2024. 3, 7

    2024

  33. [41]

    PyTorch: An Imperative Style, High-Performance Deep Learning Library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Rai- son, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, L...

    2019

  34. [42]

    Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdel- rahman Shaker, Salman Khan, Hisham Cholakkal, Rao M. Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S. Khan. GLaMM: Pixel Grounding Large Multimodal Model. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Patte...

    2024

  35. [43]

    HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face,

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face,

  36. [44]

    Language models are unsuper- vised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsuper- vised multitask learners. OpenAI blog, 1(8):9, 2019. 2

    2019

  37. [45]

    ViperGPT: Visual Inference via Python Execution for Reasoning, 2023

    Dídac Surís, Sachit Menon, and Carl V ondrick. ViperGPT: Visual Inference via Python Execution for Reasoning, 2023. arXiv:2303.08128 [cs]. 2, 3, 4, 6, 7, 1

    2023 arXiv

  38. [46]

    Gemma 2: Improving Open Language Mod- els at a Practical Size, 2024

    Gemma Team. Gemma 2: Improving Open Language Mod- els at a Practical Size, 2024. arXiv:2408.00118 [cs]. 8

    2024 arXiv

  39. [47]

    arXiv:2303.17580 [cs]. 4

    arXiv

  40. [48]

    Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as Programmers

    Aleksandar Stani ´c, Sergi Caelles, and Michael Tschannen. Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as Programmers. Transactions on Machine Learning Research, 2024. 2, 3, 6

    2024

  41. [49]

    CRIS: CLIP- Driven Referring Image Segmentation

    Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. CRIS: CLIP- Driven Referring Image Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11686–11695, 2022. 7

    2022

  42. [50]

    Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models, 2023

    Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models, 2023. arXiv:2303.04671 [cs]. 2

    2023 arXiv

  43. [51]

    Will we run out of data? Limits of LLM scaling based on human- generated data, 2024

    Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Be- siroglu, Lennart Heim, and Marius Hobbhahn. Will we run out of data? Limits of LLM scaling based on human- generated data, 2024. arXiv:2211.04325. 3

    2024 arXiv

  44. [52]

    VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks,

    Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, and Jifeng Dai. VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks,

  45. [53]

    arXiv:2305.11175 [cs]. 3

    arXiv

  46. [54]

    Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S. Ryoo, Shrikant Kendre, Jieyu Zhang, Can Qin, Shu Zhang, Chia-Chih Chen, Ning Yu, Juntao Tan, Tulika Manoj Awalgaonkar, Shelby Heinecke, Huan Wang, Yejin ...

    2024

  47. [55]

    Set-of-Mark Prompting Un- leashes Extraordinary Visual Grounding in GPT-4V, 2023

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chun- yuan Li, and Jianfeng Gao. Set-of-Mark Prompting Un- leashes Extraordinary Visual Grounding in GPT-4V, 2023. arXiv:2310.11441 [cs]. 5

    2023 arXiv

  48. [56]

    NExT-GPT: Any-to-Any Multimodal LLM, 2023

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. NExT-GPT: Any-to-Any Multimodal LLM, 2023. arXiv:2309.05519 [cs]. 2, 3

    2023 arXiv

  49. [57]

    GSV A: Generalized Segmentation via Multimodal Large Language Models

    Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. GSV A: Generalized Segmentation via Multimodal Large Language Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3858–3869, 2024. 3, 7

    2024

  50. [58]

    Florence-2: Advancing a Unified Representation for a Va- riety of Vision Tasks

    Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a Unified Representation for a Va- riety of Vision Tasks. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages ...

    2024

  51. [59]

    IdealGPT: Iteratively Decomposing Vision and Lan- guage Reasoning via Large Language Models

    Haoxuan You, Rui Sun, Zhecan Wang, Long Chen, Gengyu Wang, Hammad Ayyubi, Kai-Wei Chang, and Shih-Fu Chang. IdealGPT: Iteratively Decomposing Vision and Lan- guage Reasoning via Large Language Models. In The 2023 Conference on Empirical Methods in Natural Language Pro- cessing...

    2023

  52. [60]

    Berg, and Tamara L

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling Context in Referring Expres- sions. In Proceedings of the European Conference on Com- puter Vision (ECCV) , pages 69–85, Cham, 2016. Springer International Publishing. 1

    2016

  53. [61]

    Depth Any- thing V2, 2024

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiao- gang Xu, Jiashi Feng, and Hengshuang Zhao. Depth Any- thing V2, 2024. arXiv:2406.09414. 6

    2024 arXiv

  54. [62]

    Cross-Modal Rela- tionship Inference for Grounding Referring Expressions

    Sibei Yang, Guanbin Li, and Yizhou Yu. Cross-Modal Rela- tionship Inference for Grounding Referring Expressions. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 4145–4154, 2019. 1

    2019

  55. [63]

    Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Heng- shuang Zhao, and Philip H. S. Torr. LA VT: Language-Aware Vision Transformer for Referring Image Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 18155–18165, 2022. 7

    2022

  56. [66]

    Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts

    Yan Zeng, Xinsong Zhang, and Hang Li. Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts. In Proceedings of the 39th International Con- ference on Machine Learning, pages 25994–26009. PMLR,

  57. [67]

    PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model, 2024

    Zheng Zhang, Yeyao Ma, Enming Zhang, and Xiang Bai. PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model, 2024. arXiv:2403.14598. 6, 7

    2024 arXiv

  58. [68]

    MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models,

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models,

  59. [69]

    arXiv:2304.10592 [cs]. 2 NA VER: A Neuro-Symbolic Compositional Automaton for Visual Grounding with Explicit Logic Reasoning Supplementary Material This supplementary material provides additional details about the proposed method NA VER. In particular, it pro- vides details ab...

    arXiv

  60. [70]

    Table 9 presents a quantitative comparison on the RefCOCO, RefCOCO+, RefCOCOg [21], and Ref- Adv [1] datasets

    Additional Quantitative Comparison In this section, we report both the accuracy and Intersection over Union (IoU) performance for the referring expression detection task. Table 9 presents a quantitative comparison on the RefCOCO, RefCOCO+, RefCOCOg [21], and Ref- Adv [1] datas...

  61. [71]

    For a fair comparison, we use the same foundation models and LLM for the ex- periments

    Complexity Analysis Efficiency results for NA VER and compositional baselines are provided in Table 10, which includes average runtime, token usage, and cost per sample. For a fair comparison, we use the same foundation models and LLM for the ex- periments. From the results, w...

  62. [72]

    The results are shown in Table 11

    Ablation Studies for VLMs We evaluate the impact of four different Vision-Language Models (VLMs) [10, 26, 30, 54] on the performance of NA VER. The results are shown in Table 11. All tested mod- els achieve comparable results on both RefCOCO and Re- fCOCO+. Among the four VLMs...

  63. [73]

    In this mechanism, each time the system transitions into the self-correction state (indicated by a red arrow in Figure 2), it is counted as one retry

    Self-Correction Analysis We analyze the performance of the self-correction mecha- nism in NA VER, focusing on its ability to address errors during inference. In this mechanism, each time the system transitions into the self-correction state (indicated by a red arrow in Figure ...

  64. [74]

    The results, shown in Figure 4, indicate that NA VER consistently reaches SoTA performance compared to all baselines regardless of query length

    Query Length Analysis We analyze the impact of text query length on perfor- mance, comparing NA VER with baselines across different query lengths. The results, shown in Figure 4, indicate that NA VER consistently reaches SoTA performance compared to all baselines regardless of...

  65. [75]

    An intuitive alternative is to skip captioning and ask the VLM to predict categories directly

    Ablations for Captioner In the perception state, NA VER first converts the image into a rich caption and then lets an LLM-based entity-category extractor (ECE) decide which object classes are relevant to the query. An intuitive alternative is to skip captioning and ask the VLM...

  66. [76]

    Yes” or “No

    LLM and VLM Prompts NA VER employs LLMs and VLMs in five different roles as described in section 3: as a caption generator VLM in the Perception state, an entity extractor LLM in the Perception state, a logic query generator LLM in the Logic Generation state, a relation recogn...

  67. [2023]

    arXiv:2305.06500 [cs]. 3

    arXiv

  68. [2024]

    2, 3, 4, 6, 7, 1

    Springer Nature Switzerland. 2, 3, 4, 6, 7, 1

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.