Pith. sign in

REVIEW 4 major objections 3 minor 88 references

FLORA: Formal Language Model Enables Robust Training-free Zero-shot Object Referring Analysis

T0 review · 4 major / 3 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FLORA claims that decomposing a referring phrase into four scored slots and multiplying the probabilities beats feeding the full phrase to a pretrained grounding detector.

desk verdict Clean ablation and large reported gains, but the zero-shot claim rests on an unverified GDINO checkpoint that likely saw RefCOCO. read the letter →

arxiv 2501.09887 v1 pith:JNGZIENQ submitted 2025-01-17 cs.CV

Zhe Chen , Zijing Chen This is my paper · ORCID
classification cs.CV
keywords referringexpressioncomprehensionzero-shotobjectdetectionsegmentationlargelanguagemodelsformalmodelBayesianinferencegroundingtraining-free
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that zero-shot referring expression comprehension and segmentation can be made dramatically better, without any training, by having a large language model translate the referring expression into four structured components and then combining off-the-shelf model scores as a product. The load-bearing comparison is on RefCOCO val, where FLORA raises Grounding DINO from 50.4 to 73.7 accuracy, and up to roughly 45 percent relative improvement across tasks. The author argues the gain comes from decomposing language into object type, location, visual pattern, and relation, each scored by an appropriate interpreter, and then multiplying those probabilities under a Bayesian independence assumption. This matters because ORA training data is expensive, and the pipeline turns two generic foundation models (an LLM and a grounding detector) into a competitive zero-shot reasoner.

What carries the argument

The Formal Language Model (FLM) is the central object: a grammar, defined by an object alphabet and object syntax, that requires any valid description to start with '#' and list object type, spatial location, visual pattern, and relation to surrounding objects as separate slots. The machinery then runs CoP-FLM prompting to make an LLM, such as Llama, emit text conforming to this grammar, parses the slots, and applies Bayes' rule with an independence assumption and uniform priors, reducing the matching problem to the product in Eq. 13. Each factor is supplied by an off-the-shelf model: a visual grounding detector for type, a position-and-size relevance function for location, and a vision-language model such as CLIP for visual patterns and relations. The product is what carries the argument: it lets one weak or hallucinated slot be overruled by the other three.

What would settle it

On a random sample of RefCOCO val, run Grounding DINO with only the parsed object type as prompt and check whether the ground-truth box appears anywhere in the candidate set; if the true object is missing for a large share of failures that FLORA gets wrong, the reported gains are capped by detector recall, not by the product score. A second check: on the subset where GDINO already ranks the true box first, verify whether the product ever demotes it below rank 5, which would show that the reranking can also hurt.

Watch

Extended reading notes

Core claim

FLORA's central claim is that the product of four component probabilities, $P(O_T|x)P(O_L|x)P(O_V|x)P(O_R|x)$ from Eq. 13, identifies the referred object more reliably than feeding the full referring phrase to a pretrained grounding detector. The paper argues that the formal language model's grammar, which forces LLM output into hashtag-delimited slots for type, location, visual pattern, and relation, makes the decomposition stable enough for probabilistic scoring. With Grounding DINO as the object type interpreter, a spatial relevance function for location, and CLIP for visual patterns and relations, the product selects the object with the highest combined match. On RefCOCO val the pipeline scores 73.7 versus 50.4 for the grounding detector alone, with similar gains on RefCOCO+ and RefCOCOg, on the Who's Waldo person-link task, and on PhraseCut segmentation.

Load-bearing premise

The pipeline assumes that when the parsed object type is fed to the grounding detector, the detector's candidate list actually contains the true referred object; if the type prompt is mistranslated or the detector fails to recall that object, no later probability multiplication can bring it back.

Editorial extensions

If this is right

  • Zero-shot referring object detection and segmentation can be improved by large margins without any fine-tuning on ORA datasets, lowering the annotation burden for new scenes and domains.
  • The same FLM-plus-product recipe should transfer to other referring tasks, since segmentation is obtained simply by feeding FLORA's top box to SAM.
  • Even a weak detector like OWL, which alone scores in the low single digits on RefCOCO, becomes competitive with early supervised approaches once its outputs are reranked by the product.
  • Because each probability factor is computed independently, a hallucinated or missing relation slot does not necessarily destroy the final prediction, as long as the other slots carry evidence.
  • Formal regulation of LLM outputs, rather than free-form parsing, is what makes the improvement large, as the ablation shows 73.7 with the formal language model versus 59.6 for the same pipeline without it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's reported gains are upper-bounded by the recall of the grounding detector on the parsed object type prompt; images where the true object never appears among the detector's candidates cannot be rescued by reranking, and this failure mode is not analyzed in the paper.
  • The independence assumption behind Eq. 7 is visibly violated whenever attributes are correlated with the category (e.g., 'red car'), which may double-count shared evidence; a test that replaces the product with a learned or normalized combination could show how much this costs.
  • The same decomposition-product pattern is a natural template for other multimodal tasks where a prompt can be split into independently scorable facets, such as text-guided image editing or visual question answering; this is an extension the paper mentions only as future work.
  • Because RefCOCO+ removes location words, the 11.8-point gain there comes almost entirely from visual-pattern and relation interpreters, making that dataset a cleaner test of whether the visual and relation scores, not just the location score, are doing the work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes FLORA, a training-free zero-shot pipeline for object referring analysis (ORA). A prompted LLM, guided by a hand-defined formal language model (FLM), parses a referring expression into four components—object type, spatial location, visual pattern, and relation—and each component is scored by an off-the-shelf interpreter (Grounding DINO for type, a coordinate-based relevance function for location, and CLIP/GDINO ensembles for visual and relational descriptions). The scores are combined as a product of probabilities in Eq. (13), and the detection with the highest combined score is selected. Experiments on RefCOCO, RefCOCO+, RefCOCOg, Who's Waldo, and PhraseCut report large gains over zero-shot baselines, e.g., 73.7 versus 50.4 for GDINO-FLORA versus GDINO on RefCOCO val (Table I).

Significance. If the central claims hold, FLORA is a practically attractive modular pipeline: it requires no training or fine-tuning, gives interpretable per-component scores, and delivers consistent improvements over backbone grounding detectors in both detection and segmentation settings. The ablations in Tables V and VI are informative and support the contribution of each component, and the evaluation across five datasets is a strength. However, the headline 'training-free zero-shot' claim is load-bearing and is threatened by the likely inclusion of RefCOCO-series data in the pretraining of the Grounding DINO checkpoint used as the backbone. The Bayesian derivation is also heuristic rather than a rigorous posterior. The relative gains of FLORA over its own backbone may still hold, but the comparison to methods that were not trained on RefCOCO would not support a zero-shot state-of-the-art claim without additional evidence.

major comments (4)
  1. [Sec. IV-B1, Table I] The paper states 'we do not use the grounding detectors, especially GDINO, pre-trained on RefCOCO datasets to avoid data leak' (Sec. IV-B1), but the official Grounding DINO release is pretrained on a multi-source grounding corpus that includes RefCOCO/RefCOCO+/RefCOCOg, and the manuscript provides no evidence that a different checkpoint was used. Since both the GDINO baseline and GDINO-FLORA rely on that backbone, the RefCOCO-series results cannot be described as zero-shot, and the comparisons with ReCLIP, SS-CLIP, and GVLP, which do not use RefCOCO training data, are not on equal footing. Please specify the exact checkpoint and its pretraining data, and either re-run with a detector not exposed to these datasets or re-frame the claims as fine-tuning-free with an explicit in-distribution caveat.
  2. [Sec. III-E1, Eqs. (6)-(7)] The simplification of Eq. (6) to Eq. (7) requires conditional independence, P(OT,OL,OV,OR|x) = P(OT|x)P(OL|x)P(OV|x)P(OR|x). The stated 'Independence Assumption' in Sec. III-E1 is unconditional independence of the variables, which is neither the same nor generally true (e.g., object type and visual pattern are correlated with location). The uniform-prior assumptions on P(x) and P(O) are asserted without justification, and detector proposal distributions are not uniform over x. As written, Eq. (13) is a heuristic product of relevance scores rather than a Bayesian posterior. Please either justify the conditional-independence step or present Eq. (13) explicitly as an empirical scoring rule.
  3. [Sec. III-E2, Eq. (8)] The final prediction in Eq. (13) is restricted to the candidate set X = GroundingDetection(I, OT). If the LLM-parsed object type is incorrect, or the grounding detector fails to recall the true object, no downstream interpreter can recover. The paper does not measure or discuss this recall ceiling. Please add an analysis of this ceiling, e.g., oracle accuracy when the true object is guaranteed to be in X, or per-component recall rates, and discuss how the reported gains are bounded by it.
  4. [Sec. IV-D, Eq. (11)] Several components involve choices that are tuned on the evaluation benchmark but not fully specified: the form of sigma in Eq. (11) (the squared function is selected using RefCOCO val accuracy), the CLIP/GDINO ensemble weights for the visual and relation interpreters, the spatial-term dictionary in the validity filter, and the grounding-detector confidence threshold that defines X. Without these details, the method is difficult to reproduce, and the 'training-free' claim should be qualified by the benchmark-driven hyperparameter selection. Please report the exact settings and separate any tuning on the validation split from the zero-shot evaluation.
minor comments (3)
  1. [Table V vs. Table I] Table V reports 78.1 for 'prompt w/ FLM, w/ System-level Prompt (ours)' on RefCOCO TestA, while Table I reports 78.5 for GDINO-FLORA on the same split; please reconcile the discrepancy.
  2. [Abstract and Conclusion] The abstract and conclusion state 'up to around 45%' improvement, but the maximum relative gain in Table I is larger (e.g., RefCOCO TestB: (67.8−43.2)/43.2 ≈ 56.9%); please state the actual range or use a phrase such as 'at least 45%'.
  3. [Sec. IV-C2] In Sec. IV-C2, 'state-of-the-art' is misspelled as 'stage-of-the-art'; please correct the typo.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FLORA's Bayesian product is a reranking of off-the-shelf detector outputs and is benchmarked on held-out splits rather than reducing to fitted inputs.

full rationale

The derivation chain in Eqs. 1-13 is not circular. The final prediction Eq. 13 multiplies four externally computed probabilities: Grounding DINO confidence for the parsed object type (Eq. 8), a hand-defined geometric relevance score (Eqs. 9-11), and CLIP / grounding-detector text-image matching for visual patterns and relations (Eq. 12). None of these factors is defined in terms of the final selected object or the evaluation label, and no parameter is learned on the target benchmarks. The candidate set is produced by Grounding DINO, so FLORA is a reranker over the detector's boxes; that is a legitimate two-stage design, not a self-definitional reduction. Hyperparameter choices such as the squared sigma function and the ViT-H CLIP variant are selected from validation ablations (Sec. IV-D) and can introduce selection bias, but they are not fitted quantities renamed as predictions. The paper's self-citations ([29], [55], [70]) appear only in peripheral contexts and are not load-bearing for the central claim. The concern that the public Grounding DINO checkpoint may have been pretrained on RefCOCO-series data is a potential data-contamination or correctness issue, not a circularity reduction under the defined patterns, so it does not affect the circularity score.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central claim rests on a set of heuristic assumptions: LLM outputs follow the FLM syntax, the parsed components are independent, uniform priors are valid, and detector/CLIP scores behave as probabilities. None of these are derived or externally validated. Free parameters such as the sigma function shape and ensemble details are chosen using validation data, which introduces mild selection on the benchmark. No invented physical entities are introduced.

free parameters (4)
  • sigma function shape for spatial relevance = squared (x^2)
    Section IV-D: authors tested linear, squared, cubic, and exponential functions and found squared 'most effective' on RefCOCO val; this is a hand-chosen component of Eq. 11.
  • CLIP/GDINO ensemble weight for visual and relation interpreters = not reported; ensemble of CLIP and grounding detector scores
    Section III-E.2: 'CLIP contributes 9.5 points and the ensemble with grounding detection results contributes another 0.5 points'; no explicit weight or combination rule is given.
  • Spatial term dictionary for validity filter = not provided
    Section III-D: parsed location must match a predefined dictionary of terms like 'left' and 'right'; the dictionary contents are not specified, affecting which phrases are accepted or discarded.
  • Grounding detector candidate set and threshold = not specified
    Section III-E.2: candidates x come from GroundingDetection(I, OT), but detection confidence threshold, box count, and NMS settings are not reported.
assumptions (4)
  • domain assumption LLMs contain sufficient formal language knowledge to produce structured hashtag-prefixed outputs without fine-tuning.
    Section III-B and III-C: the method relies on Llama producing compliant outputs under CoP-FLM; no training or in-context examples beyond a system prompt are used.
  • ad hoc to paper The parsed components OT, OL, OV, and OR are conditionally independent given the object x.
    Section III-E.1 (Eq. 7): the product rule follows from this 'intuitive' assumption, which is known to be false in general (object type correlates with typical location and visual patterns).
  • ad hoc to paper Uniform priors over candidate objects and over the structured description O.
    Section III-E.1: P(x) and P(OT,OL,OV,OR) are 'assumed uniform and consistent', which simplifies Eq. 6 to Eq. 7 but discards useful prior information.
  • domain assumption Scores from off-the-shelf detectors and CLIP can be treated as calibrated probabilities for multiplication.
    Section III-E.2: detection confidence s_T(x) and CLIP relevance outputs are multiplied as if they were likelihoods P(OT|x) and P(OV|x); no calibration step is described.
invented entities (1)
  • Formal Language Model (FLM)
    purpose: Regulates LLM outputs into a fixed hashtag-prefixed syntax (type, location, visual pattern, relation) to make parsing deterministic.
    Introduced in Section III-B; it is a prompt-and-parse convention, not an independently verified formal system, and its benefits are only shown within this paper's pipeline.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FLORA: Formal Language Model Enables Robust Training-free Zero-shot Object Referring Analysis." pith.science (2026). https://pith.science/paper/JNGZIENQ

@misc{pith2026250109887,
  author       = {Pith},
  title        = {Pith review of: FLORA: Formal Language Model Enables Robust Training-free Zero-shot Object Referring Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JNGZIENQ}},
  note         = {Machine review of arXiv:2501.09887}
}
read the original abstract

Object Referring Analysis (ORA), commonly known as referring expression comprehension, requires the identification and localization of specific objects in an image based on natural descriptions. Unlike generic object detection, ORA requires both accurate language understanding and precise visual localization, making it inherently more complex. Although recent pre-trained large visual grounding detectors have achieved significant progress, they heavily rely on extensively labeled data and time-consuming learning. To address these, we introduce a novel, training-free framework for zero-shot ORA, termed FLORA (Formal Language for Object Referring and Analysis). FLORA harnesses the inherent reasoning capabilities of large language models (LLMs) and integrates a formal language model - a logical framework that regulates language within structured, rule-based descriptions - to provide effective zero-shot ORA. More specifically, our formal language model (FLM) enables an effective, logic-driven interpretation of object descriptions without necessitating any training processes. Built upon FLM-regulated LLM outputs, we further devise a Bayesian inference framework and employ appropriate off-the-shelf interpretive models to finalize the reasoning, delivering favorable robustness against LLM hallucinations and compelling ORA performance in a training-free manner. In practice, our FLORA boosts the zero-shot performance of existing pretrained grounding detectors by up to around 45%. Our comprehensive evaluation across different challenging datasets also confirms that FLORA consistently surpasses current state-of-the-art zero-shot methods in both detection and segmentation tasks associated with zero-shot ORA. We believe our probabilistic parsing and reasoning of the LLM outputs elevate the reliability and interpretability of zero-shot ORA. We shall release codes upon publication.

Figures

Figures reproduced from arXiv: 2501.09887 by the authors.

Figure 1
Figure 1. The concept behind our Formal Language-based Object Referring Analysis (FLORA) framework, which enables outstanding accuracy in a training-free [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overall pipeline of our training-free zero-shot FLORA. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Some visualized results of ORA using GDINO [12] baseline and our [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

88 extracted references · 44 canonical work pages

  1. [1]

    Referring expression comprehension: A survey of methods and datasets,

    Y . Qiao, C. Deng, and Q. Wu, “Referring expression comprehension: A survey of methods and datasets,” IEEE Transactions on Multimedia , vol. 23, pp. 4426–4440, 2020

    2020

  2. [2]

    Referring image segmentation by generative adversarial learning,

    S. Qiu, Y . Zhao, J. Jiao, Y . Wei, and S. Wei, “Referring image segmentation by generative adversarial learning,” IEEE Transactions on Multimedia, vol. 22, no. 5, pp. 1333–1344, 2019

    2019

  3. [3]

    Visual question answering: A survey of methods and datasets,

    Q. Wu, D. Teney, P. Wang, C. Shen, A. Dick, and A. Van Den Hen- gel, “Visual question answering: A survey of methods and datasets,” Computer Vision and Image Understanding , vol. 163, pp. 21–40, 2017

    2017

  4. [4]

    The multi-modal fusion in visual question answering: a review of attention mechanisms,

    S. Lu, M. Liu, L. Yin, Z. Yin, X. Liu, and W. Zheng, “The multi-modal fusion in visual question answering: a review of attention mechanisms,” PeerJ Computer Science, vol. 9, p. e1400, 2023

    2023

  5. [5]

    Imagic: Text-based real image editing with diffusion models,

    B. Kawar, S. Zada, O. Lang, O. Tov, H. Chang, T. Dekel, I. Mosseri, and M. Irani, “Imagic: Text-based real image editing with diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 6007–6017

    2023

  6. [6]

    Instructpix2pix: Learning to follow image editing instructions,

    T. Brooks, A. Holynski, and A. A. Efros, “Instructpix2pix: Learning to follow image editing instructions,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 18 392–18 402

    2023

  7. [7]

    Structured scene memory for vision-language navigation,

    H. Wang, W. Wang, W. Liang, C. Xiong, and J. Shen, “Structured scene memory for vision-language navigation,” in Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition , 2021, pp. 8455–8464. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 13

    2021

  8. [8]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763

    2021

Show all 88 references
  1. [9]

    Reproducible scaling laws for contrastive language-image learning,

    M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jitsev, “Reproducible scaling laws for contrastive language-image learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp...

    2023

  2. [10]

    Simple open-vocabulary object detection,

    M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn, A. Dosovitskiy, A. Mahendran, A. Arnab, M. Dehghani, Z. Shen et al., “Simple open-vocabulary object detection,” in European Conference on Computer Vision. Springer, 2022, pp. 728–755

    2022

  3. [11]

    Scaling open-vocabulary object detection,

    M. Minderer, A. Gritsenko, and N. Houlsby, “Scaling open-vocabulary object detection,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  4. [12]

    Grounding dino: Marrying dino with grounded pre- training for open-set object detection,

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu et al., “Grounding dino: Marrying dino with grounded pre- training for open-set object detection,” arXiv preprint arXiv:2303.05499, 2023

    2023 arXiv

  5. [13]

    Generation and comprehension of unambiguous object descriptions,

    J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy, “Generation and comprehension of unambiguous object descriptions,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 11–20

    2016

  6. [14]

    Modeling context in referring expressions,

    L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg, “Modeling context in referring expressions,” in Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14 . Springer, 2016, pp. 69–85

    2016

  7. [15]

    Reclip: A strong zero-shot baseline for referring expres- sion comprehension,

    S. Subramanian, W. Merrill, T. Darrell, M. Gardner, S. Singh, and A. Rohrbach, “Reclip: A strong zero-shot baseline for referring expres- sion comprehension,” arXiv preprint arXiv:2204.05991 , 2022

    2022 arXiv

  8. [16]

    Zero-shot referring image segmentation with global-local context features,

    S. Yu, P. H. Seo, and J. Son, “Zero-shot referring image segmentation with global-local context features,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 19 456–19 465

    2023

  9. [17]

    Cris: Clip-driven referring image segmentation,

    Z. Wang, Y . Lu, Q. Li, X. Tao, Y . Guo, M. Gong, and T. Liu, “Cris: Clip-driven referring image segmentation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 11 686–11 695

    2022

  10. [18]

    When and why vision-language models behave like bags-of-words, and what to do about it?

    M. Yuksekgonul, F. Bianchi, P. Kalluri, D. Jurafsky, and J. Zou, “When and why vision-language models behave like bags-of-words, and what to do about it?” arXiv preprint arXiv:2210.01936 , 2022

    2022 arXiv

  11. [19]

    Dense and aligned captions (dac) promote compositional reasoning in vl models,

    S. Doveh, A. Arbelle, S. Harary, R. Herzig, D. Kim, P. Cascante-Bonilla, A. Alfassy, R. Panda, R. Giryes, R. Feris et al. , “Dense and aligned captions (dac) promote compositional reasoning in vl models,” Advances in Neural Information Processing Systems , vol. 36, pp. 76 137–...

    2023

  12. [20]

    Teaching structured vision & language concepts to vision & language models,

    S. Doveh, A. Arbelle, S. Harary, E. Schwartz, R. Herzig, R. Giryes, R. Feris, R. Panda, S. Ullman, and L. Karlinsky, “Teaching structured vision & language concepts to vision & language models,” in Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2023

  13. [21]

    Visual programming: Compositional visual reasoning without training,

    T. Gupta and A. Kembhavi, “Visual programming: Compositional visual reasoning without training,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 14 953–14 962

    2023

  14. [22]

    Aligning bag of regions for open-vocabulary object detection,

    S. Wu, W. Zhang, S. Jin, W. Liu, and C. C. Loy, “Aligning bag of regions for open-vocabulary object detection,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 15 254–15 264

    2023

  15. [23]

    Comclip: Training-free composi- tional image and text matching,

    K. Jiang, X. He, R. Xu, and X. Wang, “Comclip: Training-free composi- tional image and text matching,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , 20...

    2024

  16. [24]

    Zero-shot referring expression comprehension via structural similarity between images and captions,

    Z. Han, F. Zhu, Q. Lao, and H. Jiang, “Zero-shot referring expression comprehension via structural similarity between images and captions,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 14 364–14 374

    2024

  17. [25]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  18. [26]

    A survey on deep learning in medical image analysis,

    G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. Van Der Laak, B. Van Ginneken, and C. I. S´anchez, “A survey on deep learning in medical image analysis,” Medical image analysis, vol. 42, pp. 60–88, 2017

    2017

  19. [27]

    Towards high performance human keypoint detection,

    J. Zhang, Z. Chen, and D. Tao, “Towards high performance human keypoint detection,” International Journal of Computer Vision, vol. 129, no. 9, pp. 2639–2662, 2021

    2021

  20. [28]

    Are we ready for autonomous driving? the kitti vision benchmark suite,

    A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in 2012 IEEE conference on computer vision and pattern recognition . IEEE, 2012, pp. 3354–3361

    2012

  21. [29]

    Progressive lidar adaptation for road detection,

    Z. Chen, J. Zhang, and D. Tao, “Progressive lidar adaptation for road detection,” IEEE/CAA Journal of Automatica Sinica , vol. 6, no. 3, pp. 693–702, 2019

    2019

  22. [30]

    X-linear attention networks for image captioning,

    Y . Pan, T. Yao, Y . Li, and T. Mei, “X-linear attention networks for image captioning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 10 971–10 980

    2020

  23. [31]

    From show to tell: A survey on deep learning-based image captioning,

    M. Stefanini, M. Cornia, L. Baraldi, S. Cascianelli, G. Fiameni, and R. Cucchiara, “From show to tell: A survey on deep learning-based image captioning,” IEEE transactions on pattern analysis and machine intelligence, vol. 45, no. 1, pp. 539–559, 2022

    2022

  24. [32]

    Scaling up vision-language pre-training for image captioning,

    X. Hu, Z. Gan, J. Wang, Z. Yang, Z. Liu, Y . Lu, and L. Wang, “Scaling up vision-language pre-training for image captioning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 17 980–17 989

    2022

  25. [33]

    Comprehending and ordering seman- tics for image captioning,

    Y . Li, Y . Pan, T. Yao, and T. Mei, “Comprehending and ordering seman- tics for image captioning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 17 990–17 999

    2022

  26. [34]

    Hierarchy parsing for image captioning,

    T. Yao, Y . Pan, Y . Li, and T. Mei, “Hierarchy parsing for image captioning,” in Proceedings of the IEEE/CVF international conference on computer vision , 2019, pp. 2621–2629

    2019

  27. [35]

    Referitgame: Referring to objects in photographs of natural scenes,

    S. Kazemzadeh, V . Ordonez, M. Matten, and T. Berg, “Referitgame: Referring to objects in photographs of natural scenes,” in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014, pp. 787–798

    2014

  28. [36]

    Microsoft coco: Common objects in context,

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 . Springer,...

    2014

  29. [37]

    Who’s waldo? linking people across text and images,

    Y . Cui, A. Khandelwal, Y . Artzi, N. Snavely, and H. Averbuch-Elor, “Who’s waldo? linking people across text and images,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 1374–1384

    2021

  30. [38]

    Phrasecut: Language- based image segmentation in the wild,

    C. Wu, Z. Lin, S. Cohen, T. Bui, and S. Maji, “Phrasecut: Language- based image segmentation in the wild,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 10 216–10 225

    2020

  31. [39]

    Clevr-ref+: Diagnosing visual reasoning with referring expressions,

    R. Liu, C. Liu, Y . Bai, and A. L. Yuille, “Clevr-ref+: Diagnosing visual reasoning with referring expressions,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 4185– 4194

    2019

  32. [40]

    Refego: Referring expres- sion comprehension dataset from first-person perception of ego4d,

    S. Kurita, N. Katsura, and E. Onami, “Refego: Referring expres- sion comprehension dataset from first-person perception of ego4d,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 15 214–15 224

    2023

  33. [41]

    Instructdet: Diversifying referring object de- tection with generalized instructions,

    R. Dang, J. Feng, H. Zhang, C. Ge, L. Song, L. Gong, C. Liu, Q. Chen, F. Zhu, R. Zhao et al. , “Instructdet: Diversifying referring object de- tection with generalized instructions,” arXiv preprint arXiv:2310.05136, 2023

    2023 arXiv

  34. [42]

    Modeling con- text between objects for referring expression understanding,

    V . K. Nagaraja, V . I. Morariu, and L. S. Davis, “Modeling con- text between objects for referring expression understanding,” in Com- puter Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14 . Springer, 2016, p...

    2016

  35. [43]

    Natural language object retrieval,

    R. Hu, H. Xu, M. Rohrbach, J. Feng, K. Saenko, and T. Darrell, “Natural language object retrieval,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 4555–4564

    2016

  36. [44]

    Comprehension-guided referring expres- sions,

    R. Luo and G. Shakhnarovich, “Comprehension-guided referring expres- sions,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 7102–7111

    2017

  37. [45]

    Grounding referring expressions in images by variational context,

    H. Zhang, Y . Niu, and S.-F. Chang, “Grounding referring expressions in images by variational context,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 4158–4166

    2018

  38. [46]

    A fast and accurate one-stage approach to visual grounding,

    Z. Yang, B. Gong, L. Wang, W. Huang, D. Yu, and J. Luo, “A fast and accurate one-stage approach to visual grounding,” in Proceedings of the IEEE/CVF international conference on computer vision , 2019, pp. 4683–4693

    2019

  39. [47]

    Zero-shot grounding of objects from natural language queries,

    A. Sadhu, K. Chen, and R. Nevatia, “Zero-shot grounding of objects from natural language queries,” in Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision , 2019, pp. 4694–4703. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 14

    2019

  40. [48]

    Multi-task collaborative network for joint referring expression comprehension and segmentation,

    G. Luo, Y . Zhou, X. Sun, L. Cao, C. Wu, C. Deng, and R. Ji, “Multi-task collaborative network for joint referring expression comprehension and segmentation,” in Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition , 2020, pp. 10 034–10 043

    2020

  41. [49]

    Modeling relationships in referential expressions with compositional modular networks,

    R. Hu, M. Rohrbach, J. Andreas, T. Darrell, and K. Saenko, “Modeling relationships in referential expressions with compositional modular networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 1115–1124

    2017

  42. [50]

    Neigh- bourhood watch: Referring expression comprehension via language- guided graph attention networks,

    P. Wang, Q. Wu, J. Cao, C. Shen, L. Gao, and A. v. d. Hengel, “Neigh- bourhood watch: Referring expression comprehension via language- guided graph attention networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 1960–1968

    2019

  43. [51]

    Flava: A foundational language and vision alignment model,

    A. Singh, R. Hu, V . Goswami, G. Couairon, W. Galuba, M. Rohrbach, and D. Kiela, “Flava: A foundational language and vision alignment model,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 15 638–15 650

    2022

  44. [52]

    Yolo9000: better, faster, stronger,

    J. Redmon and A. Farhadi, “Yolo9000: better, faster, stronger,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 7263–7271

    2017

  45. [53]

    Atten- tive contexts for object detection,

    J. Li, Y . Wei, X. Liang, J. Dong, T. Xu, J. Feng, and S. Yan, “Atten- tive contexts for object detection,” IEEE Transactions on Multimedia , vol. 19, no. 5, pp. 944–954, 2016

    2016

  46. [54]

    Faster r-cnn: Towards real-time object detection with region proposal networks,

    S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” IEEE transactions on pattern analysis and machine intelligence, vol. 39, no. 6, pp. 1137–1149, 2016

    2016

  47. [55]

    Transformer-based context condensation for boosting feature pyramids in object detection,

    Z. Chen, J. Zhang, Y . Xu, and D. Tao, “Transformer-based context condensation for boosting feature pyramids in object detection,” Inter- national Journal of Computer Vision , vol. 131, no. 10, pp. 2738–2756, 2023

    2023

  48. [56]

    Attention is all you need,

    A. Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems, 2017

    2017

  49. [57]

    End-to-end object detection with transformers,

    N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in European conference on computer vision . Springer, 2020, pp. 213– 229

    2020

  50. [58]

    Objects365: A large-scale, high-quality dataset for object detection,

    S. Shao, Z. Li, T. Zhang, C. Peng, G. Yu, X. Zhang, J. Li, and J. Sun, “Objects365: A large-scale, high-quality dataset for object detection,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 8430–8439

    2019

  51. [59]

    Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models,

    B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hocken- maier, and S. Lazebnik, “Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models,” in Proceedings of the IEEE international conference on computer vision , 2015, pp. ...

    2015

  52. [60]

    Open-vocabulary object detection using captions,

    A. Zareian, K. D. Rosa, D. H. Hu, and S.-F. Chang, “Open-vocabulary object detection using captions,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 14 393–14 402

    2021

  53. [61]

    Open-vocabulary object detection via vision and language knowledge distillation,

    X. Gu, T.-Y . Lin, W. Kuo, and Y . Cui, “Open-vocabulary object detection via vision and language knowledge distillation,” arXiv preprint arXiv:2104.13921, 2021

    2021 arXiv

  54. [62]

    Uninext: Exploring a unified architecture for vision recognition,

    F. Lin, J. Yuan, S. Wu, F. Wang, and Z. Wang, “Uninext: Exploring a unified architecture for vision recognition,” in Proceedings of the 31st ACM International Conference on Multimedia , 2023, pp. 3200–3208

    2023

  55. [63]

    Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models,

    I. Mirzadeh, K. Alizadeh, H. Shahrokhi, O. Tuzel, S. Bengio, and M. Farajtabar, “Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models,” arXiv preprint arXiv:2410.05229, 2024

    2024 arXiv

  56. [64]

    Transzero: Attribute-guided transformer for zero-shot learn- ing,

    S. Chen, Z. Hong, Y . Liu, G.-S. Xie, B. Sun, H. Li, Q. Peng, K. Lu, and X. You, “Transzero: Attribute-guided transformer for zero-shot learn- ing,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 36, no. 1, 2022, pp. 330–338

    2022

  57. [65]

    Contrastive embedding for gener- alized zero-shot learning,

    Z. Han, Z. Fu, S. Chen, and J. Yang, “Contrastive embedding for gener- alized zero-shot learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 2371–2381

    2021

  58. [66]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. D. M.-W. C. Kenton and L. K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of naacL-HLT, vol. 1. Minneapolis, Minnesota, 2019, p. 2

    2019

  59. [67]

    Isometric propagation network for generalized zero-shot learning,

    L. Liu, T. Zhou, G. Long, J. Jiang, X. Dong, and C. Zhang, “Isometric propagation network for generalized zero-shot learning,” arXiv preprint arXiv:2102.02038, 2021

    2021 arXiv

  60. [68]

    Scaling up visual and vision-language representation learning with noisy text supervision,

    C. Jia, Y . Yang, Y . Xia, Y .-T. Chen, Z. Parekh, H. Pham, Q. Le, Y .-H. Sung, Z. Li, and T. Duerig, “Scaling up visual and vision-language representation learning with noisy text supervision,” in International conference on machine learning . PMLR, 2021, pp. 4904–4916

    2021

  61. [69]

    Effective con- ditioned and composed image retrieval combining clip-based features,

    A. Baldrati, M. Bertini, T. Uricchio, and A. Del Bimbo, “Effective con- ditioned and composed image retrieval combining clip-based features,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 21 466–21 474

    2022

  62. [70]

    Clamp: Prompt-based contrastive learning for connecting language and animal pose,

    X. Zhang, W. Wang, Z. Chen, Y . Xu, J. Zhang, and D. Tao, “Clamp: Prompt-based contrastive learning for connecting language and animal pose,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 23 272–23 281

    2023

  63. [71]

    H. Hsia, C. Lin, B. Kung, J. Chen, D. Tan, J. Chen, and K. Hua, “Clipcam: A simple baseline for zero-shot text-guided object and action localization, icassp 2022-2022 ieee international conference on acoustics, speech and signal processing (icassp),” 2022

    2022

  64. [72]

    Open-vocabulary object detec- tion via vision and language knowledge distillation,

    X. Gu, T.-Y . Lin, W. Kuo, and Y . Cui, “Open-vocabulary object detec- tion via vision and language knowledge distillation,” in International Conference on Learning Representations

  65. [73]

    Learning to prompt for open-vocabulary object detection with vision-language model,

    Y . Du, F. Wei, Z. Zhang, M. Shi, Y . Gao, and G. Li, “Learning to prompt for open-vocabulary object detection with vision-language model,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 14 084–14 093

    2022

  66. [74]

    Scaling open-vocabulary image segmentation with image-level labels,

    G. Ghiasi, X. Gu, Y . Cui, and T.-Y . Lin, “Scaling open-vocabulary image segmentation with image-level labels,” in European Conference on Computer Vision . Springer, 2022, pp. 540–557

    2022

  67. [75]

    Adapting clip for phrase localization without further training,

    J. Li, G. Shakhnarovich, and R. A. Yeh, “Adapting clip for phrase localization without further training,” arXiv preprint arXiv:2204.03647 , 2022

    2022 arXiv

  68. [76]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24 824–24 837, 2022

    2022

  69. [77]

    Instructblip: Towards general-purpose vision- language models with instruction tuning,

    W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi, “Instructblip: Towards general-purpose vision- language models with instruction tuning,” 2023

    2023

  70. [78]

    Chameleon: Plug-and-play compositional reasoning with large language models,

    P. Lu, B. Peng, H. Cheng, M. Galley, K.-W. Chang, Y . N. Wu, S.-C. Zhu, and J. Gao, “Chameleon: Plug-and-play compositional reasoning with large language models,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  71. [79]

    Langchain: Building applications with large language models,

    “Langchain: Building applications with large language models,” https: //github.com/hwchase17/langchain, 2023, accessed: 2024-11-26

    2023

  72. [80]

    Corefqa: Coreference resolution as query-based span prediction,

    W. Wu, F. Wang, A. Yuan, F. Wu, and J. Li, “Corefqa: Coreference resolution as query-based span prediction,” in Proceedings of the 58th annual meeting of the association for computational linguistics , 2020, pp. 6953–6963

    2020

  73. [81]

    Mdetr-modulated detection for end-to-end multi-modal understanding,

    A. Kamath, M. Singh, Y . LeCun, G. Synnaeve, I. Misra, and N. Carion, “Mdetr-modulated detection for end-to-end multi-modal understanding,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 1780–1790

    2021

  74. [82]

    Grad-cam: Visual explanations from deep networks via gradient-based localization,

    R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in Proceedings of the IEEE international conference on computer vision , 2017, pp. 618–626

    2017

  75. [83]

    Groundvlp: Harnessing zero-shot visual grounding from vision-language pre-training and open-vocabulary object detection,

    H. Shen, T. Zhao, M. Zhu, and J. Yin, “Groundvlp: Harnessing zero-shot visual grounding from vision-language pre-training and open-vocabulary object detection,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 5, 2024, pp. 4766–4775

    2024

  76. [84]

    Consnet: Learning consistency graph for zero-shot human-object interaction detection,

    Y . Liu, J. Yuan, and C. W. Chen, “Consnet: Learning consistency graph for zero-shot human-object interaction detection,” in Proceedings of the 28th ACM International Conference on Multimedia , 2020, pp. 4235– 4243

    2020

  77. [85]

    Extract free dense labels from clip,

    C. Zhou, C. C. Loy, and B. Dai, “Extract free dense labels from clip,” in European Conference on Computer Vision. Springer, 2022, pp. 696– 712

    2022

  78. [86]

    Text augmented spatial aware zero-shot referring image segmentation,

    Y . Suo, L. Zhu, and Y . Yang, “Text augmented spatial aware zero-shot referring image segmentation,” in The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  79. [87]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Loet al., “Segment anything,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 4015–4026

    2023

  80. [88]

    Groundhog: Grounding large language models to holistic segmentation,

    Y . Zhang, Z. Ma, X. Gao, S. Shakiah, Q. Gao, and J. Chai, “Groundhog: Grounding large language models to holistic segmentation,” in Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, pp. 14 227–14 238. JOURNAL OF LATEX CLASS FILES, VOL....

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.