Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Understand, Think, and Answer: Advancing Visual Reasoning with Large Multimodal Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a single-pass 'understand-think-answer' process, in which the model writes its own grounded hints before answering, lets a large multimodal model beat prior chain-of-thought and tool-using systems on compositional…

desk verdict Plausible single-pass mechanism for visual reasoning, strong numbers, but attribution is under-supported by ablations and a likely train/eval overlap; deserves peer review with demands for controlled baselines. read the letter →

arxiv 2505.20753 v1 pith:KNAXNP4D submitted 2025-05-27 cs.CV cs.AI

Yufei Zhan , Hongyin Zhao , Yousong Zhu , Shurong Zheng , Fan Yang , Ming Tang , Jinqiao Wang This is my paper
classification cs.CVcs.AI
keywords visualreasoninglargemultimodalmodelschain-of-thoughtcompositionalgroundinginstructiontuningsingleforwardpassunderstand-think-answer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a single forward pass can contain the entire reasoning process if the model is trained to emit its own grounded understanding before answering. On images, Griffon-R first writes a short plan and produces hints such as object names, attributes, and normalized bounding boxes; then it reasons over those hints; then it answers. This 'understand-think-answer' sequence is trained on 334K re-annotated visual reasoning samples, and the resulting model reports 70.9 on VSR and 63.7 on CLEVR, ahead of the chain-of-thought and toolkit-based systems it compares against. The paper also reports gains on MMBench, ScienceQA, POPE, and referring expression comprehension, and a 13x latency reduction over a toolkit-based baseline. The intended lesson is that compositional visual reasoning does not need external tools or multiple passes; a model's own grounding abilities can power the reasoning.

What carries the argument

The load-bearing object is the 'Understand' step's output: structured visual cues (grounded captions and object bounding boxes with normalized coordinates) that make the needed evidence explicit for the model itself. The second carrying component is a semi-automatic data engine that produces 334K training samples: a strong off-the-shelf vision-language model drafts question analyses, understanding plans, and initial annotations, and human experts complete multi-object grounding and review or remove bad samples. The model is trained with cross-entropy loss over the full Understand-Think-Answer sequence in a single autoregressive pass.

What would settle it

Take the 'Understand' hints from held-out samples and give them, without the image, to a strong language model along with the original question; if that model's accuracy on VSR or CLEVR approaches Griffon-R's accuracy with the image, then the hints are carrying the answer and the visual reasoning step is not doing the work.

Watch

Extended reading notes

Core claim

The central claim is that compositional visual reasoning fails under the shortcut-learning paradigm—training a model to produce the final answer directly from image and question—and that the failure is avoidable by having the model generate its own intermediate reasoning context. Griffon-R's forward pass produces three blocks in order: an 'Understand' block that plans what information the question requires and emits structured visual cues such as grounded captions and object boxes with coordinates; a 'Think' block that self-prompts reasoning over those cues; and an 'Answer' block that outputs the final response. This is done with ordinary next-token prediction, without extra perception modules, tool calls, or repeated inference. The paper's evidence is that Griffon-R improves on VSR, CLEVR, V-Star spatial, TallyQA, MMBench, ScienceQA, POPE, and referring expression comprehension, and that the mechanism is faster than a toolkit-based approach while remaining more accurate.

Load-bearing premise

The method assumes the curated hints contain the visual evidence needed for the answer but do not themselves contain the answer; if a hint names the attribute or relation being asked about, the final 'think' step becomes a formality and the model may be learning to read answers off hints instead of reasoning from the image.

Editorial extensions

If this is right

  • The single-pass design means the gains come without repeated forward passes or external tool calls; the paper reports roughly 13x lower per-sample latency than a toolkit-based baseline on V-Star spatial.
  • Because training uses only cross-entropy loss on the sequence, the same mechanism can be added to other large multimodal models without task-specific losses or architectural changes.
  • The improvement is not confined to one benchmark: VSR and CLEVR gains accompany higher MMBench, ScienceQA, SEED, TextVQA, and POPE scores, suggesting the mechanism transfers to general multimodal QA.
  • The model's referring expression comprehension is at or above prior visual reasoning systems, which supports the paper's claim that better understanding is what drives better reasoning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the hints are genuinely answer-free, this design is effectively a way to inject grounded rationales into any LMM; a natural next experiment is to perturb the hints or offer multiple plausible hint sets and check whether reasoning quality tracks the evidence.
  • The mechanism sits between chain-of-thought prompting and toolkit-based methods: the model calls its own perception abilities instead of external programs, so it can be viewed as learned internal tool use.
  • The data engine's filtering criteria (yes/no questions and relationship/attribute keywords) may bias the training distribution toward certain question forms; testing on fresh, unanticipated compositional question types would measure how general the mechanism really is.
  • The ablation in which the mechanism without curated annotations gives no gain on V-Star suggests the data, not the mechanism alone, carries much of the effect; a direct comparison against randomly generated hints would isolate what curation contributes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a unified "understand-think-answer" visual reasoning mechanism for large multimodal models (LMMs), in which the model first plans and generates grounding-style hints, then self-prompts reasoning, and finally answers, all in a single forward pass without external tools. To train this behavior, the authors build a semi-automatic data engine that combines Qwen2-VL-72B annotations with human expert review to produce 334K visual reasoning samples mixing VQA, instruction, and caption data. The resulting model, Griffon-R, is built on the Griffon v2 architecture with a Gemma-9B LLM and trained in three stages. The paper reports strong results on VSR, CLEVR, GQA, V-StarSpat, TallyQA, MMBench, ScienceQA, TextVQA, SEED, LLaVA-W, and POPE, and it presents ablations on understanding quality (REC), mechanism versus toolkit-based inference, and the contribution of curated annotations.

Significance. If the attribution claims hold, the paper makes a useful contribution: a single-pass, tool-free recipe for eliciting compositional visual reasoning from LMMs by generating self-contained visual cues before answering. The 13x speed advantage over a toolkit-based method (Table 3), the strong headline numbers, and the publicly promised data/model release are concrete assets. The REC results in Table 4 demonstrate that the model has strong grounding ability. However, the paper's central mechanistic claim is currently under-supported: the only controlled ablation of the mechanism (Table 4) shows no gain without curated annotations (75.0 vs 75.0 on V-StarSpat), and the gain with curated data (77.6) is potentially confounded by train/evaluation image overlap and by answer leakage through hints. The headline comparisons in Tables 1 and 2 are across different backbones, training data, and scales, so they do not by themselves isolate the proposed mechanism.

major comments (4)
  1. [§4.5, Table 4; §B.1, Table 7; Appendix A.1] The only controlled experiment isolating the unified mechanism is not interpretable without a train/test image-overlap analysis. V-StarSpat is constructed from Visual Genome images (the V* benchmark), and Table 7 shows that Stage II and Stage III training include Visual Genome for REG, DET, and Perc, as well as GQA, whose images come from Visual Genome. The paper never reports a deduplication or overlap check between V-StarSpat and any training source. If V-StarSpat images (or near-duplicates) appeared in training, the +2.6 gain in Table 4 could reflect memorization of already-seen scenes rather than a general reasoning mechanism. Please report exact overlap counts, evaluate on a disjoint held-out subset, and also report the same split-exclusion information for GQA, ScienceQA, and TextVQA, which are evaluated in Tables 1–2 after being used as training sources in Table 7.
  2. [§4.5, Table 4; §4.3, Tables 1–2] The paper's central claim is that the unified understand-think-answer mechanism, not data quantity or backbone, drives Griffon-R's performance on VSR, CLEVR, MMBench, and ScienceQA. But the only controlled mechanism ablation is on V-StarSpat (Table 4), while the headline results in Tables 1–2 compare against models with different backbones, resolutions, and training corpora. A same-backbone shortcut-trained baseline that receives the same 334K curated data and the same multi-task mixture but is trained to answer directly, without generating understand/think segments, is needed on VSR, CLEVR, MMBench, and ScienceQA. Without such a baseline, the paper demonstrates a strong model but not that the proposed mechanism is what produces the gains on those benchmarks.
  3. [§3.2, Figure 2; §3.3, Appendix A.2] The load-bearing assumption that curated understanding hints contain relevant visual evidence but not the answer is never tested. In Figure 2, the hint includes "vegetables on top, which look like mushrooms," which comes very close to the final response content; in the second example, the hint describes the skateboarder's attire and explicitly hypothesizes "it could be summer." If the training hints systematically encode the answer's key relation or attribute, then the "think" step is a near-trivial read-off, and the model may be learning to answer from hints rather than from the image. Please report a leakage measurement (e.g., how often the hint text contains the answer token or a paraphrase), provide a human audit of a random sample, and ablate with hints stripped of relation/attribute conclusions.
  4. [§4.5, Table 4; §4.6, Figure 5] The paper should report the variance or significance of the ablation results. Table 4 reports single numbers (75.0 vs 75.0 vs 77.6; 89.1 vs 89.4 vs 89.3), and it is unclear whether the +0.3 POPE gain or even the +2.6 V-StarSpat gain exceeds run-to-run noise. Please provide multiple seeds or a statistical test, especially because the first two rows are identical on V-StarSpat and the POPE differences are within typical benchmark fluctuation.
minor comments (6)
  1. [Abstract] The phrase "Data, models, and codes will be release" is a typo; it should read "will be released."
  2. [§4.5] "high efficinet inference" should be "highly efficient inference."
  3. [Figure 2] "Indentify the presense" should be "Identify the presence."
  4. [§3.2, Eq. (3)] The notation p([XU, XT, Xans]) is introduced informally. Please define explicitly that the bracket denotes left-to-right autoregressive generation of the concatenated sequence and clarify the exact role of XT versus Xans in the loss computation.
  5. [§4.1] "resolution to 1022" is unclear: specify whether this is width, height, or both, and state the units and the interpolation method.
  6. [Appendix B.1] The sentence "while for the overlapping data with the visual reasoning data, we directly remove them" is ambiguous; it should identify which overlaps are removed (e.g., benchmark evaluation sets versus general instruction data) and how the removal is performed.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: Griffon-R's claims are empirical benchmark evaluations; the only self-citation is to the authors' prior architecture and is not load-bearing.

full rationale

The paper contains no derivation chain in which a predicted quantity is defined from, or fitted to, the quantity it purports to predict. The central claims are empirical: a model is trained on 334K curated instruction samples plus standard SFT data and evaluated on external benchmark answer labels (VSR, CLEVR, GQA, V-StarSpat., MMBench, ScienceQA, etc.). The unified 'understand-think-answer' mechanism is implemented as autoregressive generation p([XU, XT, Xans]) = product_i p(x_i | I, Q, X_<i), which is a training objective and inference scheme, not a self-referential derivation. The 334K data are produced by Qwen2-VL-72B and human annotators from public VQA pairs; the hints are answer-relevant by design, but at test time the model must generate those hints itself, and the final answers are evaluated against benchmark ground truth. The only notable self-citation is to Griffon v2 [93] for the high-resolution architecture, but this is an implementation choice that is externally validated by the REC results, and the benchmark gains do not rest on a self-cited uniqueness or correctness theorem. The ablation in Table 4 shows that the mechanism without curated annotations gives no gain on V-StarSpat. (75.0 vs. 75.0), which weakens the attribution of the improvement to the mechanism but is an empirical null result, not circularity. A benchmark-contamination risk exists because Table 7 lists GQA, TextVQA, and Visual Genome among training sources while Tables 1-2 report GQA, TextVQA, and V-Star results, and no overlap check is reported; this is a correctness/leakage concern, not a circularity of definition or fit. Score 1 reflects only the minor non-load-bearing self-citation.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

This paper is an empirical machine learning contribution: it fits no mathematical constants, and its central claim rests on the quality and non-leakage of the AI-plus-human annotations, on the representativeness of the curated data, and on the validity of the chosen benchmarks. No new physical or mathematical entities are introduced.

assumptions (4)
  • domain assumption The AI expert Qwen2-VL-72B produces accurate understanding and caption/grounding annotations for training data.
    Section 3.3 uses Qwen2-VL-72B as the AI Expert to design the understanding process and generate captions; errors propagate to Griffon-R.
  • domain assumption Human experts correctly verify and fix grounding annotations and understanding steps.
    Appendix A.3 describes human curation; human error or bias would corrupt the training set.
  • domain assumption The benchmark evaluations (VSR, CLEVR, MMBench, etc.) measure the intended visual reasoning capabilities and are not gamed by the model's training distribution.
    Section 4 uses these benchmarks as ground truth; if the model overfits to benchmark-specific priors, the reported improvements would not generalize.
  • domain assumption Supervised fine-tuning with the curated data is sufficient to instill the understand-think-answer process without task-specific losses.
    Section 3.4 states only cross-entropy loss is used; if the mechanism requires special losses or decoding, the reported results may depend on unstated details.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understand, Think, and Answer: Advancing Visual Reasoning with Large Multimodal Models." pith.science (2026). https://pith.science/paper/KNAXNP4D

@misc{pith2026250520753,
  author       = {Pith},
  title        = {Pith review of: Understand, Think, and Answer: Advancing Visual Reasoning with Large Multimodal Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KNAXNP4D}},
  note         = {Machine review of arXiv:2505.20753}
}
read the original abstract

Large Multimodal Models (LMMs) have recently demonstrated remarkable visual understanding performance on both vision-language and vision-centric tasks. However, they often fall short in integrating advanced, task-specific capabilities for compositional reasoning, which hinders their progress toward truly competent general vision models. To address this, we present a unified visual reasoning mechanism that enables LMMs to solve complicated compositional problems by leveraging their intrinsic capabilities (e.g. grounding and visual understanding capabilities). Different from the previous shortcut learning mechanism, our approach introduces a human-like understanding-thinking-answering process, allowing the model to complete all steps in a single pass forwarding without the need for multiple inferences or external tools. This design bridges the gap between foundational visual capabilities and general question answering, encouraging LMMs to generate faithful and traceable responses for complex visual reasoning. Meanwhile, we curate 334K visual instruction samples covering both general scenes and text-rich scenes and involving multiple foundational visual capabilities. Our trained model, Griffon-R, has the ability of end-to-end automatic understanding, self-thinking, and reasoning answers. Comprehensive experiments show that Griffon-R not only achieves advancing performance on complex visual reasoning benchmarks including VSR and CLEVR, but also enhances multimodal capabilities across various benchmarks like MMBench and ScienceQA. Data, models, and codes will be release at https://github.com/jefferyZhan/Griffon/tree/master/Griffon-R soon.

Figures

Figures reproduced from arXiv: 2505.20753 by the authors.

Figure 1
Figure 1. Enabled by the proposed unified mechanism, Griffon-R naturally connects the reasoning [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Detailed illustration of the unified visual reasoning mechanism with the “Understand-Think [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Illustration of the semi-automatic expert [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Ablation on understanding quality. With REC task covering object localization and attribute perception, we choose it to evaluate the quality of understanding in the mechanism. Methods RefCOCO RefCOCO+ RefCOCOg val test-A test-B val test-A test-B val-u test-u Expert Mod…
Figure 5
Figure 5. Figure 5: Visualization of Griffon-R’s reasoning results. Correct answers are highlighted in bold [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: An example UI screenshot showcasing the annotation process for a question-answer pair [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. BVS: Bayesian Visual Search with Multimodal Large Language Model for Fine-grained Perception

    cs.CV 2026-07 conditional novelty 6.0 of 10

    BVS combines early-stop attention rollout priors with a scale-aware non-stationary kernel and GP-UCB to locate tiny objects in UHR images more accurately and with fewer MLLM queries than prior visual-search methods.

Reference graph

Works this paper leans on

103 extracted references · 28 canonical work pages · cited by 1 Pith paper

  1. [1]

    Tallyqa: Answering complex counting questions

    Manoj Acharya, Kushal Kafle, and Christopher Kanan. Tallyqa: Answering complex counting questions. InAAAI, 2019

    2019

  2. [2]

    Macmillan, 2005

    John R Anderson.Cognitive psychology and its implications. Macmillan, 2005

    2005

  3. [3]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv:2308.12966, 2023

    Jinze Bai and et al. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv:2308.12966, 2023

    arXiv 2023

  4. [4]

    Graph of thoughts: Solving elaborate problems with large language models

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. InProceedings of the AAAI Conference on Artificial Intelligence, pages 17682–17690, 2024

    2024

  5. [5]

    Vizwiz: nearly real-time answers to visual questions

    Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. Vizwiz: nearly real-time answers to visual questions. InProceedings of the 23nd annual ACM symposium on User interface software and technology, pages 333–342, 2010

    2010

  6. [6]

    Due: End-to-end document understanding benchmark

    Łukasz Borchmann, Michał Pietruszka, Tomasz Stanislawek, Dawid Jurkiewicz, Michał Turski, Karolina Szyndler, and Filip Grali ´nski. Due: End-to-end document understanding benchmark. InThirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021

    2021

  7. [7]

    Allava: Harnessing gpt4v-synthesized data for a lite vision-language model, 2024

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for a lite vision-language model, 2024

    2024

  8. [8]

    Shikra:unleashing multimodal llm’s referential dialogue magic.arXiv:2306.15195, 2023

    Keqin Chen and et al. Shikra:unleashing multimodal llm’s referential dialogue magic.arXiv:2306.15195, 2023

    arXiv 2023

Show all 103 references
  1. [9]

    Sharegpt4v: Improving large multi-modal models with better captions.arXiv preprint arXiv:2311.12793, 2023

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions.arXiv preprint arXiv:2311.12793, 2023

    2023 arXiv

  2. [10]

    Microsoft coco captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325, 2015

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325, 2015

    2015 arXiv

  3. [11]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023

    2023

  4. [12]

    Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023. 10

    2023

  5. [13]

    Enhancing chat language models by scaling high-quality instructional conversations.arXiv preprint arXiv:2305.14233, 2023

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations.arXiv preprint arXiv:2305.14233, 2023

    2023 arXiv

  6. [14]

    Llama-adapter v2: Parameter-efficient visual instruction model.arXiv preprint arXiv:2304.15010, 2023

    Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model.arXiv preprint arXiv:2304.15010, 2023

    2023 arXiv

  7. [15]

    Shortcut learning in deep neural networks.Nature Machine Intelligence, 2(11):665–673, 2020

    Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks.Nature Machine Intelligence, 2(11):665–673, 2020

    2020

  8. [16]

    Flacuna: Unleashing the problem solving power of vicuna using flan fine-tuning, 2023

    Deepanway Ghosal, Yew Ken Chia, Navonil Majumder, and Soujanya Poria. Flacuna: Unleashing the problem solving power of vicuna using flan fine-tuning, 2023

    2023

  9. [17]

    Accurate, large minibatch sgd: Training imagenet in 1 hour.arXiv preprint arXiv:1706.02677, 2017

    Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour.arXiv preprint arXiv:1706.02677, 2017

    2017 arXiv

  10. [18]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Aishwarya Agrawal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. International Journal of Computer Vision, page 398–414, 2019

    2019

  11. [19]

    Visual programming: Compositional visual reasoning without training

    Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14953–14962, 2023

    2023

  12. [20]

    GREC: Generalized referring expression comprehension.arXiv preprint arXiv:2308.16182, 2023

    Shuting He, Henghui Ding, Chang Liu, and Xudong Jiang. GREC: Generalized referring expression comprehension.arXiv preprint arXiv:2308.16182, 2023

    2023 arXiv

  13. [21]

    spaCy: Industrial-strength Natural Language Processing in Python

    Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial-strength Natural Language Processing in Python. 2020

    2020

  14. [22]

    Visual program distillation: Distilling tools and programmatic reasoning into vision-language models

    Yushi Hu, Otilia Stretcu, Chun-Ta Lu, Krishnamurthy Viswanathan, Kenji Hata, Enming Luo, Ranjay Krishna, and Ariel Fuxman. Visual program distillation: Distilling tools and programmatic reasoning into vision-language models. InProceedings of the IEEE/CVF Conference on Computer...

    2024

  15. [23]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019

    2019

  16. [24]

    Vcoder: Versatile vision encoders for multimodal large lan- guage models

    Jitesh Jain, Jianwei Yang, and Humphrey Shi. Vcoder: Versatile vision encoders for multimodal large lan- guage models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27992–28002, 2024

    2024

  17. [25]

    Clevr: A diagnostic dataset for compositional language and elementary visual reasoning

    Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognitio...

    2017

  18. [26]

    Dvqa: Understanding data visualizations via question answering

    Kushal Kafle, Scott Cohen, Brian Price, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. InCVPR, 2018

    2018

  19. [27]

    Mdetr-modulated detection for end-to-end multi-modal understanding

    Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. InProceedings of the IEEE/CVF international conference on computer vision, pages 1780–1790, 2021

    2021

  20. [28]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. InComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 235–251. S...

    2016

  21. [29]

    Ocr-free document understanding transformer

    Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. InEuropean Conference on Computer Vision (ECCV), 2022

    2022

  22. [30]

    Large language models are zero-shot reasoners.Advances in neural information processing systems, 35:22199–22213, 2022

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners.Advances in neural information processing systems, 35:22199–22213, 2022. 11

    2022

  23. [31]

    Shamma, Michael S

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations.Int...

    2017

  24. [32]

    Scaffolding coordinates to promote vision-language coordination in large multi-modal models.arXiv preprint arXiv:2402.12058, 2024

    Xuanyu Lei, Zonghan Yang, Xinrui Chen, Peng Li, and Yang Liu. Scaffolding coordinates to promote vision-language coordination in large multi-modal models.arXiv preprint arXiv:2402.12058, 2024

    2024 arXiv

  25. [33]

    Seed-bench: Benchmarking multimodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023

    2023 arXiv

  26. [34]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  27. [35]

    Evaluating object hallucination in large vision-language models.arXiv preprint arXiv:2305.10355, 2023

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models.arXiv preprint arXiv:2305.10355, 2023

    2023 arXiv

  28. [36]

    V ocot: Unleashing visually grounded multi-step reasoning in large multi-modal models.arXiv preprint arXiv:2405.16919, 2024

    Zejun Li, Ruipu Luo, Jiwen Zhang, Minghui Qiu, and Zhongyu Wei. V ocot: Unleashing visually grounded multi-step reasoning in large multi-modal models.arXiv preprint arXiv:2405.16919, 2024

    2024 arXiv

  29. [37]

    Monkey: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit...

    2024

  30. [38]

    Openorca: An open dataset of gpt augmented flan reasoning traces

    Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet V ong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces. https://https://huggingface.co/ Open-Orca/OpenOrca, 2023

    2023

  31. [39]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InECCV, 2014

    2014

  32. [40]

    Sphinx-x: Scaling data and parameters for a family of multi-modal large language models

    Dongyang Liu, Renrui Zhang, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, Kaipeng Zhang, et al. Sphinx-x: Scaling data and parameters for a family of multi-modal large language models. InForty-first International Conference on Machine Learning

  33. [41]

    Visual spatial reasoning.Transactions of the Association for Computational Linguistics, 11:635–651, 2023

    Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning.Transactions of the Association for Computational Linguistics, 11:635–651, 2023

    2023

  34. [42]

    Improved baselines with visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023

    2023

  35. [43]

    Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    2024

  36. [44]

    Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    2024

  37. [45]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection.arXiv preprint arXiv:2303.05499, 2023

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection.arXiv preprint arXiv:2303.05499, 2023

    2023 arXiv

  38. [46]

    Mmbench: Is your multi-modal model an all-around player?arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player?arXiv preprint arXiv:2307.06281, 2023

    2023 arXiv

  39. [47]

    Sgdr: Stochastic gradient descent with warm restarts.arXiv preprint arXiv:1608.03983, 2016

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts.arXiv preprint arXiv:1608.03983, 2016

    2016 arXiv

  40. [48]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  41. [49]

    Deepseek-vl: towards real-world vision-language understanding.arXiv preprint arXiv:2403.05525, 2024

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseek-vl: towards real-world vision-language understanding.arXiv preprint arXiv:2403.05525, 2024

    2024 arXiv

  42. [50]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. InThe 36th Conference on Neural Information Processing Systems (...

    2022

  43. [51]

    Wizardcoder: Empowering code large language models with evol-instruct

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568, 2023

    2023 arXiv

  44. [52]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. InProceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019

    2019

  45. [53]

    ChartQA: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. InFindings of the Association for Computational Linguistics: ACL 2022, pages 2263–2279, Dublin, Ireland, 2022. Associa...

    2022

  46. [54]

    Docvqa: a dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, R Manmatha, and CV Jawahar. Docvqa: a dataset for vqa on document images. corr abs/2007.00398 (2020).arXiv preprint arXiv:2007.00398, 2020

    2020 arXiv

  47. [55]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697–1706, 2022

    2022

  48. [56]

    Schema theory revisited.Review of educational research, 75(4):531–566, 2005

    Mary B McVee, Kailonnie Dunsmore, and James R Gavelek. Schema theory revisited.Review of educational research, 75(4):531–566, 2005

    2005

  49. [57]

    Ocr-vqa: Visual question answering by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. InICDAR, 2019

    2019

  50. [58]

    Compositional chain of thought prompting for large multimodal models

    Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain of thought prompting for large multimodal models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  51. [59]

    Modeling context between objects for referring expression understanding

    Varun K Nagaraja, Vlad I Morariu, and Larry S Davis. Modeling context between objects for referring expression understanding. InComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 792–807. Springer, 2016

    2016

  52. [60]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023

    2023

  53. [61]

    Chatgpt: A large language model for natural language processing, 2024

    OpenAI. Chatgpt: A large language model for natural language processing, 2024. Accessed via OpenAI platform. URL:https://chat.openai.com/

    2024

  54. [62]

    Learning to predict visual attributes in the wild

    Khoi Pham, Kushal Kafle, Zhe Lin, Zhihong Ding, Scott Cohen, Quan Tran, and Abhinav Shrivastava. Learning to predict visual attributes in the wild. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13018–13028, 2021

    2021

  55. [63]

    Plummer, Liwei Wang, Chris M

    Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.International Journal of Computer Vision, page 74–93, 2017

    2017

  56. [64]

    Jack of all tasks master of many: Designing general-purpose coarse-to-fine vision-language model

    Shraman Pramanick, Guangxing Han, Rui Hou, Sayan Nag, Ser-Nam Lim, Nicolas Ballas, Qifan Wang, Rama Chellappa, and Amjad Almahairi. Jack of all tasks master of many: Designing general-purpose coarse-to-fine vision-language model. InProceedings of the IEEE/CVF Conference on Com...

    2024

  57. [65]

    Cogcom: Train large vision-language models diving into details through chain of manipulations.arXiv preprint arXiv:2402.04236, 2024

    Ji Qi, Ming Ding, Weihan Wang, Yushi Bai, Qingsong Lv, Wenyi Hong, Bin Xu, Lei Hou, Juanzi Li, Yuxiao Dong, et al. Cogcom: Train large vision-language models diving into details through chain of manipulations.arXiv preprint arXiv:2402.04236, 2024

    2024 arXiv

  58. [66]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  59. [67]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. InProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 35...

    2020

  60. [68]

    Faster r-cnn: Towards real-time object detection with region proposal networks.Advances in neural information processing systems, 28, 2015

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks.Advances in neural information processing systems, 28, 2015. 13

    2015

  61. [69]

    A- okvqa: A benchmark for visual question answering using world knowledge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A- okvqa: A benchmark for visual question answering using world knowledge. InEuropean conference on computer vision, pages 146–162. Springer, 2022

    2022

  62. [70]

    Visual cot: Unleashing chain-of-thought reasoning in multi-modal language models.arXiv preprint arXiv:2403.16999, 2024

    Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Unleashing chain-of-thought reasoning in multi-modal language models.arXiv preprint arXiv:2403.16999, 2024

    2024 arXiv

  63. [71]

    Objects365: A large-scale, high-quality dataset for object detection

    Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 8429–8438, 2019

    2019

  64. [72]

    Woodpecker: Hallucination correction for multimodal large language models.arXiv preprint arXiv:2310.16045, 2023

    Yin Shukang, Fu Chaoyou, Zhao Sirui, Xu Tong, Wang Hao, Sui Dianbo, Shen Yunhang, Li Ke, Sun Xing, and Chen Enhong. Woodpecker: Hallucination correction for multimodal large language models.arXiv preprint arXiv:2310.16045, 2023

    arXiv

  65. [73]

    Textcaps: a dataset for image captioning with reading comprehension

    Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 742–758. Springer, 2020

    2020

  66. [74]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019

    2019

  67. [75]

    Vipergpt: Visual inference via python execution for reasoning

    Dídac Surís, Sachit Menon, and Carl V ondrick. Vipergpt: Visual inference via python execution for reasoning. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 11888– 11898, 2023

    2023

  68. [76]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  69. [77]

    Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024

    2024 arXiv

  70. [78]

    Qwen2.5: A party of foundation models, 2024

    Qwen Team. Qwen2.5: A party of foundation models, 2024

    2024

  71. [79]

    To see is to believe: Prompting gpt-4v for better visual instruction tuning.arXiv preprint arXiv:2311.07574, 2023

    Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompting gpt-4v for better visual instruction tuning.arXiv preprint arXiv:2311.07574, 2023

    2023 arXiv

  72. [80]

    V3det: Vast vocabulary visual detection dataset

    Jiaqi Wang, Pan Zhang, Tao Chu, Yuhang Cao, Yujie Zhou, Tong Wu, Bin Wang, Conghui He, and Dahua Lin. V3det: Vast vocabulary visual detection dataset. InThe IEEE International Conference on Computer Vision (ICCV), 2023

    2023

  73. [81]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  74. [82]

    Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  75. [83]

    V?: Guided visual search as a core mechanism in multimodal llms

    Penghao Wu and Saining Xie. V?: Guided visual search as a core mechanism in multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13084– 13094, 2024

    2024

  76. [84]

    Universal instance perception as object discovery and retrieval

    Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Zehuan Yuan, Ping Luo, and Huchuan Lu. Universal instance perception as object discovery and retrieval. InCVPR, 2023

    2023

  77. [85]

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023

    2023 arXiv

  78. [86]

    Tree of thoughts: Deliberate problem solving with large language models.Advances in Neural Information Processing Systems, 36, 2024

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models.Advances in Neural Information Processing Systems, 36, 2024

    2024

  79. [87]

    Ferret: Refer and ground anything anywhere at any granularity

    Haoxuan You and et al. Ferret: Refer and ground anything anywhere at any granularity. InICLR, 2024. 14

    2024

  80. [88]

    Ferret-ui: Grounded mobile ui understanding with multimodal llms.arXiv preprint arXiv:2404.05719, 2024

    Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. Ferret-ui: Grounded mobile ui understanding with multimodal llms.arXiv preprint arXiv:2404.05719, 2024

    2024 arXiv

  81. [89]

    Modeling context in referring expressions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. InComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69–85. Springer, 2016

    2016

  82. [90]

    Metamath: Bootstrap your own mathematical questions for large language models.arXiv preprint arXiv:2309.12284, 2023

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models.arXiv preprint arXiv:2309.12284, 2023

    2023 arXiv

  83. [91]

    Osprey: Pixel understanding with visual instruction tuning

    Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28202–28211, 2024

    2024

  84. [92]

    Mam- moth: Building math generalist models through hybrid instruction tuning.arXiv preprint arXiv:2309.05653, 2023

    Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mam- moth: Building math generalist models through hybrid instruction tuning.arXiv preprint arXiv:2309.05653, 2023

    2023 arXiv

  85. [93]

    Griffon: Spelling out all object locations at any granularity with large language models

    Yufei Zhan, Yousong Zhu, Zhiyang Chen, Fan Yang, Ming Tang, and Jinqiao Wang. Griffon: Spelling out all object locations at any granularity with large language models. InECCV, 2024

    2024

  86. [94]

    Automatic chain of thought prompting in large language models.arXiv preprint arXiv:2210.03493, 2022

    Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models.arXiv preprint arXiv:2210.03493, 2022

    2022 arXiv

  87. [95]

    yes” or “no

    Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou, and Sibei Yang. Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models.Advances in Neural Information Processing Systems, 36:5168–5191, 2023. 15 Appendix Contents A Details of Expert-Supervised...

    2023

  88. [96]

    <image> is that <label id=3 relation-id=1>a team hansen car</label>?

  89. [98]

    which company is the <label id=2 relation-id=2>silver truck</label> for?

  90. [100]

    <label id=4 relation-id=3>hansen</label> Preview

  91. [101]

    <image> is that a team hansen car？

  92. [102]

    Following objects to be focused on: Team Hansen car

  93. [103]

    which company is the silver truck for?

  94. [104]

    Following objects to be focused on: silver truck

  95. [105]

    hansen Correlations <label relation-id=1 >a team hansen car </label> <label relation-id=2 >silver truck </label> <label relation-id=3 >hansen </label> 2 13 Figure 6: An example UI screenshot showcasing the annotation process for a question-answer pair requiring grounding. Afte...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.