Pith. sign in

REVIEW 5 major objections 6 minor 5 cited by

Hidden in plain sight: VLMs overlook their visual representations

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Open-source VLMs perform near chance on vision tasks that their own encoders solve, and the paper identifies the language model as the bottleneck.

desk verdict A useful diagnostic with a plausible but not fully controlled central comparison; the privileged visual readouts and supervised depth head soften the LLM-bottleneck claim, but the paper deserves serious review. read the letter →

arxiv 2506.08008 v1 pith:7D6UCQRC submitted 2025-06-09 cs.CV cs.AIcs.LG

Stephanie Fu , Tyler Bonnen , Devin Guillory , Trevor Darrell This is my paper
classification cs.CVcs.AIcs.LG
keywords vision-languagemodelsvisualrepresentationsvision-centricbenchmarkslanguagepriorsLLMbottleneckprobingVQAevaluationDINOv2
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish why open-source vision-language models fail at purely visual tasks: they are not using the visual information their own encoders already contain. Across six vision-centric benchmarks, querying the full VLM drops performance from near-ceiling to near chance, while direct readouts of the same encoder features solve the tasks. The authors rule out representation degradation and prompt sensitivity, and pin the failure on the language model, which under-attends to relevant image regions and leans on answer biases inherited from text-only pretraining. If the claim is right, VLM benchmark scores badly underestimate the visual abilities of their encoders, and improving visual tasks means improving how the LLM consumes visual features.

What carries the argument

The central device is the paired evaluation: a direct visual readout of a VLM's frozen vision encoder versus a VQA-style query of the full VLM on the same task. The readout methods are cosine similarity between patch features at manually located keypoints for correspondence tasks, Gram-matrix second-moment matching of patch features for art style, pairwise CLS-token cosine similarity for 3D object awareness, and a DPT dense-prediction head trained on NYUv2 for depth. To separate candidate bottlenecks, the authors probe task accuracy at intermediate projector and LLM layers, tune learnable prompt prefixes, and fine-tune each component with matched parameter counts. These probes establish that the information is present throughout the model, that prompt wording is not the failure, and that only the LLM's use of the visual tokens moves performance.

What would settle it

Run the six tasks on a VLM with native multi-image input and full resolution, avoiding letterbox stitching and 224-pixel resizing, and compare against the same direct visual readouts: if the gap mostly closes, input formatting rather than LLM underuse is the cause. Alternatively, train the same readout probes on the actual token sequence the LLM receives, after the projector and input embedding layers; if those probes also fail, the information is not easily accessible to any decoder and the bottleneck is earlier than the LLM.

Watch

Extended reading notes

Core claim

The paper's central claim is that open-source VLMs systematically discard visual information that their own vision encoders make readily available. On six vision-centric tasks—depth ordering, semantic correspondence, low-level matching, object affordance, art style, and 3D object awareness—the same image features that support near-ceiling accuracy when read out directly support only near-chance accuracy when the VLM answers in text. The loss is not explained by the vision encoder degrading inside the VLM, because probing intermediate projector and LLM layers still recovers task-relevant signal; nor by prompt sensitivity, because learnable prompt prefixes yield only marginal, diminishing gains. Equal-parameter fine-tuning shows the bottleneck is the language model: tuning the LLM improves accuracy and moves answer distributions toward ground truth, while tuning the projector or ViT does little. The paper concludes that VLMs inherit the answer biases of their language backbones and under-attend to the visual regions that carry the task.

Load-bearing premise

The central claim depends on the hand-built visual readouts being a fair upper bound on what the language model could extract from the same encoder features; if stitching images, lower resolution, or reading red-circled labels makes the language-model path harder in ways the probes skip, the gap would not prove the language model is the bottleneck.

Editorial extensions

If this is right

  • VLM benchmark rankings should not be read as rankings of vision encoders' visual abilities; a strong encoder can score best on direct readout yet worst inside a VLM.
  • Improving vision-centric VLM performance will require changing how the LLM consumes visual tokens, for example by increasing attention to salient regions, rather than only swapping or ensembling vision backbones.
  • Task-level fine-tuning of the LLM reduces the influence of the language prior and brings multiple-choice answer distributions closer to ground truth, while fine-tuning the vision encoder or projector does not.
  • Prompt engineering with learnable prefixes is not a cure: one or a few prefixes give only small gains, and additional prefixes do not keep improving accuracy.
  • Because representations stay informative through the projector and LLM layers, failure is a read-out problem rather than a storage problem: the final answer can be wrong even when the model still contains the right answer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the blind-baseline comparison suggests a cheap diagnostic—run the same prompt with a blank image; high overlap between the two answer distributions flags tasks where the model is not looking at the image.
  • Editorial extension: if the LLM is the bottleneck, holding the LLM fixed and varying only the encoder should barely move vision-centric accuracy, a direct test of the paper's attribution.
  • Editorial extension: the paired protocol could be reused to evaluate future methods; a method that raises VQA accuracy while the direct-readout ceiling stays fixed is genuinely improving visual-information use.
  • Editorial extension: the visual readouts are one access path, so the claim is best read as 'easily accessible information under these probes,' not a proof that no LLM could ever extract more from the same features.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper compares a VQA-style evaluation of open-source VLMs to 'direct readouts' of their (identical) frozen vision encoders across six vision-centric tasks: depth estimation, semantic correspondence, object affordance, low-level matching, art style, and 3D object awareness. It reports a universal and often near-chance drop in VLM accuracy relative to the visual readouts, shows that task-relevant visual information remains probeable in projector and LLM layers, finds that prompt prefix tuning gives only marginal gains, and concludes from component-wise LoRA fine-tuning that the LLM is the bottleneck because fine-tuning the LLM improves accuracy the most and also reduces total-variation distance to the ground-truth answer distribution. The paper further documents that VLMs' answer distributions resemble their blind (no-image) baselines, and that the rank ordering of vision encoders changes between visual and VLM evaluations.

Significance. If the central claim survives scrutiny, the paper is significant: it calls into question the practice of ranking vision encoders through VLM-based benchmarks and redirects attention to the visual-linguistic integration stage as the main source of failure on perception-centric tasks. The study has notable strengths: it uses a controlled Prismatic VLM suite in which the vision encoder weights are identical to the standalone backbones, it provides several converging analyses (layer probing, blind baselines, prompt tuning, component fine-tuning), and it builds on external benchmarks rather than self-constructed ones. The attention-map after fine-tuning and the TV-distance analysis are useful diagnostic tools. However, the headline numerical claim depends on treating the direct visual readouts as a fair upper bound on what the VLM could extract from the same representations, and that bound is not yet established due to differences in input presentation, task-specific decoder training, and privileged information in the visual probes.

major comments (5)
  1. [Sec. 2.1, Appendix C.1] The depth-estimation visual baseline trains a DPT decoder on NYUv2 for 10 epochs and then compares box-averaged depths, whereas the VLM is evaluated zero-shot with a text prompt. A supervised, task-specific decoder is not 'easily accessible' to a frozen LLM, so the gap in Table 1 (e.g., 88.7% vs. 64.0% for DINOv2) conflates decoder supervision with representation underuse. The paper should either provide a zero-shot or frozen visual readout for depth, or explicitly treat the DPT-trained baseline as an upper bound on task-relevant information rather than as a readout of 'easy access'. Without this control, the depth result cannot support the claim that the information is readily available to the LLM.
  2. [Sec. 2.2, Appendix C.2] The correspondence visual baselines are given the ground-truth image coordinates of the REF, A, B, C, and D points and sample patch features at exactly those floating-point locations. The VLM, in contrast, must internally localize the red circles and text labels in a stitched, letterboxed 224px image while also parsing the multiple-choice letters and reading the REF annotation. This asymmetry injects privileged spatial information into the visual probe: the baseline is not input-matched to the VLM. A fairer comparison would either remove the coordinate-location step from the visual probe (e.g., a patch-similarity readout over the whole image or over detected candidates) or provide the VLM with a task format that does not require text parsing and stitching. As written, the large drops on Semantic Correspondence, Object Affordance, and Low-Level Matching in Fig. 2 may be partially an artifact of the VLM's harder input format.
  3. [Sec. 3.2, Fig. 2, Table 1] The central visual-vs-VLM accuracy comparisons are reported as point estimates without error bars, confidence intervals, or significance tests. Several task accuracies for the VLMs are near chance (e.g., 0.225 for DINOv2 semantic correspondence vs. chance 0.25), and the claimed 'universal drop' rests on differences that could in places be within sampling variability. The paper should report bootstrap or per-item confidence intervals over benchmark examples (and, where applicable, repeated prompt variants) for the main comparisons in Table 1 and Figs. 2 and 3. This is load-bearing because near-chance claims and the rank-order shift in Sec. 3.3 are both point estimates.
  4. [Sec. 4.3, Fig. 7] The conclusion that the LLM is the bottleneck relies on comparing LoRA fine-tuning of the ViT, projector, and LLM with 'equal parameter counts' (16.7M). Equal parameter count does not imply equal optimization capacity: LoRA on a 7B LLM and LoRA on a ViT-L differ in the matrices being adapted, in the effective rank used by each module, and in the pretraining objectives of those modules. The larger improvement from LLM tuning could stem from the LLM having more learnable capacity to absorb the task-specific mapping, rather than from the original LLM being a uniquely underutilizing component. Additionally, task-conditional fine-tuning on the same datasets used for evaluation demonstrates an upper bound on what that component can learn, not what the frozen VLM should already be doing. A control that varies training steps, total FLOPs, or the amount of visual information provided to the LLM would strengthen the bottleneck attribution.
  5. [Sec. 3.4, Fig. 4] The blind-baseline analysis is suggestive but is used to claim that VLMs 'ignore images'. The total-variation distances in Table 2 show that the original VLM's answer distribution is close to its blind counterpart (e.g., 0.337 vs 0.399 for semantic correspondence), yet these are aggregate distributions over all items; a model that attends to the image but has a strong letter-answer prior could produce a similar aggregate distribution. Reporting per-item agreement between the with-image and no-image predictions, not just marginal distributions, would directly support the 'ignore images' interpretation. As it stands, the evidence indicates inherited answer biases, but the stronger claim that visual input is not used requires a per-item measure.
minor comments (6)
  1. [Sec. 3.1, Table 1 reference] The text refers to 'Fig. 2 and Tab. 3' when reporting visual evaluation results, but the relevant table is numbered Table 1. Please correct the cross-reference.
  2. [Sec. 3.4 and Sec. 4.3.1, Table 2] The text refers to 'Table 4.3.1' when discussing TV distances, but the table is numbered Table 2. Please fix the in-text citations.
  3. [Sec. 2.3] Typo: 'differnet' should read 'different' in the MOCHI task description.
  4. [Fig. 4 caption] Typo: 'misuing' should read 'misusing' in the caption: 'VLMs are not simply misuing their visual representations'.
  5. [Discussion] Typo: 'there there are still improvements needed' should read 'there are still improvements needed'.
  6. [Appendix C.5] The few-shot results report 'mean and standard deviation (which is zero)' for the vision model; the source of the zero standard deviation should be stated explicitly, otherwise readers may suspect a reporting error.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation identified; the visual-readout gap is an empirical comparison with external benchmarks, and the probe asymmetries are validity limitations rather than definitional or fitted-input circularity.

full rationale

The paper's central claim is comparative: direct readouts of VLM vision encoders outperform the corresponding VLM on vision-centric benchmarks, and component fine-tuning implicates the LLM. No equation in the paper defines the VLM result in terms of the visual readout; the readouts are separate evaluation protocols (cosine similarity at located keypoints, Gram matrices, CLS similarity, and a DPT head) rather than parameters fitted to the VLM output. The conclusions therefore do not reduce to their inputs by construction. The strongest circularity concern is that the visual 'ceiling' is not input-matched or decoder-matched: Appendix C.2 gives the visual probe the REF/A/B/C/D coordinates, Appendix C.1 trains a supervised DPT head on NYUv2, and the VLM is evaluated zero-shot on stitched 224px inputs. These asymmetries weaken the causal attribution 'LLM underuses easily accessible visual information,' but they are evaluation-design limitations, not a fitted parameter renamed as a prediction or a self-citation chain that forces the result. Self-citations (BLINK, MOCHI) supply benchmark data with independent provenance and do not carry a uniqueness or ansatz argument. No specific reduction of Eq. X to Eq. Y, or of a fitted value to a predicted value, can be quoted, so the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No new entities or mediators are postulated; the paper is an empirical diagnostic and introduces no new physical or architectural primitives. The main assumptions are about the validity of probes and baselines rather than about new mechanisms.

free parameters (3)
  • DPT depth head weights = Trained on NYUv2, 10 epochs, AdamW lr 5e-4
    Task-specific supervised head for the depth visual baseline; not zero-shot, so it may overstate the visual evaluation ceiling for the depth task.
  • LoRA adapter parameters = 16.7M parameters per component per task
    Used to compare fine-tuning ViT vs projector vs LLM; conclusions about the LLM bottleneck depend on this equal-parameter-count recipe, which is trained on task labels.
  • Prompt prefix embeddings = 1, 5, and 10 learnable prefixes
    Used for prompt-sensitivity analysis; results show marginal gains and do not feed the central claim, but are fitted to task data.
assumptions (3)
  • domain assumption Direct visual readouts (cosine similarity, Gram matrices, DPT head, CLS similarity) are a valid upper bound on task-relevant information accessible from the vision representations.
    The entire comparison treats the gap between these readouts and VLM accuracy as evidence of VLM integration failure; if these probes are not faithful measures, the gap could be a methodological artifact. Entered in Sections 2 and 3.
  • domain assumption Similarity between with-vision and blank-vision answer distributions implies the VLM ignores visual input and inherits LLM biases.
    TV distance near zero is interpreted causally; alternative explanations (e.g., prompt format dominating answers, or blank input not fully disabling vision) are not ruled out. Section 3.4, Table 2.
  • ad hoc to paper Larger gains from fine-tuning the LLM, compared to projector or ViT with equal parameter counts, identify the original LLM as the inference-time bottleneck.
    Fine-tuning measures adaptability under supervision, not necessarily the original model's failure point; the inference relies on an additional assumption. Section 4.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hidden in plain sight: VLMs overlook their visual representations." pith.science (2026). https://pith.science/paper/7D6UCQRC

@misc{pith2026250608008,
  author       = {Pith},
  title        = {Pith review of: Hidden in plain sight: VLMs overlook their visual representations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7D6UCQRC}},
  note         = {Machine review of arXiv:2506.08008}
}
read the original abstract

Language provides a natural interface to specify and evaluate performance on visual tasks. To realize this possibility, vision language models (VLMs) must successfully integrate visual and linguistic information. Our work compares VLMs to a direct readout of their visual encoders to understand their ability to integrate across these modalities. Across a series of vision-centric benchmarks (e.g., depth estimation, correspondence), we find that VLMs perform substantially worse than their visual encoders, dropping to near-chance performance. We investigate these results through a series of analyses across the entire VLM: namely 1) the degradation of vision representations, 2) brittleness to task prompt, and 3) the language model's role in solving the task. We find that the bottleneck in performing these vision-centric tasks lies in this third category; VLMs are not effectively using visual information easily accessible throughout the entire model, and they inherit the language priors present in the LLM. Our work helps diagnose the failure modes of open-source VLMs, and presents a series of evaluations useful for future investigations into visual understanding within VLMs.

Figures

Figures reproduced from arXiv: 2506.08008 by the authors.

Figure 1
Figure 1. Evaluating vision language models (VLMs) alongside their vision encoders reveals a failure to utilize visual information. To assess VLMs’ visual abilities, we compare their performance to the accuracy supported by a direct readout of their visual encoders. Using ‘vision-centric’ tasks (e.g., visual correspondence), we compare typical VQA-style VLM evaluation (center, bottom) with vision-only methods (center, top). A… view at source ↗
Figure 2
Figure 2. Comparing standard visual evaluation to VLMs across vision-centric tasks. Shifting from a standard vision evaluation strategy to a VLM evaluation results in a performance drop, often to chance-level accuracies. Additionally, the vision encoders that perform best at a task (often DINOv2) are not the same vision encoders in more performant VLMs. Tong et al. (2024a). Nonetheless, the behavior of VLMs in vision-centric … view at source ↗
Figure 3
Figure 3. We find the same trends as in [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: VLM choice behavior reflects the biases of their LLMs. Here we visualize the distribution of answers when models are presented with (blue) and without (orange) a valid image. We find that behaviors largely reflect the pattern of choices in the blind baselines. We take …
Figure 5
Figure 5. Figure 5: Visual evaluations for intermediate VLM layers. We probe vision representations throughout the projector (gray region) and LLM (white region) layers, finding that they generally preserve task-relevant information and show no significant degradation. We highlight two in…
Figure 6
Figure 6. Figure 6: Prompt-tuning evaluation. We tune [1, 5, 10] prefix embeddings and compare results with the original performance (x=0) and visual evaluation ceiling (dotted line). We observe minimal returns that diminish after 1-5 prefix embeddings. If vision representations are not t…
Figure 7
Figure 7. Figure 7: We find that tuning the LLM (as opposed to the projector layers or the ViT) on each task, with the same parameter count in each setting, provides the largest performance increase. Taken together with [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Visualizing the difference between atten￾tion maps before and after fine-tuning the LLM, we observe an increase in attention at the points of inter￾est (REF, A, B, C, D) for correspondence tasks. These points are most salient in attention layers 4-6; here we visualize …
Figure 9
Figure 9. Figure 9: Example failure cases for VLMs (left), vision encoders (center), and both (right). We observe a few common failure modes for both evaluation strategies: on correspondence-based tasks, similar local structure may confuse the model (see Low-level Matching: stone wall and…
Figure 10
Figure 10. Figure 10: Two examples of ‘neural style transfer’: starting with the original image, we [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Attention difference visualizations for Functional Correspondence (top two) and [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Which point in the image (right) best matches the semantics of the point in the reference image (left)? Additional examples of VLM-only, vision encoder-only, and VLM + vision encoder failures on the Semantic Correspondence task. 7 [PITH_FULL_IMAGE:figures/full_fig_p0…
Figure 13
Figure 13. Figure 13: Which point in the image (right) best matches to the reference point in the same scene (left)? Additional examples of VLM-only, vision encoder-only, and VLM + vision encoder failures on the Low-level Matching task. 8 [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: Which bounding box contains the object closer to the camera? Additional examples of VLM-only, vision encoder-only, and VLM + vision encoder failures on the Depth Estimation task. 9 [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Which point in the image (right) best matches the function of the point in the reference image (left)? Additional examples of VLM-only, vision encoder-only, and VLM + vision encoder failures on the Object Affordances task. Ground Truth Visual answer VLM answer Visual …
Figure 16
Figure 16. Figure 16: Which images (center or right) best matches the art style of the reference image (left)? Additional examples of VLM-only, vision encoder-only, and VLM + vision encoder failures on the Art Style task. 10 [PITH_FULL_IMAGE:figures/full_fig_p023_16.png]
Figure 17
Figure 17. Figure 17: Which image contains the odd-object-out? Additional examples of VLM-only, vision encoder-only, and VLM + vision encoder failures on the 3D Object Awareness task. 11 [PITH_FULL_IMAGE:figures/full_fig_p024_17.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Vision Language Models Cannot Reason About Physical Transformation

    cs.AI 2026-03 accept novelty 6.5 of 10

    Current VLMs cannot maintain transformation-invariant representations of number, length, volume or size and instead rely on textual invariance priors that reverse on matched non-conserving controls.

  2. Teaching MLLMs to Say No: Generalized Referring Expression Comprehension via Refusal Calibrated GRPO

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A recalibrated GRPO reinforcement learning method lets multimodal LLMs say 'None' for nonexistent referring expressions without sacrificing localization accuracy on objects that do exist.

  3. Visual Access Boundaries in Vision-Language Model Reasoning

    cs.AI 2026-07 conditional novelty 6.0 of 10

    CoT in VLMs extends language-side computation over early image-derived states rather than prolonging direct image-token access, with gains limited by perceptual readout reliability.

  4. Egocentric Bias in Vision-Language Models

    cs.CV 2026-02 conditional novelty 6.0 of 10

    Most vision-language models fail Level-2 visual perspective taking: they report the camera's view rather than the 180°-rotated string, even though they often recognize that another agent sees differently.

  5. PictOBI-20k: Unveiling Large Multimodal Models in Visual Decipherment for Pictographic Oracle Bone Characters

    cs.CV 2025-09 conditional novelty 6.0 of 10

    PictOBI-20k, a new 15k-question benchmark, shows top large multimodal models reach only 53.7% accuracy at matching oracle bone pictographs to object photos, with vision encoders often outperforming the full models.

Reference graph

Works this paper leans on

42 extracted references · 5 canonical work pages · cited by 5 Pith papers

  1. [1]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matt...

    arXiv 2024

  2. [2]

    Flamingo: a Visual Language Model for Few - Shot Learning , November 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski,...

    arXiv 2022

  3. [3]

    Openflamingo: An open-source framework for training large autoregressive vision-language models, 2023

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open-source framework for training large autoregressive vision-language models, 2023. URL https://...

    arXiv 2023

  4. [4]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023. URL https://arxiv.org/abs/2308.12966

    arXiv 2023

  5. [5]

    Hpatches: A benchmark and evaluation of handcrafted and learned local descriptors

    Vassileios Balntas, Karel Lenc, Andrea Vedaldi, and Krystian Mikolajczyk. Hpatches: A benchmark and evaluation of handcrafted and learned local descriptors. In CVPR, 2017

    work page 2017

  6. [6]

    Probing the 3D Awareness of Visual Foundation Models , April 2024

    Mohamed El Banani, Amit Raj, Kevis-Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3D Awareness of Visual Foundation Models , April 2024. URL http://arxiv.org/abs/2404.08636. arXiv:2404.08636 [cs]

    arXiv 2024

  7. [7]

    Paligemma: A versatile 3b vlm for transfer, 2024

    Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bau...

    arXiv 2024

  8. [8]

    Tenenbaum, and Alexei A

    Tyler Bonnen, Stephanie Fu, Yutong Bai, Thomas O'Connell, Yoni Friedman, Nancy Kanwisher, Joshua B. Tenenbaum, and Alexei A. Efros. Evaluating multiview object consistency in humans and image models, 2024. URL https://arxiv.org/abs/2409.05862

    arXiv 2024

Show all 42 references
  1. [9]

    Omni3d: A large benchmark and model for 3d object detection in the wild, 2023

    Garrick Brazil, Abhinav Kumar, Julian Straub, Nikhila Ravi, Justin Johnson, and Georgia Gkioxari. Omni3d: A large benchmark and model for 3d object detection in the wild, 2023. URL https://arxiv.org/abs/2207.10660

    2023 arXiv

  2. [10]

    Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu

    Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shapenet: An information-rich 3d model repository, 2015. URL https://arxiv.org/abs/1512.03012

    2015 arXiv

  3. [11]

    An empirical study of training self-supervised vision transformers, 2021

    Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers, 2021. URL https://arxiv.org/abs/2104.02057

    2021 arXiv

  4. [12]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vis...

    2024

  5. [13]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90\ URL https://lmsys.org/blog/2023-03-30-vicuna/

    2023

  6. [14]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp.\ 248--255, 2009. doi:10.1109/CVPR.2009.5206848

    2009

  7. [15]

    An Image is Worth 16x16 Words : Transformers for Image Recognition at Scale , June 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words : Transformers for Image Recognition a...

    2021 arXiv

  8. [16]

    Mousi: Poly-visual-expert vision-language models, 2024

    Xiaoran Fan, Tao Ji, Changhao Jiang, Shuo Li, Senjie Jin, Sirui Song, Junke Wang, Boyang Hong, Lu Chen, Guodong Zheng, Ming Zhang, Caishuang Huang, Rui Zheng, Zhiheng Xi, Yuhao Zhou, Shihan Dou, Junjie Ye, Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and...

    2024 arXiv

  9. [17]

    Smith, Wei-Chiu Ma, and Ranjay Krishna

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. BLINK : Multimodal Large Language Models Can See but Not Perceive , April 2024. URL http://arxiv.org/abs/2404.12390. arXiv:2404.12390 [cs]

    2024 arXiv

  10. [18]

    Gatys, Alexander S

    Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. A neural algorithm of artistic style. ArXiv, abs/1508.06576, 2015. URL https://api.semanticscholar.org/CorpusID:13914930

    2015 arXiv

  11. [19]

    Prismatic VLMs : Investigating the Design Space of Visually - Conditioned Language Models , May 2024

    Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic VLMs : Investigating the Design Space of Visually - Conditioned Language Models , May 2024. URL http://arxiv.org/abs/2402.07865. arXiv:2402.07865 [cs]

    2024 arXiv

  12. [20]

    The functional correspondence problem

    Zihang Lai, Senthil Purushwalkam, and Abhinav Gupta. The functional correspondence problem. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 15772--15781, 2021

    2021

  13. [21]

    What matters when building vision-language models?, May 2024

    Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models?, May 2024. URL http://arxiv.org/abs/2405.02246. arXiv:2405.02246 [cs]

    2024 arXiv

  14. [22]

    The Power of Scale for Parameter - Efficient Prompt Tuning , September 2021

    Brian Lester, Rami Al-Rfou, and Noah Constant. The Power of Scale for Parameter - Efficient Prompt Tuning , September 2021. URL http://arxiv.org/abs/2104.08691. arXiv:2104.08691 [cs]

    2021 arXiv

  15. [23]

    Improved Baselines with Visual Instruction Tuning , October 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved Baselines with Visual Instruction Tuning , October 2023. URL http://arxiv.org/abs/2310.03744. arXiv:2310.03744 [cs]

    2023 arXiv

  16. [24]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https://llava-vl.github.io/blog/2024-01-30-llava-next/

    2024

  17. [25]

    Transformer-based neural texture synthesis and style transfer

    Jiahao Lu. Transformer-based neural texture synthesis and style transfer. In Proceedings of the 2022 4th Asia Pacific Information Technology Conference, APIT '22, pp.\ 88–95, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450395571. doi:10.1145/3512353....

    2022

  18. [26]

    Spair-71k: A large-scale benchmark for semantic correspondence, 2019

    Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Spair-71k: A large-scale benchmark for semantic correspondence, 2019. URL https://arxiv.org/abs/1908.10543

    2019 arXiv

  19. [27]

    Indoor segmentation and support inference from rgbd images

    Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012

    2012

  20. [28]

    DINOv2 : Learning Robust Visual Features without Supervision , February 2024

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabba...

    2024 arXiv

  21. [29]

    Learning Transferable Visual Models From Natural Language Supervision , February 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision , February 2021. URL ...

    2021 arXiv

  22. [30]

    Vision Transformers for Dense Prediction

    Rene Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision Transformers for Dense Prediction . In 2021 IEEE / CVF International Conference on Computer Vision ( ICCV ) , pp.\ 12159--12168, Montreal, QC, Canada, October 2021. IEEE. ISBN 978-1-66542-812-5. doi:10.1109/ICCV48922....

    2021

  23. [31]

    Eagle: Exploring the design space for multimodal llms with mixture of encoders, 2024

    Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, Bryan Catanzaro, Andrew Tao, Jan Kautz, Zhiding Yu, and Guilin Liu. Eagle: Exploring the design space for multimodal llms with mixture of ...

    2024 arXiv

  24. [32]

    How to train your vit? data, augmentation, and regularization in vision transformers, 2022

    Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers, 2022. URL https://arxiv.org/abs/2106.10270

    2022 arXiv

  25. [33]

    Cambrian-1: A Fully Open , Vision - Centric Exploration of Multimodal LLMs , June 2024 a

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A Fully Open , Vision - Centric Exploration of Multimodal LLMs , J...

    2024 arXiv

  26. [34]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 9568--9578, 2024 b

    2024

  27. [35]

    Disn: Deep implicit surface network for high-quality single-view 3d reconstruction

    Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann. Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. In NeurIPS, 2019

    2019

  28. [36]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi, 2024

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2024 arXiv

  29. [37]

    Sigmoid Loss for Language Image Pre - Training , September 2023

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid Loss for Language Image Pre - Training , September 2023. URL http://arxiv.org/abs/2303.15343. arXiv:2303.15343 [cs]

    2023 arXiv

  30. [38]

    A General Protocol to Probe Large Vision Models for 3D Physical Understanding , June 2024

    Guanqi Zhan, Chuanxia Zheng, Weidi Xie, and Andrew Zisserman. A General Protocol to Probe Large Vision Models for 3D Physical Understanding , June 2024. URL http://arxiv.org/abs/2310.06836. arXiv:2310.06836 [cs]

    2024 arXiv

  31. [39]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  32. [40]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  33. [41]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  34. [42]

    A, B, C, D

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.