Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Fine-tuning only the last four blocks of a frozen DINOv2, with a symmetric contrastive loss plus a teacher-anchoring loss, gives a single encoder that embeds RGB, depth, and segmentation of the same scene into nearly the same point.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 20:03 UTC pith:XYS6WLT2

load-bearing objection Useful post-hoc modality-alignment recipe with credible downstream support, but the headline retrieval claim rests on an unresolved contradiction about whether evaluation inputs were colorized with the paired RGB palette — leaving a color-statistics shortcut uncontrolled. the 3 major comments →

arxiv 2602.24181 v2 pith:XYS6WLT2 submitted 2026-02-27 cs.CV cs.AI

A Mixed Diet Makes DINO An Omnivorous Vision Encoder

Rishabh Kabra , Maks Ovsjanikov , Drew A. Hudson , Ye Xia , Skanda Koppula , Andre Araujo , Joao Carreira , Niloy J. Mitra This is my paper
classification cs.CV cs.AI
keywords cross-modal alignmentmodality-agnostic encoderDINOv2contrastive learningparameter-efficient fine-tuningnatural colorizationzero-shot modality transfercross-modal retrieval
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that a frozen DINOv2 backbone can be made 'omnivorous' by fine-tuning only its last four transformer blocks with a symmetric contrastive loss and a distillation-anchoring loss. The result is a single encoder whose embeddings for RGB, depth, and segmentation of the same scene are nearly identical, while different scenes stay far apart. The authors show this empirically: cross-modal retrieval on ScanNet jumps from 4.6% to 46.1% Recall@1, and a depth head trained only on RGB works when fed segmentation or NOCS maps. If true, it means foundation models keep their semantics while becoming usable across modalities without retraining the backbone.

Core claim

On the paper's own terms, the discovery is that cross-modal alignment does not require co-training a large backbone from scratch. A 'student' initialized from a frozen DINOv2, with the final four blocks trainable, can be pushed toward a modality-agnostic feature space using symmetric InfoNCE over modality pairs plus a teacher-anchoring distillation term. With depth and segmentation inputs rendered with a color palette taken from the paired RGB image and then stochastically blended with RGB, the student learns to place the same scene's RGB, depth, and segmentation embeddings close together while preserving discrimination between scenes. The paper reports that the adapted encoder beats frozen

What carries the argument

The 'adapter' is the student head g: the final four blocks of a ViT-B/14 that are fine-tuned while the first eight blocks and the teacher head stay frozen. It is driven by two objectives: a symmetric InfoNCE loss over all three modality pairs (RGB, depth, segmentation) that pulls same-scene embeddings together while pushing different scenes apart, and an anchoring loss that keeps the student's outputs close to the frozen teacher's outputs to prevent representational collapse. Two data ingredients make the alignment task hard rather than trivial: natural colorization, which renders depth/segmentation maps with a per-bin color palette derived from the paired RGB image, and modality mixup, whic

Load-bearing premise

The colorization step derives the palette from the paired RGB image and is applied identically at evaluation time, so the alignment gains may come from matching per-bin color statistics between inputs rather than from genuinely modality-invariant structure; the paper never tests the encoder on raw, grayscale, or independently colorized depth/segmentation inputs.

What would settle it

Run the cross-modal retrieval evaluation with depth and segmentation maps rendered as raw grayscale, or with each query using a different random RGB-derived palette than the gallery. If Recall@1 drops back toward the frozen-baseline level, the alignment was carried by color statistics rather than by a structural shared space. A second test: train the depth head on RGB and evaluate on NOCS maps with palette randomization removed—if the NOCS advantage persists, the transfer is structural.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single frozen backbone plus a four-block adapter can support RGB, depth, and segmentation at inference with one shared set of parameters.
  • Task heads trained on one modality can be deployed on another without retraining; the paper demonstrates this by training a depth head on RGB and evaluating on segmentation and NOCS maps.
  • Cross-modal retrieval improves dramatically on real and synthetic datasets, e.g., ScanNet Median Rank drops from 401.8 to 2.0 with GAP features.
  • The adapter preserves and sometimes improves the original encoder's discriminative semantics: linear-probe ImageNet top-1 rises from 80.4% to 83.8%, while k-NN accuracy stays at parity.
  • The method transfers to a different teacher (TIPS) and to an adapter-on-top parametrization, suggesting it is not specific to DINOv2's architecture.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The reported retrieval numbers use the same natural colorization at evaluation as in training, so the model could partially satisfy the retrieval task by matching per-bin color statistics rather than true geometric structure; evaluating on raw grayscale maps or with independently chosen palettes would reveal how much of the alignment is structural.
  • If the colorization shortcut is real, the zero-shot 'modality transfer' result on segmentation might largely be an RGB-histogram matching artefact; the NOCS result, where inputs are raw coordinate maps, is the cleaner test and is worth isolating.
  • The recipe suggests a general post-hoc alignment procedure: given paired signals, a small trainable adapter plus an anchoring loss may align any two modalities into an existing foundation model's space, provided the input rendering removes trivial cues.
  • One testable extension is to measure whether the aligned space supports cross-modal generation (e.g., depth-conditioned RGB synthesis) — the paper hints at but does not demonstrate such applications.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes the Omnivorous Vision Encoder, a post-hoc adaptation of a frozen DINOv2 ViT-B/14 backbone in which only the last four transformer blocks are fine-tuned. Training uses a symmetric InfoNCE loss over paired modalities (RGB, colorized depth, colorized segmentation) plus a teacher-anchoring cosine loss that prevents representational collapse. The inputs are preprocessed with a 'natural colorization' scheme that re-renders depth and segmentation maps using a palette derived from the paired RGB image, followed by stochastic modality mixup. The authors claim that the resulting encoder is modality-agnostic, with strong cross-modal retrieval (e.g., ScanNet R@1 from 4.6% to 46.1% and MedR to 2.0), improved or preserved downstream task performance (depth, segmentation, classification), and zero-shot modality transfer to segmentation and NOCS inputs. The main evidence for the core claim is Table 1, supported by ablations in Section 4.4 and the appendix.

Significance. If the central claim is established, the paper would make a useful contribution: a lightweight, parameter-efficient recipe for aligning visual modalities without retraining a large backbone, with released weights and a broad evaluation suite. The teacher-anchoring loss and the 'natural colorization' idea are interesting, and the downstream evaluations (Tables 2, 3, 4, 9, 10, 11, 12) are extensive and mostly well executed. The main weakness is that the headline cross-modal retrieval results are confounded by the RGB-derived colorization applied at evaluation time; as written, Table 1 does not cleanly demonstrate structural alignment rather than low-level color matching. The paper also contains a direct internal contradiction about whether retrieval inputs are colorized or raw. The claim can likely be repaired with additional control experiments, but it is not established by the current evidence.

major comments (3)
  1. [§3.2, §6.2.1, Algorithm 1] The retrieval evaluation is confounded by the colorization scheme. Section 3.2 states that colorization is applied at both training and eval, and Algorithm 1 constructs a per-scene palette P from the RGB pixel values in each intensity bin of the depth/segmentation map. At eval, the colorized depth/seg of scene i is therefore built from colors extracted from RGB_i, so a retrieval model can rank the correct scene by matching scene-specific chromatic statistics without needing to understand geometry. Appendix 6.2.1, by contrast, says retrieval inputs are 'single-channel structural inputs ... tiled to 3 channels,' with no colorization, directly contradicting Sec 3.2. The authors must clarify which protocol was actually used for Table 1. If colorization was used, they should add control evaluations with raw grayscale, a fixed colormap, or a palette borrowed from a different scene; without suc
  2. [§4.3 / Appendix 6.2.7] The zero-shot transfer to Segmentation in Table 5 is not a clean modality shift. Appendix 6.2.7 says segmentation maps are preprocessed using the same Natural Colorization scheme (Algorithm 1), i.e., they are re-rendered with the palette of the paired RGB image from PACE. The depth head trained on RGB may therefore succeed on segmentation inputs partly because the input carries RGB-like chromatic statistics, not because the encoder has a genuine shared structural space. The NOCS result is cleaner, but it is the only out-of-distribution modality that avoids this issue. Please report the exact preprocessing for the 'seg' column and evaluate on raw or fixed-colormap segmentation inputs to make the transfer claim convincing.
  3. [§4.4 / §6.1.2] The paper motivates natural colorization as a defense against shortcut learning ('hard positives'), but it never ablates the colorization choice. There is no comparison to grayscale, jet, a fixed colormap, or a randomly assigned palette. Such an ablation is essential because Algorithm 1 is a core data-centric contribution and because Table 1's validity depends on the claim that the model cannot solve retrieval using color statistics. Without this ablation, the claim that natural colorization prevents shortcuts is unsupported, and the reader cannot tell whether the retrieval gains reflect structural alignment or a palette-matching shortcut.
minor comments (5)
  1. [§5 / Appendix 7.2] The Discussion states that Omnivorous 'exceed[s] DINOv2's performance on all 3D-relevant tasks in the Probe3D framework,' but Table 9 shows normals estimation at parity or with mixed small differences (e.g., absrel 197.8 vs 197.9, diff 11.25 43.6 vs 43.5). Please soften this claim to 'at par or better' or qualify it.
  2. [§6.2.7] The caption for the zero-shot transfer section says NOCS maps are '3-channel coordinate maps' and are normalized with ImageNet statistics, but the natural colorization description in the same appendix could be misread as applying to NOCS. Clarify explicitly that NOCS inputs are not colorized.
  3. [Algorithm 1] In Step 3, the 'uniform kernel w' is defined with each entry 1 but the convolution is not divided by the kernel sum; the later normalization by N_tilde is sufficient, but a one-sentence note would avoid confusion.
  4. [§3.3] Equation (1) uses exp_tau(x) = exp(x/tau) but the denominator is written with exp_tau(sim(...)); this is fine, but the notation is slightly nonstandard. Consider writing exp(sim(...)/tau) directly for readability.
  5. [Table 1] The table does not state whether the reported R@1/mAP/MedR for DINOv2 are computed on the same colorized inputs as Omnivorous or on raw/grayscale inputs. This matters for interpreting the baseline comparison; please state the exact input protocol in the caption.

Circularity Check

2 steps flagged

Retrieval evidence is partially self-definitional: the colorized depth/segmentation inputs at eval are re-rendered from the query RGB palette, so the measured alignment is partly baked into the preprocessing; an internal inconsistency about eval inputs leaves the confound unresolved.

specific steps
  1. self definitional [Sec 3.2 (Data pipeline) with Algorithm 1 in Sec 6.1.2; impacts Table 1 in Sec 4.1]
    "Colorization (training and eval). For a given (photometrically augmented) RGB image x_aug_r, we quantize its pixel values into 64 bins. These can then be used to colorize the corresponding segmentation or depth map, so the colorized maps x_s and x_d resemble the RGB image. ... The colorized map x_m is generated by mapping each pixel in the raw map to its corresponding palette entry: x_m(u, v) = P[b_{u,v}]."

    The depth/segmentation inputs are defined from the paired RGB image via a per-bin palette. Because the same colorization is stated to be applied at evaluation, the colorized structural map of scene i is composed of colors aggregated from RGB_i. A retrieval model can therefore rank the correct scene by matching palette/color-bin statistics without learning a structural or modality-invariant space. The large improvement in Table 1 (ScanNet R@1 4.6% to 46.1%) is thus partly guaranteed by the input construction, not purely by the adapter's learned alignment. No control with raw grayscale, fixed colormap, or cross-scene palette is reported, so the claimed 'hard positives' evidence does not establish the structural-attention mechanism it asserts.

  2. other [Appendix 6.2.1 vs Sec 3.2 and Sec 6.2.7]
    "Single-channel structural inputs (depth and segmentation) are tiled to 3 channels and normalized using standard ImageNet statistics (Appendix 6.2.1). Yet Sec 3.2 states: 'Colorization (training and eval).' And Sec 6.2.7 says: 'segmentation maps are preprocessed using our Natural Colorization scheme (Algorithm 1) to match the spectral statistics of RGB images.'"

    The paper is internally inconsistent about which preprocessing was used for cross-modal retrieval. If the Appendix protocol (raw single-channel tiled to 3 channels) is the one actually used, then the Sec 3.2 claim of eval colorization is false and the shortcut is moot, but the paper must say so. If the Sec 3.2 protocol was used, the self-definitional confound applies to the central retrieval numbers. Either way, the experiments as written do not establish the claimed 'regardless of input modality' behavior, and the unresolved contradiction is load-bearing evidence rather than a cosmetic inconsistency.

full rationale

The paper's central derivation is an empirical training recipe, not a formal theorem; most of the downstream evaluation is external and independent. ImageNet linear/k-NN, NYUv2 depth, ADE20k/Cityscapes/Pascal VOC segmentation, and especially zero-shot NOCS transfer provide genuine out-of-distribution support for the claim that the adapter preserves semantics and generalizes to unseen modalities. There is no load-bearing self-citation chain: citations to DINOv2, Probe3D, TIPS, etc. are external or used only as ablations, and no 'uniqueness theorem' is invoked to forbid alternatives. However, the flagship cross-modal retrieval results, which are the primary evidence for 'modality-agnostic' behavior, are compromised. Algorithm 1 defines the structural input as a function of the paired RGB image's color palette, and Sec 3.2 explicitly applies this colorization at eval. A model can match scene-specific palette statistics to solve retrieval without learning structural invariance. The Supplementary's conflicting description of retrieval inputs (single-channel tiles) makes it impossible to determine which protocol was actually used, and the paper provides no control condition. This makes the retrieval-based alignment claim partially circular-by-construction, although the existence of independent external benchmarks and the NOCS result keep the overall work from being wholly circular. Score 6 reflects one substantial 'prediction' (cross-modal retrieval) whose measured success is partly baked into its inputs, while the remaining claims retain independent content.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

No new physical or architectural entities are introduced; the method is a training recipe over an existing frozen backbone. The load-bearing choices are hyperparameters tuned on the paper's own validation tasks and the assertion that RGB-derived colorization of depth/seg is a neutral hard-positive transform rather than a label-leaking preprocessing. No graviton-style entities are postulated.

free parameters (5)
  • λ_anchor = 10
    Hand-selected using the Fig 4b frontier (linear-head depth vs segmentation performance); controls the alignment-vs-drift trade-off and prevents collapse (Sec 3.3, Sec 4.4).
  • α_max = 0.5
    Chosen to balance across tasks from the Table 6 ablation; most tasks keep improving toward 1.0 but depth is the outlier (Sec 3.2, Sec 4.4).
  • L_stop-gradient = 8 of 12
    Chosen from the Table 12 ablation; variations 4–10 give comparable results, so the choice is weakly constrained (Sec 3.1, Appendix 7.3.3).
  • Colorization bins B and smoothing kernel K = B=64, K=5
    Hand-chosen in Algorithm 1; directly shape input statistics of depth/seg and thus difficulty and potential leakage (Sec 6.1.2).
  • InfoNCE temperature τ = learned, clipped to [0,100]
    Standard learned temperature in Eq. 1; controls effective sharpness of the contrastive loss.
axioms (5)
  • domain assumption DINOv2 teacher features form a semantically rich anchor whose space is the right target for distillation.
    The anchoring loss (Eq. 3) pulls the student toward g* ◦ f* on the assumption that the teacher space is worth preserving; supported indirectly by k-NN parity (Table 4) but never directly validated (Sec 3.3).
  • domain assumption The six training datasets (mostly synthetic/simulated) transfer to natural-image downstream tasks.
    The paper itself concedes regressions on iNaturalist and GLDv2 'could be explained by our training mix... simulated multi-object data' (Sec 4.2), so transfer is partial.
  • ad hoc to paper Natural colorization creates 'hard positives' and does not create an exploitable cross-modal shortcut.
    Colorized depth/seg share the paired RGB's palette by construction (Algorithm 1, Sec 6.1.2); the claim is asserted but never tested against raw, grayscale, or jet-colored inputs.
  • domain assumption Retrieval test splits (MOVi, ScanNet, TartanAir) are disjoint from training and results are representative of generalization.
    MOVi, ScanNet, TartanAir appear in both the training list (Table 7) and the retrieval evaluation (Table 1); the train/test split is never stated, so in-domain leakage cannot be excluded.
  • standard math Standard InfoNCE/contrastive machinery behaves as expected (no collapse when λ_anchor>0).
    Eq. 1 with L2-normalized features and temperature clipping is standard; the paper's own ablation shows λ_anchor=0 collapses performance, so non-degeneracy relies on the anchoring term.

pith-pipeline@v1.3.0-alltime-deepseek · 4918 in / 9259 out tokens · 258066 ms · 2026-08-02T20:03:11.107107+00:00 · methodology

0 comments
read the original abstract

Pre-trained vision encoders like DINOv2 have demonstrated exceptional performance on unimodal tasks. However, we observe that their features are poorly aligned across different visual modalities. For instance, the feature embedding for an RGB image and its corresponding depth map of the same scene exhibit a cosine similarity that is nearly identical to that of two random, unrelated images. To address this, we propose the Omnivorous Vision Encoder, a post-training framework that learns a modality-agnostic feature space. We fine-tune the encoder with a dual objective: first, to maximize the feature alignment between different modalities of the same scene; and second, a distillation objective that anchors the learned representations to a fully frozen teacher. The resulting student encoder becomes "omnivorous" by producing more consistent embeddings for a given scene, regardless of the input modality (RGB, Depth, Segmentation, etc.). This approach enables robust cross-modal understanding while retaining the discriminative semantics of the original foundation model. Omnivorous model weights are available at https://github.com/google-deepmind/representations4d.

Figures

Figures reproduced from arXiv: 2602.24181 by Andre Araujo, Drew A. Hudson, Joao Carreira, Maks Ovsjanikov, Niloy J. Mitra, Rishabh Kabra, Skanda Koppula, Ye Xia.

Figure 1
Figure 1. Figure 1: Off-the-shelf vision encoders like DINO show poor cross-modal alignment. We show the similarity in feature space between randomly paired RGB images (top), between RGB im￾ages and depth maps of the same scene (middle), and between RGB and grayscale images of the same scene (bottom). While the numbers vary depending on the dataset, the pattern of misalign￾ment between visual modalities remains consistent. Ou… view at source ↗
Figure 2
Figure 2. Figure 2: Omnivorous Vision Encoder architecture. A frozen encoder f ∗ extracts features zm = f ∗ (xm) from a spectrum of modalities denoted m (Segmentation, RGB, Depth). A trainable modality-agnostic adapter g maps these features into a common, aligned embedding space, producing a modality-invariant repre￾sentation h = g(zm). A convenient implementation of this archi￾tecture uses the early layers of a pretrained ne… view at source ↗
Figure 3
Figure 3. Figure 3: illustrates our training data spanning six datasets (detailed further in Appendix 6). 3.3. Loss Symmetric Cross-Modal Alignment To create a unified representation space, we employ a symmetric alignment strategy. We aim for the student embeddings from the same scene but different modalities to be close, while embeddings from different scenes should be distinct. We use the InfoNCE (Information Noise-Contrast… view at source ↗
Figure 4
Figure 4. Figure 4: Analysis of the anchoring loss. (a) Trade-off between cross-modal alignment and cross-scene discernibility, controlled by λanchor. The x-axis measures alignment (cosine sim of <RGB, Depth>) and the y-axis measures discernibility (1 - cosine simi￾larity of distinct RGB scenes) on ScanNet. Frozen DINOv2 (light blue) is discriminative but poorly aligned. (b) To pick a value for λanchor, we examine its effect … view at source ↗
Figure 5
Figure 5. Figure 5: PCA visualizations of frozen (DINO ViT-B/14) and [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative comparison (Omnivorous vs DINOv2) on depth prediction using a linear head. Please compare a versus b, and c versus d. We highlight notable differences using a black oval. 5 [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative comparison (Omnivorous vs DINOv2) on segmentation prediction using a linear head. We highlight notable differences using a white oval. 6 [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Qualitative comparison (Omnivorous vs DINOv2) on segmentation prediction (contd.) using a linear head. We highlight notable differences using a white oval [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Behavior of Omnivorous TIPS. 10 [PITH_FULL_IMAGE:figures/full_fig_p021_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Depth-Regularized JEPA World Models Learn More Transferable Representations from Real Outdoor Robot Data

    cs.CV 2026-07 conditional novelty 5.0

    Adding a training-only depth-alignment loss to a compact LeWM JEPA world model improves downstream representation quality and out-of-domain transfer on real agricultural robot data.

Reference graph

Works this paper leans on

59 extracted references · 7 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Cross-modal knowledge transfer with- out task-relevant source data

    Faheem Ahmed et al. Cross-modal knowledge transfer with- out task-relevant source data. InComputer Vision – ECCV 2022, 2022. 2

    2022

  2. [2]

    Unsupervised neural machine trans- lation.CoRR, abs/1710.11041, 2017

    Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural machine trans- lation.CoRR, abs/1710.11041, 2017. 1

    Pith/arXiv arXiv 2017

  3. [3]

    Spatial-aware multi-modal contrastive learning for rgb-d salient object detection and beyond.Information Fu- sion, 124:103362, 2025

    Hao Chen, Zichao Chen, Yongliang Wu, and Hongzhuo Chen. Spatial-aware multi-modal contrastive learning for rgb-d salient object detection and beyond.Information Fu- sion, 124:103362, 2025. 2

    2025

  4. [4]

    Vision transformer adapter for dense predictions

    Zhe Chen, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Zi- hang Dai, Dongdong Chen, Lu Yuan, Lei Zhang, Zicheng Liu, Baining Guo, and Jingdong Wang. Vision transformer adapter for dense predictions. InProceedings of the Inter- national Conference on Learning Representations (ICLR),

  5. [5]

    Cross-lingual language model pretraining

    Alexis CONNEAU and Guillaume Lample. Cross-lingual language model pretraining. InAdvances in Neural Infor- mation Processing Systems. Curran Associates, Inc., 2019. 1

    2019

  6. [6]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. InProceed- ings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016. 5

    2016

  7. [7]

    Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner

    Angela Dai, Angel X. Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2017. 5, 2

    2017

  8. [8]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 7, 3

    2009

  9. [9]

    An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020. 3

    Pith/arXiv arXiv 2010

  10. [10]

    Prob- ing the 3d awareness of visual foundation models

    Mohamed El Banani, Amit Raj, Kevis-Kokitsi Maninis, Ab- hishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Prob- ing the 3d awareness of visual foundation models. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21795–21806, 2024. 5, 2

    2024

  11. [11]

    Mark Everingham, Luc Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge.International Journal of Computer Vision, 88(2):303–338, 2010. 5

    2010

  12. [12]

    Dantas, Luigi Di Caro, and Dino Ienco

    Roger Ferrod, C ´assio F. Dantas, Luigi Di Caro, and Dino Ienco. Revisiting cross-modal knowledge distillation: A disentanglement approach for rgbd semantic segmentation,

  13. [13]

    Omnivore: A sin- gle model for many visual modalities

    Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A sin- gle model for many visual modalities. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16102–16112, 2022. 2

    2022

  14. [14]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 15180–15190,

  15. [15]

    Kubric: A scalable dataset generator

    Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapra- gasam, Florian Golemo, Charles Herrmann, et al. Kubric: A scalable dataset generator. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3749–3761, 2022. 5, 2

    2022

  16. [16]

    The inaturalist species classification and de- tection dataset, 2018

    Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and de- tection dataset, 2018. 7

    2018

  17. [17]

    Mask3d: Pre-training 2d vision transformers by learning masked 3d priors

    Ji Hou, Xiaoliang Dai, Zijian He, Angela Dai, and Matthias Nießner. Mask3d: Pre-training 2d vision transformers by learning masked 3d priors. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 2

    2023

  18. [18]

    Tianyu Huang, Bowen Dong, Yunhan Yang, Xiaoshui Huang, Rynson W. H. Lau, Wanli Ouyang, and Wangmeng Zuo. Clip2point: Transfer clip to point cloud classifica- tion with image-depth pre-training. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. 2

    2023

  19. [19]

    Perceiver: General perception with iterative attention

    Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. InProceedings of the 38th International Conference on Machine Learning, pages 4651–4664. PMLR, 2021. 2

    2021

  20. [20]

    Perceiver io: A general architecture for structured inputs & outputs

    Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Maksym Andriushchenko, Sander Dieleman, and et al. Perceiver io: A general architecture for structured inputs & outputs. InProceedings of the International Con- ference on Learning Representations (ICLR), 2022. 2

    2022

  21. [21]

    Multi- modal contrastive masked autoencoders: A two-stage pro- gressive pre-training approach for rgbd datasets

    Muhammad Abdullah Jamal and Omid Mohareri. Multi- modal contrastive masked autoencoders: A two-stage pro- gressive pre-training approach for rgbd datasets. InProceed- ings of the Computer Vision and Pattern Recognition Con- ference (CVPR), pages 17947–17957, 2025. 2

    2025

  22. [22]

    NA VI: Category- agnostic image collections with high-quality 3d shape and pose annotations

    Varun Jampani, Kevis-Kokitsi Maninis, Andreas Engel- hardt, Arjun Karpur, Karen Truong, Kyle Sargent, Stefan Popov, Andre Araujo, Ricardo Martin-Brualla, Kaushal Pa- tel, Daniel Vlasic, Vittorio Ferrari, Ameesh Makadia, Ce Liu, Yuanzhen Li, and Howard Zhou. NA VI: Category- agnostic image collections with high-quality 3d shape and pose annotations. InNeur...

    2023

  23. [23]

    Google’s multilin- gual neural machine translation system: Enabling zero-shot translation.Transactions of the Association for Computa- tional Linguistics, 5:339–351, 2017

    Melvin Johnson, Mike Schuster, Quoc Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Vi´egas, Martin Wattenberg, Greg Corrado, et al. Google’s multilin- gual neural machine translation system: Enabling zero-shot translation.Transactions of the Association for Computa- tional Linguistics, 5:339–351, 2017. 1 9

    2017

  24. [24]

    Dy- namicstereo: Consistent dynamic depth from stereo videos

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Dy- namicstereo: Consistent dynamic depth from stereo videos. CVPR, 2023. 2

    2023

  25. [25]

    The retinex theory of color vision.Scientific american, 237(6):108–129, 1977

    Edwin H Land. The retinex theory of color vision.Scientific american, 237(6):108–129, 1977. 1

    1977

  26. [26]

    Uni-perceiver v2: A gener- alist model for large-scale vision and vision-language tasks

    Hao Li, Jinguo Zhu, Xiaohu Jiang, Xizhou Zhu, Hongsheng Li, Chun Yuan, Xiaohua Wang, Yu Qiao, Xiaogang Wang, Wenhai Wang, and Jifeng Dai. Uni-perceiver v2: A gener- alist model for large-scale vision and vision-language tasks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2691–2700,

  27. [27]

    A closer look at invari- ances in self-supervised pre-training for 3d vision

    Lanxiao Li and Michael Heizmann. A closer look at invari- ances in self-supervised pre-training for 3d vision. InCom- puter Vision – ECCV 2022, 2022. 2

    2022

  28. [28]

    Contrastive multimodal fu- sion with tupleinfonce

    Yunze Liu, Qingnan Fan, Shanghang Zhang, Hao Dong, Thomas Funkhouser, and Li Yi. Contrastive multimodal fu- sion with tupleinfonce. InProceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 754–763,

  29. [29]

    Unified-io: A unified model for vision, language, and multi-modal tasks

    Yuan Lu, Boyang Li, Philipp Randen, Marcella Cornia, Shiry Schiff, Ming-Wei Chang, and et al. Unified-io: A unified model for vision, language, and multi-modal tasks. InPro- ceedings of the International Conference on Learning Rep- resentations (ICLR), 2023. 2

    2023

  30. [30]

    Unified-io 2: Scaling autoregressive multimodal models with vision, language, au- dio, and action

    Yuan Lu, Boyang Li, Philipp Randen, Marcella Cornia, Shiry Schiff, Ming-Wei Chang, and et al. Unified-io 2: Scaling autoregressive multimodal models with vision, language, au- dio, and action. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2

    2024

  31. [31]

    Ma-avt: Modality alignment for parameter-efficient audio-visual transformers

    Tanvir Mahmud, Yizhe Tong, Dongliang Du, Ying Cao, Deliang Wang, and Deng Cai. Ma-avt: Modality alignment for parameter-efficient audio-visual transformers. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2024. 2

    2024

  32. [32]

    TIPS: Text-Image Pretraining with Spatial Awareness

    Kevis-Kokitsi Maninis, Kaifeng Chen, Soham Ghosh, Ar- jun Karpur, Koert Chen, Ye Xia, Bingyi Cao, Daniel Salz, Guangxing Han, Jan Dlabal, Dan Gnanapragasam, Mojtaba Seyedhosseini, Howard Zhou, and Andr ´e Araujo. TIPS: Text-Image Pretraining with Spatial Awareness. InICLR,

  33. [33]

    Large scale visual food recognition, 2023

    Weiqing Min, Zhiling Wang, Yuxin Liu, Mengjiang Luo, Liping Kang, Xiaoming Wei, Xiaolin Wei, and Shuqiang Jiang. Large scale visual food recognition, 2023. 7

    2023

  34. [34]

    Indoor segmentation and support inference from rgbd images

    Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. InECCV, 2012. 2, 5

    2012

  35. [35]

    Repre- sentation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018. 4

    Pith/arXiv arXiv 2018

  36. [36]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 1

    Pith/arXiv arXiv 2023

  37. [37]

    Maxime Oquab, Timoth ´ee Darcet, Th´eo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Je- gou, Julien Mairal, Patr...

    2024

  38. [38]

    Rp2k: A large- scale retail product dataset for fine-grained image classifica- tion.arXiv preprint arXiv:2006.12634, 2020

    Jingtian Peng, Chang Xiao, and Yifan Li. Rp2k: A large- scale retail product dataset for fine-grained image classifica- tion.arXiv preprint arXiv:2006.12634, 2020. 7

    Pith/arXiv arXiv 2006

  39. [39]

    Susskind

    Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M. Susskind. Hypersim: A photorealistic syn- thetic dataset for holistic indoor scene understanding. In International Conference on Computer Vision (ICCV) 2021,

    2021

  40. [40]

    Deep metric learning via lifted structured feature embedding

    Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 7

    2016

  41. [41]

    Sun rgb-d: A rgb-d scene understanding benchmark suite

    Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 567–576, 2015. 2

    2015

  42. [42]

    Sequence to sequence learning with neural networks.Advances in neural information processing systems, 27, 2014

    Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks.Advances in neural information processing systems, 27, 2014. 1

    2014

  43. [43]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023. 1

    Pith/arXiv arXiv 2023

  44. [44]

    Con- trastive multiview coding

    Yonglong Tian, Dilip Krishnan, and Phillip Isola. Con- trastive multiview coding. InComputer Vision – ECCV 2020, pages 776–794, Cham, 2020. Springer International Publish- ing. 1

    2020

  45. [45]

    Normalized object coordinate space for category-level 6d object pose and size estimation

    He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2642–2651,

  46. [46]

    Tartanair: A dataset to push the limits of visual slam

    Wenshan Wang, Delong Zhu, Xiangwei Wang, Yaoyu Hu, Yuheng Qiu, Chen Wang, Yafei Hu, Ashish Kapoor, and Se- bastian Scherer. Tartanair: A dataset to push the limits of visual slam. In2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4909–4916. IEEE, 2020. 5, 2

    2020

  47. [47]

    Weyand, A

    T. Weyand, A. Araujo, B. Cao, and J. Sim. Google Land- marks Dataset v2 - A Large-Scale Benchmark for Instance- Level Recognition and Retrieval. InProc. CVPR, 2020. 7

    2020

  48. [48]

    Comae: Single model hybrid pre-training on small-scale rgb- d datasets

    Jiange Yang, Sheng Guo, Gangshan Wu, and Limin Wang. Comae: Single model hybrid pre-training on small-scale rgb- d datasets. InProceedings of the AAAI Conference on Artifi- cial Intelligence (AAAI), 2023. 2 10

    2023

  49. [49]

    Pace: A large-scale dataset with pose annotations in cluttered environments

    Yang You, Kai Xiong, Zhening Yang, Zhengxiang Huang, Junwei Zhou, Ruoxi Shi, Zhou Fang, Adam W Harley, Leonidas Guibas, and Cewu Lu. Pace: A large-scale dataset with pose annotations in cluttered environments. InEu- ropean Conference on Computer Vision, pages 473–489. Springer, 2024. 6

    2024

  50. [50]

    Colour coding in the cerebral cortex: the re- action of cells in monkey visual cortex to wavelengths and colours.Neuroscience, 9(4):741–765, 1983

    Semir Zeki. Colour coding in the cerebral cortex: the re- action of cells in monkey visual cortex to wavelengths and colours.Neuroscience, 9(4):741–765, 1983. 1

    1983

  51. [51]

    Dauphin, and David Lopez-Paz

    Hongyi Zhang, Moustapha Ciss ´e, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimiza- tion.CoRR, abs/1710.09412, 2017. 3

    Pith/arXiv arXiv 2017

  52. [52]

    Towards uni- fied representation of invariant-specific features in missing modality face anti-spoofing

    Wen Zheng, Zhanpeng Zhang, Kai Zhang, Guangwei Zhou, Jie Yu, Yongdong Zhang, and Jian Cheng. Towards uni- fied representation of invariant-specific features in missing modality face anti-spoofing. InComputer Vision – ECCV 2024, 2024. 2

    2024

  53. [53]

    Harley, Bokui Shen, Gordon Wet- zstein, and Leonidas J

    Yang Zheng, Adam W. Harley, Bokui Shen, Gordon Wet- zstein, and Leonidas J. Guibas. Pointodyssey: A large-scale synthetic dataset for long-term point tracking. InICCV,

  54. [54]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 633–641,

  55. [55]

    Uni-perceiver: Pre- training unified architecture for generic perception for zero- shot and few-shot tasks

    Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng Li, Xiaohua Wang, and Jifeng Dai. Uni-perceiver: Pre- training unified architecture for generic perception for zero- shot and few-shot tasks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16804–16815, 2022. 2 11 A Mixed Diet Makes DINO An Omnivorous V...

    2022

  56. [56]

    hard positives

    Training and Evaluation Details We first present a summary of our training configuration in Table 7. Next, we describe our training data pipeline in Sec- tion 6.1. Finally, we elaborate on our evaluation protocols in Section 6.2. 6.1. Data Pipeline Below we elaborate on all elements of the training-data pro- cessing steps previously introduced in Section ...

  57. [58]

    Diagnostic Metrics Expanding on Fig 1, we report detailed cross-modal align- ment and cross-scene discernibility metrics before and after Omnivorous training

    Extended Results 7.1. Diagnostic Metrics Expanding on Fig 1, we report detailed cross-modal align- ment and cross-scene discernibility metrics before and after Omnivorous training. Table 8 shows that our default check- point of Omnivorous DINO greatly improves cross-modal alignment while sacrificing some cross-scene discernibility (e.g., from 0.198 to 0.2...

  58. [59]

    adapter-on-top

    We present two evaluations that were omitted in the main paper (normals estimation and multiview 3D corre- spondence), and add qualitative results for those already presented in the main paper (e.g., depth estimation and seg- mentation): 7.2.1. Normals Estimation See Table 9. Omnivorous is consistently at par with DI- NOv2 across all metrics. 7.2.2. Multi...

    arXiv

  59. [2048]

    pad- to-patch

    The rank for a given query is determined by counting the number of gallery items with a similarity score strictly greater than or equal to the ground-truth pair’s score (using a numerical stability thresholdϵ= 10 −6). As our evalu- ation setup assumes a strict one-to-one mapping between modalities (i.e., exactly one positive match per query), the Mean Ave...