REVIEW 3 major objections 5 minor 36 references
Probing the 3D Object-Level Understanding of Pre-Trained Detection Transformers
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Pre-trained 2D detection transformers encode object depth and 3D location in their query embeddings, readable with small probes to within fractions of a meter of task-specific 3D models.
desk verdict A genuinely new probing target with a solid depth result, but the 3D location claim needs a 2D-box-only control before it can support the '3D understanding' interpretation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the DETR query embedding $q_i \in \mathbb{R}^D$ — the per-object vector output by the transformer decoder before the class and box heads. The paper freezes the detector and trains simple probes (one affine map, or a two-layer ReLU MLP with hidden dimension 256) to map $q_i$ to a target property: depth sampled at the predicted box center, or the KITTI 3D box center in camera coordinates. Greedy IoU matching links each embedding to ground-truth annotations, and a second alignment step ensures that all models are scored on the same detected objects. The mechanism's power is demonstrated by the ablation in Table 4: swapping the embedding for the 4-dimensional predicted
What would settle it
Train the same MLP probe on the concatenation of class probabilities and the 2D box (center, width, height) instead of the embedding, with identical matching, alignment, and capacity. If that input reaches the same depth MAE and 3D-center error as the embedding probe, the central claim collapses into a 2D-geometry effect.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that the query embeddings produced by pre-trained DETR-family detectors carry recoverable, object-level 3D information despite never being trained on depth or 3D boxes. The evidence is a probing comparison: an MLP probe on DETR embeddings reaches 0.54 m mean absolute error on Virtual KITTI2 and 0.51 m on NYUv2, while zero-shot Depth Anything 2 reaches 1.49 m and 0.61 m; on KITTI, the probe predicts 3D object-center location with 0.50 m mean absolute error versus 0.30 m for MonoDETR, whose architecture and training are specialized for this task. The paper also reports that the two real-time variants (RT-DETR v2 and LW-DETR) carry noticeably l
Load-bearing premise
The load-bearing premise is that probe accuracy reflects 3D structure inside the embeddings, not shallow 2D geometry such as vertical image position, box scale, and category priors; the paper's only control is a 4-dimensional bounding-box probe, so a probe that also includes class information could easily overturn the 3D reading.
Editorial extensions
If this is right
- Probing frozen DETR embeddings can produce an object-centric depth estimate without retraining the detector, which could feed monocular 3D detection or depth-aware reasoning as a cheap prior.
- A 2D detector plus a small readout may serve as a pragmatic substitute for fully 3D-supervised detectors in settings where only 2D labels are abundant and a few meters of location error is tolerable.
- The real-time DETR variants' weaker 3D readout implies that latency optimizations can trade away geometric understanding, not just detection accuracy.
- PCA compression results indicate the 3D-relevant signal sits in a low-dimensional subspace of the embedding, so compact probes or embedding compression can preserve much of the depth information.
- The small absolute gap to MonoDETR (about 0.2 m center MAE at a 25.3 m median object distance) suggests the remaining 3D deficit is modest, motivating architectural changes that add explicit 3D objectives to close it.
Reading between the lines
- Editorial inference: the reported depth accuracy may be partly attributable to geometric priors like vertical image position, box scale, and class-specific depth distributions; a probe trained jointly on class logits and the 2D box would isolate the 3D content.
- Editorial extension: the same protocol could probe yaw, pitch, and 3D extents on KITTI; strong readouts there would show the embeddings encode a full 3D box rather than just a location.
- Editorial extension: the real-time variant deficit could be a capacity effect rather than a fundamental limit; probing a scaled-up RT-DETR or LW-DETR would separate those explanations.
- Editorial inference: if 2D detection pretraining is sufficient to organize object embeddings around 3D location, then multi-task or self-supervised detection pipelines may already be learning geometric structure that could be exposed by similar probes rather than learned from scratch.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper probes object-level query embeddings from five frozen DETR-family 2D detectors (DETR, Deformable DETR, Conditional DETR, RT-DETR v2, LW-DETR) for two 3D properties: object-center depth and 3D object-center location. Using linear and two-layer MLP probes trained on public COCO-pretrained checkpoints, the authors report that depth can be decoded comparably to or better than a zero-shot Depth Anything 2 baseline, and that 3D location can be decoded to within 0.2 m MAE of MonoDETR, a model trained with 3D supervision. They also include PCA-based compressibility analysis and a bounding-box-probe ablation for depth. The central claim is that 2D DETR embeddings implicitly encode substantial 3D structure despite no 3D supervision.
Significance. If substantiated, the finding would be a significant empirical contribution: it would show that standard 2D detection transformers acquire non-trivial 3D object knowledge as a byproduct of 2D detection training, with implications for transfer learning and 3D-aware architecture design. The study has notable strengths: a standard frozen-representation probing protocol, public and reproducible checkpoints, multiple DETR variants, and a code release. These are positive aspects of the work. However, the interpretation that probe success reflects 3D understanding rather than 2D geometric or categorical shortcuts is currently under-supported, and one numerical inconsistency in the headline 3D location result requires attention.
major comments (3)
- [§4.2, Table 3] The 3D location experiment has no non-embedding control. The only bounding-box-only probe is for depth (Table 4). For KITTI, objects are essentially on a ground plane and the target is the bottom center of the 3D box; depth is strongly constrained by the vertical position of the 2D box bottom edge, and x/y follow from the box center and depth via the camera model. A 4-dimensional MLP probe on (cx, cy, w, h) could plausibly match or approach DETR's 0.50 m Center MAE. Without this control, the claim that query embeddings encode 3D location beyond projective 2D geometry is not established. Please add a bbox-only probe (and, ideally, a bbox+class probe) to Table 3.
- [§4.2, Table 3] There is an internal inconsistency in the reported 3D location numbers. For DETR, x MAE = 0.33 m, y MAE = 0.12 m, and z MAE = 1.03 m, yet Center MAE is reported as 0.50 m. Since the per-sample Euclidean distance is at least |dz|, the mean center distance cannot be smaller than the mean |dz|. This suggests an error in either the component MAE computation or the Center MAE computation. Because the headline '0.5 m vs 0.3 m' comparison to MonoDETR depends on this number, it must be corrected and re-evaluated.
- [§4.1, Table 4; §4.3] The depth ablation is incomplete for the multi-class NYUv2 setting. The bbox-only probe is a useful control, but it does not control for class distribution or category priors. The authors themselves attribute part of the embedding probe's advantage to 'semantic context, category priors' (§4.3). A probe on class logits alone, and a class+bbox probe, would determine how much of the depth accuracy is due to knowing the object category versus genuine geometric information in the embeddings. This is directly relevant to the paper's central interpretation.
minor comments (5)
- [§3.2 / Appendix C] Training hyperparameters for the 3D location MLP probe (epochs, warmup, learning-rate schedule) are not specified. Please include them for reproducibility.
- [§3.3 / §4.1] Dataset alignment uses DETR as the anchor model and keeps only objects detected by all models. This may affect cross-model comparisons. Reporting unaligned results or a sensitivity check with a different anchor would increase robustness.
- [Figure 2 (Right)] Layerwise depth MAE curves are shown without error bars or significance tests. Given the modest test sizes, some of the apparent layer-to-layer differences may be noise.
- [Table 2] In the NYUv2 block, MonoDETR's Zero-shot AbsRel is 497.72% and δ1 is 0.00%; this is likely a formatting artifact or an extreme domain-shift result. Consider a footnote to clarify.
- [Throughout] Minor language issues: 'out-performed' should be 'outperformed'; 'DETR3D' should be 'DETR 3D' for consistency; some hyphenation of 'pre-trained' is inconsistent.
Circularity Check
No significant circularity: probing is a standard supervised readout of frozen public DETR embeddings against external 3D labels; the missing 3D-location bbox control is a validity gap, not a circular reduction.
full rationale
The paper's derivation chain is a probing study, not a closed-form derivation. Frozen DETR query embeddings are taken from public Hugging Face checkpoints, and probe targets are external annotations: dense depth maps (Virtual KITTI2, NYUv2) and KITTI 3D bounding-box centers matched by IoU. There is no equation in the paper that defines the target in terms of the probe output or that defines the input representation in terms of the predicted 3D property. The probe parameters are trained on a disjoint training split and evaluated on held-out test objects, so the reported MAE/δ1 values are not fitted quantities renamed as predictions. The paper includes a bounding-box-only control for the monocular depth task (Table 4), which directly addresses the main 2D-geometry confound by showing that embedding probes outperform bbox probes on the same protocol. The absence of an analogous bbox-only control for the Table 3 3D-location result is a legitimate threat to the interpretation that the embeddings encode 3D structure beyond projective geometry, but this is an experimental-validity concern, not a circularity: the target 3D center is an independent KITTI annotation, and no fitted parameter or self-citation forces the result. The paper cites no prior work by its own authors, invokes no uniqueness theorem, and does not smuggle an ansatz in via citation. Therefore no circular step can be exhibited, and the appropriate score is 0.
Assumptions & free parameters
free parameters (4)
- objectness threshold tau =
0.5
- IoU matching threshold =
0.5
- Depth validity ranges =
0.5-100m (VKITTI2), 0.1-10m (NYUv2)
- Dataset alignment anchor =
DETR
assumptions (4)
- domain assumption Probing informativeness: low test error of a trained probe on frozen features implies the features encode the target property.
- domain assumption Depth at the 2D box center is a meaningful object-level 3D property, and the pinhole camera model links screen position and scale to depth.
- domain assumption Public pretrained checkpoints are representative instances of the five DETR architectures and were trained without 3D supervision.
- domain assumption Ground-truth depth and 3D annotations in VKITTI2, NYUv2, and KITTI are accurate for training and evaluation.
Cite this review
Pith. "Pith review of Probing the 3D Object-Level Understanding of Pre-Trained Detection Transformers." pith.science (2026). https://pith.science/paper/ALIMDVZQ
@misc{pith2026260801495,
author = {Pith},
title = {Pith review of: Probing the 3D Object-Level Understanding of Pre-Trained Detection Transformers},
year = {2026},
howpublished = {\url{https://pith.science/paper/ALIMDVZQ}},
note = {Machine review of arXiv:2608.01495}
}
read the original abstract
Detection transformer models, including DETR and its extensions, learn to output a set of object-level embeddings that can be simultaneously decoded into 2D bounding boxes and class distributions. In this paper, we investigate what pre-trained 2D detection transformers understand about the 3D properties of objects. Specifically, we investigate the extent to which properties including the depth of objects from the camera and the 3D location of objects relative to the camera can be recovered from object-level embeddings using linear and non-linear probes. Across a range of detection transformer models, our results show a surprisingly strong and previously unknown ability of 2D DETR models to represent useful information about the 3D properties of objects, despite the complete lack of 3D supervision during model pre-training.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
arXiv preprint arXiv:1610.01644 (2016)
Alain, G., Bengio, Y.: Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644 (2016)
arXiv 2016
-
[2]
Azaria, A., Mitchell, T.: The internal state of an llm knows when it’s lying. In: Findings of the Association for Computational Linguistics: Conference on Empir- ical Methods in Natural Language Processing 2023. pp. 967–976 (2023)
work page 2023
-
[3]
Compu- tational Linguistics48(1), 207–219 (2022)
Belinkov, Y.: Probing classifiers: Promises, shortcomings, and advances. Compu- tational Linguistics48(1), 207–219 (2022)
2022
-
[4]
arXiv preprint arXiv:2001.10773 (2020)
Cabon, Y., Murray, N., Humenberger, M.: Virtual kitti 2. arXiv preprint arXiv:2001.10773 (2020)
arXiv 2001
-
[5]
In: European Conference on Computer Vision
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End- to-end object detection with transformers. In: European Conference on Computer Vision. pp. 213–229. Springer (2020) Probing 3D Understanding of DETRs 15
work page 2020
-
[6]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9650–9660 (2021)
2021
-
[7]
arXiv preprint arXiv:2406.03459 (2024)
Chen, Q., Su, X., Zhang, X., Wang, J., Chen, J., Shen, Y., Han, C., Chen, Z., Xu, W., Li, F., et al.: Lw-detr: A transformer replacement to yolo for real-time detection. arXiv preprint arXiv:2406.03459 (2024)
arXiv 2024
-
[8]
In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W
Cheng, B., Schwing, A., Kirillov, A.: Per-pixel classification is not all you need for semantic segmentation. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems. vol. 34, pp. 17864–17875. Curran Associates, Inc. (2021)
work page 2021
Show all 36 references
-
[9]
In: Proceedings of the Com- puter Vision and Pattern Recognition Conference
Danier, D., Aygün, M., Li, C., Bilen, H., Mac Aodha, O.: Depthcues: Evaluating monocular depth perception in large vision models. In: Proceedings of the Com- puter Vision and Pattern Recognition Conference. pp. 20049–20059 (2025)
2025
-
[10]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
El Banani, M., Raj, A., Maninis, K.K., Kar, A., Li, Y., Rubinstein, M., Sun, D., Guibas,L.,Johnson,J.,Jampani,V.:Probingthe3dawarenessofvisualfoundation models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 21795–21806 (2024)
2024
-
[11]
In: International Conference on Learning Representations (2025)
Gairola, S., Böhle, M., Locatello, F., Schiele, B.: How to probe: Simple yet effective techniques for improving post-hoc explanations. In: International Conference on Learning Representations (2025)
2025
-
[12]
Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? the kitti visionbenchmarksuite.In:2012IEEEConferenceonComputerVisionandPattern Recognition. pp. 3354–3361. IEEE (2012)
2012
-
[13]
He,K.,Zhang,X.,Ren,S.,Sun,J.:Deepresiduallearningforimagerecognition.In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 770–778 (2016)
2016
-
[14]
ACM Computing Surveys54(10s), 1–41 (2022)
Khan, S., Naseer, M., Hayat, M., Zamir, S.W., Khan, F.S., Shah, M.: Transformers in vision: A survey. ACM Computing Surveys54(10s), 1–41 (2022)
2022
-
[15]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision
Li, Y., Zhang, S., Wang, Z., Yang, S., Yang, W., Xia, S.T., Zhou, E.: Token- pose: Learning keypoint tokens for human pose estimation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11313–11322 (Oc- tober 2021)
2021
-
[16]
In: International Conference on Learning Representations (2023)
Li, Y., Bornschein, J., Hutter, M.: Evaluating representations with readout model switching. In: International Conference on Learning Representations (2023)
2023
-
[17]
Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick,C.L.:Microsoftcoco:Commonobjectsincontext.In:EuropeanConference on Computer Vision. pp. 740–755. Springer (2014)
2014
-
[18]
arXiv preprint arXiv:2407.17140 (2024)
Lv, W., Zhao, Y., Chang, Q., Huang, K., Wang, G., Liu, Y.: Rt-detrv2: Improved baseline with bag-of-freebies for real-time detection transformer. arXiv preprint arXiv:2407.17140 (2024)
2024 arXiv
-
[19]
arXiv preprint arXiv:2310.06824 (2023)
Marks, S., Tegmark, M.: The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824 (2023)
2023 arXiv
-
[20]
In: Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision
Meng, D., Chen, X., Fan, Z., Zeng, G., Li, H., Yuan, Y., Sun, L., Wang, J.: Con- ditional detr for fast training convergence. In: Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision. pp. 3651–3660 (2021)
2021
-
[21]
In: World Con- ference on Explainable Artificial Intelligence
Mikriukov, G., Schwalbe, G., Hellert, C., Bade, K.: Evaluating the stability of semantic concept representations in cnns for robust explainability. In: World Con- ference on Explainable Artificial Intelligence. pp. 499–524. Springer (2023) 16 R. Kim et al
2023
-
[22]
Transactions on Machine Learning Research (2024)
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research (2024)
2024
-
[23]
In: Proceedings of the 41st International Con- ference on Machine Learning
Park, K., Choe, Y.J., Veitch, V.: The linear representation hypothesis and the geometry of large language models. In: Proceedings of the 41st International Con- ference on Machine Learning. pp. 39643–39666 (2024)
2024
-
[24]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision
Ranftl, R., Bochkovskiy, A., Koltun, V.: Vision transformers for dense prediction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 12179–12188 (2021)
2021
-
[25]
In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 779–788 (2016)
2016
-
[26]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10684–10695 (2022)
2022
-
[27]
In: European Conference on Computer Vision
Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from rgbd images. In: European Conference on Computer Vision. pp. 746–760. Springer (2012)
2012
-
[28]
Advances in Neural Information Pro- cessing Systems30(2017)
Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in Neural Information Pro- cessing Systems30(2017)
2017
-
[29]
In: 2024 32nd European Signal Processing Conference
Wagner, B., Pellerin, D., Huet, S.: Forgetting analysis by module probing for on- line object detection with faster r-cnn. In: 2024 32nd European Signal Processing Conference. pp. 576–580. IEEE (2024)
2024
-
[30]
In: Conference on Robot Learning
Wang, Y., Guizilini, V.C., Zhang, T., Wang, Y., Zhao, H., Solomon, J.: Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In: Conference on Robot Learning. pp. 180–191. PMLR (2022)
2022
-
[31]
In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al.: Transformers: State-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: S...
2020
-
[32]
Advances in Neural Information Processing Systems37, 21875–21911 (2024)
Yang, L., Kang, B., Huang, Z., Zhao, Z., Xu, X., Feng, J., Zhao, H.: Depth anything v2. Advances in Neural Information Processing Systems37, 21875–21911 (2024)
2024
-
[33]
Advances in Neural Information Pro- cessing Systems37, 43468–43498 (2024)
Zhan, G., Zheng, C., Xie, W., Zisserman, A.: A general protocol to probe large vision models for 3d physical understanding. Advances in Neural Information Pro- cessing Systems37, 43468–43498 (2024)
2024
-
[34]
Zhang, R., Qiu, H., Wang, T., Guo, Z., Cui, Z., Qiao, Y., Li, H., Gao, P.: Monodetr: Depth-guidedtransformerformonocular3dobjectdetection.In:Proceedingsofthe IEEE/CVF International Conference on Computer Vision. pp. 9155–9166 (2023)
2023
-
[35]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Zhao, Y., Lv, W., Xu, S., Wei, J., Wang, G., Dang, Q., Liu, Y., Chen, J.: Detrs beat yolos on real-time object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16965–16974 (2024)
2024
-
[36]
arXiv preprint arXiv:2010.04159 (2020) Probing 3D Understanding of DETRs 17 T able 5:Exact checkpoints used in our experiments
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.: Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159 (2020) Probing 3D Understanding of DETRs 17 T able 5:Exact checkpoints used in our experiments. Model Checkpoint URL DE...
2010 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.