Pith. sign in

REVIEW 3 major objections 5 minor 31 references

Vision Foundation Model Embedding-Based Semantic Anomaly Detection

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read The paper claims that a detector built from frozen DINOv2 embeddings can match GPT-4o at spotting semantic driving anomalies on CARLA and can localize them as well.

desk verdict A useful engineering extension of prior embedding-based anomaly detection, but the headline GPT-4o comparison rests on thresholds tuned on the test set and needs a validation split before it can be believed. read the letter →

arxiv 2505.07998 v1 pith:5D2VBML5 submitted 2025-05-12 cs.CV cs.LG

classification cs.CVcs.LG
keywords semanticanomalydetectionvisionfoundationmodelDINOv2embeddingsautonomousdrivingout-of-distributioninstancesegmentationlocalizationCARLAsimulation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Semantic anomalies are contextually invalid pairings of familiar objects, such as a truck carrying a traffic light or a stop sign on a billboard, and they can trigger unsafe behavior in autonomous vehicles. The paper claims these anomalies can be detected by comparing local image embeddings from the vision foundation model DINOv2 against a database of embeddings from nominal driving frames, without needing anomaly examples or VLM reasoning. Its instance-based variant, which segments objects with OWLv2 and SAM2 and scores each object, plus a small-component filter, reaches frame-level F1 0.51 on CARLA-simulated anomalies versus 0.47 for GPT-4o, while also producing spatial localization masks. This matters because embedding-based detectors are much lighter than large vision-language models and can supply a pixel-level alarm in addition to a frame-level flag.

What carries the argument

The load-bearing object is the DINOv2 patch embedding: each $14\times14$ pixel region of an image maps to a 384-dimensional vector, giving 256 patch embeddings per frame. These are compared against a cache of patch embeddings from nominal frames using the score $s(e_t;\mathcal{D}_e):=-\max_{e_i\in\mathcal{D}_e}\max_{j,k}\frac{e_t^{(j)\top} e_i^{(k)}}{\|e_t^{(j)}\|\,\|e_i^{(k)}\|}$, i.e., the negative maximum cosine similarity over all patch pairs, and exceeding a threshold $\tau$, set as an $\alpha$-quantile of nominal scores in leave-one-out fashion, flags the frame as anomalous. In the instance-based variant, OWLv2 supplies object boxes, SAM2 converts them into masks, and DINOv2 patch embeddings within each mask are averaged into an object-level embedding. A post-processing filter removes small connected components from the anomaly map to suppress isolated false positives.

What would settle it

Re-run the CARLA evaluation with thresholds fixed on a held-out validation split, or report a threshold-free metric such as AUROC; if the instance-with-filter F1 no longer reaches or exceeds GPT-4o's 0.47, the claimed equivalence is an artifact of threshold selection rather than of the embeddings themselves.

Watch

Extended reading notes

Core claim

The paper's central discovery is that semantic anomalies, which involve familiar objects in unusual arrangements, leave a detectable trace in the embedding space of a frozen vision foundation model. The proposed framework computes the negative maximum cosine similarity between every patch of a runtime image and the nearest patch in a cache of nominal embeddings; an object-centric variant averages patch embeddings inside instance masks produced by OWLv2 and SAM2, and then filters out small connected components in the binary anomaly map. On the CARLA evaluation, this instance-based method with filtering achieves a full-dataset F1 of 0.51 against 0.47 for GPT-4o, and it outperforms GPT-4o specifically on the traffic-light and stop-sign scenarios while providing sharp localization. The authors argue that these results make embedding-based detection a credible path toward real-time semantic anomaly detection in autonomous systems.

Load-bearing premise

The headline comparison rests on the anomaly threshold and the minimum patch size being chosen empirically on the evaluation set itself; if those thresholds were tuned to maximize test-set F1, the reported parity or advantage over GPT-4o may not carry over to deployment.

Editorial extensions

If this is right

  • A semantic-anomaly detector can be built without anomaly examples, using only a cache of nominal driving frames and a frozen vision encoder.
  • The instance-based variant localizes anomalies to object masks, so a downstream planner can react to the specific offending object rather than only to a frame-level flag.
  • On the CARLA evaluation, the filtered instance method reaches F1 0.51 versus 0.47 for GPT-4o overall, and beats it on the traffic-light and stop-sign semantic anomaly scenarios.
  • Filtering small connected components raises F1 by about 0.11 for the instance method while cutting false positives, making post-processing a cheap robustness lever.
  • Score distributions separate nominal from anomalous objects in most scenarios, but the ranges differ per anomaly type, so a single global threshold will not work equally well everywhere.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An immediate test the paper leaves open: fix the threshold on a separate validation split or report AUROC; if the edge over GPT-4o vanishes, the headline depends on threshold selection rather than on the embedding representation.
  • Because the method mainly responds to visual novelty, a natural extension is to combine local embedding distance with a relational or graph model of object co-occurrence, which the paper names in its outlook.
  • Although the evaluation is on synthetic CARLA data, the same pipeline is directly portable to real driving logs; the paper's own failure analysis suggests domain shift is likely to hurt, so real-data performance is an open question.
  • A latency comparison against GPT-4o would quantify the real-time motivation, since the paper frames embedding-based detection as a way around the several-second response times of VLMs.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript proposes two DINOv2-embedding-based semantic anomaly detectors for autonomous driving imagery: a grid-patch variant and an instance-mask variant that uses OWLv2 proposals and SAM2 masks, together with a connected-component filtering post-process. On the CARLA-based dataset introduced in [4] and [7], the instance-based variant with filtering is reported to reach a full-dataset F1 of 0.51 versus 0.47 for GPT-4o, and to outperform GPT-4o on the Traffic Light and Stop Sign semantic-anomaly scenarios while also producing spatial localization heatmaps. The paper further analyzes failure modes of GPT-4o and of the embedding-based pipelines.

Significance. If the quantitative claims were validated, the paper would contribute a lightweight, localizable alternative to VLM-based semantic anomaly monitoring, with potential for real-time use. The strengths include a clear pipeline description, standard components (DINOv2, OWLv2, SAM2), a transparent evaluation protocol, honest discussion of failure modes, and qualitative evidence that score distributions separate anomalous from nominal instances. The central numerical comparison, however, relies on thresholds selected on the same evaluation data, and the localization claim is only qualitatively supported; these issues currently prevent the abstract's headline claims from being substantiated.

major comments (3)
  1. [IV; III-B.2] The headline comparison to GPT-4o is not validated because the anomaly threshold and the filtering patch-size threshold are selected on the same evaluation data: Section IV states only that "Thresholds for anomaly score and patch size are determined empirically," and Figure 5 is a threshold sweep on the test scenarios. The anomaly threshold is an α-quantile of leave-one-out nominal scores, which by construction controls the false-positive rate; choosing α on test labels can improve reported F1 without any improvement in the embedding representation. I request a held-out validation split with a pre-specified operating point (or a report of full PR/ROC curves) and separate test-set results, so that the comparison with the fixed, zero-shot GPT-4o baseline is fair.
  2. [III-A; Table I] The abstract's claim of "precise anomaly localization" is not supported by any quantitative localization metric. The only localization-related criterion is the TP definition in Section III-A that requires IoU with the ground-truth mask of at least 0.3, and Table I reports only frame-level TPR, FPR, and F1. Please add per-frame or per-instance IoU, pixel accuracy, or a localization F1/AP metric; without such numbers, the localization contribution remains qualitative.
  3. [Table I; Section IV-A] Table I reports a full-dataset F1 of 0.51 for Instance (F) versus 0.47 for GPT-4o and a Stop Sign F1 of 0.23 versus 0.19, all at a single operating point on one simulated dataset, with no confidence intervals, repeated runs, or significance tests. Given the small magnitude of these differences and the test-selected operating point, the numerical "comparable" and "outperforms" claims cannot be interpreted as robust. I request bootstrap confidence intervals or error bars over scenarios, and ideally multiple dataset variants or a statistical test of the F1 differences.
minor comments (5)
  1. [Equation (1)] The patch indices j and k are not defined; please state explicitly that j ranges over the p patch embeddings of the runtime image and k over the p patch embeddings of each cached nominal image.
  2. [Abstract; Introduction] The term "real-time" appears in the last sentence of the Introduction and in the Abstract, but no runtime or latency measurements are reported; either add timing numbers or soften the claim to "potentially suitable for real-time."
  3. [Table I caption] The table caption should expand "NF" and "F" to "No Filter" and "Filter" so the table is self-contained.
  4. [References] Reference [13] uses ellipses in the author list ("R. Bommasani, D. A. Hudson, ..., D. Demszky, ..."); a complete author list is required before publication.
  5. [References] References [8] and [14] are duplicate citations of the same CLIP paper; this duplication should be consolidated or cross-referenced to avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the embedding-based detector is an independent empirical method compared against an external GPT-4o baseline.

full rationale

The paper's derivation chain is self-contained. The proposed anomaly score (Eq. 1) is a nearest-neighbor cosine dissimilarity between DINOv2 patch/instance embeddings and a nominal cache; the threshold is an alpha-quantile of leave-one-out nominal scores; filtering removes small connected components. None of these quantities is defined in terms of the evaluation labels or of the GPT-4o comparison, and no equation reduces to another by construction. The self-citations [4], [7] supply the semantic-anomaly definition, dataset, and a prior framework, but the central claim—that instance-level embeddings plus filtering reach GPT-4o-level F1 on CARLA—is an external empirical comparison, not an analytical consequence of those citations. The only fitted quantities are operating-point parameters (alpha and patch-size threshold), described as 'determined empirically'; to the extent they are selected on the test set, that is an evaluation-bias limitation, not circularity, because the method's representational content does not depend on those values.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new entities are introduced; the 'semantic anomaly' concept and all model components are from prior work. The free parameters are the detection thresholds, which are fit to data and under-specified.

free parameters (2)
  • Anomaly threshold α-quantile = not reported (α unspecified)
    Set as α-quantile of leave-one-out nominal scores (Section III-B.2); α is not reported and appears to be chosen on the evaluation data (Section IV).
  • Filtering pixel threshold = not reported
    Removes small connected components; size threshold 'determined empirically' (Section IV).
assumptions (4)
  • domain assumption Semantic anomalies are detectable as low similarity to a fixed library of nominal embeddings (Section III-A).
    The entire framework assumes that contextually invalid scenes produce embeddings far from safe-scene embeddings.
  • domain assumption The CARLA-simulated dataset from [4],[7] is representative of semantic anomalies for autonomous driving.
    All conclusions are drawn from this simulated dataset; no real-world validation.
  • domain assumption OWLv2/SAM2 instance segmentation on synthetic CARLA images is reliable enough to produce object-centric embeddings.
    The paper itself reports oversegmentation and false detections on synthetic images (Section IV-A, Appendix V-E), so this is a fragile premise.
  • standard math Leave-one-out estimation on nominal sequences gives a valid threshold for false-positive control.
    Statistical assumption that nominal score distribution is representative; but the α choice is not principled.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Vision Foundation Model Embedding-Based Semantic Anomaly Detection." pith.science (2026). https://pith.science/paper/5D2VBML5

@misc{pith2026250507998,
  author       = {Pith},
  title        = {Pith review of: Vision Foundation Model Embedding-Based Semantic Anomaly Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5D2VBML5}},
  note         = {Machine review of arXiv:2505.07998}
}
read the original abstract

Semantic anomalies are contextually invalid or unusual combinations of familiar visual elements that can cause undefined behavior and failures in system-level reasoning for autonomous systems. This work explores semantic anomaly detection by leveraging the semantic priors of state-of-the-art vision foundation models, operating directly on the image. We propose a framework that compares local vision embeddings from runtime images to a database of nominal scenarios in which the autonomous system is deemed safe and performant. In this work, we consider two variants of the proposed framework: one using raw grid-based embeddings, and another leveraging instance segmentation for object-centric representations. To further improve robustness, we introduce a simple filtering mechanism to suppress false positives. Our evaluations on CARLA-simulated anomalies show that the instance-based method with filtering achieves performance comparable to GPT-4o, while providing precise anomaly localization. These results highlight the potential utility of vision embeddings from foundation models for real-time anomaly detection in autonomous systems.

Figures

Figures reproduced from arXiv: 2505.07998 by the authors.

Figure 1
Figure 1. Examples of semantic anomalies (a,c) and their [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the proposed vision-based semantic [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Qualitative comparison of anomaly detections. Each method (embedding-based left, instance-based right) shows two [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Distribution of anomaly scores for anomalies (T) and nominal objects (N) across scenarios. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Threshold sweep showing metric trends (IoU, F1, TPR, FPR) for both methods across all scenarios. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Example images from the different scenarios used for [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Embedding-based anomaly detection pipeline: (a) computes anomaly scores from embedding distances. (b) shows the final classification after thresholding. (a) Object detection (b) Instance segmentation.. (c) Anomaly scoring heatmap (d) Anomaly classification result [PIT…
Figure 8
Figure 8. Figure 8: Instance-based anomaly detection pipeline: (a) shows the object detection output. (b) visualizes the segmentation mask of SAM2 that used the bounding boxes for prompting. (c) illustrates the anomaly score heatmap. (d) presents the final classification output, highlight…
Figure 9
Figure 9. Figure 9: GPT-4o failure cases where anomalies are either [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Examples where false positives in the object detector [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 23 canonical work pages

  1. [4]

    Semantic anomaly detection with large language models,

    A. Elhafsi, R. Sinha, C. Agia, E. Schmerling, I. A. D. Nesnas, and M. Pavone, “Semantic anomaly detection with large language models,” Autonomous Robots , vol. 47, no. 8, pp. 1035–1055, Oct

  2. [7]

    Real-time anomaly detection and planning with large language models,

    R. Sinha, A. Elhafsi, C. Agia, M. Foutter, E. Schmerling, and M. Pavone, “Real-time anomaly detection and planning with large language models,” in Robotics: Science and Systems , Delft, Netherlands, Jul. 2024. [Online]. Available: https://arxiv.org/abs/2407. 08735

  3. [1]

    Scaling waymo one safely across four cities this year,

    Waymo, “Scaling waymo one safely across four cities this year,” https://waymo.com/blog/2024/03/ scaling-waymo-one-safely-across-four-cities-this-year, March 2024, accessed: 2025-04-07. [Online]. Available: https://waymo.com/blog/ 2024/03/scaling-waymo-one-safely-across-four-cities-this-year

  4. [2]

    Any tesla driver can now join full self-driving beta regardless of safety score,

    A. Nedelea, “Any tesla driver can now join full self-driving beta regardless of safety score,” InsideEVs, November 2022, accessed: 2025-04-07. [Online]. Available: https://insideevs.com/news/623469/ tesla-fsd-beta-no-safety-score-required/

  5. [3]

    A system-level view on out-of-distribution data in robotics,

    R. Sinha, S. Sharma, S. Banerjee, T. Lew, R. Luo, S. M. Richards, Y . Sun, E. Schmerling, and M. Pavone, “A system-level view on out-of-distribution data in robotics,” 2022. [Online]. Available: https://arxiv.org/abs/2212.14020

  6. [5]

    Watch tesla autopilot get bamboozled by a truck hauling traffic lights,

    D. Robitzski, “Watch tesla autopilot get bamboozled by a truck hauling traffic lights,” June 2021, accessed: 2025-04-07. [Online]. Available: https://futurism.com/the-byte/ tesla-autopilot-bamboozled-truck-traffic-lights

  7. [6]

    (2021, April) Tesla keeps ”slamming on the brakes” when it sees stop sign on billboard

    ——. (2021, April) Tesla keeps ”slamming on the brakes” when it sees stop sign on billboard. Accessed: 2025-04-16. [Online]. Available: https://futurism.com/the-byte/ tesla-slamming-brakes-sees-stop-sign-billboard

  8. [9]

    Masked autoencoders are scalable vision learners,

    K. He, X. Chen, S. Xie, Y . Li, P. Doll ´ar, and R. Girshick, “Masked autoencoders are scalable vision learners,” in 2022 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR) , 2022, pp. 15 979–15 988

Show all 31 references
  1. [10]

    Dinov2: Learning robust visual features without supervision,

    M. Oquab, T. Darcet, T. Moutakanni, H. V . V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, R. Howes, P.-Y . Huang, H. Xu, V . Sharma, S.-W. Li, W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, J. Mairal, P. Labatut,...

  2. [11]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,

    L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu, “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,” ACM Trans. Inf. Syst. , vol. 43, no. 2, Jan. 2025. [Online]. Available...

  3. [12]

    It’s not just a stop sign,

    B. R. Kelly, “It’s not just a stop sign,” Kentucky Teacher , October 2017, accessed: 2025-04-16. [Online]. Available: https: //www.kentuckyteacher.org/features/2017/10/its-not-just-a-stop-sign/

  4. [13]

    On the opportunities and risks of foundation models,

    R. Bommasani, D. A. Hudson, ..., D. Demszky, ..., and P. Liang, “On the opportunities and risks of foundation models,” arXiv preprint arXiv:2108.07258, Jul. 2021

  5. [14]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in Proceedings of the 38th International Conference on Machine L...

  6. [15]

    Emerging properties in self-supervised vision trans- formers,

    M. Caron, H. Touvron, I. Misra, H. J ´egou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision trans- formers,” in Proceedings of the International Conference on Computer Vision (ICCV), 2021

  7. [16]

    Vision transform- ers need registers,

    T. Darcet, M. Oquab, J. Mairal, and P. Bojanowski, “Vision transform- ers need registers,” 2023

  8. [17]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, P. Doll ´ar, and R. Gir- shick, “Segment anything,” arXiv:2304.02643, 2023

  9. [18]

    SAM 2: Segment anything in images and videos,

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R ¨adle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V . Alwala, N. Carion, C.-Y . Wu, R. Girshick, P. Dollar, and C. Feichtenhofer, “SAM 2: Segment anything in images and videos,” in The Thirteenth Inter...

  10. [19]

    Simple open-vocabulary object detection,

    M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn, A. Dosovitskiy, A. Mahendran, A. Arnab, M. Dehghani, Z. Shen, X. Wang, X. Zhai, T. Kipf, and N. Houlsby, “Simple open-vocabulary object detection,” in Computer Vision – ECCV 2022 , S. Avidan, G. Brostow, M. Ciss´...

  11. [20]

    Scaling open- vocabulary object detection,

    M. Minderer, A. A. Gritsenko, and N. Houlsby, “Scaling open- vocabulary object detection,” in Thirty-seventh Conference on Neural Information Processing Systems , 2023. [Online]. Available: https://openreview.net/forum?id=mQPNcBWjGc

  12. [21]

    Segment every out- of-distribution object,

    W. Zhao, J. Li, X. Dong, Y . Xiang, and Y . Guo, “Segment every out- of-distribution object,” in 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024, pp. 3910–3920

  13. [22]

    Anomalyclip: Object- agnostic prompt learning for zero-shot anomaly detection,

    Q. Zhou, G. Pang, Y . Tian, S. He, and J. Chen, “Anomalyclip: Object- agnostic prompt learning for zero-shot anomaly detection,” in The Twelfth International Conference on Learning Representations , 2023. APPENDIX This appendix provides additional data to support the paper’s f...

  14. [24]

    7 visualizes interme- diate steps of the embedding-based anomaly detection

    Embedding-Based Anomaly Detection Visualization: To better understand the pipeline, Fig. 7 visualizes interme- diate steps of the embedding-based anomaly detection. The left image shows the anomaly score heatmap, where higher values (yellow) indicate stronger anomalies. As see...

  15. [25]

    8 shows all intermediate steps

    Instance-Based Anomaly Detection Visualization: To better understand the instance-based pipeline, Fig. 8 shows all intermediate steps. The first image presents object de- tection. While the boat is correctly detected, the streetlight is detected multiple times. The second imag...

  16. [26]

    Is this common to see while driving?

  17. [27]

    Can this influence the vehicle’s behavior?

  18. [28]

    Can the vehicle drive safely in its presence?

  19. [29]

    • Anomaly: May pose risks, lead to unsafe be- havior, or cause violations

    Can this cause the vehicle to make unpre- dictable or unsafe maneuvers? Based on the analysis, I will classify each object as either: • Nominal (Normal) Observation : Does not im- pact safety or traffic rule compliance. • Anomaly: May pose risks, lead to unsafe be- havior, or ...

  20. [30]

    Individual Object Assessments: Answer the four questions and classify each object

  21. [31]

    Overall Scenario Analysis: Reasoning based on all objects

  22. [32]

    Example: The vehicle observes: • A cyclist riding on the sidewalk Classification: Normal (a) Anomaly scoring heatmap (b) Anomaly classification result

    Overall Scenario Classification: Nominal or Anomaly. Example: The vehicle observes: • A cyclist riding on the sidewalk Classification: Normal (a) Anomaly scoring heatmap (b) Anomaly classification result. Fig. 7: Embedding-based anomaly detection pipeline: (a) computes anomaly...

  23. [2023]

    Available: https://arxiv.org/abs/2305.11307

    [Online]. Available: https://arxiv.org/abs/2305.11307

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.