Pith. sign in

REVIEW 5 major objections 5 minor 26 references

CoT-Segmenter: Enhancing OOD Detection in Dense Road Scenes via Chain-of-Thought Reasoning

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CoT-generated prompts lift road anomaly segmentation to 0.91 mIoU

desk verdict New idea, but oracle threshold tuning makes the SOTA claim unverifiable. read the letter →

arxiv 2507.03984 v2 pith:UMMTPFQR submitted 2025-07-05 cs.CV

Jeonghyo Song , Kimin Yun , DaeUng Jo , Jinyoung Kim , Youngjoon Yoo This is my paper
classification cs.CV
keywords out-of-distributiondetectionsemanticsegmentationchain-of-thoughtreasoningroadanomalyopen-vocabularyGroundedSAMGPT-4prompt-based
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that chain-of-thought reasoning by a large language model can make open-vocabulary segmentation detect road anomalies better than specialized out-of-distribution models. It identifies three road-scene situations where current OOD segmentation fails—dense overlapping objects, distant small objects, and large foreground-dominant objects—and proposes using GPT-4 to reason step by step about what is abnormal, turning that reasoning into text prompts. A reader should care because the approach needs no retraining or category-specific data; it only needs a prompt that names the anomaly. On the RoadAnomaly benchmark and a harder subset the paper defines, the prompted pipeline reports higher mIoU and F1 scores than the compared methods.

What carries the argument

The load-bearing mechanism is the CoT-based prompt generation module: GPT-4 is prompted to analyze the scene, identify deviations along appearance, semantic, and spatial axes, and output a short state+noun prompt (V1) and a noun-only prompt (V2). These prompts are then resolved into boxes by GroundingDINO and into masks by SAM, with the final prediction being the pixel union of the two prompt-driven masks. The three-step reasoning and the dual-prompt union are what the paper credits for the improvement; ablations show performance rising from 0.088 mIoU with one step to 0.838 with all three steps.

What would settle it

Take a fixed set of images from RoadAnomaly and run GroundedSAM twice, once with CoT prompts and once with the generic prompt 'object', keeping the same box and text thresholds for both; if the mIoU gap shrinks to near zero, the paper's claim that CoT reasoning drives the improvement is falsified.

Watch

Extended reading notes

Core claim

The central claim is that structured visual reasoning, expressed as language, is enough to steer an open-vocabulary segmenter to out-of-distribution objects. The framework breaks reasoning into three steps, S1 scene analysis, S2 deviation detection, S3 final candidate selection, and produces two complementary prompts, a state+noun phrase and a plain noun. Feeding both into GroundedSAM (GroundingDINO for boxes, SAM for masks) and taking the union of the masks yields the final OOD prediction. In the paper's experiments this reaches 0.912 mIoU and 0.941 F1 on RoadAnomaly, and 0.838 mIoU and 0.892 F1 on the newly defined challenging subset, compared with 0.803 and 0.862 for the same model using the generic prompt 'object'.

Load-bearing premise

The comparison assumes that using different per-image threshold values for the detector is fair, and that the baselines were not given the same per-image tuning; if thresholds rather than the CoT prompts explain the gains, the main claim does not follow.

Editorial extensions

If this is right

  • OOD segmentation can be improved at inference time by asking a language model to name the anomaly, without retraining the segmenter.
  • The three challenging scenarios define a test subset that future road-anomaly methods should report separately, since average scores hide large drops on these cases.
  • Prompt phrasing matters quantitatively: describing the anomaly as an OOD object or with contextual state beats the generic prompt 'object' by a large margin.
  • The union of a descriptive prompt and a plain noun prompt outperforms either prompt alone, so prompt diversity is an easy robustness lever.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not compare against using per-image threshold optimization for the baseline prompt 'object' with the same tuning budget, so the marginal gain specifically attributable to CoT reasoning over generic open-vocabulary grounding is not isolated; a fixed-threshold study would be the direct test.
  • The same prompt-generation idea should transfer to other text-conditioned segmenters and other anomaly domains, since the method only needs a vision-language model and a grounded detector; that is an extrapolation, not a claim in the paper.
  • The paper's own conclusion notes that overly long prompts hurt performance, suggesting a future system could iteratively refine prompts using the segmenter's feedback instead of one-shot generation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes CoT-Segmenter, a two-stage framework that uses GPT-4 with Chain-of-Thought prompting to generate textual prompts for GroundedSAM, an open-vocabulary detector and segmenter, and applies it to out-of-distribution (OOD) detection in road scenes. The authors identify three challenging scenarios on the RoadAnomaly dataset, introduce a curated 'Challenging' subset, and report that their method outperforms existing OOD segmentation methods on both the standard and challenging subsets. They also present ablations on prompt types and CoT reasoning stages.

Significance. If the reported gains are real, the paper would demonstrate a useful application of LLM-based visual reasoning to OOD segmentation, and its identification of three failure regimes could be of practical interest. The paper includes a clear motivation, a two-prompt design (V1 state+noun and V2 noun-only), and ablation experiments that attempt to isolate the effect of CoT reasoning. However, the central empirical claim is currently not supported because the evaluation protocol uses per-image threshold optimization on the test set for the proposed method, and the contributions are not accompanied by released prompts or code. The paper's value would be substantially increased by a fixed-threshold evaluation and by providing the exact prompts for reproducibility.

major comments (5)
  1. [Section 4.1] The per-image optimization of the box and text thresholds for GroundedSAM is an oracle protocol: the test ground-truth masks are used to select two thresholds per image that maximize the same mIoU and F1 scores that are then reported. This protocol appears to be applied only to the GroundedSAM rows in Table 1, while the compared methods (S2M, RbA, Mask2Anomaly, SynBoost, RPL+CoroCL, PEBAL) have fixed inference settings. As a result, Table 1 does not establish that the CoT-derived prompts cause the reported improvement, because the per-image oracle tuning alone could account for the margin. The authors should report results with fixed thresholds, for both the proposed method and all baselines, and also report results in which the baselines receive a comparable per-image tuning procedure.
  2. [Tables 2 and 3] The ablation comparisons in Tables 2 and 3 are also performed with per-image threshold optimization for every row. Under that protocol, the relative ordering of prompt types may reflect the optimizer's ability to fit thresholds to the available prompts rather than the semantic quality of the prompts themselves. Notably, the low scores for 'only GPT-4' and '1 step' could be caused by threshold optimization failing for those prompts, not by the absence of CoT reasoning. The authors should rerun these ablations with a single fixed threshold set or with a validation-based threshold selection and report the results.
  3. [Section 4.1, Challenging subset] The 'Challenging' subset is defined post hoc after observing performance drops, but the paper does not specify the exact selection criteria, the number of images, the class distribution, or the overlap with the standard RoadAnomaly split. Because the subset is used to support the claim of robustness, the selection protocol must be fully described and, ideally, predefined before evaluation to avoid cherry-picking. The authors should also report the performance of all methods on the full RoadAnomaly test set without threshold optimization, in addition to the subset results.
  4. [Section 3.2 and reproducibility] The CoT prompts themselves are the core method, yet the paper does not provide the exact prompt templates, the GPT-4 version and decoding settings, or the generated prompts for the test images. Without this information, the results cannot be independently reproduced or compared across works. The authors should release the full set of prompts and the code for the reasoning and grounding pipeline.
  5. [Section 4.1 and Table 1] The paper gives no error bars, variance estimates, or multiple-run statistics for any of the reported numbers. Given that the proposed method depends on stochastic LLM sampling and per-image threshold selection, the reported single-run values, especially the 0.912 vs. 0.803 margin, may not be stable. The authors should report means and standard deviations over multiple runs or, at minimum, over multiple GPT-4 decoding seeds.
minor comments (5)
  1. [Section 1] The phrase 'criti-cal' in the abstract is a line-breaking artifact that should be corrected to 'critical'.
  2. [Section 2.2] The name 'LLaV A' contains an unintended space and should be 'LLaVA'.
  3. [Section 4.2] The sentence contains 'Section. 4.1' with an extra period; it should read 'Section 4.1'.
  4. [Section 3.2] The notation T Vi ∈ {TV1, TV2} is slightly inconsistent because the subscript i is used without a bound; it would be clearer to define i ∈ {V1, V2}.
  5. [Table 3] The table rows 'only GPT-4' and '1 step' lack a clear description of what output is passed to GroundedSAM; the text should clarify whether these rows use the GPT-4 output as the text prompt or as the final segmentation.

Circularity Check

1 steps flagged · score 6.0 of 10

Per-image oracle threshold tuning in Sec. 4.1 makes the reported mIoU/F1 an optimized quantity rather than a prediction, confounding the SOTA comparison and CoT ablations.

  1. fitted input called prediction [Section 4.1 Experimental Setup]
    "For each image, the box threshold and text threshold values in GroundedSAM were individually optimized to maximize segmentation accuracy, reflecting the sensitivity of open-vocabulary grounding to threshold tuning."

    The reported mIoU and F1 scores in Tables 1-3 are the same metrics used as the objective for per-image threshold selection. Because the box/text thresholds are fitted on the test images to maximize the reported accuracy, the resulting numbers are optimized values, not predictions of the CoT method. The compared baselines (S2M, RbA, Mask2Anomaly, etc.) use fixed inference-time settings and receive no comparable per-image oracle, so Table 1's SOTA comparison cannot attribute the margin to CoT prompts. Tables 2-3 also threshold-tune every row, so the prompt and CoT ablations are confounded by threshold overfitting. The central claim that CoT prompts improve OOD segmentation is therefore not independently established by the reported numbers.

full rationale

The CoT-Segmenter framework is not circular in a definitional sense: the CoT prompts are generated by GPT-4, the segmentation is performed by GroundedSAM, and neither component is defined in terms of the reported metric. There is no load-bearing self-citation chain or imported uniqueness theorem. However, the evaluation protocol makes the central empirical claim partly circular by construction: Section 4.1 explicitly states that the box and text thresholds were individually optimized per image to maximize segmentation accuracy. This means the reported mIoU/F1 values are directly fitted to the test set rather than predicted. Because the same optimization is not applied to the compared OOD segmentation baselines, the claimed consistent superiority in Table 1 is not a valid measure of CoT prompt quality. The prompt and CoT ablations in Tables 2 and 3 are similarly affected, since every row receives per-image threshold tuning. The correct remedy would be fixed thresholds or matched tuning for all methods, plus reporting with multiple threshold settings. This is a fitted-input-called-prediction pattern rather than pure self-definition, so the score is 6 rather than higher.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the reliability of GPT-4's reasoning and the suitability of noun-phrase prompts for open-vocabulary grounding. The only explicit free parameters are the per-image GroundingDINO thresholds, which are fitted to maximize test accuracy and thus directly inflate the reported metrics.

free parameters (2)
  • GroundingDINO box threshold = per-image optimized to maximize accuracy
    Section 4.1 states thresholds are individually optimized per image, a fitted hyperparameter that directly influences reported metrics.
  • GroundingDINO text threshold = per-image optimized to maximize accuracy
    Same as the box threshold; also per-image optimized, further inflating reported performance.
assumptions (3)
  • domain assumption GPT-4's chain-of-thought reasoning reliably identifies OOD objects in road scenes and produces prompts that GroundedSAM can ground.
    The method depends on the LLM's visual reasoning; no analysis of failure cases or prompt distribution is provided.
  • domain assumption OOD objects are expressible as short noun or state+noun phrases that an open-vocabulary detector can localize.
    The dual-prompt strategy assumes anomalies can be named in language that GroundingDINO can ground, which may not hold for all anomalies.
  • domain assumption The RoadAnomaly dataset and the custom Challenging subset are representative benchmarks for OOD segmentation.
    Evaluation relies on these datasets without discussion of label noise, selection bias, or how the subset was curated beyond the three scenario types.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CoT-Segmenter: Enhancing OOD Detection in Dense Road Scenes via Chain-of-Thought Reasoning." pith.science (2026). https://pith.science/paper/UMMTPFQR

@misc{pith2026250703984,
  author       = {Pith},
  title        = {Pith review of: CoT-Segmenter: Enhancing OOD Detection in Dense Road Scenes via Chain-of-Thought Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UMMTPFQR}},
  note         = {Machine review of arXiv:2507.03984}
}
read the original abstract

Effective Out-of-Distribution (OOD) detection is criti-cal for ensuring the reliability of semantic segmentation models, particularly in complex road environments where safety and accuracy are paramount. Despite recent advancements in large language models (LLMs), notably GPT-4, which significantly enhanced multimodal reasoning through Chain-of-Thought (CoT) prompting, the application of CoT-based visual reasoning for OOD semantic segmentation remains largely unexplored. In this paper, through extensive analyses of the road scene anomalies, we identify three challenging scenarios where current state-of-the-art OOD segmentation methods consistently struggle: (1) densely packed and overlapping objects, (2) distant scenes with small objects, and (3) large foreground-dominant objects. To address the presented challenges, we propose a novel CoT-based framework targeting OOD detection in road anomaly scenes. Our method leverages the extensive knowledge and reasoning capabilities of foundation models, such as GPT-4, to enhance OOD detection through improved image understanding and prompt-based reasoning aligned with observed problematic scene attributes. Extensive experiments show that our framework consistently outperforms state-of-the-art methods on both standard benchmarks and our newly defined challenging subset of the RoadAnomaly dataset, offering a robust and interpretable solution for OOD semantic segmentation in complex driving environments.

Figures

Figures reproduced from arXiv: 2507.03984 by the authors.

Figure 1
Figure 1. Visualization of three challenging scenarios affecting [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the proposed CoT-Segmenter framework. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Comparison of segmentation results using different prompts: (a), (c) use a generic prompt (‘object’), while (b), (d) use CoT [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

26 extracted references · 19 canonical work pages

  1. [1]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Alt- man, S. Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Alayrac, J

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning.NeuriPS, 2022

    work page 2022

  3. [3]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Ka- plan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sas- try, A. Askell, et al. Language models are few-shot learners.NeuriPS, 2020

    work page 2020

  4. [4]

    R. Chan, M. Rottmann, and H. Gottschalk. En- tropy maximization and meta classification for out- of-distribution detection in semantic segmentation. In ICCV, 2021

    work page 2021

  5. [5]

    K. Chen, Z. Zhang, W. Zeng, R. Zhang, F. Zhu, and R. Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023

    arXiv 2023

  6. [6]

    Di Biase, H

    G. Di Biase, H. Blum, R. Siegwart, and C. Ca- dena. Pixel-wise anomaly detection in complex driv- ing scenes. InCVPR, 2021

    work page 2021

  7. [7]

    Grci ´c, P

    M. Grci ´c, P. Bevandi´c, and S. ˇSegvi´c. Densehybrid: Hybrid anomaly detection for dense open-set recogni- tion. InECCV, 2022

    work page 2022

  8. [8]

    Hendrycks, S

    D. Hendrycks, S. Basart, M. Mazeika, A. Zou, J. Kwon, M. Mostajabi, J. Steinhardt, and D. Song. Scaling out-of-distribution detection for real-world settings.arXiv preprint arXiv:1911.11132, 2019

    arXiv 1911

Show all 26 references
  1. [9]

    Ilyas, I

    S. Ilyas, I. Freeman, and M. Rottmann. On the potential of open-vocabulary models for object de- tection in unusual street scenes.arXiv preprint arXiv:2408.11221, 2024

    2024 arXiv

  2. [10]

    Kirillov, E

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.- Y . Lo, et al. Segment anything. InICCV, 2023

    2023

  3. [11]

    Liang, W

    C. Liang, W. Wang, J. Miao, and Y . Yang. Gmmseg: Gaussian mixture based generative semantic segmen- tation models.NeuriPS, 2022

    2022

  4. [12]

    K. Lis, K. Nakka, P. Fua, and M. Salzmann. Detecting the unexpected via image resynthesis. InICCV, 2019

    2019

  5. [13]

    H. Liu, C. Li, Q. Wu, and Y . J. Lee. Visual instruction tuning.NeuriPS, 2023

    2023

  6. [14]

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. InECCV, 2024

    2024

  7. [15]

    Y . Liu, C. Ding, Y . Tian, G. Pang, V . Belagiannis, I. Reid, and G. Carneiro. Residual pattern learning for pixel-wise out-of-distribution detection in seman- tic segmentation. InICCV, 2023

    2023

  8. [16]

    Z. Liu, Y . Dong, Y . Rao, J. Zhou, and J. Lu. Chain- of-spot: Interactive reasoning improves large vision- language models.arXiv preprint arXiv:2403.12966, 2024

    2024 arXiv

  9. [17]

    Mitra, B

    C. Mitra, B. Huang, T. Darrell, and R. Herzig. Com- positional chain-of-thought prompting for large multi- modal models. InCVPR, 2024

    2024

  10. [18]

    Nayal, M

    N. Nayal, M. Yavuz, J. F. Henriques, and F. G ¨uney. Rba: Segmenting unknown regions rejected by all. In ICCV, 2023

    2023

  11. [19]

    Z. Peng, W. Wang, L. Dong, Y . Hao, S. Huang, S. Ma, and F. Wei. Kosmos-2: Grounding multimodal large language models to the world.arXiv preprint arXiv:2306.14824, 2023

    2023 arXiv

  12. [20]

    S. N. Rai, F. Cermelli, D. Fontanel, C. Masone, and B. Caputo. Unmasking anomalies in road-scene seg- mentation. InICCV, 2023

    2023

  13. [21]

    T. Ren, S. Liu, A. Zeng, J. Lin, K. Li, H. Cao, J. Chen, X. Huang, Y . Chen, F. Yan, et al. Grounded sam: As- sembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159, 2024

    2024 arXiv

  14. [22]

    H. Shao, S. Qian, H. Xiao, G. Song, Z. Zong, L. Wang, Y . Liu, and H. Li. Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning.NeuriPS, 2024

    2024

  15. [23]

    Sinhamahapatra, F

    P. Sinhamahapatra, F. Schwaiger, S. Bose, H. Wang, K. Roscher, and S. Guennemann. Finding dino: A plug-and-play framework for unsupervised detection of out-of-distribution objects using prototypes.arXiv preprint arXiv:2404.07664, 2024

    2024 arXiv

  16. [24]

    Y . Tian, Y . Liu, G. Pang, F. Liu, Y . Chen, and G. Carneiro. Pixel-wise energy-biased abstention learning for anomaly segmentation on complex urban driving scenes. InECCV, 2022

    2022

  17. [25]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. NeuriPS, 2022

    2022

  18. [26]

    W. Zhao, J. Li, X. Dong, Y . Xiang, and Y . Guo. Seg- ment every out-of-distribution object. InCVPR, 2024

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.