{"id":"0ecf16a5-dc0e-4b9b-a9ec-1523d9e2575d","arxiv_id":"2501.16410","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"DynAlign aligns source and target taxonomies by using GPT-4 to map labels, SAM for mask proposals, and CLIP to reassign fine-grained target labels, improving unsupervised cross-domain segmentation under label-space shift.","lead":"A new framework, DynAlign, combines an unsupervised domain adaptation model with GPT-4, SAM, and CLIP to segment street scenes into a new set of class labels without target-domain annotations. On GTA-to-Mapillary and GTA-to-IDD benchmarks it improves mIoU over the paper's open-vocabulary baselines, though the gains are modest and several evaluation choices need scrutiny.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Candidate-set restriction from coarse UDA labels, not CLIP confusion, likely explains the per-class failures and undermines the novel-class discovery claim.","rationale":"The reader's weakest assumption points at SAM+CLIP reassignment accuracy, and the per-class failures support that concern. I partially agree but sharpen the mechanism: the method constrains CLIP's decision to the candidate set induced by the coarse UDA label. Even a perfect CLIP model cannot recover if the correct target class is outside that set. This is a more load-bearing condition than raw CLIP discriminability because it is structural and follows directly from Eq. 1, Eq. 7, and the mapping tables in Appendix A.7. The per-class results for banner, phone booth, traffic sign frame, and bench are consistent with candidate-set exclusion. The paper's statement that novel classes are mapped to each source label conflicts with the actual mapping tables, so either the implementation is more restrictive than described or the description would make the CLIP classification task harder; either way, the novel-class discovery claim is not established. I do not think this warrants rejection because the concern is addressable with the proposed candidate-recall analysis and an expanded-candidate experiment, and the reported overall improvements may survive such a check. The reader's CONDITIONAL verdict already captures the need for additional verification; my read does not move that verdict. I also note secondary issues such as the lack of code, absent error bars, and manual refinement of the taxonomy mapping, but these do not change the conditional assessment.","tokens_in":24542,"tokens_out":6145,"duration_ms":59382,"concrete_test":"Compute candidate-set recall on the target validation sets: for each ground-truth mask region, record the source label predicted by the UDA model for that region, then check whether the true target class appears in the corresponding candidate subset from Appendix A.7. Report recall per class and compare against the low-IoU classes in Tables 9 and 10. If bench, banner, phone booth, and traffic sign frame have near-zero candidate recall, the failures are structural. Then rerun the fusion with the candidate set expanded to include all target classes for every mask; if the mIoU of those classes rises substantially, the candidate-set restriction, not CLIP feature quality, is the binding constraint.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that DynAlign can reassign mask regions to correct target classes, including classes absent from the source label space. The mechanism in Section 4.4 does not let CLIP decide freely: for each SAM mask, the majority UDA prediction c_s selects the candidate subset C_t^i via Eq. 1, and Eq. 7 takes the argmax only over that subset. Therefore correct reassignment is possible only if the coarse UDA label for the region is one of the source labels to which the true target class is mapped. Section 4.2 states that novel classes are mapped to 'each source label,' but Appendix A.7 (Tables 11, 12) actually maps each novel class to one or two source labels (e.g., bench only under unlabeled; boat only under car). This internal inconsistency makes the method's ability to discover novel classes depend on the unverified accuracy of the coarse UDA prediction. The per-class failures in Tables 9 and 10 (banner 0.5, phone booth 0.8, traffic sign frame 0.1, bench 1.2 on Mapillary) are consistent with candidate-set exclusion rather than CLIP feature confusion; if the correct class is not in the candidate set, no threshold or text prompt can recover it. This weakens the central claim that the framework can accurately adapt to arbitrary new taxonomies in a fully unsupervised manner.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DynAlign, a framework for unsupervised taxonomy-adaptive cross-domain semantic segmentation in which the source and target label sets differ and no target annotations are available. The method combines a UDA segmentation model (HRDA) that produces coarse predictions in the source label space with foundation models: GPT-4 is used to map source labels to target labels and to generate context-aware text descriptions, SAM generates fine-grained mask proposals, and CLIP is used to assign each mask a target label by text-image similarity. The core technical novelty is the knowledge-fusion mechanism that conditions CLIP's classification on the candidate target classes retrieved from the coarse UDA prediction. Experiments are reported on GTA to Mapillary Vistas and GTA to IDD, with headline mIoU values of 36.7 and 41.7, respectively, compared with 32.9 and 40.2 for the best naive HRDA-plus-open-vocabulary baselines. The paper also includes ablations, per-class tables, computational-efficiency measurements, and a pseudo-label training variant.","tokens_in":24820,"tokens_out":4940,"duration_ms":49611,"significance":"If the central claims hold, the paper addresses a genuinely useful and understudied problem: adapting a semantic segmentation model across both image-level domain shift and label-taxonomy mismatch without target supervision. The modular design, integrating any UDA model with SAM and CLIP, is an attractive and likely to be influential recipe. The manuscript is commendably explicit about its problem definition, provides per-class results in the appendix, and reports an ablation study for each main design choice; the planned code release is also a strength. However, the novelty-discovery claim is weakened by an internal inconsistency in the candidate-set mechanism, and the headline evaluation excludes a substantial fraction of the target label space. These issues are load-bearing for the paper's 'fully unsupervised taxonomy adaptation' claim and need to be addressed before the results can be accepted at face value.","major_comments":[{"comment":"The mechanism in Eq. (7) does not let CLIP reassign a mask to the full target label space: the argmax is taken only over the candidate subset C_t^i induced by the majority source label of the mask through Eq. (1). Therefore correct novel-class assignment is possible only if the coarse UDA prediction is one of the source labels to which the true target class is mapped. Section 4.2 states that novel classes are mapped to 'each source label,' but Appendix A.7 (Tables 11 and 12) actually maps each novel class to one or two source labels (e.g., bench only under 'unlabeled' on Mapillary, boat only under 'car'). This is an internal inconsistency, and it makes the claimed discovery of novel classes depend on the unverified accuracy of the coarse UDA prediction. The per-class failures in Table 9 (banner 0.5, phone booth 0.8, traffic sign frame 0.1, bench 1.2) are consistent with candidate-set exclusion rather than CLIP feature confusion: if the correct target class is not in the candidate set, no confidence threshold or text prompt can recover it. Please either allow CLIP to consider all target labels (or a broader learned candidate set) or provide an explicit analysis showing that candidate retrieval succeeds for novel classes; as written, the central novelty-discovery claim is not supported.","section":"§4.4, Eq. (7), and Appendix A.7, Tables 11–12"},{"comment":"The evaluation covers only 45 of the 66 Mapillary Vistas classes and 24 of the 25 IDD classes, with the excluded categories described only as 'small-scale or less informative' (Section 5.1). Since the target taxonomy is a central input to the problem, omitting 21 of 66 classes can materially inflate the reported mIoU and weakens the claim of seamless adaptation to arbitrary taxonomies. In addition, the 'known' versus 'unknown' split used in Table 1 is never defined in the main text or appendix; the per-class tables do not indicate which classes are considered unknown. Please report results over the full label space (or justify the exclusion with per-class numbers for the omitted classes) and state precisely how the known/unknown partition is computed.","section":"§5.1, Table 1, and Tables 9–10"},{"comment":"The paper repeatedly describes the method as 'fully unsupervised' and 'without requiring any manual annotations,' but Appendix A.7 states that the GPT-4 taxonomy mapping is 'introduce[d] with human intervention to refine these mappings.' The taxonomy mapping is a core component of the method, so human refinement of this mapping is a form of dataset-specific supervision, even if it is not pixel-level annotation. Please clarify the exact role of human intervention, report the prompts and refinement protocol, and temper or qualify the 'fully unsupervised' claim accordingly.","section":"Appendix A.7"},{"comment":"The comparison is limited to two open-vocabulary systems and their naive combination with HRDA. No comparison is made to existing taxonomy-adaptive DA methods (Gong et al., 2022; Fan et al., 2023a) even under their supervised or few-shot settings, nor to a closed-set UDA oracle in which source labels are manually mapped to target labels. Moreover, the reported gains over the best baseline are 3.8 mIoU on Mapillary but only 1.5 mIoU on IDD, and all numbers appear to come from a single run with no error bars or multiple seeds. Given the known run-to-run variance of self-training UDA models, the 'significant improvement' claim on IDD would be more convincing with repeated trials or statistical significance information.","section":"§5.2, Table 1"}],"minor_comments":[{"comment":"The citation placeholder '(Ghiasi et al., 2022; ?)' should be replaced with the intended reference.","section":"§2.2"},{"comment":"The word 'publically' should be 'publicly'.","section":"Abstract"},{"comment":"The sentence beginning 'Typically, In the field of cross-domain semantic segmentation' has a capitalization and punctuation error; 'Typically,' should be lowercase and integrated into the sentence.","section":"§2.1"},{"comment":"The text above Table 4 reads 'Table 4 shows shows the performance'; the duplicated word should be removed.","section":"§5.4, Table 4"},{"comment":"The reference 'Matthias Minderer, Alexey Gritsenko. Scaling open-vocabulary object detection. NeurIPS, 2023' is incomplete; the full author list and venue information are needed.","section":"References"},{"comment":"Please add a note in the appendix specifying which target classes are treated as 'known' and which as 'unknown' in Table 1, since this partition is used throughout the results discussion.","section":"Appendix A.6"}],"recommendation":"major_revision","confidential_remarks":"The paper is already accepted at ICLR 2025, but for this journal the evidence gaps are substantial. The most serious issue is the mismatch between Section 4.2's claim that novel classes are mapped to 'each source label' and the actual tables in Appendix A.7, which map several novel classes to a single source label; combined with the candidate-set restriction in Eq. (7), this undermines the central novelty-discovery claim in a way that is not merely a presentation issue. The evaluation subset and lack of multi-seed results further reduce confidence. I believe the work is salvageable and the direction is valuable, but the authors should be asked to either revise the mechanism or provide explicit evidence that candidate retrieval succeeds for novel classes, and to report results on the full target label space."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper defines a sensible problem—adapting a segmentation model to a new label taxonomy without target annotations—and builds a modular pipeline that works better than the naive open-vocabulary baselines. The ablations are informative, and the pseudo-label training results are a nice addition. The per-class tables are a credit: they show exactly where the method fails.\n\nThe stress-test note is on target. Section 4.2 says novel classes are mapped to \"each source label,\" but Appendix A.7 maps each novel class to only one or two source labels (bench under 'unlabeled', boat under 'car'). Because the candidate set for a mask region is selected by the majority UDA prediction, a region can only be reassigned to a novel class if the coarse UDA label happens to fall in that narrow mapping. This is a structural constraint, not just CLIP confusion. The per-class failures (bench 1.2, phone booth 0.8, traffic sign frame 0.1) are consistent with candidate-set exclusion. The claim that DynAlign discovers arbitrary novel classes in a fully unsupervised way is weaker than the headline suggests.\n\nOther soft spots: no error bars or multiple seeds; baselines are limited to HRDA plus two open-vocabulary models; 21 of 66 Mapillary classes and 1 of 25 IDD classes are excluded from evaluation. The taxonomy mapping relies on GPT-4 with human refinement, which is external prior knowledge—so \"fully unsupervised\" is oversold. These are addressable in revision, but the candidate-set restriction is load-bearing: if the correct target class cannot enter the candidate set, no threshold or prompt tuning can recover it.\n\nWhat holds up: the known-class numbers are solid, the fine-grained splitting of source classes (road to lane-marking, curb, etc.) works well, and the modular design makes the contribution easy to build on. The paper is honest enough to show per-class results that expose its own limitations.\n\nThis paper is for researchers in UDA and open-vocabulary segmentation who want a concrete integration strategy and a benchmark for taxonomy-adaptive DA. It deserves serious refereeing, but it needs major revision: open up the candidate set (or verify novel-class discovery under coarse-label errors), add baselines and seeds, and temper the 'fully unsupervised' claim. I would not desk-reject it; I would send it back for a careful revision.","headline":"A useful integration of UDA with foundation models for taxonomy shift, but the novel-class discovery claim is undermined by a candidate-set restriction that depends on coarse UDA predictions.","tokens_in":25327,"tokens_out":2505,"would_cite":true,"duration_ms":25501,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DynAlign claims a fully unsupervised pipeline can segment images into a target label space that differs from the source label space, using GPT-4 taxonomy mapping, SAM masks, and CLIP fusion.","keywords":["unsupervised domain adaptation","semantic segmentation","taxonomy alignment","open-vocabulary segmentation","foundation models","CLIP","SAM","pseudo-labeling"],"falsifier":"Run the per-class evaluation in Tables 9 and 10 and check the classes the framework itself fails on: banner (0.5 mIoU), phone booth (0.8), traffic sign frame (0.1), bench (1.2) on Mapillary, and drivable fallback (2.9) on IDD. A controlled experiment that replaces the GPT-4 context names with oracle synonyms for exactly those classes and measures whether mIoU jumps would pinpoint whether the bottleneck is the text descriptions or CLIP's visual discrimination; if the gap persists with oracle names, the central reassignment mechanism is the limiting factor.","tokens_in":24334,"feed_emoji":"🏷️","tokens_out":5710,"duration_ms":52373,"temperature":0.7,"pith_summary":"DynAlign claims that a fully unsupervised pipeline can adapt a semantic segmentation model to a target domain whose label set differs from the source's—finer-grained, renamed, or entirely new classes—without a single target annotation. It combines a standard UDA model, which handles the image-level shift and predicts in the source label space, with foundation-model components: GPT-4 builds a source-to-target taxonomy mapping and context descriptions, SAM carves precise mask proposals, and CLIP re-labels each mask by matching multi-scale visual features to context-aware text features. On GTA to Mapillary Vistas and GTA to IDD, it reports mIoU of 36.7 and 41.7, beating naive HRDA-plus-open-vocabulary baselines of 32.9 and 40.2. The paper presents this as the first fully unsupervised formulation of taxonomy-adaptive domain adaptation, where changing the target label set only requires updating the mapping rather than recollecting annotations.","feed_headline":"DynAlign lifts GTA-to-Mapillary mIoU to 36.7 without target labels","feed_subtitle":"It maps source classes to finer or novel target classes, beating HRDA-plus-open-vocabulary baselines on two street benchmarks.","key_machinery":"The load-bearing mechanism is the knowledge-fusion equation: each SAM mask $m$ gets a multi-scale visual feature $F_V$, formed as a similarity-weighted sum of local and global CLIP features (Eq. 5–6), and is assigned to the target class whose GPT-4-context-averaged text feature maximizes $\\langle F_V, F_T\\rangle$ (Eq. 7). The taxonomy mapping $c_s^i \\to C_t^i \\subseteq C_t$ lets a source label open onto several target labels, and Eq. 2 encodes each target label as the average CLIP text feature of context descriptions such as “road bridge” versus “pedestrian bridge.” SAM supplies the regions, CLIP supplies the semantic comparison, and the UDA model supplies the initial source label that selects both the candidate target set and the padding scale for global context.","core_discovery":"On its own terms, the paper's discovery is that the label-level domain gap can be closed by turning the UDA model's source-space prediction into a prior for a foundation-model reassignment step. Given a target image, the UDA model produces a source-space label map; SAM segments the image into fine regions; for each region the majority source label selects a subset of allowed target labels via the LLM taxonomy mapping; and CLIP reclassifies the region by comparing a multi-scale regional visual embedding against context-averaged text embeddings of the candidate target classes. This produces pseudo-labels in the target taxonomy with no target annotations, and the paper reports that they are accurate enough to improve over naive HRDA-plus-open-vocabulary combinations and to train a target-space Mask2Former that is faster and slightly more accurate than direct DynAlign inference.","pith_inferences":["The per-class tables show reassignment fails precisely where CLIP's text names are visually confusable (banner, phone booth, traffic sign frame, bench on Mapillary); a likely next step is replacing global CLIP features with region-specific or fine-grained visual features rather than richer text.","The taxonomy mapping is initiated by GPT-4 but human-refined in the appendix, so the pipeline is unsupervised in pixel labels but not fully autonomous in mapping construction; automating this refinement is a natural extension.","The modularity suggests the same UDA-plus-SAM-plus-CLIP recipe could transfer to other dense prediction settings where target label sets evolve, such as medical imaging or remote sensing, provided the text descriptions carry enough context."],"forward_implications":["A model trained on one taxonomy can produce predictions in a completely different target label space without any target-domain annotations.","Unseen classes receive concrete labels rather than a generic “unknown” class, with unknown-class mIoU more than three points above the strongest HRDA-plus-open-vocabulary baseline on IDD.","When the target taxonomy changes, only the taxonomy mapping and context names need to be updated; direct inference works without retraining, and pseudo-label training yields both higher accuracy and much faster inference.","The same framework preserves or slightly improves traditional UDA performance when source and target label spaces coincide, as shown by GTA-to-Cityscapes mIoU of 75.9 versus 74.9 for the HRDA baseline."],"supporting_citations":[{"why":"Supplies the HRDA UDA segmentation model and training protocol that produce the source-space predictions DynAlign refines.","marker":"Hoyer et al. (2022b)"},{"why":"SAM generates the fine-grained mask proposals that DynAlign reclassifies.","marker":"Kirillov et al. (2023)"},{"why":"CLIP's text and vision encoders provide the open-vocabulary similarity used in knowledge fusion.","marker":"Radford et al. (2021)"},{"why":"ConvCLIP is the vision encoder chosen for multi-scale visual features and gives the best results in ablations.","marker":"Yu et al. (2024)"},{"why":"GPT-4 generates the taxonomy mapping proposals and context-aware descriptions for target labels.","marker":"Achiam et al. (2023)"},{"why":"Prior taxonomy-adaptive DA method that needs few-shot target labels; defines the task family DynAlign makes fully unsupervised.","marker":"Gong et al. (2022)"},{"why":"Grounded-SAM is the open-vocabulary baseline whose naive HRDA combination is the main comparison to beat.","marker":"Ren et al. (2024)"}],"fun_headline_variants":["DynAlign aligns taxonomies unsupervised in cross-domain segmentation","DynAlign closes the label gap in cross-domain segmentation without labels","DynAlign maps source classes to novel target classes with no annotations","DynAlign uses foundation models to align label spaces for segmentation","DynAlign adapts segmentation to new taxonomies with no manual labeling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework works only if CLIP's text-image similarity, using GPT-4's context phrases, can tell apart the fine-grained target classes; when it cannot, the reassignment fails and those classes collapse to near-zero accuracy.","fun_headline_variants_meta":{"raw":{"variants":["DynAlign aligns taxonomies unsupervised in cross-domain segmentation","DynAlign closes the label gap in cross-domain segmentation without labels","DynAlign maps source classes to novel target classes with no annotations","DynAlign uses foundation models to align label spaces for segmentation","DynAlign adapts segmentation to new taxonomies with no manual labeling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000444,"raw_usage":{"total_tokens":2265,"prompt_tokens":979,"completion_tokens":1286,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":1195}},"tokens_in":595,"tokens_out":1286,"duration_ms":11799,"temperature":1.0,"reasoning_tokens":1195,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:32:02.495241+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the per-class evaluation in Tables 9 and 10 and check the classes the framework itself fails on: banner (0.5 mIoU), phone booth (0.8), traffic sign frame (0.1), bench (1.2) on Mapillary, and drivable fallback (2.9) on IDD. A controlled experiment that replaces the GPT-4 context names with oracle synonyms for exactly those classes and measures whether mIoU jumps would pinpoint whether the bottleneck is the text descriptions or CLIP's visual discrimination; if the gap persists with oracle names, the central reassignment mechanism is the limiting factor.","supporting_citations":[],"review_version":1}