{"id":"7fb19452-62d5-4977-aa25-b9b48ef3e15c","arxiv_id":"2502.06219","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"HFIT fuses frozen vision foundation model features with a side adapter that ingests relative depth maps, achieving top results on Cityscapes and KITTI driving scene parsing.","lead":"This paper introduces a new network, HFIT, that combines a frozen vision foundation model with a lightweight adapter to fuse RGB images and relative depth maps for driving-scene semantic segmentation. The authors report improved accuracy on Cityscapes and KITTI compared to earlier segmentation methods, using predicted relative depth instead of LiDAR depth.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline training protocols are unspecified; the 2.63 mIoU edge over ViT-Adapter could vanish under equal training budgets.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: baseline training protocols are not described, leaving open the possibility that the reported improvements over ViT-Adapter and other methods come from unequal training settings rather than the proposed architecture. This concern is concrete and testable, and it directly affects the central claim of superior performance. The paper does contain internal ablations with a fixed protocol (Tables III–VI), which give some support for the design choices and for the usefulness of relative depth, but those ablations do not validate the headline comparison against external SOTA methods. The 'generalizable' wording is also not backed by cross-dataset experiments, though this is secondary to the protocol issue. Given that the concern is addressable by releasing training configurations or rerunning baselines, a conditional acceptance is appropriate rather than rejection; the reader's verdict already reflects this, so no change is needed.","tokens_in":14810,"tokens_out":4483,"duration_ms":39550,"concrete_test":"Retrain the key baselines (ViT-Adapter, ViT-CoMer, OCRNet, K-Net) under the exact HFIT protocol: same 448×448 crops, 20,000 iterations, same optimizer and learning rate, and for adapter methods use the same frozen Depth Anything V1 Large backbone and decoder head. If ViT-Adapter under these conditions reaches or exceeds 84.74 mIoU on Cityscapes val, the claimed advantage of HFIT is not established. Releasing the exact training configs and seeds for all baselines would also settle whether the comparison was protocol-matched.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central SOTA claim is undercut by a lack of controlled baseline comparisons. Section IV-A specifies training details only for HFIT (20,000 iterations, 448×448 random crops, RTX 3090); no equivalent settings are given for OCRNet, K-Net, EMANet, SNE-RoadSeg, OFFNet, MFNet, FuseNet, ViT-Adapter, or ViT-CoMer in Tables I–II. It is unclear which backbone ViT-Adapter and ViT-CoMer used, whether numbers are the authors' own runs or quoted from other papers, and what optimizer, learning rate, or decoder head was applied. Since HFIT's advantage over ViT-Adapter is only 2.63 mIoU on Cityscapes (84.74 vs 82.11) and 0.96 mIoU on KITTI (80.24 vs 79.28), a modest difference in training budget, resolution, or backbone choice could erase the claimed gain. The internal ablation in Table III does provide a clean protocol-matched comparison of RGB alone versus RGB plus relative depth, and it supports the value of the depth modality, but it does not establish superiority over ViT-Adapter or other SOTA methods. The title's 'generalizable' claim is also unsupported because only Cityscapes and KITTI are used, with no cross-dataset or cross-domain evaluation; this is a secondary concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HFIT, a side-adapter architecture for RGB-depth driving scene parsing that keeps a frozen vision foundation model (ViT) backbone and adds a trainable adapter composed of a Duplex Spatial Prior Extractor (DSPE), Recalibrated Heterogeneous Feature Fusion (RHFF) modules, and Holistic Gated Feature Integration (HGFI) modules. Relative depth maps predicted by a VFM (e.g., Depth Anything V2) are used as the depth input, avoiding reliance on metric depth sensors. Experiments on Cityscapes and KITTI Semantics report state-of-the-art mIoU values (84.74% on Cityscapes, 80.24% on KITTI) and ablations showing that RGB plus relative depth outperforms RGB alone and is comparable to using metric depth. The authors claim that this is the first work to adapt VFMs for RGB-D driving scene parsing and that the approach is generalizable.","tokens_in":15116,"tokens_out":6376,"duration_ms":52570,"significance":"If the experimental claims hold, the paper makes a useful contribution: it demonstrates a parameter-efficient way to adapt frozen vision foundation models to RGB-D scene parsing and shows that VFM-predicted relative depth can serve as a substitute for metric depth. The public release of code is a concrete strength, as is the internal protocol-matched ablation in Table III that isolates the contribution of the depth modality. However, the significance is limited by two gaps: the comparison with prior methods is not protocol-matched, so the magnitude of the reported gains is uncertain, and the 'generalizable' claim is not tested across datasets or domains. The paper also uses a heuristic confidence formulation that is not well justified and may not behave as described.","major_comments":[{"comment":"The training and evaluation protocols for the baseline methods are not specified. Section IV-A reports 20,000 iterations and 448x448 random crops only for HFIT; no equivalent settings are given for OCRNet, KNet, EMANet, SNE-RoadSeg, OFFNet, MFNet, FuseNet, ViT-Adapter, or ViT-CoMer. Because HFIT's advantage over ViT-Adapter is only 2.63 mIoU on Cityscapes and 0.96 mIoU on KITTI, differences in training budget, crop size, backbone, optimizer, or evaluation protocol could erase the claimed gain. Please provide protocol-matched comparisons (ideally all methods trained under the same pipeline) or clearly state that the numbers are quoted from the original papers and cite them.","section":"IV-A, Tables I and II"},{"comment":"The claim that HFIT is 'generalizable' is not supported by the experiments. Only Cityscapes and KITTI Semantics are used, and there is no cross-dataset or cross-domain evaluation (e.g., training on Cityscapes and testing on KITTI, or evaluating on unseen driving scenes). The authors themselves note in Section IV-B that KITTI's smaller size limits generalization. A cross-dataset experiment is needed to justify the title and abstract claims.","section":"Title, Abstract, Section I, Section IV-B"},{"comment":"The confidence maps C^V and C^S are defined as sigma(Conv(-L * log L)), where L is the output of a ReLU. The text states that 'a higher confidence level suggests the pixel significantly contributes to accurate segmentation' (Section III-C). However, for L > 1, -L log L is negative and the sigmoid output approaches 0, meaning that strong activations are treated as low-confidence. This contradicts the described semantics. Moreover, no justification or reference is given for this entropy-like transform. Since these maps directly implement the recalibration in Eq. (7), the RHFF module requires either a corrected formula, a normalization that bounds L to [0,1], or an ablation demonstrating that the particular nonlinearity is essential to the reported performance.","section":"III-C, Eqs. (2) and (5)"},{"comment":"The main experimental configuration is not fully specified. Section IV-A does not state which EfficientNet variant is used in the DSPE for the main results; Table IV shows EfficientNet-B7 achieving the highest mIoU in the ablation, but it is not confirmed as the default choice. In addition, the number of interaction stages N (set to 4 in Section III-A) is not ablated, and hyperparameters such as learning rate, batch size, and optimizer are omitted from the main text. These details are necessary for reproducibility and for fair comparison with the baselines.","section":"IV-A, III-A, Table IV"}],"minor_comments":[{"comment":"The 'error range (%)' columns are not defined in Section IV-A; please describe how the range is computed from the three independent runs.","section":"Tables I and II"},{"comment":"The phrase 'increase in mIoU by 17.45-45.54%' should say 'percentage points' rather than '%' to avoid confusion.","section":"Section IV-B"},{"comment":"The paper argues that the adapter approach has a 'significantly lower number of trainable parameters', but no parameter counts or FLOPs are reported for HFIT or the baselines. Please include these numbers.","section":"Tables I and II, Section I"},{"comment":"The rows 'Depth' and 'Relative Depth' should be explicitly defined (e.g., metric depth from RAFT-Stereo versus relative depth from Depth Anything V2), and the source of the depth maps used in the 'Depth' row should be stated.","section":"Table III"},{"comment":"The paper states 'More details are given in the supplementary material' but does not list the learning rate, batch size, number of classes, or the train/val split. Please provide these details in the main text or ensure the supplementary material is publicly accessible.","section":"Section IV-A"},{"comment":"Please define the base of the logarithm and specify the kernel size k and dilation rate r used in the atrous convolution, as these are not stated in the text.","section":"III-C, Eqs. (2) and (5)"}],"recommendation":"major_revision","confidential_remarks":"The core idea of the paper is interesting and the internal ablations are helpful, but the lack of protocol-matched baseline comparisons is a load-bearing weakness for the main SOTA claim, and the 'generalizable' title is not yet justified. The confidence-map derivation in Section III-C should be revisited, as the sign behavior seems inconsistent with the stated interpretation. I recommend a major revision with controlled experiments and clear documentation of the comparison setup."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: the central SOTA claim is weaker than it looks, but the paper still has a genuinely useful result. HFIT is a frozen-ViT side adapter for RGB-D driving scene parsing, and the cleanest finding is in the ablation: replacing RAFT-Stereo metric depth with Depth Anything's relative depth costs less than 0.12% mIoU. That is a practical result for anyone who wants RGB-D parsing without LiDAR.\n\nWhat's new: the specific combination of a duplex EfficientNet spatial-prior extractor, a recalibrated heterogeneous fusion module (RHFF) that uses confidence maps to gate the injection of depth features into the ViT, and a holistic gated feature integration (HGFI) that mixes multi-level features. The modules are plausible and the ablations are decent — each component earns its place, and the paper releases code. The internal protocol-matched comparison in Table III is the most trustworthy evidence in the paper.\n\nSoft spots, in proportion. First, the comparison against ViT-Adapter and other baselines in Tables I–II is not backed by any training protocol details for those baselines. We don't know the backbone version, crop size, iterations, or optimizer. Since the Cityscapes edge over ViT-Adapter is 2.63 mIoU (84.74 vs 82.11), an unequal training budget could erase it. This is the load-bearing weakness for the 'SOTA' claim. Second, the title says 'generalizable' but both datasets are driving scenes from the same visual domain and there is no cross-dataset or cross-domain evaluation. That word should be softened or tested. Third, the confidence-map formula C = σ(Conv(−L ⊙ log L)) is heuristic. It looks like an entropy-like measure but L is not normalized; the paper doesn't explain why this is the right reliability signal. It works empirically in the ablation, so it's not fatal, but it needs a sentence of grounding. Fourth, a mild self-referential smell: using Depth Anything both to produce the depth inputs and as the backbone could be a hidden bias, though not circular.\n\nWho this is for: researchers working on parameter-efficient adapters for dense prediction or depth-based driving perception. It's a worthy submission for a good venue, not a breakthrough.\n\nRecommendation: send it to peer review, but insist that the authors report baseline training protocols, either add a cross-dataset experiment or change 'generalizable' to something defensible, and comment on the confidence-map equation. With those changes, the practical claim about relative depth is solid.","headline":"A practical adapter paper whose real contribution is showing VFM-predicted relative depth can replace metric depth in RGB-D parsing; the SOTA claims need tighter baseline control.","tokens_in":15610,"tokens_out":2715,"would_cite":true,"duration_ms":24077,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a frozen vision foundation model plus a lightweight depth side adapter can outperform all compared RGB-D scene parsers, reaching 84.74% mIoU on Cityscapes.","keywords":["RGB-D scene parsing","vision foundation models","side adapter","relative depth estimation","heterogeneous feature fusion","driving scene parsing","semantic segmentation","vision transformer"],"falsifier":"Retrain the strongest ViT-Adapter baseline under HFIT's exact protocol (same iterations, crop size, decoder, and depth supervision) and check whether HFIT's reported 2.6-point mIoU lead on Cityscapes and 0.96-point lead on KITTI survive.","tokens_in":14651,"feed_emoji":"🚗","tokens_out":4847,"duration_ms":37605,"temperature":0.7,"pith_summary":"This paper argues that a vision foundation model (VFM) trained only on RGB images can be turned into a strong RGB-depth driving-scene parser without retraining the foundation model. The proposed network, HFIT, freezes the VFM backbone and adds a lightweight side adapter that digests the RGB image together with a relative depth map produced by the VFM itself. On Cityscapes and KITTI Semantics, HFIT reports the best mIoU among all compared single-modal and data-fusion methods, reaching 84.74% on Cityscapes with a Depth Anything V1 backbone. The paper's key practical claim is that relative depth, which any modern VFM can supply, is nearly as useful as metric depth: swapping in relative depth costs less than 0.12% mIoU. If the claim holds, depth sensors become optional for this task, because the adapter needs no LiDAR-accurate depth.","feed_headline":"Frozen ViT plus depth adapter tops RGB-D scene parsing","feed_subtitle":"Relative depth from a foundation model, not metric depth, stays within 0.12% mIoU of full fusion.","key_machinery":"The load-bearing object is the HFIT side adapter, made of three modules: the Duplex Spatial Prior Extractor (DSPE), which uses two identical EfficientNet blocks to build a multi-scale pyramid from RGB and relative depth; the Recalibrated Heterogeneous Feature Fusion (RHFF) module, which computes per-pixel confidence maps from both branches and gates the depth priors by multiplying with the complement of the ViT confidence before injecting them into the ViT with cross-attention; and the Holistic Gated Feature Integration (HGFI) module, which reweights multi-level features from both branches so lower-level detail flows upward only when gates open. The relative-depth input is itself part of the machinery: because the VFM generates the depth map, the system never needs calibrated metric depth.","core_discovery":"The central discovery is that the profound prior knowledge of a frozen vision foundation model can be harvested for RGB-D driving scene parsing by treating the VFM as a fixed feature extractor and attaching a purpose-built side adapter, rather than re-training the ViT or training a second encoder. HFIT feeds RGB and the VFM's own relative depth output through a duplex EfficientNet backbone, then recalibrates those heterogeneous spatial priors with confidence maps before injecting them into the frozen transformer via cross-attention. Multi-level features are then combined by gates that move fine-grained details upward only when they are useful. The result is that HFIT outperforms all compared traditional RGB-D fusion networks, plain pre-trained VFMs, and ViT adapters, reaching 84.74% mean IoU on Cityscapes and 80.24% on KITTI; its ablation shows the gains disappear if either the recalibration weights or the gated integration are removed.","pith_inferences":["A natural extension is to test whether the same frozen-backbone/adapter recipe transfers to other RGB-X tasks, such as RGB-thermal parsing, where the DSPE's duplex branches would ingest a different second modality.","The under-0.12% gap between relative and metric depth is measured only with RAFT-Stereo as the metric source; comparing against LiDAR-grade metric depth from KITTI would sharpen the claim about relative depth being sufficient.","One could put the confidence maps to work at inference time by thresholding low-confidence pixels and querying them for active learning, a use the paper does not discuss."],"forward_implications":["Freezing the ViT and training only the adapter keeps trainable parameters low, so the approach can ride on future, larger foundation models without a matching growth in training cost.","Because relative depth from the VFM substitutes for metric depth within 0.12% mIoU, deployment no longer requires LiDAR or calibrated stereo on the vehicle.","The confidence-based gating in RHFF suggests the model can suppress unreliable depth regions, which should make the system more robust to calibration errors and noisy depth estimates.","On KITTI, the gains over VFM baselines are smaller than on Cityscapes, so the method's advantage grows with dataset diversity and scale."],"supporting_citations":[{"why":"Supplies the frozen DINOv2 backbone whose pre-trained semantic features HFIT adapts for scene parsing.","marker":"[2]"},{"why":"Depth Anything V1 is the backbone that yields HFIT's best Cityscapes result (84.74% mIoU) and the relative depth input.","marker":"[3]"},{"why":"Depth Anything V2 provides the relative depth predictions used to replace metric depth in the side adapter.","marker":"[19]"},{"why":"ViT-Adapter is the single-modal adapter baseline HFIT extends and outperforms on both datasets.","marker":"[24]"},{"why":"ViT-CoMer is the other ViT-adapter baseline compared, with its feature fusion mechanism discussed as failing to bridge the semantic gap.","marker":"[30]"},{"why":"Represents the traditional RGB-D data-fusion segment that HFIT is compared against for freespace and scene parsing.","marker":"[10]"},{"why":"EfficientNet blocks form the DSPE and are selected over ResNet and Swin baselines for spatial-prior extraction.","marker":"[47]"},{"why":"Supplies the metric depth baseline (RAFT-Stereo) against which relative depth is measured to stay within 0.12% mIoU.","marker":"[61]"}],"fun_headline_variants":["Frozen ViT plus depth adapter beats RGB-D fusion","HFIT harnesses VFM depth priors for driving parsing","Relative depth from VFM powers RGB-D scene parser","Side adapter unlocks foundation model for RGB-D"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes every baseline was trained under the same schedule and resolution as HFIT (20,000 iterations, 448x448 crops), but those settings are only stated for HFIT itself.","fun_headline_variants_meta":{"raw":{"variants":["Frozen ViT plus depth adapter beats RGB-D fusion","HFIT harnesses VFM depth priors for driving parsing","Relative depth from VFM powers RGB-D scene parser","Side adapter unlocks foundation model for RGB-D"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000238,"raw_usage":{"total_tokens":1519,"prompt_tokens":961,"completion_tokens":558,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":494}},"tokens_in":577,"tokens_out":558,"duration_ms":5150,"temperature":1.0,"reasoning_tokens":494,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T16:22:06.808131+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the strongest ViT-Adapter baseline under HFIT's exact protocol (same iterations, crop size, decoder, and depth supervision) and check whether HFIT's reported 2.6-point mIoU lead on Cityscapes and 0.96-point lead on KITTI survive.","supporting_citations":[{"cited_title":"Depth Anything: Unleashing the power of large-scale unlabeled data,","cited_arxiv_id":null,"evidence_quote":"Depth Anything V1 is the backbone that yields HFIT's best Cityscapes result (84.74% mIoU) and the relative depth input."},{"cited_title":"Vision Transformer adapter for dense predictions,","cited_arxiv_id":null,"evidence_quote":"ViT-Adapter is the single-modal adapter baseline HFIT extends and outperforms on both datasets."},{"cited_title":"ViT-CoMer: Vision Transformer with convolutional multi- scale feature interaction for dense predictions,","cited_arxiv_id":null,"evidence_quote":"ViT-CoMer is the other ViT-adapter baseline compared, with its feature fusion mechanism discussed as failing to bridge the semantic gap."},{"cited_title":"SNE-RoadSeg: Incorporating surface normal informa- tion into semantic segmentation for accurate freespace detection,","cited_arxiv_id":null,"evidence_quote":"Represents the traditional RGB-D data-fusion segment that HFIT is compared against for freespace and scene parsing."},{"cited_title":"EfficientNet: Rethinking model scaling for con- volutional neural networks,","cited_arxiv_id":null,"evidence_quote":"EfficientNet blocks form the DSPE and are selected over ResNet and Swin baselines for spatial-prior extraction."},{"cited_title":"RAFT-Stereo: Multilevel recurrent field transforms for stereo matching,","cited_arxiv_id":null,"evidence_quote":"Supplies the metric depth baseline (RAFT-Stereo) against which relative depth is measured to stay within 0.12% mIoU."}],"review_version":1}