{"id":"245ce9f1-a711-4aae-8707-85ab1ccd6a2f","arxiv_id":"2508.07552","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A CLIP-based network predicts a feature-map score defined as 80% NDS ratio plus 20% similarity to SOTA features; using it as an auxiliary loss gives a 3.89% average NDS gain on BEVFormer.","lead":"The paper scores intermediate feature maps by how close they are to a state-of-the-art model's feature maps, blends this with the normalized NDS metric, and uses the score as an extra loss when training BEVFormer. It reports a 3.89% average NDS gain on nuScenes, but the score is partly built from NDS itself, so the gain does not independently validate the quality metric.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FMQS auxiliary loss is 80% normalized NDS, so the reported NDS gain is expected even if feature-quality scoring adds nothing; Table 4 also contradicts joint-integration claim.","rationale":"The paper's central claim is that integrating CLIP-FMQE-Net with FMQS as an auxiliary loss improves 3D detection because it evaluates and guides feature-map quality. The most load-bearing assumption is not just that similarity to one SOTA checkpoint measures quality, but that the validation can distinguish the quality signal from the target-metric signal. Eq. 6 makes those inseparable: FMQS is 80% NDS/NDS_sota. Therefore the reported NDS improvement is confounded by direct metric optimization. This is a circularity concern, not a mere disagreement with consensus. The paper also contains internal empirical inconsistencies: Table 4 shows combined IFEM+BFEM integration is sometimes worse than single-module integration, contradicting the text and Figure 3's claim that joint integration is consistently better. Without error bars, multiple seeds, or a baseline such as 'use NDS itself as auxiliary loss,' the 3.89% average gain cannot be attributed to the proposed quality score. The component ideas—CLIP-based alignment, feature similarity, and auxiliary losses—are plausible, and the regression experiments show internal fit, but they do not isolate the causal role of feature-quality scoring. A w=0 ablation would settle whether the micro-level similarity term contributes at all. Because the manuscript's central validation is confounded and currently unsupported, the reader's REJECT verdict is appropriate; the concern does not require changing that verdict but sharpens why the evidence fails.","tokens_in":10583,"tokens_out":5020,"duration_ms":61593,"concrete_test":"Retrain CLIP-FMQE-Net with FMQS labels using only the micro-level CS-CosSim term (set w=0 in Eq. 6), then repeat the Section 4.4 integration under identical settings and seeds. Compare the average NDS gain over the eight configurations. If the gain drops from 3.89% to near zero or is not consistently positive, the reported improvement is attributable to the 80% NDS leakage in the loss, not to feature-map quality scoring. As a secondary control, permute the stage-level macro scores randomly; if gains persist, the macro term acts only as a generic regularizer.","verdict_should_be":"REJECT","load_bearing_attack":"The load-bearing weakness is the circular validation in Eqs. (6) and (12)-(13). FMQS = 0.8·(NDS/NDS_sota) + 0.2·CS-CosSim. The auxiliary loss is 1 − FMQS_pred, which, even ignoring prediction error, is 0.8·(1 − NDS/NDS_sota) + 0.2·(1 − CS-CosSim). Thus 80% of the auxiliary training signal is a monotone transform of the exact target metric (NDS). Adding such a loss should improve NDS even if the feature-similarity term and CLIP alignment are completely uninformative. The regression labels are also dominated by a stage-level constant (NDS/NDS_sota is shared by all samples in a stage), so the predictor can learn a maturity/configuration proxy rather than per-sample feature-map quality; the reported R²≈0.85 is therefore not evidence that the score measures quality. Additionally, Table 4 undercuts the joint-integration claim: for Res50-SCA-TSA, individual gains are 5.72% and 5.74% but combined gain is 4.46%; for VoV-SCA-RCF, 1.58%/1.63% individually vs 1.32% combined. With no error bars or repeated seeds, the average 3.89% gain is not established. The central claim—that a learned feature-quality score improves detection—requires separating the NDS component from the similarity component.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Feature Map Quality Score (FMQS) for evaluating intermediate functional modules in BEVFormer-style autonomous driving models. FMQS combines a macro-level model-performance score (normalized NDS) with a micro-level feature-map similarity score (CS-CosSim against a SOTA checkpoint), weighted 0.8/0.2. A CLIP-based network, CLIP-FMQE-Net, is trained to regress FMQS from feature maps and ground-truth text, and the predicted score is then used as an auxiliary loss during BEVFormer training. Experiments on nuScenes report high regression accuracy (average R² ≈ 0.855 for IFEM, 0.794 for BFEM) and an average 3.89% NDS improvement when the auxiliary loss is integrated. The central claim is that FMQS provides an independent, interpretable quality signal that improves detection performance.","tokens_in":10980,"tokens_out":3969,"duration_ms":46267,"significance":"If validated, the idea of a decoupled, quantitative quality score for intermediate feature maps would be useful for modular perception systems: it could enable independent module monitoring, configuration selection, and auxiliary supervision without changing the task head. The paper also addresses a real gap in the BEV perception literature, where intermediate feature quality is rarely evaluated separately from final task metrics. The authors provide a concrete scoring formula, a regression network, and a training-integration recipe. However, the paper currently lacks the experimental controls needed to establish that the proposed quality signal, rather than the NDS component embedded in the score, drives the reported gains. The significance is therefore conditional on a component-level ablation that the manuscript does not provide.","major_comments":[{"comment":"The auxiliary loss is not independent of the target metric. Eq. (6) defines FMQS = 0.8·Score_model + 0.2·Score_feature, and Eq. (2) defines Score_model = NDS/NDS_sota. The training loss in Eq. (12) is L_FMQS = 1 − FMQS_pred. Even if the predictor were perfect, 80% of the training signal is a monotone transform of NDS, so the reported NDS gain is expected even if both the CS-CosSim term and the CLIP alignment contribute nothing. The paper needs an ablation that isolates the 0.2 similarity component, e.g., comparing against a loss using only Score_model, only Score_feature, and a directly injected NDS-based reward. Without this, the central claim that the learned feature-quality score improves detection is not established.","section":"§3.2.3, Eqs. (2), (6), (12)"},{"comment":"The text claims that joint integration 'consistently achieves higher NDS than single-module integration,' but Table 4 contradicts this for three of eight configurations: Res50-SCA-TSA has individual gains 5.72% and 5.74% but a combined gain of 4.46%; VoV-SCA-RCF has 1.58%/1.63% individually vs 1.32% combined; VoV-GKT-TSA has 2.24%/2.36% vs 0.70% combined. Moreover, the table reports no baseline NDS values, no standard deviations, and no number of seeds, so the average 3.89% gain is not statistically supported. At minimum, the paper should report mean ± std over multiple runs and provide the baseline NDS for each configuration.","section":"§4.4, Table 4"},{"comment":"The regression labels are dominated by a stage-level constant. For a given configuration–stage pair, Score_model is identical for all samples, and it contributes 80% of the FMQS label. The per-sample variation comes only from the 0.2-weighted CS-CosSim term. The reported R² ≈ 0.85/0.79 may therefore reflect the predictor learning configuration/stage identity rather than per-sample feature-map quality. The paper should report the R² after removing the stage-level component (e.g., predicting only the CS-CosSim residual) or compare against a baseline that uses one-hot configuration/stage features.","section":"§4.1–§4.2, Eqs. (1)–(6)"},{"comment":"The loss weights w_BEV, w_IFEM, and w_BFEM in Eq. (13) are never reported, nor is the relative scale of the FMQS losses to the BEV detection loss. Because the auxiliary loss includes a large NDS-normalized term, the reported improvements could be an artifact of weighting rather than of the proposed quality scoring. The paper should report these weights, the gradient scales, and a sensitivity analysis over at least a small range of weights.","section":"§4.4, Eq. (13)"},{"comment":"The FMQS regression dataset is built from nuScenes-mini (404 keyframe samples), while the NDS values in Table 1 are presumably computed on the full nuScenes validation set. The paper does not clarify whether the NDS used in Eq. (2) is computed on the same mini split used to store feature maps. If the NDS comes from the full validation set, then the macro-level score does not correspond to the stored mini-split feature maps, undermining the label construction. Please specify the evaluation protocol for the NDS values used in FMQS.","section":"§4.2 (FMQS-Dataset) and §4.1 (Table 1)"}],"minor_comments":[{"comment":"Notation issue: 'Score isota,jsota model' should be formatted as Score_{i_sota, j_sota}^{model}; also 'isota' appears without proper subscripting.","section":"§3.2.2, Eq. (2)"},{"comment":"Typo: 'activvation' should be 'activation'.","section":"§4.2"},{"comment":"The header 'IEFM' should read 'IFEM' for consistency with the Image Feature Extraction Module.","section":"Table 4"},{"comment":"Typo: 'constructe' should be 'construct'.","section":"§3.2"},{"comment":"The temperature τ in the contrastive loss is introduced but its value is never reported. Please provide the setting used in the experiments.","section":"§3.3, Eq. (9)"},{"comment":"No code or trained model weights are provided. Given that the method relies on a custom scoring pipeline and the nuScenes-mini regression dataset, releasing code would substantially aid reproducibility.","section":"General"}],"recommendation":"reject","confidential_remarks":"The central issue is not incremental novelty but the inability to attribute the reported NDS gains to the proposed feature-quality scoring mechanism. Eq. (6) makes FMQS 80% normalized NDS, and Eq. (12) converts it into the training loss, so the headline result is confounded by construction. The Table 4 inconsistencies further undermine the joint-integration claim. If the authors can provide a component-level ablation and full training details, a resubmission could be reconsidered, but in its current form the manuscript does not support its central claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline is that this is a reasonable extension of the authors' earlier RSS work [19], but the main claimed result—that the FMQS auxiliary loss improves BEVFormer—is confounded by construction. The reported 3.89% NDS gain should not be read as evidence that the feature-quality score works as advertised.\n\nThe genuinely new piece is converting the evaluation score into a trainable auxiliary loss, using CLIP to align feature maps with text-encoded ground truth. That is a plausible direction, and the regression pipeline is described clearly enough to reproduce the FMQS labels. The regression numbers (R2 ~0.85 for IFEM) look respectable, though they don't tell you much once you notice the label structure.\n\nThe load-bearing problem is Eqs. (6) and (12). FMQS is 80% normalized NDS and 20% CS-CosSim. The auxiliary loss is 1 minus the predicted FMQS. Since the predictor is trained to track the true FMQS, the gradient signal is dominated by a monotone function of the target metric. So the loss would push NDS up even if the CLIP alignment and similarity term were pure noise. The paper does not ablate the NDS component, which is the one control that would separate the quality-score hypothesis from the trivial \"optimize a proxy for NDS\" explanation. The R2 values are also inflated by the fact that NDS/NDS_sota is a stage-level constant shared by all samples in a configuration; the model can learn the maturity curve without measuring per-sample feature-map quality.\n\nThe integration results are weaker than presented. For several configurations, the combined IFEM+BFEM gain is below the individual gains (e.g., Res50-SCA-TSA: 5.72% and 5.74% individually, 4.46% combined; VoV-SCA-RCF: 1.58/1.63 vs 1.32). The text says the joint curve is \"consistently\" better, but Table 4 contradicts that. With no baseline NDS values, only relative percentages, and no variance across seeds, the average 3.89% is not established.\n\nMinor issues: the loss weights w_BEV, w_IFEM, w_BFEM and the temperature tau are not reported. A comparison to a simple auxiliary loss that directly uses NDS, or to a standard feature-distillation baseline, is absent. That would be the real test.\n\nTo be fair, the paper is a direct and honest extension of the authors' prior work, and the writing is mostly clear. The core idea of using a learned feature-quality score as supervision is worth exploring. But as presented, the evidence doesn't separate the interesting part from the trivial NDS proxy.\n\nI would not cite this in its current form, but I would send it to peer review rather than desk-reject: the idea is significant enough, and the flaw is fixable with the right control experiments. A referee can ask for an NDS-ablated loss, absolute metric values, error bars, and the missing hyperparameters.","headline":"The paper's central claim is confounded: the auxiliary loss is 80% a normalized NDS, so the 3.89% gain isn't evidence for the feature-quality score; still worth a referee.","tokens_in":11475,"tokens_out":2953,"would_cite":false,"duration_ms":32684,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a Feature Map Quality Score, computed from NDS ratios and cosine similarity to a top checkpoint and predicted by a CLIP-based network, can serve as an auxiliary loss that improves BEVFormer 3D detection on nuScenes by","keywords":["feature map quality scoring","functional module evaluation","autonomous driving perception","3D object detection","BEVFormer","CLIP","auxiliary loss","nuScenes"],"falsifier":"Run BEVFormer training with the FMQS auxiliary loss replaced by (a) a constant, (b) a random score, or (c) the macro-level NDS ratio alone. If any of these reproduce the 3.89% NDS gain, the micro-level feature-similarity signal is not what carries the improvement. Separately, freeze a module's weights after random initialization, or feed shuffled input, and check that its FMQS drops below the SOTA's; a single case where a degraded or random feature map scores at or above the SOTA map falsifies the quality interpretation.","tokens_in":10489,"feed_emoji":"🚗","tokens_out":7139,"duration_ms":77417,"temperature":0.7,"pith_summary":"End-to-end driving models lack explicit supervision for their internal modules, so developers cannot tell how mature a module is or guide it directly. This paper proposes a Feature Map Quality Score (FMQS) that combines a macro-level NDS ratio—how close a module configuration's detection score is to the best of 64 configuration-stage runs—with a micro-level channel-spatial cosine similarity between the module's feature map and that best checkpoint's feature map. It then trains CLIP-FMQE-Net, a lightweight network that aligns feature maps with CLIP text embeddings of ground-truth annotations and regresses FMQS, and uses the predicted score as an auxiliary loss during BEVFormer training. On nuScenes, this yields an average 3.89% NDS gain in 3D object detection, with the image and BEV feature modules helping most when applied together. If correct, the result means hidden module quality can be quantified and optimized without task-level labels for the intermediate representations.","feed_headline":"Feature-map quality scoring lifts 3D detection by 3.89% NDS","feed_subtitle":"A CLIP-based score grades each module's feature map, then trains BEVFormer to keep that grade high.","key_machinery":"The object that carries the argument is the Feature Map Quality Score (FMQS), computed as the weighted fusion (with $w=0.8$) of a macro-level score—the current configuration-stage NDS divided by the SOTA NDS—and a micro-level score, the Channel-Spatial Cosine Similarity between the module's feature map and the SOTA checkpoint's feature map, averaging cosine similarity over channels and spatial positions. Around this scalar, the paper builds a dual-granularity scoring system that generates training labels for stored feature maps, a CLIP-based encoder pair that projects feature maps and textualized ground truth into one semantic space, and a Transformer-decoder head that predicts the score. Th","core_discovery":"The central claim is that a functional module's training maturity can be captured by a scalar score, FMQS, built as a weighted fusion (with weight $w=0.8$) of a task-level score—the module's NDS divided by the NDS of the best configuration-stage checkpoint—and a structure-level score, the Channel-Spatial Cosine Similarity between the module's feature map and that checkpoint's feature map. The paper further claims that this score is learnable: CLIP-FMQE-Net, using a CLIP text encoder for ground truth and a light CNN for feature maps, aligns the two modalities in a shared semantic space and regresses FMQS with a Transformer decoder, reaching average $R^2 = 0.855$ for the image feature extracti","pith_inferences":["The method's reference is a single SOTA checkpoint; if that checkpoint is unrepresentative, the score inherits its biases. Replacing it with an ensemble or a task-derived ideal feature distribution is a natural stress test the paper does not run.","The CLIP text encoder may be doing less semantic work than claimed: because ground-truth text is template-based, a fixed learned embedding of annotation classes might align feature maps just as well, at lower cost.","The 3.89% average gain mixes large gains (up to 7.36%) with much smaller ones (e.g. 0.70% in one configuration), so the mechanism's benefit may depend on which backbone and head are used.","The same FMQS formulation would apply to other tensor-producing modules—for example, occupancy or trajectory predictors—provided their ground truth can be rendered as text templates; testing that extension would clarify whether the score is generic or BEV-specific."],"forward_implications":["Developers can monitor module maturity during training from a single predicted score, without waiting for final task metrics.","The auxiliary loss can be applied during BEVFormer training with frozen CLIP-FMQE-Net parameters, so the evaluator does not need to be retrained for each run.","Jointly supervising both the image-level and BEV-level feature modules gives a larger average gain (3.89% NDS) than either alone, suggesting multi-module quality signals compound.","Because FMQS correlates with final NDS, it offers a cheap selection criterion for configuration and checkpoint choice in multi-module learning pipelines.","The approach transfers CLIP's cross-modal alignment to a regression setting where only a few hundred feature-map samples are needed to predict quality."],"supporting_citations":[{"why":"Supplies the feature map-truth representation similarity-based evaluation framework and RSS that this paper extends, plus the experimental setup.","marker":"[19]"},{"why":"Provides the pretrained CLIP model and cross-modal semantic alignment used by the ground-truth text encoder.","marker":"[13]"},{"why":"Defines BEVFormer, the perception architecture whose feature maps are evaluated and whose training receives the FMQS auxiliary loss.","marker":"[9]"},{"why":"Introduces Multi-Module Learning, the decoupled training paradigm and notion of module configurations used to generate the FMQS labels.","marker":"[4]"},{"why":"Earlier feature map convergence evaluation method that this work extends from single-module to cascaded functional modules.","marker":"[18]"},{"why":"Supplies the nuScenes dataset, NDS metric, and ground-truth labeling schema used for both scoring and evaluation.","marker":"[2]"},{"why":"Cited as evidence that CLIP-style alignment and auxiliary losses improve BEV detectors, motivating the loss design.","marker":"[12]"},{"why":"Cited as an example where auxiliary losses improve BEV detection performance.","marker":"[8]"}],"fun_headline_variants":["Feature-map scoring lifts 3D detection NDS by 3.89%","CLIP-based feature-map grader improves 3D detection by 3.89% NDS","A trainable feature-map quality score boosts NDS 3.89%","Scoring feature maps with CLIP enhances detection NDS 3.89%","Module maturity via feature-map score raises NDS 3.89%"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The pipeline depends on the premise that a feature map's quality is faithfully measured by its channel-spatial cosine similarity to one chosen best checkpoint's feature map, supplemented by an NDS ratio, and that CLIP text embeddings of annotations share a semantic space with BEV feature maps; if either premise fails, the score, its regression labels, and the auxiliary loss lose their meaning.","fun_headline_variants_meta":{"raw":{"variants":["Feature-map scoring lifts 3D detection NDS by 3.89%","CLIP-based feature-map grader improves 3D detection by 3.89% NDS","A trainable feature-map quality score boosts NDS 3.89%","Scoring feature maps with CLIP enhances detection NDS 3.89%","Module maturity via feature-map score raises NDS 3.89%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000596,"raw_usage":{"total_tokens":2635,"prompt_tokens":760,"completion_tokens":1875,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":1769}},"tokens_in":504,"tokens_out":1875,"duration_ms":14212,"temperature":1.0,"reasoning_tokens":1769,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T22:01:33.848581+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run BEVFormer training with the FMQS auxiliary loss replaced by (a) a constant, (b) a random score, or (c) the macro-level NDS ratio alone. If any of these reproduce the 3.89% NDS gain, the micro-level feature-similarity signal is not what carries the improvement. Separately, freeze a module's weights after random initialization, or feed shuffled input, and check that its FMQS drops below the SOTA's; a single case where a degraded or random feature map scores at or above the SOTA map falsifies the quality interpretation.","supporting_citations":[{"cited_title":"Bevdepth: Acquisition of reliable depth for multi-view 3d object detection","cited_arxiv_id":null,"evidence_quote":"Cited as an example where auxiliary losses improve BEV detection performance."},{"cited_title":"Unveiling the Black Box: Independent Functional Module Evaluation for Bird's-Eye-View Perception Model","cited_arxiv_id":"2409.11969","evidence_quote":"Supplies the feature map-truth representation similarity-based evaluation framework and RSS that this paper extends, plus the experimental setup."},{"cited_title":"Learning transferable visual models from natural lan- guage supervision","cited_arxiv_id":null,"evidence_quote":"Provides the pretrained CLIP model and cross-modal semantic alignment used by the ground-truth text encoder."},{"cited_title":"Bev- former: learning bird’s-eye-view representation from lidar-camera via spatiotemporal transformers","cited_arxiv_id":null,"evidence_quote":"Defines BEVFormer, the perception architecture whose feature maps are evaluated and whose training receives the FMQS auxiliary loss."},{"cited_title":"Hierarchical and Decoupled BEV Perception Learning Framework for Autonomous Driving","cited_arxiv_id":"2407.12491","evidence_quote":"Introduces Multi-Module Learning, the decoupled training paradigm and notion of module configurations used to generate the FMQS labels."},{"cited_title":"Feature map convergence evaluation for functional mod- ule","cited_arxiv_id":null,"evidence_quote":"Earlier feature map convergence evaluation method that this work extends from single-module to cascaded functional modules."},{"cited_title":"nuscenes: A multimodal dataset for autonomous driving","cited_arxiv_id":null,"evidence_quote":"Supplies the nuScenes dataset, NDS metric, and ground-truth labeling schema used for both scoring and evaluation."},{"cited_title":"Clip-bevformer: Enhancing multi-view image-based bev detector with ground truth flow","cited_arxiv_id":null,"evidence_quote":"Cited as evidence that CLIP-style alignment and auxiliary losses improve BEV detectors, motivating the loss design."}],"review_version":1}