{"id":"0a125557-3d6a-414c-b947-d81a8d223478","arxiv_id":"2506.07412","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The first compressed feature quality assessment benchmark is released, and three standard similarity metrics are shown to correlate inconsistently with task-level semantic distortion.","lead":"This paper introduces a benchmark dataset for evaluating how well quality metrics measure semantic distortion in compressed features, covering 300 original and 12,000 compressed features from three vision tasks and four codecs. It reports that standard metrics such as MSE, cosine similarity, and CKA fail to consistently track task performance, motivating better feature-quality metrics.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth labels for Seg and Dpt are output-drift measures, not task-performance drops, so the benchmark's central semantic-distortion claim is unsupported for 2 of 3 tasks.","rationale":"The reader's weakest_assumption points to the general risk that the task-specific proxies are noisy or discontinuous. My stress test identifies a more fundamental and more specific issue: for Seg and Dpt, the published labels are not task-performance measures at all. The paper's own formulation (Sec 3.1) requires a performance metric A_T evaluated on the task head output (implicitly against ground truth), but Sec 4.6 compares the compressed-feature prediction to the original-feature prediction. This is an internal inconsistency, not a matter of consensus. If confirmed, the benchmark's headline contribution for 2/3 tasks collapses from 'semantic distortion labels' to 'prediction-sensitivity labels,' and Table 4's correlations no longer support the claim that simple metrics fail to predict semantic distortion. The proposed test is feasible because the released code and dataset include the original and compressed features and the task heads; ground-truth annotations for VOC and NYUv2 are public. Since the issue is addressable (labels could be recomputed with ground-truth references, or the claims narrowed to output-drift assessment), the correct verdict remains conditional rather than accept or reject. The reader's verdict is already CONDITIONAL, so my recommendation does not change it.","tokens_in":11516,"tokens_out":7594,"duration_ms":85358,"concrete_test":"On a random subset of 50 Seg and 50 Dpt source features from the released dataset, recompute the ground-truth distortion using the original task ground truth: for Seg, D_seg = mIoU(pred_from_f, GT) - mIoU(pred_from_f_hat, GT); for Dpt, D_dpt = RMSE(pred_from_f_hat, GT) - RMSE(pred_from_f, GT). Then recompute the per-feature PLCC and SROCC for MSE, cosine similarity, and CKA against these performance-based labels and compare with Table 4. If the average absolute SROCC shift exceeds ~0.2, or if the relative ranking of the three metrics changes for either task, the paper's labels do not measure task performance for Seg/Dpt, and the dataset's stated ground truth is invalid for those tasks.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sec 3.1 defines the ground-truth semantic utility as s = A_T(h_T(f_hat)), where A_T is a task performance metric. Sec 4.6, however, operationalizes Seg distortion as the 'mIoU difference between the segmentation masks predicted from f_hat and f' and Dpt distortion as the 'RMSE difference between the depth maps predicted from f_hat and f.' These are output-drift or prediction-sensitivity measures, not task-performance degradation against ground truth. For example, a compressed feature that changes an incorrect original segmentation into a different incorrect segmentation can receive a large mIoU-difference label, while a compressed feature that preserves an incorrect prediction is labeled as having zero distortion. Consequently, the PLCC/SROCC values in Table 4 for Seg and Dpt quantify how well MSE/cosine/CKA track prediction drift, not semantic utility loss. The abstract's claim that the dataset provides 'task-specific performance degradation' as ground truth is thus unsupported for two of the three tasks, and the conclusion that existing metrics 'fail to predict semantic distortion' (Sec 5.5) rests on labels that do not match the problem formulation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper formalizes the task of Compressed Feature Quality Assessment (CFQA), introduces a benchmark dataset of 300 original and 12,000 compressed features derived from three vision tasks (classification, segmentation, depth) and four feature codecs, proposes task-specific semantic distortion labels, and evaluates three baseline metrics (MSE, cosine similarity, CKA) using PLCC and SROCC. The authors conclude that none of the three metrics consistently captures semantic distortion and release the dataset and code.","tokens_in":11704,"tokens_out":9185,"duration_ms":101711,"significance":"The proposed dataset and evaluation protocol are a potentially valuable community resource: if the ground-truth labels are valid, the paper provides the first public CFQA benchmark with paired semantic-distortion labels across multiple tasks and codecs, and the systematic baseline comparison addresses a real gap. The manuscript is an empirical benchmark rather than a derivation, so circularity is not an issue, and the release of data and code is a concrete strength. However, the significance is conditional on the correctness and unambiguous definition of the ground-truth labels, and on the statistical reliability of the reported correlations; the current manuscript does not fully establish either.","major_comments":[{"comment":"The definition of ground-truth semantic distortion is internally inconsistent. Sec. 3.1 defines the ground truth as task performance s = A_T(h_T(f_hat)), and the abstract claims the dataset provides 'task-specific performance degradation.' Sec. 4.6, however, operationalizes Seg distortion as 'the mIoU difference between the segmentation masks predicted from f_hat and f' and Dpt distortion as 'the RMSE difference between the depth maps predicted from f_hat and f.' These are output-drift measures, not task-performance degradation against ground truth: a compressed feature that changes an incorrect original prediction into a different incorrect prediction receives a large label, while one that preserves the incorrect prediction receives zero. In contrast, Tables 2 and 3 report standard task-performance numbers (Acc., mIoU, RMSE), and the sign pattern in Table 4 is consistent with task-performance labels. Please disambiguate: if the actual labels are output drift, the abstract's claim is unsupported for two of the three tasks and the conclusions in Sec. 5.5 must be rephrased; if the labels are true task-performance drops, Sec. 4.6 must be rewritten to say 'difference in mIoU with ground truth' and 'difference in RMSE with ground truth.'","section":"Sec. 4.6 vs Sec. 3.1"},{"comment":"For Cls, the distortion label is 'the rank in the softmax function generated from f_hat.' This is not equivalent to task-performance degradation: if the true class has rank 2, the top-1 prediction is still correct, so a conventional accuracy-based performance measure would show no degradation. The paper should either report top-1 accuracy drop as an additional or alternative label, or provide a clear justification for why softmax rank is a valid proxy for semantic utility loss. As written, the 'true semantic distortion' claim for the Cls task is weaker than stated.","section":"Sec. 4.6 (Cls label)"},{"comment":"Per-feature PLCC and SROCC are computed over only 10 rate points per codec. Table 4 then reports averages over the 100 source features, but no standard deviations, confidence intervals, or significance tests are given. With n=10 per correlation, the reported differences are potentially dominated by sampling noise; for example, the task-specific Hyperprior Cls SROCC values range from 0.1303 (MSE) to -0.6084 (cosine), and it is unclear whether these are statistically distinguishable. Please report variance or bootstrap confidence intervals for the Table 4 averages and for the Figure 2 distributions, and add appropriate significance tests (e.g., paired tests across the 100 features).","section":"Sec. 5.2 and Table 4"},{"comment":"The Introduction claims 'there is no public dataset that provides compressed features with corresponding task performance across multiple tasks and codecs,' but reference [22] (Feature Coding in the Era of Large Models: Dataset, Test Conditions, and Benchmark) appears to be a closely related public resource by the same group. Please explain how the proposed CFQA dataset differs from [22] and justify the 'first benchmark' claim more carefully; if [22] already provides compressed features with task-performance labels, the novelty claim needs to be narrowed.","section":"Introduction and Ref. [22]"}],"minor_comments":[{"comment":"There is a typo in the sentence 'The dataset is designed to support the analysis ofsemantic distortion' — 'ofsemantic' should be 'of semantic.'","section":"Sec. 4.1"},{"comment":"The paper should state the sign convention for the correlation coefficients explicitly. For similarity metrics such as cosine similarity and CKA, negative PLCC/SROCC values are expected when the ground truth is a distortion measure, while for MSE positive values are expected; without this clarification, readers may misinterpret the signs in Table 4.","section":"Table 4 and Sec. 5.4"},{"comment":"The axis labels in Figure 2 appear garbled (e.g., '1 0 Correlation'), and the caption does not fully explain how the histograms are constructed from per-feature PLCC values. Please redraw the figure with clear axis labels and describe the rounding/binning procedure in the caption.","section":"Figure 2"},{"comment":"The manuscript does not specify how MSE, cosine similarity, and CKA are computed for features of shape 2×4×1611×1536, particularly how the tensor is flattened or pooled before computing these metrics and what kernel/HSIC estimator is used for CKA. Please provide these details in the dataset or supplementary material to support reproducibility.","section":"Sec. 5.1 and reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The overlap with reference [22] should be verified by the editor: if [22] already provides a public dataset of compressed features with task-performance labels, the 'first benchmark' claim needs substantial revision. I also recommend asking the authors to clarify the ground-truth label definitions before acceptance, since the current text is ambiguous between output drift and task-performance degradation. The dataset and code URLs should be checked for availability."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You'll want to know two things about this one. First, it ships a genuinely useful artifact: 300 original features, 12,000 compressed versions, three tasks, four codecs, released code and data, plus a simple evaluation protocol for CFQA metrics. That is a real contribution to the feature coding community, and the rate-accuracy tables give a decent picture of how the codecs behave. Second, the semantic-distortion labels for segmentation and depth do not measure what the paper says they measure. Section 4.6 defines Seg distortion as the mIoU difference between masks predicted from the compressed feature and the original feature, and Dpt similarly as RMSE difference between the two predicted depth maps. Those are output-drift measures, not task-performance degradation against ground truth. A compressed feature that changes one wrong segmentation into another wrong segmentation gets a large distortion label; one that preserves a wrong prediction gets zero. The problem formulation in Section 3.1 defines semantic utility as A_T(h_T(f_hat)) against ground truth, so the operationalization does not match the stated goal for two of three tasks. The stress-test note is right, and this undercuts the conclusion that MSE, cosine, and CKA fail to predict semantic distortion across all tasks. The claim is solid for classification, where softmax rank is a reasonable task measure, but not for Seg or Dpt. That is the main soft spot. There are two smaller issues. Correlations are computed per feature over only 10 rate points, with no variance or significance statistics; the histograms in Figure 2 give some sense of spread, but not enough to judge whether the differences between metrics are real. And the paper claims the dataset is representative without comparing to any broader distribution of features; the rate-accuracy diversity shown is necessary but not sufficient for that. On balance, the dataset is worth having and the paper deserves a serious referee, but it needs a major revision before publication. The authors should either recompute the Seg and Dpt labels against ground-truth annotations (they have the source datasets), or explicitly reframe the benchmark as measuring prediction consistency and soften the claims accordingly. I'd bring it to our reading group because the flaw is instructive, and I'd cite the dataset with caution after the labels are corrected.","headline":"A real first benchmark for compressed feature quality, but the ground-truth labels for two of the three tasks measure output drift rather than task-performance drop, so the headline negative result is only solid for classification.","tokens_in":684,"tokens_out":797,"would_cite":true,"duration_ms":42791,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper establishes a benchmark for compressed feature quality assessment and shows that MSE, cosine similarity, and CKA fail to consistently measure the semantic distortion introduced by feature compression.","keywords":["compressed feature quality assessment","feature coding","semantic distortion","benchmark dataset","DINOv2","cosine similarity","CKA","coding for machines"],"falsifier":"Recompute the ground-truth labels using more than one task-head checkpoint or repeated inference, then re-run the PLCC and SROCC evaluation; if the rankings of the ten compressed versions change materially, or if expanding to, say, 50 versions per source shifts the correlations by more than about 0.1, the benchmark's labels are too noisy to support the reported conclusions.","tokens_in":11314,"feed_emoji":"📊","tokens_out":4249,"duration_ms":40948,"temperature":0.7,"pith_summary":"This paper defines a research problem: estimating the semantic quality of a compressed feature without running the downstream model. It builds the first benchmark for the problem, with 300 original DINOv2 features and 12,000 compressed versions spanning image classification, semantic segmentation, and depth estimation, produced by four codecs at many bitrates. For every compressed feature the authors attach a task-specific ground-truth distortion label, then test whether MSE, cosine similarity, and CKA can predict those labels. The outcome is that no standard metric predicts semantic distortion consistently across tasks and codecs, with cosine similarity the strongest but still unstable for learned codecs. A sympathetic reading is that the dataset and protocol make CFQA a tractable, measurable problem and motivate task-aware or learned quality metrics.","feed_headline":"Benchmark: standard metrics can't judge compressed features","feed_subtitle":"12,000 compressed features across three tasks expose why quality needs its own metric, not pixel-level error.","key_machinery":"The load-bearing object is the paired dataset with its semantic-distortion labels, together with the evaluation protocol of PLCC and SROCC computed per source feature over its ten compressed versions. The dataset's 300 original features come from DINOv2's intermediate blocks; the four codecs are HM, VTM, a multi-task Hyperprior, and a task-specific Hyperprior; the labels are task-head performance drops. The protocol turns CFQA into a supervised correlation problem: a metric is good if its scores rank the ten distortions the same way the task-level labels do.","core_discovery":"The central discovery is that semantic distortion caused by feature compression is a measurable but poorly predicted quantity. The authors define true semantic distortion as performance degradation of task heads: softmax rank for classification, mIoU difference for segmentation, and RMSE difference for depth, computed by comparing outputs from compressed and original features. Their benchmark supplies paired feature and distortion data across three tasks and four codecs. Evaluated by PLCC and SROCC, the three candidate metrics have no stable winning behaviour: correlation signs and strengths flip across codecs and tasks, and for segmentation all metrics occasionally show near-zero or even opposite correlation. The paper's conclusion is that conventional signal-similarity measures do not capture the semantic degradation introduced by feature coding, so CFQA needs its own quality estimators.","pith_inferences":["The softmax rank label for classification can jump from 1 to 1000 with tiny probability changes, so a smoother surrogate such as confidence margin or cross-entropy might yield more stable correlation measurements.","The same dataset could support a learned CFQA regressor, and reporting its per-codec cross-generalisation would directly test whether any task-agnostic estimator exists.","Extending labels to video features or to features from other backbones would test whether the findings generalise beyond DINOv2.","If PLCC and SROCC values are unstable across the ten samples, averaging over more compressed versions per source would settle whether the reported correlations are reliable."],"forward_implications":["Researchers can train or tune CFQA metrics against the released labels without running downstream inference.","Codec training can use such a metric as a semantic-aware loss, replacing end-to-end task labels.","Edge-cloud systems could use the metric to decide when to re-encode a feature at a higher bitrate before transmission.","Baseline results identify cosine similarity as the default starting point for handcrafted codecs, while exposing learning-based codecs as the harder regime.","The benchmark's three tasks span coarse-to-fine semantics, so a metric that performs well on all three would be a strong candidate for general CFQA."],"supporting_citations":[{"why":"Supplies the Hyperprior architecture used for both learning-based codec variants.","marker":"[4]"},{"why":"Source of the 100 classification images sampled for the Cls task.","marker":"[12]"},{"why":"Source of the 100 validation images for the segmentation task.","marker":"[15]"},{"why":"Provides the training protocols and test conditions for the Hyperprior codecs.","marker":"[22]"},{"why":"Defines CKA, one of the three baseline metrics evaluated.","marker":"[28]"},{"why":"Source of the indoor images for the depth estimation task.","marker":"[32]"},{"why":"The backbone feature extractor whose intermediate features populate the entire dataset.","marker":"[34]"},{"why":"Cited as the basis for expecting cosine similarity to reflect semantic similarity in high-dimensional representations.","marker":"[35]"}],"fun_headline_variants":["Standard metrics fail to gauge compressed feature quality","New benchmark shows why feature compression needs its own metric","Compressed features break MSE, cosine, and CKA as quality judges","12k compressed features prove old metrics can't measure semantic loss"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's ground truth assumes that task-head performance drops—softmax rank, mIoU difference, and RMSE difference—are stable, precise measures of semantic distortion, and that ten compressed versions per source suffice for reliable correlation; no variance or significance statistics are reported to back either assumption.","fun_headline_variants_meta":{"raw":{"variants":["Standard metrics fail to gauge compressed feature quality","New benchmark shows why feature compression needs its own metric","Compressed features break MSE, cosine, and CKA as quality judges","12k compressed features prove old metrics can't measure semantic loss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000493,"raw_usage":{"total_tokens":2415,"prompt_tokens":933,"completion_tokens":1482,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":1415}},"tokens_in":549,"tokens_out":1482,"duration_ms":13094,"temperature":1.0,"reasoning_tokens":1415,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:34:09.820352+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the ground-truth labels using more than one task-head checkpoint or repeated inference, then re-run the PLCC and SROCC evaluation; if the rankings of the ten compressed versions change materially, or if expanding to, say, 50 versions per source shifts the correlations by more than about 0.1, the benchmark's labels are too noisy to support the reported conclusions.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the 100 validation images for the segmentation task."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the training protocols and test conditions for the Hyperprior codecs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the indoor images for the depth estimation task."}],"review_version":1}