{"id":"97fae6b0-0e51-4946-8e04-b2b3836aba9f","arxiv_id":"2504.16952","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A last-layer committee of 100 randomly initialized linear heads trained on shared features yields uncertainty estimates for benthic imagery similar to Monte Carlo dropout at a fraction of the inference cost.","lead":"This study tests a cheap way to measure how confident a computer vision model is when classifying seafloor images: train 100 simple linear classifiers on top of one shared image encoder instead of running the full model many times. The method gives uncertainty rankings roughly as good as much costlier Bayesian and Monte Carlo dropout approaches while using far fewer parameters at prediction time, which could help marine scientists prioritize images for human review.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The epistemic-uncertainty claim rests on committee diversity that the paper's own metric places near zero; predictive diversity, not weight-space CV, is the untested load-bearing assumption.","rationale":"The reader's weakest assumption identifies the same load-bearing issue: random initialization must leave enough functional diversity for the committee variance to represent epistemic uncertainty. I agree. The paper's internal evidence, Section 4.1, measures only weight-space diversity and reports values near 0.003; this does not establish predictive diversity, and the diversity was measured on a different configuration (LLCM-10, MNIST) than the one used for the headline results (LLCM-100, benthic imagery). A direct predictive-diversity test would settle whether the committee's per-sample variance is meaningful or whether the method reduces to a calibrated softmax confidence. The efficiency claim is arithmetically supported and the accuracy tables are internally consistent, so the reader's CONDITIONAL verdict should be kept: the method may still be a useful efficient confidence tool, but the abstract should be tempered if the proposed test shows negligible predictive disagreement. The missing deep-ensemble baseline is a separate claims-accuracy issue, but the diversity question is more load-bearing because it concerns the method's stated epistemic-uncertainty contribution.","tokens_in":24301,"tokens_out":6483,"duration_ms":64855,"concrete_test":"Re-implement the LLCM-100 configuration in Table 2 for German Bank and Substrate; for every test image, compute each member's softmax vector and measure (a) the mean pairwise Jensen-Shannon divergence between members and (b) the standard deviation of the max-class probability across members. Compare these with the same quantities for MCD-100 and with a single-CNN softmax entropy ranking. If the median pairwise JS divergence is below 0.001 on the low-confidence test images, or if LLCM uncertainty rankings correlate with single-model softmax entropy at Spearman rho > 0.99, the committee adds no epistemic uncertainty beyond calibrated confidence. Report the same statistics for M=100 on the benthic datasets, since Fig. 1 only covers M=10 MNIST.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central contribution (Contribution 3, Section 1) is that random initialization of 100 last-layer heads yields epistemic model uncertainty. That requires the members' predictive distributions to disagree on inputs where the model is uncertain. The only quantitative evidence is weight-space diversity: Section 4.1 reports sigma_CVM in Table 3 and CV||SV||F values of 2.7e-3 to 3.6e-3 in Fig. 1b-d for an LLCM-10 on MNIST. A 0.3% relative spread of singular-value norms is not predictive diversity; near-identical weight magnitudes are compatible with either large or negligible differences in the functions p_m(y|x). The measurement was also made only for M=10 on MNIST, not for the deployed LLCM-100 on German Bank or Substrate. Because all heads share the feature extractor and receive summed gradients from Eq. 4, training drives them toward the same optimum, and Fig. 1 shows diversity decreasing during training. If the residual head differences are negligible, the variance in Eq. 5 collapses and the uncertainty scores reduce to the entropy of the shared softmax, which is not epistemic uncertainty. The accuracy and efficiency comparisons would survive, but the stated epistemic-uncertainty contribution would not be established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a last-layer committee machine (LLCM) in which a shared feature extractor feeds M independently initialized linear heads, trained by summed cross-entropy losses, and per-sample uncertainties are obtained in a single forward pass by averaging the head softmax outputs. The authors compare LLCM-100 with Bayesian model averaging (BMA-100), Monte Carlo dropout (MCD-100), and deterministic CNNs on MNIST and on two benthic datasets, German Bank 2010 and Substrate (depth 2), using accuracy, NLL, Brier score, ECE, reliability diagrams, and threshold-based accuracy/efficiency curves. They also report weight-space diversity analyses based on coefficients of variation and SVD singular values, and they claim that random initialization of the committee heads is sufficient for epistemic (model) uncertainty. The headline empirical result is that LLCM-100 accuracy is within roughly one point of MCD-100 on the two benthic datasets while requiring one forward pass instead of 100 stochastic forward passes.","tokens_in":24443,"tokens_out":8038,"duration_ms":78286,"significance":"If the accuracy and uncertainty-ranking results hold, the LLCM is a useful, low-complexity tool for prioritizing ambiguous benthic images for human review. The paper's comparisons are internally consistent, the data and pretrained models are public, calibration is performed on the validation set, and the uncertainty metric plots provide a transparent way to display selective accuracy and efficiency. The work also engages appropriately with recent last-layer Bayesian methods. However, the epistemic-uncertainty contribution is not established by the reported weight-space diversity evidence, the '95% reduction of network parameters' claim is not literally a parameter-count reduction, and the near-identical performance claim lacks seed-level variance. These issues do not invalidate the accuracy/efficiency comparison, but they require revision before the paper can be recommended.","major_comments":[{"comment":"The evidence for Contribution 3 (random initialization yields epistemic model uncertainty) is weight-space diversity only, and that evidence is not persuasive. Fig. 1b-d reports CV||SV||F between 2.7e-3 and 3.6e-3 for an LLCM-10 on MNIST, and Fig. 1 shows this metric decreasing during training; Table 3 reports sigma_CVM values ranging from about 9.4 to 8372 depending on hyperparameters, which reflects logit magnitude scaling rather than functional diversity. None of these metrics measures whether the M members' predictive distributions p(y|x; w^g_m) disagree on inputs where the model is uncertain. If residual member differences are negligible, the variance across members in Eq. (5) collapses and the per-sample uncertainty reduces to the entropy of the committee-mean softmax, which is not epistemic uncertainty. Please provide a direct predictive-diversity measurement (e.g., per-sample standard deviation of member softmax probabilities, average pairwise KL or Jensen-Shannon divergence, or fraction of members agreeing with the ensemble mean) computed for the deployed LLCM-100 models on German Bank 2010 and Substrate, and show that these quantities are non-negligible relative to the ensemble-mean entropy. Without this, the epistemic-uncertainty claim is unsupported.","section":"Section 4.1, Eq. (5)"},{"comment":"The abstract's '>95% reduction of network parameters' is not a parameter-count reduction. MCD evaluates the same set of parameters with different dropout masks; counting 100 sampled forward passes as 40M parameters (Section 4.2) is a nonstandard accounting. The ResNet-LLCM adds 100 linear classifiers to the shared ResNet-50 backbone, so its actual parameter count is larger than that of the corresponding single-head CNN, not 95% smaller. The legitimate efficiency claim is that LLCM obtains uncertainties with one backbone forward pass plus M head evaluations, rather than 100 stochastic forward passes. Please restate the efficiency claim in terms of inference-time compute or parameter-evaluations throughout the paper.","section":"Abstract; Section 4.2"},{"comment":"The claim that LLCM-100 performs 'near-identically' to MCD-100 rests on single runs with no reported variance across random seeds or training repetitions. For example, German Bank 2010 accuracy is 0.786 for LLCM versus 0.794 for MCD on 500 test images, a difference of about 4 images, and Substrate accuracy is 0.879 versus 0.885. Without seed-level standard deviations, confidence intervals, or a paired comparison, the equal-performance claim is not statistically supported. Please report means and standard deviations over at least several seeds for all ensemble methods, or explicitly present the comparison as descriptive rather than as evidence of equivalence.","section":"Tables 5 and 6"},{"comment":"The uncertainty metric plots do not include a baseline that is cheaper than LLCM, such as the max-softmax or softmax entropy of the deterministic CNN. Since the paper's practical contribution is a prioritized list of uncertain samples for human review, the reader needs to know how much of the selective-accuracy benefit comes from the 100-headed architecture rather than from simple confidence scores of a single trained network. Adding this baseline to Figs. 5 and 7 would directly address the concern that, if committee members are nearly identical, LLCM uncertainty scores reduce to the entropy of a shared softmax.","section":"Figures 5 and 7"}],"minor_comments":[{"comment":"The symbol M is used both for the number of committee members and for the number of calibration bins in Eq. (8); please rename the bin index or the bin count to avoid confusion.","section":"Eq. (8)"},{"comment":"The phrase 'ca.400K network parameters which is sampled 100x (i.e., 40M network parameters)' should be replaced by a statement about parameter-evaluations or multiply-accumulate operations, since multiplying a parameter count by the number of stochastic forward passes is not a standard parameter count.","section":"Section 4.2"},{"comment":"The BMA-100 results for the benthic datasets use a last-layer Bayesian network, not a fully Bayesian neural network; please clarify this in the text and in the table labels to avoid terminological confusion.","section":"Section 3.3; Tables 5 and 6"},{"comment":"The captions for Figs. 2 and 4 say 'using class weights/logit normalization/label smoothing' without indicating which panel corresponds to which combination; please specify this in each panel caption or in the figure legend.","section":"Figure captions"},{"comment":"There are several typos and nonstandard phrasings, including 'resulted with' for 'resulted in', 'int he next section' for 'in the next section', 'dateset' for 'dataset', 'preformed' for 'performed', and 'to identity' for 'to identify'; a careful proofreading pass is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The main risk is overclaiming epistemic uncertainty from tiny weight-space diversity; the accuracy and efficiency parts are salvageable. The authors should be asked to either provide predictive-diversity evidence on the deployed LLCM-100 models or soften Contribution 3. Adding a deterministic CNN softmax-entropy baseline to the uncertainty metric plots would strengthen the practical claim. The manuscript fits the journal's scope, and the benthic application is useful, but the methodological novelty relative to existing last-layer ensembles (TreeNets, deep sub-ensembles, VBLL) is incremental; the empirical comparison is the main contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this paper is a practical engineering contribution, not a theoretical one. The headline result holds up. Across MNIST, German Bank 2010, and Substrate depth 2, LLCM-100 accuracy is within one point of MCD-100, and the parameter-evaluation arithmetic is sound. That is a real, useful result for anyone needing per-sample uncertainty rankings on large benthic collections with limited compute.\n\nWhat is genuinely new is the specific configuration: 100 single linear heads on a shared trunk, trained jointly with summed cross-entropy, no diversity-promoting loss, random initialization only. The authors honestly review TreeNets, deep sub-ensembles, VBLL, and POVI. The efficiency framing is fair.\n\nWhere it gets soft: the paper claims the committee provides epistemic model uncertainty (Contribution 3). The evidence for that is weight-space diversity: CV of Frobenius norms of SVD singular values between 2.7e-3 and 3.6e-3 for LLCM-10 on MNIST. That is not predictive diversity. Two functions can have similar weight norms and still disagree, or have different norms and agree. The measurement was only on M=10, not the deployed LLCM-100 on benthic data. The authors never measure disagreement among committee members' predictive distributions, such as variance of softmax probabilities, pairwise KL, or vote disagreement. If the members converge to nearly the same function, Eq. 5 variance collapses and the uncertainty scores reduce to the entropy of the ensemble mean softmax, which is not epistemic uncertainty. So the central claim is unproven. The accuracy and efficiency claims survive; the epistemic interpretation is what needs work.\n\nOther soft spots are minor but real: the abstract promises a comparison to deep ensembles that never appears in the tables; there are no seed-level error bars anywhere; no code, seeds, or config files are released; the loss-scaling factor for M is mentioned as a hyperparameter but never specified. None of these are fatal, but they are the usual revision asks.\n\nWho is this for? Applied marine-imagery people who want a cheap uncertainty ranking for human-in-the-loop review, and people working on last-layer ensembles. It is not a methods breakthrough, and the epistemic claim should be softened or properly tested. A serious referee should engage with it; with code, repeated runs, a real deep-ensemble baseline, and either predictive-diversity measurements or a reworded claim, it would be a solid application paper.","headline":"A useful applied comparison showing a 100-head last-layer committee matches MCD accuracy on benthic images at a fraction of the inference cost, but the epistemic-uncertainty claim is not supported by the diversity evidence in the paper.","tokens_in":25071,"tokens_out":2119,"would_cite":false,"duration_ms":20842,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A last-layer committee machine estimates per-image uncertainty for benthic imagery with over 95% fewer parameters, matching the accuracy of Monte Carlo dropout and Bayesian approaches.","keywords":["benthic habitat mapping","last-layer ensemble","committee machine","uncertainty estimation","Monte Carlo dropout","Bayesian neural network","human-in-the-loop","singular value decomposition"],"falsifier":"Compute the correlation between LLCM-100 per-sample standard deviation and prediction error on held-out benthic images, including artificially mislabeled and geographically out-of-distribution images; if the correlation is near zero or the spread collapses to the entropy of the mean softmax, the epistemic-uncertainty claim would not be supported.","tokens_in":24012,"feed_emoji":"🌊","tokens_out":10156,"duration_ms":84519,"temperature":0.7,"pith_summary":"The paper claims that attaching 100 thin linear classifiers to a shared pre-trained feature extractor can give per-image uncertainty estimates for seafloor imagery at a fraction of the cost of standard Bayesian approaches. Each committee member is trained with its own cross-entropy loss, then the member softmax outputs are averaged in one forward pass. On the German Bank 2010 and Substrate depth 2 benthic datasets, the reported accuracies are within about one percentage point of Monte Carlo dropout (0.786 vs 0.794 and 0.879 vs 0.885), while the number of network parameters evaluated for uncertainty drops by more than 95%. If correct, this gives marine scientists a practical way to generate prioritized lists of ambiguous, mislabeled, or out-of-distribution images for human review.","feed_headline":"100 small heads deliver per-image uncertainty at 95% fewer parameters","feed_subtitle":"One pass through 100 linear heads on a shared network rivals Monte Carlo dropout on seafloor images.","key_machinery":"The load-bearing object is the last-layer committee machine: $M$ linear layers $g_m$ appended to a shared feature extractor $f(x; w_f)$, each with randomly initialized weights and its own cross-entropy loss $\\ell_m(y, g_m(f; w_{g_m}))$, with gradients backpropagated independently for each member. At inference, the per-sample uncertainty estimate is the average of the $M$ softmax outputs, giving an approximation to the posterior predictive distribution in one forward pass. Diversity is quantified by the coefficient of variation of the Frobenius norms of singular values across members, which remains non-zero but small (about $2.7\\times10^{-3}$ to $3.6\\times10^{-3}$) after training, and this residual diversity is what the authors lean on for the epistemic-uncertainty claim.","core_discovery":"The paper's central claim is that a last-layer committee machine can replace computationally expensive uncertainty estimators such as Bayesian neural networks and Monte Carlo dropout without meaningful loss in classification accuracy or calibration. The method keeps one shared feature extractor and adds $M=100$ single linear layers as committee members; each member is randomly initialized and trained with its own cross-entropy objective, and inference averages the $M$ softmax distributions. The authors report near-identical accuracy to Monte Carlo dropout on both benthic datasets, with a single forward pass and a reduction in parameters evaluated by about 20 times on the benchmark network and more than 95% overall. They also provide diversity analyses based on singular value decompositions of member weights, arguing that random initialization alone leaves the members sufficiently distinct to support epistemic uncertainty.","pith_inferences":["Inference: Because the committee members end up nearly identical (singular-value coefficient of variation around 0.003), the per-sample uncertainty could in practice be only slightly more informative than the softmax entropy of the averaged output; this should be checked explicitly before interpreting the spread as epistemic uncertainty.","Inference: The efficiency-threshold plots the paper introduces could be used as a model-agnostic protocol to audit any uncertainty estimator on any dataset, not just benthic imagery.","Inference: A sharper test of the method would be to measure whether LLCM variance increases on out-of-distribution images from geographic regions absent from training; the paper's experiments focus on in-distribution held-out partitions.","Inference: The same single-layer committee could be plugged into an active-learning loop by re-annotating the lowest-confidence samples and adding them to the training pool, which the paper mentions only as a long-term goal."],"forward_implications":["Single-pass uncertainty: an LLCM-100 produces per-sample standard deviations over committee members in one forward pass, so uncertainty-aware review can scale to the millions of images in large benthic compilations.","Parameter reduction: with a shared feature extractor and 100 single-layer heads, the number of network parameters evaluated for uncertainty drops by more than 95% relative to sampling a full network 100 times for Monte Carlo dropout.","Accuracy parity: LLCM accuracy is within about one point of Monte Carlo dropout on German Bank 2010 (0.786 vs 0.794) and Substrate depth 2 (0.879 vs 0.885), with calibration restored by temperature scaling.","Simplicity: random initialization of the committee members is enough to create diversity; no specialized loss, logit averaging, or diversity-promoting regularizer is required."],"supporting_citations":[{"why":"Supplies the Monte Carlo dropout baseline the LLCM is compared against.","marker":"(Gal and Ghahramani, 2015)"},{"why":"Defines deep ensembles, the expensive strategy the LLCM is designed to approximate.","marker":"(Lakshminarayanan et al., 2017)"},{"why":"Introduces last-head ensembles and evidence that random initialization can provide diversity, the basis for the LLCM design.","marker":"(Lee et al., 2015)"},{"why":"Provides the variational Bayesian last-layer baseline used on the benthic datasets.","marker":"(Harrison et al., 2024)"},{"why":"Provides the global benthic imagery compilation that supplies the German Bank and Substrate sub-datasets.","marker":"(Lowe et al., 2025)"},{"why":"Supplies the dataset partition labels and the two challenging sub-datasets used for training and testing.","marker":"(Misiuk et al., 2024)"},{"why":"Provides the pre-trained ResNet-50 feature extractor the LLCM heads are attached to.","marker":"(Xu et al., 2024)"},{"why":"Supplies temperature scaling and calibration metrics (NLL, Brier score, ECE) used for evaluation.","marker":"(Guo et al., 2017)"}],"fun_headline_variants":["100 linear heads match MC dropout with 95% fewer parameters","Seafloor uncertainty at 95% less cost via committee heads","One pass, 100 heads: cheap uncertainty for benthic images","Committee of 100 heads rivals Bayesian nets on seafloor images","Last-layer committees cut uncertainty cost 95% on benthic data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim stands on the assumption that the 100 committee members, after training on the same features, remain different enough that the spread of their predictions is a meaningful measure of model uncertainty rather than numerical noise.","fun_headline_variants_meta":{"raw":{"variants":["100 linear heads match MC dropout with 95% fewer parameters","Seafloor uncertainty at 95% less cost via committee heads","One pass, 100 heads: cheap uncertainty for benthic images","Committee of 100 heads rivals Bayesian nets on seafloor images","Last-layer committees cut uncertainty cost 95% on benthic data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000215,"raw_usage":{"total_tokens":1442,"prompt_tokens":970,"completion_tokens":472,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":384}},"tokens_in":586,"tokens_out":472,"duration_ms":4586,"temperature":1.0,"reasoning_tokens":384,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:10:20.814167+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the correlation between LLCM-100 per-sample standard deviation and prediction error on held-out benthic images, including artificially mislabeled and geographically out-of-distribution images; if the correlation is near zero or the spread collapses to the entropy of the mean softmax, the epistemic-uncertainty claim would not be supported.","supporting_citations":[],"review_version":1}