{"id":"b9f2f584-d787-4aba-80fe-c08debb493f3","arxiv_id":"2412.20918","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A per-class Gaussian process over DNN features, scored by KL divergence between predictive distributions, detects out-of-distribution images using only in-distribution data to set the threshold.","lead":"This paper builds a Gaussian process model on top of a neural network's internal features and output scores, and uses prediction uncertainty to flag inputs that look unfamiliar, without needing examples of unfamiliar data during training. The method performs well on MNIST-style benchmarks, but its advantage over earlier detectors is inconsistent on larger real-world image datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's proof algebra is reversed: the derived sufficient condition is B < -2a_k, not B < 2a_k, so the stated distance threshold does not follow and the theoretical guarantee is vacuous.","rationale":"The reader's weakest_assumption concerns the empirical feature-space separation, but the most decisive and concrete problem is internal to the proof. Theorem 1's derivation contains a sign error: with a_k positive as asserted, the condition B < -2a_k is impossible, so the theorem is vacuous. This removes the claimed theoretical support for the method independently of any experiment. I also note the paper's Table 4 shows the multi-class GP below ODIN on all six real-world OOD datasets, which undermines the abstract's unqualified 'outperforms' claim; however, the theorem error is the more objective and directly checkable flaw. The reader's rejection remains appropriate, so the verdict is unchanged.","tokens_in":14861,"tokens_out":17402,"duration_ms":171491,"concrete_test":"Independently re-derive the algebraic step in the Appendix between Eq. (10) and the display 'phi(d_min)^2 < 2 a_k lambda_min / m_GP^k' using the paper's definition of a_k. If the correct condition is B < -2a_k, equivalently phi^2 < -2a_k lambda_min / m_GP^k, then Theorem 1 is false. A numeric check with tau^2=1, m_GP=1, lambda_min=1, gamma=0, and average log sigma^2 = -1 will show that the theorem's condition can trigger while the proof's own lower bound is still below the threshold.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Theorem 1 claims x' is flagged OOD whenever d_min,k(x')^2 > -0.5 log(2 a_k lambda_min(Phi_k)/m_GP^k), with a_k = gamma_k - (1/mValid_k) sum_x log[1 - Phi_k(x) Phi_k^{-1} Phi_k(x)] and a_k > 0. In the Appendix proof, the KL score is bounded below by -log tau_k^2 + (1 - B)/2, where B = m_GP^k phi(d_min)^2 / lambda_min(Phi_k). For g(x')=1, the proof requires -log tau_k^2 + (1 - B)/2 > gamma_k + 1/2 - (1/mValid_k) sum_x log sigma_k^2(x). Rearranging this inequality gives B < -2a_k, not B < 2a_k. Since a_k is asserted positive, the actual condition B < -2a_k cannot hold, so the stated sufficient condition is not implied by the derivation. This theorem is the paper's theoretical justification for separating InD and OOD by the Mahalanobis-like distance in Eq. (7), so the claimed no-OOD guarantee is not established.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an OOD detection method for DNN classifiers that fits a separate Gaussian process to each class's unconstrained softmax scores using only in-distribution (InD) feature representations from an intermediate layer. A test point is scored by the average KL divergence between the GP predictive distribution at that point and the predictive distributions at InD validation points of the DNN's predicted class, and it is flagged as OOD when the score exceeds an InD-derived quantile threshold. The authors provide Theorem 1 as a sufficient-distance theoretical guarantee and report experiments on MNIST and ImageNet-based benchmarks, claiming state-of-the-art performance without using OOD data during training.","tokens_in":15078,"tokens_out":9000,"duration_ms":83122,"significance":"If the proposed method worked as advertised, it would offer a useful plug-in OOD detector that avoids using OOD data for training or threshold selection, and the GP uncertainty-quantification perspective is a natural and potentially appealing idea. The algorithm is clearly specified in Algorithm 1, the implementation relies on the standard laGP package, and the experiments use widely recognized benchmarks. However, the paper's central theoretical result is invalid as stated, the claim that no OOD data are used in validation is contradicted by the paper's own UMAP-based layer selection, and the real-world comparison in Table 4 does not support the abstract's claim of outperforming state-of-the-art methods. These are load-bearing issues rather than presentation concerns.","major_comments":[{"comment":"The proof of Theorem 1 contains a sign error that invalidates the stated sufficient condition. After bounding the left-hand side of the detection inequality by -log(tau_k^2) + (1 - B)/2 with B = m_GP^k * phi(d_min,k(x'))^2 / lambda_min(Phi_k), and writing the right-hand side as a_k + 1/2 - log(tau_k^2), the proof requires (1 - B)/2 > a_k + 1/2, which rearranges to B < -2a_k, not B < 2a_k. Since the paper correctly notes that a_k > 0, the condition B < -2a_k cannot hold for the nonnegative quantity B, so the claimed sufficient distance condition d_min,k(x')^2 > -1/2 log(2 a_k lambda_min(Phi_k) / m_GP^k) is not implied by the derivation. This error is load-bearing because Theorem 1 is the paper's only theoretical justification for separating InD and OOD inputs by the Mahalanobis-like distance in Eq. (7).","section":"Appendix A, Theorem 1 proof"},{"comment":"The abstract and Section 4.2 claim that the proposed method outperforms state-of-the-art OOD detection methods, but Table 4 shows that Multi-class GP underperforms both ODIN and Mahalanobis on every real-world dataset in both TNR and AUROC. For example, on DTD the reported values are 0.4168/0.6815 for Multi-class GP versus 0.4956/0.8017 for ODIN and 0.6100/0.9003 for Mahalanobis, and on SVHN the values are 0.1781/0.5631 versus 0.3095/0.6889 for ODIN and 0.6484/0.9383 for Mahalanobis. The text's characterization of the result as 'comparable TNR and AUROC with ODIN' is therefore inaccurate, and the central empirical claim of outperforming the state of the art is not supported by the paper's own data.","section":"Table 4 and Section 4.2"},{"comment":"The paper states that 'All setups above ensure that OOD data was not involved in any training or validation process,' but earlier in Section 4.1 the dimension of the intermediate layer was 'validated' using UMAP visualization of InD and OOD data. This is a hyper-parameter choice informed by OOD data, which directly contradicts the central claim that the method requires no OOD samples for training or hyper-parameter tuning. Section 3.3 explicitly permits t-SNE or UMAP visualization only when OOD data are available, so the experimental setup in Section 4.1 is not a pure no-OOD validation procedure.","section":"Section 4.1 and Section 3.3"},{"comment":"The detection mechanism relies on the assumption that OOD inputs are far from their predicted class's InD points in the feature space measured by the GP kernel metric, but the paper states this separability as a goal rather than proving or empirically validating it for near-OOD inputs. Because Theorem 1 is vacuous due to the sign error, the paper does not currently provide a theoretical rationale for why the GP-based score will separate InD from OOD when the OOD distribution is close to the InD distribution.","section":"Section 3.3, Eq. (7)"}],"minor_comments":[{"comment":"The text says that increasing 1-alpha from 0.9 to 0.95 improves TPR 'from 0.8795 to 0.9420', but the TPR values reported in Table 1 are 0.9036 and 0.9546, respectively; these numbers should be reconciled.","section":"Section 4.1, Table 1"},{"comment":"The text says the intermediate-layer dimension was validated using UMAP, but the caption of Figure 3 says the visualization was produced with t-SNE; one of these statements is incorrect.","section":"Figure 3 and Section 4.1"},{"comment":"There is a typo in the discussion of ODIN: 'perturbations to the input data using a gradient-based method' appears in a sentence that refers to the 'DMM' instead of the DNN.","section":"Section 1"},{"comment":"The statement that FashionMNIST performance 'remains high' is relative: the AUROC values are 0.8661 and 0.8551, which are markedly lower than the values above 0.95 obtained on the other OOD datasets, so the wording could be more precise.","section":"Section 4.1, Table 1"}],"recommendation":"reject","confidential_remarks":"The manuscript's central theoretical guarantee is invalid due to the reversed inequality in the proof of Theorem 1, and the headline empirical claim is contradicted by Table 4. In addition, the use of UMAP visualization with OOD data to validate the intermediate-layer dimension undercuts the claimed no-OOD setting. These are not local presentation problems: a revision would need a different theoretical argument, an OOD-free model-selection protocol, and substantially tempered claims about state-of-the-art performance. I therefore recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The method—per-class GP on an intermediate DNN layer, KL-divergence score, InD-only quantile threshold—is a legitimate and reasonably clean construction. The paper is well organized, and on the MNIST-vs-everything setup it shows genuinely strong separation for Cifar10, SVHN, and ImageNet. That part of the empirical story is credible.\n\nThe problems are concentrated in the theory and the overclaiming. I checked the Appendix proof of Theorem 1. The stress-test note is right: the algebra reverses the inequality. The proof's final equivalence states phi(d_min)^2 < 2 a_k lambda_min / m_GP^k, but following their own inequalities gives m_GP^k phi(d_min)^2 / lambda_min < -2 a_k. Since a_k is positive by their own claim, no nonnegative B can satisfy that. So the sufficient condition is vacuous and the theorem, as stated, does not follow. That is a load-bearing flaw because the paper advertises statistical theory as a contribution and uses the theorem to motivate the distance-based score.\n\nThe empirical overreach is also real. Table 4, the large-scale ImageNet setup, shows the proposed method below ODIN on every OOD set, and often well below Mahalanobis. The paper acknowledges this but then falls back on Table 2, where ODIN and Mahalanobis are handicapped by averaging over untuned hyperparameters. That's not a fair comparison for the 'outperforms state-of-the-art' claim. And Section 4.1 says the intermediate layer dimension p=32 was 'validated' via UMAP on InD and OOD data, which leaks OOD information into model selection—minor, but it weakens the 'no OOD data' selling point. No code is provided.\n\nSo: the core idea is worth a serious look, but the theoretical guarantee is currently invalid and the performance claims need to be scaled back to 'competitive no-OOD baseline at best.' I'd send this to review only if the authors can fix the proof and re-frame the claims. As is, I would not cite it in my own work, and I wouldn't trust the abstract. But it's worth a reading-group discussion to see the sign error and debate the baseline comparison.","headline":"The GP-OOD method is sensible and the MNIST experiments are decent, but the proof of Theorem 1 has a sign error that voids the theoretical guarantee, and the paper's own Table 4 undercuts its headline claim.","tokens_in":695,"tokens_out":2317,"would_cite":false,"duration_ms":54733,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62M45","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"A Gaussian process fitted only to in-distribution features can detect out-of-distribution inputs to deep classifiers, outperforming OOD detectors that require OOD data for tuning.","keywords":["out-of-distribution detection","Gaussian processes","uncertainty quantification","Kullback-Leibler divergence","deep neural networks","in-distribution-only calibration","multi-class GP"],"falsifier":"Take a fixed pre-trained classifier and a fixed intermediate layer, then evaluate the detector on a family of OOD sets ordered by similarity to the InD data, for example corrupted versions of the InD test set and a semantically similar but distinct dataset. Record the per-class threshold $\\gamma_k$ and the GP score of every point: the paper's claim predicts that all points with $d_{\\min,k}(x')^2$ above the Theorem 1 bound are flagged, so any such point that fails to be flagged would refute the sufficiency theorem, and an OOD set whose AUROC falls to chance while the InD true positive rate stays at 95% would refute the InD-only decision boundary.","tokens_in":14618,"feed_emoji":"🔍","tokens_out":13626,"duration_ms":105459,"temperature":0.7,"pith_summary":"Deep classifiers tend to make confident predictions even on inputs from outside their training distribution, and common safeguards need actual out-of-distribution (OOD) examples to tune thresholds. This paper proposes an OOD detector that never sees OOD data: it fits one Gaussian process per class to the intermediate-layer features of a pre-trained network, treating the network's unconstrained softmax scores as the GP responses, and then flags any test point whose average Kullback-Leibler divergence from the class-conditional predictive distributions exceeds a quantile threshold computed from in-distribution validation data. The paper argues that because GPs revert to their prior mean and inflate variance far from training data, OOD points generate larger scores than InD points, and it proves a sufficient distance condition under which detection is guaranteed. On MNIST-trained and ImageNet-trained classifiers, the method reports higher true-negative rates than several score-based detectors under the constraint that no OOD data are used for tuning.","feed_headline":"No OOD data needed: GP detector flags out-of-distribution inputs","feed_subtitle":"Fitting one GP per class to DNN features turns predictive uncertainty into a decision boundary, no OOD examples needed.","key_machinery":"The carrying object is the multi-class Gaussian process predictive distribution. For class $k$, the GP maps the DNN's intermediate features to the unconstrained softmax score $f_k$ through a squared exponential kernel $\\Phi_k(\\xi(x),\\xi(x')) = \\exp\\left(-\\sum_{j=1}^p (\\xi_j(x)-\\xi_j(x'))^2/\\theta_{k,j}\\right)$. The mechanism does the work: away from the class training data the predictive mean decays toward the zero prior and the predictive variance grows, so the average KL divergence between a test point and the class's validation points becomes large exactly when the input is unlike that class. The threshold $\\gamma_k$ is the $1-\\alpha$ quantile of those InD validation scores, so no OOD sample enters the calibration. Theorem 1 converts this intuition into a sufficient condition on the scaled distance $\\|\\Theta_k^{-1}(\\xi(x)-\\xi(x'))\\|_2$, using the minimum eigenvalue of the kernel matrix to control how far is far enough.","core_discovery":"The central claim is that a multi-class Gaussian process emulator, trained and thresholded exclusively on in-distribution data, can draw a decision boundary that separates InD and OOD inputs and that transfers to unseen OOD test sets. For each class $k$, a zero-mean GP is trained on the DNN's intermediate features $\\xi(x)$ and the unconstrained softmax score $f_k(x)$; the detection score $s_k(x')$ is the average KL divergence between the predictive distribution at $x'$ and at each InD validation point of the predicted class. Theorem 1 states that $x'$ is flagged as OOD whenever its scaled minimum feature distance satisfies $d_{\\min,k}(x')^2 > -\\frac{1}{2}\\log\\left(\\frac{2 a_k \\lambda_{\\min}(\\Phi_k)}{m_k^{GP}}\\right)$, where $\\lambda_{\\min}(\\Phi_k)$ is the smallest eigenvalue of the class kernel matrix and $a_k$ is defined from the threshold and the validation predictive variances. In the paper's experiments, this InD-only rule matches or beats ODIN, Mahalanobis distance, Energy, and VOS on MNIST-versus-other and ImageNet-versus-other benchmarks, with the largest advantage on FashionMNIST, the near-OOD set most similar to MNIST.","pith_inferences":["The paper states the feature separation between InD and OOD as a goal rather than proving it for the chosen layer; a direct test would be to sweep the layer index of a fixed network and measure the GP score margin on a near-OOD set, predicting that the margin tracks the scaled distance of Theorem 1.","Theorem 1 is only sufficient, so the practical decision boundary may be looser than the theorem's bound; computing, for detected and missed OOD points, the empirical distance at which flagging begins would quantify how conservative the bound is.","Because $\\gamma_k$ is estimated from a validation sample, the false-positive rate is only as reliable as that sample; small validation sets will make the quantile noisy, so a per-class sample-size correction is a natural follow-up.","The same decomposition could be applied to regression or time-series monitoring by replacing the unconstrained softmax response with a scalar target and using a single GP instead of one per class; the paper only gestures at this in its closing discussion."],"forward_implications":["A pre-trained classifier can be wrapped with this detector as a plug-in module; no re-training and no OOD collection is needed, so deployment becomes possible where OOD data are scarce or not yet defined.","The user can set the admissible false negative rate in advance: the threshold is fixed from the $1-\\alpha$ quantile of InD validation scores, and the experiments report TNR and AUROC at 90% and 95% true positive rates.","For any test point whose scaled distance to the class-$k$ training set exceeds the Theorem 1 bound, being flagged as OOD is guaranteed rather than merely likely.","Near-OOD data are the hardest case, yet the method keeps a reported AUROC above 0.85 on FashionMNIST versus MNIST, where the comparators fall below 0.82 in the same no-OOD tuning setup.","On large-scale ImageNet-based problems, the reported TNR drops to roughly 30-70% at a 90% TPR, showing the boundary degrades when each InD class contains wide internal variation."],"supporting_citations":[{"why":"Supplies the Gaussian process predictive mean and variance equations used to build each class-conditional predictive distribution.","marker":"Rasmussen and Williams (2006)"},{"why":"Provides the profile log-likelihood estimation of kernel hyperparameters and the interpolation behavior that motivates the GP/validation split.","marker":"Gramacy (2020)"},{"why":"Supports the claim that GPs revert to the prior mean and inflate variance away from training data, the mechanism behind the score separation.","marker":"Santner et al. (2018)"},{"why":"Supplies the proof technique for Theorem 1, bounding predictive variance using the minimum eigenvalue of the kernel matrix.","marker":"Sung et al. (2018)"},{"why":"ODIN, the temperature-scaled softmax baseline whose hyperparameters need OOD tuning and which the proposed method is compared against.","marker":"Liang et al. (2018)"},{"why":"Mahalanobis-distance baseline, a score-based detector that the paper compares against.","marker":"Lee et al. (2018)"},{"why":"Energy-based OOD detection baseline, evaluated with nominal hyperparameter values in the comparison.","marker":"Liu et al. (2020)"},{"why":"Virtual Outlier Synthesis baseline, a generative approach that synthesizes virtual OOD data during training.","marker":"Du et al. (2022)"},{"why":"DenseNet architecture used as the pre-trained classifier in the conventional image classification experiments.","marker":"Huang et al. (2017)"},{"why":"ResNet architecture used as the pre-trained classifier in the large-scale real-world image experiments.","marker":"He et al. (2016)"}],"fun_headline_variants":["GP uncertainty flags OOD without any OOD data","In-distribution-only GP detector rivals OOD methods","No OOD needed: multi-class GP separates InD and OOD","KL divergence from GP emulator detects OOD inputs","GP on softmax scores makes OOD boundary from InD only"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that any out-of-distribution input, once mapped to the chosen intermediate layer, will land far enough from the in-distribution points of the class it is assigned to that the Gaussian process becomes noticeably more uncertain; near-OOD inputs that embed close to in-distribution features would get confident predictions and slip past the threshold.","fun_headline_variants_meta":{"raw":{"variants":["GP uncertainty flags OOD without any OOD data","In-distribution-only GP detector rivals OOD methods","No OOD needed: multi-class GP separates InD and OOD","KL divergence from GP emulator detects OOD inputs","GP on softmax scores makes OOD boundary from InD only"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000268,"raw_usage":{"total_tokens":1669,"prompt_tokens":1050,"completion_tokens":619,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":666,"completion_tokens_details":{"reasoning_tokens":536}},"tokens_in":666,"tokens_out":619,"duration_ms":5987,"temperature":1.0,"reasoning_tokens":536,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:07:35.553991+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed pre-trained classifier and a fixed intermediate layer, then evaluate the detector on a family of OOD sets ordered by similarity to the InD data, for example corrupted versions of the InD test set and a semantically similar but distinct dataset. Record the per-class threshold $\\gamma_k$ and the GP score of every point: the paper's claim predicts that all points with $d_{\\min,k}(x')^2$ above the Theorem 1 bound are flagged, so any such point that fails to be flagged would refute the sufficiency theorem, and an OOD set whose AUROC falls to chance while the InD true positive rate stays at 95% would refute the InD-only decision boundary.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Gaussian process predictive mean and variance equations used to build each class-conditional predictive distribution."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the profile log-likelihood estimation of kernel hyperparameters and the interpolation behavior that motivates the GP/validation split."},{"cited_title":"J., Williams, B","cited_arxiv_id":null,"evidence_quote":"Supports the claim that GPs revert to the prior mean and inflate variance away from training data, the mechanism behind the score separation."},{"cited_title":"B., and Haaland, B","cited_arxiv_id":null,"evidence_quote":"Supplies the proof technique for Theorem 1, bounding predictive variance using the minimum eigenvalue of the kernel matrix."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ODIN, the temperature-scaled softmax baseline whose hyperparameters need OOD tuning and which the proposed method is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Mahalanobis-distance baseline, a score-based detector that the paper compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Energy-based OOD detection baseline, evaluated with nominal hyperparameter values in the comparison."}],"review_version":1}