{"id":"a58f3dbc-380a-474b-b66f-7e02d2506b2f","arxiv_id":"2412.07520","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A per-sample confidence score derived from the pNML min-max regret is applied to linear regression and neural networks, and improves OOD detection, adversarial robustness, and active learning.","lead":"This PhD thesis applies the predictive normalized maximum likelihood (pNML) principle to linear regression and neural networks, deriving a per-test-point regret that measures prediction confidence without assuming a data distribution. It reports improved out-of-distribution detection, adversarial robustness, and active learning efficiency based on this regret.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 5.11 is not the true pNML regret for a single-layer softmax NN: the recursive update (5.5) is an approximate linear-layer step, and the softmax 'inverse activation' in Eq. 5.7 is not well-defined, so the min-max guarantee does not follow.","rationale":"I agree with the reader that the approximate update is the weakest link, but I sharpen it: the update is not merely approximate, it is the wrong estimator for softmax, so the gap is not just numerical precision but a category error in the ERM definition. The linear-regression chapters (2–4) are unaffected and remain sound. The proposed brute-force test is decisive because it evaluates the exact genie and checks whether Eq. 5.11 equals the true normalization. If the test shows a mismatch, the central claim for NN confidence measures must be downgraded to a heuristic, which is consistent with the reader's CONDITIONAL verdict.","tokens_in":62605,"tokens_out":10006,"duration_ms":91856,"concrete_test":"Train a small single-layer softmax model (e.g., C=2, M=4, N=30) on random data. For a fixed test x, compute Γ_exact by, for each y' ∈ {0,1}, retraining the model from the ERM initialization on the augmented set (D_N ∪ (x,y')) using full-batch gradient descent until convergence, evaluating p_{θ̂(y')}(y'|x), and forming log Σ_{y'} p_{θ̂(y')}(y'|x). Compare to Eq. 5.11 with the same training set and x. If Γ_exact differs materially from Eq. 5.11 (or if the converged θ̂(y') differs from the update (5.5)), Theorem 5.2.2 is not the true pNML regret.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.2 claims an analytical pNML regret for a single-layer softmax neural network (Theorem 5.2.2, Eq. 5.11). The derivation hinges on the recursive update (5.5) from Zhuang et al. [130], which the paper itself concedes is not the exact retrained solution but an iterative step. More seriously, the update is an online least-squares update for a linear layer with an invertible activation; a softmax layer is not of that form. The 'inverse activation' z in Eq. 5.7 includes the unknown log-sum-exp term Σ_j e^{θ_j^T x}, so Eq. 5.3's closed-form θ̂ = X_N^+ f^{-1}(Y_N) does not minimize the stated MSE objective for softmax. Consequently, the genie probabilities used in Lemma 5.2.1 are not the maximum-likelihood genie for the hypothesis class, and Γ from Eq. 5.11 is not the min-max regret for a single-layer NN. The OOD experiments (Tables 5.1–5.4) therefore lack the claimed theoretical backing; the empirical improvements may come from the heuristics (embedding normalization, fine-tuning) rather than pNML optimality.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This PhD thesis studies the predictive normalized maximum likelihood (pNML) solution for the individual setting, where no distributional assumption links training and test data. It derives exact pNML regret formulas for ordinary linear regression and for regularized/luckiness-based variants, gives an upper bound for over-parameterized linear regression, and then proposes pNML-based confidence scores for neural network last layers, out-of-distribution detection, adversarial defense, and active learning. The central claim is that the per-instance regret, computed from the training set and the single test input, quantifies learnability and confidence, and that this measure improves OOD detection, adversarial robustness, and active learning. The linear-regression parts are largely self-contained and analytically clean; the neural-network part is where the pNML interpretation breaks down.","tokens_in":62861,"tokens_out":9859,"duration_ms":98285,"significance":"If the central claims held, the thesis would offer a distribution-free, per-instance uncertainty measure for modern ML models, with direct practical value for OOD detection and active learning. The linear-regression derivations in Chapters 2-4 are a genuine strength: the regret formula Γ = log(1 + xᵀ(X_NᵀX_N)⁻¹x) follows from Gaussian integration, the learnable-space eigenvector analysis is insightful, and the LpNML extension is a nontrivial and well-executed idea. The experimental breadth is also substantial, covering multiple architectures, datasets, and metrics. However, the neural-network confidence measure, which is the headline application, is currently not a min-max regret for the stated hypothesis class, so the theoretical backing for the OOD and active-learning claims is missing. The empirical results may still be useful, but they need to be reframed as heuristic and supported by ablations that isolate the source of the gains.","major_comments":[{"comment":"The claimed pNML regret for a single-layer softmax neural network is not the min-max regret for that hypothesis class. Eq. (5.3) sets θ̂_N = X_N⁺ f⁻¹(Y_N), which is only a closed-form least-squares solution when the activation f is linear and invertible; for a softmax layer, the inverse activation in Eq. (5.7) contains the unknown log-sum-exp term, and applying f⁻¹ to one-hot labels is not well-defined. The recursive update in Eq. (5.5), taken from Zhuang et al., is, as the text itself states one paragraph after Eq. (5.5), an iterative step rather than the exact ERM solution on the augmented dataset. Consequently, the genie probabilities in Lemma 5.2.1 and the sum in Eq. (5.11) do not correspond to the pNML genie for the stated model class, and the min-max guarantee does not follow. The OOD detection application in Tables 5.1-5.4 therefore lacks the claimed theoretical backing. Please either derive the exact pNML for a hypothesis class on which the update is exact, or present Eq. (5.11) explicitly as an approximation/heuristic and remove the optimality claims.","section":"Section 5.2, Theorem 5.2.2 / Eq. (5.11)"},{"comment":"The L2 normalization of all embeddings is introduced as an empirical observation, not derived from the pNML principle, and the regret in Eq. (5.11) is scale-dependent. Normalizing the test input changes xᵀg and hence Γ, so the reported AUROC gains in Tables 5.1-5.4 may be due to this normalization or to the fine-tuning in Section 5.3 rather than to the pNML score. Please add ablations: (i) pNML score with and without normalization, (ii) normalization applied to baseline scores alone, and (iii) the standalone pNML detector compared with the combined scores. Without such ablations, the empirical improvement over baselines is not attributable to the proposed regret.","section":"Section 5.2, Deep neural network adaptation"},{"comment":"DIAL is introduced as the solution of the min-max criterion in Eq. (7.5), but the algorithm replaces the posterior p(θ|z^{n-1}) by a uniform distribution over MC-dropout samples and then maximizes the product of two softmax probabilities over that discrete set. No argument is given that this maximization approximates the pNML regret in Eq. (7.5) or that the uniform-weight approximation is adequate. As a result, the active-learning contribution is an interesting heuristic rather than a derived pNML procedure; the min-max framing in Sections 7.2 and 7.5 overstates the theoretical status. Please state the approximation explicitly, provide supporting ablations or approximation bounds, or weaken the claims accordingly.","section":"Section 7.2 / Algorithm 2"},{"comment":"The Adversarial pNML scheme uses a refinement stage that performs a targeted adversarial attack, not maximum-likelihood training on the augmented dataset, so the normalization factor K in Eq. (6.8) is not the pNML normalization for any stated hypothesis class. The empirical results are of interest and the adversarial-subspace interpretation is suggestive, but the chapter should not claim the pNML min-max regret interpretation without a derivation. Please either provide a formal link between the refinement step and the pNML genie, or describe the method as a pNML-inspired heuristic and remove the optimality language from the abstract and introduction.","section":"Chapter 6, Eqs. (6.5)-(6.8)"}],"minor_comments":[{"comment":"There appears to be a sign error: with P_Nλ defined as in Eq. (2.18), the normalization factor in q_pNML should involve (1 − xᵀP_Nλx), not (1 + xᵀP_Nλx). As printed, the density does not integrate to 1.","section":"Eq. (2.20)"},{"comment":"The symbol f⁻¹(Y_N) is used before the inverse activation is defined in Eq. (5.7), and for the softmax case the definition is ambiguous for one-hot label matrices. Please clarify the domain of f⁻¹ or avoid this notation until the activation is fixed.","section":"Section 5.2, after Eq. (5.3)"},{"comment":"The phrase 'In section 5.2 we show...' appears twice in quick succession and the second occurrence should reference the relevant experiment subsection rather than the same section.","section":"Section 5.2, remarks after Eq. (5.12)"},{"comment":"The notation in the regret upper bound is dense; in particular, the third term inside the logarithm should be displayed with explicit parentheses so that the reader can see what is inside the cube root. A small derivation sketch in the main text would also improve readability.","section":"Section 3.4.1, Eq. (3.27)"},{"comment":"The text 'Moor-Penrose' should be 'Moore-Penrose', both here and where the same typo appears near Eq. (5.2).","section":"Section 5.3"}],"recommendation":"major_revision","confidential_remarks":"This manuscript is a PhD thesis compilation, and several chapters appear to be based on previously published papers (e.g., refs. [10, 11]). The editor may wish to verify that the thesis contains sufficient new material relative to those papers and that the arXiv version is an appropriate submission for this venue. The main technical concern is concentrated in the neural-network pNML derivation, which is a load-bearing part of the abstract and the OOD application; the linear-regression chapters are sound and could form the basis of a strong revision if the NN claims are appropriately scoped."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this is a PhD thesis, not a focused paper, and its weight is uneven. The linear-regression half is genuinely solid; the neural-network half makes a claim that outruns its derivation.\n\nWhat's actually new: the norm-constrained over-parameterized regret bound (Thm 3.4.3), the LpNML ridge shift (Thm 4.3.3), the adversarial refinement class in Ch. 6, and the DIAL active-learning criterion in Ch. 7. The OLS pNML regret formula itself is not new (Fogel-Feder and the author's own ISIT paper), but the derivation is clean and the learnable-space interpretation is well explained. The UCI double-descent experiments are a nice touch: the regret tracks the test log-loss curve, and the upper bound behaves the same way. The PMLB results for LpNML are also credible, with 48/50 sets improved and the distribution-shift benchmark beats Tripuraneni-Mackey on four of five sets. These chapters deserve a serious referee.\n\nThe soft spot is Chapter 5. The derivation of the \"analytical pNML regret for a single-layer NN\" (Eq. 5.11) hinges on the recursive update (5.5) from Zhuang et al. The paper itself says that update is an iterative step, not the exact retrained solution. Worse, the inverse softmax in Eq. 5.7 contains the unknown log-sum-exp term, so Eq. 5.3's closed form does not actually minimize the stated MSE objective for a softmax layer. That means Lemma 5.2.1 and Theorem 5.2.2 are not computing the true genie for the softmax hypothesis class, and Gamma in Eq. 5.11 is not the min-max regret. The OOD experiments then lack the advertised theoretical backing; the gains may come from embedding normalization and fine-tuning rather than from pNML optimality. This is a load-bearing flaw for the NN confidence claims, though not for the linear-regression chapters.\n\nThe other issues are smaller. Several chapters reproduce prior work (Ch. 2, and Ch. 5 largely from the Deep pNML preprint). The experiments have no code and no confidence intervals. The embedding L2 normalization is introduced as an empirical observation and then used in all OOD tests, which is a post hoc heuristic.\n\nVerdict: conditional. Send it to review, but the referee should push hard on Chapter 5. If the authors either fix the softmax derivation or explicitly reframe the score as an approximation without the min-max guarantee, the thesis is a useful contribution. As it stands, the NN claims are overstated.\n\nRecommendation: accept for peer review with the expectation of major revision on Chapter 5.","headline":"A PhD thesis with genuinely new linear-regression pNML theory, but the neural-network confidence claims rest on an approximate update that is not the true min-max solution; worth a careful referee if Chapter 5 is reframed.","tokens_in":63444,"tokens_out":1898,"would_cite":true,"duration_ms":18778,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62J05","68T07","94A15"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a per-input regret score—the pNML regret—quantifies when a model's prediction can be trusted, with closed-form expressions for linear regression and neural networks.","keywords":["predictive normalized maximum likelihood","individual setting","confidence estimation","out-of-distribution detection","linear regression","neural networks","active learning","adversarial robustness"],"falsifier":"Train a small single-layer softmax network on a low-dimensional dataset where exact retraining is feasible, enumerate every possible test label, retrain the last layer from scratch for each label, compute the exact pNML normalization factor, and compare it with formula (5.11); a large discrepancy on inputs with $x^\\top g$ near 1 would show that the approximate update does not deliver the claimed regret.","tokens_in":62322,"feed_emoji":"🎯","tokens_out":4790,"duration_ms":51262,"temperature":0.7,"pith_summary":"This thesis tries to establish a single distribution-free measure of prediction confidence that works for a specific test input, not just on average over a test distribution. That measure is the predictive normalized maximum likelihood (pNML) regret, which compares the learner to a 'genie' that knows the true test label but must still use the given hypothesis class. The paper derives closed forms for linear regression and for a single-layer neural network, and then uses the regret as a confidence score for out-of-distribution detection, adversarial defense, and active learning. A sympathetic reader would take the central claim to be that a model can say 'I do not know' about an individual example, with the mathematical backing of a min-max optimality criterion that makes no distributional assumptions.","feed_headline":"A regret score tells a model when it doesn't know","feed_subtitle":"Per-input pNML regret flags out-of-distribution samples, resists attacks, and guides active learning.","key_machinery":"The load-bearing object is the pNML learner and its regret. For a hypothesis class $p_\\theta(y|x)$ and a training set $D_N$, the pNML probability assignment is $q_{\\text{pNML}}(y|x) = p_{\\hat\\theta(D_N;x,y)}(y|x) / \\sum_{y'} p_{\\hat\\theta(D_N;x,y')}(y'|x)$, and its min-max regret is the logarithm of the normalization factor, $\\Gamma = \\log \\sum_{y'} p_{\\hat\\theta(D_N;x,y')}(y'|x)$. For linear regression the genie update is solved with recursive least squares, producing the closed form $\\Gamma = \\log(1 + x^\\top(X_N^\\top X_N)^{-1}x)$. For neural networks the paper uses a recursive update rule for the weights to approximate the genie, leading to the single-layer regret formula and to the quantity $x^\\top g$ that measures alignment with the training data subspace. This single regret quantity carries all three applications: thresholding it detects unexpected inputs, refining an adversarial example toward each hypothesized label yields a defense, and minimizing the worst-case regret over the test set selects which unlabeled sample to query.","core_discovery":"The central claim is that the pNML regret, computed from the training set and a single test input, is a per-instance learnability and confidence measure. For linear regression the regret is exact: the pNML predictive distribution is Gaussian with the same mean as the empirical risk minimizer, and its variance is inflated by a factor whose logarithm is Γ = $\\log(1 + x^\\top(X_N^\\top X_N)^{-1}x)$; low regret occurs when the test input lies mostly in the subspace spanned by the large eigenvectors of the training correlation matrix. For a single-layer softmax neural network the paper derives Γ = $\\log \\sum_{i=1}^C \\frac{p_i}{p_i + p_i^{x^\\top g}(1-p_i)}$, where $p_i$ is the network's probability for class $i$ and $x^\\top g$ measures how aligned the test input is with the training data subspace. Low regret is shown to occur when the test input aligns with large-eigenvalue directions or sits far from the decision boundary. Applied to the last layer of a pretrained deep network, this regret is then used as a confidence score for detecting out-of-distribution samples, as a defense mechanism against adversarial perturbations, and as an active-learning acquisition criterion.","pith_inferences":["Because the regret depends only on the training set and the single test input, it could serve as a plug-in calibration or misclassification-detection score beyond the OOD task the paper evaluates; the paper does not test that use directly.","The neural-network regret formula rests on an iterative update that the paper explicitly says is not the exact retrained solution, so a cheap exactness check on small networks—comparing formula (5.11) against full leave-one-out retraining—would show how large the gap is in practice.","The luckiness mechanism suggests that replacing the Gaussian prior with other priors, such as a Laplace prior for $\\ell^1$ regularization, would yield analogous shift-toward-zero predictions under distribution shift, extending the same framework beyond ridge regression."],"forward_implications":["A pretrained classifier can gain an out-of-distribution detector without retraining or extra data, simply by computing the pNML regret on the last layer's embeddings.","The linear-regression regret gives a per-point generalization measure that explains why over-parameterized models can still generalize when test inputs lie in the 'learnable space' of large training eigenvalues.","The luckiness-augmented pNML shifts ridge-regression predictions toward zero outside the training support, which the paper shows improves distribution-shift performance.","The active-learning criterion selects samples that minimize worst-case regret on the unlabeled test set, and in the presence of out-of-distribution pool samples it needs fewer labeled examples to reach a given accuracy."],"supporting_citations":[{"why":"Introduces universal batch learning with log-loss and defines the pNML as the min-max regret solution that the thesis extends.","marker":"[32]"},{"why":"Establishes the individual-setting framework and log-loss regret that motivates treating the test input as an arbitrary individual value.","marker":"[74]"},{"why":"Provides the normalized maximum likelihood principle and the notion of luckiness that the thesis adapts for the LpNML formulation.","marker":"[100]"},{"why":"Supplies the recursive least squares update used to derive the exact linear-regression genie and the closed-form regret.","marker":"[45]"},{"why":"Gives the recursive neural-network update rule on which the single-layer pNML regret formula is built.","marker":"[130]"},{"why":"Defines the single-point transductive prediction benchmark and method that the luckiness-augmented pNML is compared against for distribution shift.","marker":"[107]"},{"why":"Provides the adversarial training and PGD attack framework used as the base model and comparison for the adversarial defense chapter.","marker":"[73]"},{"why":"Supplies MC-dropout as the approximate posterior used in the deep individual active learning algorithm.","marker":"[34]"}],"fun_headline_variants":["Per-input pNML regret gives reliable confidence scores","Regret score flags OOD samples and boosts adversarial defense","Model says 'unsure' with pNML regret for each input","pNML gives exact uncertainty for linear regression, robust for NNs","Active learning with per-sample regret from pNML"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The neural-network regret formula depends on an iterative update rule that the paper itself says does not compute the exact solution for the updated dataset, so the derived regret is not literally the min-max regret for the full network hypothesis class.","fun_headline_variants_meta":{"raw":{"variants":["Per-input pNML regret gives reliable confidence scores","Regret score flags OOD samples and boosts adversarial defense","Model says 'unsure' with pNML regret for each input","pNML gives exact uncertainty for linear regression, robust for NNs","Active learning with per-sample regret from pNML"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00063,"raw_usage":{"total_tokens":2920,"prompt_tokens":966,"completion_tokens":1954,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":1869}},"tokens_in":582,"tokens_out":1954,"duration_ms":13540,"temperature":1.0,"reasoning_tokens":1869,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:46:09.277191+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a small single-layer softmax network on a low-dimensional dataset where exact retraining is feasible, enumerate every possible test label, retrain the last layer from scratch for each label, compute the exact pNML normalization factor, and compare it with formula (5.11); a large discrepancy on inputs with $x^\\top g$ near 1 would show that the approximate update does not deliver the claimed regret.","supporting_citations":[{"cited_title":"Universal prediction","cited_arxiv_id":null,"evidence_quote":"Establishes the individual-setting framework and log-loss regret that motivates treating the test input as an arbitrary individual value."},{"cited_title":"Universal sequential coding of single messages.Prob- lemy Peredachi Informatsii, 23(3):3–17, 1987","cited_arxiv_id":null,"evidence_quote":"Provides the normalized maximum likelihood principle and the notion of luckiness that the thesis adapts for the LpNML formulation."},{"cited_title":"שאיביואלדרפליישמןבית הספרלתאריםמתקדמיםע","cited_arxiv_id":null,"evidence_quote":"Gives the recursive neural-network update rule on which the single-layer pNML regret formula is built."},{"cited_title":"Single point transductive prediction","cited_arxiv_id":null,"evidence_quote":"Defines the single-point transductive prediction benchmark and method that the luckiness-augmented pNML is compared against for distribution shift."}],"review_version":1}