{"id":"555d71f4-f98c-4e18-a2a9-7d3803f3c2dc","arxiv_id":"2412.17411","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Random-noise pretraining calibrates small networks' confidence to chance level before data training, reducing expected calibration error and improving OOD detection on CIFAR-10/SVHN.","lead":"This paper reports that pretraining a small neural network on pure random noise and random labels, before training on real data, makes its confidence scores match its accuracy more closely and improves detection of out-of-distribution inputs. The authors argue that the usual random initialization contributes to overconfidence and offer their pretraining as a simple fix needing no extra post-processing.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing trivial controls: the claimed pre-calibration effect may be due to lower initial confidence, not to the specific random-noise pretraining procedure.","rationale":"The reader's verdict is CONDITIONAL, and my concern supports keeping it conditional: the paper's central claim is plausible but under-supported because it lacks the standard calibration baselines and a proper control for the initialization effect. The reader's weakest_assumption focused on whether pretraining converges to a uniform-output solution without memorizing re-sampled random labels; that assumption is actually quite secure because the expected gradient of cross-entropy with uniformly random labels drives the output distribution to uniform, and the paper reports the corresponding loss decrease and chance-level accuracy. The more load-bearing gap is the absence of controls that would establish that the specific pretraining procedure, rather than simply lowering initial confidence, is responsible for the reported calibration and OOD improvements. This is consistent with the reader's rationale, which already notes missing baseline comparisons, but it is not the same as the reader's stated weakest_assumption. For a paper whose headline contribution is a new initialization/pretraining method, showing that trivial confidence-reducing alternatives do not reproduce the effect is essential. Until then, CONDITIONAL acceptance is appropriate.","tokens_in":14122,"tokens_out":7308,"duration_ms":79442,"concrete_test":"Run the same CIFAR-10 experiments with three additional control groups: (1) standard random initialization followed by post-hoc temperature scaling; (2) standard random initialization plus label smoothing during real-data training; (3) an initialization with final-layer weights and biases scaled so that the initial softmax confidence is near-uniform (equivalent to a low-temperature initial state) without any noise pretraining. Train all groups with identical optimizer settings and compare ECE on the CIFAR-10 test set and OOD AUROC on SVHN. If any control group matches or beats the random-noise-pretrained network on both metrics, the central claim of a specific pretraining effect is refuted; if the pretrained network is significantly better, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that pretraining with random noise and random labels pre-calibrates network uncertainty, and that random initialization is an underlying cause of miscalibration. What the experiments show is that pretraining drives the network toward a constant uniform-output solution, i.e., the lowest-confidence initial state, and that this improves ECE and OOD AUROC for small MLPs. However, the paper never compares against simpler ways of reducing initial confidence: smaller final-layer weight scale, a low softmax temperature, or label smoothing during real-data training. Without these controls, the observed benefit could be caused by starting from low confidence rather than by the specific noise/label pretraining. In particular, a post-hoc temperature scaling on the standard, non-pretrained network is the standard calibration baseline (Guo et al., 2017) and is not included. The authors explicitly acknowledge (Discussion) that no real-world or large-scale validation was performed, so the broad claim that random-noise pretraining is a general solution is not supported by the evidence. The causal statement that random initialization is the cause of miscalibration is also weakened because random initialization simply produces peaked softmax outputs; the paper does not vary initialization scale or temperature to show this initial peak drives final miscalibration rather than being a benign starting point.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes that random weight initialization is an underlying cause of confidence miscalibration in deep networks, and that pretraining with random Gaussian noise and randomly re-sampled uniform labels drives the network's softmax outputs toward the uniform distribution, thereby pre-calibrating uncertainty. The authors report that this pretraining step reduces Expected Calibration Error (ECE) and improves OOD detection AUROC across a grid of small fully-connected networks (2-6 layers) trained on CIFAR-10 subsets, and that the benefit also appears with a biologically plausible feedback-alignment rule. The paper explicitly acknowledges that no real-world or large-scale validation was performed and that all experiments use relatively simple tasks and small-scale networks.","tokens_in":14363,"tokens_out":5045,"duration_ms":48293,"significance":"The empirical mapping is systematic: ECE improvements are reported across depths (2-6) and data sizes (500-32,000) with appropriate nonparametric tests, and the method is simple and requires no post-hoc calibration. The inclusion of a biologically plausible learning rule and the public code are positive aspects. However, the central mechanistic claim is substantially weakened by the fact that training against uniform random labels mathematically drives the softmax output to the uniform distribution; the observation that confidence after pretraining is near chance is a by-construction property of the loss, not an emergent discovery. In addition, the paper does not compare against standard calibration baselines such as temperature scaling, nor against other low-confidence initializations, so the specific benefit of random-noise pretraining over simply starting from a low-confidence state is not established. These gaps are load-bearing for the paper's broad conclusions.","major_comments":[{"comment":"Because labels are sampled uniformly and re-sampled each iteration, the conditional label distribution given any input is uniform, so the population optimum of the cross-entropy objective is the constant softmax output (1/K, ..., 1/K). Consequently, the observations in Fig. 3c, Fig. 3f-h, and Supplementary Fig. 6 that confidence after pretraining is near chance are direct consequences of the loss function rather than evidence of a novel 'pre-calibration' mechanism. The manuscript should state this explicitly and add a control that starts from any other uniform-output initial condition (e.g., zero or near-zero final-layer weights) to isolate what random-noise pretraining contributes beyond minimizing entropy at initialization.","section":"Methods: Pretraining with random noise"},{"comment":"The paper's central practical claim is that pretraining removes the need for 'additional pre- or post-processing,' but all comparisons are against an uncalibrated baseline (w/o pretraining). The standard post-hoc calibration baseline, temperature scaling (Guo et al., 2017), is not included, nor are label smoothing or other simple low-confidence initializations. Without these comparisons, the reader cannot tell whether the ECE and OOD AUROC improvements are comparable to a one-temperature-parameter post-hoc calibration, which would undercut the claim of a novel no-post-processing solution. Please add these baselines to Figures 2 and 5.","section":"Figure 2f inset and Figure 5d"},{"comment":"The claim that random initialization is an 'underlying cause' of miscalibration is not supported by the experimental design. The paper shows that untrained networks have high initial confidence and that pretraining changes this, but it does not vary initialization scale or output temperature to test whether the initial peakedness is causal for final miscalibration or merely a starting point that training can overcome. A concrete test would be to train normally from an initialization with a much smaller final-layer weight scale (or with an initial logit offset) and measure whether the final ECE matches the pretrained network; if it does, the effect is attributable to starting at low confidence, not to the random-noise pretraining procedure itself.","section":"Abstract and Discussion"}],"minor_comments":[{"comment":"The text 'w/o vs. zero, Wilcoxon signed-rank test, NS, P < 10−3' is internally contradictory; if P < 10^−3, the test is significant, so 'NS' should be removed or corrected.","section":"Figure 2e legend"},{"comment":"The text states that the primary analysis uses 1000 training samples, but Fig. 2g reports conditions from 500 to 32,000; please clarify which panels correspond to the primary analysis and which are the extended grid.","section":"Methods: Subsequent training with real data"},{"comment":"The class-bias measure, std([N0, N1, ..., NM]/sum Ni), is not normalized for the number of classes; for a fair comparison across different output dimensions, consider reporting a normalized entropy or dividing by the maximum possible standard deviation.","section":"Eq. (4)"},{"comment":"The statement that the method 'is applicable to all types of network models' is stronger than the evidence supports; the paper tests only fully-connected networks with ReLU activations and one biologically plausible learning rule, so please temper the generality claim.","section":"Discussion"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is transparent about its limitations, and the empirical ECE benefit across depths and dataset sizes is potentially useful. However, the novelty framing should be revised to acknowledge the by-construction nature of the pre-calibration effect, and the missing standard baselines make the current version unsuitable for acceptance. I would be willing to review a revised version that includes the proposed controls and baselines."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this paper shows a genuine, reproducible effect—random-noise pretraining lowers expected calibration error and improves confidence-based OOD detection on small feedforward networks. The code is public, the experiments cover several depths and dataset sizes, and the effect is consistent (P < 0.001 across conditions). That part deserves credit. The method itself isn't new—it comes from the authors' own NeurIPS 2024 paper—but the calibration application and the OOD result are new empirical findings.\n\nThe main soft spot is the missing control group. The paper frames pretraining as a replacement for post-hoc calibration, but it never compares against temperature scaling, label smoothing, or simply starting from a lower-confidence initialization. Without those controls, the benefit could come from starting the network at a lower confidence rather than from the specific random-noise procedure. The causal claim that random initialization is an underlying cause of miscalibration is also too strong: they don't vary initialization scale or temperature to show that the peaked softmax at initialization actually drives final miscalibration. The pre-calibration-to-chance observation is largely by construction—optimizing against random labels drives outputs to uniform—so the authors present that as a discovery when it's really a consequence of the loss function.\n\nTo their credit, the Discussion explicitly acknowledges the lack of real-world and large-scale validation. That is honest, but it also means the broad applicability claims—\"applicable to all types of network models,\" helping with LLM hallucinations—are speculative extrapolations from small MLPs.\n\nWho's this for? Someone working on calibration or initialization who wants a quick empirical hint that random-noise pretraining might help in low-data regimes. It's not a settled result. A serious referee should engage with it, but the paper needs major revision: add standard baselines, soften the causal language, and ideally test on at least one CNN or transformer. If those comparisons come back favorable, this becomes a useful contribution. As it stands, it's a promising but incomplete study.","headline":"A real, reproducible calibration effect on small MLPs, but the paper never compares to temperature scaling or other standard baselines, so the headline claim is unproven.","tokens_in":14884,"tokens_out":2123,"would_cite":false,"duration_ms":22508,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Random-noise pretraining can calibrate neural-network confidence by driving untrained networks to chance-level output, aligning confidence with accuracy during later training.","keywords":["uncertainty calibration","expected calibration error","random initialization","random noise pretraining","out-of-distribution detection","confidence calibration","feedforward neural networks","developmental neuroscience"],"falsifier":"Train the same feedforward architecture at increasing widths on random noise and record average SoftMax confidence after pretraining: the claim predicts it falls to the chance level $1/10$ for every width. A width at which average confidence stays well above chance, or at which training accuracy on random labels exceeds chance, would refute the load-bearing premise.","tokens_in":13903,"feed_emoji":"🎲","tokens_out":4279,"duration_ms":40775,"temperature":0.7,"pith_summary":"This paper tries to establish that conventional random initialization is a root cause of miscalibrated confidence in deep networks: untrained networks already output overconfident, class-biased probabilities. The proposed fix is a pretraining stage on randomly sampled Gaussian noise with freshly re-sampled uniform random labels, which pushes the network to chance-level confidence before any real data are seen. After real-data training, these pre-calibrated networks show lower Expected Calibration Error across model depths and dataset sizes, and their raw confidence scores detect out-of-distribution SVHN inputs more accurately. If the claim holds, confidence calibration can be improved with no extra calibration step, only a different initialization strategy.","feed_headline":"Random-noise pretraining calibrates neural network confidence","feed_subtitle":"A simple initialization change lowers calibration error and sharpens out-of-distribution detection with no post-processing.","key_machinery":"The central mechanism is a pretraining stage in which the network is trained on Gaussian noise inputs and independent uniform random labels, with both re-sampled every iteration. This drives the SoftMax output toward the uniform distribution, so confidence over the whole input space collapses to the chance level $1/C$ and class bias disappears. The paper uses this 'pre-calibration' as a replacement for conventional random initialization; it is the object that carries the later alignment of confidence and accuracy.","core_discovery":"Conventional random initialization leaves an untrained network's SoftMax confidence high and unevenly distributed across the input space, so when real data arrive the network starts from an overconfident prior and stays miscalibrated. The paper claims that pretraining the same network on Gaussian noise inputs with freshly re-sampled uniform random labels drives the output distribution to a uniform, chance-level solution; after that pre-calibration, subsequent training on CIFAR-10 keeps confidence and accuracy aligned throughout learning. The reported result is lower Expected Calibration Error across depths and dataset sizes, and lower confidence on SVHN out-of-distribution inputs, so raw confidence alone separates in-distribution from out-of-distribution data.","pith_inferences":["If the mechanism is a general flattening of the output prior, the same pretraining might apply to transformer-based models by feeding random token sequences and uniform labels; the paper does not test this.","The method could be combined with existing post-hoc calibration, but the paper's value is removing the need; whether gains would be additive or redundant is an open question.","The load-bearing assumption that the optimizer does not memorize noise-label pairs becomes harder to satisfy as capacity grows, so the method's applicability to very large models is an open question.","A practical extension would be to use the post-pretraining confidence map as a diagnostic for whether an architecture is overparameterized relative to its data, since overconfidence appears exactly in that regime."],"forward_implications":["Calibration error should be lower after random-noise pretraining for feedforward networks of depths 2 to 6 and training-set sizes 500 to 32,000 on CIFAR-10.","Raw confidence, without temperature scaling or post-hoc calibration, should separate CIFAR-10 in-distribution from SVHN out-of-distribution inputs above chance, as measured by AUROC.","The benefit should persist when training uses feedback alignment instead of backpropagation, so it is not tied to a specific learning rule.","Untrained networks initialized by the pretraining procedure should show no class bias and no overconfidence on held-out inputs before data training.","Pretraining should reduce test loss during subsequent training more than data-only training at matched accuracy or matched training epochs."],"supporting_citations":[{"why":"Supplies the random-noise pretraining procedure that the paper re-purposes for confidence calibration.","marker":"[34]"},{"why":"Establishes the overconfidence and Expected Calibration Error baseline that the paper compares against.","marker":"[9]"},{"why":"Defines Expected Calibration Error, the paper's main calibration metric.","marker":"[37]"},{"why":"Provides the confidence-based out-of-distribution detection baseline that raw calibrated confidence is measured against.","marker":"[12]"},{"why":"Is the standard random initialization whose overconfidence is identified as the cause of miscalibration.","marker":"[44]"},{"why":"Provides the CIFAR-10 in-distribution dataset used for training and evaluation.","marker":"[35]"},{"why":"Provides the SVHN out-of-distribution dataset used for testing OOD detection.","marker":"[45]"},{"why":"Tests whether the calibration benefit survives a biologically plausible learning rule (feedback alignment).","marker":"[43]"}],"fun_headline_variants":["Pretrain on noise to cure overconfident nets","Random-noise pretraining aligns confidence with accuracy","Noise pretraining marks unknown data with low confidence","Random-noise pretraining fixes confidence calibration","Calibrate networks by pretraining on pure noise"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire effect rests on the pretraining optimizer driving the network to a uniform-output solution instead of exploiting the randomly re-sampled noise-label pairs; if any batch-specific correlation gets memorized, confidence after pretraining would rise rather than fall.","fun_headline_variants_meta":{"raw":{"variants":["Pretrain on noise to cure overconfident nets","Random-noise pretraining aligns confidence with accuracy","Noise pretraining marks unknown data with low confidence","Random-noise pretraining fixes confidence calibration","Calibrate networks by pretraining on pure noise"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000788,"raw_usage":{"total_tokens":3407,"prompt_tokens":807,"completion_tokens":2600,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":423,"completion_tokens_details":{"reasoning_tokens":2529}},"tokens_in":423,"tokens_out":2600,"duration_ms":17167,"temperature":1.0,"reasoning_tokens":2529,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:27:03.874751+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same feedforward architecture at increasing widths on random noise and record average SoftMax confidence after pretraining: the claim predicts it falls to the chance level $1/10$ for every width. A width at which average confidence stays well above chance, or at which training accuracy on random labels exceeds chance, would refute the load-bearing premise.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the random-noise pretraining procedure that the paper re-purposes for confidence calibration."},{"cited_title":"& Weinberger, K","cited_arxiv_id":null,"evidence_quote":"Establishes the overconfidence and Expected Calibration Error baseline that the paper compares against."},{"cited_title":"P ., Cooper, G","cited_arxiv_id":null,"evidence_quote":"Defines Expected Calibration Error, the paper's main calibration metric."},{"cited_title":"& Gimpel, K","cited_arxiv_id":null,"evidence_quote":"Provides the confidence-based out-of-distribution detection baseline that raw calibrated confidence is measured against."},{"cited_title":"& Sun, J","cited_arxiv_id":null,"evidence_quote":"Is the standard random initialization whose overconfidence is identified as the cause of miscalibration."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CIFAR-10 in-distribution dataset used for training and evaluation."},{"cited_title":"Reading digits in natural images with unsupervised feature learning","cited_arxiv_id":null,"evidence_quote":"Provides the SVHN out-of-distribution dataset used for testing OOD detection."},{"cited_title":"P ., Cownden, D., Tweed, D","cited_arxiv_id":null,"evidence_quote":"Tests whether the calibration benefit survives a biologically plausible learning rule (feedback alignment)."}],"review_version":1}