{"id":"d989fe7b-82d8-42b0-abe3-d520c056fe70","arxiv_id":"2608.01142","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"EulerLoRA samples over the rank-one directions of shared LoRA adapters to generate an implicit ensemble that preserves the deterministic LoRA output in expectation and matches a 16-adapter LoRA-Ensemble with 69% fewer parameters.","lead":"EulerLoRA adds random 'jumps' to the low-rank adapter weights used in fine-tuning, so a single trained model can produce many different predictions for the same image. It gives useful uncertainty estimates and out-of-distribution detection, matching a 16-model LoRA ensemble with about 69 percent fewer trainable parameters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unbiasedness guarantee is projection-level only: rank sampling at Q/K/V passes through softmax, so the full attention block is not mean-preserving; the paper's own supplementary concedes this, and stochastic inference is not consistently better than deterministic inference.","rationale":"The reader's weakest assumption identifies exactly the point where the paper's technical guarantee is weakest: the projection-level unbiasedness result does not survive the attention nonlinearity, and the paper says so in the supplementary. That is the load-bearing concern because the method's identity as an expectation-preserving stochastic LoRA, rather than a generic noise-regularized adapter, depends on the full model being an unbiased or nearly unbiased ensemble around the deterministic LoRA block. The paper's own limitation statement is the clearest evidence that this concern is real and not manufactured. I agree with the reader's conditional verdict: the theoretical derivation is sound at the conditional level, and the empirical results are promising, but the broad 'preserving deterministic LoRA in expectation' claim should be qualified to the block level, and the usefulness of stochastic inference should be checked against deterministic inference on each benchmark. The proposed test—measuring full-network bias against MC spread at a fixed checkpoint—would settle whether the missing theoretical guarantee is practically material or merely a formal caveat. I do not see grounds for rejection: the reproduced matched-parameter LoRA-Ensemble comparison, the explicit supplementary limitation, and the consistent improvements of at least one inference mode over baselines all support a conditional acceptance contingent on this analysis and on code release.","tokens_in":16150,"tokens_out":13000,"duration_ms":121468,"concrete_test":"Take a trained EulerLoRA R20 Ad2 checkpoint and, on a held-out set, compute for each input the deterministic logits z_det (all rank components active) and the mean of N=100 stochastic forward passes \\bar z_stoch using the training-time mask distribution. Report (i) the average L2 bias ||E_s[\\bar z] - z_det|| and (ii) the average within-sample spread std_s(z_s). If the bias is a substantial fraction of the spread (e.g., >20%), the full network is not acting as an unbiased ensemble around the deterministic LoRA model, and the expectation-preserving narrative is not supported at the model level; this would require reframing the central claims as empirical regularization rather than unbiased stochastic LoRA. If the bias is negligible, the concern is resolved and the conditional acceptance can proceed with minor wording changes.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central theoretical result, Eq. (17), establishes E[Y_{ℓ,T}|C_ℓ] for the compensated output projection only. But the actual forward pass also samples rank masks at Q, K, and V (Eq. 24), and those stochastic projections feed into the attention softmax before C_ℓ is formed. Since softmax is nonlinear, the expectation of the complete stochastic attention block is not the deterministic attention block. The supplementary states this explicitly: 'this projection-level expectation result does not imply that the output of the complete stochastic attention block is an unbiased estimator of the corresponding deterministic attention block.' Thus the abstract's claim of 'preserving the deterministic LoRA transformation in expectation' holds only for a conditional sub-block, not for the model whose predictive distribution is averaged at inference. The practical value of EulerLoRA therefore rests on the empirical assumption that this bias is small enough that the Monte Carlo ensemble is a faithful ensemble around the deterministic LoRA prediction, and that the remaining stochasticity is useful. The tables do not establish this: on CIFAR-10 and CIFAR-100 the deterministic mode dominates the stochastic mode on every proper-scoring metric (e.g., CIFAR-10 NLL 0.116 vs 0.126, ECE 0.004 vs 0.016; CIFAR-100 NLL 0.559 vs 0.589, ECE 0.013 vs 0.058). The only clear stochastic-inference benefits are on HAM10000 calibration and SVHN FPR@95TPR. Without measuring the actual full-block bias, the calibration and OOD gains could be artifacts of a biased smoothing rather than a faithful ensemble, and the central claim of useful predictive diversity from shared adapters remains incompletely supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EulerLoRA, a stochastic extension of LoRA. At the output projection of each attention block, an auxiliary state is evolved over T internal Euler steps; at each step a random subset of rank-one LoRA components is activated with weights r/K, and the centered fluctuations are added with stochastic scaling. The paper proves (Eq. 17) that, conditional on the input to the output projection, the expected terminal state equals the deterministic LoRA block. During stochastic inference, multiple forward passes are averaged to form an implicit ensemble; setting all rank components active gives deterministic inference. Experiments on ViT-B/32 compare one- and two-adapter rank-20 configurations against LoRA-Ensemble baselines on CIFAR-10, CIFAR-100, HAM10000, and SVHN OOD detection, reporting accuracy, macro-F1, ECE, NLL, Brier score, and OOD metrics. The headline claim is that two rank-20 adapters (~3M trainable parameters) match or beat a rank-8, 16-adapter LoRA-Ensemble (~10M parameters), a reduction of about 69%.","tokens_in":16490,"tokens_out":7785,"duration_ms":69874,"significance":"If the claims were fully established, the main contribution would be a simple and parameter-efficient way to obtain predictive diversity from shared LoRA adapters, with a clean projection-level expectation guarantee and a deterministic inference mode. The paper is transparent in several respects: it reports five-seed means and standard deviations, reproduces rank-20 baselines, gives detailed hyperparameters and training settings, and explicitly states in the supplementary that the expectation guarantee does not extend through the attention softmax. However, the theoretical guarantee is narrower than the abstract suggests, and the stochastic-inference advantage is not uniform across benchmarks. The value of the paper after revision would be as an honest demonstration that stochastic training of LoRA can improve deterministic performance and sometimes improve calibration and OOD detection, rather than as a general unbiased-ensemble method.","major_comments":[{"comment":"The central theoretical claim is proven only for the output projection conditional on C_l. Eq. (24) samples rank configurations at the Q, K, and V projections in every stochastic forward pass, and those stochastic inputs pass through the attention softmax before C_l is formed. Since softmax is nonlinear, the expectation of the complete stochastic attention block is not the deterministic attention block; the supplementary explicitly concedes this. The abstract and contributions, however, state that EulerLoRA preserves 'the deterministic LoRA transformation in expectation' without this qualification. This is a load-bearing overclaim: the Monte Carlo ensemble at inference is not guaranteed to be centered on the deterministic LoRA prediction, and Tables 1 and 4 show that deterministic inference often beats stochastic inference on proper-scoring metrics. The authors should either qualify the theoretical claims throughout or add an empirical analysis quantifying the full-block bias, for example by comparing the mean stochastic attention output with the deterministic attention output over the validation set.","section":"Compensated Rank-Driven Jump Dynamics, Eq. (17); supplementary 'Random Active-Rank Sampling at Q, K, and V'"},{"comment":"The empirical case for stochastic inference as a source of 'useful predictive diversity' is mixed. On CIFAR-10 with two adapters, the stochastic mode is worse than the deterministic mode on every proper-scoring metric (NLL 0.126 vs 0.116, ECE 0.016 vs 0.004, Brier 0.064 vs 0.059). On CIFAR-100, the stochastic mode has substantially higher ECE (0.058 vs 0.013) and NLL (0.589 vs 0.559). The clear stochastic-inference wins are HAM10000 calibration/NLL and SVHN FPR@95TPR. The paper's own 'complementary benefits' conclusion is defensible, but the abstract's general claim that 'useful predictive diversity can be obtained' needs to be scoped, ideally with statistical significance tests or effect sizes over the five seeds and with a discussion of when stochastic inference is expected to help.","section":"Experiments, Tables 1, 3, and 4"},{"comment":"The headline comparison with the 16-adapter LoRA-Ensemble uses numbers taken from the original LoRA-Ensemble paper rather than re-run under the same protocol and seeds. The reproduced controls are rank-20 LoRA and rank-20 two-adapter LoRA-Ensemble. Given the central parameter-efficiency claim, a controlled reimplementation of the rank-8/16-adapter baseline with the same five seeds, training script, and evaluation pipeline would substantially strengthen the claim; as it stands, the cross-paper comparison may conflate implementation differences with method differences.","section":"Experiments, Baselines and Tables 1, 3, 4"}],"minor_comments":[{"comment":"The sentence 'Figure 1 gives an overview our proposed method' should read 'gives an overview of our proposed method'.","section":"Figure 1"},{"comment":"The reproducibility summary says 'Compute the metrics in Section' without a section number or name; this should be completed.","section":"Supplementary Material, Evaluation Metrics"},{"comment":"No statistical significance testing is reported; given five seeds and small standard deviations, statements comparing close values such as CIFAR-100 accuracy 82.6 +/- 0.2 vs 82.6 +/- 0.1 should avoid over-interpreting small gaps.","section":"Experiments, Tables 1 and 4"},{"comment":"The paper does not report diversity diagnostics such as average pairwise disagreement or variance of the Monte Carlo predictions, even though predictive diversity is the core mechanism; reporting one such measure would directly support the narrative.","section":"Overall analysis"},{"comment":"The training procedure uses four stochastic samples per adapter, but the compute overhead of training is not reported, so the parameter-efficiency comparison alone does not fully characterize the total cost relative to LoRA-Ensemble.","section":"Training protocol"}],"recommendation":"major_revision","confidential_remarks":"The paper is salvageable. The core derivation in Eq. (17) is correct under the stated assumptions, and the empirical work is careful in several respects. The main issue is that the abstract and contributions overstate the theoretical guarantee, while the empirical evidence for stochastic inference is mixed. I would send the paper back for revision rather than reject it. I see no concern about novelty disclosure or citation patterns; the relationship to LoRA-Ensemble is clearly acknowledged."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"EulerLoRA is a genuinely new mean-preserving stochastic LoRA construction, and the math is cleaner than the packaging suggests. The compensated dynamics in Eq. (15), with the centered rank fluctuation and the T-independent expectation in Eq. (17), is a real construction, not a restatement of rank dropout or a dressed-up ensemble. The paper also earns credit for stating the limitation in the supplementary: the unbiasedness is projection-level, and because softmax is nonlinear it does not survive the full attention block. That kind of honesty is rare.\n\nWhere the paper does well: the derivation is sound, the parameter count is transparent, the five-seed protocol and reproduced LoRA-Ensemble rank-20 controls are the right kind of evidence, and the deterministic inference mode is a legitimate practical contribution on its own. On CIFAR-10, CIFAR-100, and HAM10000, the all-ranks-active deterministic checkpoint trained with rank sampling beats the reproduced LoRA-Ensemble on most proper-scoring metrics, with far fewer parameters. That is a solid empirical result even before considering stochastic inference.\n\nSoft spots, in proportion. The central claim of \"useful predictive diversity\" is only half-supported. Stochastic inference beats deterministic on HAM10000 calibration and SVHN FPR@95TPR, but on CIFAR-10 and CIFAR-100 deterministic inference dominates on every proper-scoring metric, sometimes by a wide margin (NLL 0.116 vs 0.126; ECE 0.004 vs 0.016). So the Monte Carlo rank sampling is not consistently giving ensemble-like uncertainty benefits over the deterministic limit. The paper does not measure the full-block bias from the Q/K/V sampling, so the calibration and OOD gains could be a byproduct of biased smoothing rather than a faithful ensemble. This is a real gap, but not a fatal one: the method's practical value does not collapse without stochastic inference, and the authors have already flagged the projection-level scope. The other weaknesses are modest: the 16-adapter baseline numbers are borrowed from LoRA-Ensemble without significance tests, several hyperparameters (T, sigma, Kmin, S) are hand-chosen without sensitivity analysis, and code is promised but not released. None of those make the contribution invalid.\n\nWho this is for: anyone working on parameter-efficient fine-tuning and uncertainty, especially people who want a cheap way to get OOD detection or calibration from LoRA adapters. It deserves a serious referee. I would send it out, with demands for code, a sensitivity analysis on the two or three most important hyperparameters, and either a direct estimate of the full-block bias or a much more careful comparison of stochastic vs deterministic inference across all benchmarks.","headline":"EulerLoRA is a genuinely new mean-preserving stochastic LoRA construction with clean math and honest caveats; the full-block unbiasedness is only projection-level, and stochastic inference gains are narrower than the abstract implies, but the deterministic trained model alone is competitive and the paper deserves a serious referee.","tokens_in":17031,"tokens_out":2324,"would_cite":true,"duration_ms":19961,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Rank-sampled LoRA matches a 16-model ensemble at 69% lower cost.","keywords":["parameter-efficient fine-tuning","LoRA","predictive uncertainty","ensemble methods","rank-driven dynamics","model calibration","out-of-distribution detection","vision transformers"],"falsifier":"Measure the gap between the stochastic block's average output and the deterministic block's output on a real attention block: compute $\\|\\mathbb{E}[\\mathrm{softmax}(Y_{\\ell,T})]-\\mathrm{softmax}(\\mathbb{E}[Y_{\\ell,T}])\\|$ with a large number of Monte Carlo samples; if this gap stays large while trajectory diversity is high and stochastic inference never improves calibration or OOD detection over deterministic inference across ranks and datasets, the claim that shared-adapter rank sampling produces useful predictive diversity would be refuted.","tokens_in":15916,"feed_emoji":"🎲","tokens_out":7922,"duration_ms":62131,"temperature":0.7,"pith_summary":"This paper tries to establish that predictive diversity—the usual benefit of training many separate models—can be obtained from a single shared low-rank adapter by adding structured randomness along its rank-one components. The method, EulerLoRA, evolves an auxiliary state through a few internal Euler steps, sampling a subset of rank directions at each step and rescaling their coefficients so that every direction contributes with expected coefficient one; the average over trajectories is exactly the deterministic LoRA transformation. If this holds, then calibration, negative log-likelihood, and out-of-distribution detection can be improved without training one adapter per ensemble member. The paper reports that two rank-20 adapters (about 3 million trainable parameters) match or beat a 16-adapter rank-8 LoRA-Ensemble (about 10 million parameters) on CIFAR-10, CIFAR-100, HAM10000, and SVHN out-of-distribution detection.","feed_headline":"Rank-sampled LoRA matches a 16-model ensemble at 69% lower cost","feed_subtitle":"One shared low-rank adapter produces many predictive trajectories while preserving exact LoRA on average.","key_machinery":"The central object is the compensated rank-driven jump dynamics at the output projection. An auxiliary state $Y_{\\ell,s}$ evolves for $T$ internal Euler steps with $\\Delta t=1/T$; each step samples a rank configuration $(K_{\\ell,s},S_{\\ell,s})$ and applies the coefficient $M_{\\ell,s,k}=r/K_{\\ell,s}$ to selected rank-one directions. Writing $M=1+(M-1)$ separates the predictable LoRA increment (accumulated once over the unit interval via $\\Delta t$ scaling) from centered zero-mean noise (accumulated with $\\sqrt{\\Delta t}$ scaling), which is what makes the terminal expectation exactly LoRA while individual trajectories fluctuate. In deterministic inference the sampled subset is all $r$ components, so $M=1$ and the stochastic term vanishes; in stochastic inference, four Monte Carlo samples per adapter are averaged. The construction is applied to the output projection with $T=2$; query, key, and value projections use a single expectation-preserving rank-sampled update per forward pass.","core_discovery":"At the output projection of a transformer block, EulerLoRA replaces the deterministic LoRA correction $B_\\ell A_\\ell C_\\ell$ with a finite-step stochastic process. At each of $T$ internal steps it samples an active rank $K_{\\ell,s}$ and a uniformly chosen subset of rank-one directions, giving component coefficients $M_{\\ell,s,k}=\\frac{r}{K_{\\ell,s}}\\mathbf{1}\\{k\\in S_{\\ell,s}\\}$ with conditional expectation one. The update separates the deterministic LoRA term (scaled by $\\Delta t$) from the centered fluctuation $M_{\\ell,s,k}-1$ (scaled by $\\sqrt{\\Delta t}$), so the expected terminal state satisfies $\\mathbb{E}[Y_{\\ell,T}\\mid C_\\ell]=h_\\ell+W^0_{O,\\ell}C_\\ell+B_\\ell A_\\ell C_\\ell$, exactly the deterministic LoRA block, independent of $T$. Trajectories differ around that mean, and averaging softmax probabilities over them produces an implicit ensemble. The paper's empirical claim is that this shared-adapter diversity is useful: two rank-20 adapters outperform or match a much larger 16-adapter LoRA-Ensemble in accuracy, calibration, NLL, Brier score, and OOD detection, with about 69% fewer trainable adapter parameters.","pith_inferences":["The mean-preservation theorem is proved only at the linear projection level; because self-attention applies softmax, the full block output is not guaranteed to be an unbiased estimate of the deterministic block. A direct extension would be to quantify the bias $\\mathbb{E}[\\mathrm{softmax}(Y_{\\ell,T})]-\\mathrm{softmax}(\\mathbb{E}[Y_{\\ell,T}])$ and test whether it shrinks with more internal steps or","EulerLoRA's rank-sampling schedule is a free design choice: the paper fixes $r=20$, $K_{\\min}=10$, and uniform $K$. Varying the minimum active rank or making the coefficient distribution non-uniform would change trajectory diversity, so the reported parameter-efficiency gain might extend to even smaller adapters or collapse if trajectories become correlated.","Because the expected terminal state is exactly the deterministic LoRA block, EulerLoRA can be layered on top of existing LoRA training pipelines as a drop-in stochastic head: the same checkpoint can serve as a deterministic predictor and, at inference time, as an uncertainty estimator, which suggests applications to active learning or selective prediction where calibrated confidence is needed."],"forward_implications":["The number of Monte Carlo predictions is decoupled from the number of learned adapters: four samples per adapter yield the reported uncertainty estimates, and more samples can be added without training anything new.","One trained EulerLoRA checkpoint supports two inference modes: deterministic inference (all rank components active) gives the best accuracy, NLL, and Brier score, while stochastic inference gives the best calibration on HAM10000 and the best OOD FPR@95TPR on SVHN.","Two rank-20 adapters (about 2.95 million adapter parameters) match or beat a 16-adapter rank-8 LoRA-Ensemble (about 9.44 million adapter parameters), a reduction of about 69% in trainable adapter parameters.","On CIFAR-10, EulerLoRA with two adapters reaches 96.10% accuracy and 0.116 NLL in deterministic mode, improving on the reproduced two-adapter LoRA-Ensemble (95.43%, 0.138) and the published 16-adapter ensemble (95.9%, 0.128).","On SVHN OOD detection, stochastic EulerLoRA lowers FPR@95TPR to 37.42%, a 7.76-percentage-point improvement over the reproduced two-adapter LoRA-Ensemble."],"supporting_citations":[{"why":"Defines LoRA's low-rank parameterization $W_0+\\Delta W$ with $\\Delta W=BA$, the object EulerLoRA makes stochastic.","marker":"Hu et al. 2022"},{"why":"Provides the LoRA-Ensemble baseline and the exact training configuration and seeds that EulerLoRA reuses for its comparisons.","marker":"Mühlematter et al. 2026"},{"why":"Defines expected calibration error, the calibration metric whose improvement is a central empirical claim.","marker":"Guo et al. 2017"},{"why":"Defines maximum softmax probability, the out-of-distribution score used in the SVHN experiment.","marker":"Hendrycks and Gimpel 2017"},{"why":"Supplies the neural-ODE and Euler-discretization viewpoint that motivates the finite-step internal dynamics.","marker":"Chen et al. 2018"},{"why":"Provides the explicit deep-ensemble baseline that motivates the goal of obtaining ensemble-like uncertainty without training multiple full models.","marker":"Lakshminarayanan, Pritzel, and Blundell 2017"}],"fun_headline_variants":["EulerLoRA: 69% fewer params, implicit ensemble diversity","One shared adapter, multiple models, 69% less cost","Rank-sampled LoRA matches 16-model ensemble at 31% cost","Stochastic LoRA: calibrated fine-tuning with 69% fewer parameters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The practical gains rest on the assumption that the nonlinear operations inside self-attention, especially softmax, do not destroy the diversity or bias the average of the rank-sampled trajectories, so the projection-level expectation guarantee still yields useful predictive diversity at the block output.","fun_headline_variants_meta":{"raw":{"variants":["EulerLoRA: 69% fewer params, implicit ensemble diversity","One shared adapter, multiple models, 69% less cost","Rank-sampled LoRA matches 16-model ensemble at 31% cost","Stochastic LoRA: calibrated fine-tuning with 69% fewer parameters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001508,"raw_usage":{"total_tokens":6077,"prompt_tokens":1008,"completion_tokens":5069,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":4990}},"tokens_in":624,"tokens_out":5069,"duration_ms":30234,"temperature":1.0,"reasoning_tokens":4990,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:11:08.340896+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the gap between the stochastic block's average output and the deterministic block's output on a real attention block: compute $\\|\\mathbb{E}[\\mathrm{softmax}(Y_{\\ell,T})]-\\mathrm{softmax}(\\mathbb{E}[Y_{\\ell,T}])\\|$ with a large number of Monte Carlo samples; if this gap stays large while trajectory diversity is high and stochastic inference never improves calibration or OOD detection over deterministic inference across ranks and datasets, the claim that shared-adapter rank sampling produces useful predictive diversity would be refuted.","supporting_citations":[],"review_version":1}