{"id":"37b3b612-b551-4b2d-869e-b82fefe58074","arxiv_id":"2502.06173","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"LoRA ensembles and Bayesian LoRA improve accuracy and calibration of LLM-based protein-protein interaction prediction on disease-specific datasets, though gains are small and baselines are limited.","lead":"Researchers fine-tuned large language models to predict whether proteins interact in disease contexts, adding uncertainty estimates to flag low-confidence answers. The uncertainty-aware versions improved both accuracy and calibration on three disease-focused datasets, but gains were modest and limited by weak baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LoRA rank is selected on the test set (Appendix B), so the reported accuracy and calibration gains may be inflated by test-set overfitting and do not yet support the central claim.","rationale":"The reader's weakest assumption was the train/test split (protein vs. interaction). That is a real concern, but it is an omission that could be clarified; if the split is by interaction, the task may still be meaningful for predicting interactions between known proteins. The hyperparameter selection on the test set, however, is an explicit and unambiguous violation of proper evaluation protocol. It appears in Appendix B and Implementation Details, and it directly affects the numbers that support the central claim. The same-rank constraint across methods mitigates some bias, but the selection rule remains a form of test-set leakage, and the statistical tests in the paper do not account for multiple comparisons over ranks. A validation-based re-run is the natural fix. Therefore the reader's CONDITIONAL verdict remains appropriate; the concern does not by itself warrant rejection, but it must be addressed before the accuracy/calibration gains can be taken as evidence. I partially agree with the reader because both concerns target the reliability of the empirical evaluation, but I would prioritize the rank-selection issue as the single most load-bearing one.","tokens_in":12680,"tokens_out":8515,"duration_ms":76126,"concrete_test":"Hold out 10% of the current training set as validation; for each backbone-dataset combination and each r in {8,16,32}, train Single LoRA, LoRA Ensemble, and Bayesian LoRA; pick the best r on validation using the same criterion as in Appendix B; then evaluate the chosen models on the untouched 20% test split. If the test accuracy/NLL/ECE gaps between Single LoRA and the UQ variants shrink or reverse compared to Table I, the reported gains are artifacts of selection on the test set. Also check whether any test protein appears in the training set, and if so, rerun with a protein-disjoint split to rule out memorization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that uncertainty-aware LoRA (LoRA Ensemble and Bayesian LoRA) improves PPI accuracy and calibration rests entirely on Table I's test metrics. In 'Implementation Details' the authors state that the LoRA rank r is selected via hyperparameter search (Appendix B), and Appendix B shows that r is chosen by comparing test-set performance for each backbone-dataset combination. No validation set is used. This is test-set leakage: the reported Acc/NLL/ECE are the best over three hyperparameter choices, not unbiased estimates of generalization. Because the same r is then applied to all three methods, the relative comparison is not necessarily unfair, but the selection rule 'overall best performance across LoRA-based models' is ambiguous and can systematically favor whichever method peaks at the chosen r. For example, on M-PPI with LLaMA-3, r=16 is selected; Bayesian LoRA has its best accuracy at r=16 (83.41) and much lower at r=8 (82.37), so the comparison cherry-picks favorable settings. The paper must either re-select r on a validation split or report results averaged across all r values. Until then, the empirical evidence for 'enhanced accuracy and calibration' is not trustworthy. The missing split specification (by protein vs. by interaction) is a related but secondary problem; the test-set-based selection is an explicit methodological flaw that independently undermines the strongest claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes uncertainty-aware parameter-efficient fine-tuning of large language models for protein-protein interaction (PPI) prediction. It applies LoRA ensembles and Bayesian LoRA (Laplace approximation with K-FAC) to LLaMA-3-8B and BioMedGPT-LM-7B on three disease-focused balanced PPI datasets (neurodegenerative, metabolic, cancer), comparing accuracy, negative log-likelihood, expected calibration error, and several secondary metrics against a single-LoRA baseline. The central claim is that uncertainty-aware LoRA adaptation improves both PPI prediction accuracy and confidence calibration.","tokens_in":12941,"tokens_out":3979,"duration_ms":36026,"significance":"If the empirical claim is supported, the paper would provide a practical and computationally efficient recipe for uncertainty quantification in biomedical LLM applications, with potential value for trustworthy PPI screening and downstream disease-mechanism studies. The authors report three independent runs, multiple complementary metrics, reliability diagrams, and use publicly available datasets and established UQ methods. However, the central empirical claim is currently undermined by a test-set-based hyperparameter selection protocol and by an unspecified train/test split granularity; these issues are load-bearing for the conclusion and must be fixed before the results can be interpreted as evidence of improved generalization.","major_comments":[{"comment":"The LoRA rank r is selected by comparing test-set performance in the hyperparameter search shown in Table II, and the chosen r is then fixed across methods. This is test-set leakage: the Acc/NLL/ECE values in Table I are the best over three hyperparameter choices rather than unbiased estimates of generalization. The problem is visible in M-PPI with LLaMA-3, where Bayesian LoRA accuracy rises from 82.37 at r=8 to 83.41 at r=16 and the selected r=16 favors that method. The authors should either select r on a held-out validation split or report results averaged over all r values; without this, the reported accuracy and calibration gains are not trustworthy.","section":"§IV 'Implementation Details' and Appendix B"},{"comment":"The text states that each dataset is partitioned into an 80% training set and a 20% testing set, but it never specifies whether the split is at the protein level or at the interaction level. If the same proteins appear in both training and test pairs, the model can memorize protein-specific features and the reported accuracy does not measure generalization to unseen proteins. This distinction is essential for the reliability claim and should be stated explicitly; if the current split is interaction-level, a protein-level split should be added to validate the generalization claim.","section":"§IV 'PPI Datasets'"},{"comment":"The statistical evidence is weaker than the conclusion claims. The text acknowledges that LLaMA-3 ND-PPI accuracy, BioMedGPT M-PPI accuracy, and all BioMedGPT C-PPI metrics are not statistically significant (p>0.05), yet the abstract and conclusion state that uncertainty-aware adaptation 'enhances PPI prediction accuracy' and 'consistently improves' performance. The conclusion should be qualified to the settings where the gains are significant, and the actual p-values, together with a multiple-testing correction for the large number of metrics, should be reported.","section":"§IV 'Results'"},{"comment":"The selection rule is described as choosing the rank with 'overall best performance across LoRA-based models', but this criterion is not precisely defined: it is unclear which metric is optimized, whether accuracy or calibration is prioritized, and how ties are broken. Because the selected rank can systematically favor one method (as the M-PPI LLaMA-3 example shows), the comparison among Single LoRA, LoRA Ensemble, and Bayesian LoRA is not fully controlled. The selection rule should be specified precisely, ideally using a validation metric.","section":"Appendix B"}],"minor_comments":[{"comment":"In the Bayesian LoRA derivation, 'AdamW with its weight deca' appears to be a typo for 'weight decay'.","section":"§III-B"},{"comment":"The text says 'we set |B_m| = 15', but the ECE formula uses |B_m| to denote bin size; if 15 is the number of bins, the notation should be M=15 or should be clarified.","section":"Appendix A"},{"comment":"The standard deviations are printed as concatenated subscripts (e.g., 87.421.64), which is difficult to read; please use proper subscripts or separate columns for means and standard deviations.","section":"Table I"},{"comment":"Reliability diagrams are shown only for LLaMA-3, with BioMedGPT omitted for brevity; given that the conclusions cover both backbones, the BioMedGPT diagrams should be included in a supplement or their ECE values should be discussed in the text.","section":"Fig. 2"},{"comment":"No data or code availability statement is provided, which makes the exact preprocessing and split procedure impossible to reproduce from the paper alone.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is an application-oriented empirical study of existing UQ methods rather than a methodological advance, which is acceptable if the empirical evidence is solid. The test-set-based rank selection is a serious evaluation-protocol flaw that currently blocks acceptance; it is fixable within the manuscript's scope by introducing a validation split or averaging over ranks, and by clarifying the protein-level vs. interaction-level split. The overstated conclusion for non-significant C-PPI results should also be tempered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here is my read on arXiv:2502.06173. The paper does something straightforward but useful: it applies two established uncertainty-quantification recipes (LoRA ensembles and Bayesian LoRA) to LLM-based protein–protein interaction prediction on three disease-focused datasets, using LLaMA-3 and BioMedGPT. The new content is the empirical comparison itself, not the methods. Credit where due: the experiments are reported fairly, with means and standard deviations over three seeds, and the authors explicitly flag which comparisons are not statistically significant (e.g., all BioMedGPT C-PPI metrics). The reliability diagrams are a nice addition.\n\nThe soft spot is real and load-bearing. In Appendix B the LoRA rank r is selected on the test set. Table I then reports the best performance among the three r values. That is test-set leakage: the headline numbers are hyperparameter-tuned on the evaluation set, not unbiased estimates of generalization. The stress-test example is on point: on M-PPI with LLaMA-3, r=16 is chosen, and Bayesian LoRA's accuracy is best at that rank and much worse at r=8, so the comparison cherry-picks favorable settings. The relative ranking across methods is probably not dramatically unfair since the same r is used for all three, but the absolute gains over single LoRA are inflated and the central claim that UQ \"enhances accuracy and calibration\" is not yet supported.\n\nThe second issue the reader flagged is also worth stating plainly: the paper never says whether the 80/20 split is by protein or by interaction. If proteins overlap between train and test, the model may memorize protein-specific patterns, and the reported accuracy does not measure generalization to unseen proteins. This is a missing experimental detail that needs fixing before the results can be trusted.\n\nThese are not fatal flaws in the sense that the methods are broken. They are fixable with a validation split and a clarified protocol. The paper is a reasonable recipe for practitioners who want to add calibrated confidence to LLM-based PPI screens. But as written, the evidence for the strongest claims is not trustworthy.\n\nWho is this for? People in biomedical ML who want a worked example of LoRA ensembles and Bayesian LoRA on protein interaction tasks. It does not open new science, but it is a competent application. A serious referee should see it, but the authors need to re-run rank selection on a validation split and state the split granularity before the results can be accepted. That is a major-revision request, not a reject.\n\nIn short: engage with it, but require the fix.","headline":"A useful empirical comparison of two existing UQ methods on disease-specific PPI prediction, but the central accuracy and calibration claims are undermined by test-set-based rank selection and an unspecified train/test split; worth a major revision, not a desk reject.","tokens_in":13492,"tokens_out":1505,"would_cite":false,"duration_ms":15507,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Uncertainty-aware fine-tuning of large language models is shown to improve protein-protein interaction prediction while producing better-calibrated confidence estimates.","keywords":["Large Language Models","Low-Rank Adaptation","Uncertainty Quantification","Bayesian Inference","Deep Ensemble","Protein-Protein Interaction","Confidence Calibration","BioMedGPT"],"falsifier":"Re-run the three PPI experiments with an 80/20 split that holds out entire proteins, so that no protein appearing in the training set appears in the test set; if the accuracy and calibration advantages of the uncertainty-aware variants shrink or vanish, the original results partly reflect protein-level memorization rather than robust interaction prediction.","tokens_in":12479,"feed_emoji":"🧬","tokens_out":7561,"duration_ms":57456,"temperature":0.7,"pith_summary":"This paper sets out to show that the reliability of large language models in predicting protein-protein interactions can be improved by making the fine-tuning procedure uncertainty-aware. It compares a standard low-rank adapter (LoRA) against two uncertainty-aware variants—an ensemble of LoRA adapters and a Bayesian LoRA that approximates a posterior over the adapter weights—on balanced task-specific datasets for neurodegenerative, metabolic, and cancer-related interactions. Across most settings, the uncertainty-aware variants achieve higher accuracy and lower negative log-likelihood than the single LoRA baseline, and they produce confidence scores that are better calibrated to actual correctness. The reason this matters is that biomedical users need to know when a prediction is trustworthy, not just whether the model is often right.","feed_headline":"Uncertainty-aware LoRA sharpens LLM protein-interaction calls","feed_subtitle":"LoRA ensembles and Bayesian LoRA improve accuracy and calibration across three disease-specific PPI datasets.","key_machinery":"The central object is the low-rank adapter: with the backbone frozen, the forward pass becomes $h = (W_0 + BA)a$, so only the small matrices $B$ and $A$ are trained. Uncertainty is introduced in two ways. First, a LoRA ensemble trains $M$ such adapters independently and averages their output probabilities, capturing epistemic diversity at a fraction of the cost of full-model ensembles. Second, Bayesian LoRA places a zero-mean Gaussian prior on the adapter parameters, finds the MAP estimate, approximates the posterior as $N(\\theta_{\\mathrm{MAP}}, H^{-1})$ using a Laplace expansion with a Kronecker-factored (K-FAC) Fisher information matrix, and then linearizes the model at the MAP so that the predictive distribution over logits is Gaussian and can be sampled by Cholesky decomposition. This machinery is what lets the paper attach calibrated confidence to each yes/no prediction.","core_discovery":"The central claim, stated in the paper's own terms, is that 'incorporating UQ strategies not only enhances PPI prediction accuracy but also yields better-calibrated confidence measures.' Concretely, on the ND-PPI, M-PPI, and C-PPI datasets with LLaMA-3-8B and BioMedGPT-LM-7B backbones, the LoRA ensemble attains the highest accuracy, F1, MCC, and AUROC in most configurations, and Bayesian LoRA consistently delivers the lowest expected calibration error in the majority of settings (e.g., ECE as low as 0.027 on C-PPI with LLaMA-3). The paper presents this as evidence that principled uncertainty quantification can be integrated into parameter-efficient fine-tuning without sacrificing predictive performance.","pith_inferences":["The paper's train/test partition is described only as an 80/20 split with no statement of whether proteins are shared across the split; my inference is that a held-out-protein evaluation would be the decisive test of whether the accuracy gains reflect generalization to unseen proteins rather than memorization of protein identifiers.","The comparison baseline is a single deterministic LoRA; the reported advantage over that baseline does not establish superiority over other uncertainty-estimation strategies, so a broader benchmark would be needed before adopting this as the default.","The negative interaction pairs are pairs with no documented evidence of interaction, which makes the classification task easier than real discovery settings where absence of evidence is noisy; under noisier labels, the calibration gains may be smaller, which is a testable prediction.","If the calibrated confidence truly tracks correctness, then a practical use is to rank protein pairs by uncertainty and send only the low-confidence tail to experimental validation, which the paper suggests but does not demonstrate."],"forward_implications":["In most experimental configurations, replacing a single LoRA with either a LoRA ensemble or Bayesian LoRA improves or matches accuracy while reducing negative log-likelihood and calibration error, so the uncertainty-aware variants can serve as drop-in replacements in the same fine-tuning pipeline.","The calibration gains are consistent across two different LLM backbones (LLaMA-3-8B and BioMedGPT-LM-7B), indicating the benefit is tied to the uncertainty mechanism rather than to a specific pretrained model.","Reliability diagrams show a visibly reduced gap between confidence and observed accuracy for both uncertainty-aware methods relative to the single LoRA, supporting the claim that these methods mitigate the overconfidence problem.","Because the adapters are lightweight and the backbone is frozen, the added uncertainty estimation keeps computational overhead close to standard LoRA training, which matters for scaling to larger models.","The approach can be extended to other biomedical tasks that use language-like protein representations, including agentic AI workflows and prompt optimization, as the authors note."],"supporting_citations":[{"why":"Defines low-rank adaptation (LoRA), the base fine-tuning method that all variants build on.","marker":"[43]"},{"why":"Supplies the Bayesian LoRA recipe: Laplace approximation, K-FAC Fisher information, and linearized predictive posterior.","marker":"[44]"},{"why":"Introduces LoRA ensembles as an efficient way to quantify uncertainty by training multiple low-rank adapters.","marker":"[48]"},{"why":"Provides the disease-specific PPI datasets and the experimental protocol (batch sizes, binary classification framing) that the paper follows.","marker":"[29]"},{"why":"Source of the ND-PPI and M-PPI dataset construction via symmetric logistic matrix factorization.","marker":"[53]"},{"why":"Source of the C-PPI dataset and the network-based PPI prediction framework with the 80/20 split convention.","marker":"[54]"},{"why":"The LLaMA-3-8B backbone used in the experiments.","marker":"[17]"},{"why":"The BioMedGPT-LM-7B backbone used in the experiments.","marker":"[28]"}],"fun_headline_variants":["Bayesian LoRA boosts LLM protein-interaction trust","LoRA ensembles calibrate LLM PPI predictions","Uncertainty-aware fine-tuning sharpens PPI forecasts","Confidence-aware LLMs improve protein interaction calls"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the 80/20 train/test split separates interactions without leaking the same proteins into both sets; the paper never states whether the split is by protein or by interaction, and if proteins overlap, accuracy gains could come from memorizing protein identities rather than from learning to generalize.","fun_headline_variants_meta":{"raw":{"variants":["Bayesian LoRA boosts LLM protein-interaction trust","LoRA ensembles calibrate LLM PPI predictions","Uncertainty-aware fine-tuning sharpens PPI forecasts","Confidence-aware LLMs improve protein interaction calls"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000151,"raw_usage":{"total_tokens":1161,"prompt_tokens":864,"completion_tokens":297,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":480,"completion_tokens_details":{"reasoning_tokens":233}},"tokens_in":480,"tokens_out":297,"duration_ms":3118,"temperature":1.0,"reasoning_tokens":233,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T16:29:20.992423+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the three PPI experiments with an 80/20 split that holds out entire proteins, so that no protein appearing in the training set appears in the test set; if the accuracy and calibration advantages of the uncertainty-aware variants shrink or vanish, the original results partly reflect protein-level memorization rather than robust interaction prediction.","supporting_citations":[{"cited_title":"LoRA: Low-rank adaptation of large language models,","cited_arxiv_id":null,"evidence_quote":"Defines low-rank adaptation (LoRA), the base fine-tuning method that all variants build on."},{"cited_title":"Bayesian low-rank adaptation for large language models,","cited_arxiv_id":null,"evidence_quote":"Supplies the Bayesian LoRA recipe: Laplace approximation, K-FAC Fisher information, and linearized predictive posterior."},{"cited_title":"Evaluating large language models for predicting protein behavior under radiation exposure and disease conditions,","cited_arxiv_id":null,"evidence_quote":"Provides the disease-specific PPI datasets and the experimental protocol (batch sizes, binary classification framing) that the paper follows."},{"cited_title":"Predicting protein–protein interactions using symmetric logistic matrix factorization,","cited_arxiv_id":null,"evidence_quote":"Source of the ND-PPI and M-PPI dataset construction via symmetric logistic matrix factorization."},{"cited_title":"Network-based protein- protein interaction prediction method maps perturbations of cancer interactome,","cited_arxiv_id":null,"evidence_quote":"Source of the C-PPI dataset and the network-based PPI prediction framework with the 80/20 split convention."}],"review_version":1}