Pith. sign in

REVIEW 3 major objections 5 minor 81 references

Temporal Distribution Shift in Real-World Pharmaceutical Data: Implications for Uncertainty Quantification in QSAR Models

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Temporal shifts in real pharmaceutical assay data degrade uncertainty estimates, most severely for target-based screens, so i.i.d. evaluation overstates the reliability of QSAR uncertainty quantification.

desk verdict A useful, well-executed empirical benchmark of UQ under temporal shift in QSAR, but the headline causal claim about shift magnitude is confounded with assay category, size, and imbalance. read the letter →

arxiv 2502.03982 v1 pith:2HM7PNQ7 submitted 2025-02-06 cs.LG

Hannah Rosa Friesacher , Emma Svensson , Susanne Winiwarter , Lewis Mervin , Adam Arany , Ola Engkvist This is my paper · ORCID
classification cs.LG
keywords temporaldistributionshiftuncertaintyquantificationQSARprobabilitycalibrationlabeldescriptordeepensemblespharmaceuticaldata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

QSAR models are routinely evaluated by random or chemical splits, which assume the future looks like the past. This paper trains classifiers on 15 real pharmaceutical assays split by measurement date and finds that the assumption fails: target-based (TB) project assays show large shifts over time in both the fraction of active compounds and the chemical space explored, while ADME-T property assays stay comparatively stable. It then shows that when the shift is large, common uncertainty quantification methods—deep ensembles, MC dropout, Bayesian neural networks, and post hoc calibration—stop delivering calibrated probabilities, sometimes no better than a plain neural network. The work matters because it means i.i.d. benchmarking of uncertainty estimates overstates their reliability exactly in the settings where drug-discovery decisions are riskiest.

What carries the argument

The temporal split is the central object: each assay's measurements are ordered by date and divided into five roughly equal time spans, with one, two, or three spans used for training, the next for validation and calibration, and the first following span for testing. Against this split the paper measures label shift as the difference in preferred-class ratios and descriptor shift as the maximum mean discrepancy with a Tanimoto kernel on 4096-bit ECFP fingerprints, then compares RF, MLP, deep ensembles (MLPE), MC dropout (MLPMC), Bayes-by-Backprop BNN, Platt scaling, and Venn-ABERS predictors on AUC, binary cross-entropy, and adaptive calibration error. This machinery isolates time as the only variable and lets the authors tie uncertainty quality to measured shift magnitude.

What would settle it

Take a stable ADME-T assay and reweight its time spans to reproduce the label-shift trajectory of TB-7 while holding dataset size and class balance fixed; if deep ensembles then still match the plain MLP in calibration, shift magnitude is confirmed as the driver, whereas if they keep their calibration advantage the paper's causal story is refuted.

Watch

Extended reading notes

Core claim

The central claim is that temporal distribution shift in real-world pharmaceutical data is assay-dependent and that this shift, not model architecture alone, determines whether uncertainty quantification methods improve on a baseline classifier. In the seven target-based assays, the preferred-class ratio swings from span to span (for example TB-1 drops from 0.68 in the first span to 0.24 in the fourth, and TB-7 falls to 0.08 in the fifth), and maximum mean discrepancy between training and test ECFP fingerprints is larger than in the eight ADME-T assays. Under these shifts, deep ensembles and Bayesian neural networks, which are the best-calibrated methods on the stable ADME-T assays, rarely beat the plain MLP on TB assays; post hoc calibration with Platt scaling or Venn-ABERS improves calibration only when the calibration-to-test MMD is small. The paper concludes that the i.i.d. assumption is questionable for target-specific assays and that recommended practice should account for the magnitude and nature of assay-specific shifts.

Load-bearing premise

The results assume the worse uncertainty quality on target-based assays is caused by the larger distribution shift, not by confounds that happen to align with the target-based versus ADME-T split, such as smaller dataset size, more extreme class imbalance, and noisier labels.

Editorial extensions

If this is right

  • Uncertainty estimates from popular methods should not be trusted as reported on target-based assays, because a plain MLP can match or beat them under large temporal shift.
  • Model performance at one point in an assay's history is a poor guide to later performance, so models must be re-evaluated when new temporal data arrives.
  • Post hoc calibration methods such as Platt scaling and Venn-ABERS should be applied only when the calibration set is close in descriptor space to the test set, since under large MMD they can leave calibration unchanged or make it worse.
  • For stable ADME-T assays, deep ensembles and Bayesian neural networks give well-calibrated probabilities, with the BNN preferable if compute is limited.
  • The label-space shift, rather than descriptor shift alone, is the likely cause of uncertainty quantification failure in TB assays, because classification thresholds discard the magnitude information that regression models retain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If label shift is the dominant failure mode, then label-shift correction, such as reweighting the validation distribution to match the test period, is a cheap experiment the paper does not run; on TB assays it should restore the calibration advantage of deep ensembles over the MLP.
  • The TB versus ADME-T contrast is entangled with assay size, class imbalance, and measurement noise; matching a stable ADME-T assay to a TB assay on size and imbalance while inducing an artificial temporal label shift would separate shift magnitude from these confounds.
  • Practitioners evaluating uncertainty quantification methods should report temporal MMD and preferred-class-ratio trajectories as standard metadata, since the paper shows these predict when calibration gains will materialize.
  • A natural extension is to test distribution-free conformal prediction under the same temporal splits, using the paper's shift metrics as a difficulty score for selecting a conformal strategy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This manuscript uses 15 internal pharmaceutical assays to study temporal distribution shifts and their effect on uncertainty quantification for QSAR classification. The authors split each assay into five time spans, train on one to three spans, validate on the next, and test on the following span. They quantify label shift by the difference in preferred-class ratios and descriptor shift by maximum mean discrepancy on ECFP fingerprints, then compare RF, MLP, deep ensembles, MC dropout, and Bayes-by-backprop, with and without Platt/Venn-ABERS calibration, using AUC, BCE, and ACE. The reported results show larger label and descriptor shifts for target-based assays and a tendency for uncertainty methods to improve calibration on ADME-T but not target-based assays.

Significance. The study's main asset is its realistic and industrially grounded evaluation: true measurement dates, assay-specific thresholds, per-assay hyperparameter selection on validation data, ten model repetitions, and public code. If the causal claim were established, the paper would be an important caution that i.i.d. evaluation overstates UQ reliability for target-based assays. As it stands, the descriptive shift analysis is valuable, but the central causal conclusion is entangled with assay-size and class-imbalance confounds and lacks formal trend tests; the manuscript needs additional analyses before the claim can be accepted.

major comments (3)
  1. [Abstract and §3.2 (Table 1)] The abstract's claim that 'pronounced distribution shifts impair the performance of popular uncertainty estimation methods' is supported only by the TB-vs-ADME-T contrast. The two categories differ systematically in assay size (Table 1: TB 5k–14k vs ADME-T 12k–92k), class balance (PC ratios 0.16–0.82 for TB vs 0.15–0.84 for ADME-T), and likely label noise, and §3.2 itself offers small dataset size as an alternative explanation for TB-1/TB-2. Because the shift variable is collinear with category and dataset size, the observed performance pattern cannot be attributed to shift magnitude without covariate-adjusted or matched analysis. Please add, at minimum, a within-category analysis of performance gains versus shift metrics and a regression or matched-assay check that includes dataset size and class balance as covariates, and soften the abstract and conclusion causality accordingly.
  2. [Figure 8 and §3.2 (Post hoc Probability Calibration)] The claimed decrease in post hoc calibration quality with increasing MMD between calibration and test set is not verifiable from the manuscript. The text says 'The MMD between the calibration and test dataset is reported in Table 5', but Table 5 reports the difference in preferred-class ratios (label shift), not MMD; the MMD values used to order the assays in Figure 8 are not reported. No trend test or correlation coefficient is provided. Please report the MMD point estimates (ideally with bootstrap intervals), add a quantitative association test (e.g., Spearman rank correlation between MMD and calibration gain), and correct the table reference and the figure caption (which says 'calibration and the training set').
  3. [§2.3, §3.1, and statistical reporting] The paper states 'significant shifts over time' and 'a clear connection' but does not report tests for the shift metrics: Figure 4 shows means without confidence intervals, MMD is a single point estimate per assay/setting, and only one test fold is used per temporal setting. In addition, the model comparisons use two-sided t-tests over 45 assay-settings without any multiplicity correction. Please add confidence intervals (or bootstrap resampling of MMD and label shifts), state the single-fold limitation explicitly, and either apply a multiple-testing correction or label the t-test results as exploratory.
minor comments (5)
  1. [§3.2, text near Figure 8] The text refers to MLPE models calibrated with Venn-ABERS predictors as 'MLP-V A'; this should be 'MLPE-V A'.
  2. [Figure 8] The x-axis label reads 'MDD' in the figure panel; it should read 'MMD'.
  3. [§2.3, Eq. (5)] The statement that Tanimoto-kernel MMD lies between 0 and 1 requires the kernel to be normalized; please give the exact kernel definition and clarify the range condition.
  4. [Table 5] The table title says 'label distribution shifts between the calibration and test set' but the columns contain differences in preferred-class ratios; consider renaming the table to 'label shift (difference in preferred-class ratios)' to avoid confusion with the MMD-based descriptor shift.
  5. [Figure 7 caption] The caption repeats 'trained with compounds from three time spans' although the figure covers all three temporal settings; it should say 'one, two, or three time spans'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: shift metrics are descriptive statistics, model performance is evaluated on held-out temporal folds, and no fitted parameter is relabeled as a prediction.

full rationale

The paper's central claims are empirical rather than derivational. Distribution shifts are quantified directly from the data using preferred-class ratio differences and MMD computed between training and test sets, while uncertainty quantification performance is measured by AUC, BCE, and ACE on a temporally held-out test fold. Hyperparameters are tuned on a separate validation fold using BCE loss, which is standard experimental practice and does not force the reported test-set results. The conclusion that pronounced distribution shifts impair uncertainty estimation is an observed contrast between Target-Based and ADME-T assays, not an identity or a fitted result; the paper itself acknowledges confounds such as small dataset size in Section 3.2. Citations of prior work by the same authors (references [49], [50], and [51]) are used to establish that the internal dataset has been studied before and to compare with a regression study, but they do not supply a uniqueness theorem, a fitted parameter later called a prediction, or an ansatz that is smuggled in as an external result. The Figure 8 versus Table 5 mismatch (MMD ordering versus a table captioned as label shift) is a reporting inconsistency and a correctness risk, but it is not a reduction of the conclusion to its inputs. The paper is self-contained as an empirical benchmark comparison, so no circular step is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on hand-set assay thresholds (which define label shift), grid-searched hyperparameters and the fixed BNN prior (which shape model performance), and domain assumptions about measurement dates, the MMD metric, and the single-test-fold design. No new entities are postulated; all results are empirical measurements on proprietary data.

free parameters (4)
  • Assay-specific activity thresholds T = TB: pIC50/pEC50 > 6; CYP/hERG: pIC50 < 5; A-PM: logP > 1; A-SOL: logS > 2; A-logD: logD > 3; A-MS: logMS < 1
    Table 1 defines these hand-set thresholds that convert continuous measurements to binary preferred-class labels. They directly determine the label-space shift ratios and all downstream calibration results.
  • MLP and RF hyperparameters (grid-searched per assay and temporal setting) = Explored ranges in Table 3: RF estimators 50-1500, max depth 5-10000; MLP weight decay 0-0.0005, dropout 0-0.75…
    Section 2.2 and Table 3: hyperparameters are tuned for each of the 45 assay-settings using validation BCE, so the reported test results depend on these per-setting choices.
  • BNN Gaussian prior variance = Fixed, not tuned
    Section 2.2 states a fixed variance was chosen for the Gaussian prior; the discussion in Section 3.2 notes tuning this hyperparameter could improve BNN performance.
  • Number of MC dropout forward passes = 400
    Section 2.2: 400 forward passes are aggregated for MC dropout, an ad hoc choice that affects the variance estimates.
assumptions (4)
  • domain assumption Measurement dates are a valid temporal ordering for splitting assay data
    Section 2.1 'Temporal Split' divides each assay into five folds using the date of each measurement. If timestamps do not reflect true experiment dates or include re-tests, the shift measurements are distorted.
  • domain assumption Tanimoto-kernel MMD is a reliable distance between ECFP fingerprint distributions
    Section 2.3 Eq. (5) uses MMD with the Tanimoto kernel to quantify descriptor shift; the TB vs ADME-T shift conclusions rely on this metric being a meaningful distance in chemical space.
  • domain assumption Equal-size folds from temporal ordering provide a realistic train/validation/test regime
    Section 2.1 and Figure 2: using one subsequent fold for validation and the next for test, with only one test fold per setting, is a design choice not derived from deployment practice.
  • standard math Independent t-tests across 10 repetitions approximate significance for model comparisons
    Section 2.3 uses a two-sided independent t-test per assay and temporal setting; normality and independence are assumed, and no multiple-comparison correction is applied, so some bolded or asterisked 'best' flags may be false positives.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Temporal Distribution Shift in Real-World Pharmaceutical Data: Implications for Uncertainty Quantification in QSAR Models." pith.science (2026). https://pith.science/paper/2HM7PNQ7

@misc{pith2026250203982,
  author       = {Pith},
  title        = {Pith review of: Temporal Distribution Shift in Real-World Pharmaceutical Data: Implications for Uncertainty Quantification in QSAR Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2HM7PNQ7}},
  note         = {Machine review of arXiv:2502.03982}
}
read the original abstract

The estimation of uncertainties associated with predictions from quantitative structure-activity relationship (QSAR) models can accelerate the drug discovery process by identifying promising experiments and allowing an efficient allocation of resources. Several computational tools exist that estimate the predictive uncertainty in machine learning models. However, deviations from the i.i.d. setting have been shown to impair the performance of these uncertainty quantification methods. We use a real-world pharmaceutical dataset to address the pressing need for a comprehensive, large-scale evaluation of uncertainty estimation methods in the context of realistic distribution shifts over time. We investigate the performance of several uncertainty estimation methods, including ensemble-based and Bayesian approaches. Furthermore, we use this real-world setting to systematically assess the distribution shifts in label and descriptor space and their impact on the capability of the uncertainty estimation methods. Our study reveals significant shifts over time in both label and descriptor space and a clear connection between the magnitude of the shift and the nature of the assay. Moreover, we show that pronounced distribution shifts impair the performance of popular uncertainty estimation methods used in QSAR models. This work highlights the challenges of identifying uncertainty quantification methods that remain reliable under distribution shifts introduced by real-world data.

Figures

Figures reproduced from arXiv: 2502.03982 by the authors.

Figure 1
Figure 1. In addition, Table 1 provides detailed information on the assays used in this study. The assays were assigned [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 1
Figure 1. Overview of dataset sizes. The left panel plots the size of the individual assays ordered according to assay size. The striped areas in the bar indicate the amount of compounds belonging to the preferred class (PC) in each assay. The right panel shows the amount of training data in each temporal setting across all assays, with 1, 2, or 3 time spans used for training [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Overview of the temporal split and model training. The data in each assay was assigned to 5 time spans to create three temporal settings, each with increasing amounts of training (Training) data. The subsequent two folds were used for validation (Valid.) and testing (Test). The validation data also served as a calibration set used in post hoc calibration approaches. 2.2 Models [PITH_FULL_IMAGE:figures/full_fig_p006… view at source ↗
Figures from the paper (8 more)
Figure 3
Figure 3. Figure 3: provides an overview of the models compared in this study. All architectures used in this work stem from a Random Forest (RF) or a multilayer perceptron (MLP). Both approaches are commonly used in research addressing uncertainty estimation in QSAR modeling [35, 36, 30]…
Figure 4
Figure 4. Figure 4: Quantification of the distribution shifts between the training and test datasets over time. The shift in label space and in the descriptor space is illustrated for each temporal setting, using the data of 1, 2, or 3 time spans for training. Results are shown for each a…
Figure 5
Figure 5. Figure 5: T-SNE plots of the ECFP space. T-SNE plots of the ECFP space are shown for one example of each assay category to illustrate how the explored chemical space changes over time. Compounds are colored according to the time span that they were assigned to. The t-SNE plot of…
Figure 6
Figure 6. Figure 6: Summary of BCE and ACE scores for the third temporal setting. The first column shows the results for TB assays, while the second one reports the performance of models trained on ADME-T assays. BCE scores are plotted in the first, and ACE scores in the second row. Resul…
Figure 7
Figure 7. Figure 7: Summary of BCE and ACE scores across all temporal settings. The first two columns show the ACE and BCE scores for TB assays, while the last two report the performance of models trained on ADME-T assays. The temporal setting is indicated in brackets after the assay abbr…
Figure 8
Figure 8. Figure 8: Summary of ACE scores of post-hoc probability calibration approaches using the third temporal setting. The left panel shows the ACE scores of models trained on TB assays, while the right panel reports the ACE performance of ADME-T models. The assays in each panel are o…
Figure 9
Figure 9. Figure 9: T-SNE plots of the ECFP space for all TB assays. T-SNE plots of the ECFP space are shown for all TB assays. Compounds are colored according to the time span that they were assigned to [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: T-SNE plots of the ECFP space for all ADME-T assays. T-SNE plots of the ECFP space are shown for all ADME-T assays. Compounds are colored according to the time span that they were assigned to. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

81 extracted references · 64 canonical work pages

  1. [1]

    Drug discovery and development: introduction to the general public and patient groups

    Natesh Singh, Philippe Vayer, Shivalika Tanwar, Jean-Luc Poyet, Katya Tsaioun, and Bruno O Villoutreix. Drug discovery and development: introduction to the general public and patient groups. Frontiers in Drug Discovery, 3: 1201419, 2023

    2023

  2. [2]

    Innovation crisis in the pharmaceutical industry? a survey

    Ute Laermann-Nguyen and Martin Backfisch. Innovation crisis in the pharmaceutical industry? a survey. SN Business & Economics, 1(12):164, 2021

    work page 2021

  3. [3]

    Computational approaches streamlining drug discovery

    Anastasiia V Sadybekov and Vsevolod Katritch. Computational approaches streamlining drug discovery. Nature, 616(7958):673–685, 2023

    work page 2023

  4. [4]

    High-Throughput Screening: New Technology for the 21st Century

    Robert P Hertzberg and Andrew J Pope. High-Throughput Screening: New Technology for the 21st Century. Curr. Opin. Chem. Biol., 4(4):445–451, 2000

    work page 2000

  5. [5]

    Supervised Prediction of Drug–Target Interactions Using Bipartite Local Models

    Kevin Bleakley and Yoshihiro Yamanishi. Supervised Prediction of Drug–Target Interactions Using Bipartite Local Models. Bioinform., 25(18):2397–2403, 2009

    work page 2009

  6. [6]

    The Concept of Probability in Safety Assessments of Technological Systems

    George Apostolakis. The Concept of Probability in Safety Assessments of Technological Systems. Science, 250 (4986):1359–1364, 1990

    work page 1990

  7. [7]

    Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods

    Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Machine Learning, 110:457 – 506, 2019. doi:10.1007/s10994-021-05946-3

    doi:10.1007/s10994-021-05946-3 2019

  8. [8]

    Sources of uncertainty in machine learning – a statisticians’ view, 2023

    Cornelia Gruber, Patrick Oliver Schenk, Malte Schierholz, Frauke Kreuter, and Göran Kauermann. Sources of uncertainty in machine learning – a statisticians’ view, 2023

    work page 2023

Show all 81 references
  1. [9]

    Bayesian learning for neural networks, volume 118

    Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012

    2012

  2. [10]

    Weight uncertainty in neural network

    Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning, pages 1613–1622. PMLR, 2015

    2015

  3. [11]

    What are bayesian neural network posteriors really like? In International conference on machine learning, pages 4629–4640

    Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Gordon Wilson. What are bayesian neural network posteriors really like? In International conference on machine learning, pages 4629–4640. PMLR, 2021

    2021

  4. [12]

    Bayesian neural network with pretrained protein embedding enhances prediction accuracy of drug-protein interaction

    QHwan Kim, Joon-Hyuk Ko, Sunghoon Kim, Nojun Park, and Wonho Jhe. Bayesian neural network with pretrained protein embedding enhances prediction accuracy of drug-protein interaction. Bioinformatics, 37(20): 3428–3435, 2021

    2021

  5. [13]

    Simple and scalable predictive uncer- tainty estimation using deep ensembles

    Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncer- tainty estimation using deep ensembles. In I. Guyon, U. V on Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Informat...

    2017

  6. [14]

    Dropout as a bayesian approximation: Representing model uncertainty in deep learning

    Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of M...

    2016

  7. [15]

    A gentle introduction to conformal prediction and distribution-free uncertainty quantification

    Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021

    2021 arXiv

  8. [16]

    Mapie: an open-source library for distribution-free uncertainty quantification

    Vianney Taquet, Vincent Blot, Thomas Morzadec, Louis Lacombe, and Nicolas Brunel. Mapie: an open-source library for distribution-free uncertainty quantification. arXiv preprint arXiv:2207.12274, 2022

    2022 arXiv

  9. [17]

    Evidential deep learning to quantify classification uncertainty

    Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. Advances in neural information processing systems, 31, 2018

    2018

  10. [18]

    An uncertainty- guided deep learning method facilitates rapid screening of cyp3a4 inhibitors

    Ruixuan Wang, Zhikang Liu, Jiahao Gong, Qingping Zhou, Xiaoqing Guan, and Guangbo Ge. An uncertainty- guided deep learning method facilitates rapid screening of cyp3a4 inhibitors. Journal of Chemical Information and Modeling, 63(24):7699–7710, 2023

    2023

  11. [19]

    Learning with uncertainty to accelerate the discovery of histone lysine-specific demethylase 1a (kdm1a/lsd1) inhibitors

    Dong Wang, Zhenxing Wu, Chao Shen, Lingjie Bao, Hao Luo, Zhe Wang, Hucheng Yao, De-Xin Kong, Cheng Luo, and Tingjun Hou. Learning with uncertainty to accelerate the discovery of histone lysine-specific demethylase 1a (kdm1a/lsd1) inhibitors. Briefings in Bioinformatics, 24(1):...

    2023

  12. [20]

    Improving evidential deep learning via multi-task learning

    Dongpin Oh and Bonggun Shin. Improving evidential deep learning via multi-task learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 7895–7903, 2022. 16 Temporal Distribution Shift in Real-World Pharmaceutical Data: Implications for Uncer...

    2022

  13. [21]

    Evidential deep learning for guided molecular property prediction and discovery

    Ava P Soleimany, Alexander Amini, Samuel Goldman, Daniela Rus, Sangeeta N Bhatia, and Connor W Coley. Evidential deep learning for guided molecular property prediction and discovery. ACS central science, 7(8): 1356–1367, 2021

    2021

  14. [22]

    Simple and principled uncertainty estimation with deterministic deep learning via distance awareness

    Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in neural information processing systems, 33:7498–7512, 2020

    2020

  15. [23]

    Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods

    John Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Adv. Large Margin Classif., 10, 06 1999

    1999

  16. [24]

    Venn-abers predictors, 2014

    Vladimir V ovk and Ivan Petej. Venn-abers predictors, 2014

    2014

  17. [25]

    Transforming classifier scores into accurate multiclass probability estimates

    Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 694–699, 2002

    2002

  18. [26]

    Mervin, Simon Johansson, Elizaveta Semenova, Kathryn A

    Lewis H. Mervin, Simon Johansson, Elizaveta Semenova, Kathryn A. Giblin, and Ola Engkvist. Un- certainty quantification in drug design. Drug Discovery Today , 26(2):474–489, 2021. ISSN 1359-6446. doi:10.1016/j.drudis.2020.11.027

    2021 doi

  19. [27]

    Uncertainty quantification: Can we trust artificial intelligence in drug discovery? Iscience, 25(8), 2022

    Jie Yu, Dingyan Wang, and Mingyue Zheng. Uncertainty quantification: Can we trust artificial intelligence in drug discovery? Iscience, 25(8), 2022

    2022

  20. [28]

    A large-scale study of probabilistic calibration in neural network regression

    Victor Dheur and Souhaib Ben Taieb. A large-scale study of probabilistic calibration in neural network regression. In International Conference on Machine Learning, pages 7813–7836. PMLR, 2023

    2023

  21. [29]

    Quantifica- tion of uncertainty with adversarial models.Advances in Neural Information Processing Systems, 36:19446–19484, 2023

    Kajetan Schweighofer, Lukas Aichberger, Mykyta Ielanskyi, Günter Klambauer, and Sepp Hochreiter. Quantifica- tion of uncertainty with adversarial models.Advances in Neural Information Processing Systems, 36:19446–19484, 2023

    2023

  22. [30]

    Probabilistic random forest improves bioactivity predictions close to the classification threshold by taking into account experimental uncertainty

    Lewis H Mervin, Maria-Anna Trapotsi, Avid M Afzal, Ian P Barrett, Andreas Bender, and Ola Engkvist. Probabilistic random forest improves bioactivity predictions close to the classification threshold by taking into account experimental uncertainty. Journal of Cheminformatics, 1...

    2021

  23. [31]

    An ensemble-based approach to estimate confidence of predicted protein–ligand binding affinity values

    Milad Rayka, Morteza Mirzaei, and Ali Mohammad Latifi. An ensemble-based approach to estimate confidence of predicted protein–ligand binding affinity values. Molecular Informatics, 43(4):e202300292, 2024

    2024

  24. [32]

    Reducing overconfident errors in molecular property classification using posterior network

    Zhehuan Fan, Jie Yu, Xiang Zhang, Yijie Chen, Shihui Sun, Yuanyuan Zhang, Mingan Chen, Fu Xiao, Wenyong Wu, Xutong Li, et al. Reducing overconfident errors in molecular property classification using posterior network. Patterns, 2024

    2024

  25. [33]

    Achieving well-informed decision-making in drug discovery: A comprehensive calibration study using neural network-based structure- activity models

    Hannah Rosa Friesacher, Ola Engkvist, Lewis Mervin, Yves Moreau, and Adam Arany. Achieving well-informed decision-making in drug discovery: A comprehensive calibration study using neural network-based structure- activity models. arXiv preprint arXiv:2407.14185, 2024

    2024 arXiv

  26. [34]

    Uncertainty quantification using neural networks for molecular property prediction

    Lior Hirschfeld, Kyle Swanson, Kevin Yang, Regina Barzilay, and Connor W Coley. Uncertainty quantification using neural networks for molecular property prediction. Journal of Chemical Information and Modeling, 60(8): 3770–3780, 2020

    2020

  27. [35]

    Mervin, Avid M

    Lewis H. Mervin, Avid M. Afzal, Ola Engkvist, and Andreas Bender. Comparison of scaling methods to obtain calibrated probabilities of activity for protein–ligand predictions. Journal of Chemical Information and Modeling, 60(10):4546–4559, 2020. doi:10.1021/acs.jcim.0c00476. PM...

    2020 doi

  28. [36]

    Large-scale evaluation of k-fold cross-validation ensembles for uncertainty estimation

    Thomas-Martin Dutschmann, Lennart Kinzel, Antonius Ter Laak, and Knut Baumann. Large-scale evaluation of k-fold cross-validation ensembles for uncertainty estimation. Journal of Cheminformatics, 15(1):49, 2023

    2023

  29. [37]

    Time-split cross-validation as a method for estimating the goodness of prospective prediction

    Robert P Sheridan. Time-split cross-validation as a method for estimating the goodness of prospective prediction. J. Chem. Inf. Model., 53(4):783–790, 2013. doi:10.1021/ci400084k

    2013 doi

  30. [38]

    Simpd: an algorithm for generating simulated time splits for validating machine learning approaches

    Gregory A Landrum, Maximilian Beckers, Jessica Lanini, Nadine Schneider, Nikolaus Stiefl, and Sereina Riniker. Simpd: an algorithm for generating simulated time splits for validating machine learning approaches. Journal of cheminformatics, 15(1):119, 2023

    2023

  31. [39]

    Learning classifiers when the training data is not iid

    Murat Dundar, Balaji Krishnapuram, Jinbo Bi, and R Bharat Rao. Learning classifiers when the training data is not iid. In IJCAI, volume 2007, pages 756–61. Citeseer, 2007

    2007

  32. [40]

    Beyond iid: Non-iid thinking, informatics, and learning

    Longbing Cao. Beyond iid: Non-iid thinking, informatics, and learning. IEEE Intelligent Systems, 37(4):5–17, 2022. 17 Temporal Distribution Shift in Real-World Pharmaceutical Data: Implications for Uncertainty Quantification in QSAR Models

    2022

  33. [41]

    Sculley, Sebastian Nowozin, Joshua Dillon, Bal- aji Lakshminarayanan, and Jasper Snoek

    Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Bal- aji Lakshminarayanan, and Jasper Snoek. Can you trust your model 's uncertainty? evaluating predic- tive uncertainty under dataset shift. In H. Wallach, H. Larochelle, A. Beyg...

    2019

  34. [42]

    Wilds: A benchmark of in-the-wild distribution shifts

    Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International conference on machine learning, p...

    2021

  35. [43]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh, editors,Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 132...

    2017

  36. [44]

    Three Useful Dimensions for Domain Applicability in QSAR Models Using Random Forest

    Robert P Sheridan. Three Useful Dimensions for Domain Applicability in QSAR Models Using Random Forest. J. Chem. Inf. Model., 52(3):814–823, 2012

    2012

  37. [45]

    Predicting good probabilities with supervised learning

    Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference on Machine learning, pages 625–632, 2005

    2005

  38. [46]

    The chembl database in 2023: a drug discovery platform spanning multiple bioactivity data types and time periods.Nucleic acids research, 52(D1):D1180–D1192, 2024

    Barbara Zdrazil, Eloy Felix, Fiona Hunter, Emma J Manners, James Blackshaw, Sybilla Corbett, Marleen de Veij, Harris Ioannidis, David Mendez Lopez, Juan F Mosquera, et al. The chembl database in 2023: a drug discovery platform spanning multiple bioactivity data types and time ...

    2023

  39. [47]

    Multispecies machine learning predictions of in vitro intrinsic clearance with uncertainty quantification analyses

    Raquel Rodríguez-Pérez, Markus Trunzer, Nadine Schneider, Bernard Faller, and Gregori Gerebtzoff. Multispecies machine learning predictions of in vitro intrinsic clearance with uncertainty quantification analyses. Molecular Pharmaceutics, 20(1):383–394, 2022

    2022

  40. [48]

    Computational predictions of nonclinical pharmacokinetics at the drug design stage

    Raya Stoyanova, Paul Maximilian Katzberger, Leonid Komissarov, Aous Khadhraoui, Lisa Sach-Peltason, Katrin Groebke Zbinden, Torsten Schindler, and Nenad Manevski. Computational predictions of nonclinical pharmacokinetics at the drug design stage. Journal of Chemical Informatio...

    2023

  41. [49]

    Enhancing uncertainty quantification in drug discovery with censored regression labels

    Emma Svensson, Hannah Rosa Friesacher, Susanne Winiwarter, Lewis Mervin, Adam Arany, and Ola En- gkvist. Enhancing uncertainty quantification in drug discovery with censored regression labels. arXiv preprint arXiv:2409.04313, 2024

    2024 arXiv

  42. [50]

    Towards reliable uncertainty estimates for drug discovery: A large-scale temporal study of probability calibration

    Hannah Rosa Friesacher, Emma Svensson, Adam Arany, Lewis Mervin, and Ola Engkvist. Towards reliable uncertainty estimates for drug discovery: A large-scale temporal study of probability calibration. In ICML 2024 AI for Science Workshop, 2024

    2024

  43. [51]

    Temporal evaluation of probability calibration with experimental errors

    Hannah Rosa Friesacher, Emma Svensson, Adam Arany, Lewis Mervin, and Ola Engkvist. Temporal evaluation of probability calibration with experimental errors. In International Workshop on AI in Drug Discovery, pages 13–20. Springer, 2024

    2024

  44. [52]

    Risks in new drug development: approval success rates for investigational drugs

    Joseph A DiMasi. Risks in new drug development: approval success rates for investigational drugs. Clinical Pharmacology & Therapeutics, 69(5):297–307, 2001

    2001

  45. [53]

    Admet in silico modelling: towards prediction paradise? Nature reviews Drug discovery, 2(3):192–204, 2003

    Han Van De Waterbeemd and Eric Gifford. Admet in silico modelling: towards prediction paradise? Nature reviews Drug discovery, 2(3):192–204, 2003

    2003

  46. [54]

    Mechanisms of cyp450 inhibition: understanding drug-drug interactions due to mechanism- based inhibition in clinical practice

    Malavika Deodhar, Sweilem B Al Rihani, Meghan J Arwood, Lucy Darakjian, Pamela Dow, Jacques Turgeon, and Veronique Michaud. Mechanisms of cyp450 inhibition: understanding drug-drug interactions due to mechanism- based inhibition in clinical practice. Pharmaceutics, 12(9):846, 2020

    2020

  47. [55]

    Cytochromes P450: metabolic and toxicological aspects

    Costas Ioannides. Cytochromes P450: metabolic and toxicological aspects. Crc Press, 1996

    1996

  48. [56]

    Cytochrome p450 enzymes in drug metabolism and chemical toxicology: An introduction

    Laura Lowe Furge and F Peter Guengerich. Cytochrome p450 enzymes in drug metabolism and chemical toxicology: An introduction. Biochemistry and Molecular Biology Education, 34(2):66–74, 2006

    2006

  49. [57]

    Role of caco-2 cell monolayers in prediction of intestinal drug absorption

    Pranav Shah, Viral Jogani, Tamishraha Bagchi, and Ambikanandan Misra. Role of caco-2 cell monolayers in prediction of intestinal drug absorption. Biotechnology progress, 22(1):186–198, 2006

    2006

  50. [58]

    Bridging solubility between drug discovery and development

    Li Di, Paul V Fish, and Takashi Mano. Bridging solubility between drug discovery and development. Drug discovery today, 17(9-10):486–495, 2012

    2012

  51. [59]

    Molecular genetic insights into cardiovascular disease

    Mark T Keating and Michael C Sanguinetti. Molecular genetic insights into cardiovascular disease. Science, 272 (5262):681–685, 1996

    1996

  52. [60]

    Lipophilicity in drug discovery

    Michael J Waring. Lipophilicity in drug discovery. Expert Opinion on Drug Discovery, 5(3):235–248, 2010. 18 Temporal Distribution Shift in Real-World Pharmaceutical Data: Implications for Uncertainty Quantification in QSAR Models

    2010

  53. [61]

    High lipophilicity and high daily dose of oral medications are associated with significant risk for drug-induced liver injury

    Minjun Chen, Jürgen Borlak, and Weida Tong. High lipophilicity and high daily dose of oral medications are associated with significant risk for drug-induced liver injury. Hepatology, 58(1):388–396, 2013

    2013

  54. [62]

    In vitro high throughput screening of compounds for favorable metabolic properties in drug discovery

    Collen M Masimirembwa, Richard Thompson, and Tommy B Andersson. In vitro high throughput screening of compounds for favorable metabolic properties in drug discovery. Combinatorial chemistry & high throughput screening, 4(3):245–263, 2001

    2001

  55. [63]

    Optimization of a higher throughput microsomal stability screening assay for profiling drug discovery candidates

    Li Di, Edward H Kerns, Yan Hong, Teresa A Kleintop, Oliver J Mc Connell, and Donna M Huryn. Optimization of a higher throughput microsomal stability screening assay for profiling drug discovery candidates. SLAS Discovery, 8(4):453–462, 2003

    2003

  56. [64]

    Rdkit: Open-source cheminformatics, 2006

    Greg Landrum. Rdkit: Open-source cheminformatics, 2006

    2006

  57. [65]

    SMILES, a Chemical Language and Information System

    David Weininger. SMILES, a Chemical Language and Information System. 1. Introduction to Methodology and Encoding Rules. J. Chem. Inf. Comput. Sci., 28(1):31–36, 1988

    1988

  58. [66]

    Registries in Machine Learning-Based Drug Discovery: A Shortcut to Code Reuse

    Peter B R Hartog, Emma Svensson, Lewis Mervin, Samuel Genheden, Ola Engkvist, and Igor V Tetko. Registries in Machine Learning-Based Drug Discovery: A Shortcut to Code Reuse. In International Workshop on AI in Drug Discovery, pages 98–115. Springer, 2024. URL https://doi.org/1...

    2024 doi

  59. [67]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine Learning in Python. Journal of Machine Learni...

    2011

  60. [68]

    PyTorch: An Imperative Style, High-Performance Deep Learning Library

    Adam Paszke et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019

    2019

  61. [69]

    Adam: A Method for Stochastic Optimization

    Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations, 2015

    2015

  62. [70]

    On information and sufficiency

    Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79–86, 1951

    1951

  63. [71]

    Excape wp1-probabilistic prediction, 2016

    Paolo Toccaceli, Ilia Nouretdinov, Zhiyuan Luo, Vladimir V ovk, Lars Carlsson, and Alex Gammerman. Excape wp1-probabilistic prediction, 2016

    2016

  64. [72]

    A kernel two-sample test

    Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723–773, 2012

    2012

  65. [73]

    Grouping of coefficients for the calculation of inter-molecular similarity and dissimilarity using 2d fragment bit-strings

    John D Holliday, CY Hu, and Peter Willett. Grouping of coefficients for the calculation of inter-molecular similarity and dissimilarity using 2d fragment bit-strings. Combinatorial chemistry & high throughput screening, 5(2):155–166, 2002

    2002

  66. [74]

    The meaning and use of the area under a receiver operating characteristic (roc) curve

    James A Hanley and Barbara J McNeil. The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology, 143(1):29–36, 1982

    1982

  67. [75]

    Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran

    Jeremy Nixon, Michael W. Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measuring calibration in deep learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019. doi:10.48550/arXiv.1904.01685

    2019 doi arXiv

  68. [76]

    Strictly proper scoring rules, prediction, and estimation

    Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association , 102(477):359–378, 2007. doi:10.1198/016214506000001437. URL https://doi.org/10.1198/016214506000001437

    2007 doi

  69. [77]

    Reliability, sufficiency, and the decomposition of proper scores

    Jochen Bröcker. Reliability, sufficiency, and the decomposition of proper scores. Quarterly Journal of the Royal Meteorological Society: A journal of the atmospheric sciences, applied meteorology and physical oceanography, 135(643):1512–1519, 2009

    2009

  70. [78]

    A unified view of label shift estimation

    Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary Lipton. A unified view of label shift estimation. Advances in Neural Information Processing Systems, 33:3290–3300, 2020

    2020

  71. [79]

    Evaluating scalable bayesian deep learning methods for robust computer vision

    Fredrik K Gustafsson, Martin Danelljan, and Thomas B Schon. Evaluating scalable bayesian deep learning methods for robust computer vision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 318–319, 2020

    2020

  72. [80]

    Benchmarking common uncertainty estimation methods with histopathological images under domain shift and label noise

    Hendrik A Mehrtens, Alexander Kurz, Tabea-Clara Bucher, and Titus J Brinker. Benchmarking common uncertainty estimation methods with histopathological images under domain shift and label noise. Medical image analysis, 89:102914, 2023

    2023

  73. [81]

    Uncertainty quantification and deep ensembles

    Rahul Rahaman and Alexandre Thiery. Uncertainty quantification and deep ensembles. In M. Ranzato, A. Beygelz- imer, Y . Dauphin, P.S. Liang, and J. Wortman Vaughan, editors,Advances in Neural Information Processing Systems, volume 34, pages 20063–20075. Curran Associates, Inc....

    2021

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.