{"id":"686ac72e-695a-40f4-b7ff-78b33ac6ed8b","arxiv_id":"1908.01288","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A knowledge graph embedding plus convolutional-LSTM pipeline predicts drug-drug interactions with 0.94 AUPR on a new 2.9 million pair dataset.","lead":"This paper builds a large knowledge graph from several drug databases, turns it into vector embeddings, and uses a convolutional-LSTM network plus an ensemble to predict drug-drug interactions. If the reported accuracy holds, it could help screen dangerous drug combinations earlier and more cheaply than clinical trials.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground truth includes 145k predicted DDIs from previous models; reported AUPR/F1/MCC may reflect reproducing those predictions rather than validated interactions.","rationale":"The paper reports strong metrics, but the validity of the evaluation hinges on the ground-truth labels. The inclusion of predicted DDIs from prior models is the most load-bearing weakness because the reported accuracy could be an artifact of the model learning to replicate earlier predictions, especially since the embedding features are derived from overlapping side-effect data. This is not an internal inconsistency; the authors transparently describe the data construction. However, it is a correctness risk for the central claim. The reader's weakest assumption correctly identifies this issue. A straightforward ablation settles it. Other concerns (missing error bars, unmatched comparisons) are real but secondary; they affect interpretation, not the core validity of the evaluation. Therefore the conditional verdict is appropriate; no verdict change is needed.","tokens_in":16748,"tokens_out":5391,"duration_ms":55457,"concrete_test":"Re-run the Section 4 pipeline (ComplEx-PBG embeddings, GBT/RF/Conv-LSTM, MAE, 5-fold CV) on a version of the dataset that excludes all 145,108 interactions contributed by Zhang et al. [50] and Sridhar et al. [40], leaving only DrugBank, KEGG, TWOSIDES, and DDIExtraction corpus positives. Compare AUPR, F1, and MCC with Table 3. A drop of more than ~5 percentage points in AUPR would confirm that the headline numbers are substantially inflated by predicted labels; a negligible drop would refute this concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3.2.2 the authors add 145,068 DDIs from Zhang et al. [50], produced by label propagation on clinical side effects, and 10 top-ranked predictions from Sridhar et al. [40] to the positive set (Table 1 lists 145,108 for this category). These are model outputs, not experimentally validated interactions. The KG embeddings are trained on a graph that includes OFFSIDES side-effect data (Section 3.2.3), which is closely related to the side-effect information used by Zhang et al. The classifier can therefore achieve high scores by predicting which drug pairs the earlier label-propagation model flagged. Because 5-fold CV splits this combined positive set randomly, the predicted positives appear in both training and test folds, so the reported 0.94 AUPR / 0.92 F1 / 0.80 MCC are not a clean estimate of performance on true, curated DDIs. The central claim that the method 'predicts potential DDIs' is thus not yet established for the portion of the ground truth that is itself model-generated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a drug-drug interaction (DDI) prediction pipeline that integrates multiple drug-related data sources into a knowledge graph, embeds the graph nodes using several knowledge-graph embedding methods (with ComplEx trained via PyTorch-BigGraph reported as best), and then trains a convolutional-LSTM network together with classic machine-learning baselines on concatenated drug-pair embedding vectors. A model-averaging ensemble of the three best classifiers is reported to achieve AUPR 0.94, F1-score 0.92, and MCC 0.80 under 5-fold cross-validation. The authors claim the largest DDI dataset to date (2,898,937 interaction pairs) and a large integrated knowledge graph. The paper also compares embedding methods, analyzes learning curves, and discusses state-of-the-art comparisons.","tokens_in":16956,"tokens_out":4746,"duration_ms":46793,"significance":"If the results hold, the paper would provide a useful demonstration that multi-source knowledge-graph embeddings, especially those from PyTorch-BigGraph's ComplEx, can feed effective classifiers for DDI prediction, and the released code/scripts would support reproducibility. The evaluation is fairly broad: six embedding methods, six classical baselines plus the Conv-LSTM architecture, and multiple metrics (AUPR, F1, MCC) are considered. However, the central quantitative claim is compromised by the inclusion of model-generated predictions in the ground-truth positive set, so the significance of the reported accuracy numbers is currently unclear.","major_comments":[{"comment":"The positive set includes 145,068 DDIs predicted by label propagation in Zhang et al. [50] and the top ten predictions from Sridhar et al. [40], totaling at least 145,078 model-generated pairs out of the 145,108 in the 'MEDLINE, other sources' category. Because the 5-fold cross-validation in Section 4 splits this combined positive set randomly, these predicted positives appear in both training and test folds, so the classifier can achieve high AUPR/F1/MCC by learning to reproduce the earlier models' outputs rather than by identifying true, curated interactions. The reported 0.94/0.92/0.80 therefore do not cleanly estimate performance on validated DDIs. The authors should re-run the experiments with these predicted pairs excluded from the positive set, or at minimum report performance separately on the curated and predicted subsets.","section":"Section 3.2.2 / Table 1"},{"comment":"The Introduction states that the integrated knowledge graph has '1.2 billion triples,' but Table 2 reports a total of 11,281,434 triples across all sources, a discrepancy of roughly two orders of magnitude. This inconsistency affects the paper's scale claims, including the 'largest available' DDI dataset assertion. The authors must correct either the text or the table and ensure all quantitative descriptions of the data artifacts are consistent.","section":"Section 1 contribution bullet / Table 2"},{"comment":"The headline results are reported as point estimates with no standard deviations, no fold-level breakdown, and no explanation of how the 'best' configuration was selected among the 5 runs and hyperparameter searches. The abstract's use of 'up to' suggests best-case selection rather than average performance. The authors should report mean and standard deviation over folds and independent runs, and state explicitly whether the Table 3 values are means, best folds, or best runs.","section":"Section 4 / Table 3"},{"comment":"The negative-sampling ratio sigma is handled in a way that is unclear and potentially post hoc: the text says sigma was set to 15, then further varied in {5,10,20,25}, with sigma=20 giving about a 1% AUPR boost, yet it is not stated which sigma produced the results in Table 3 and whether this selection was made using a validation set or the test set. The authors should describe a clear model-selection protocol (e.g., nested cross-validation) so that the reported test numbers are not optimized after seeing test performance.","section":"Section 4.4"}],"minor_comments":[{"comment":"The phrase 'interaction exits between drugs' should be corrected to 'interaction exists between drugs.'","section":"Equation (1)"},{"comment":"The embedding method is referred to as 'SimpleIE' in the text and Table 3, but the reference [25] and the standard name are 'SimplE'; please use a consistent spelling.","section":"Section 3.3 and Table 3"},{"comment":"The sentence describing the data split is ambiguous: it says 70% of the data is used for training, 30% for evaluating, and 10% from the training set for validation, which does not sum to a clear protocol. Please specify the actual proportions, e.g., 70% train / 10% validation / 20% test, and how the 5 runs are defined.","section":"Section 3.5"},{"comment":"The Pearson product-moment correlation coefficient of 0.70 is mentioned without defining which variables are correlated; please define the variables or remove the statement.","section":"Section 4.1"},{"comment":"For the interactions from Zhang et al. and Sridhar et al., the verb 'extract' is not appropriate because these are model predictions rather than extracted facts; consider saying 'we included these predicted interactions' to avoid implying experimental verification.","section":"Section 3.2.2"},{"comment":"Reference [44] appears to duplicate reference [43] (the same Tatonetti et al. paper); please remove the duplicate or distinguish the two citations with different page numbers or versions if intended.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The ground-truth contamination issue is the main risk: the authors need to re-run without the predicted positives from Zhang et al. and Sridhar et al., or at least report a clear breakdown. Please also verify the billion-triple versus 11-million-triple discrepancy before any acceptance decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know about this paper: it builds the largest DDI dataset I've seen (2.9M pairs) and runs a broad, honest comparison of embedding methods feeding a Conv-LSTM ensemble. The reported numbers (AUPR 0.94, F1 0.92, MCC 0.80) are probably not real, because the positive set includes 145k interactions that are themselves predictions from an earlier label-propagation model (Zhang et al.) plus a handful from Sridhar et al. Section 3.2.2 says so explicitly. Since the 5-fold CV splits are random, those predicted positives appear in both training and test folds, and since the KG contains OFFSIDES side-effect data—the same signal Zhang used—the model can learn to copy the label-propagation output. So the headline results are inflated by construction.\n\nWhat's genuinely new: the integrated KG (drug, gene, protein, pathway, phenotype relations) and the systematic comparison of six embedding methods with several classifiers, with code and data links provided. The embedding stage is unsupervised with respect to DDI labels, so there is no circularity in the narrow sense; the problem is the ground truth, not the fitted parameters. The finding that ComplEx/PBG embeddings work best is plausible and useful.\n\nSoft spots, in order: (1) the label leakage above; (2) no error bars, and the best numbers appear to be selected post hoc across many runs; (3) internal inconsistencies in the counts—abstract says 1.2 billion triples but Table 2 sums to ~11.3 million, and the KEGG interaction count in the text (58,205) differs from Table 1 (40,540). These are fixable but they undermine confidence. The comparison with prior work is also not on matched data, though the authors acknowledge that.\n\nThe paper deserves a serious referee: the dataset and evaluation scaffold could be useful if the ground truth is cleaned and the experiments redone. I wouldn't cite it as-is, but I'd keep it in mind as a cautionary example of predicted labels entering ground truth. Recommend: send to peer review, but expect major revision.\n\nBest.","headline":"Useful large-scale DDI dataset and honest embedding comparison, but headline results are inflated by putting prior model predictions into the ground truth.","tokens_in":17476,"tokens_out":2622,"would_cite":false,"duration_ms":25065,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A knowledge-graph embedding pipeline with a convolutional-LSTM ensemble predicts drug-drug interactions with AUPR 0.94, F1 0.92, and MCC 0.80 in five-fold cross-validation.","keywords":["drug-drug interaction prediction","knowledge graph embeddings","ComplEx embeddings","convolutional-LSTM","model averaging ensemble","link prediction","imbalanced classification","multi-source drug data"],"falsifier":"Retrain the best-performing ensemble after deleting every interaction that entered the dataset from a prediction algorithm rather than from a curated or pharmacist-verified source, then measure AUPR and MCC on a held-out set of verified interactions; if the scores fall well below 0.94 and 0.80, the original accuracy was carried by predicted labels rather than by genuine interaction knowledge.","tokens_in":16596,"feed_emoji":"💊","tokens_out":9581,"duration_ms":90102,"temperature":0.7,"pith_summary":"The paper argues that drug-drug interactions can be predicted more accurately by fusing several public drug-information repositories into a knowledge graph, embedding the graph's nodes into dense vectors, and training classifiers on the concatenated vectors of each drug pair. The authors assembled 2,898,937 interaction pairs and an integrated knowledge graph with 11,281,434 triples, which they believe is the largest such resource, and evaluated six embedding methods with several classifiers. Their best configuration, ComplEx embeddings trained at scale followed by a convolutional-LSTM network whose probabilities are averaged with random forest and gradient-boosted trees, reaches an AUPR (area under the precision-recall curve) of 0.94, an F1-score of 0.92, and a Matthews correlation coefficient of 0.80 in five-fold cross-validation. If this holds, drug development and co-prescription screening could prioritize unknown interactions computationally before resorting to costly clinical evaluation.","feed_headline":"Knowledge-graph model predicts drug-drug interactions at 0.94 AUPR","feed_subtitle":"An ensemble of tree models and a CNN-LSTM scores higher than single-source baselines on 2.9 million drug pairs.","key_machinery":"The load-bearing mechanism is a three-stage pipeline. First, a knowledge graph is assembled from drug, gene, protein, pathway, enzyme, and phenotype data, deliberately excluding explicit interaction edges so that the learned representations carry background knowledge rather than the target labels. Second, ComplEx embeddings, a knowledge-graph embedding model that scores triples with the Hermitian dot product of complex vectors, allowing it to model symmetric and antisymmetric relations, are trained at scale, producing 300-dimensional vectors for 12,439 drugs; each drug pair is represented by concatenating its two vectors. Third, a convolutional-LSTM classifier processes that representation: a 1D convolutional layer with 100 filters and kernel size 4 extracts local feature patterns, max pooling downsamples them, and an LSTM layer treats the flattened features as timesteps to carry global dependencies; its output passes through dense, dropout, and Gaussian-noise layers to a softmax. The final predictor averages the probabilities of this network with random forest and gradient-boosted trees, and this ensemble defines the reported accuracy.","core_discovery":"The central discovery is that the choice of knowledge-graph embedding, rather than classifier sophistication alone, drives DDI prediction quality. The paper reports that ComplEx embeddings, a model that scores triples with the Hermitian dot product of complex vectors and can represent both symmetric and antisymmetric relations, yield the most informative drug-pair features; the same classifiers perform markedly worse on random-walk or translation-based embeddings. On those features, a convolutional-LSTM network, which uses 1D convolution for local feature patterns and an LSTM for global dependencies, outperforms every baseline classifier, and averaging its predictions with random forest and gradient-boosted trees improves the F1-score by about 1.5 percentage points over the best individual model. The quantitative claim is Table 3: AUPR 0.94, F1 0.92, MCC 0.80 during five-fold cross-validation, with the ensemble best across all six embedding methods.","pith_inferences":["Editorial extension: because part of the positive set consists of earlier model predictions, a prospective evaluation on only pharmacist-verified, post-market interactions would test whether the 0.94 AUPR carries over to genuine clinical discovery.","Editorial extension: the same integrate-then-embed pipeline could be applied to other biomedical link-prediction tasks, such as drug-target binding or adverse-event prediction, where multi-source background knowledge is available.","Editorial extension: an ablation that removes each data source from the integrated graph would reveal which sources drive the gain; the paper does not isolate this, but it is a natural next experiment.","Editorial extension: a temporal holdout, training only on interactions known before a cutoff date and testing on later-reported ones, would measure prospective utility more realistically than random cross-validation splits."],"forward_implications":["If the claim is correct, unknown drug pairs can be ranked from public knowledge-graph data alone, allowing laboratories and regulators to concentrate scarce testing resources on the most plausible interactions.","The large gap between embedding methods means that scalable, relation-aware embeddings are a higher-leverage investment than classifier choice alone for this problem.","The Conv-LSTM's consistent advantage over classical baselines suggests that combining local feature extraction with sequential and global modeling captures complementary signals in drug-pair data.","Reporting AUPR and MCC, rather than AUC alone, is the appropriate standard for this imbalanced task, and the reported numbers supply a concrete benchmark for later methods.","The assembled dataset and integrated knowledge graph are positioned as a reusable benchmark, so future work can compare methods on identical training and test splits."],"supporting_citations":[{"why":"Supplies the primary drug database: drug entries, targets, and roughly 2.6 million of the known interaction pairs used as labels.","marker":"[48]"},{"why":"Contributes drug entries and interaction relations to both the interaction label set and the integrated knowledge graph.","marker":"[23]"},{"why":"Supplies drug, gene, and pathway knowledge used to build the integrated knowledge graph.","marker":"[47]"},{"why":"Supplies drug side-effect data to the integrated knowledge graph.","marker":"[43]"},{"why":"Supplies the polypharmacy adverse-event pairs that become part of the interaction dataset.","marker":"[44]"},{"why":"The scalable graph-embedding implementation used to train the embeddings that yield the best drug features.","marker":"[27]"},{"why":"Defines the embedding model whose Hermitian dot-product scoring is the basis of the best-performing feature vectors.","marker":"[45]"},{"why":"Introduces the convolutional-LSTM architecture that the paper adapts for drug-pair classification.","marker":"[37]"},{"why":"Supplies 145,068 label-propagation-predicted interactions added to the ground-truth set; load-bearing for the label set and for the weakest assumption.","marker":"[50]"},{"why":"Adds the top ten model-predicted interactions to the ground-truth label set, influencing the evaluation.","marker":"[40]"}],"fun_headline_variants":["ComplEx embeddings boost drug-drug interaction predictions to 0.94 AUPR","Ensemble of CNN-LSTM and trees predicts drug interactions with 0.94 AUPR","Knowledge graph embeddings key to drug-drug interaction prediction accuracy","CNN-LSTM ensemble hits 0.94 AUPR on drug-drug interaction prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the list of known interactions used for training and testing is correct, even though about 145,000 of its entries were themselves produced by a label-propagation prediction algorithm and a few more came from another model's top-ranked guesses; if those predicted labels are biased, the reported accuracy is inflated.","fun_headline_variants_meta":{"raw":{"variants":["ComplEx embeddings boost drug-drug interaction predictions to 0.94 AUPR","Ensemble of CNN-LSTM and trees predicts drug interactions with 0.94 AUPR","Knowledge graph embeddings key to drug-drug interaction prediction accuracy","CNN-LSTM ensemble hits 0.94 AUPR on drug-drug interaction prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000623,"raw_usage":{"total_tokens":2910,"prompt_tokens":994,"completion_tokens":1916,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":1833}},"tokens_in":610,"tokens_out":1916,"duration_ms":13233,"temperature":1.0,"reasoning_tokens":1833,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:16:40.081947+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the best-performing ensemble after deleting every interaction that entered the dataset from a prediction algorithm rather than from a curated or pharmacist-verified source, then measure AUPR and MCC on a held-out set of verified interactions; if the scores fall well below 0.94 and 0.80, the original accuracy was carried by predicted labels rather than by genuine interaction knowledge.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the primary drug database: drug entries, targets, and roughly 2.6 million of the known interaction pairs used as labels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes drug entries and interaction relations to both the interaction label set and the integrated knowledge graph."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies drug, gene, and pathway knowledge used to build the integrated knowledge graph."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies drug side-effect data to the integrated knowledge graph."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the polypharmacy adverse-event pairs that become part of the interaction dataset."},{"cited_title":"Journal of cheminformatics 9, 1 (2017), 16","cited_arxiv_id":null,"evidence_quote":"Defines the embedding model whose Hermitian dot-product scoring is the basis of the best-performing feature vectors."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the convolutional-LSTM architecture that the paper adapts for drug-pair classification."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies 145,068 label-propagation-predicted interactions added to the ground-truth set; load-bearing for the label set and for the weakest assumption."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Adds the top ten model-predicted interactions to the ground-truth label set, influencing the evaluation."}],"review_version":1}