LLMs for Drug-Drug Interaction Prediction: A Comprehensive Comparison

Athanasios Angelakis; Filomena Ferrucci; Gabriele De Vito

REVIEW 4 major objections 4 minor 2 cited by

LLMs for Drug-Drug Interaction Prediction: A Comprehensive Comparison

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuned language models can predict drug-drug interactions from plain-text drug data, with a small 2.7B-parameter model reaching 97.8% sensitivity and matching an established logistic regression baseline.

desk verdict Useful LLM-for-DDI benchmark with reproducible code and a sensible small-model finding, but the 'state-of-the-art' claim is unsupported because the only baseline is a 2021 logistic regression that receives fewer input features than the LLMs. read the letter →

arxiv 2502.06890 v1 pith:KFS7FU7S submitted 2025-02-09 cs.LG cs.AIq-bio.QM

Gabriele De Vito , Filomena Ferrucci , Athanasios Angelakis This is my paper · ORCID

classification cs.LGcs.AIq-bio.QM

keywords drug-druginteractionpredictionlargelanguagemodelsfine-tuningzero-shotlearningSMILESgenetargetprofilesexternalvalidation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether large language models can predict drug-drug interactions directly from textual descriptions of drug pairs, and whether fine-tuning makes them competitive with conventional machine learning. It evaluates 18 models without any task-specific training (zero-shot) and then fine-tunes five selected models on a balanced training set built from a major drug database, using prompts that combine drug names, SMILES strings, target organisms, and gene-target vectors. The central result is that fine-tuning transforms modest zero-shot performance into strong classification: the best small model, Phi-3.5 2.7B, reaches 0.978 sensitivity and 0.919 accuracy on the balanced validation set, and its sensitivity advantage persists across 13 external datasets. If this holds, small fine-tuned language models could serve as locally deployable, privacy-preserving screening tools for polypharmacy safety without specialized infrastructure.

What carries the argument

The load-bearing object is the serialized drug-pair prompt: a fixed template that turns each ordered pair into drug names, SMILES strings (linear text encodings of molecular structure), target organisms, and gene-target indicator vectors, followed by a forced binary classification. This prompt is what lets a language model apply its attention over chemical and biological features simultaneously and what makes the task a text problem rather than a tabular one. The argument is carried by coupling this representation with LoRA (low-rank adaptation) fine-tuning of small open-weight models, and by comparing the resulting classifiers against $\ell^2$-regularized logistic regression on the same gene-target features across 13 external datasets.

What would settle it

Compare the fine-tuned Phi-3.5 classifier against the logistic regression baseline on a held-out set of drug pairs whose non-interaction status has been experimentally verified rather than inferred from absence in databases; if the LLM's sensitivity advantage disappears or reverses on those clean labels, the central comparative claim fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that drug-drug interaction prediction can be reframed as a text-classification task and that fine-tuned LLMs are the best performers at that task. The authors construct a balanced benchmark from the primary drug database plus 13 external sources, represent each ordered drug pair as a prompt containing the drugs' names, SMILES notation, target organisms, and binary gene-interaction vectors, and ask models to output 'interaction' or 'no interaction.' They report that zero-shot LLMs are too unreliable for screening (average sensitivity about 0.55), but after LoRA fine-tuning on only 1,000 training pairs, smaller models match or exceed an $\ell^2$-regularized logistic regression baseline trained on gene-target profiles; Phi-3.5 2.7B attains the highest external sensitivity average among all models tested. The paper concludes that model size is less important than task-specific adaptation and that the textual representation lets LLMs exploit structural and genetic information jointly.

Load-bearing premise

The study treats a drug pair as a negative example simply because it is absent from known interaction databases, and the reported performance depends on those unverified labels being true non-interactions.

Editorial extensions

If this is right

Small fine-tuned LLMs, at 1.5B to 3B parameters, can deliver high-sensitivity DDI screening on commodity hardware, which the paper argues makes local and private deployment practical.
The sensitivity advantage of Phi-3.5 over the logistic regression baseline appears across the average of 13 external datasets, not only on the original validation set.
Zero-shot prompting alone is not a reliable DDI screening strategy; task-specific fine-tuning changes average sensitivity from roughly 0.55 to above 0.9.
Model size is not the main driver of DDI performance: a 2.7B model outperforms much larger models after fine-tuning.
Because the input is plain text, the same pipeline can absorb additional drug features, such as metabolic pathways and pharmacokinetic properties, without changing the model family.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same text-serialization recipe likely transfers to adjacent prediction problems, such as drug-food, drug-gene, or drug-supplement interactions, because those inputs also reduce to paired descriptions; the paper does not test these settings.
The 1,000-pair training set suggests a very low data requirement, but the paper does not test how performance degrades as training size shrinks; an ablation would reveal whether the model learns general molecular interaction patterns or memorizes a small set of drug names.
Ablating the SMILES, organism, and gene fields from the prompt could isolate which modality carries the predictive signal and whether the language model is using the intended molecular information rather than name co-occurrence patterns.
Because smaller models match larger ones, a practical next step would be to train even smaller or distilled models on the same serialized prompts and measure the accuracy-versus-size trade-off.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Worth knowing: this is a useful, mostly honest empirical benchmark of 18 LLMs for DDI prediction, with a real headline finding — after fine-tuning on 1,000 examples, small models (Phi-3.5 2.7B) match or beat the implemented l2 logistic regression on sensitivity. But the "new state-of-the-art" claim in the abstract and contributions is not supported: the only non-LLM comparator is one 2021 logistic regression, and that baseline gets only gene-target profiles while the LLM prompt also includes SMILES and organisms. So the comparison conflates model class with input features. The stress-test note has this right.

What is genuinely good: the 18-model sweep, the text representation combining three drug features, the fine-tuning comparison, and the external validation across 13 datasets. They shipped code and processed datasets, did a hyperparameter search with Optuna, and ran five repeated classifications that showed no variability — that is real reproducibility care. Their RQ1 finding, that zero-shot LLMs are weak (best sensitivity around 0.59), is a useful negative result. The RQ2 result, that fine-tuning small models changes everything, is the paper's actual contribution.

Soft spots, in order of severity. First, the baseline. The paper's own related work lists DeepDDI, DPDDI, STNN-DDI, and DDI-GPT, but none is evaluated. A new state-of-the-art claim requires at least one modern strong predictor, not a re-implementation of [13]. Second, the feature confound: even the logistic regression comparison is unfair because the LLM gets SMILES and organism text plus genes. To isolate LLM value, the baseline needs the same input features. Third, the numbers: abstract's 0.919 accuracy doesn't match Table 4 (0.913 on validation) or Table 6 (0.881 external average), and the prose around sensitivity is loose. These need correcting. Fourth, negative examples are random drug pairs absent from known databases; if some are undocumented interactions, both LLM and baseline metrics inflate. The reader calls this the weakest assumption, and it is real, but since it hits both arms it doesn't by itself invalidate the LLM-vs-baseline comparison — it makes the absolute numbers optimistic, not the relative claim.

Bottom line: the paper deserves a serious referee. It is a solid empirical contribution if reframed as "fine-tuned small LLMs are competitive for DDI screening with text-only features," with modern baselines and corrected numbers. As written, the SOTA claim is unsupported, but the work underneath is careful and reproducible.

Referee Report

4 major / 4 minor

Summary. This paper presents an empirical study of 18 large language models for binary drug-drug interaction (DDI) prediction. Drug pairs are described to the models with SMILES strings, target organisms, and gene-target vectors, and the models are evaluated zero-shot and after fine-tuning. The best fine-tuned model, Phi-3.5 2.7B, is reported to achieve high sensitivity on external datasets, and the paper claims that fine-tuned LLMs improve over state-of-the-art machine-learning DDI predictors. The only non-LLM comparator is an l2-regularized logistic regression baseline taken from prior work.

Significance. If the central claim were established, the finding that a 2.7B fine-tuned LLM can approximately match or exceed a standard logistic-regression DDI predictor while consuming only textual drug features would be practically useful, especially for local deployment and privacy-sensitive settings. The paper also provides reproducible artifacts: a public repository with data-processing scripts, fine-tuning code, and repeated classifications that show no prediction variability. However, the headline claim of superiority over state-of-the-art machine-learning methods is not currently supported. The sole baseline uses a different input representation, no recent DDI-specific methods are compared, and there is a potential evaluation leakage issue between the baseline and the LLM validation set. These problems directly affect the paper's main contribution, so the manuscript needs substantial revision before the claim can be accepted.

major comments (4)

[Abstract; §5.3; §4.5.2] The claim that fine-tuned LLMs 'improve over state-of-the-art machine-learning methods' is not supported by the reported experiments. The only non-LLM baseline is l2-regularized logistic regression [13], trained on gene-target profiles only, while the LLM prompt (Section 4.4.2) additionally contains SMILES strings and target organisms. The comparison therefore conflates the model class with the input representation. In addition, no recent DDI-specific methods (e.g., DeepDDI [42], DPDDI [44], STNN-DDI [14], DDI-GPT [21]) are evaluated. Please add baselines that use the same features as the LLMs, include modern DDI predictors, and temper or remove the 'new state-of-the-art' wording until such comparisons are available.
[§4.5.2] The baseline evaluation appears to be subject to training/validation leakage. The logistic regression is trained on 95% of the balanced DrugBank dataset and then evaluated on 'the same LLM validation set,' which was randomly extracted from that same balanced DrugBank dataset (Section 4.3). If the 1,090 validation pairs were not explicitly excluded from the 95% training split, the baseline may have seen them during training, making the RQ3 comparison invalid. The paper must state exactly how the 95%/5% split relates to the LLM training and validation sets, and if leakage occurred, the baseline must be retrained on a split that excludes the LLM validation pairs.
[§4.3] Negative examples are defined as random drug pairs absent from the known interaction databases and are treated as true non-interactions. This is a strong label assumption. Absence of an interaction from DrugBank and the 13 external sources is not evidence that the pair does not interact, especially for widely studied drugs. If a meaningful fraction of these pairs actually interact, both the training data and the external validation metrics are corrupted, inflating accuracy and sensitivity. The authors should validate the negative labels using a curated non-interaction source, or at least quantify the likely false-negative rate in the negative set.
[Abstract; Table 4; Table 6] The headline numbers in the abstract are internally inconsistent. The abstract states that Phi-3.5 achieves a sensitivity of 0.978 and an accuracy of 0.919 on balanced datasets, but Table 4 reports a validation accuracy of 0.913 and Table 6 reports an external average accuracy of 0.881; the sensitivity 0.978 appears in Table 5 as an external average. The value 0.919 appears only as one dataset row (HEP) in Table 6, not as a global result. Please specify the exact evaluation condition for each reported number and make the abstract consistent with the tables.

minor comments (4)

[§4.4.3; Tables 4-7] The text says GPT-4 was fine-tuned, while the tables label the column 'GPT-4o' and '+250B'. Please standardize the model name and clarify which GPT-4 variant was actually used.
[§3.4] The paragraph describing the ESM-2/ChemBERTa framework contains a duplicated sentence: 'A framework leveraging protein language models (ESM-2) and chemical language models (ChemBERTa) ... has been introduced in [20]. Their approach demonstrated that ...' appears twice. Please remove the duplicate.
[Tables 5-7] The column headers in Tables 5-7 are formatted without separators (e.g., 'Phi3.5Qwen2.5Gemma2GPT-4o Deepseek'), making the table difficult to read. Please reformat the headers so each model name is clearly separated.
[§4.4.3; §5.2] The fine-tuning hyperparameters were selected by minimizing validation loss on a 1,000-example training set, and the reported validation metrics are for that same validation set. While the external datasets mitigate this concern, the paper should report confidence intervals or repeated fine-tuning runs with different seeds to assess the stability of the validation results.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the study is an empirical benchmark with independent training, validation, and external test sets; the main weaknesses are experimental confounds, not definitional circularity.

full rationale

The paper makes no derived mathematical prediction whose output is equivalent to its input by construction. Fine-tuned LLMs are trained on 1,000 DrugBank pairs and evaluated on a held-out 1,090-pair validation set and on 13 external datasets whose positive pairs were filtered to remove overlap with DrugBank, so the reported external metrics are not forced by the training procedure. The l2-regularized logistic regression baseline is trained separately on the same source data and evaluated on the same external sets, so the LLM-vs-baseline comparison is not circular. The negative-example construction (random pairs absent from known DDI databases) is a shared validity threat affecting both LLMs and the baseline approximately equally; it is a labeling limitation, not a circular step. The only self-references are to the authors' GitHub repository for code and results, which is not load-bearing for any claim. The claim that fine-tuned LLMs improve over 'state-of-the-art' methods is weakened by an input-feature confound: the only non-LLM comparator uses gene-target profiles only, whereas LLM prompts also include SMILES and organisms, and modern DDI-specific baselines are not reproduced. That is an experimental-design and generalizability concern, not a definitional equivalence. No equation in the paper reduces to its own input, and no fitted parameter is renamed as a prediction, so the circularity score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central evaluation rests on DrugBank's completeness and on the assumption that random unlisted pairs are true negatives. Hyperparameters are fitted to the validation split. No new physical, chemical, or formal entities are introduced.

free parameters (5)

Fine-tuning hyperparameters for Phi-3.5 2.7B = layers=16, lr=2e-4, rank=16, alpha=16, scale=4.0, dropout=0.0
Selected by Optuna to minimize validation loss (Section 4.4.3).
Fine-tuning hyperparameters for Qwen2.5 3B = layers=16, lr=2e-4, rank=16, alpha=16, scale=4.0, dropout=0.0
Selected by Optuna to minimize validation loss (Section 4.4.3).
Fine-tuning hyperparameters for Deepseek R1 Qwen 1.5B = layers=20, lr=2.2e-4, rank=32, alpha=64, scale=4.0, dropout=0.009
Selected by Optuna to minimize validation loss (Section 4.4.3).
Fine-tuning hyperparameters for Gemma2 9B = layers=16, lr=1e-5, rank=16, alpha=16, scale=4.0, dropout=0.1
Selected by Optuna to minimize validation loss (Section 4.4.3).
Regularization parameter C for l2-regularized logistic regression = Tuned via stratified 10-fold CV over [2^-16, 2^16]
Follows the original paper [13] and is fit on the DrugBank training data (Section 4.5.2).

assumptions (5)

domain assumption DrugBank interaction labels are treated as ground truth for positive drug pairs.
All training and validation positives come from DrugBank; if these labels are incomplete or wrong, metrics are affected (Section 4.3).
domain assumption Pairs absent from known interaction databases are treated as true negatives.
Negative examples were generated by sampling pairs with no overlap with known interactions, assuming absence of evidence means evidence of absence (Section 4.3, step 3).
domain assumption The textual prompt (SMILES, organisms, gene lists) carries enough information for DDI prediction.
The method provides no learned features beyond this text; if gene target or organism annotations are incomplete, prediction performance is capped (Section 4.4.2).
domain assumption External dataset labels remain valid after removing DrugBank-overlapping interactions.
The paper filters external interactions to keep only novel pairs and treats the remaining labels as correct (Section 4.3, step 2).
domain assumption LLM outputs are reliably parseable into the two classes.
The system prompt asks for only 'interaction' or 'no interaction'; parsing edge cases are not discussed (Section 4.4.2).

how reviews work

0 comments

Cite this review

Pith. "Pith review of LLMs for Drug-Drug Interaction Prediction: A Comprehensive Comparison." pith.science (2026). https://pith.science/paper/KFS7FU7S

@misc{pith2026250206890,
  author       = {Pith},
  title        = {Pith review of: LLMs for Drug-Drug Interaction Prediction: A Comprehensive Comparison},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KFS7FU7S}},
  note         = {Machine review of arXiv:2502.06890}
}

read the original abstract

The increasing volume of drug combinations in modern therapeutic regimens needs reliable methods for predicting drug-drug interactions (DDIs). While Large Language Models (LLMs) have revolutionized various domains, their potential in pharmaceutical research, particularly in DDI prediction, remains largely unexplored. This study thoroughly investigates LLMs' capabilities in predicting DDIs by uniquely processing molecular structures (SMILES), target organisms, and gene interaction data as raw text input from the latest DrugBank dataset. We evaluated 18 different LLMs, including proprietary models (GPT-4, Claude, Gemini) and open-source variants (from 1.5B to 72B parameters), first assessing their zero-shot capabilities in DDI prediction. We then fine-tuned selected models (GPT-4, Phi-3.5 2.7B, Qwen-2.5 3B, Gemma-2 9B, and Deepseek R1 distilled Qwen 1.5B) to optimize their performance. Our comprehensive evaluation framework included validation across 13 external DDI datasets, comparing against traditional approaches such as l2-regularized logistic regression. Fine-tuned LLMs demonstrated superior performance, with Phi-3.5 2.7B achieving a sensitivity of 0.978 in DDI prediction, with an accuracy of 0.919 on balanced datasets (50% positive, 50% negative cases). This result represents an improvement over both zero-shot predictions and state-of-the-art machine-learning methods used for DDI prediction. Our analysis reveals that LLMs can effectively capture complex molecular interaction patterns and cases where drug pairs target common genes, making them valuable tools for practical applications in pharmaceutical research and clinical settings.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

MARD: Mirror-Augmented Reasoning Distillation for Mechanism-Level Drug-Drug Interaction Prediction
cs.CL 2026-06 unverdicted novelty 6.0 of 10

MARD-7B outperforms baselines and GPT-4o on novel drug pairs for mechanism-level DDI prediction via a new distillation pipeline with verifiable process rewards and releases all resources.
DeepSeek in Healthcare: A Survey of Capabilities, Risks, and Clinical Applications of Open-Source Large Language Models
cs.CL 2025-06 conditional

A narrative review of DeepSeek-R1's healthcare capabilities, risks, and applications, without new experiments.

Reference graph

Works this paper leans on

4 extracted references · 1 canonical work pages · cited by 2 Pith papers

[3]

Prevalence and factors associated with polypharmacy: a systematic review and meta-analysis

Mahin Delara, Lauren Murray, Behnaz Jafari, Anees Bahji, Zahra Goodarzi, Julia Kirkham, Mohammad Chowd- hury, and Dallas P Seitz. Prevalence and factors associated with polypharmacy: a systematic review and meta-analysis. BMC geriatrics, 22(1):601, 2022. [4] Hyesung Lee, Yeon-Hee Baek, Ju Hwan Kim, Tzu-Chi Liao, Wallis CY Lau, Kenneth KC Man, Xiwen Qin, S...

work page 2022
[24]

Sequence-dependence of cisplatin and 5-fluorouracil in advanced and recurrent gastric cancer

Wasaburo Koizumi, Minoru Kurihara, Koichi Hasegawa, Akimichi Chonan, Yasuhiko Kubo, Ryuichiro Maekawa, Ryozo Iwasaki, Tadashi Sasai, Yoshio Fukuyama, Kunitsugu Ishikawa, et al. Sequence-dependence of cisplatin and 5-fluorouracil in advanced and recurrent gastric cancer. Oncology reports, 12(3):557–561, 2004. [25] Magda Bahcall, Yanan Kuang, Cloud P Pawele...

arXiv 2004
[46]

Iscmf: Integrated similarity-constrained matrix factoriza- tion for drug–drug interaction prediction

Narjes Rohani, Changiz Eslahchi, and Ali Katanforoush. Iscmf: Integrated similarity-constrained matrix factoriza- tion for drug–drug interaction prediction. Network Modeling Analysis in Health Informatics and Bioinformatics, 9:1–8, 2020. [47] Guy Shtar, Lior Rokach, and Bracha Shapira. Detecting drug-drug interactions using artificial neural networks and ...

arXiv 2020
[68]

Element Labs

Inc. Element Labs. Lm studio website. [Online; accessed 15 November 2024]. [69] OpenAI. Gpt models fine-tuning. [Online; accessed 15 November 2024]. [70] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. [7...

arXiv 2024

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [3]

Prevalence and factors associated with polypharmacy: a systematic review and meta-analysis

Mahin Delara, Lauren Murray, Behnaz Jafari, Anees Bahji, Zahra Goodarzi, Julia Kirkham, Mohammad Chowd- hury, and Dallas P Seitz. Prevalence and factors associated with polypharmacy: a systematic review and meta-analysis. BMC geriatrics, 22(1):601, 2022. [4] Hyesung Lee, Yeon-Hee Baek, Ju Hwan Kim, Tzu-Chi Liao, Wallis CY Lau, Kenneth KC Man, Xiwen Qin, S...

work page 2022

[2] [24]

Sequence-dependence of cisplatin and 5-fluorouracil in advanced and recurrent gastric cancer

Wasaburo Koizumi, Minoru Kurihara, Koichi Hasegawa, Akimichi Chonan, Yasuhiko Kubo, Ryuichiro Maekawa, Ryozo Iwasaki, Tadashi Sasai, Yoshio Fukuyama, Kunitsugu Ishikawa, et al. Sequence-dependence of cisplatin and 5-fluorouracil in advanced and recurrent gastric cancer. Oncology reports, 12(3):557–561, 2004. [25] Magda Bahcall, Yanan Kuang, Cloud P Pawele...

arXiv 2004

[3] [46]

Iscmf: Integrated similarity-constrained matrix factoriza- tion for drug–drug interaction prediction

Narjes Rohani, Changiz Eslahchi, and Ali Katanforoush. Iscmf: Integrated similarity-constrained matrix factoriza- tion for drug–drug interaction prediction. Network Modeling Analysis in Health Informatics and Bioinformatics, 9:1–8, 2020. [47] Guy Shtar, Lior Rokach, and Bracha Shapira. Detecting drug-drug interactions using artificial neural networks and ...

arXiv 2020

[4] [68]

Element Labs

Inc. Element Labs. Lm studio website. [Online; accessed 15 November 2024]. [69] OpenAI. Gpt models fine-tuning. [Online; accessed 15 November 2024]. [70] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. [7...

arXiv 2024