{"id":"fa93a112-e72c-42be-9d23-34010aacfd7e","arxiv_id":"2506.21085","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CovDocker provides a large ML-ready benchmark for covalent protein-ligand docking by decomposing the problem into reactive site prediction, covalent reaction prediction, and covalent pose prediction, with baseline models and data.","lead":"This paper presents CovDocker, a machine-learning benchmark that splits covalent drug docking into three tasks: finding where a drug will bond to a protein, predicting the new molecule after the bond forms, and docking that new molecule. It releases processed structures from two existing databases and reports baseline results from adapted models such as Uni-Mol and Chemformer.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The new RMSD(IB) metric is largely determined by the §3.4 postprocessing step that resamples the covalent bond length, so the Table 4 cov-docking results do not establish the benchmark's claimed pose-prediction rigor.","rationale":"The reader's weakest-assumption pick (post-reactive ligand reconstruction in Appendix A.2) is a genuine risk: if those heuristics mis-assign bond orders, atoms, or protonation states, both Task 2 SMILES labels and Task 3 pose labels would be corrupted. However, it remains a hypothetical risk—the paper gives no validation sample, but also no direct evidence of error, and the pipeline is adapted from a peer-reviewed method (CovBinderInPDB). The RMSD(IB)/postprocessing issue is different: it is an internal inconsistency demonstrated by the paper's own numbers. Section 3.4's postprocessing samples a bond length from the global N(μ,σ) distribution and moves the bonded atom to it; Section 4.3's RMSD(IB) then measures exactly this quantity. Table 7 isolates postprocessing and shows it alone raises RMSD(IB)<0.5Å by ~18 points while standard RMSD is unchanged. This means the new metric, presented as a rigorous evaluation of covalent-bond precision, can be satisfied by a one-atom bond-length correction that does not improve the docked pose. I therefore keep the reader's CONDITIONAL verdict, but shift the primary justification to the evaluation metric. Both concerns should be addressed before the benchmark's evaluation claims are accepted: the RMSD(IB) metric should be reworked or removed, and the reconstruction pipeline should be validated on a manually curated sample. The dataset itself remains potentially useful, which is why the verdict stays conditional rather than moving to reject.","tokens_in":21584,"tokens_out":11376,"duration_ms":125747,"concrete_test":"Re-run Table 4 for Ours-p and Ours with the §3.4 postprocessing disabled, and separately compute RMSD(IB) for a trivial baseline that only places the bonded ligand atom at a randomly sampled N(μ,σ) bond length while keeping the rest of the pose at the raw Uni-Mol output (or at a random pose). If the trivial baseline matches Ours-p's RMSD(IB) numbers (~79% below 0.5Å), the metric is not informative for pose quality, and Tables 4/7 should be re-reported without RMSD(IB) or with a pose-invariant metric (e.g., full-ligand RMSD computed after removing the bonded atom, or RMSD that ignores the one-atom bond-length correction).","verdict_should_be":"UNCHANGED","load_bearing_attack":"In §3.4, after Uni-Mol predicts a pose, the authors sample l′ from N(μ,σ), the global distribution of all covalent bond lengths in the dataset, and move the bonded ligand atom toward the reactive residue until the bond length equals l′. The new metric RMSD(IB) defined in §4.3 measures the covalent inter-bond distance, i.e., essentially the predicted bond length. Because the postprocessor directly sets this length to a value drawn from the same distribution that contains the true bond length, the error |l′ − l_true| is bounded by roughly σ of the global bond-length distribution. This is visible in the paper's own ablation: Table 7 rows (a) vs (b) show the postprocessing step alone raises RMSD(IB)<0.5Å from 59.2% to 77.1%, while the whole-ligand RMSD<2Å stays at 37.7%. In Table 4, Ours-p jumps from 60.1% to 79.1% (RMSD(IB)<0.5Å) purely from postprocessing, with RMSD<2Å unchanged at 37.2%. A model that produces a completely wrong pose but places the warhead atom at a typical covalent bond length from the reactive residue would score near-perfectly on RMSD(IB). Thus the metric does not measure pose accuracy or 'covalent bond precision'; it measures whether the postprocessor sampled a plausible bond length. This inflates the headline cov-docking baseline numbers and weakens the claim that the benchmark rigorously evaluates covalent docking accuracy. The reconstruction risk identified by the reader is real but hypothetical; the RMSD(IB) issue is an internal inconsistency demonstrable from the paper's own reported ablations.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CovDocker, a benchmark for covalent drug design that decomposes covalent docking into three tasks: reactive location prediction, covalent reaction prediction, and covalent docking pose prediction. The authors construct a dataset of 2,754 complexes from CovPDB and CovBinderInPDB, provide time-based splits, adapt Uni-Mol and Chemformer as baselines, introduce an auxiliary covalent loss and a postprocessing step, and release the preprocessed data, code, and trained weights. The paper claims that this provides a comprehensive, rigorous, and reproducible framework for advancing covalent drug design.","tokens_in":21940,"tokens_out":5342,"duration_ms":56333,"significance":"The dataset release and task decomposition are genuinely valuable: 2,754 entries, 22 reaction mechanisms, 10 target amino acids, time-based splits, and publicly available code and weights are a clear step beyond the existing evaluation-only covalent docking benchmarks. If the reconstructed post-reactive labels are reliable, Tasks 1 and 2 provide useful ML-ready testbeds for reactive-site and reaction-product prediction. The open availability of the resource is a concrete strength that should be credited. However, the Task 3 evaluation is weakened by the RMSD(IB) metric and postprocessor issue described below, so the benchmark's claim to rigorously evaluate covalent docking accuracy is not yet fully established.","major_comments":[{"comment":"The RMSD(IB) metric is not a measure of pose accuracy and is almost entirely determined by the postprocessor. In Section 3.4, the postprocessor samples a bond length l' from the dataset-wide normal distribution N(mu, sigma) and moves the bonded ligand atom so that the predicted covalent bond length equals l'. RMSD(IB) as defined in Section 4.3 is the distance between the bonded ligand atom and the bonded protein atom, which is exactly this bond length. Therefore a model that places the warhead atom at any plausible bond length from the reactive residue will score near-perfectly on RMSD(IB) regardless of whether the rest of the pose is correct. The paper's own ablation in Table 7 demonstrates this: rows (b) to (a), postprocessing alone raises RMSD(IB)<0.5 Å from 59.2% to 77.1% while whole-ligand RMSD<2 Å stays at 37.7%; Table 4 shows the same pattern for Ours-p (60.1% to 79.1% with RMSD<2 Å unchanged at 37.2%). Since Table 4 presents RMSD(IB) as the 'covalent bond precision' result, these numbers are mechanical and do not support the claim that the benchmark rigorously evaluates covalent docking. Please make whole-ligand RMSD the primary pose-quality criterion and introduce a warhead-position RMSD that compares the predicted bonded ligand atom with the ground-truth bonded ligand atom after alignment; the bond length can be reported only as a sanity check, not as a pose-accuracy score.","section":"Sections 3.4, 4.3, and Tables 4 and 7"},{"comment":"The post-reactive ligand labels underpin both the Task 2 reaction targets and the Task 3 pose labels, but they are reconstructed heuristically: stable ligand forms from the Chemical Component Dictionary are aligned to PDB coordinates, atoms are deleted or added, bond orders are patched, and hydrogen atoms are added manually. The paper does not provide quantitative validation of these reconstructions. Errors or ambiguities in this step would propagate to the reaction-prediction SMILES and to the docking pose labels, invalidating the reported baseline numbers. Please validate a sample against hand-curated covalent complexes such as the Keseru benchmark, report alignment failure rates and the frequency of each patch type, and assess how much the benchmark results change under alternative reconstruction choices.","section":"Appendix A.2"},{"comment":"The postprocessing step samples l' randomly from a normal distribution at inference, which makes the Ours-p results stochastic and not exactly reproducible without a fixed seed or multiple sampling. Please either fix the random seed, average results over multiple samples, or replace the random sample with a fixed quantile, and report the resulting variance.","section":"Section 3.4 and Table 4 reproducibility"}],"minor_comments":[{"comment":"The text says the pocket is defined as a circle with radius 20 Å around the pocket center; this should be a sphere in three-dimensional space.","section":"Section 3.2"},{"comment":"There is a typo: 'entry numer' should be 'entry number'.","section":"Section 4.4"},{"comment":"The caption contains the typo 'ligand aotm' and should be 'ligand atom'; also, the asterisk for AutoDock4(cov) should be defined in the caption before it appears in the table.","section":"Table 4 caption"},{"comment":"The dataset name 'USTPO' appears to be a typo for 'USPTO'.","section":"Section 5.2"},{"comment":"The main text states that chains exceeding 1,024 amino acids are excluded for Task 1, while Appendix A.2 says the cutoff is 1,022 residues; please clarify which value was actually used.","section":"Section 4.1 and Appendix A.2"},{"comment":"The notation is dimensionally unclear: d_ij is described as a scalar bond distance while D_inter is a distance map matrix; please specify whether the loss is computed on the single matrix entry corresponding to the covalent pair or aggregated over all ligand-pocket pairs.","section":"Equation (5)"}],"recommendation":"major_revision","confidential_remarks":"The dataset and code release are substantial contributions, and the central benchmark idea is worth publishing. My main concern is the circularity of RMSD(IB): the postprocessor directly sets the quantity being measured, and the paper's own ablation makes that visible. This is fixable by re-centering the docking evaluation on whole-ligand RMSD and adding a true warhead-position metric, so I would not reject. I would also ask the area chair to ensure the authors address the reconstruction validation concern in Appendix A.2, since the benchmark labels depend on it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [colleague],\n\nThe one thing to know: CovDocker is a genuinely useful resource, but the paper's headline cov-docking metric doesn't measure what it claims. The postprocessing step moves the bonded ligand atom to a bond length sampled from the global distribution, so the RMSD(IB) numbers mostly reflect that mechanical correction. The paper's own ablation shows it: postprocessing alone raises RMSD(IB)<0.5Å from 59.2% to 77.1% (Table 7, a vs b) while whole-ligand RMSD<2Å stays flat at 37.7%. In Table 4, Ours-p jumps from 60.1% to 79.1% on RMSD(IB) with no change in whole-ligand RMSD. So the claim that the auxiliary loss or the benchmark rigorously evaluates covalent bond precision is not supported by that metric.\n\nThat said, the assembled dataset is the real contribution. It takes two existing databases (CovPDB, CovBinderInPDB), merges and cleans them into 2,754 complexes with time-based splits, covers 22 reaction mechanisms and 10 amino acids, and releases preprocessed structures plus code and weights. That is a substantial step for an under-served problem, and the three-task decomposition (reactive location, reaction prediction, cov-docking) is a sensible framing that will help future work. The blind-docking pipeline evaluation is refreshingly honest: Table 5 shows blind performance drops to 0.4% RMSD<3Å because of cumulative errors, which is a useful reality check.\n\nThe soft spots beyond the metric: the ground-truth post-reactive ligands are reconstructed via a heuristic CCD-alignment pipeline (Appendix A.2). That is a real risk — if those labels are wrong, both Task 2 targets and Task 3 poses are wrong. The authors should provide a validation set or manual-curation comparison. Also, the main text and appendix describe different postprocessing rules (random sample vs 10σ threshold), and that inconsistency should be fixed. The evaluation would be stronger if they reported RMSD(IB) with the postprocessor disabled, or reported the postprocessor's effect separately for every result.\n\nWho is this for? Anyone building or testing deep learning models for covalent docking. The dataset and task definitions deserve to be used and cited, but the evaluation claims should be treated with caution until the metric issue is addressed. I'd take it for peer review rather than desk reject it, but I'd send it back with a request to separate postprocessor effects from learned pose accuracy.","headline":"The dataset is the contribution; the headline RMSD(IB) metric is largely measuring the postprocessor, not the learned pose.","tokens_in":22486,"tokens_out":4215,"would_cite":true,"duration_ms":38648,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CovDocker decomposes covalent docking into three learnable tasks and releases 2,754 curated complexes with baselines.","keywords":["molecular docking","covalent drug design","deep learning benchmarks","protein-ligand interaction","covalent docking","reactive site prediction","covalent reaction prediction","pose prediction"],"falsifier":"Reconstruct the post-reactive ligand for a random sample of CovDocker entries directly from the raw PDB LINK records and the published preprocessing rules, without consulting the released labels; if the resulting SMILES and bond orders fail to match the release on even a few entries, the benchmark targets inherit those errors. A second check is to retrain the Task 3 model without the $L_{cov}$ loss and compare RMSD (IB) on the same test split: the paper reports a large gap at tight thresholds, so an independent run should reproduce that gap if the loss is doing the claimed work.","tokens_in":21391,"feed_emoji":"💊","tokens_out":8594,"duration_ms":79129,"temperature":0.7,"pith_summary":"Most docking tools and deep-learning models treat protein–ligand binding as non-covalent, so they miss the covalent bond that makes many drugs bind strongly and persistently. This paper argues that covalent docking should be decomposed into three separate learnable steps: locating the reactive pocket and residue, predicting the product of the covalent reaction, and predicting the docked pose of the post-reactive ligand. To support that decomposition, it assembles 2,754 covalent complexes from two public structural databases, reconstructs the post-reactive ligands, and releases preprocessed data, code, and trained weights. It also adapts existing models—Uni-Mol for site and pose prediction, Chemformer for reaction prediction—as baselines, including an auxiliary loss that keeps the covalent bond at the shortest ligand–pocket distance. If the benchmark is sound, deep learning can be trained and compared on covalent docking in a reproducible way, which would speed up the design of selective covalent inhibitors.","feed_headline":"Covalent docking gets a three-task benchmark and open data","feed_subtitle":"Reactive-site, reaction, and pose baselines on time-split data make covalent drug design trainable and reproducible.","key_machinery":"The machinery is the task decomposition plus the data pipeline that feeds it. CovDocker converts PDB LINK records and ligand metadata from CovPDB and CovBinderInPDB into a consistent set of pre-reactive SMILES, post-reactive SMILES, and docked structures; the post-reactive ligand is reconstructed by aligning the stable Chemical Component Dictionary form to the PDB coordinates, deleting extra atoms, patching bond orders, and adding hydrogens. On top of this data, each task has a defined model: a residue-level Uni-Mol encoder with cross-attention predicts the pocket center and reactive residue; Chemformer, a Transformer sequence model, predicts reaction products; and a finetuned Uni-Mol docking model with the covalent-distance auxiliary loss $L_{cov}$ and optional bond-length postprocessing predicts poses. The time-based split is the guard against data leakage.","core_discovery":"On its own terms, the paper’s central claim is that covalent docking is not one monolithic prediction problem but three coupled tasks, and that a benchmark built from PDB-derived covalent complexes can make each task tractable for deep learning. Task 1 predicts the pocket center and the reactive residue; Task 2 predicts the post-reactive product SMILES from the pre-reactive ligand and reactive residue; Task 3 predicts the docked pose of the product within a pocket, with a loss term $L_{cov} = \\mathrm{ReLU}(d_{ij} - D_{inter})$ that biases the predicted covalent bond toward the shortest inter-molecular distance. The paper reports that this decomposition, together with time-based splits of 2,308 training, 223 validation, and 223 test entries, yields baselines that beat traditional non-covalent and covalent docking tools on pose accuracy, and that the covalent constraint plus a bond-length postprocessing step nearly saturates the new covalent-bond RMSD metric.","pith_inferences":["If the benchmark labels hold up, the same three-task decomposition could be applied to other irreversible binding modalities, such as covalent protein–DNA or protein–carbohydrate cross-links, by swapping the residue vocabulary and reaction templates.","A direct next experiment is to feed oracle reactive-site and oracle reaction labels into the pose model and measure how much each stage’s error contributes to the final blind-docking score, which would locate where the pipeline loses the most accuracy.","The generality of the $L_{cov}$ loss could be tested on a non-covalent docking benchmark by imposing an artificial close-contact anchor between ligand and pocket, to see whether the gain comes from a chemistry-specific signal or from a generic short-distance prior."],"forward_implications":["A machine-learning model can now be trained on covalent docking from a single downloadable dataset, with held-out splits chosen by discovery date rather than by curation quality.","Covalent bond quality becomes directly measurable: the new RMSD (IB) metric checks whether the bonded ligand atom lands on the bonded protein atom, not just whether the whole pose is close.","Reaction prediction on this dataset is harder than on standard small-molecule reaction sets, so gains scored on CovDocker may transfer to real covalent inhibitor chemistry.","The blind-docking pipeline result (0.4% under 3 Å RMSD versus 41.3% for site-specific docking) shows that the three stages must be improved jointly rather than in isolation."],"supporting_citations":[{"why":"Supplies the larger portion of covalent protein–ligand complexes, with reaction mechanism and amino-acid annotations that define the benchmark’s coverage.","marker":"[17]"},{"why":"Supplies the second data source and the covalently bonded ligand reconstruction workflow that the paper adapts for post-reactive SMILES generation.","marker":"[21]"},{"why":"Provides the Uni-Mol architecture and pretrained checkpoints that are adapted for Task 1 reactive location and Task 3 covalent docking.","marker":"[64]"},{"why":"Provides the Chemformer sequence model used as the Task 2 reaction-prediction baseline.","marker":"[24]"},{"why":"Provides AutoDock4 (cov), the traditional covalent docking method used as a comparison baseline in the Task 3 evaluation.","marker":"[5]"},{"why":"Defines the Cys-focused Keseru evaluation set that the new time-based test set is designed to complement with more diverse and realistic data.","marker":"[50]"},{"why":"Supplies the time-based training/validation/test split strategy adopted to prevent data leakage.","marker":"[54]"},{"why":"Provides the Chemical Component Dictionary ligand forms used to align and reconstruct post-reactive ligands in preprocessing.","marker":"[14]"},{"why":"Supplies the PDB structure files used to supplement entries where the source databases lacked coordinate data.","marker":"[4]"}],"fun_headline_variants":["Covalent docking broken into three deep-learning tasks","New benchmark makes covalent docking trainable and split","Reactive site, reaction, pose: CovDocker decomposes docking","Open covalent docking benchmark with time-split baselines","CovDocker: three tasks, one benchmark for covalent binding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire benchmark depends on the hand-reconstructed post-reactive ligand labels: for each complex, the authors align the stable ligand form from the Chemical Component Dictionary to PDB coordinates, delete or add atoms, patch bond orders, and add hydrogens by hand, so if those reconstructed products are wrong or ambiguous, the reaction-prediction targets and the pose labels built from them are wrong.","fun_headline_variants_meta":{"raw":{"variants":["Covalent docking broken into three deep-learning tasks","New benchmark makes covalent docking trainable and split","Reactive site, reaction, pose: CovDocker decomposes docking","Open covalent docking benchmark with time-split baselines","CovDocker: three tasks, one benchmark for covalent binding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00016,"raw_usage":{"total_tokens":1224,"prompt_tokens":932,"completion_tokens":292,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":211}},"tokens_in":548,"tokens_out":292,"duration_ms":3665,"temperature":1.0,"reasoning_tokens":211,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:34:27.998482+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reconstruct the post-reactive ligand for a random sample of CovDocker entries directly from the raw PDB LINK records and the published preprocessing rules, without consulting the released labels; if the resulting SMILES and bond orders fail to match the release on even a few entries, the benchmark targets inherit those errors. A second check is to retrain the Task 3 model without the $L_{cov}$ loss and compare RMSD (IB) on the same test split: the paper reports a large gap at tight thresholds, so an independent run should reproduce that gap if the loss is doing the claimed work.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the larger portion of covalent protein–ligand complexes, with reaction mechanism and amino-acid annotations that define the benchmark’s coverage."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the second data source and the covalently bonded ligand reconstruction workflow that the paper adapts for post-reactive SMILES generation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Uni-Mol architecture and pretrained checkpoints that are adapted for Task 1 reactive location and Task 3 covalent docking."},{"cited_title":"Goodsell, and Arthur J","cited_arxiv_id":null,"evidence_quote":"Provides AutoDock4 (cov), the traditional covalent docking method used as a comparison baseline in the Task 3 evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Cys-focused Keseru evaluation set that the new time-based test set is designed to complement with more diverse and realistic data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the time-based training/validation/test split strategy adopted to prevent data leakage."},{"cited_title":"Berman, and John D","cited_arxiv_id":null,"evidence_quote":"Provides the Chemical Component Dictionary ligand forms used to align and reconstruct post-reactive ligands in preprocessing."}],"review_version":1}