{"id":"272baac2-97a3-43b9-9c6f-430d8c6224f1","arxiv_id":"2411.08306","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A synthesizability metric that reconstructs a molecule from its predicted synthetic route, called the round-trip score, beats search success rate and ranks seven generative drug-design models.","lead":"This paper proposes a round-trip score that tests whether a molecule from a drug-design model can be recreated by retrosynthetic planning followed by forward reaction prediction. Seven generative models are benchmarked with this score, which the authors argue is more reliable than earlier search-success metrics.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Round-trip score recall may reflect memorization: the forward model is trained on the same USPTO reactions used to define feasible routes, so Table 1 does not establish generalization to generated molecules.","rationale":"The reader's weakest-assumption analysis identifies the overlap between the forward model's training set and the reaction network used to label test routes as the key threat to the central claim. I agree: this is the single most load-bearing concern. The high recall in Table 1 is expected if the forward model has memorized the exact reactions that constitute feasible routes, so it does not by itself demonstrate that the round-trip score can evaluate novel molecules proposed by generative models. The paper includes a manual evaluation subset that partially addresses out-of-distribution feasibility, but the subset is small, unblinded, and not reported separately; the aggregate numbers suggest a meaningful recall drop (79.5% vs 97.0%), which is consistent with memorization. The benchmark in §4.2 is a useful contribution and the relative ranking of generative models may still be informative, but the absolute round-trip percentages should be interpreted cautiously until the leakage concern is resolved. Since the reader already issued a conditional verdict, my read does not change that verdict; it does, however, sharpen the specific condition that should be met: demonstrate non-memorization via a disjoint reaction split or a separate report of the manual subset. I do not see grounds for rejection, because the proposed metric is reasonable and the benchmark is valuable, but acceptance should remain conditional on this validation.","tokens_in":14385,"tokens_out":6316,"duration_ms":60927,"concrete_test":"Re-run the §4.1 evaluation with a forward model trained on a reaction split that excludes all reactions appearing in the test reference routes (e.g., split the 916k reactions by reaction class or by product scaffold, ensuring no test route reaction is in training), and report recall separately for reference-matched versus manually-labeled feasible routes. If recall on reference-matched routes drops substantially or the gap between the two subsets persists, the reported 93.9% recall is inflated by memorization and the generalization claim fails. A minimal version using existing data: report the recall on the 112 manually labeled feasible routes separately from the 526 reference-matched routes, with confidence intervals.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the round-trip score 'surpasses the search success rate' (Table 1) rests on the forward model's ability to distinguish feasible from infeasible routes. The evaluation protocol in §4.1 creates a circularity: the reaction network used to define feasible routes is built from the same cleaned USPTO set on which the forward Transformer is trained ('training it on about 916k reactions'). A test route is labeled feasible if its starting materials match a reference route in the test set; those reference routes are extracted from this same network, so every reaction in them is present in the forward model's training data. Thus the 93.9% recall (510/526 for reference-matched feasible routes) largely measures memorization, not chemical generalization. The manual subset is meant to break this circularity, but only 112 routes are labeled feasible manually (44 CAS-documented + 68 expert-judged), and the paper does not report their recall separately; aggregating the given numbers gives 89/112 ≈ 79.5%, notably below the 97.0% on reference-matched routes. Moreover, the manual labels were assigned by the authors using CAS and 'domain expertise' without a blinded protocol, so label noise is uncontrolled. Consequently, the empirical support for using the round-trip score on out-of-distribution generated molecules—the actual use case in §4.2—is weak, and the absolute top-k percentages in Table 2 lack a validated calibration.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a round-trip score S(m) = Sim(m, f(g(m))) for evaluating the synthesizability of molecules generated by drug-design models. A retrosynthetic planner g first proposes a synthetic route from purchasable starting materials; a forward reaction prediction model f then simulates the route in the forward direction; the Tanimoto similarity between the original molecule and the reconstructed molecule is the score. The authors validate the score on a test set of 1,068 molecules drawn from a reaction network built from about 916k cleaned USPTO reactions, labeling predicted routes as feasible either by matching starting materials to reference routes or by manual CAS/expert evaluation. They report that the round-trip score has precision 76.4%, recall 93.9%, and F1 84.2%, compared with search success rate precision 62.1% (Table 1). They then benchmark seven structure-based drug design generative models and report top-k round-trip success percentages (Table 2), concluding that the metric is more reliable than the search success rate and that existing models still generate mostly molecules without feasible synthetic routes.","tokens_in":14645,"tokens_out":4878,"duration_ms":50440,"significance":"If the validation were sound, the round-trip score would be a valuable, parameter-free evaluation metric that combines retrosynthetic planning and forward reaction prediction, directly addressing a known weakness of SA-score-based and search-success-based synthesizability measures. The paper is also useful in assembling a large-scale benchmark of 1,068 test molecules and applying the metric to seven SBDD models, producing a concrete ranking. The score itself has no fitted parameters beyond the trained retrosynthesis and forward models, which is a genuine strength. However, the empirical validation is the load-bearing part of the paper, and that validation has a training/test overlap problem that substantially weakens the central claim. The metric idea is defensible and the comparison with search success rate is meaningful, but the current evidence does not yet establish generalization to out-of-distribution generated molecules.","major_comments":[{"comment":"The forward reaction model is trained on 'about 916k reactions' from the cleaned USPTO set, and the same set is used to build the reaction network from which the reference routes and the feasibility labels are derived. A predicted route is labeled feasible when its starting materials match a reference route in the test set, so every reaction in that reference route is present in the forward model's training data. The 93.9% recall (510/526) therefore reflects, at least in part, the forward model's ability to recall reactions it has already seen, not its ability to simulate novel chemistry. This is load-bearing for the claim that the round-trip score 'surpasses the search success rate' as a general synthesizability metric. Please retrain the forward model on reactions excluding all reactions that appear in the test reference routes (or otherwise report recall on test routes whose reactions were not in the training set), and report the corresponding Table 1 statistics.","section":"Section 4.1, Settings and Evaluation Protocol"},{"comment":"The manually labeled feasible routes provide a partially independent check, but the paper does not report their recall separately. Aggregating the stated numbers gives (36 + 53)/(44 + 68) = 89/112 ≈ 79.5% for the manual subset, which is materially below the 97.0% recall on reference-matched routes. This gap is consistent with the memorization concern raised above and should be reported and discussed. In addition, the manual labels were assigned by the authors using CAS and their own domain expertise without a blinded protocol or inter-annotator agreement, so the label noise is uncontrolled. At minimum, report recall on the manual subset separately and specify the criteria used to decide that a partial CAS match 'is also accurate'.","section":"Section 4.1, Evaluation Protocol"},{"comment":"The absolute top-k percentages in Table 2 are presented as measures of synthesizability for generated molecules, but their calibration depends entirely on the Table 1 validation, which is weakened by the training/test overlap described above. For out-of-distribution generated molecules there are no reference routes to match, so the forward model is being applied in a regime the current experiment does not validate. At most the ranking across models may be supported, and even that assumes error rates are similar across the seven generative models. Please add a calibrated evaluation on a held-out set of novel reactions or generated molecules, and report uncertainty estimates for the Table 2 percentages.","section":"Section 4.2, Table 2"}],"minor_comments":[{"comment":"The term 'data points' is used for molecules; please clarify that the 105,218/1,068/1,068 split is over target molecules, not reactions, and state explicitly how reactions from the test molecules are handled when building the forward model's training set.","section":"Section 4.1, Dataset Construction"},{"comment":"The claim that the scale is 'highly convincing' because 1,068 exceeds 189 by a factor of five is not by itself an argument for reliability; please supplement with confidence intervals or a statistical comparison of the precision/recall estimates.","section":"Section 4.1, Evaluation Protocol"},{"comment":"The notation m_tar&, m_I1&, and m_I2& is used in the figure but not defined in the text; please define the reconstructed intermediates and clarify how they are matched to the original route.","section":"Figure 3"},{"comment":"The statement that the authors 'verify the validity and plausibility of these molecules' is not backed by a description of the filters used; please specify the validity and plausibility criteria.","section":"Section 4.2, Settings"},{"comment":"The conclusion that the SA score 'lacks the ability to differentiate' because the average SA scores are 2.68 and 2.73 would be stronger with a reported standard deviation or significance test, given that the two groups are large.","section":"Section 4.1, Results"}],"recommendation":"major_revision","confidential_remarks":"The central idea is timely and the benchmark is substantial, but the validation protocol has a load-bearing training/test overlap: the forward model is trained on the same reaction set used to construct the feasible-route labels. This is fixable within the scope of the paper by retraining on a non-overlapping split and reporting the manual-subset recall separately. I would support publication after that revision. The manual labeling by the authors should also be described transparently, since it is the only partially independent ground truth in the current design."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The idea is a natural extension of Schwaller's round-trip accuracy to multi-step routes, and the benchmark across seven SBDD models is a useful reference point. But the headline validation result in Table 1 is not trustworthy as reported, because the forward model was trained on the same 916k USPTO reactions used to build the test routes. That makes the 93.9% recall largely a memorization score, not a generalization result.\n\nWhat is actually new: the multi-step round-trip score S(m)=Sim(m, f(g(m))) composed from a retrosynthetic planner and a forward predictor, and the evaluation of seven SBDD generative models on it. The paper is clearly written, the arithmetic is internally consistent, and the comparison with search success rate is a legitimate improvement in principle. The motivation is sound: search success rate is too lenient, and SA score is not route-aware.\n\nThe soft spot is the validation. The paper trains the forward Transformer on 'about 916k reactions' — the same cleaned USPTO set used to construct the reaction network and extract reference routes for the test molecules. So a large fraction of the 'feasible' test routes are composed of reactions the forward model has memorized. The manual subset is supposed to break the circularity, but it is small (112 feasible routes) and was labeled by the authors with CAS and expertise, without a blinding protocol. The paper does not report the recall on that manual subset; if you aggregate the given numbers, it is about 79.5%, well below the 97% on reference-matched routes. The stress-test note gets this right. This means the absolute top-k percentages in Table 2 lack a validated calibration, though the ranking of models is probably informative.\n\nWho should read this: anyone working on synthesizability scoring or generative drug design. It is a fair contribution to the subfield, not a breakthrough. As a referee, I'd send it out, but I would insist the authors retrain or reselect the forward model so that test routes are strictly out-of-distribution, and report recall separately for matched and manually labeled routes. If they fix the leak, the central claim becomes credible. Right now, the concept is fine and the execution is not.","headline":"A sensible round-trip synthesizability metric, but the headline recall number is inflated by a training-set leak that the authors need to fix.","tokens_in":15192,"tokens_out":3278,"would_cite":false,"duration_ms":30791,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes the round-trip score—plan a route backward, simulate it forward, and measure how well the target comes back—and shows it outperforms the search success rate for judging whether generated molecules are synthesizable.","keywords":["synthesizability evaluation","round-trip score","retrosynthetic planning","forward reaction prediction","structure-based drug design","generative models","synthetic accessibility","Tanimoto similarity"],"falsifier":"Retrain the forward model on a cleaned USPTO split that excludes the reactions used to build the test-set reference routes, recompute round-trip scores on the same 1,068 targets, and check whether the 93.9% recall and the ranking of the seven generative models persist; if recall collapses or rankings change, the reported discrimination depends on memorization rather than generalizable forward prediction. A complementary wet-lab check would be to take a sample of generated molecules with round-trip score 1, follow the predicted routes in a real laboratory, and measure how many actually produce the target.","tokens_in":141,"feed_emoji":"🧪","tokens_out":7413,"duration_ms":146416,"temperature":0.7,"pith_summary":"The paper proposes a new metric, the round-trip score, for deciding whether a molecule generated by a drug-design model can actually be synthesized. The score is computed in three stages: a retrosynthetic planner finds a route backward from the molecule to purchasable starting materials, a forward reaction model tries to simulate that route forward from those materials, and the Tanimoto similarity between the original molecule and the simulated product becomes the score. The paper argues this score distinguishes feasible from infeasible routes far better than the search success rate, reporting 76.4% precision versus 62.1% on a 1,068-molecule test set, and shows the SA score cannot separate the two groups. It then uses the metric to benchmark seven structure-based drug-design generative models, finding large differences in the share of molecules with at least one reproducible route.","feed_headline":"Round-trip score beats search rate for feasible synthesis routes","feed_subtitle":"The metric checks whether a designed molecule's predicted route can actually rebuild it.","key_machinery":"The load-bearing object is the round-trip score $S(m) = \\mathrm{Sim}(m, m')$, where $m'$ is the product obtained by running a retrosynthetic planner backward from $m$ to starting materials and then a forward reaction model forward from those starting materials back to a molecule. Tanimoto similarity here is the Jaccard similarity between molecular fingerprint bit vectors, so a score of 1 means the forward simulation reproduced the target exactly; lower scores mean the simulated route leads somewhere else. The forward model is a Transformer decoder trained on about 916k cleaned USPTO reactions, and the retrosynthetic planner is the template-based Neuralsym with beam search. The argument works because the forward model acts as a wet-lab proxy: if the planner proposes a reaction that is not chemically faithful, the forward simulation will not return the target, and the route is penalized. Reference routes from the same reaction network provide the feasible/infeasible labels used to measure the metric's discrimination.","core_discovery":"The central claim is that synthesizability can be quantified by round-tripping: if the starting materials of a predicted synthetic route can be fed through a learned forward reaction model to reproduce the target molecule, the route is feasible; if not, it is not. Concretely, the paper defines $S(m) = \\mathrm{Sim}(m, f_\\Phi(g_\\Theta(m)))$, where $g$ is a retrosynthetic planner, $f$ is a forward reaction predictor, and $\\mathrm{Sim}$ is Tanimoto similarity. On 1,027 molecules for which the planner produced routes, the round-trip score identifies feasible routes with 76.4% precision and 93.9% recall (F1 84.2%), while the search success rate treats all 1,027 as found and has 62.1% precision. The paper also reports that the SA score gives nearly identical averages for feasible and infeasible routes (2.68 vs 2.73), so it cannot discriminate. Applied to 10,000 molecules from seven generative models, the round-trip score ranks Pocket2Mol highest, with 22.05% of its molecules having at least one route among the top five that the forward model can reproduce, versus 2.87% for LiGAN.","pith_inferences":["The reliability ceiling of the round-trip score is set by forward-model accuracy, so an explicit out-of-distribution benchmark—evaluating the forward model on reactions with unseen functional groups—would quantify how much of the reported recall transfers to truly novel molecules.","The same round-trip procedure could be turned into a training reward for generative models, so molecules are optimized not just for docking scores but for surviving forward simulation; the paper stops at evaluation.","The score could also be used to compare route quality continuously, rather than thresholding at 1, by retaining the Tanimoto value as a graded measure of how close the simulated route gets to the target.","The observed near-zero round-trip rates for most models (top-5 under 22%) suggest that current SBDD training objectives ignore synthesizability; a round-trip filter during sampling could be a cheap way to boost the fraction of synthesizable hits."],"forward_implications":["The round-trip score can replace SA score and search success rate as the primary synthesizability check for generative drug-design models, since it directly verifies route reproducibility instead of relying on structural heuristics or mere route existence.","A model's synthesizability is measured by the fraction of its generated molecules for which at least one of the top five predicted routes reproduces the molecule, so multi-route evaluation gives a fuller picture than single-route success.","Because the score is computed entirely from reaction data, expanding reaction databases should make the metric stricter and more reliable without changing the evaluation protocol.","Hallucinated retrosynthetic reactions are penalized automatically: a route counts only if the forward model can walk it back to the target, fixing a known weakness of the search success rate."],"supporting_citations":[{"why":"Defines the SA score, the heuristic synthesizability measure the paper argues cannot discriminate feasible from infeasible routes.","marker":"Ertl & Schuffenhauer, 2009"},{"why":"Supplies the USPTO reaction dataset (about 916k reactions) used to train both the retrosynthetic and forward models and to construct reference routes for ground-truth feasibility labels.","marker":"Lowe, 2014"},{"why":"Provides the template-based Neuralsym retrosynthesis model used as the planner whose predicted routes the round-trip score evaluates.","marker":"Segler & Waller, 2017"},{"why":"Introduces matching-based route evaluation against literature reference routes, which the round-trip score extends to molecules without reference routes.","marker":"Liu et al., 2023b"},{"why":"Reports forward reaction prediction models with top-1 accuracy exceeding 90%, motivating use of a reaction model as a wet-lab proxy.","marker":"Bi et al., 2021"},{"why":"Defines the retrosynthetic planning benchmark and search success rate evaluation that the paper compares against and criticizes as overly lenient.","marker":"Chen et al., 2020"}],"fun_headline_variants":["Round-trip synthesis score beats search rate for feasibility","Retrosynthesis plus forward reaction predicts synthesizability","Feasible routes found via round-trip score, not search","Round-trip score outperforms SA score for synthesis","Generative models ranked by round-trip synthesizability"],"cache_read_input_tokens":17280,"weakest_assumption_plain":"The metric is only as reliable as the forward reaction model, which is assumed to predict wet-lab outcomes on newly designed molecules even though it was trained on about 916k literature reactions; if that model memorizes training reactions or fails on out-of-distribution molecules, a high round-trip score will not guarantee a real synthesis.","fun_headline_variants_meta":{"raw":{"variants":["Round-trip synthesis score beats search rate for feasibility","Retrosynthesis plus forward reaction predicts synthesizability","Feasible routes found via round-trip score, not search","Round-trip score outperforms SA score for synthesis","Generative models ranked by round-trip synthesizability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000268,"raw_usage":{"total_tokens":1629,"prompt_tokens":965,"completion_tokens":664,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":586}},"tokens_in":581,"tokens_out":664,"duration_ms":6130,"temperature":1.0,"reasoning_tokens":586,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:44:41.079725+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the forward model on a cleaned USPTO split that excludes the reactions used to build the test-set reference routes, recompute round-trip scores on the same 1,068 targets, and check whether the 93.9% recall and the ranking of the seven generative models persist; if recall collapses or rankings change, the reported discrimination depends on memorization rather than generalizable forward prediction. A complementary wet-lab check would be to take a sample of generated molecules with round-trip score 1, follow the predicted routes in a real laboratory, and measure how many actually produce the target.","supporting_citations":[],"review_version":1}