{"id":"9876e657-e352-4ff6-9ae4-293716d14cad","arxiv_id":"2501.09103","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SQRL trains molecular property models to predict activity differences between similar molecule pairs, improving neural network accuracy in low-data and activity cliff settings.","lead":"This paper introduces SQRL, a training method that predicts the difference in activity between similar pairs of molecules instead of the raw activity of each molecule. It reports consistent gains for neural network models on public and proprietary drug discovery benchmarks, especially on difficult 'activity cliff' pairs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 3 applies the delta model to nearest neighbors beyond the training threshold α and inherits anchor-label noise; Figure 6 does not resolve whether Table 1 gains survive for test molecules without a close, clean training anchor.","rationale":"The reader's weakest assumption matches my main concern. In good faith, the paper's hypothesis is plausible and the empirical pattern in Table 1 is large for most architectures; Figure 1 supports the thresholding mechanism for MLP, and the KNN baseline shows SQRL is not just the anchor. However, the inference rule Eq. 3 is the bridge from training pairs (d <= α) to deployment (any nearest neighbor). The paper does not prove that bridge. Figure 6 is close, but pooled distance bins and lack of confidence intervals leave the >α regime unquantified. If the >α regime contains many test molecules and SQRL's gains there are negative, the low-data generalization claim is overstated. This is a reproducibility and validation gap, not a demonstrated failure, so UNCHANGED is the right verdict. I would not reject because the central tables are internally consistent and the proposed test may well confirm the method. I also considered the lack of random-seed reporting, but the per-architecture gains are mostly large enough that the main risk is the inference mechanism, not noise.","tokens_in":11674,"tokens_out":10494,"duration_ms":122415,"concrete_test":"Reproduce the MoleculeACE results for all models in Table 1 with 5 seeds, splitting test predictions by whether d(x_new, NN1) <= 0.7 or > 0.7 (Tanimoto), and report Spearman and sample count per bin. Additionally, in the SQRL inference, perturb each anchor y_NN by independent Gaussian noise with sigma = 0.3 (a typical assay error) and recompute Spearman. If SQRL's advantage over Standard disappears or reverses in the >0.7 bin, or if the gains are not robust to anchor noise, then Eq. 3's out-of-threshold extrapolation and anchor assumption are load-bearing for the claimed improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim rests on the inference rule in Eq. 3. The training distribution Drel (Eq. 1) contains only pairs with d(xi, xj) <= α, but inference always uses y_NN + f(g(x_NN) − g(x_new)) with n = 1. For any test molecule whose nearest training neighbor is farther than α, f is asked to extrapolate on a representation difference that never appeared in training, and the prediction directly inherits the experimental noise of y_NN. A standard absolute model does not carry that anchor-noise term. Figure 6 partially addresses this by showing Spearman as a function of nearest-neighbor distance, but it pools over tasks and does not report how many test molecules fall beyond α, nor the anchor noise level per task. If the majority of test points have NN distance > α, the Table 1 gains could come from the oracle-like observed anchor y_NN plus in-distribution local pairs, not from learned relative differences. The paper's stated limitation (availability of meaningful distance metrics) does not cover this failure mode, which is the most direct threat to the 'low-data generalization' claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Similarity-Quantized Relative Learning (SQRL), a framework in which molecular activity prediction is reformulated as learning relative activity differences between structurally similar molecule pairs. Training pairs are restricted by a distance threshold α on a molecular similarity metric, and test predictions are formed by adding the predicted difference to the activity of the nearest training molecule. The authors benchmark SQRL against standard absolute-value training across 30 MoleculeACE tasks, the MoleculeACE-Cliff subset, and five proprietary internal targets, using tree-based baselines, MLPs, GNNs, and transformer-based models. The central empirical claim is that SQRL consistently and substantially improves Spearman rank correlation for all deep learning architectures tested, with the largest gains for models that perform poorly under standard training, while not helping tree-based baselines. The paper also provides a mechanism analysis showing that SQRL's improvements concentrate on test points with nearby training neighbors.","tokens_in":11949,"tokens_out":2550,"duration_ms":28607,"significance":"If the results hold, the paper makes a useful empirical contribution: a simple, architecture-agnostic training reformulation that consistently improves ranking performance for neural molecular property predictors, with particular relevance to activity-cliff tasks and low-data regimes. The strengths of the paper are its breadth (multiple public benchmarks plus proprietary data), the inclusion of negative results for tree-based models, and the mechanistic evidence in Figure 6 that gains are localized to test points with close training neighbors. The framework is not circular in the sense that the pairwise training objective differs from the absolute-value evaluation metric, and the main gains are not forced by construction. However, the central claim rests on two load-bearing assumptions that are not fully validated: that α=0.7 is selected without test-set leakage, and that the nearest-neighbor anchoring step in Eq. (3) remains valid when the nearest neighbor lies outside the training-pair distance threshold α. These issues require additional analysis before the reported gains can be taken at face value.","major_comments":[{"comment":"The inference rule in Eq. (3) uses the nearest training molecule with no restriction on its distance to the test molecule, while the training distribution in Eq. (1) contains only pairs with d(xi,xj) ≤ α. For any test molecule whose nearest training neighbor is farther than α, the model f is asked to extrapolate on a representation difference that never appeared in training, and the prediction directly inherits the noise of the anchor label y_NN. Figure 6 partially addresses this by showing Spearman correlation as a function of nearest-neighbor distance, but it pools across tasks and does not report how many test molecules fall beyond α, nor the anchor-label noise level per task. This is the most direct threat to the paper's 'low-data generalization' claim. The authors should report, per task and in aggregate, the fraction of test points with nearest-neighbor distance > α, and should condition the Table 1 results on whether the nearest neighbor lies within α. If the gains persist only for the within-α subset, the claim that SQRL improves general ranking should be substantially qualified.","section":"Section 3, Eq. (3)"},{"comment":"The threshold α=0.7 used for all main results appears to be selected from the same threshold sweeps shown in Figure 1 and Appendix Figure 5, without any described nested validation or an explicit selection rule. Figure 1 shows MAE as a function of α for MLP models, and Table 1 then reports results at α=0.7; if α was chosen after inspecting test-set performance across thresholds, the reported gains are optimistically biased. The paper states a heuristic (choose α smaller than the average pairwise distance) but does not operationalize it into a reproducible selection procedure. The authors should either provide a nested validation protocol that selects α on held-out tasks or validation folds, or report results for a range of α values and show that the conclusions are robust.","section":"Section 4.2 and Table 1"},{"comment":"The standard and SQRL variants differ not only in training objective but also in hyperparameters, hidden sizes, number of layers, dropout, learning rate, and batch size. For example, the standard MLP uses [256,256] with learning rate 1e-4, while MLP-SQRL uses [512,256] with learning rate 1e-5; similar differences appear for GINE, PNA, and COATI. The paper does not establish that the standard baselines were tuned with comparable effort; the reported Uni-Mol standard Spearman of 0.26 is far below typical published performance for stronger configurations, which raises the concern that the headline gains partly reflect undertrained baselines rather than a benefit of the SQRL objective. The authors should report hyperparameter search budgets for both settings, and ideally include a stronger standard baseline (e.g., fine-tuned Uni-Mol or a published reference value) to confirm that SQRL improves over reasonably tuned standard training.","section":"Appendix A.2, Table 2"}],"minor_comments":[{"comment":"There is a typo in 'Morgan fingeprints' (missing 'r'), and the phrase 'Due to the computationally expensive conformation generation, Uni-Mol-SQRL was evaluated on the following subset' is duplicated across two places in the appendix.","section":"Appendix A.1"},{"comment":"The symbol d is overloaded: it denotes both the input-space distance function in Eq. (1) and the dimensionality of the representation space in the definition of g : X → R^d in Section 3. This makes the notation confusing; one of the two should be renamed.","section":"Section 3, Eq. (1)-(2)"},{"comment":"The top panel of Figure 1 plots MAE but the y-axis label appears incomplete ('MAE ( )'), and the caption does not state which model and features were used for the sweep; Appendix A.5 clarifies that this is an MLP, but the main text should say so.","section":"Figure 1"},{"comment":"The text says the PNA implementation follows Ref. 28 (FS-Mol), but Ref. 28 is a few-shot learning dataset paper and is unlikely to be the correct source for the PNA implementation; the citation should be checked and corrected.","section":"Appendix A.1, PNA"},{"comment":"The paper provides no code repository, data availability statement, or detailed training configuration (e.g., optimizer, early stopping, number of seeds) beyond the hyperparameter table; given the empirical nature of the contribution, sharing code or at least a detailed configuration file would materially strengthen reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical study with a plausible mechanism, but the two main threats—test-set-related α selection and the unconstrained nearest-neighbor anchor at inference—are exactly the kinds of issues that a careful reader will probe. If the authors can show per-task fractions of beyond-α test points and demonstrate that the gains survive for within-α points, the paper would be a reasonable candidate for acceptance. I would also gently nudge the authors to be more cautious in the abstract and conclusion about 'low-data generalization,' since the current experiments do not include a systematic low-data regime analysis beyond the already-small MoleculeACE tasks."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real news here is simple: similarity-thresholded relative difference learning helps deep models on molecular activity prediction, and the help is consistent across architectures and datasets. Table 1 is the paper's backbone, and it holds up — every GNN and transformer gains in Spearman rank correlation on MoleculeACE, MoleculeACE-Cliff, and the internal targets, with the biggest jumps on activity cliffs. Figure 6 gives some mechanism: the gains concentrate on test points with close training neighbors, which is what you'd expect if learning local deltas is the operative effect. That is a genuinely new empirical result, going beyond the indiscriminate pairwise training in Tynes, Wetzel, and Fralish. The paper also cites that prior work fairly and is honest about the assumption that good distance metrics exist. There's no circularity: the pairwise objective is not the same as the ranking evaluation, so the gains are not forced by construction.\n\nThe soft spots are real but addressable. The threshold α = 0.7 appears to be selected from the same sweep shown in Figure 1, and I don't see a nested validation that protects against overfitting to the test set. That's a common problem in this literature, and it needs stating. The bigger issue is the inference rule in Eq. 3: it always uses the nearest training neighbor, even when that neighbor is farther than α. The delta model never saw such large representation differences during training, and the prediction directly inherits the noise of the anchor label. Figure 6 partially covers this by showing Spearman as a function of nearest-neighbor distance, but it pools across tasks and doesn't report how many test points fall beyond α or how noisy the anchors are per task. Without that, the aggregate gains could be driven mainly by the oracle-like anchor for locally dense test molecules, with the learned delta contributing less than the headline implies. Those are fixable with a few extra experiments, not fatal.\n\nThere's also no code or data release and no seed-level variance, only standard deviations across tasks. That makes the numbers harder to trust than they should be.\n\nBottom line: this is a useful paper for anyone working on molecular activity prediction, especially low-data regimes and activity cliffs. It deserves a serious peer review, but I'd want the authors to show α selection via nested validation, report results conditioned on anchor distance, and release artifacts. Not a paradigm shift, but a solid, citable step forward.","headline":"SQRL is a solid empirical extension of pairwise difference learning, with consistent Spearman gains for deep models on activity benchmarks, but the underspecified α selection and the inference rule that applies the delta model to neighbors beyond the training threshold need closer scrutiny before the low-data generalization claim is taken at face value.","tokens_in":692,"tokens_out":1899,"would_cite":true,"duration_ms":35305,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Similarity-thresholded relative difference learning lifts neural molecular activity prediction across every tested architecture.","keywords":["molecular activity prediction","relative difference learning","similarity threshold","graph neural networks","activity cliffs","low-data drug discovery","pairwise learning","Tanimoto similarity"],"falsifier":"Split SQRL's test predictions by the distance from each test molecule to its nearest training neighbor; if the reported Spearman gains over standard training disappear or reverse for test molecules whose anchor lies beyond the training threshold α, the anchor-extrapolation assumption fails. A concrete experiment would rerun Table 1 while excluding such test molecules and check whether the gains vanish.","tokens_in":11538,"feed_emoji":"🧪","tokens_out":5140,"duration_ms":51619,"temperature":0.7,"pith_summary":"The paper claims that molecular activity prediction improves when a model is trained to predict the difference in potency between pairs of similar molecules, rather than the absolute potency of each molecule. It introduces SQRL, a framework that builds training pairs from molecules whose structural distance falls below a threshold α, trains a network on representation differences, and at inference anchors a new molecule's prediction to its nearest training neighbor's observed activity. On 30 MoleculeACE activity tasks, the activity-cliff subset, and five proprietary drug discovery targets, every tested deep learning architecture showed consistent Spearman rank-correlation gains, with the largest jumps for models that start weak, for example Uni-Mol rising from 0.26 to 0.69. The paper argues this works because structurally similar pairs carry the most informative local signal in low-data, noisy settings.","feed_headline":"Similar-pair difference training lifts neural activity ranking","feed_subtitle":"Every tested deep network gains Spearman rank correlation; models that start weakest gain the most.","key_machinery":"The load-bearing mechanism is the similarity-quantized relative representation: a paired dataset restricted by d(xi, xj) ≤ α and a model that consumes the representation difference g(xi) − g(xj). The threshold α controls a quantity-versus-relevance trade-off; the paper proposes selecting it from the training distance distribution, choosing a value smaller than the average pairwise distance, and shows that MAE improves as α grows to a point and then degrades as more dissimilar pairs are added. At inference the same distance function selects the anchor molecule, converting a global regression problem into a local, anchored one.","core_discovery":"The paper's central discovery is that reformulating molecular activity prediction as similarity-thresholded relative difference learning converts weak neural predictors into competitive ones. Given a distance function d and threshold α, SQRL trains on the paired dataset Drel = {((xi, xj), yi − yj) | d(xi, xj) ≤ α}, minimizing mean squared error over f(g(xi) − g(xj)). At test time a new molecule xnew is assigned the prediction y_new = (1/n) Σ_{xi ∈ NNn(xnew)} (yi + f(g(xi) − g(xnew))). The empirical result is that this objective improves ranking accuracy and mean absolute error for MLPs, graph neural networks, and transformer-based models across public and internal activity datasets, while tree-based and nearest-neighbor baselines are not helped. The improvement is largest on activity-cliff molecules, where structurally similar compounds have very different activities, indicating that the method captures local structure-activity relationships that absolute regression objectives miss.","pith_inferences":["SQRL's inference-time anchor, the nearest training neighbor, is used even when that neighbor lies beyond the training threshold α; a natural extension is to condition predictions on anchor distance or to abstain when no sufficiently close anchor exists.","The method's architecture-agnostic formulation suggests it could transfer to other low-data molecular endpoints, such as ADMET or solubility prediction, with any embedding-based similarity metric.","Because tree models are not helped by the difference representation, concatenating full molecular representations with difference representations may extend SQRL's gains to non-neural models, a possibility the paper hints at but does not test."],"forward_implications":["Neural networks trained with SQRL show consistent Spearman rank-correlation gains across all tested architectures and datasets, with the weakest models gaining the most: Uni-Mol from 0.26 to 0.69, GINE from 0.33 to 0.76, MolCLR from 0.35 to 0.77, and MLP from 0.32 to 0.73 on MoleculeACE.","SQRL produces the largest gains for test molecules close to training data, while roughly maintaining standard performance for more distant points, as shown in the distance-conditioned analysis.","The improvements transfer to proprietary internal drug discovery targets, not just public benchmarks, supporting real-world applicability.","Tree-based baselines such as XGBoost, random forest, and k-nearest neighbors do not benefit from SQRL, which the paper attributes to the simple difference-fingerprint representation discarding the rest of the molecular structure.","Distance metrics with left-skewed or high-kurtosis pairwise distance distributions, such as Tanimoto distance on Morgan fingerprints and COATI embeddings, give the best results under SQRL."],"supporting_citations":[{"why":"Motivates relative structure-activity reasoning from matched molecular pair analysis, the conceptual basis for learning potency differences between similar compounds.","marker":"[15]"},{"why":"Supplies the MoleculeACE dataset of 30 ChEMBL activity tasks and the activity-cliff subset used for all main benchmark comparisons.","marker":"[23]"},{"why":"Establishes the all-pairs pairwise difference regression setting that SQRL's similarity thresholding is designed to improve upon.","marker":"[33]"},{"why":"Shows that paired-compound deep learning improves ADME property prediction, the direct prior approach that SQRL extends with similarity thresholds.","marker":"[34]"},{"why":"Provides the MolCLR contrastive GNN embeddings used both as a learned representation and as a distance metric in the benchmarks.","marker":"[42]"},{"why":"Provides the COATI embedding model and COATI distance metric, one of the best-performing similarity measures in the threshold analysis.","marker":"[43]"},{"why":"Provides the Uni-Mol 3D transformer embeddings, the model showing the largest SQRL rank-correlation improvement.","marker":"[45]"}],"fun_headline_variants":["Similarity-paired relative learning sharpens activity prediction","Relative difference objective lifts molecular activity ranking","SQRL: training on similar pairs improves neural predictors","Weak models gain most from similarity-quantized pair training","Reformulating activity prediction as relative differences helps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The test-time prediction assumes that the nearest training neighbor's measured activity is a reliable anchor and that the model trained on pairs within distance α can be extrapolated to that neighbor even when the neighbor lies farther than α.","fun_headline_variants_meta":{"raw":{"variants":["Similarity-paired relative learning sharpens activity prediction","Relative difference objective lifts molecular activity ranking","SQRL: training on similar pairs improves neural predictors","Weak models gain most from similarity-quantized pair training","Reformulating activity prediction as relative differences helps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1321,"prompt_tokens":841,"completion_tokens":480,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":457,"completion_tokens_details":{"reasoning_tokens":407}},"tokens_in":457,"tokens_out":480,"duration_ms":5143,"temperature":1.0,"reasoning_tokens":407,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:10:04.255736+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Split SQRL's test predictions by the distance from each test molecule to its nearest training neighbor; if the reported Spearman gains over standard training disappear or reverse for test molecules whose anchor lies beyond the training threshold α, the anchor-extrapolation assumption fails. A concrete experiment would rerun Table 1 while excluding such test molecules and check whether the gains vanish.","supporting_citations":[{"cited_title":"Exposing the limitations of molecular machine learning with activity cliffs","cited_arxiv_id":null,"evidence_quote":"Supplies the MoleculeACE dataset of 30 ChEMBL activity tasks and the activity-cliff subset used for all main benchmark comparisons."},{"cited_title":"Twin neural network regression","cited_arxiv_id":null,"evidence_quote":"Shows that paired-compound deep learning improves ADME property prediction, the direct prior approach that SQRL extends with similarity thresholds."},{"cited_title":"Uni-Mol: A universal 3D molecular representation learning framework","cited_arxiv_id":null,"evidence_quote":"Provides the Uni-Mol 3D transformer embeddings, the model showing the largest SQRL rank-correlation improvement."}],"review_version":1}