Pith. sign in

REVIEW 4 major objections 4 minor 23 references

CaliciBoost: Performance-Driven Evaluation of Molecular Representations for Caco-2 Permeability Prediction

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An AutoGluon-based pipeline called CaliciBoost reports state-of-the-art Caco-2 permeability prediction (MAE 0.2560) and claims 3D descriptors add value.

desk verdict A useful benchmark and a newly curated OCHEM dataset, but the headline MAE and the 15.73% 3D claim are test-set-fitted minima that should not be used as stated. read the letter →

arxiv 2506.08059 v1 pith:FWZ5P5KN submitted 2025-06-09 q-bio.QM cs.AIcs.LG

Huong Van Le , Weibin Ren , Junhong Kim , Yukyung Yun , Young Bin Park , Young Jun Kim , Bok Kyung Han , Inho Choi
show 3 more authors
Jong IL Park Hwi-Yeol Yun Jae-Mun Choi
This is my paper
classification q-bio.QMcs.AIcs.LG
keywords caco-2descriptorsfeaturepermeabilitypredictionrepresentationscaliciboostdrug
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This study tests eight ways of turning a drug molecule's structure into numbers that a machine learning model can use, then predicts a laboratory measure called Caco-2 permeability, which estimates how well a drug would pass through intestinal cells. The authors use two datasets: the standard TDC benchmark with 906 compounds and a larger curated set from OCHEM with 5,481 compounds. For each molecular representation, they train an automated machine learning system called AutoGluon and compare mean absolute error (MAE).

The best number reported is a MAE of 0.2560 on the TDC test set, using a subset of PaDEL descriptors, and the paper calls this model CaliciBoost. The authors also claim that adding 3D descriptors to 2D descriptors reduces MAE by 15.73%. There is a problem with how that best number is produced. The authors rank features, then try keeping the top 1 through 100 or 200 features and watch which MAE on the test set looks best. Choosing the number of features based on test-set results means the test set is used to tune the model, so the reported 0.2560 is optimistic. Also, the 15.73% improvement appears only when comparing PaDEL top features to PaDEL 2D features; for Mordred the improvement is about 10.4%, and using 3D features alone makes predictions worse.

The paper releases code and a curated dataset, which is useful for future benchmarks. The qualitative ranking of representations, such as PaDEL, Mordred, and RDKit working well, is plausible. However, the specific state-of-the-art claim and the 3D benefit claim need re-evaluation with feature selection done inside cross-validation, without touching the test set.

Extended reading notes

Core claim

The load-bearing assertion is that CaliciBoost, trained on top-ranked PaDEL features on the TDC scaffold split, achieves MAE 0.2560, RMSE 0.3224, R2 0.7788, and Pearson r 0.8839, outperforming all previously reported methods, and that adding 3D descriptors to 2D reduces MAE by 15.73% for PaDEL and Mordred (Abstract, Section 3.2.2, Table 1 rows 12-14 and 16-18). If correct, it demonstrates that AutoML plus descriptor selection is the best strategy for Caco-2 prediction on small data.

Load-bearing premise

The evaluation assumes the top-feature count N can be chosen by scanning MAE on the test set without invalidating the reported performance. Section 3.2.2 says models were trained with the top N features, with N ranging from 1 to 100 or 200, and Figures 6 and 7 plot MAE versus N. If N and the feature subset are selected using test-set outcomes, the reported 0.2560 MAE and the 15.73% improvement are fitted values, not unbiased predictions.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript presents a systematic comparison of eight molecular representation types (Morgan, Avalon, ErG, RDKit, MACCS, PaDEL, Mordred, CDDD) for regression of Caco-2 permeability values, using AutoGluon-based AutoML on the TDC benchmark and a custom-curated OCHEM dataset. The authors apply permutation importance and SHAP to select top-ranked features for each representation, retrain with Bayesian optimization, and report that a model named CaliciBoost, trained on top PaDEL features with the TDC scaffold split, achieves MAE 0.2560, RMSE 0.3224, R2 0.7788, and Pearson r 0.8839. They further claim that incorporating 3D descriptors with 2D descriptors reduces MAE by 15.73% for both PaDEL and Mordred, and conclude that AutoML plus descriptor selection is a state-of-the-art strategy for Caco-2 prediction.

Significance. If the results were valid, the paper would provide a practically useful benchmark of molecular representations for ADMET modeling in a data-limited setting, with the notable strength of making code, curated data, and pretrained models publicly available. The two-dataset design, the breadth of representations, and the use of modern AutoML tools are valuable. However, the central quantitative claims are compromised by the evaluation protocol: the number of top features N is selected using test-set outcomes (Section 3.2.2, Figures 6-7), so the reported best MAEs are fitted values rather than unbiased predictions. In addition, the abstract's 15.73% claim is not supported by Table 1 for Mordred, and the comparison underlying the 3D-benefit claim confounds descriptor addition with dimensionality reduction. These issues are correctable with re-analysis, but as presented the headline results do not support the state-of-the-art conclusion.

major comments (4)
  1. [Section 3.2.2, Figures 6-7] The optimal number of top features N is selected by inspecting MAE curves computed on the test set, with no internal validation split described in Sections 2.2.1 or 2.2.4. As a consequence, the 'Top' rows in Table 1 (e.g., PaDEL Top MAE 0.2560 and Mordred Top MAE 0.2613) are minima over N chosen using test outcomes, so these values are fitted rather than unbiased predictions. The state-of-the-art claim in Section 4 and the Abstract is therefore not supported as stated. The authors should either fix N and the feature subset before seeing the test set, or use nested cross-validation to report honest prediction error.
  2. [Abstract and Section 3.2.2] The claim that adding 3D descriptors reduces MAE by 15.73% for both PaDEL and Mordred is not consistent with Table 1. PaDEL 2D (0.3037) to PaDEL Top (0.2560) gives about 15.7%, but Mordred 2D (0.2916) to Mordred Top (0.2613) gives about 10.4%. In addition, '2D' is compared with 'Top from 2D+3D', which confounds the addition of 3D descriptors with dimensionality reduction. The abstract overstates the quantitative result; a controlled comparison (e.g., 2D-top versus 2D+3D-top with matched feature counts) is needed.
  3. [Table 1, rows 13 and 17] The conclusion in Section 3.2.2 that 3D structural information 'plays a significant role' is difficult to reconcile with the reported 3D-only results, which are markedly worse than 2D-only for both PaDEL (MAE 0.4277 vs 0.3037) and Mordred (MAE 0.3883 vs 0.2916). The paper should explain this discrepancy or soften the 3D-benefit claim accordingly.
  4. [Tables 1-2] No error bars, standard deviations, or repeated-seed evaluations are reported for any model configuration, so it is unclear whether differences of 0.01-0.05 in MAE between representations are statistically meaningful. Given that the headline claims concern small margins, the authors should report variability across at least several independent runs or bootstrap resamples of the test set.
minor comments (4)
  1. [Figures 10-11] Figures 10 and 11 are captioned as 'using Mordred descriptors', but the accompanying text in Section 3.2.2 states the experiments cover both PaDEL and Mordred; the PaDEL panels appear to be missing or mislabeled.
  2. [Section 2.2.2] The preprocessing imputes missing pH as 7.4 and removes records with missing temperature; these choices could introduce systematic bias and should be justified or subjected to a sensitivity analysis.
  3. [References] Several references are cited in the text with formats not clearly matched to the reference list (e.g., 'Z. Wu et al. (2019)' and 'H. Moriwaki et al. (2018)'), and the reference list mixes styles; please use a consistent citation format throughout.
  4. [Sections 2.2.4 and 3.2.2] The naming of the model is inconsistent: 'AutoGluon' is used in Section 2.2.4, while the headline results refer to 'CaliciBoost'; the relationship between the AutoGluon ensemble, Bayesian optimization, and the CaliciBoost model should be stated explicitly.

Circularity Check

2 steps flagged · score 6.0 of 10

Reported state-of-the-art MAE is a test-set-fitted minimum: N is chosen by scanning MAE-versus-top-feature-count on the held-out test set, and the 3D-benefit number is computed from that selected configuration.

  1. fitted input called prediction [Section 3.2.2 (Using top features), Figures 6-7; Table 1 rows 11-18; Abstract]
    "we conducted a screening experiment, evaluating model performance using the top N features, where N ranged from 1 to 100 or 1 to 200, depending on the dimensionality of each representation. This approach allowed us to identify the optimal number of informative features required to match or exceed the performance of the full model. Performance trends for each representation (MAE vs. number of top features) are provided in Figure 6 (for TDC dataset) and Figure 7 (for OCHEM dataset)."

    The manuscript defines only train/test splits (Section 2.2.1 gives TDC 728/182; Section 2.2.2 gives OCHEM 4,377/1,095) and never defines an inner validation set for the N-scan. As written, the curves in Figures 6 and 7 are therefore MAE on the test set as N varies, and the 'optimal subset' is the argmin of those test-set curves. Table 1's PaDEL Top MAE 0.2560, Mordred Top MAE 0.2613, and the Abstract's 'best MAE performance' are thus the selected minima of this scan, i.e., fitted values reported as predictions. The SOTA claim (MAE 0.2560) is forced by the selection rule: choose N to minimize test MAE.

  2. fitted input called prediction [Abstract; Section 3.2.2 (3D vs 2D comparison); Table 1 rows 12-14 and 16-18]
    "Furthermore, for both PaDEL and Mordred representations, the incorporation of 3D descriptors resulted in a 15.73% reduction in MAE compared to using 2D features alone, as confirmed by feature importance analysis."

    By construction, the comparison is between all 2D features and a top subset drawn from the combined 2D+3D pool after test-set MAE minimization. The 15.73% figure matches only PaDEL (0.3037 to 0.2560); for Mordred the same comparison gives roughly 10.4% (0.2916 to 0.2613), so the Abstract's 'for both PaDEL and Mordred' is not supported. Since 3D-only models are worse (Table 1: PaDEL 3D MAE 0.4277, Mordred 3D MAE 0.3883), the 3D-benefit claim is not an independent measurement; it is the same test-selected configuration whose MAE was minimized, confounded with dimensionality reduction.

full rationale

CaliciBoost is not equation-level circular and it does not lean on load-bearing self-citations; the descriptor families, AutoML framework, and datasets are external inputs. The circularity is confined to the headline evaluation: only train/test splits are defined, and Section 3.2.2 uses MAE-versus-N curves to choose the top-feature subset that is later labeled 'Top'. As written, those curves are test-set MAEs, so the PaDEL Top MAE 0.2560 and Mordred Top MAE 0.2613 in Table 1 are the minima of a test-set feature-count scan, i.e., fitted values presented as predictions. The abstract's 15.73% 3D-benefit figure is calculated from that same selected PaDEL configuration, is not matched by Mordred (~10.4%), and is confounded with dimensionality reduction. A corrected protocol with an inner validation split could support the qualitative ranking, but the state-of-the-art and 3D-benefit claims as stated are not self-contained predictions.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on two fitted choices: the number of top features selected on test data and the implicit AutoML hyperparameters. No new physical entities are postulated; CaliciBoost is a named pipeline, not an independent hypothesis.

free parameters (4)
  • Number of top features N per representation = Not fully reported per representation; ranges 1 to 100 or 1 to 200
    Chosen by scanning MAE on the test set in Section 3.2.2 and Figures 6-7. This is a free parameter fitted to test data and directly inflates the reported MAE.
  • Feature subset composition = Varies per representation
    Top features selected by permutation importance and SHAP rankings, then refined by test-set performance. The composition is not prespecified and depends on the model and data.
  • AutoGluon best-quality preset and Bayesian optimization hyperparameters = Not specified
    AutoGluon selects algorithms and hyperparameters automatically during training. Without fixed configurations, the reported numbers include model-selection artifacts.
  • OCHEM preprocessing thresholds (pH 7.4, temperature 37 C, A-to-B direction) = Fixed by curation rules
    Section 2.2.2 removes entries with non-standard pH or temperature and assumes missing values reflect standard conditions. These choices change the dataset composition and therefore all downstream results.
assumptions (5)
  • domain assumption TDC scaffold split is a valid unbiased evaluation of generalization to novel structures
    The paper trusts TDC's predefined scaffold split and checks label distribution overlap, but scaffold splits can still overestimate performance when test compounds are close to training structures.
  • domain assumption AutoGluon's best-quality preset and Bayesian optimization produce models that can be compared fairly across representations
    Automated model selection is treated as neutral, but AutoML itself is data-dependent and may favor certain feature types, confounding the representation comparison.
  • domain assumption Feature importance from SHAP and permutation on the trained model identifies features that improve held-out performance
    Section 3.2.2 uses these rankings to select top features. Training-set importance does not guarantee test-set improvement, and here it is combined with test-set screening.
  • ad hoc to paper Missing OCHEM metadata values (direction, pH, temperature) reflect standard assay conditions
    Section 2.2.2 assumes missing direction is A-to-B, imputes missing pH as 7.4, and discards non-37 C entries. If missingness is non-random, the curated dataset is biased.
  • domain assumption 3D descriptors generated by PaDEL and Mordred capture meaningful molecular geometry
    3D descriptor quality depends on conformer generation and protonation states. The paper provides no validation of the 3D structures used.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CaliciBoost: Performance-Driven Evaluation of Molecular Representations for Caco-2 Permeability Prediction." pith.science (2026). https://pith.science/paper/FWZ5P5KN

@misc{pith2026250608059,
  author       = {Pith},
  title        = {Pith review of: CaliciBoost: Performance-Driven Evaluation of Molecular Representations for Caco-2 Permeability Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FWZ5P5KN}},
  note         = {Machine review of arXiv:2506.08059}
}
read the original abstract

Caco-2 permeability serves as a critical in vitro indicator for predicting the oral absorption of drug candidates during early-stage drug discovery. To enhance the accuracy and efficiency of computational predictions, we systematically investigated the impact of eight molecular feature representation types including 2D/3D descriptors, structural fingerprints, and deep learning-based embeddings combined with automated machine learning techniques to predict Caco-2 permeability. Using two datasets of differing scale and diversity (TDC benchmark and curated OCHEM data), we assessed model performance across representations and identified PaDEL, Mordred, and RDKit descriptors as particularly effective for Caco-2 prediction. Notably, the AutoML-based model CaliciBoost achieved the best MAE performance. Furthermore, for both PaDEL and Mordred representations, the incorporation of 3D descriptors resulted in a 15.73% reduction in MAE compared to using 2D features alone, as confirmed by feature importance analysis. These findings highlight the effectiveness of AutoML approaches in ADMET modeling and offer practical guidance for feature selection in data-limited prediction tasks.

Figures

Figures reproduced from arXiv: 2506.08059 by the authors.

Figure 1
Figure 1. Overall workflow for Caco-2 permeability prediction using AutoML and multi-representation molecular features 2.1. Datasets 2.1.1. TDC TDC·Caco2 Wang dataset provided by the Therapeutics Data Commons (TDC) was used to train AutoML models for Caco-2 permeability prediction. This dataset contains 906 compounds with experimentally measured Caco-2 Papp values, curated from Wang et al. N.-N. Wang et al. (2016). It adopts … view at source ↗
Figure 2
Figure 2. Overview of data distribution and structural clustering in the TDC dataset. (A) Distribution of Caco-2 values in the training and test sets. (B) PCA projection of molecular structures colored by structural clusters. (C) PCA projection showing separation between train and test sets. Principal Component Analysis (PCA) and clustering reveal that chemical structures are well spread and that the test set adequately cover… view at source ↗
Figure 3
Figure 3. Overview of data distribution and structural clustering in the OCHEM dataset. (A) Distribution of Caco-2 values in the training and test sets. (B) PCA projection of molecular structures colored by structural clusters. (C) PCA projection showing separation between train and test sets. 2.2.4. AutoML Training and Optimization To develop predictive models for Caco-2 permeability, we employed AutoGluon-Tabular (v0.7.0), … view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: MAE of models trained with the TDC dataset across different feature representations using both all and top features [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: MAE of models trained with the OCHEM dataset across different feature representations using both all and top features In summary, the TDC benchmark provides a more stable, balanced, and reproducible environment for evaluating model performance, making it well-suited fo…
Figure 6
Figure 6. Figure 6: Effect of the number of top features on model MAE across eight molecular representations on the TDC dataset. (A) Morgan fingerprints (B)Avalon fingerprints (C) ErG fingerprints (D) RDKit descriptors (E) MACCS fingerprints (F) PaDEL descriptors (G) Mordred descriptors (…
Figure 7
Figure 7. Figure 7: Effect of the number of top features on model MAE across eight molecular representations on the OCHEM dataset. (A) Morgan fingerprints (B)Avalon fingerprints (C) ErG fingerprints (D) RDKit descriptors (E) MACCS fingerprints (F) PaDEL descriptors (G) Mordred descriptors…
Figure 8
Figure 8. Figure 8: Model performance across eight molecular feature representations on the TDC dataset. (A) Morgan fingerprints (B)Avalon fingerprints (C) ErG fingerprints (D) RDKit descriptors (E) MACCS fingerprints (F) PaDEL descriptors (G) Mordred descriptors (H) CDDD embeddings [PIT…
Figure 9
Figure 9. Figure 9: Model performance across eight molecular feature representations on the OCHEM dataset. (A) Morgan fingerprints (B) Avalon fingerprints (C) ErG fingerprints (D) RDKit descriptors (E) MACCS fingerprints (F) PaDEL descriptors (G) Mordred descriptors (H) CDDD embeddings […
Figure 10
Figure 10. Figure 10: Comparative model performance using Mordred descriptors on the TDC dataset. (A) Model trained with only 2D descriptors. (B) Model trained with only 3D descriptors. (C) Model trained with both 2D and 3D descriptors using top feature selection and Bayesian optimization …
Figure 11
Figure 11. Figure 11: Comparative model performance using Mordred descriptors on the OCHEM dataset. (A) Model trained with only 2D descriptors. (B) Model trained with only 3D descriptors. (C) Model trained with both 2D and 3D descriptors using top feature selection and Bayesian optimizatio…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 21 canonical work pages

  1. [1]

    2001, Machine learning, 45, 5

    Breiman, L. 2001, Machine learning, 45, 5

    2001

  2. [2]

    2020, arXiv preprint arXiv:2003.06505

    Erickson, N., Mueller, J., Shirkov, A., et al. 2020, arXiv preprint arXiv:2003.06505

    arXiv 2020

  3. [3]

    2022, Journal of Machine Learning Research, 23, 1

    Hutter, F. 2022, Journal of Machine Learning Research, 23, 1

    work page 2022

  4. [4]

    2006, Journal of chemical information and modeling, 46, 1924

    Gedeck, P., Rohde, B., & Bartels, C. 2006, Journal of chemical information and modeling, 46, 1924

    work page 2006

  5. [5]

    2024, Information Sciences, 679, 121056

    Gui, Y., Zhan, D., & Li, T. 2024, Information Sciences, 679, 121056

    work page 2024

  6. [6]

    T., & Hu, P

    Hadipour, H., Liu, C., Davis, R., Cardona, S. T., & Hu, P. 2022, BMC bioinformatics, 23, 132

    work page 2022

  7. [7]

    2021, Journal of cheminformatics, 13, 1

    Kuwahara, H., & Gao, X. 2021, Journal of cheminformatics, 13, 1

    work page 2021

  8. [8]

    2020, in Proceedings of the AutoML Workshop at ICML, Vol

    LeDell, E., & Poirier, S. 2020, in Proceedings of the AutoML Workshop at ICML, Vol. 2020, 24

    work page 2020

Show all 23 references
  1. [9]

    M., & Lee, S.-I

    Lundberg, S. M., & Lee, S.-I. 2017, Advances in neural information processing systems, 30

    2017

  2. [10]

    2021, in 2021 12th International Conference on Information, Intelligence, Systems & Applications (IISA), IEEE, 1–8

    Malu, M., Dasarathy, G., & Spanias, A. 2021, in 2021 12th International Conference on Information, Intelligence, Systems & Applications (IISA), IEEE, 1–8

    2021

  3. [11]

    Morgan, H. L. 1965, Journal of chemical documentation, 5, 107

    1965

  4. [12]

    2018, Journal of cheminformatics, 10, 4

    Moriwaki, H., Tian, Y.-S., Kawashita, N., & Takagi, T. 2018, Journal of cheminformatics, 10, 4

    2018

  5. [13]

    S., & Moore, J

    Olson, R. S., & Moore, J. H. 2016, in Workshop on automatic machine learning, PMLR, 66–74 rdkit. 2025, rdkit, https://www.rdkit.org/

    2016

  6. [14]

    A., Baumann, K., & Zaliani, A

    Stiefl, N., Watson, I. A., Baumann, K., & Zaliani, A. 2006, Journal of chemical information and modeling, 46, 208

    2006

  7. [15]

    2011, Journal of computer-aided molecular design, 25, 533 TDC.Caco2 Wang

    Sushko, I., Novotarskyi, S., K¨ orner, R., et al. 2011, Journal of computer-aided molecular design, 25, 533 TDC.Caco2 Wang. 2025, TDC.Caco2 Wang, https: //tdcommons.ai/benchmark/admet group/01caco2/

    2011

  8. [16]

    2021, Proceedings of Machine Learning and Systems, 3, 434

    Wang, C., Wu, Q., Weimer, M., & Zhu, E. 2021, Proceedings of Machine Learning and Systems, 3, 434

    2021

  9. [17]

    2016, Journal of chemical information and modeling, 56, 763

    Wang, N.-N., Dong, J., Deng, Y.-H., et al. 2016, Journal of chemical information and modeling, 56, 763

    2016

  10. [18]

    2019, Chemical science, 10, 1692

    Winter, R., Montanari, F., No´ e, F., & Clevert, D.-A. 2019, Chemical science, 10, 1692

    2019

  11. [19]

    J., Valeri, J

    Wong, F., Zheng, E. J., Valeri, J. A., et al. 2024, Nature, 626, 177

    2024

  12. [20]

    2019, Journal of chemical information and modeling, 59, 4587

    Wu, Z., Lei, T., Shen, C., et al. 2019, Journal of chemical information and modeling, 59, 4587

    2019

  13. [21]

    2019, Journal of chemical information and modeling, 59, 3370

    Yang, K., Swanson, K., Jin, W., et al. 2019, Journal of chemical information and modeling, 59, 3370

    2019

  14. [22]

    Yap, C. W. 2011, Journal of computational chemistry, 32, 1466

    2011

  15. [23]

    1997, Pharmaceutical research, 14, 763

    Yee, S. 1997, Pharmaceutical research, 14, 763

    1997

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.