REVIEW 2 major objections 3 minor 33 references
A CNN-Transformer for Classification of Longitudinal 3D MRI Images -- A Case Study on Hepatocellular Carcinoma Prediction
T0 review · 2 major / 3 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read This paper claims that a 3D CNN-Transformer using a patient's longitudinal diffusion-weighted MRI history predicts hepatocellular carcinoma at the next screening, with an average AUROC of 0.928 and AUPRC of 0.687.
desk verdict A legitimate architectural contribution is undermined by a label leak in the positional encoding: the reported AUROC/AUPRC do not measure predictive ability, but the ideas are worth testing properly. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the HCCNet architecture itself: a 3D adaptation of ConvNeXt (a modern convolutional network) whose depthwise kernels are inflated to three dimensions and shortened, paired with a Transformer encoder with pre-layer normalization. Temporal position is encoded with fixed sinusoids of the square root of the time interval in months between each scan and an anchor date (the HCC diagnosis date for patients who develop HCC, the last registered screening otherwise), which is what lets the model handle irregular and variable-length patient histories. Before fine-tuning, the backbone is pre-trained with a teacher-student self-supervised objective in which different MRI sequences of the same patient act as natural augmentations, and the Transformer is pre-trained to tell shuffled sequences of image embeddings from intact ones. The [cls] token's final hidden state is passed through a small pooling head to produce the risk probability.
What would settle it
Recompute the same test-set evaluation using positional encodings anchored only to each patient's most recent available scan, with no reference to the future diagnosis date, and compare the AUROC and AUPRC to the reported 0.928 and 0.687; a large drop would show that the diagnosis-date anchor, not genuine temporal prediction, is doing the work.
Extended reading notes
Core claim
The paper's central claim is that temporal structure in repeated whole-volume MRIs carries usable signal for future HCC diagnosis, and that a 3D convolutional-plus-sequence model can extract it. HCCNet embeds every 3D scan with a 3D ConvNeXt backbone, orders the embeddings by time, adds sinusoidal positional encodings derived from time intervals, and lets a Transformer encoder read the resulting sequence; the [cls] token's final hidden state is classified as 'HCC at the next examination' or not. According to the paper's retrospective evaluation, DW-MRI variants achieve an average AUROC of 0.928 and AUPRC of 0.687, outperform the same architecture trained from scratch by roughly 30 percent in AUROC and up to 156 percent in AUPRC, and produce better-calibrated confidence scores even when discrimination gains are modest.
Load-bearing premise
For patients who develop HCC, the time intervals used in the positional encoding are computed backward from the HCC diagnosis date, so the model receives the very outcome it is supposed to predict during training and evaluation.
Editorial extensions
If this is right
- A working DW-MRI predictor could let surveillance programs rank cirrhosis patients by next-screening risk and tailor scan intervals instead of using one fixed schedule.
- The two-stage pre-training appears to carry most of the benefit on DW-MRI, so the recipe may transfer to other small longitudinal-imaging datasets with sparse outcome labels.
- Because the model accepts variable-length records and irregular gaps, it can serve as a template for monitoring chronic diseases beyond HCC.
- Across the four model sizes, imaging modality matters far more than parameter count, suggesting a small model is enough for deployment if the right sequence type is used.
Reading between the lines
- The paper's evaluation is retrospective: because the positional encoding uses the diagnosis date, the reported 0.928 AUROC is an upper bound on what a clinician would see at decision time, not a prospective estimate.
- An ablation that replaces the diagnosis-date anchor with last-scan date would separate 'time until diagnosis' from 'time since last surveillance', and would clarify whether the model learns true risk or proximity to diagnosis.
- Treating MRI sequences (e.g., different b-values) as natural augmentations is a cheap pre-training trick that other multi-sequence imaging studies could reuse without hand-designed color augmentations.
- If the modality-dependent pretraining gains are real, a single universal longitudinal-MRI checkpoint may be the wrong target; modality-specific pretraining is the likelier route.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes HCCNet, a hybrid 3D CNN-Transformer architecture for predicting hepatocellular carcinoma (HCC) from longitudinal MRI data. The model uses a 3D ConvNeXt backbone with DINO-style pretraining and a Transformer encoder pretrained via sequence-order prediction, and it encodes irregular screening intervals through time-based positional encodings. The authors evaluate on a retrospective cohort of cirrhosis patients, reporting average AUROC 0.928 and AUPRC 0.687 for DW-MRI, and claim that pre-training improves performance and calibration over a randomly initialized baseline.
Significance. If the reported results were valid, the work would be a useful contribution to longitudinal medical imaging: it addresses a realistic clinical task, handles irregular timing, and provides a reproducible codebase. The pre-training strategy for 3D MRI is a reasonable extension of existing self-supervised methods. However, the central evaluation is compromised by a label-leaking positional encoding, so the claimed predictive performance is not supported. The paper's contribution therefore reduces to an architectural proposal whose empirical validation remains to be demonstrated.
major comments (2)
- [Section 3.2, Eq. (4)] The positional encoding in Eq. (4) uses the date of HCC diagnosis tp_HCC to compute time intervals for patients who develop HCC, while for patients without HCC it uses the last registered MRI. Since tp_HCC is exactly the outcome the model is meant to predict, the input contains the label at training and test time. Concretely, for a positive patient the final positional-encoding element is sqrt(t_HCC - t_last) > 0, whereas for a negative patient the analogous element is 0 (the distance from the last MRI to itself). Because the positional encoding is added to the image embeddings in Eq. (6), a model can discriminate the two classes from the positional features alone, without any image content. All reported performance figures (Tables 1 and 2, Figures 2 and 3) are therefore uninformative about genuine predictive ability. The baseline comparisons in Table 2 share the same leak, so the claimed benefit of pre-training is also not established.
- [Section 3.1, Eq. (1)] The task stated in the abstract and Section 1 is to predict whether the patient will be diagnosed with HCC at the next examination. However, Eq. (1) assigns yp = 1 to any patient with tp_HCC != None, regardless of how many screenings separate the last observed MRI from the diagnosis. A positive patient whose last record is several years before diagnosis is scored as a success even though the 'next examination' after the last observation may not have occurred or may not have led to a diagnosis. The evaluation should be aligned with the stated clinical scenario, e.g., by defining positives based on the next screening result or by using time-to-event modeling.
minor comments (3)
- [Section 4.1] The sentence 'we additionally exclude all patients without a definitive diagnosis of HCC yielding a total of 243 patients of which 37 develop HCC' is ambiguous; please clarify the exclusion criterion and report the number of excluded patients.
- [Section 3.2, Eq. (4)] The definition of Δtp_i is hard to parse: the index i appears on the left, while the right-hand side mixes tp_{i+1} and tp_HCC. Please state explicitly how the vector is constructed for the last time point and how it is aligned with the sequence of image embeddings.
- [Table 1 note] The note states that AUROC ranges from 0.50 to 1; this is not a general property of AUROC and should be revised, or the observed range should be reported separately.
Circularity Check
Eq. (4) encodes the future HCC diagnosis date into the input positional encoding, so the reported AUROC/AUPRC do not measure prospective next-examination prediction.
-
self definitional
[Section 3.2, Eq. (4) (positional encoding), with labels in Eq. (1) and evaluation in Sections 4.3-4.4]
"we substitute the conventionally employed position in the sequence with the distance (i.e., the time in months) between the date of diagnosis and all previous examinations for patients with developing HCC or the patient’s last registered MRI screening and all previous examinations for patients without developing HCC respectively. ... ∆tp i = (√ti+1 − ti, if tp HCC = None, p tp HCC − ti, otherwise, (4)"
The label in Eq. (1) is defined directly from tp_HCC: yp = 1 whenever tp_HCC is not None. Equation (4) then uses that same tp_HCC as the reference point for the positional encoding of every positive patient, and Eq. (6) adds this positional encoding to the image embeddings at both training and test time. Consequently, at test time the model receives a feature that encodes how far each scan lies from the future diagnosis date, and the final positional element is sqrt(tp_HCC - t_last) > 0 for every positive patient but 0 for every negative patient. The classes can therefore be separated from the positional encoding alone, without any MRI content, so the average AUROC of 0.928 and AUPRC of 0.687 do not measure the stated task of predicting HCC at the next examination.
full rationale
The central claim of the paper, that HCCNet significantly improves predictive accuracy and reliability for personalized HCC surveillance, is invalidated by target leakage in the input representation. For patients who develop HCC, Eq. (4) computes every positional interval relative to tp_HCC, the date of diagnosis, while for patients who do not develop HCC the intervals are computed relative to the last registered MRI. Since tp_HCC is also the source of the label in Eq. (1), the model's input contains information about the very outcome it is supposed to predict, and this leak is present at test time because the test set is encoded retrospectively. This is a concrete, quotable reduction of the prediction to its own target and not merely a statistical or generalizability concern. The same leak affects the baseline comparison in Table 2, so the claimed benefit of pre-training is also not supported by the reported numbers. There is no load-bearing self-citation chain: the external methods cited (DINO, ConvNeXt, Transformer) are standard and independently established, so the circularity is concentrated in the problem definition and input encoding rather than in the citation network. The separate issue that Eq. (1) labels any patient with eventual HCC as positive rather than only patients diagnosed at the next examination reinforces the mismatch between the stated task and the evaluated task, but the positional-encoding leak alone is sufficient to invalidate the central claim. Score 8 reflects that the result is substantially forced by construction, though the model could in principle still extract some image-based signal; the reported performance figures, however, cannot be interpreted as prospective predictive accuracy.
Assumptions & free parameters
free parameters (3)
- Square-root time interval transform =
0.5 exponent
- Teacher and student temperatures in DINO pre-training =
0.04 and 0.1
- Fine-tuning batch size =
chosen from {16, 32, 64} by 5-fold CV
assumptions (3)
- ad hoc to paper The HCC diagnosis date tp_HCC is available when computing positional encodings for positive patients during model evaluation.
- domain assumption The single-center retrospective cohort yields performance estimates that transfer to other patient populations.
- domain assumption MRI scans are treated as i.i.d. during CNN pre-training.
Cite this review
Pith. "Pith review of A CNN-Transformer for Classification of Longitudinal 3D MRI Images -- A Case Study on Hepatocellular Carcinoma Prediction." pith.science (2026). https://pith.science/paper/CMLJK2JT
@misc{pith2026250110733,
author = {Pith},
title = {Pith review of: A CNN-Transformer for Classification of Longitudinal 3D MRI Images -- A Case Study on Hepatocellular Carcinoma Prediction},
year = {2026},
howpublished = {\url{https://pith.science/paper/CMLJK2JT}},
note = {Machine review of arXiv:2501.10733}
}
read the original abstract
Longitudinal MRI analysis is crucial for predicting disease outcomes, particularly in chronic conditions like hepatocellular carcinoma (HCC), where early detection can significantly influence treatment strategies and patient prognosis. Yet, due to challenges like limited data availability, subtle parenchymal changes, and the irregular timing of medical screenings, current approaches have so far focused on cross-sectional imaging data. To address this, we propose HCCNet, a novel model architecture that integrates a 3D adaptation of the ConvNeXt CNN architecture with a Transformer encoder, capturing both the intricate spatial features of 3D MRIs and the complex temporal dependencies across different time points. HCCNet utilizes a two-stage pre-training process tailored for longitudinal MRI data. The CNN backbone is pre-trained using a self-supervised learning framework adapted for 3D MRIs, while the Transformer encoder is pre-trained with a sequence-order-prediction task to enhance its understanding of disease progression over time. We demonstrate the effectiveness of HCCNet by applying it to a cohort of liver cirrhosis patients undergoing regular MRI screenings for HCC surveillance. Our results show that HCCNet significantly improves predictive accuracy and reliability over baseline models, providing a robust tool for personalized HCC surveillance. The methodological approach presented in this paper is versatile and can be adapted to various longitudinal MRI screening applications. Its ability to handle varying patient record lengths and irregular screening intervals establishes it as an invaluable framework for monitoring chronic diseases, where timely and accurate disease prognosis is critical for effective treatment planning.
Figures
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in ":" * " " * FUNCTION f...
-
[2]
author Caron, M. , author Touvron, H. , author Misra, I. , author Jégou, H. , author Mairal, J. , author Bojanowski, P. , author Joulin, A. , year 2021 . title Emerging Properties in Self - Supervised Vision Transformers . http://arxiv.org/abs/2104.14294. note arXiv:2104.14294 [cs]
arXiv 2021
-
[3]
author Chen, T. , author Kornblith, S. , author Norouzi, M. , author Hinton, G. , year 2020 . title A Simple Framework for Contrastive Learning of Visual Representations . http://arxiv.org/abs/2002.05709. note arXiv:2002.05709 [cs, stat]
arXiv 2020
-
[4]
author Dadsetan, S. , author Arefan, D. , author Berg, W.A. , author Zuley, M.L. , author Sumkin, J.H. , author Wu, S. , year 2022 . title Deep learning of longitudinal mammogram examinations for breast cancer risk prediction . journal Pattern Recognition volume 132 , pages 108919 . https://www.sciencedirect.com/science/article/pii/S0031320322004009, :10....
-
[5]
author Dammu, H. , author Ren, T. , author Duong, T.Q. , year 2023 . title Deep learning prediction of pathological complete response, residual cancer burden, and progression-free survival in breast cancer patients . journal PLOS ONE volume 18 , pages e0280148 . https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0280148, :10.1371/journal.po...
-
[6]
author El-Nouby, A. , author Izacard, G. , author Touvron, H. , author Laptev, I. , author Jegou, H. , author Grave, E. , year 2021 . title Are Large -scale Datasets Necessary for Self - Supervised Pre -training? http://arxiv.org/abs/2112.10740, :10.48550/arXiv.2112.10740. note arXiv:2112.10740 [cs]
-
[7]
author Gao, Y. , author Ghodrati, V. , author Kalbasi, A. , author Fu, J. , author Ruan, D. , author Cao, M. , author Wang, C. , author Eilber, F.C. , author Bernthal, N. , author Bukata, S. , author Dry, S.M. , author Nelson, S.D. , author Kamrava, M. , author Lewis, J. , author Low, D.A. , author Steinberg, M. , author Hu, P. , author Yang, Y. , year 20...
work page 2021
-
[8]
author Gong, W. , author Beckmann, C.F. , author Smith, S.M. , year 2024 . title Individualised prediction of longitudinal change in multimodal brain imaging . journal Imaging Neuroscience volume 2 , pages 1--19
work page 2024
Show all 33 references
-
[9]
, author Zohren, S
author Granziol, D. , author Zohren, S. , author Roberts, S. , . title Learning Rates as a Function of Batch Size : A Random Matrix Theory Approach to Neural Network Training
-
[10]
, author Strub, F
author Grill, J.B. , author Strub, F. , author Altché, F. , author Tallec, C. , author Richemond, P.H. , author Buchatskaya, E. , author Doersch, C. , author Pires, B.A. , author Guo, Z.D. , author Azar, M.G. , author Piot, B. , author Kavukcuoglu, K. , author Munos, R. , auth...
2020 arXiv
-
[11]
, author Pareek, A
author Huang, S.C. , author Pareek, A. , author Jensen, M. , author Lungren, M.P. , author Yeung, S. , author Chaudhari, A.S. , year 2023 . title Self-supervised learning for medical image classification: a systematic review and implementation guidelines . journal npj Digital ...
2023 doi
-
[12]
, author Kadir, T
author Jamaludin, A. , author Kadir, T. , author Zisserman, A. , year 2017 . title Self-supervised Learning for Spinal MRIs , in: editor Cardoso, M.J. , editor Arbel, T. , editor Carneiro, G. , editor Syeda-Mahmood, T. , editor Tavares, J.M.R. , editor Moradi, M. , editor Brad...
2017
-
[13]
, author Yu, H
author Jin, C. , author Yu, H. , author Ke, J. , author Ding, P. , author Yi, Y. , author Jiang, X. , author Duan, X. , author Tang, J. , author Chang, D.T. , author Wu, X. , author Gao, F. , author Li, R. , year 2021 a. title Predicting treatment response from longitudinal im...
2021 doi
-
[14]
, author Yu, H
author Jin, C. , author Yu, H. , author Ke, J. , author Ding, P. , author Yi, Y. , author Jiang, X. , author Duan, X. , author Tang, J. , author Chang, D.T. , author Wu, X. , et al., year 2021 b. title Predicting treatment response from longitudinal images using multi-task dee...
2021
-
[15]
, author Marttinen, P
author Kumar, Y. , author Marttinen, P. , year 2024 . title Improving medical multi-modal contrastive learning with expert annotations . journal arXiv preprint arXiv:2403.10153
2024 arXiv
-
[16]
, author Chen, M
author Lan, Z. , author Chen, M. , author Goodman, S. , author Gimpel, K. , author Sharma, P. , author Soricut, R. , year 2020 . title ALBERT : A Lite BERT for Self -supervised Learning of Language Representations . http://arxiv.org/abs/1909.11942. note arXiv:1909.11942 [cs]
2020 arXiv
-
[17]
, author Hu, Y.c
author Lee, D. , author Hu, Y.c. , author Kuo, L. , author Alam, S. , author Yorke, E. , author Li, A. , author Rimner, A. , author Zhang, P. , year 2022 . title Deep learning driven predictive treatment planning for adaptive radiotherapy of lung cancer . journal Radiotherapy ...
2022 doi
-
[18]
, author Chen, S
author Li, X. , author Chen, S. , author Yang, J. , year 2020 . title Understanding the disharmony between weight normalization family and weight decay , in: booktitle Proceedings of the AAAI Conference on Artificial Intelligence , pp. pages 4715--4722 . https://ojs.aaai.org/i...
2020
-
[19]
, author Mao, H
author Liu, Z. , author Mao, H. , author Wu, C.Y. , author Feichtenhofer, C. , author Darrell, T. , author Xie, S. , year 2022 . title A ConvNet for the 2020s . http://arxiv.org/abs/2201.03545. note arXiv:2201.03545 [cs]
2022 arXiv
- [20]
-
[21]
, author Dercle, L
author Lu, L. , author Dercle, L. , author Zhao, B. , author Schwartz, L.H. , year 2021 . title Deep learning for the prediction of early on-treatment response in metastatic colorectal cancer from serial medical imaging . journal Nature Communications volume 12 , pages 6654 . ...
2021 doi
-
[22]
, author Mehta, A.S
author Parikh, N.D. , author Mehta, A.S. , author Singal, A.G. , author Block, T. , author Marrero, J.A. , author Lok, A.S. , year 2020 . title Biomarkers for the early detection of hepatocellular carcinoma . journal Cancer Epidemiology, Biomarkers & Prevention volume 29 , pag...
2020
-
[23]
, author Wang, J
author Ren, H. , author Wang, J. , author Zhao, W.X. , author Wu, N. , year 2021 . title RAPT : Pre -training of Time - Aware Transformer for Learning Robust Healthcare Representation , in: booktitle Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data M...
2021
-
[24]
, author Schmidt-Erfurth, U
author Rivail, A. , author Schmidt-Erfurth, U. , author Vogl, W.D. , author Waldstein, S.M. , author Riedl, S. , author Grechenig, C. , author Wu, Z. , author Bogunovic, H. , year 2019 . title Modeling Disease Progression in Retinal OCTs with Longitudinal Self -supervised Lear...
2019 doi
-
[25]
, author Yang, D
author Tang, Y. , author Yang, D. , author Li, W. , author Roth, H. , author Landman, B. , author Xu, D. , author Nath, V. , author Hatamizadeh, A. , year 2021 . title Self-supervised pre-training of swin transformers for 3d medical image analysis . journal arXiv preprint arXi...
2021 arXiv
-
[26]
, author Shazeer, N
author Vaswani, A. , author Shazeer, N. , author Parmar, N. , author Uszkoreit, J. , author Jones, L. , author Gomez, A.N. , author Kaiser, L. , author Polosukhin, I. , year 2023 . title Attention Is All You Need . http://arxiv.org/abs/1706.03762. note arXiv:1706.03762 [cs]
2023 arXiv
-
[27]
, author Alam, S.R
author Wang, C. , author Alam, S.R. , author Zhang, S. , author Hu, Y.C. , author Nadeem, S. , author Tyagi, N. , author Rimner, A. , author Lu, W. , author Thor, M. , author Zhang, P. , year 2020 . title Predicting spatial esophageal changes in a multimodal longitudinal imagi...
2020 doi
-
[28]
, author Rimner, A
author Wang, C. , author Rimner, A. , author Hu, Y.C. , author Tyagi, N. , author Jiang, J. , author Yorke, E. , author Riyahi, S. , author Mageras, G. , author Deasy, J.O. , author Zhang, P. , year 2019 . title Toward predicting the evolution of lung tumors during radiotherap...
2019 doi
-
[29]
, author Yang, Y
author Xiong, R. , author Yang, Y. , author He, D. , author Zheng, K. , author Zheng, S. , author Xing, C. , author Zhang, H. , author Lan, Y. , author Wang, L. , author Liu, T.Y. , year 2020 . title On Layer Normalization in the Transformer Architecture . http://arxiv.org/abs...
2020 arXiv
-
[30]
, author Hosny, A
author Xu, Y. , author Hosny, A. , author Zeleznik, R. , author Parmar, C. , author Coroller, T. , author Franco, I. , author Mak, R.H. , author Aerts, H.J. , year 2019 . title Deep Learning Predicts Lung Cancer Treatment Response from Serial Medical Imaging . journal Clinical...
2019 doi
-
[31]
, author Wu, Y
author Zeng, D. , author Wu, Y. , author Hu, X. , author Xu, X. , author Yuan, H. , author Huang, M. , author Zhuang, J. , author Hu, J. , author Shi, Y. , year 2021 . title Positional contrastive learning for volumetric medical image segmentation . journal arXiv preprint arXi...
2021 arXiv
-
[32]
, author Li, Z
author Zhang, S. , author Li, Z. , author Zhou, H.Y. , author Ma, J. , author Yu, Y. , year 2022 a. title Advancing 3D Medical Image Analysis with Variable Dimension Transform based Supervised 3D Pre -training . http://arxiv.org/abs/2201.01426. note arXiv:2201.01426 [cs, eess]
2022 arXiv
-
[33]
, author Hu, C
author Zhang, Y. , author Hu, C. , author Zhong, L. , author Song, Y. , author Sun, J. , author Li, M. , author Dai, L. , author Zhou, Y. , author Yang, W. , year 2022 b. title Spatiotemporal Attention for Early Prediction of Hepatocellular Carcinoma Based on Longitudinal Ultr...
2022
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.