{"id":"185c7fa1-aa74-4756-933a-dbd715436c1e","arxiv_id":"2509.06396","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Brain metastasis trajectories cluster into five patterns, and one-year complete response can be predicted with 0.90 AUC once the 60-day follow-up scan is included.","lead":"An automated pipeline assembled 896 brain metastases in 177 patients from routine MRI and predicted one-year response to stereotactic radiosurgery with up to 0.90 AUC once the first follow-up scan was included. If the evaluation is unbiased, this would support early, automatic response assessment in clinical practice.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-validation lacks patient-level grouping; lesion-level splitting likely inflates the 0.90 AUC.","rationale":"The reader identified several potential weaknesses. The most load-bearing for the central claim is the missing patient-level cross-validation, because it directly undermines the unbiasedness of the 0.90 AUC. The segmentation concern (16% partial failures) is real but secondary: even if segmentation were perfect, leakage would still inflate performance. The paper itself provides code and data, so the test is feasible. The reader's verdict of CONDITIONAL is appropriate; the condition should explicitly require patient-level evaluation. My read therefore does not change the reader's verdict.","tokens_in":8868,"tokens_out":3737,"duration_ms":38653,"concrete_test":"Using the released code and the same features, rerun the CR vs non-CR classification for the t0:t1 → t6 configuration with patient-level grouped cross-validation (e.g., GroupKFold or repeated group shuffle-split on patient ID). Report pooled AUC with 95% CI. Also rerun the t0-only model the same way. If the grouped t0:t1 AUC drops more than ~0.05 relative to Table 2 or its CI overlaps the grouped t0-only AUC, the headline result is not attributable to genuine early predictive signal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that t0 and t1 MRI predict one-year response with AUC 0.90—depends on an unbiased evaluation. The paper specifies a 5-fold cross-validation (Section 2.3) but does not state that folds were split by patient. With 896 lesions in 177 patients, random lesion-level folds will put lesions from the same patient in both training and test sets. The feature set includes patient-level variables (primary tumor site, histology, total lesion burden) and lesions from one patient share systemic treatment and imaging conditions. The model can therefore exploit patient identity to boost test AUC. This is a form of leakage that directly affects the reported number, and it is not addressed in the text. The paper also notes that most one-year CR lesions are already CR at t1 (Section 3.3); if the t1 volume is zero, the model trivially predicts CR. That persistence effect is acknowledged, but the leakage is not. Thus the strongest claim is unsupported unless patient-level cross-validation yields similar results.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an automated pipeline for curating longitudinal MRI/CT/RTStruct data of brain metastases treated with stereotactic radiosurgery, yielding 896 lesions in 177 patients with follow-up to one year. The authors use Gaussian-mixture clustering of normalized volume trajectories to identify five response patterns, and then predict lesion-level one-year response (CR vs. non-CR and responding vs. non-responding) using gradient boosting (LGBM) and graph attention networks on time-point graphs. The headline result is an AUC of 0.90 (CI95%=0.88–0.92) for CR vs. non-CR using only t0 and t1 features with LGBM, with GML reaching 0.88. The paper emphasizes the scalability of the curation pipeline and the potential for early-response decision support.","tokens_in":9072,"tokens_out":2538,"duration_ms":31079,"significance":"If the evaluation is unbiased, this is a useful contribution: it demonstrates that a fully automated longitudinal curation pipeline can assemble a large real-world lesion-level dataset, and that early follow-up imaging carries substantial predictive signal for one-year response. The trajectory clustering is clinically interpretable, and the code is made available, which supports reproducibility. The reported AUCs are high but rest on a small number of methodological choices; the central claim therefore hinges on whether the evaluation protocol excludes leakage and whether the automated volumetry is reliable enough to support both the labels and the features.","major_comments":[{"comment":"The 5-fold cross-validation is described without stating that folds were split by patient. With 896 lesions from only 177 patients, random lesion-level splits will place lesions from the same patient in both training and test folds. Since the feature set includes patient-level variables (primary tumor site, histology, total lesion burden, lesion count) and lesions from one patient share systemic treatment and imaging conditions, the model can exploit patient identity or patient-level confounding to inflate the test AUC. The manuscript reports pooled test predictions across folds without any patient-level grouping. This directly affects the headline 0.90 AUC. Please re-run the evaluation with patient-stratified folds and report the patient-level cross-validated AUC; alternatively, justify why lesion-level independence is acceptable here and quantify the amount of same-patient leakage.","section":"Section 2.3 (Cross-validation)"},{"comment":"The CR vs. non-CR prediction at t0:t1 likely benefits from the trivial persistence of complete response. The authors note in Section 3.2 and Figure 2 that more than 50% of lesions achieving one-year CR are already in CR at the first follow-up. Since the t1 volume (specifically zero volume) is a feature and the one-year label is derived from the same volumetric resegmentation pipeline, a lesion already at CR at t1 is almost deterministically labeled CR at t6. The paper acknowledges this pattern only as a possible contributor to separability, but does not quantify it. To support the early-prediction claim, report AUC on the subset of lesions not already in CR at t1, or at least report the fraction of test predictions that are explained by this persistence effect. Without this, the 0.90 AUC cannot be interpreted as novel predictive information beyond what is already visible at the first fol","section":"Section 3.3 / Table 2"},{"comment":"The quality control subsection reports that roughly 16% of lesion time series had partial segmentation or other failures on a random subset of 66 series. The response labels and the volume/radiomic features are both derived from these resegmentations. If segmentation errors are systematic (e.g., more frequent for small lesions or for lesions near the skull), they can create a shared error component in labels and features that inflates the apparent predictive performance. The paper also applies a 'semi-automatic sample rejection based on suspicious trajectory detection and marginal manual observation' without reporting how many lesions were excluded or on what exact criteria. Please provide a sensitivity analysis excluding QC-flagged time points or lesions, and report the rejection rate and its possible effect on the reported AUC.","section":"Section 3.1 / Section 2.3"},{"comment":"The permutation test is used only to compare models, and the reported confidence intervals are obtained by bootstrapping pooled fold predictions. This procedure treats predictions as independent, which is not valid when lesions are nested in patients and when folds may contain the same patient. The significance of the improvement from adding t1 is also tested with multiple comparisons across time-point configurations, but no correction is applied. At minimum, the statistical analysis should account for patient-level clustering and should state whether any multiple-comparison correction was used.","section":"Section 2.3 / Section 3.3 (statistical inference)"}],"minor_comments":[{"comment":"Duplicate phrase 'Graph Machine Learning Graph Machine Learning (GML)' in the abstract. Also 'machine learning Machine Learning (ML)' in Section 1. Please proofread the notation for expanded acronyms.","section":"Abstract and Section 1"},{"comment":"The choice of target cluster count is fixed to 5, but no sensitivity analysis is provided. Since the clustering is exploratory, a small paragraph reporting stability across k values would strengthen the interpretation of the five trajectories.","section":"Section 2.2"},{"comment":"The definition of 'responding' vs. 'non-responding' groups is clinically meaningful, but the class imbalance and the exact RANO-BM thresholds are stated only in Section 1. It would help to restate the volumetric criteria near the prediction task definition, since the thresholds drive both the labels and the feature engineering.","section":"Section 2.3"},{"comment":"The sentence 'For many patients (76.5%), RTStruct files were stored on separate databases that cannot be queried' is important for the generalizability of the curation pipeline, but it is not clear whether this refers to all 1135 initial patients or to the 177-patient cohort. Please clarify.","section":"Section 3.1"},{"comment":"The clusters are labeled only by number in the figure and text. Naming them descriptively (e.g., 'fast responders', 'pseudoprogression-like', 'moderate responders', 'rapid progressors', 'late progressors') would improve readability, especially for clinical readers.","section":"Section 3.2 / Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central claim is plausible and the dataset is valuable, but the evaluation currently lacks a patient-level split, which is a standard and necessary safeguard for lesion-level studies. The persistence of CR at t1 is acknowledged but not quantified, and the 16% segmentation failure rate warrants a robustness check. I do not see this as a rejection: the issues are addressable by re-running the experiments with patient-stratified CV and reporting subgroup analyses. The paper would benefit from clearer reporting of the exclusion criteria and the statistical clustering of predictions. I also note that the scope (an application paper in a vision/imaging venue) is appropriate, and the code release is a strength."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The dataset and curation pipeline are the real contribution here. 896 lesions in 177 patients with longitudinal follow-up, automatic segmentation and tracking, plus the five-trajectory clustering—that is genuinely useful and well executed. The t0-only prediction at 0.77 AUC is an honest baseline, and the code is public, which helps a lot.\n\nThat said, the headline 0.90 AUC for t0:t1 is likely inflated. The 5-fold CV is described without any patient-level grouping. With 177 patients and 896 lesions, random lesion-level splits will put lesions from the same patient in both training and test folds. The features include patient-level clinical variables (primary tumor site, histology, total lesion burden), and lesions from one patient share systemic treatment and imaging conditions. So the model can exploit patient identity to boost test AUC. That is leakage, and it directly affects the reported number.\n\nThe cohort filtering is another soft spot. The inclusion criteria mention 'semi-automatic sample rejection based on suspicious trajectory detection (time point swings into CR) and marginal manual observation.' That removes exactly the hard-to-predict cases, so the test set is biased toward clean trajectories.\n\nThe paper itself acknowledges in Section 3.3 that most one-year CR lesions are already CR at t1. That means a large fraction of the CR predictions are almost deterministic once t1 shows zero volume. The authors were honest about this, and it is a real phenomenon, but combined with the CV issue it makes the 0.90 AUC more of an upper bound than a realistic estimate of early prediction performance.\n\nI want to be fair: the paper also reports negative results (GML not outperforming LGBM, permutation tests not always significant), and the t0-only AUC of 0.77 is a genuine finding. The trajectory analysis is descriptive but novel for this scale. So the work has value beyond the predictive claim.\n\nWho is this for? Anyone working on longitudinal brain metastases imaging, automated curation, or response prediction. The dataset and pipeline are worth citing. But the central predictive claim needs a re-analysis with patient-level CV and performance on the unfiltered cohort. If those numbers hold, the paper becomes much stronger; if they drop, the data side still stands.\n\nI would send this to peer review—it deserves a careful referee—but the referee should ask for that re-analysis before accepting the headline result.","headline":"Useful longitudinal BM dataset and pipeline, but the 0.90 AUC overstates early prediction because the CV split is not patient-level and CR persistence dominates.","tokens_in":9579,"tokens_out":1758,"would_cite":true,"duration_ms":20414,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that routine longitudinal MRI, processed by a fully automated pipeline, can accurately assess and predict one-year response of brain metastases to stereotactic radiosurgery.","keywords":["longitudinal MRI","brain metastases","stereotactic radiosurgery","response prediction","growth trajectories","graph machine learning","radiomics","UNet resegmentation"],"falsifier":"Recompute one-year response labels on a random subset of lesions from manual expert segmentations of the same MRIs, then retrain and evaluate the same features and models. If AUC for CR vs non-CR drops substantially below 0.90, the automated-label noise is the source of the performance. An external multi-centre cohort would also settle generalizability.","tokens_in":8764,"feed_emoji":"🧠","tokens_out":4804,"duration_ms":52260,"temperature":0.7,"pith_summary":"The paper claims that a fully automated imaging pipeline can turn routine follow-up scans of brain metastases into reliable lesion-level response labels, and that one-year response can be predicted as early as the first follow-up scan. On a curated cohort of 896 lesions in 177 patients, gradient boosting on pre-treatment and 60-day post-treatment MRI reaches 0.90 AUC for complete response versus not, and 0.82 AUC for the broader responding/non-responding split; a graph model reaches 0.88 AUC with one flexible model across input time-point configurations. The same data-driven analysis finds five characteristic volume trajectories, including an early-shrinking group, a pseudoprogression-like group, moderate responders, and progressive groups. If the evaluation is unbiased, these results imply that routine clinical images alone can support early, personalized treatment decisions without manual annotation.","feed_headline":"A 60-day scan predicts brain metastasis response at 0.90 AUC","feed_subtitle":"Automated pipeline on 896 lesions shows the first follow-up MRI plus pre-treatment imaging forecasts one-year outcome.","key_machinery":"The load-bearing mechanism is the lesion-level longitudinal volume trajectory: pre-treatment delineations from radiosurgery planning are propagated through registered follow-up MRIs by a UNet-based resegmentation model, individual lesions are matched by overlap and centroid distance, and volumes are resampled to seven uniform 60-day intervals. Relative volume trajectories are clustered with a Gaussian-mixture-style model into five patterns, and per-time-point radiomic and volume features feed either gradient-boosted trees or a graph attention network whose nodes are time points. The graph formulation lets one model take any subset of follow-up time points, while the classical model needs a s","core_discovery":"The central claim is that the one-year response of a brain metastasis to stereotactic radiosurgery is largely legible in the first post-treatment MRI, and that a data-curation pipeline can extract this signal at scale. Lesions are followed across uniformly resampled 60-day time points; clustering identifies five dominant growth trajectories. For CR vs non-CR, classification reaches 0.90 AUC (CI 0.88–0.92) using only t0 and t1 features with gradient boosting; a graph attention network reaches 0.88 AUC (CI 0.86–0.90) and handles variable input configurations in one model. Performance rises with each additional time point but the largest jump comes from adding t1.","pith_inferences":["If the 0.90 AUC holds under external validation, response prediction could be recast as a screening tool to select lesions for early salvage, rather than only retrospective assessment.","The pseudoprogression-like cluster suggests a trajectory-based classifier could eventually separate radiation necrosis from true progression using only volumes, a clinically urgent distinction the paper does not directly test.","The reported 16% partial segmentation failure rate implies that a human-in-the-loop correction step or uncertainty filter may be needed before clinical deployment; the reported AUC could be optimistic if label noise from automated volumes is correlated with features.","Because most one-year CR lesions are already CR at t1, the high AUC may partly reflect an early binary state rather than true predictive signal; testing on lesions not already CR at t1 would isolate genuine prediction."],"forward_implications":["One-year complete response can be predicted with high AUC at 60 days post-treatment from routine MRI, so treatment decisions could be adapted a year earlier.","Five data-driven trajectory clusters give clinicians a compact vocabulary for interpreting lesion growth after radiosurgery.","Automated curation makes large longitudinal cohorts feasible without manual annotation, enabling larger studies of growth patterns and biomarkers.","Including more follow-up time points improves prediction monotonically, with the largest gain from the first follow-up scan.","The graph model trades a small amount of accuracy for deployment flexibility, using a single model across different available time-point configurations."],"supporting_citations":[{"why":"Supplies the UNet-based resegmentation model that propagates t0 delineations to all follow-ups; its reported Dice of 0.78 is the segmentation quality foundation.","marker":"[2]"},{"why":"Provides the volumetric adaptation of response categories used to label each lesion; the paper states it adapts these criteria.","marker":"[3]"},{"why":"Supplies the affine registration implementation used to align CT/ROI to MRI and all images to the treatment scan.","marker":"[4]"},{"why":"Prior work showing that incorporating follow-up MRI improves predictive performance, the comparison point for using t1.","marker":"[6]"},{"why":"Supplies the radiomics feature extraction used to compute the 107 per-lesion features.","marker":"[9]"},{"why":"Defines the RANO-BM response criteria that structure the complete/partial/stable/progressive categories.","marker":"[13]"},{"why":"Benchmark showing the difficulty of brain metastasis segmentation, motivating the resegmentation-based approach.","marker":"[14]"},{"why":"Provides the pseudo-likelihood Gaussian mixture implementation used for trajectory clustering.","marker":"[15]"},{"why":"Defines the graph attention network encoder used for the time-point graph models.","marker":"[20]"}],"fun_headline_variants":["First MRI forecast at 0.90 AUC","One scan predicts one-year metastasis response","AI predicts SRS outcome from early imaging","Clustering finds 5 brain metastasis trajectories","0.90 AUC from pre-treatment and first follow-up"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The automated volume measurements, lesion matching, and feature extraction are accurate enough that the response labels and the predictors are not corrupted by shared segmentation or registration errors; the paper's own quality check reports about 16% partial segmentation failures and 1% misregistrations.","fun_headline_variants_meta":{"raw":{"variants":["First MRI forecast at 0.90 AUC","One scan predicts one-year metastasis response","AI predicts SRS outcome from early imaging","Clustering finds 5 brain metastasis trajectories","0.90 AUC from pre-treatment and first follow-up"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000154,"raw_usage":{"total_tokens":1109,"prompt_tokens":865,"completion_tokens":244,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":175}},"tokens_in":609,"tokens_out":244,"duration_ms":3320,"temperature":1.0,"reasoning_tokens":175,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T23:38:13.755705+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute one-year response labels on a random subset of lesions from manual expert segmentations of the same MRIs, then retrain and evaluate the same features and models. If AUC for CR vs non-CR drops substantially below 0.90, the automated-label noise is the source of the performance. An external multi-centre cohort would also settle generalizability.","supporting_citations":[{"cited_title":"Scientific reports14(1), 31603 (2024)","cited_arxiv_id":null,"evidence_quote":"Supplies the UNet-based resegmentation model that propagates t0 delineations to all follow-ups; its reported Dice of 0.78 is the segmentation quality foundation."},{"cited_title":"Neuro- Oncology Advances7(1), vdae216 (01 2025)","cited_arxiv_id":null,"evidence_quote":"Provides the volumetric adaptation of response categories used to label each lesion; the paper states it adapts these criteria."},{"cited_title":"Insight j2(365), 1–35 (2009)","cited_arxiv_id":null,"evidence_quote":"Supplies the affine registration implementation used to align CT/ROI to MRI and all images to the treatment scan."},{"cited_title":"Scientific Reports14(1), 11085 (May 2024)","cited_arxiv_id":null,"evidence_quote":"Prior work showing that incorporating follow-up MRI improves predictive performance, the comparison point for using t1."},{"cited_title":"The Lancet Oncology16(6), e270–e278 (2015)","cited_arxiv_id":null,"evidence_quote":"Defines the RANO-BM response criteria that structure the complete/partial/stable/progressive categories."},{"cited_title":"The Brain Tumor Segmentation (BraTS-METS) Challenge 2023: Brain Metastasis Segmentation on Pre-treatment MRI","cited_arxiv_id":"2306.00838","evidence_quote":"Benchmark showing the difficulty of brain metastasis segmentation, motivating the resegmentation-based approach."},{"cited_title":"StepMix: A Python Package for Pseudo-Likelihood Estimation of Generalized Mixture Models with External Variables","cited_arxiv_id":"2304.03853","evidence_quote":"Provides the pseudo-likelihood Gaussian mixture implementation used for trajectory clustering."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the graph attention network encoder used for the time-point graph models."}],"review_version":1}