{"id":"7e8520a6-5a87-492d-a52f-f06476a8caf6","arxiv_id":"2504.13645","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"PEMMA adapts a CT-only transformer segmentation model to CT+PET with LoRA/DoRA, reaching early-fusion-level Dice while training only 0.5-8% of parameters, and extends the same adapters to prognosis.","lead":"An AI tumor segmentation model trained on CT scans can be adapted to also accept PET scans by training only a few percent of its parameters using low-rank adapters and small PET-specific modules. The paper reports accuracy close to training on both scans together, and shows the same recipe can add survival prediction with PET and electronic health records.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No backward-transfer evaluation supports the headline no-catastrophic-forgetting claim; Table 3 only shows new-domain scores after CT-only fine-tuning, so retention on MDA remains unmeasured.","rationale":"The paper's strongest contribution is narrow: a frozen CT-backbone plus PEFT modules on attention plus a PET skip path can match early fusion on HECKTOR with roughly 0.5-8% trainable parameters. The Table 3 point estimates are consistent with that claim, and the consistent DoRA-over-LoRA ordering is a useful signal. The reader's CONDITIONAL verdict is appropriate. Among the reader's three concerns, I single out the missing backward-transfer evaluation as most load-bearing, because the abstract and Section 3.2.3 promise no catastrophic forgetting when adapting with one modality, while the continual-learning protocol in Section 4.3.2 never re-tests the original MDA distribution. PET-only performance on HGJ/HMR after CT-only fine-tuning is a forward-transfer measurement on a new domain, not a retention measurement. This is not a claim that the method forgets; it is a claim that the paper provides no evidence on a central advertised advantage. A revision should add MDA re-evaluation after each continual-learning step and ideally multi-seed error bars, since the reported gaps are otherwise unquantified. The self-reported limitation in Section 6 (registered inputs required) is orthogonal to this concern. My read therefore leaves the CONDITIONAL verdict unchanged.","tokens_in":25268,"tokens_out":6042,"duration_ms":55896,"concrete_test":"Re-run the HGJ continual-learning checkpoint (PEMMA + DoRA, Swin UNETR, CT-only fine-tuning from Section 4.3.2) and evaluate it on the same MDA held-out test split used for the MDA columns of Table 3, recording tumor, lymph-node, and average Dice for CT-only, PET-only, and CP inputs. Compare PET-only average Dice against the MDA row before fine-tuning (0.68 in Table 3). If PET-only average Dice on MDA drops by more than a pre-registered margin (e.g., 0.05) or by more than the seed-to-seed spread across three runs, the no-catastrophic-forgetting claim is not supported. Repeat for HMR and for the CT+PET fine-tuning condition.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim that PEMMA 'allows updates using only one modality without causing catastrophic forgetting in the other' (Abstract; Section 3.2.3) is not actually tested. The continual-learning experiments in Section 4.3.2 fine-tune MDA-adapted models on HGJ and HMR using CT-only or CT+PET data, but every row in Table 3 after the MDA columns is evaluated on the new center's data only. Backward transfer to the MDA distribution, which is what catastrophic forgetting is about, is never measured. High PET-only Dice on HGJ or HMR after CT-only fine-tuning cannot establish retention of PET knowledge from the MDA adaptation phase, because those are different domains and the comparison lacks a pre-fine-tuning MDA baseline. Thus the no-forgetting claim is currently unsupported, not contradicted. This gap is the most load-bearing because it concerns a unique benefit of PEMMA over standard fusion, rather than a tuning detail or a secondary efficiency claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PEMMA, a parameter-efficient framework for adapting a CT-only pre-trained transformer-based segmentation model (UNETR or Swin UNETR) to also use PET images, with an extension to prognosis. The approach freezes the base model and adds PET-specific patch embeddings, a PET-specific skip connection, LoRA/DoRA adapters on attention weights, and, for Swin UNETR, an adapter module that reshapes single-modality inputs to the expected two-channel format and supports modality dropout. Experiments on the HECKTOR dataset cover multi-modal adaptation on the MDA center, continual fine-tuning on the HGJ and HMR centers (using CT-only or CT+PET data), and prognosis with the Swin UNETR encoder. The authors report that PEMMA matches or exceeds early fusion with 0.5%-8% of the trainable parameters, shows large PET-only Dice improvements after CT-only fine-tuning, and improves the concordance index as PET and EHR modalities are added.","tokens_in":25460,"tokens_out":7926,"duration_ms":69409,"significance":"If the claims are validated, PEMMA addresses a practically important deployment problem: upgrading a widely deployable CT-only model to exploit PET scans when they become available, with a parameter overhead an order of magnitude smaller than early fusion. The Swin UNETR adapter with modality dropout is a concrete architectural contribution, and the prognosis extension tests task-level transfer. The point estimates in Table 3 on the public HECKTOR data make the parameter-efficiency claim plausible, and the DoRA comparison is a useful addition. However, the paper's most distinctive claims—no catastrophic forgetting and cross-modal knowledge distillation through self-attention—are not actually measured, and the absence of uncertainty quantification weakens the quantitative conclusions. The requested experiments would make the contribution substantially stronger.","major_comments":[{"comment":"The abstract and Section 3.2.3 claim that PEMMA 'allows updates using only one modality without causing catastrophic forgetting in the other,' but the continual-learning experiments in Section 4.3.2 do not evaluate backward transfer. Every row in Table 3 after the MDA columns reports performance on the new dataset (HGJ or HMR) only; no result is reported on the MDA distribution after fine-tuning on HGJ or HMR. Catastrophic forgetting is by definition a loss of performance on previously learned data, so the current evidence supports only forward transfer to new centers, not retention of the originally adapted MDA knowledge. I request a post-fine-tuning evaluation on the MDA test set for all continual-learning variants, with comparison to the pre-fine-tuning MDA baseline, before the no-forgetting claim is made.","section":"Section 3.2.3, Section 4.3.2, Table 3"},{"comment":"The architecture sends only the N CT tokens to the decoder and relies on the assertion that 'the self-attention architecture ensures that the knowledge from the PET tokens gets distilled into the CT tokens.' This mechanism is never measured. Table 4 compares three token-selection strategies for reducing the encoder output to the decoder's expected size, but it does not isolate the contribution of the PET tokens through self-attention from the contribution of the PET-specific skip connection (z_P scaled by beta). Without ablations that remove the PET patch embedding while keeping the skip connection, and vice versa, the paper does not establish the claimed distillation mechanism, which also underlies the Swin UNETR adapter design in Section 3.2.2.","section":"Section 3.2.1, Figure 1, Table 4"},{"comment":"All results in Tables 3 and 5 are single-run point estimates with no uncertainty quantification. The abstract describes the results as 'significant' (for example, '+28% Dice score improvement'), and Section 5 ranks methods on differences as small as 0.01-0.02 average Dice (e.g., DoRA versus LoRA on Swin UNETR in the MDA stage), but no standard deviations, confidence intervals, number of seeds, or significance tests are reported. This makes it impossible to assess whether the claimed improvements over early fusion and the DoRA-versus-LoRA ranking are reliable. Please provide results across multiple seeds with variance and appropriate statistical comparisons, and remove the word 'significant' unless it is supported by a test.","section":"Tables 3 and 5, Abstract"},{"comment":"The prognosis experiments report C-index improvements of +10% (CT to CP) and +23% (CT to CPT), but these are within-method comparisons relative to the same method's CT-only result. No alternative fusion baseline (early fusion, late fusion, or full fine-tuning of the encoder) is evaluated for prognosis, so the paper does not establish that parameter-efficient adaptation is beneficial for prognosis relative to standard approaches. The absence of baseline comparisons also makes the abstract's +10% and +23% claims difficult to interpret. Additionally, the LoRA/DoRA parameter counts for the text encoder and the full EHR preprocessing pipeline are not described in enough detail to reproduce the CPT experiments.","section":"Section 4.4, Table 5"},{"comment":"The headline '+28% Dice score improvement on PET scans when trained with a single modality' is not tied to a specific baseline or table cell. The Discussion states 'minimum Average Dice scores of approximately 19% and 28% increases for two new datasets' without specifying which baseline, which inference modality, or which rows in Table 3 underlie these numbers. In addition, Section 3.1 defines three PET initialization strategies for the early fusion baseline (random, zero, cross-modal), but Section 4.3 does not state which one was used in Table 3; this choice can materially affect early fusion performance and is essential for a fair comparison with PEMMA.","section":"Abstract, Section 3.1, Section 4.3.1"}],"minor_comments":[{"comment":"There is a typo: 'segementation' should be 'segmentation'.","section":"Abstract"},{"comment":"The word 'allowimg' should be 'allowing'.","section":"Section 3.2.1"},{"comment":"The table layout is very dense and the 'Train Modalities' row is ambiguous; clearer grouping of the MDA, HGJ, and HMR blocks and of the inference-modality columns would improve readability. The caption also contains an incomplete parenthetical: 'relative toΦ orΨ)' should be 'relative to Φ or Ψ'.","section":"Table 3"},{"comment":"The modality-sampling probabilities are written as '('ct' = 0.2 , 'pet' = 0.2 , or 'ctpet' 0.6)'; the missing '=' after 'ctpet' should be fixed, and the sentence should clarify whether these probabilities apply to the multi-modal adaptation stage, the continual-learning stage, or both.","section":"Section 4.2.3"},{"comment":"The capitalization of 'DoRA' is inconsistent: the text sometimes uses 'DoRa' (e.g., 'DoRa consistently outperforms LoRA' in Section 5). Please standardize.","section":"Section 5"},{"comment":"The text encoder ClipMD is cited but not described; a sentence on its architecture, output dimension, and whether it is frozen or fine-tuned would aid reproducibility.","section":"Section 4.2.2 / Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper builds heavily on the authors' own prior PEMMA work (Saadi et al., arXiv:2404.13704) for the UNETR baseline; the authors should clearly state which Table 3 numbers are new versus replicated from that prior paper. The central no-catastrophic-forgetting claim is currently unsupported but is fixable with additional backward-transfer experiments, which is why I recommend major revision rather than rejection. If the authors cannot provide such measurements, the claim should be removed or substantially weakened in a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this one. First, the core efficiency claim is probably right: on the public HECKTOR data, PEMMA matches or beats early fusion on Swin UNETR while training only about 0.5% of the parameters, and the UNETR results replicate their own earlier PEMMA work. Second, the more interesting claim — that the method updates on one modality without catastrophic forgetting of the other — is not tested anywhere in the paper. The stress-test note is correct: the continual learning experiments in Section 4.3.2 fine-tune on HGJ/HMR and then evaluate only on HGJ/HMR. There is no backward-transfer evaluation back to the MDA distribution, so retention is unmeasured. That is a load-bearing gap because avoiding catastrophic forgetting is one of the two headline advantages.\n\nWhat is genuinely new: the Swin UNETR adapter for missing-modality handling, the DoRA comparison, and the prognosis extension with EHR text. These are architectural extensions of the earlier PEMMA framework, not a conceptual breakthrough, but they are useful and the paper says so plainly in Section 2.6. The experiments on HECKTOR are on a public dataset, and the point estimates in Table 3 support the parameter-efficiency claim. The self-citation to [58] is appropriate here because the UNETR/LoRA baseline is their own prior work.\n\nWhere it is soft, in proportion: (1) No uncertainty quantification — every score is a single run, so the gaps between DoRA and LoRA, or between PEMMA and early fusion, may not be significant. (2) The abstract's \"+28% Dice improvement on PET scans when trained with a single modality\" is not traceable to any number in Table 3. The Discussion mentions ~19% and ~28% increases for the two new datasets, but the derivation is not shown. (3) The Section 3.2.1 assertion that self-attention distills PET knowledge into CT tokens is plausible but never measured; the Table 4 ablation gives indirect support, but a direct probe of PET information retention would be stronger. (4) No code or data release is mentioned, so reproducibility rests on the description alone.\n\nWho this is for: people working on parameter-efficient fine-tuning for medical imaging, missing-modality robustness, or continual adaptation across centers. It deserves a serious referee. My recommendation: send it to peer review, but require multi-seed results, a proper backward-transfer evaluation on MDA after each continual learning step, and a clear definition of how the +28% (and +19%) numbers are computed. With those additions the paper would be a solid contribution; without them the headline claims outrun the evidence.","headline":"The efficiency story is plausible and worth referee time, but the paper's headline no-catastrophic-forgetting claim is never actually tested, and the +28% PET-only improvement is not traceable to any reported number.","tokens_in":26076,"tokens_out":1732,"would_cite":true,"duration_ms":18278,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A frozen CT-only segmentation transformer can be upgraded to use PET scans with just 8% of the trainable parameters.","keywords":["Multi-modal Adaptation","Low-rank Adaptation","Parameter-Efficiency","Cross-modal Entanglement","3D Medical Image Segmentation","Prognosis","LoRA","DoRA"],"falsifier":"Set the PET skip-connection weight $\\beta$ to zero, keep the PET tokens in the attention stream, and measure PET-only Dice on the adaptation test set; if PET Dice collapses to the CT-only baseline, then the claim that PET knowledge is distilled into CT tokens is false and the skip connection is doing the work. Conversely, remove the PET tokens from attention while keeping the skip connection to see whether the skip path alone accounts for the PET gain.","tokens_in":25007,"feed_emoji":"🩻","tokens_out":6323,"duration_ms":53959,"temperature":0.7,"pith_summary":"PEMMA claims that a segmentation transformer trained on CT images alone can be upgraded to also use PET scans by freezing the original network and training only a small set of added components: a PET patch embedding, a PET-specific skip connection, and low-rank (LoRA or DoRA) updates to the attention weights. On the HECKTOR head-and-neck dataset this upgrade matches or beats early fusion while using 0.5% to 8% of the trainable parameters, and it delivers a +28% Dice gain on PET-only inference after single-modality training. Because the PET pathway is kept separate from the CT pathway, the model can later be fine-tuned with only one modality without wiping out the other. The same recipe extends to prognosis, where adding PET improves the concordance index and adding EHR text improves it further. The paper's central wager is that transformer self-attention can transfer PET knowledge into the CT tokens the decoder actually consumes.","feed_headline":"Frozen CT model gains PET with just 8% extra parameters","feed_subtitle":"PEMMA adds low-rank adapters and a PET skip path, matching early fusion at a fraction of the cost.","key_machinery":"The mechanism that carries the argument is a parallel PET pathway grafted onto a frozen transformer. A new PET patch-embedding layer produces PET tokens, a PET-specific skip connection $\\theta^P_{SK}$ carries low-level PET features directly into the decoder with a learned weight $\\beta$, and LoRA or DoRA low-rank matrices placed in the query/value attention projections are the only updated encoder weights. For Swin UNETR, an adapter layer reshapes single-modality inputs into the expected two-channel token format so that missing-modality inference remains possible. The conceptual load-bearer is the claim, borrowed from visual prompt tuning, that PET tokens can be ignored by the decoder yet still influence it because self-attention distills their information into the CT tokens.","core_discovery":"The central claim is that the modularity of transformer encoders lets a CT-only model become a CT+PET model through a parameter-efficient adaptation that never touches the frozen base weights. PEMMA adds PET tokens via a new patch embedding, adds a direct PET skip connection to the decoder, and trains only LoRA or DoRA matrices in the attention blocks. The decoder continues to receive only CT tokens; the paper asserts that self-attention distills the PET information into those CT tokens, so the PET stream acts as visual context rather than as a second input channel. Experiments on HECKTOR with UNETR and Swin UNETR backbones report segmentation performance comparable to early fusion at a fraction of the trainable parameters, robust single-modality inference, and continual-learning updates on new centers that preserve earlier capabilities. In the prognosis extension, the same frozen-encoder recipe with LoRA/DoRA raises the time-dependent concordance index from 0.61-0.66 on CT alone to 0.67-0.68 with CT+PET and 0.75-0.80 with CT+PET+EHR.","pith_inferences":["If the distillation-through-attention premise holds, even lighter adapters—lower rank, fewer layers, or a shared PET projector—should retain most of the gain, since the PET stream does not need its own representational capacity.","The same recipe could likely adapt a pretrained model to a third imaging modality or a new segmentation target without growing a new decoder, as long as the modality can be tokenized and attend to the CT tokens.","A direct check of the distillation assumption is missing from the paper: masking PET tokens from attention in selected blocks and measuring PET-only Dice would isolate how much information travels through attention versus through the skip connection.","Because PEMMA assumes pre-registered CT/PET pairs, the next natural stress test is unregistered or misaligned data, where the PET skip connection would need an alignment module before tokenization."],"forward_implications":["A clinical deployment can start with a CT-only model and later add PET capability by training roughly 0.5-8% of the parameters, avoiding a full retrain on paired data.","The adapted model remains usable when only one modality is available at inference, which matches real clinical workflow where PET is comparatively scarce.","New imaging centers can be incorporated by updating only the low-rank adapters, and CT-only updates do not erase previously learned PET segmentation.","The same frozen-encoder recipe transfers across tasks: a segmentation encoder can be repurposed for survival prognosis by adding a small head, and adding PET and EHR modalities raises the concordance index.","DoRA consistently outperforms LoRA under PEMMA in both segmentation and prognosis, so the magnitude-direction decomposition is the recommended PEFT choice."],"supporting_citations":[{"why":"Defines LoRA, the low-rank attention update that is the core PEFT mechanism for UNETR.","marker":"[30]"},{"why":"Defines DoRA, the magnitude-direction weight decomposition that yields the stronger PEFT variant in experiments.","marker":"[48]"},{"why":"Visual prompt tuning supplies the premise that extra context tokens can be ignored by the decoder while still informing the retained tokens.","marker":"[38]"},{"why":"Provides the UNETR backbone that PEMMA adapts from CT-only to CT+PET.","marker":"[26]"},{"why":"Provides the Swin UNETR backbone and motivates the adapter layer for windowed attention.","marker":"[25]"},{"why":"The earlier PEMMA paper establishes the original UNETR version that this work extends to Swin UNETR, DoRA, and prognosis.","marker":"[58]"},{"why":"Supplies the HECKTOR multi-center CT/PET/EHR dataset on which every reported result is measured.","marker":"[52]"},{"why":"Defines the time-dependent concordance index used to evaluate prognosis.","marker":"[5]"},{"why":"Supplies the survival-analysis loss used to train the prognosis head.","marker":"[44]"},{"why":"Provides the text encoder that embeds EHR sentences for the CT+PET+EHR prognosis setting.","marker":"[21]"}],"fun_headline_variants":["CT model gains PET vision with only 8% extra parameters","Add PET to frozen CT model via low-rank adapters","Match early fusion with just 8% trainable params","PEMMA: CT-only model, PET-ready with 8% params","Frozen CT backbone, PET adaptation in 8% of weights"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's results stand on the assertion that self-attention transfers enough PET information into the CT tokens for the decoder to use; if that transfer is weak, the PET skip connection alone must carry the entire PET signal, and the paper never measures either contribution directly.","fun_headline_variants_meta":{"raw":{"variants":["CT model gains PET vision with only 8% extra parameters","Add PET to frozen CT model via low-rank adapters","Match early fusion with just 8% trainable params","PEMMA: CT-only model, PET-ready with 8% params","Frozen CT backbone, PET adaptation in 8% of weights"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000693,"raw_usage":{"total_tokens":3212,"prompt_tokens":1100,"completion_tokens":2112,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":716,"completion_tokens_details":{"reasoning_tokens":2025}},"tokens_in":716,"tokens_out":2112,"duration_ms":13376,"temperature":1.0,"reasoning_tokens":2025,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:03:47.592152+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Set the PET skip-connection weight $\\beta$ to zero, keep the PET tokens in the attention stream, and measure PET-only Dice on the adaptation test set; if PET Dice collapses to the CT-only baseline, then the claim that PET knowledge is distilled into CT tokens is false and the skip connection is doing the work. Conversely, remove the PET tokens from attention while keeping the skip connection to see whether the skip path alone accounts for the PET gain.","supporting_citations":[{"cited_title":"Visual prompt tuning, in: European conference on computer vision, Springer","cited_arxiv_id":null,"evidence_quote":"Visual prompt tuning supplies the premise that extra context tokens can be ignored by the decoder while still informing the retained tokens."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Swin UNETR backbone and motivates the adapter layer for windowed attention."},{"cited_title":"PEMMA: Parameter-Efficient Multi-Modal Adaptation for Medical Image Segmentation","cited_arxiv_id":"2404.13704","evidence_quote":"The earlier PEMMA paper establishes the original UNETR version that this work extends to Swin UNETR, DoRA, and prognosis."},{"cited_title":"grand-challenge.org/Data/","cited_arxiv_id":null,"evidence_quote":"Supplies the HECKTOR multi-center CT/PET/EHR dataset on which every reported result is measured."},{"cited_title":"A time-dependent discrimination index for survival data","cited_arxiv_id":null,"evidence_quote":"Defines the time-dependent concordance index used to evaluate prognosis."},{"cited_title":"DeepHit: A deep learning approach to survival analysis with competing risks","cited_arxiv_id":null,"evidence_quote":"Supplies the survival-analysis loss used to train the prognosis head."}],"review_version":1}