{"id":"1a9880b8-39db-4605-9ca4-2606a6cf2862","arxiv_id":"2608.08713","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A controlled study across four vision encoders, five compressors, and five language models shows ROI cropping is the most consistent improvement for CT report generation, reaching new SOTA clinical F1 of 49.5.","lead":"To make radiology report generation from 3D CT scans cheaper, the authors compare how to crop, resize, and compress the image features before feeding them to a language model. They find that cropping to the relevant organs is the most reliable win, and that higher resolution only helps when paired with certain compressors.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ROI-cropping benefit is confounded with label-set alignment: both datasets crop to the exact anatomy the evaluation metric scores, so the headline 'generally applicable' recommendation is not yet supported.","rationale":"The reader's weakest assumption and my own analysis converge on the same load-bearing concern: the ROI in each dataset is precisely aligned with the label set used by the clinical metric, so the observed ROI benefit may not generalize to open-set or full-body report generation. This is not an internal inconsistency: the 19-of-20 result is credible as an empirical finding on these benchmarks. Rather, it is a threat to the paper's broader design recommendation, which the reader's strongest claim extends to 'generally applicable intervention.' The paper's own Section 5 explicitly flags that the lung-crop removes anatomy described in reference reports, and my proposed test would determine whether that removed anatomy is clinically consequential in the evaluated outputs. I considered other concerns, including the fact that hyperparameters were tuned only on the ROI regime (Section 3.4) and that final SOTA numbers lack error bars, but these affect the magnitude and precision of the reported gains rather than the fundamental generalizability of the recommendation. The hyperparameter confound is further mitigated by the consistency of the ROI benefit across two datasets and five projectors, and it is acknowledged in Section 5 as a limitation of lower-bound comparisons. Because the internal claim is defensible and the external-validity concern is already reflected in the CONDITIONAL verdict, I do not recommend changing the verdict; the condition should be that the authors address the label-alignment confound with the proposed out-of-ROI analysis or an equivalent test.","tokens_in":18709,"tokens_out":9904,"duration_ms":103931,"concrete_test":"On the CT-RATE test set, extract from the reference reports a set of out-of-ROI findings (e.g., hepatic, renal, osseous, or nodal abnormalities not among the 18 RadBERT classes), then compare the recall of these findings in reports generated under Whole vs ROI cropping at the same token budget and with the same projectors and LLMs. If ROI cropping substantially reduces recall of out-of-ROI findings while increasing the 18-class macro F1, the headline ROI benefit is largely an artifact of discarding unscored anatomy. A complementary check is to run the Whole-vs-ROI comparison on a dataset whose reference labels include findings outside the chosen ROI (for instance, a full-body or multi-region CT benchmark) and test whether the ROI advantage persists.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central recommendation—ROI cropping is the most consistent intervention at matched token budgets—is supported empirically in 19 of 20 cells, but both supporting datasets use a crop that coincides with the abnormality classes used for evaluation. On CT-RATE (Section 3.1), the ROI is the lung-lobe bounding box, and the RadBERT-based metric scores exactly 18 pulmonary and cardiovascular classes (Section 3.2.1); on Merlin, the foreground ROI covers the abdominal organs corresponding to the 30 scored classes. The paper itself notes in Section 5 that a lung-crop 'removes anatomy the reference reports describe, which can encourage statements about structures the input no longer contains.' This is a label-alignment confound: the F1 gain may partly reflect the model being prevented from generating or attending to extra-ROI findings that the metric neither rewards nor penalizes, rather than a general improvement in report quality. Internal consistency of the 19-of-20 result is not at issue; the threat is to external validity of the design recommendation for full-body or open-set report generation, where findings outside the crop (e.g., liver lesions or bone metastases in a chest CT) are clinically important.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a systematic empirical study of how to allocate the vision-token budget in 3D radiology report generation. It compares three input regimes (whole-volume encoding, anatomy-guided ROI cropping, and higher-resolution encoding), four frozen vision encoders (Primus, Curia, CT-FM, Merlin), five token-reducing projectors plus an uncompressed MLP baseline, and five instruction-tuned LLMs on CT-RATE and Merlin. The central finding is that ROI cropping improves clinical macro F1 in 19 of 20 VE-projector-dataset configurations at matched LLM token budgets, and that further resolution gains are projector-dependent, benefiting mainly TokenPacker and PerceiverResampler. The paper reports state-of-the-art test-set clinical macro F1 of 49.5 on CT-RATE and 49.0 on Merlin, and it releases code, models, and a ModernBERT classifier for Merlin.","tokens_in":18864,"tokens_out":7421,"duration_ms":75004,"significance":"This is a carefully controlled, large-scale study (three seeds, roughly 66,000 GPU-hours) that provides useful evidence on a design question rarely isolated in the 3D VLM literature: where the fixed vision-token budget should be spent. The internal evidence for the ROI-cropping effect is strong and clearly tabulated (Table 4), with standard deviations and per-cell comparisons. The paper also gives a clean characterization of projector behavior under compression (Table 5) and makes concrete recommendations, including the caution that most token-reducing projectors do not beat an MLP baseline. The public release of code, models, and the Merlin classifier is a concrete asset. The significance is tempered by two issues: the ROI crops coincide with the abnormality classes used for evaluation, which limits external validity of the 'most consistent strategy' claim, and the final SOTA comparison (Table 7) does not report uncertainty, so the 1.5-point lead over AdaRAG-CT may be within seed noise.","major_comments":[{"comment":"The headline ROI-cropping claim is confounded with label-set alignment. On CT-RATE the ROI is the bounding box of the five lung lobes (Section 3.1) and the clinical metric scores exactly 18 pulmonary and cardiovascular classes (Section 3.2.1); on Merlin the ROI is the whole-body foreground and the metric scores 30 abdominal classes. The paper itself states in Section 5 that a lung-crop 'removes anatomy the reference reports describe, which can encourage statements about structures the input no longer contains.' The F1 gain may therefore partly reflect the crop removing findings that the metric neither rewards nor penalizes, rather than a general improvement in report quality. To support the broad recommendation, please provide a concrete test on a setting where the crop is not aligned with the evaluation label set (for example, scoring extra-ROI findings such as liver lesions or bone metastases in chest CT, or evaluating on a full-body/open-set dataset), or substantially qualify the recommendation and the wording 'generally applicable' in the abstract and Section 5.","section":"§3.1 and §5"},{"comment":"The state-of-the-art comparison reports seed-averaged test-set macro F1 values without standard deviations or confidence intervals. The claimed lead over AdaRAG-CT is 1.5 points (49.5 vs. 48.0), while seed-to-seed standard deviations in the validation results of Table 4 are typically 0.5-1.5 F1 points for the same configurations. As reported, the SOTA claim is not statistically distinguishable from a tie. Please report test-set standard deviations/confidence intervals across seeds for both of the 'Ours' rows and for any recomputed baselines, or temper the SOTA claim to 'comparable or best in our evaluation setup.'","section":"Table 7 and §4.4"},{"comment":"The claims that the vision encoder is the dominant axis and that LLM choice has a smaller effect rely on configurations for which hyperparameters were selected only for Primus-1B on CT-RATE and then transferred to all other VEs, datasets, and LLMs. The paper acknowledges this in Section 5, but the limitation is load-bearing for the relative-importance conclusions in Section 4.3: the comparison is between a tuned development configuration and untuned transferred configurations. Please either retune the most important off-development settings (at least the LLM rows in Table 6) or present the VE/LLM relative-importance statement explicitly as a lower-bound comparison under shared hyperparameters.","section":"§3.4 and §4.3"}],"minor_comments":[{"comment":"The '19 of 20' count treats the Curia ConcatAdapter cell on CT-RATE (48.0 to 48.0) as a non-improvement; please state explicitly that one cell is a tie rather than an improvement so readers can verify the count.","section":"Table 4"},{"comment":"The Merlin baseline rows in Table 7 are recomputed with the authors' ModernBERT/GPT-OSS evaluation framework; the caption should state whether the published Merlin and Jolia papers used the same label extraction procedure, since differences in label extraction could affect comparability.","section":"§3.2.1 and Table 7"},{"comment":"The Limitations paragraph already discloses the ROI-label alignment issue; consider moving a one-sentence version of this caveat to the abstract or Section 4.1 so readers do not encounter the strong recommendation without the associated scope restriction.","section":"§5"},{"comment":"For the MLP baseline rows, the token count is much larger than the compressed rows (for example, 8,192 vs. 1,024 for Curia-8S); the table caption should remind readers that MLP rows are not at the same token budget as the compressed rows, even though they appear in the same table.","section":"§4.2 / Table 5"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical paper with unusually transparent reporting, and the internal consistency of the main table is not in doubt. The main risk is that the ROI-cropping recommendation is partly an artifact of label-set alignment, which reviewers will likely see immediately. I recommend requiring either an additional out-of-alignment experiment or a decisive reframing of the claim, plus uncertainty quantification for the SOTA table, before accepting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this is a serious, large-scale empirical study that maps where vision-token budgets should go in 3D radiology report generation. The headline result—ROI cropping at matched LLM token budgets improves clinical macro F1 in 19 of 20 configurations—holds up internally, but the crop and the evaluation labels are aligned in both datasets, so the 'general recipe' claim is weaker than it looks.\n\nWhat is actually new: a controlled comparison that holds the LLM token budget fixed while varying FOV, resolution, projector, vision encoder, and LLM. Four VEs, five token-reducing projectors up to 64x compression plus an MLP baseline, five LLMs, two datasets with more than 25k studies each, three seeds per configuration. Table 4 is clean and directly supports the ROI-cropping result across 20 cells with means and standard deviations. The paper also releases a ModernBERT classifier for the Merlin 30-class labels, which is a useful community asset. The authors are unusually candid about their own limitations: hyperparameters were tuned only for Primus-1B on CT-RATE and transferred elsewhere, one LLM collapse is reported, and Section 5 openly states that a lung-crop removes anatomy the reference reports describe.\n\nThe main soft spot is the label-alignment confound. On CT-RATE the ROI is the lung-lobe bounding box and the RadBERT metric scores exactly 18 pulmonary and cardiovascular classes. On Merlin the foreground ROI covers the abdomen and the evaluation uses 30 abdominal classes. So part of the F1 gain from cropping may come from suppressing findings outside the ROI that the metric neither rewards nor penalizes. This does not invalidate the internal comparison, but it does mean the recommendation 'crop to the ROI' is proven only for closed-set, label-aligned evaluations, not for open-set or full-body report generation. That caveat should be in the abstract, not just in Section 5.\n\nSecondary issues are proportionate: Table 7 reports test-set numbers without standard deviations, so the 1.5-point lead over AdaRAG-CT is within plausible seed noise; cross-LLM results use transferred hyperparameters and are explicitly lower bounds; code is not yet released. None of these are fatal. The clinical metric is a proxy trained on LLM-extracted labels, and the authors say so.\n\nWho this is for: anyone building 3D medical VLMs and deciding between whole-volume, ROI, higher-resolution inputs, and projector families. It deserves a serious referee. I would send it to peer review, ask for test-set error bars, a more prominent statement of the label-alignment scope, and ideally a code release before acceptance.","headline":"A large, careful empirical map of token-budget choices for 3D radiology report generation; the ROI-cropping result is real inside the evaluation setup, but label alignment limits how general the recommendation can be.","tokens_in":19544,"tokens_out":2918,"would_cite":true,"duration_ms":29312,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"At a fixed vision-token budget, anatomy-guided cropping is the most consistent lever for 3D radiology report generation, improving clinical macro F1 in 19 of 20 configurations.","keywords":["3D radiology report generation","vision-language model","token compression","vision-to-language projector","anatomy-guided ROI cropping","clinical macro F1","CT report generation"],"falsifier":"Retrain the best CT-RATE configuration with a whole-body or foreground ROI covering all organs and evaluate it with a classifier whose label set includes extrapulmonary findings (liver, kidney, bone); if the lung-crop configuration no longer beats the whole-volume baseline, the ROI gain was an artifact of label–crop alignment.","tokens_in":18422,"feed_emoji":"🩻","tokens_out":8379,"duration_ms":81588,"temperature":0.7,"pith_summary":"Generating radiology reports from whole 3D CT scans is expensive because a single scan can produce tens of thousands of visual tokens for the language model. This paper isolates one design question: given a fixed number of tokens, where should the budget be spent—on more of the body, on finer detail, or on better compression? Across two large CT-report datasets, four vision encoders, five token-reducing projectors, and five instruction-tuned language models, the answer is consistent: cropping to the anatomy of interest before encoding is the most reliable improvement, raising clinical macro F1 in 19 of 20 configurations by +3.7 points on average for the 3D ViT encoder and +1.1 for the 2D ViT encoder. Pushing input resolution beyond that only helps when the projector can turn many fine tokens into the same output without losing detail; the best configurations reach 49.5 macro F1 on CT-RATE and 49.0 on Merlin.","feed_headline":"Anatomy-guided cropping beats resolution gains in 19 of 20 CT settings","feed_subtitle":"Fixed-token CT report models improve most from cropping to relevant anatomy, not from higher resolution.","key_machinery":"The central object is the per-volume vision-token budget, held fixed across all comparisons, paired with three input regimes (whole volume, ROI crop, higher resolution) and projector families that reduce tokens either by local grouping or by learned global queries. The budget-matching design is what lets the paper separate 'how much context from where' from 'how many tokens the LLM sees.' The load-bearing mechanisms are the anatomy-guided crop (from segmentation masks) and the aggregation depth of query-based projectors, especially multiple Perceiver layers; the paper's empirical finding is that only projectors that can compress many fine tokens via iterative attention benefit from higher-resolution input.","core_discovery":"The paper's central claim is that at a matched vision-token budget, the allocation of that budget dominates projector and language-model choices for 3D radiology report generation. Anatomy-guided ROI cropping—cropping the volume to a segmentation-derived bounding box of the relevant anatomy before encoding—is the most consistent intervention, improving clinical macro F1 in 19 of 20 vision-encoder-projector-dataset settings, with average gains of +3.7 for Primus and +1.1 for Curia. Higher-resolution inputs only pay off when combined with projectors that can aggregate many encoder tokens under strong compression: the PerceiverResampler and, for Primus on Merlin, TokenPacker; grid- and pooling-based projectors often degrade. The vision encoder is the largest performance axis (over 15 F1 points between best and weakest), while language-model choice matters less (2–5 F1 points once retuned). At low compression, an uncompressed MLP projector remains a strong reference; among token-reducing projectors, the PerceiverResampler stands out, matching or exceeding the MLP baseline for Curia at high compression. The best reported configurations achieve 49.5 clinical macro F1 on CT-RATE and 49.0 on Merlin, exceeding the strongest published comparison systems on those benchmarks.","pith_inferences":["Editorial extension: if the CT-RATE label set were expanded beyond the lung and cardiovascular classes, the crop advantage could shrink or reverse; the paper's own limitation note identifies this risk.","Editorial extension: the same token-budget control could be applied to other 3D vision-language tasks, suggesting that ROI selection and projector aggregation depth should be co-designed rather than chosen independently.","Editorial extension: because only 1.7B–4B LLMs were tested, the finding that the vision encoder dominates may not hold at larger scales, where the LLM can extract more per token; re-running the matrix at 7B+ would settle it."],"forward_implications":["For any 3D radiology VLM with a fixed LLM context limit, replacing whole-volume resizing with a segmentation-based anatomy crop is likely the cheapest reliable win, worth roughly 3–5 macro-F1 points on chest CT and about 1 point on slice-based encoders, without changing downstream token count.","When compression is unavoidable, PerceiverResampler and TokenPacker are the projector families to prefer; ConcatAdapter and AveragePool degrade sharply at high compression ratios.","The dominant design choice is the vision encoder, not the language model; spending effort on encoder selection or pretraining should precede LLM swaps.","If higher-resolution inputs are desired, they must be paired with a projector able to aggregate the extra tokens; otherwise the larger input hurts performance.","The best configurations set new state-of-the-art clinical F1 on CT-RATE (49.5) and Merlin (49.0), showing that token-compressed pipelines can match or beat uncompressed ones when the projector is chosen correctly."],"supporting_citations":[{"why":"Supplies the CT-RATE dataset and the RadBERT-based clinical evaluator that defines the primary macro F1 metric.","marker":"[7]"},{"why":"Supplies the Merlin abdominal CT dataset and the Merlin vision encoder, grounding the cross-dataset and CNN-encoder comparisons.","marker":"[8]"},{"why":"Supplies the Primus 3D ViT vision encoder whose single-block and four-block token grids carry the main ROI and resolution results.","marker":"[20]"},{"why":"Supplies the Curia 2D ViT encoder used for the slice-based and higher-axial-resolution experiments.","marker":"[21]"},{"why":"Supplies the Qwen3-VL-2B language-model backbone and M-RoPE positioning used in the development configuration.","marker":"[25]"},{"why":"Supplies the TokenPacker grid-based projector that is one of the two projectors robust to high compression.","marker":"[26]"},{"why":"Supplies the PerceiverResampler query-based projector, the strongest token-reducing configuration for Curia.","marker":"[27]"},{"why":"Supplies the TotalSegmentator masks used to define the anatomy-guided ROI crops.","marker":"[28]"},{"why":"Supplies the token-preserving MLP projector baseline that most compressed projectors are measured against.","marker":"[5]"},{"why":"Supplies RadBERT, the classifier used to compute clinical macro F1 for CT-RATE report evaluation.","marker":"[39]"}],"fun_headline_variants":["Cropping anatomy outperforms higher res in 19 of 20 CT settings","For 3D CT reports, vision token allocation matters most","Crop anatomy, not pixels: winning move for CT report models","Allocate vision tokens to anatomy, not resolution, for 3D CT","Budget hints: anatomy crop beats resolution for CT report AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline cropping benefit assumes the clinical metric scores exactly the anatomy kept in the crop: if CT-RATE were scored on extrapulmonary findings too, the lung crop would remove structures the reference reports describe.","fun_headline_variants_meta":{"raw":{"variants":["Cropping anatomy outperforms higher res in 19 of 20 CT settings","For 3D CT reports, vision token allocation matters most","Crop anatomy, not pixels: winning move for CT report models","Allocate vision tokens to anatomy, not resolution, for 3D CT","Budget hints: anatomy crop beats resolution for CT report AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000899,"raw_usage":{"total_tokens":3963,"prompt_tokens":1131,"completion_tokens":2832,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":747,"completion_tokens_details":{"reasoning_tokens":2740}},"tokens_in":747,"tokens_out":2832,"duration_ms":20735,"temperature":1.0,"reasoning_tokens":2740,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:26:40.896888+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the best CT-RATE configuration with a whole-body or foreground ROI covering all organs and evaluate it with a classifier whose label set includes extrapulmonary findings (liver, kidney, bone); if the lung-crop configuration no longer beats the whole-volume baseline, the ROI gain was an artifact of label–crop alignment.","supporting_citations":[{"cited_title":"Merlin: A computed tomography vision-language foundation model and dataset,","cited_arxiv_id":null,"evidence_quote":"Supplies the Merlin abdominal CT dataset and the Merlin vision encoder, grounding the cross-dataset and CNN-encoder comparisons."},{"cited_title":"Primus: Enforcing attention usage for 3d medical image segmentation,","cited_arxiv_id":null,"evidence_quote":"Supplies the Primus 3D ViT vision encoder whose single-block and four-block token grids carry the main ROI and resolution results."},{"cited_title":"Tokenpacker: Efficient visual projector for multimodal llm,","cited_arxiv_id":null,"evidence_quote":"Supplies the TokenPacker grid-based projector that is one of the two projectors robust to high compression."},{"cited_title":"Flamingo: a visual language model for few-shot learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the PerceiverResampler query-based projector, the strongest token-reducing configuration for Curia."},{"cited_title":"Radbert: adapting transformer-based language models to radiology,","cited_arxiv_id":null,"evidence_quote":"Supplies RadBERT, the classifier used to compute clinical macro F1 for CT-RATE report evaluation."}],"review_version":1}