{"id":"fa0d9b7c-99e0-449a-a7af-3b4a40ac6005","arxiv_id":"2604.13561","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Random unbalanced batch sampling outperforms explicit class balancing for zero-shot learning in a reproduced 3D CT image-text model, with sub-linear gains from more data.","lead":"The paper reproduces a CLIP-style dual-encoder model for aligning 3D abdominal CT volumes with radiology reports and tests how the mix of normal versus abnormal cases in training batches affects zero-shot diagnostic performance. A smart generalist might read it to learn practical lessons on data sampling when building medical AI systems that must work with small batches of large 3D images.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.3","headline":"Uncontrolled differences in training dynamics could explain the observed gaps between random and balanced sampling","rationale":"The reader's weakest assumption is precisely the load-bearing uncertainty identified above. Because the full text was not supplied in the initial query, the current UNVERDICTED status already reflects this gap; the concrete test above would directly address it without requiring new data collection.","tokens_in":1827,"tokens_out":351,"duration_ms":27235,"concrete_test":"Re-run the 50:50 balanced configuration while explicitly logging and enforcing identical total optimizer steps, identical number of unique volumes per epoch, and identical subsection alternation frequency as the unbalanced baseline; if the 2–3 point gap disappears under these matched dynamics, the claim that batch composition itself drives the difference is weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that random sampling plus Merlin's alternating anatomical batching regularizes better than explicit normal:abnormal ratios at small batch sizes. The experiments compare an unbalanced random baseline against three section-level balanced ratios (25:75, 50:50, 75:25) and a 50:50 ablation on a data subset. All balanced variants underperform by 2.4–2.8 F1 points. However, section-level balanced sampling necessarily changes which volumes are drawn together and how often each anatomical subsection appears within a batch. The paper does not state whether the balanced runs preserve the exact same total training steps, the same per-epoch unique volume count, the same subsection alternation schedule, or the same effective learning-rate schedule as the baseline. If any of these quantities differ, the performance delta cannot be attributed solely to class ratio.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript reproduces the Merlin dual-encoder model for contrastive alignment of 3D abdominal CT volumes with radiology reports using symmetric InfoNCE loss, reaching 74.45% zero-shot macro F1 across 30 findings (original 73.00%). It then ablates batch composition on the full dataset by comparing random sampling against section-level balanced sampling at normal:abnormal ratios of 25:75, 50:50, and 75:25, finding all balanced variants underperform the random baseline by 2.4–2.8 F1 points (best balanced: 72.02%). A data-scaling study on a 4,362-study subset shows sub-linear gains from 65.26% (20% data) to 71.88% (100% data), with 50:50 balancing on the subset further dropping to 68.01%. The authors conclude that stochastic diversity from random sampling plus Merlin’s alternating anatomical subsection batching regularizes better than engineered class ratios at the small batch sizes required by 3D volumes.","tokens_in":1967,"tokens_out":730,"duration_ms":35090,"significance":"If the observed gaps are shown to arise from batch composition rather than uncontrolled training dynamics, the result would be significant for medical vision-language modeling: it would indicate that explicit class balancing is counterproductive for contrastive pre-training on imbalanced 3D CT data and that preserving natural distributions with stochastic sampling plus anatomical alternation is preferable. The reproduction (comparable F1, same directional balancing effect on full and subset data) and the finding that individual findings vary sharply in data sensitivity are concrete contributions.","major_comments":[{"comment":"The central comparison of random versus section-level balanced sampling (abstract and experimental results) does not state whether the balanced runs preserve the same total training steps, the same per-epoch unique volume count, the same subsection alternation schedule, or the same effective learning-rate schedule as the random baseline. Because section-level balancing necessarily changes co-occurrence statistics and sampling frequencies, any of these differences could produce the reported 2.4–2.8 point F1 gaps; the performance delta therefore cannot yet be attributed solely to class ratio.","section":"batch-composition experiments"},{"comment":"No error bars, standard deviations across runs, or statistical tests are reported for the F1 differences between random and balanced configurations (abstract and results). Given that the reproduction reaches only 74.45% and the reader notes limited implementation detail, demonstrating that the gaps are statistically reliable would be required to support the claim that random sampling plus alternation is superior regularization.","section":"results"}],"minor_comments":[{"comment":"The abstract refers to “Merlin’s alternating batching over anatomical subsections” without specifying how the alternation is implemented or whether it is held constant across the random and balanced conditions.","section":"abstract"},{"comment":"The 4,362-study subset and the 20%/40%/100% splits are mentioned but not described (e.g., whether splits are stratified by finding prevalence or by patient).","section":"data-scaling ablation"}],"recommendation":"major_revision","confidential_remarks":"The work is primarily a reproduction plus controlled ablation; its fit to a high-impact CV venue will depend on how convincingly the authors close the training-dynamics confound. Citation of the original Merlin paper appears appropriate, but the manuscript should explicitly delineate what is new versus reproduced."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments on our reproduction and ablation study. We address each major comment below, indicating planned revisions where appropriate.","responses":[{"response":"We appreciate this clarification request. All runs used the same total number of optimization steps, the same learning-rate schedule, and the identical subsection alternation schedule from the Merlin reproduction. The per-epoch unique volume count necessarily differs under balancing, as this is inherent to the class-ratio ablation. We will revise the experimental setup section to explicitly document these controls, enabling clearer attribution of the observed gaps to batch composition while acknowledging the changes in sampling frequencies.","revision_made":"yes","referee_comment":"The central comparison of random versus section-level balanced sampling (abstract and experimental results) does not state whether the balanced runs preserve the same total training steps, the same per-epoch unique volume count, the same subsection alternation schedule, or the same effective learning-rate schedule as the random baseline. Because section-level balancing necessarily changes co-occurrence statistics and sampling frequencies, any of these differences could produce the reported 2.4–2.8 point F1 gaps; the performance delta therefore cannot yet be attributed solely to class ratio."},{"response":"We agree that variability metrics would strengthen the claims. Due to the high computational cost of 3D CT contrastive training, results are reported from single runs. The directional effect is consistent across the full dataset and the independent 4,362-study subset. We will add a limitations paragraph discussing single-run results and the cross-experiment consistency, while noting that multiple seeds would be preferable if resources permit.","revision_made":"partial","referee_comment":"No error bars, standard deviations across runs, or statistical tests are reported for the F1 differences between random and balanced configurations (abstract and results). Given that the reproduction reaches only 74.45% and the reader notes limited implementation detail, demonstrating that the gaps are statistically reliable would be required to support the claim that random sampling plus alternation is superior regularization."}],"tokens_in":1634,"tokens_out":441,"duration_ms":43500,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper reproduces Merlin on abdominal CT volumes and reports, then shows that three section-level balanced batch ratios all lose to the unbalanced random baseline. The same directional hit appears when they repeat the 50:50 test on a 4,362-study subset. They also give scaling curves from 20% to 100% of that subset, with performance moving from 65% to 72% and big per-finding differences in data hunger. The reproduction lands at 74.45% macro F1, a bit above the original 73% number. The batch-composition ablations and the scaling plots are the new empirical pieces; nothing like this level of ratio testing was in the prior Merlin work. The consistent direction across full and subset data is the cleanest part of the evidence. The main soft spot is exactly the one the stress test flags: we do not know whether the balanced runs kept the same total steps, the same per-epoch unique volume count, or the exact same subsection alternation schedule as the baseline. Changing the sampling rule necessarily changes which volumes travel together, so the performance gap could come from altered training dynamics rather than the ratio itself. No error bars or statistical tests are reported, and the abstract stays light on hyperparameter matching. That leaves the 2-3 point differences looking plausible but not yet locked down. This is useful reading for anyone training 3D medical vision-language models who has to decide whether to force class balance at small batch sizes. It is worth sending to peer review because the experiments are straightforward and the result pushes back on a common assumption, but any referee will ask for explicit confirmation that the training schedules were held constant.","headline":"Random sampling with Merlin's alternation beats explicit normal-abnormal balancing by 2.4-2.8 F1 points in this 3D CT CLIP setup, with sublinear scaling curves as a side observation.","tokens_in":2487,"tokens_out":424,"would_cite":false,"duration_ms":33550,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Random sampling with alternating anatomical batches outperforms class-balanced sampling for training 3D CT vision-language models.","keywords":["batch composition","zero-shot learning","3D CT imaging","vision-language models","contrastive learning","abdominal CT","data scaling","class balancing"],"falsifier":"Re-running the full training protocol multiple times with identical batch strategies but different random seeds to check whether the gap between random and balanced sampling stays consistent.","tokens_in":2690,"feed_emoji":"📊","tokens_out":706,"duration_ms":50204,"temperature":0.7,"pith_summary":"The paper reproduces a dual-encoder model that aligns 3D abdominal CT volumes with radiology reports through contrastive learning. It tests controlled normal-to-abnormal ratios in batches and finds every balanced setup underperforms the original random sampling baseline by 2.4 to 2.8 points in zero-shot macro F1. Separate scaling experiments on a data subset show performance rising sub-linearly with added studies, while forcing balance on the subset still lowers results. These patterns indicate that natural stochastic variety in random batches, paired with subsection alternation, supplies stronger regularization than deliberate class ratios when batch sizes stay small due to 3D volume memory limits.","feed_headline":"Random sampling beats class balancing in 3D CT vision-language models","feed_subtitle":"Unbalanced batches with anatomical alternation reach 74.45% zero-shot F1 while fixed ratios cut performance by more than 2 points and data,","key_machinery":"Merlin's dual-encoder architecture using symmetric InfoNCE loss together with its alternating batching over anatomical subsections.","core_discovery":"The paper establishes that the stochastic diversity of random sampling, combined with alternating batching over anatomical subsections, provides more effective regularization than engineered class ratios at the small batch sizes required by 3D medical volumes, as shown by balanced ratios dropping macro F1 from 74.45% to as low as 72.02% and by sub-linear gains from 65.26% to 71.88% across data fractions.","pith_inferences":["The advantage of random batching may extend to other 3D medical imaging tasks where GPU memory forces small batches.","The interaction between stochastic batch diversity and anatomical alternation could be tested by ablating the alternation step alone.","Sub-linear scaling implies that targeted data collection for underperforming findings may be more efficient than uniform dataset growth."],"forward_implications":["All tested balanced ratios (25:75, 50:50, 75:25) reduce zero-shot macro F1 by 2.4–2.8 points relative to the unbalanced baseline.","Performance on a 4,362-study subset rises sub-linearly from 65.26% at 20% data to 71.88% at 100% data.","Individual findings vary sharply in how much additional data improves their zero-shot detection.","Enforcing 50:50 balance on the subset further drops performance to 68.01%, confirming the pattern holds at different scales."],"fun_headline_variants":["Random sampling with anatomical alternation yields higher F1 than balancing in 3D CT","Balanced sampling reduces zero-shot macro F1 in abdominal CT vision-language models","Sub-linear scaling from 65.26% to 71.88% in abdominal CT data ablations","Small batch sizes favor random sampling over class balancing in 3D CT models"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"That measured performance gaps arise from the batch composition choices themselves rather than from uncontrolled differences in training dynamics, hyperparameters, or dataset properties.","fun_headline_variants_meta":{"raw":{"variants":["Random sampling with anatomical alternation yields higher F1 than balancing in 3D CT","Balanced sampling reduces zero-shot macro F1 in abdominal CT vision-language models","Sub-linear scaling from 65.26% to 71.88% in abdominal CT data ablations","Small batch sizes favor random sampling over class balancing in 3D CT models"]},"model":"grok-4.3","cost_usd":0.012429,"raw_usage":{"total_tokens":5462,"prompt_tokens":766,"num_sources_used":0,"completion_tokens":87,"cost_in_usd_ticks":124287000,"prompt_tokens_details":{"text_tokens":766,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":4609,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":766,"tokens_out":87,"duration_ms":46843,"temperature":1.0,"reasoning_tokens":4609,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-10T13:54:38.818083+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Re-running the full training protocol multiple times with identical batch strategies but different random seeds to check whether the gap between random and balanced sampling stays consistent.","supporting_citations":[],"review_version":1}