{"id":"1a5f1560-ff5c-4302-8e28-287504088172","arxiv_id":"2607.28125","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"Two-level validator-and-algorithm agreement selects a single UDA checkpoint without target labels and halves the gap to the labeled oracle versus the best individual validator on seven medical transfers.","lead":"A label-free scoring rule picks both the UDA algorithm and its hyperparameters by matching each candidate to a two-level agreement reference built only from unlabeled target images. That matters because clinics often cannot label the deployment scanner or site, yet still need a single deployable model.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Reported gains are in-sample on the same target val used to build the reference; held-out target generalization of the agreement ranking is untested.","rationale":"The reader’s weakest assumption (majority-vote reference tracks true target risk; shared failures do not dominate) is exactly the load-bearing premise of Eqs. 2–5. The manuscript’s own protocol choice—no held-out target test, scores recomputed on the evaluation split—makes that premise untested at the point where the strongest claim is quantified. Relative ordering versus individual validators may still hold under a matched protocol, so the contribution remains accept-shaped and CONDITIONAL is the right bar; the concern does not justify REJECT. It does confirm the reader’s request for a held-out target evaluation protocol before unqualified acceptance. No stronger internal inconsistency (math error, label leakage into selection, or contradictory ablation) was found. Code pointer is a placeholder, which limits pin-down reproducibility but is secondary to the selection/eval confound for the central claim.","tokens_in":10133,"tokens_out":625,"duration_ms":47766,"concrete_test":"For each of the seven scenarios, stratify-split the target domain 50/50 into selection-val and held-out test. Build ˜y and run Eqs. 4–5 only on selection-val; report the chosen checkpoint’s balanced accuracy on held-out test (and the same for Class-AMI / Oracle). If on held-out test Ours–Oracle gap returns near the single-validator gaps or Ours drops below Class-AMI on average, the headline Table 2 claim does not transfer off the selection set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim (Table 2: 86.3% vs 81.0% best validator, Oracle gap 5.1% vs 10.4%) rests on Eqs. 3–5 producing a ranking that tracks true target risk. Both the Level-2 reference ˜y_j (Eq. 3) and the agreement score s(θ) (Eq. 4) are computed on the identical target validation split on which balanced accuracy is then reported (§3 Experimental Setup: “We do not hold out a separate target test set”). Selection and evaluation therefore share the same unlabeled target samples. A checkpoint can maximize agreement with consensus errors or idiosyncrasies of that finite val set and still post a strong Table 2 number without being near-best under the true target distribution. This is the operational form of the shared-failure-mode risk in §2.2–2.3: when validators/algorithms err together on the evaluation samples, in-sample s(θ) and reported accuracy both look good while the selected ˆθ is suboptimal for deployment. Ablations in Fig. 2 vary pool size and checkpoint density but never break the selection/eval data identity, so they do not close this gap.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes a label-free criterion for jointly selecting a UDA algorithm and its hyperparameters for medical imaging classification. From a pool of checkpoints spanning multiple algorithms and hyperparameter settings, it builds a two-level agreement reference without target labels: (Level 1) each of several existing validators nominates the best checkpoint within each algorithm; (Level 2) those nominations are aggregated by majority vote over unlabeled target samples to form a reference prediction. Every candidate is then scored by macro per-class agreement with this reference (Eqs. 2–5), and the single highest-scoring checkpoint is deployed. On seven brain-MRI and CXR transfer scenarios the method reports 86.3% average target balanced accuracy versus 81.0% for the best individual validator (Class-AMI), cutting the gap to a labeled Oracle from 10.4% to 5.1% (Table 2), with ablations on aggregation structure, algorithm-pool size/composition, and checkpoint retention (Table 3, Fig. 2).","tokens_in":10461,"tokens_out":1487,"duration_ms":45357,"significance":"Joint algorithm-and-hyperparameter selection under unlabeled targets is a genuine clinical bottleneck that most UDA validator work leaves open. The two-level construction is a clear, implementable idea; the empirical scope (two modalities, seven transfers, different CXR backbones, five-fold means±std, many established validators, Oracle upper bound, and pool/density ablations) is stronger than typical medical UDA selection papers. Open-sourced code further supports reproducibility. If the selection quality holds under a stricter evaluation protocol, the work is a useful step toward practical UDA deployment rather than another adaptation loss.","major_comments":[{"comment":"§3 Experimental Setup explicitly states that no separate target test set is held out and that target performance is measured on the same target validation split used to form the Level-2 reference (Eq. 3) and agreement scores s(θ) (Eq. 4). Selection and reported accuracy therefore share the identical unlabeled samples. A checkpoint can maximize agreement with consensus errors or finite-sample idiosyncrasies of that split and still inflate Table 2 numbers without being near-best under the broader target distribution. This is load-bearing for the central claim (86.3% vs 81.0%, Oracle gap 5.1% vs 10.4%). Please re-evaluate with a protocol that separates the unlabeled data used to build ˜y and s(θ) from the labeled target split used only for reporting accuracy (e.g., select on target-train unlabeled features, report on target-val), or provide an equivalent held-out analysis. Fig. 2 ablations","section":"§3 Experimental Setup; Eqs. 3–5; Table 2"},{"comment":"The method’s premise (§2.2–2.3) is that majority vote among validator-nominated checkpoints across heterogeneous algorithms yields a reference correlated enough with true labels to rank a near-best single checkpoint. Shared failure modes among algorithms/validators would make both s(θ) and in-sample accuracy look good while ˆθ is suboptimal. Beyond overall averages, the paper should quantify reference quality (e.g., agreement of ˜y with true target labels per scenario, or cases where the selected checkpoint matches a systematically wrong consensus) and report how often the selected ˆθ is within a small margin of the Oracle versus merely better than Class-AMI. The remaining 5.1% Oracle gap and the larger shortfall on RSNA→Child CXR (Table 2) are natural places for this analysis; without it, the weakest assumption stays largely untested.","section":"§2.2–2.3; Table 2; Limitations"},{"comment":"Table 3 shows that the flat “All checkpoints” baseline already reaches 82.4% overall (vs 81.0% best validator and 86.3% two-level). The discussion offers a plausible diversity argument, but there is no statistical test or fold-level comparison establishing that the two-level gain over All checkpoints / Per-validator is reliable rather than scenario-noise. Given that the paper’s novelty is precisely the two-level structure, please report paired fold-wise differences (or confidence intervals) for Table 3 and clarify whether Level-1 validator nomination is necessary when a simple all-checkpoint vote is already competitive.","section":"Table 3; §4 Discussion"}],"minor_comments":[{"comment":"Abstract and elsewhere: “algori-thms” line-break hyphenation appears in the compiled text; clean up for camera-ready.","section":"Abstract"},{"comment":"Eq. (4): the macro per-class match to ˜y is reasonable, but for the imbalanced CXR sets (Table 1) it would help to state why macro agreement was preferred over micro agreement or balanced accuracy against the reference, and whether ranking is sensitive to that choice.","section":"§2.3 Eq. (4)"},{"comment":"The validator pool V is listed in §2.1 but which subset is used in the main experiments is not stated in one place; a short explicit list (and optionally a leave-one-validator-out check) would aid reproducibility.","section":"§2.1; §3"},{"comment":"Fig. 1 is helpful but the “Config.” / algorithm / validator layout is dense; ensuring the Level-1 vs Level-2 boundary is visually sharper would help non-specialist readers.","section":"Fig. 1"},{"comment":"Claim of being “the first” joint algorithm-and-hyperparameter selection method (§1, §5) should briefly contrast the sequential algorithm-then-hyperparameter protocol of Yang et al. [32] so the novelty boundary is precise.","section":"§1; §5"},{"comment":"Code link is given as “Complete UDA Pipeline” without a stable URL/DOI in the text; add a persistent link in the camera-ready version.","section":"Abstract / footnote"}],"recommendation":"major_revision","confidential_remarks":"The evaluation-protocol issue (same target val for building the agreement reference and for reporting accuracy) is the main reason I chose major_revision rather than minor_revision. Empirically the method looks promising and the problem is real; if the authors can show the gains survive a split where agreement scores and reported labels use disjoint target samples, I would expect this to clear a subsequent round. Scope fits a solid medical-imaging / MICCAI-style venue; not overclaiming theory."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing worth knowing: this is a clean empirical fix for a real clinical pain—picking which UDA algorithm and λ to deploy when the target has no labels. The two-level trick (validators nominate inside each algorithm, then majority vote across algorithms builds a reference, then you pick the single checkpoint that matches it) is the actual novelty. Hu et al. stayed inside one algorithm; Yang/TransScore staged algorithm then hyperparams. Joint selection via this agreement reference is new enough to matter for the people who actually ship adapters.\n\nWhat they did well is the evidence package. Seven transfers, MRI and CXR, different backbones on the X-ray side, ten algorithms, five-fold means, Oracle upper bound, and ablations on aggregation structure, pool size/composition, and checkpoint density. Table 2’s jump from 81.0% (best validator) to 86.3%, cutting the Oracle gap roughly in half, is the load-bearing result and it holds across scenarios. The aggregation ablation (Table 3) actually supports their design choice rather than just restating it. Limitations section is honest: binary tasks, compute cost, residual gap on RSNA→Child.\n\nSoft spot, in proportion: they evaluate balanced accuracy on the same unlabeled target validation split used to build the reference and the agreement scores. They say so explicitly and justify it by the label-free setting, but that still means Table 2 numbers are in-sample w.r.t. the selection data. Shared failure modes among validators/algorithms on that finite set could inflate agreement and reported accuracy together. Fig. 2 never breaks that identity. It is not a collapse of the claim—gains over strong individual validators are still informative—but a held-out target split (or at least a sensitivity check) would make the deployment story tighter. Minor relative to the rest: binary-only scope and the usual “train many algorithms” cost.\n\nMath is straightforward (majority vote + macro per-class match); citations cover the validator and selection literature fairly; no obvious citation games. Code pointer is a placeholder in the PDF, which is annoying but not unique.\n\nThis is for clinical-ML and medical UDA people who already run multiple adapters and need a label-free pick rule. Not a theory paper. I would send it to peer review; the contribution is scoped and the experiments are serious enough to deserve referees. Engage if you care about deployment under shift; skim the tables if you only need the selection idea.","headline":"Solid practical selection rule for medical UDA; real gains over validators, but selection and reported accuracy share the same target val split.","tokens_in":11081,"tokens_out":604,"would_cite":true,"duration_ms":20055,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A two-level agreement vote among UDA models can pick algorithm and hyperparameters without any target labels.","keywords":["unsupervised domain adaptation","algorithm selection","medical imaging","label-free model selection","domain shift","brain MRI","chest X-ray"],"falsifier":"On a held-out medical UDA transfer, if the checkpoint with highest agreement to the two-level reference is systematically far below both the best individual validator and the labeled oracle—especially when algorithms share failure modes—the selection claim fails.","tokens_in":10985,"feed_emoji":"🏥","tokens_out":841,"duration_ms":16850,"temperature":0.7,"pith_summary":"In clinical imaging, unsupervised domain adaptation (UDA) can transfer a model from a labeled hospital to an unlabeled one, but practitioners cannot score candidates on the target domain because labels are missing. This paper claims that joint algorithm-and-hyperparameter selection is still possible: build a label-free reference by letting many existing validators each nominate a checkpoint inside every algorithm, then majority-vote those nominations across algorithms on every unlabeled target image. Score every candidate by how closely its predictions match that reference, and deploy the single highest-scoring checkpoint. On seven brain-MRI and chest X-ray transfers the method reaches higher average target accuracy than any individual validator and roughly halves the remaining gap to the best checkpoint one could pick with labels. The same ranking stays stable when the algorithm pool shrinks or when fewer checkpoints are kept per run, so the procedure is meant to be usable under realistic compute budgets.","feed_headline":"No-label vote picks the best UDA model for clinical scans","feed_subtitle":"Two-level agreement among algorithms halves the gap to oracle on brain MRI and chest X-ray transfers","key_machinery":"Two-level agreement reference: Level 1 lets each validator pick its best checkpoint inside every algorithm; Level 2 majority-votes those nominations per target sample to form a pseudo-label reference; each candidate is then scored by mean per-class agreement with that reference and the top scorer is deployed.","core_discovery":"Given a heterogeneous pool of UDA checkpoints spanning algorithms, hyperparameters, and training iterations, a two-level agreement reference—validators nominating within each algorithm, then majority vote across algorithms—produces a label-free target against which candidates can be ranked, selecting a single deployable model whose target accuracy substantially beats individual validators and closes much of the gap to the labeled oracle.","pith_inferences":["The same two-level vote could be tried as a stopping rule during a single long UDA run, not only as a post-hoc selector across finished runs.","If shared failure modes among adversarial or pseudo-label methods dominate the vote, weighting validators by diversity of their nominated algorithms may be a natural next control.","Extending the agreement score from balanced accuracy on binary tasks to sensitivity-weighted or multi-class metrics would test whether the reference still ranks clinically preferred operating points."],"forward_implications":["A clinic can train several UDA algorithms with a few hyperparameter settings, keep periodic checkpoints, and pick one model for deployment without labeling target scans.","Adding algorithms from different paradigms improves the upper bound the selector can reach, while the same criterion still works on smaller pools when compute is limited.","Selection quality does not require dense checkpointing; retaining a modest fraction of snapshots per run remains effective.","The same label-free ranking can be recomputed on any new unlabeled target batch, so the chosen model can be refreshed when data drift."],"fun_headline_variants":["Two-level agreement picks best UDA model without target labels","Label-free vote ranks UDA checkpoints for clinical scan transfer","Agreement reference selects deployable UDA model across MRI and X-ray","Validators plus cross-algorithm vote close gap to oracle on UDA","Pool of UDA models ranked by label-free two-level agreement score"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The majority vote among validator-picked checkpoints from different algorithms is assumed not to share the same systematic mistakes, so ranking by agreement with that vote still tracks true target performance.","fun_headline_variants_meta":{"raw":{"variants":["Two-level agreement picks best UDA model without target labels","Label-free vote ranks UDA checkpoints for clinical scan transfer","Agreement reference selects deployable UDA model across MRI and X-ray","Validators plus cross-algorithm vote close gap to oracle on UDA","Pool of UDA models ranked by label-free two-level agreement score"]},"model":"grok-4.5","effort":"low","cost_usd":0.00187,"raw_usage":{"total_tokens":846,"prompt_tokens":747,"num_sources_used":0,"completion_tokens":79,"cost_in_usd_ticks":18704000,"prompt_tokens_details":{"text_tokens":747,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":20,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":747,"tokens_out":79,"duration_ms":2575,"temperature":1.0,"reasoning_tokens":20,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T17:04:14.632447+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On a held-out medical UDA transfer, if the checkpoint with highest agreement to the two-level reference is systematically far below both the best individual validator and the labeled oracle—especially when algorithms share failure modes—the selection claim fails.","supporting_citations":[],"review_version":1}