{"id":"20077a10-cdd3-4e9c-b11e-aeb760b1b825","arxiv_id":"2502.02438","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An adversarial domain alignment method steals a medical multimodal LLM's radiology report generation using natural images and an oracle LLM, without medical data.","lead":"The paper presents ADA-STEAL, a black-box attack that copies a medical multimodal model's radiology report generation using only natural images. By adding adversarial noise and using an open-source LLM to diversify reports, it reaches up to 99% of the victim's clinical metrics with no medical data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that adversarial noise 'overcomes the data distribution gap' is not directly evidenced; Table 4 contradicts it, as real X-rays (Pa=MIMIC-CXR) perform worse than CIFAR-100 after ADA-STEAL.","rationale":"The reader identified the proxy-transfer assumption as the weakest point. I agree that surrogate fidelity is critical, but the more load-bearing issue is that the paper's central explanatory claim—'domain alignment'—is never tested and is contradicted by its own Table 4. If the mechanism were truly aligning Pa to Pv, using Pv's own images as the starting distribution should give the best or near-best result; instead it gives the worst among the workable distributions. The authors' own explanation says adversarial noise on real X-rays distorts them, which undercuts the notion that the noise is aligning the domains. This matters because the abstract's headline claim ('data augmentation with adversarial noise is sufficient to overcome the data distribution gap') is a causal statement. The empirical attack may be valid and useful, but the claimed insight requires either a direct measurement of the gap or a revised, more modest framing (e.g., 'adversarial noise increases the diversity and medical relevance of victim outputs'). The proposed FID/MMD check is easy to run and would settle whether the alignment story holds. Until then, the paper should be accepted only conditionally, with the claim adjusted or evidence added.","tokens_in":1138,"tokens_out":774,"duration_ms":57198,"concrete_test":"Use the victim's vision encoder to compute a feature-space divergence (e.g., FID or MMD) between (a) real MIMIC-CXR train images and (b) CIFAR-100 images before and after ADA-STEAL perturbation. If the divergence does not decrease, the alignment claim is falsified. In addition, repeat Table 4 with Pa=MIMIC-CXR across epsilon in {0.05,0.1,0.2,0.3} and compare the best RAD-S to the best CIFAR-100 run. If real X-rays never match CIFAR-100, the 'distortion' explanation is insufficient and the central mechanism must be reframed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is mechanistic: FGSM perturbations (Eq. 6) align the attacker's natural-image distribution Pa with the victim's medical distribution Pv. The paper only reports downstream metrics; it never measures whether the distribution gap actually closes. Table 4 provides a direct falsifier: if ADA-STEAL aligns Pa to Pv, then starting from Pa = MIMIC-CXR (the victim's own training distribution) should be the easiest case, yet it yields RAD-S 7.9 vs 14.1 for CIFAR-100 on MIMIC-CXR test (and 10.4 vs 22.3 on IU X-RAY) at the same epsilon=0.05. The paper's explanation—'the synthesized adversarial images on the actual CXRs distort the image distribution'—concedes that FGSM pushes inputs out of distribution. Thus the observed benefit may come not from closing the domain gap but from a transferable perturbation that elicits diverse, clinically phrased outputs from the victim; the 'domain alignment' explanation is unsupported. Since the abstract claims that adversarial noise is 'sufficient to overcome the data distribution gap,' this is a load-bearing weakness, not a stylistic one. No error bars are provided, so effect sizes are uncertain.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ADA-STEAL, a black-box model stealing attack against medical multimodal large language models (MLLMs) for radiology report generation. The attacker queries the victim with natural images (e.g., CIFAR-100) to obtain pseudo-reports, then iteratively (I) fine-tunes a surrogate MLLM on these pseudo-labels, (II) enriches the pseudo-reports using an open-source LLM oracle that composes typical chest X-ray findings, and (III) generates FGSM perturbations on the query images to elicit more diverse, medically relevant victim outputs. Experiments on IU X-RAY and MIMIC-CXR with CheXagent as victim and IDEFICS/CHEXAGENT* as attacker report ROUGE-L, BERTScore, and RadGraph-Score; ADA-STEAL consistently outperforms a Knockoff baseline and reaches up to 0.99x of the victim's RAD-S on IU X-RAY.","tokens_in":14611,"tokens_out":4786,"duration_ms":45185,"significance":"The attack is timely and, if validated, has practical security implications: it demonstrates that a medical MLLM can be functionally replicated without any medical training data, using only public natural images and a publicly available oracle LLM. The paper's strengths include evaluation on two standard radiology datasets, two attacker architectures, ablations of the oracle and of the adversarial component, a sensitivity study over epsilon, and a careful discussion of why CheXbert is unreliable for out-of-distribution reports. The central explanatory claim that adversarial noise performs 'domain alignment' is, however, not directly supported and is in tension with one of the paper's own experiments.","major_comments":[{"comment":"The paper's central claim that the FGSM perturbation 'overcome[s] the data distribution gap' is contradicted by the authors' own distribution-swap experiment. At epsilon=0.05, starting from Pa=MIMIC-CXR, i.e., the victim's own training distribution, performs substantially worse than starting from Pa=CIFAR-100 (RAD-S 7.9 vs 14.1 on MIMIC-CXR; 10.4 vs 22.3 on IU X-RAY). The explanation given in Section 'Image distribution Pa' — that adversarial images on actual CXRs 'distort the image distribution from the actual CXRs' — states that the perturbation moves inputs out of distribution, which is the opposite of the alignment mechanism claimed in the abstract and in Eq. (4). The authors should either provide direct evidence that the perturbation closes the distribution gap (e.g., feature-space distance between Pa+delta and Pv, or a plot analogous to Figure 5 but showing image embeddings) or revise the claim to state that the benefit comes from a transferable perturbation that elicits diverse victim outputs, not from domain alignment.","section":"Domain Alignment (Eq. 6) and Table 4"},{"comment":"The method relies on the assumption that the surrogate Ma is a faithful proxy for computing gradients that transfer to the victim Mv, but no transferability measurement is reported. The authors do not quantify how often the victim's output for Xa+delta differs from its output for Xa, nor whether the differences correspond to the oracle-desired findings. A minimal experiment would compare ADA-STEAL's adversarial perturbation to random noise of the same L-infinity norm, or measure label flips/test-time agreement between Ma and Mv on the perturbed query set. Without such evidence, the contribution of the 'domain alignment' component to the attack's success is not isolated.","section":"Domain Alignment, Eq. (6)"},{"comment":"All reported metrics are from a single run, without error bars. Some comparisons in Table 3 are close (e.g., CHEXAGENT* +KNOCKOFF vs +ADA-STEAL on MIMIC-CXR RG-L: 23.2 vs 24.7; BERT-S 43.7 vs 44.5). The claimed advantages would be more convincing if the main tables reported at least 3 random seeds or bootstrap confidence intervals, since MLLM fine-tuning is known to be sensitive to initialization and data ordering.","section":"Tables 3 and 4, Figure 3"},{"comment":"The oracle report composition probabilities (80% abnormal, 10% normal, 10% original) and the oracle prompt To are introduced without sensitivity analysis. Since the report enrichment is one of the two core novel components, the authors should report how the attack performance varies with these probabilities (e.g., 50/25/25 or 100/0/0) or provide a rationale for the chosen values.","section":"Implementation details"}],"minor_comments":[{"comment":"The acronym 'ADA-STEAL' is inconsistently spaced (e.g., 'ADA-S TEAL'); please standardize.","section":"Throughout"},{"comment":"The text contains the phrase 'abrogation study'; this should be 'ablation study'.","section":"Ablative analysis"},{"comment":"The row Pa=∅ is not defined in the notation table; clarify that this row corresponds to the untuned open-access IDEFICS model, not an actual attacker distribution.","section":"Table 4"},{"comment":"Eq. (4) includes L(Mv(Xa+delta), \\bar{Y}), but the implementation in Eq. (6) replaces Mv with Ma; the approximation should be stated explicitly in the text rather than only in the narrative of the domain alignment section.","section":"Domain Alignment, Eq. (4) vs Eq. (6)"},{"comment":"The caption says 'performance of ADA-STEAL on IDEFICS with different ε', but the curves may show ratios to the victim; please clarify whether the y-axis is absolute or normalized.","section":"Figure 3a"},{"comment":"The discussion states that extending to stochastic decoding 'sees no fundamental limitation' without supporting evidence; this is speculative and should be labeled as such.","section":"Extended scope"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely within scope for a security venue. My main concern is the disconnect between the attack's strong empirical results and the unsupported 'domain alignment' mechanism; I believe this is fixable with additional experiments or a revised framing, hence major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a solid, first-of-its-kind empirical demonstration that a black-box medical MLLM can be functionally stolen using only natural images plus an oracle LLM. The core tables support the claim: ADA-STEAL beats Knockoff on every metric on both IU X-RAY and MIMIC-CXR, with ratios up to 0.99 RAD-S in one setup. The ablations earn their keep, showing both the oracle enrichment and the adversarial perturbation contribute. The paper is also honest about CheXbert being unusable for out-of-distribution reports, which I appreciate.\n\nThe main soft spot is the mechanistic story. The abstract says adversarial noise 'overcomes the data distribution gap,' but the paper never measures the gap closing, and Table 4 complicates the narrative: starting from real MIMIC-CXR radiographs performs worse than CIFAR-100 after ADA-STEAL. The paper's explanation—that perturbations on actual CXRs distort the image distribution—concedes that the perturbation is not aligning with Pv in any straightforward sense. What the data actually show is that FGSM perturbations computed against the proxy transfer to the victim and elicit more diverse, clinically phrased outputs. That's still a useful and interesting effect, but the 'domain alignment' framing oversells it. The authors should either revise the claim or provide direct evidence, such as distribution distance measures or an analysis of which perturbations transfer.\n\nSecond, there are no error bars anywhere—single runs. For a security claim where effect sizes matter (0.75x vs 0.99x), I would want repeated runs or at least a statement about variance. This is fixable.\n\nThird, the 'without expert knowledge' claim is slightly generous. The oracle prompt lists anatomical structures and normal/abnormal categories; that is not nothing. It is not medical image data, but it is domain priors. A more precise claim would be 'without access to medical image datasets.'\n\nMinor: epsilon differs between the main experiments (0.2) and Table 4 (0.05), which makes cross-table comparisons awkward.\n\nOverall, the central empirical claim holds up, the novelty is real, and the writing is clear. I would send this to peer review with a request to add error bars, soften the distribution-gap claim, and acknowledge the prompt priors. It is a strong candidate for a security or medical-AI venue.","headline":"First practical stealing attack on medical MLLMs; the empirical result holds up, but the 'domain alignment' explanation is shakier than the data warrant.","tokens_in":15137,"tokens_out":2585,"would_cite":true,"duration_ms":26814,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"With black-box access and public natural images, an attacker can replicate a medical multimodal large language model's radiology report generation, reaching up to 99% of the victim's clinical efficacy without any medical data.","keywords":["medical multimodal large language models","model stealing","adversarial domain alignment","radiology report generation","black-box attack","data-free model extraction","CIFAR-100","FGSM"],"falsifier":"Run the identical ADA-STEAL pipeline with ImageNet or Places365 instead of CIFAR-100 while keeping the victim, oracle, and epsilon fixed: if the RadGraph-Score ratio on IU X-RAY and MIMIC-CXR drops well below 0.85, the 'natural images suffice' claim is specific to CIFAR-100 rather than a general property of natural images.","tokens_in":14131,"feed_emoji":"🩻","tokens_out":7313,"duration_ms":66660,"temperature":0.7,"pith_summary":"The paper introduces ADA-STEAL, a model-stealing attack that copies a medical multimodal large language model's radiology report generation from black-box queries alone. The central claim is that data augmentation with adversarial noise is enough to bridge the distribution gap between natural images and the victim's medical image distribution, so the attacker never needs medical data. Using CIFAR-100 images, an oracle LLM to diversify reports, and FGSM perturbations computed on the attacker's own surrogate model, the stolen model reaches between 74% and 99% of the victim's clinical efficacy (RadGraph-Score) on IU X-RAY and MIMIC-CXR.","feed_headline":"Noise on natural images steals a medical X-ray AI","feed_subtitle":"Black-box queries with public CIFAR-100 photos recover up to 99% of the victim's clinical accuracy.","key_machinery":"The load-bearing mechanism is the iterative Adversarial Domain Alignment loop built on the hypothesis $X = M_a^{-1}(\\bar{Y})$: the attacker's surrogate $M_a$, trained on victim pseudo-labels, is used as a differentiable proxy to invert the image-to-report mapping, so FGSM noise on natural images pushes them toward images that elicit the oracle's diversified medical reports from the victim. FGSM (fast gradient sign method) provides the perturbation $\\delta = \\epsilon \\cdot \\text{sign}(\\nabla_{X_a} L(M_a(X_a), \\bar{Y}))$, and the oracle LLM provides the medically relevant target reports $\\bar{Y}$ that the perturbation is optimized toward.","core_discovery":"ADA-STEAL is the first functionality-stealing attack against medical MLLMs for radiology report generation. The attacker trains a surrogate model on pseudo-labels obtained by querying the victim with natural images, then iteratively fine-tunes the surrogate on those labels, uses an open-source text-only LLM prompted with chest X-ray anatomy to synthesize diverse medical reports, and computes an FGSM perturbation against the surrogate, treating the surrogate as an approximate inverse of the victim mapping. The perturbed images are queried on the victim and added to the transfer set. The paper reports that this pipeline yields a stolen model whose RadGraph-Score reaches 0.99 times the victim's on IU X-RAY and 0.90 times on MIMIC-CXR, outperforming a direct Knockoff Nets adaptation.","pith_inferences":["The counterintuitive result that starting from real MIMIC-CXR images performs worse than CIFAR-100 suggests the adversarial perturbation is not literally aligning domains but rather creating surrogate features the victim happens to respond to; that would mean the attack exploits quirks of the victim's decision boundary rather than true domain transfer.","The 'data-free' framing depends on an unstated external resource: the oracle LLM's medical knowledge. If a future attack replaced the oracle with a generic, non-medical text generator, the high RAD-S ratios would likely collapse, so 'no medical data' really means 'no medical imaging data.'","A direct testable extension: run the same pipeline with ImageNet or Places365 instead of CIFAR-100; if the RAD-S ratio stays high, the 'natural images suffice' claim generalizes, and if it fails, the effect is specific to CIFAR-100's particular class structure.","The paper's Table 4 shows random Gaussian noise as a query source fails completely, which suggests the perturbation is not merely adding texture but must operate on structured natural-image features, constraining which public data an attacker could practically use."],"forward_implications":["Black-box access to a medical MLLM's report-generation API is enough to build a functional clone; no medical images, labels, or domain expertise are required.","Existing model-stealing defenses that add noise to logit predictions do not apply here because the attacker only receives discrete generated text.","The same recipe could be extended to other MLLM tasks by swapping the oracle prompt, such as image captioning in other specialized domains.","Medical institutions cannot rely on data scarcity or privacy regulations to protect MLLM intellectual property once the model is served publicly.","The reported 0.90–0.99 RAD-S ratios imply the stolen model is clinically near-equivalent on radiology report generation, making downstream attacks such as transfer jailbreaks more plausible."],"supporting_citations":[{"why":"Supplies the victim model CheXagent and the CheXbench evaluation protocol (RAD-S, GPT-4) used for all main results.","marker":"Chen et al. 2024b"},{"why":"Provides the FGSM perturbation rule that the domain alignment step is built on.","marker":"Goodfellow, Shlens, and Szegedy 2015"},{"why":"Knockoff Nets is the baseline stealing method that ADA-STEAL extends and compares against.","marker":"Orekondy, Schiele, and Fritz 2019a"},{"why":"Supplies MIMIC-CXR, one of the two medical test datasets and part of the victim's training distribution.","marker":"Johnson et al. 2019"},{"why":"Supplies IU X-RAY, the second medical test dataset.","marker":"Demner-Fushman et al. 2016"},{"why":"Defines RadGraph-Score (RAD-S), the clinical efficacy metric used to measure stealing success.","marker":"Jain et al. 2021"},{"why":"Supplies Zephyr-7B, the oracle LLM used for medical report enrichment.","marker":"Tunstall et al. 2023"},{"why":"Supplies IDEFICS, the public MLLM used as one of the attacker models.","marker":"Laurencon et al. 2024"}],"fun_headline_variants":["Natural photos plus noise clone medical X-ray AI","Adversarial noise enables stealing medical image AI","First attack steals radiology LLM using public images","Domain alignment steals medical AI without patient data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The attack works only if the attacker's surrogate model is a faithful-enough proxy for the victim that FGSM noise computed against the surrogate transfers to the victim and elicits diverse, medically relevant reports.","fun_headline_variants_meta":{"raw":{"variants":["Natural photos plus noise clone medical X-ray AI","Adversarial noise enables stealing medical image AI","First attack steals radiology LLM using public images","Domain alignment steals medical AI without patient data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000469,"raw_usage":{"total_tokens":2321,"prompt_tokens":917,"completion_tokens":1404,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":1345}},"tokens_in":533,"tokens_out":1404,"duration_ms":9852,"temperature":1.0,"reasoning_tokens":1345,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T12:07:12.014459+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical ADA-STEAL pipeline with ImageNet or Places365 instead of CIFAR-100 while keeping the victim, oracle, and epsilon fixed: if the RadGraph-Score ratio on IU X-RAY and MIMIC-CXR drops well below 0.85, the 'natural images suffice' claim is specific to CIFAR-100 rather than a general property of natural images.","supporting_citations":[{"cited_title":"J.; Shlens, J.; and Szegedy, C","cited_arxiv_id":null,"evidence_quote":"Provides the FGSM perturbation rule that the domain alignment step is built on."},{"cited_title":"E.; Pollard, T","cited_arxiv_id":null,"evidence_quote":"Supplies MIMIC-CXR, one of the two medical test datasets and part of the victim's training distribution."},{"cited_title":"D.; Rosenman, M","cited_arxiv_id":null,"evidence_quote":"Supplies IU X-RAY, the second medical test dataset."},{"cited_title":"Q.; Duong, D","cited_arxiv_id":null,"evidence_quote":"Defines RadGraph-Score (RAD-S), the clinical efficacy metric used to measure stealing success."}],"review_version":1}