{"id":"d34322f6-c1d5-43ac-9be4-ef02e9b91446","arxiv_id":"2411.09933","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"JRadiEvo shows that evolutionary model merging can adapt a vision-language model to generate Japanese chest X-ray reports using only 50 translated samples, beating larger baselines on ROUGE-L and METEOR.","lead":"The authors created a Japanese radiology report generator, called JRadiEvo, by merging a general vision-language model with medical and Japanese text models using an evolutionary algorithm, and using only 50 translated chest X-ray reports. The 8-billion-parameter model can run locally in hospitals and, on standard text-matching metrics, outperformed larger models such as CheXagent.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The quantitative comparison may be circular: CMA-ES is optimized against a ROUGE-L reference that is never explicitly identified as the 50 training reports rather than the test set; if test references were used, Table 2 proves selection, not generalization.","rationale":"The reader's weakest assumption concerns the validity of ROUGE-L and METEOR against machine-translated references as a measure of clinical quality. I agree that is a weakness, but I identify a more fundamental, prior condition: the same n-gram metric may have been used as the evolutionary search objective on data that is not separated from the test set. The paper's text is ambiguous at exactly the point where it matters, namely the optimization objective in Section 3.2, the test construction in Section 4.1, and the CMA-ES run in Section 4.2. If the test references were used for selection, every number in Table 2 is an artifact of optimizing on the evaluation set, and the comparison with CheXagent and GPT-4o is not a comparison of generalization. Even under the more favorable reading, where CMA-ES is evaluated on the 50 training reports, the absence of a stated test-set size and error bars means the small reported margins are not shown to be statistically meaningful. The paper's own Conclusion acknowledges that expert clinical judgment is still needed, so the abstract's accurate Japanese reports claim goes beyond what the evidence supports. I recommend keeping the conditional posture but adding the optimization and evaluation separation as a hard condition; if the authors cannot demonstrate that the test set was never used in the 600-iteration search, the verdict should move to rejection. This is not an accusation of misconduct; it is a request for the one piece of information that determines whether the central claim is interpretable.","tokens_in":14006,"tokens_out":6149,"duration_ms":65931,"concrete_test":"Ask the authors to state explicitly whether the ROUGE-L objective in Section 4.2 was computed on the 50 translated training reports or on the test set, and to release the exact split. Then rerun the CMA-ES search using only a held-out validation split (e.g., 40 training and 10 validation reports from the 50, or a fresh Japanese validation set) and recompute Table 2 on test references never touched during optimization. If JRadiEvo's ROUGE-L and METEOR no longer exceed CheXagent and GPT-4o, or if the differences fall within overlap, the central outperformance claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 and Section 4.2 state that the parameters were suggested by the evolutionary algorithm to maximize the ROUGE-L score between the generated text and the reference text, but they never state which reference set feeds that objective. Section 4.1 defines a held-out test set translated with GPT-3.5; a reader cannot tell whether CMA-ES's 600 iterations selected JRadiEvo on those same test references. If so, the headline claim that JRadiEvo outperformed leading models is circular: the model was explicitly optimized for the metric and references used in Table 2. Even if the EA used only the 50 translated training reports, the paper reports no test-set size, no error bars, and no repeated-seed variance, so a ROUGE-L gap of 0.013 over CheXagent and a METEOR gap of 0.003 over GPT-4o are not established as significant. The authors' own Conclusion concedes that human judgment by medical experts or further refinement may be needed, which undercuts the abstract's claim of accurate reports; the qualitative examples in Table 3 show generic templates and missed or hallucinated findings. The load-bearing assumption is therefore not just metric validity but the separation between optimization data and evaluation data.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes JRadiEvo, a Japanese chest X-ray report generation model built by evolutionary optimization of model merging. Starting from Bunny-v1_1-Llama-3-8B-V2 as a non-medical vision-language model, the authors merge it with two medical text LLMs (MMed-Llama-3-8B-EnIns, OpenBioLLM-Llama3-8B) and a Japanese LLM (Llama-3-Swallow-8B-Instruct-v0.1) using DARE, TIES-Merging, and CMA-ES, with only 50 human-reviewed, GPT-3.5-translated MIMIC-CXR training reports. The resulting 8B model is evaluated with BLEU, ROUGE-L, and METEOR on a held-out MIMIC-CXR test set whose Japanese references were produced by GPT-3.5, and compared against LoRA instruction-tuned versions of the same VLM, CheXagent, and GPT-4o. The authors report the highest ROUGE-L (0.212) and METEOR (0.191) among the compared models and argue that this demonstrates efficient low-resource adaptation, direct Japanese generation, local deployability, and superiority over instruction-tuning.","tokens_in":14240,"tokens_out":4062,"duration_ms":43612,"significance":"If the empirical claims are validated, the contribution is meaningful: it would be the first demonstration that evolutionary model merging can adapt a multimodal foundation model to a non-English medical domain using only 50 translated examples, avoiding backpropagation and catastrophic forgetting while producing a locally deployable 8B model. The manuscript is also commendable for describing the merge recipe in detail, for including instruction-tuning baselines and recent strong baselines such as CheXagent and GPT-4o, and for analyzing the relative contributions of the merged LLMs. However, the current evaluation does not support the headline claims: the optimization/evaluation separation is not explicitly stated, no test-set size or statistical significance is reported, the reference translations are machine-generated, and the qualitative examples show template-like outputs that miss or hallucinate findings. These are load-bearing issues for the central claim that JRadiEvo generates accurate reports and outperforms leading models.","major_comments":[{"comment":"The evolutionary optimization objective is not fully specified. Section 3.2 states that CMA-ES maximizes the ROUGE-L score between the generated text and \"the reference text,\" but it does not state whether this reference is the 50 translated training reports or the held-out test references introduced in Section 4.1. Since the algorithm runs for 600 iterations, optimizing on the test references would make the Table 2 comparison circular: the model would be selected for the exact metric and references used in the evaluation. The authors must state explicitly that only the 50 training reports were used in the CMA-ES objective, and report the size of the test set used in Table 2.","section":"Section 3.2 and Section 4.1"},{"comment":"The comparison lacks any measure of uncertainty. No test-set size, standard deviation, confidence interval, or significance test is reported, and the largest claimed margins are small: JRadiEvo's ROUGE-L exceeds CheXagent's by 0.013 and its METEOR exceeds GPT-4o's by 0.003. Without repeated evaluations, bootstrap intervals, or paired significance tests, the claim that JRadiEvo \"outperformed\" these models is not statistically established. The authors should provide error bars or significance tests, and ideally report results over multiple CMA-ES runs or random seeds.","section":"Section 4.2.2, Table 2"},{"comment":"The evaluation metric may not reflect clinical report quality. The test references were translated by GPT-3.5 without the human review applied to the training translations, so the metric scores compare against machine-generated Japanese. More importantly, the qualitative examples in Table 3 contradict the abstract's claim of \"accurate\" reports: Example 1 misses the ground-truth findings of reduced lung volume, scarring, and fibrosis; Example 2 misses the sternotomy wires and mediastinal clips and instead reports left ventricular enlargement and possible heart failure; Example 3 misses pleural effusion and metastatic nodules. The generated texts are short stereotyped templates, so the ROUGE-L/METEOR advantage may reflect template overlap rather than factual correctness. The authors should add human expert evaluation or clinically grounded metrics (e.g., finding-level factuality or CheXpert-label consistency) before claiming clinical accuracy or superiority.","section":"Section 4.1 and Table 3"}],"minor_comments":[{"comment":"The base VLM name is inconsistent: Section 4.1 and footnote 2 give \"Bunny-v1_1-Llama-3-8B-V2\" while Section 4.2.1 and footnote 6 give \"Bunny-v1_1-Llama-3-8B-V6\"; both footnotes point to the same URL, so the intended version should be clarified.","section":"Section 4.1 and Section 4.2.1"},{"comment":"The description \"randomly selected 50 samples from both views\" should specify how many AP and PA images were selected, and whether the 50 samples are images, reports, or image-report pairs; this matters because the paper emphasizes that only 50 cases were used.","section":"Section 4.1"},{"comment":"The LoRA baselines are said to be trained on 2,000 translated samples, but the paper does not state whether the same test set and the same GPT-3.5-translated references were used for their evaluation; this should be stated explicitly.","section":"Section 4.2.1"},{"comment":"Figure 1 is described as showing \"density and weight parameters after optimization,\" but the caption does not define \"density\" (presumably the retained percentage k_t) or the units of the weight c_t; axis labels and a legend should be added.","section":"Section 4.3, Figure 1"},{"comment":"There are several typographical errors: \"from scrach\" in Section 1, \"was was set\" in Section 4.2.1, \"and and revised\" in Section 4.1, \"a extremely limited dataset\" in Section 4.2.2, and inconsistent spacing in model names such as \"LLaV A\" and \"Med-PaLM M\". A careful proofread is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is a workshop-length submission, and the core idea—evolutionary model merging for a non-English medical VLM with very limited data—is timely and potentially interesting. My main concern is not novelty but the gap between the reported numbers and the strength of the claims. The optimization/evaluation separation must be clarified before the Table 2 comparisons can be interpreted, and the absence of uncertainty estimates is a serious issue for a paper whose headline is \"outperformed leading models.\" The qualitative examples also suggest that the clinical-accuracy framing in the abstract is too strong. I would not recommend rejection because these issues are fixable: clarifying the objective reference set, reporting test-set size and confidence intervals, and adding a small human evaluation or finding-level factuality check would substantially strengthen the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper has a legitimate new application — evolutionary model merging, borrowed from Akiba et al., used to adapt a VLM to Japanese radiology report generation with only 50 translated examples — but the evaluation is not strong enough to back the headline claims.\n\nWhat is actually new: applying the evolutionary TIES/DARE merging recipe to a medical vision-language task and a non-English language, with open models only. The method section is clear, the source models are named, and the idea is practical for privacy-constrained local deployment. The comparison against LoRA instruction-tuning on the same base VLM is a sensible baseline, and the analysis of per-model contributions (Figure 1) is a nice diagnostic. Credit where due: the paper is straightforward about its limitations in the conclusion.\n\nThe soft spots are real and load-bearing. First, the evaluation is thin: no test set size, no error bars, no significance tests. A ROUGE-L lead of 0.013 over CheXagent and a METEOR lead of 0.003 over GPT-4o are not meaningful without variance. Second, the reference texts for the test set are GPT-3.5 translations with no human review; the training translations were human-checked, but the test references were not, so the metrics themselves are noisy. Third, the qualitative examples contradict the 'accurate reports' framing — the model produces short templates, misses findings, and hallucinates (e.g., Example 2 invents left ventricle enlargement and heart failure on a post-sternotomy film with clear lungs). Fourth, the paper never states which reference set fed the CMA-ES objective. Section 3.2 says only that ROUGE-L was maximized against 'the reference text.' If that was the test set, Table 2 is selection, not generalization. I suspect it was the 50 training reports, but the omission is inexcusable in a paper whose entire claim is data-efficient generalization.\n\nThe core idea is worth pursuing, and the paper is a reasonable workshop contribution. But the performance claims need a proper evaluation: a stated test set, multiple seeds, bootstrap confidence intervals or significance tests, human expert assessment, and an explicit statement that the evolutionary objective used only training references. As it stands, the paper demonstrates a plausible method, not a proven result.\n\nRecommendation: I would not desk-reject it. Send it to reviewers, but tell them to focus on the evaluation gaps. The method is new enough and the application practical enough that it deserves referee time — but only with heavy revision.","headline":"A plausible data-efficient adaptation trick, but the evaluation is too thin to support the performance claims.","tokens_in":14818,"tokens_out":2932,"would_cite":false,"duration_ms":30207,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"JRadiEvo merges four off-the-shelf models with an evolutionary search over 50 translated samples and claims Japanese chest X-ray reports that beat CheXagent and GPT-4o on ROUGE-L and METEOR with no fine-tuning.","keywords":["Japanese radiology report generation","evolutionary model merging","vision-language model","chest X-ray interpretation","low-resource medical AI","TIES-Merging","DARE","catastrophic forgetting"],"falsifier":"Have board-certified radiologists blindly rate JRadiEvo's reports against the GPT-3.5-translated references for factual completeness and correctness, or recompute the ROUGE-L and METEOR comparison using human-translated rather than machine-translated references. If expert preference does not favor JRadiEvo, or the score gap over CheXagent and GPT-4o collapses with human references, the central claim that 50 samples suffice for accurate report generation would fail.","tokens_in":13766,"feed_emoji":"🩻","tokens_out":7102,"duration_ms":68953,"temperature":0.7,"pith_summary":"JRadiEvo is a Japanese chest X-ray report generator built without large Japanese training data: the authors merge four off-the-shelf Llama-3-based models—a vision-language model, two medical text models, and a Japanese text model—using an evolutionary algorithm to optimize TIES/DARE model merging, with only 50 GPT-3.5-translated MIMIC-CXR samples as the entire medical dataset. The paper's central claim is that this 8-billion-parameter model produces Japanese radiology reports that score higher on ROUGE-L and METEOR than the English-trained CheXagent, than GPT-4o, and than the same base model instruction-tuned with LoRA on 2,000 translated samples, without backpropagation and without catastrophic forgetting. This matters because non-English medical datasets are scarce and privacy rules often bar API use; a merged model small enough for local hospital deployment would remove the translation step and the need for large translated corpora. The paper is careful to say that expert human judgment is still needed before clinical use.","feed_headline":"Merging four models yields Japanese X-ray reports from 50 samples","feed_subtitle":"An 8-billion-parameter fusion of off-the-shelf models tops bigger English-trained systems on Japanese report metrics — no fine-tuning…","key_machinery":"The central mechanism is evolutionary model merging: a task vector $\\tau_t = \\theta_{\\mathrm{ft}}^t - \\theta_{\\mathrm{init}}$ is formed for each source fine-tune; DARE randomly drops and rescales entries; TIES-Merging trims small entries, elects a sign per parameter, and averages only matching-sign task vectors; and the final weights are $\\theta_{\\mathrm{final}} = \\theta_{\\mathrm{init}} + \\lambda \\tau_{\\mathrm{merged}}$. The evolutionary search, CMA-ES, optimizes the DARE drop rates, TIES retained percentages, per-task weights, and scaling parameter $\\lambda$, treating the whole LLM component as a single merged layer, with ROUGE-L as the fitness function. This lets the pipeline recombine skills—vision-to-text, medical text, and Japanese text—without backpropagation.","core_discovery":"On the paper's own terms, JRadiEvo establishes that evolutionary optimization of model merging can transplant medical knowledge and Japanese language ability into a vision-language model without any training-time gradient updates. Starting from Bunny-v1_1-Llama-3-8B-V2, the authors form task vectors from the fine-tuned weights of MMed-Llama-3-8B-EnIns, OpenBioLLM-Llama3-8B, and Llama-3-Swallow-8B-Instruct, apply DARE drop-and-rescale and TIES-Merging trim/elect/merge steps, and use CMA-ES to search the recipe parameters against ROUGE-L on the 50 translated reference reports. The optimized model reaches ROUGE-L 0.212 and METEOR 0.191 on the test set, the best among the compared systems, and the learned weights show OpenBioLLM carrying most of the medical signal while MMed-Llama contributes little. The qualitative examples, however, show short template-like sentences that omit several ground-truth findings and, in some cases, introduce abnormalities not present in the reference.","pith_inferences":["A testable extension the authors do not run is a sample-size curve: repeating the evolutionary merge with 10, 100, and 500 translated examples would show whether the 50-sample result is a threshold or a plateau.","The same merging recipe should transfer to other low-resource languages and other imaging domains, but such transfer is untested and likely depends on the availability of a language-capable base LLM and domain-specific medical LLMs.","The paper's own qualitative table suggests the metric lead may reflect fluency and template overlap rather than clinical completeness, since generated reports are generic while references list specific findings; human expert evaluation would be the real test, and the authors themselves note this gap.","Because the search optimizes ROUGE-L, the model is explicitly tuned to overlap with reference phrasing; a different fitness function, such as clinical entity correctness, might yield different merging weights and more specific reports."],"forward_implications":["If correct, JRadiEvo shows that a non-English medical vision-language model can be built from 50 translated cases, substantially lowering the data barrier for languages without public radiology report corpora.","Because the merge involves no gradient updates and the resulting model has 8 billion parameters, the same recipe could run on a single hospital GPU, keeping patient data in-house for privacy-sensitive environments.","The comparison suggests that for a model of this size, evolutionary merging is a more stable adaptation route than LoRA instruction-tuning, which the authors report suffered catastrophic forgetting when the training set was enlarged to 10,000 samples.","Direct Japanese generation eliminates the generate-in-English-then-translate workflow that CheXagent requires, making the model immediately usable in Japanese clinical settings.","The learned merging weights identify OpenBioLLM as the dominant source of medical knowledge and the Japanese model as secondary but necessary, indicating which source models future medical merges should prioritize."],"supporting_citations":[{"why":"Supplies the chest X-ray images and English reports from which the 50 training samples and the test set are drawn and translated.","marker":"[10]"},{"why":"Introduces evolutionary optimization of model merging recipes, the framework the paper adapts from natural images to medical chest X-rays.","marker":"[13]"},{"why":"Provides the TIES-Merging trim/elect/merge steps used to resolve interference among task vectors.","marker":"[33]"},{"why":"Provides DARE's random drop-and-rescale of task vector entries, a component of the merged recipe.","marker":"[34]"},{"why":"The base vision-language model whose LLM parameters are merged, and the model used for the LoRA instruction-tuning baselines.","marker":"[46]"},{"why":"The merged medical text model (OpenBioLLM-Llama3-8B) that the optimized weights rely on most according to the analysis.","marker":"[48]"},{"why":"The Japanese-language text model (Llama-3-Swallow-8B-Instruct) merged in to supply Japanese fluency.","marker":"[49]"},{"why":"CheXagent is the leading English chest X-ray baseline that JRadiEvo is compared against and outperforms on all reported metrics.","marker":"[30]"},{"why":"GPT-4o is the large API-based vision-language baseline; JRadiEvo beats it on ROUGE-L and METEOR but not on BLEU.","marker":"[54]"},{"why":"ROUGE-L is the fitness function used by the evolutionary search and a primary evaluation metric for the generated reports.","marker":"[36]"}],"fun_headline_variants":["Japanese X-ray reports from model merging, trained on just 50 samples","Model merging, not fine-tuning, learns Japanese radiology from 50 samples","Merging four models yields Japanese radiology reports from 50 examples","8B parameter model writes Japanese X-ray reports after seeing only 50 examples","CMA-ES merges models to generate Japanese X-ray reports from 50 cases"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quantitative claim depends on ROUGE-L and METEOR against GPT-3.5-translated Japanese references being a valid proxy for clinical report quality; the paper's own examples show generated reports that are short, templated, miss reference findings, and occasionally hallucinate abnormalities, so if the metrics reward that style, the 'accurate reports' claim overstates what the scores show.","fun_headline_variants_meta":{"raw":{"variants":["Japanese X-ray reports from model merging, trained on just 50 samples","Model merging, not fine-tuning, learns Japanese radiology from 50 samples","Merging four models yields Japanese radiology reports from 50 examples","8B parameter model writes Japanese X-ray reports after seeing only 50 examples","CMA-ES merges models to generate Japanese X-ray reports from 50 cases"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00075,"raw_usage":{"total_tokens":3402,"prompt_tokens":1074,"completion_tokens":2328,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":690,"completion_tokens_details":{"reasoning_tokens":2228}},"tokens_in":690,"tokens_out":2328,"duration_ms":18426,"temperature":1.0,"reasoning_tokens":2228,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:08:16.244549+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have board-certified radiologists blindly rate JRadiEvo's reports against the GPT-3.5-translated references for factual completeness and correctness, or recompute the ROUGE-L and METEOR comparison using human-translated rather than machine-translated references. If expert preference does not favor JRadiEvo, or the score gap over CheXagent and GPT-4o collapses with human references, the central claim that 50 samples suffice for accurate report generation would fail.","supporting_citations":[{"cited_title":"MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports,","cited_arxiv_id":null,"evidence_quote":"Supplies the chest X-ray images and English reports from which the 50 training samples and the test set are drawn and translated."},{"cited_title":"TIES-merging: Resolving interference when merging models,","cited_arxiv_id":null,"evidence_quote":"Provides the TIES-Merging trim/elect/merge steps used to resolve interference among task vectors."},{"cited_title":"Language models are super mario: Absorbing abilities from homologous models as a free lunch,","cited_arxiv_id":null,"evidence_quote":"Provides DARE's random drop-and-rescale of task vector entries, a component of the merged recipe."},{"cited_title":"OpenBioLLMs: Advancing open-source large language models for healthcare and life sciences,","cited_arxiv_id":null,"evidence_quote":"The merged medical text model (OpenBioLLM-Llama3-8B) that the optimized weights rely on most according to the analysis."},{"cited_title":"Continual pre-training for cross-lingual LLM adaptation: Enhancing japanese language capabilities,","cited_arxiv_id":null,"evidence_quote":"The Japanese-language text model (Llama-3-Swallow-8B-Instruct) merged in to supply Japanese fluency."},{"cited_title":"Chexagent: Towards a foundation model for chest x-ray interpretation,","cited_arxiv_id":null,"evidence_quote":"CheXagent is the leading English chest X-ray baseline that JRadiEvo is compared against and outperforms on all reported metrics."},{"cited_title":"GPT-4o system card,","cited_arxiv_id":null,"evidence_quote":"GPT-4o is the large API-based vision-language baseline; JRadiEvo beats it on ROUGE-L and METEOR but not on BLEU."},{"cited_title":"ROUGE: A package for automatic evaluation of summaries,","cited_arxiv_id":null,"evidence_quote":"ROUGE-L is the fitness function used by the evolutionary search and a primary evaluation metric for the generated reports."}],"review_version":1}