{"id":"dc055b25-345b-45a4-be64-5ec0838e68fa","arxiv_id":"2412.04954","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A LLaVA-style radiology report generator using LoRA fine-tuning and stitched chest X-ray inputs placed fourth in the RRG24 shared task.","lead":"This paper fine-tunes a vision-language model, built from a CLIP image encoder and a Vicuna-7B language model, to write Findings and Impressions sections of radiology reports from chest X-rays. The system placed fourth in the RRG24 shared task on large-scale radiology report generation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported hidden-test Findings F1-RadGraph (24.13) is identical to the public-test value; verify against the official RRG24 leaderboard before trusting the 4th-place claim.","rationale":"The reader's verdict is CONDITIONAL, and I agree with that overall disposition, but for a different primary reason. The reader identified the image truncation/stitching preprocessing as the weakest assumption. That is a valid limitation and should be addressed with an ablation, but it does not invalidate the factual claim that the model scored what it scored on the leaderboard. A more load-bearing issue is the internal inconsistency in Table 5: the Findings F1-RadGraph is identical (24.13) on public and hidden test sets while all neighboring metrics differ. If this is a reporting error, the central claim of a 4th-place finish (which depends on the hidden score) is not supported by the paper's own data. The proposed test is simple and decisive: compare Table 5 against the official leaderboard. If the score checks out, the current CONDITIONAL verdict could be maintained while asking for ablations; if it does not, the verdict should move toward REJECT. I marked agreement as 'partial' because the reader's detailed load-bearing concern and mine are distinct, though both point to the reliability of the reported evaluation.","tokens_in":13010,"tokens_out":7149,"duration_ms":73197,"concrete_test":"Retrieve the official RRG24 leaderboard at https://vilmedic.app/misc/bionlp24/leaderboard (or the task organizers' published results) and look up the Gla-AI4BioMed entry. Compare the hidden-test Findings F1-RadGraph value and the reported rank against the values in Table 5. If the leaderboard shows exactly 24.13 for hidden-test Findings, the identical public/hidden value is a coincidence and the claim stands. If the leaderboard shows a different value, then the paper's central numerical claim is inaccurate and the manuscript must be corrected; depending on the corrected value and rank, the verdict would move toward REJECT. Also cross-check the hidden-test Impressions score and the public-test entries if the leaderboard exposes them.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central empirical claim is the hidden-test F1-RadGraph scores in Section 5 and Table 5: 24.13 for Findings and 22.10 for Impressions, which the authors state places them 4th on the leaderboard. The most load-bearing assumption is that these numbers are accurately reported. A concrete anomaly undermines this assumption: the Findings F1-RadGraph is exactly 24.13 on both test-public and test-hidden, while every other metric in Table 5 differs between the two sets (e.g., BLEU4 8.07 vs 7.65, ROUGEL 24.90 vs 24.35, Bertscore 53.45 vs 52.69, F1-cheXbert 45.91 vs 46.21). Exact equality to two decimal places on two independent test sets is an unlikely coincidence and is more plausibly explained by a copy-paste or logging error. If the hidden Findings score is not actually 24.13, then the headline result and the associated rank are unsupported, and the paper's main numerical evidence fails. The reader's identified weak spot, the truncation and stitching of images, is a genuine limitation but does not threaten the factual accuracy of the leaderboard result; an inconsistent headline score does. The paper should be accepted only after the hidden-test values are confirmed against the official leaderboard.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a radiology report generation system developed for the RRG24 shared task. The authors fine-tune Vicuna-7B with a CLIP vision encoder and an MLP adapter in two stages: first aligning chest X-ray features with the language model, then applying LoRA fine-tuning for radiology report generation. Separate models are trained for Findings and Impressions sections, and up to four input images are horizontally stitched into a single image. The system is evaluated on the RRG24 validation, public test, and hidden test splits across five metrics. The central empirical claim is that the hidden-test F1-RadGraph scores are 24.13 for Findings and 22.10 for Impressions, placing the system 4th on the leaderboard at submission time.","tokens_in":13215,"tokens_out":4121,"duration_ms":44858,"significance":"If the reported hidden-test scores are accurate, the paper demonstrates a competitive recipe for adapting a general-purpose visual language model to radiology report generation with modest computational resources, and it provides a useful data point for the RRG24 benchmark. The manuscript is commendably transparent about its limitations and links to public code. However, the technical novelty is limited: the two-stage adaptation follows LLaVA-1.5 and LLaVA-Med closely, and the multi-image handling is a simple truncation-and-stitch heuristic with no supporting ablation. The main load-bearing evidence is a small set of leaderboard numbers, and at least one of those numbers appears to be internally inconsistent, which is why the result cannot be accepted without verification.","major_comments":[{"comment":"The hidden-test Findings F1-RadGraph is reported as exactly 24.13, identical to the public-test Findings F1-RadGraph, while every other metric in Table 5 differs between the two test sets (e.g., BLEU4 changes from 8.07 to 7.65 and ROUGEL from 24.90 to 24.35). An exact match to two decimal places on two independent test sets is highly implausible and is more likely explained by a copy-paste or logging error. Because the headline claim of a 4th-place finish rests on the hidden-test value, the authors must verify this number against the official RRG24 leaderboard and correct Table 5 and the text accordingly. If the value is confirmed, please provide evidence; otherwise the central empirical claim is unsupported.","section":"Section 5, Table 5"},{"comment":"Section 4.1 states that the maximum length is 1024 for both text input and inference output, but Section 4.3 states that inference decodes up to 150 tokens, consistent with the baseline. These two statements contradict each other. This is not a cosmetic issue: Table 3 reports an average Findings word count of 380 on test-public, so a 150-token cap could truncate a large fraction of generated reports and directly affect all reported metrics. The authors must clarify the actual decoding limit and, if 150 tokens was used, discuss the effect of truncation on the reported scores.","section":"Section 4.1 vs. Section 4.3"},{"comment":"The preprocessing choice of taking only the first four images and horizontally stitching them is not validated by any ablation or quantitative analysis. The paper claims this strategy is 'proven to be robust in our experiments' in Section 4.1, but no such experiment is reported, and Section 6 later concedes that performance 'may be compromised in multi-image inference scenarios where it does not account for superfluous images.' The authors should provide at least a comparison of stitching versus separate encoding, or an analysis of image-order sensitivity and the effect of discarding images beyond the fourth, to support the claim that the method preserves clinically relevant information.","section":"Section 4.1, Section 6"},{"comment":"There is an internal contradiction in the training description. Section 3 says the authors follow LLaVA-1.5 protocols 'including a joint tuning phase for the LLM and adapter,' but Section 3.3, Stage 2, states that 'the visual encoder weights and adapter are kept frozen while continuing to update the pre-trained LLM weights using LoRA.' These descriptions cannot both be true. Since the exact parameter-update scheme is essential for reproducing the method, the authors must state unambiguously whether the MLP adapter is updated in Stage 2 or frozen.","section":"Section 3, Section 3.3"}],"minor_comments":[{"comment":"There are several language errors, including 'the ability of model' in the Abstract, 'model ability of to mimic' in Section 3.1, and 'visual instrumental tuning' in Section 3.3, which should read 'visual instruction tuning.'","section":"Abstract, Section 3.1, Section 3.3"},{"comment":"The model Med-CXRGen-I achieves a much higher validation F1-RadGraph (26.65) than test-public (22.79) or test-hidden (22.10), but this drop is not discussed; a brief comment on possible distribution shift or overfitting would help the reader interpret the results.","section":"Table 5"},{"comment":"The dataset statistics in Table 2 are presented without a citation to the original source of the test-hidden split; please clarify whether the hidden split is the official RRG24 hidden split and how it relates to the MIMIC-CXR, CheXpert, PadChest, BIMCV-COVID19, and OpenI collections.","section":"Section 4.1"},{"comment":"No baseline or comparison system is included in the results table. Since the shared task provides a ViLMedic baseline, the authors should report that baseline (or at least the leaderboard range) so that the reader can assess whether the reported scores are strong or merely moderate.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"This is a workshop-style system description with a modest technical contribution. The main blocker is the suspicious identity of the public and hidden Findings F1-RadGraph scores in Table 5, which directly affects the headline result. I would ask the authors to verify the number against the official leaderboard before considering acceptance. The paper may be more appropriate for a workshop proceedings venue than for a broad journal, but that is a scope decision for the editor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Straightforward shared-task system paper: LLaVA-style two-stage training with an MLP adapter then LoRA, separate models for Findings and Impressions, and horizontal stitching of up to four images. The methodology is clearly described, the limitations section is unusually candid, and the hidden-test numbers come from the task organizers rather than the authors. That is real evidence and worth taking seriously.\n\nWhat is new is limited: the architecture and training schedule are inherited from LLaVA-1.5 and LLaVA-Med, and LoRA is standard. The task-specific choices are the two separate models and the image stitching; neither is ablated. The paper also has no baseline comparison in Table 5, no error bars or significance tests, and no analysis of what happens when a study has more than four images or when image order changes. The reader is right about those gaps. The Discussion itself concedes that performance may drop when superfluous images are included, and the Limitations section admits disease prevalence and modality imbalance.\n\nThe stress-test note flags something specific: the public and hidden Findings F1-RadGraph are both 24.13 to two decimals, while every other metric differs between the two sets. That is suspicious. It could be a copy-paste or logging error, and the paper should be asked to confirm the hidden value against the official leaderboard before the 4th-place claim is taken at face value. I would not call it a load-bearing flaw in the method: the model's overall performance is still in the same band regardless of one decimal place, and the rank claim can be verified externally.\n\nThe paper's real weakness is that it is a competent incremental system description, not a study that isolates the effect of any design choice. That is fine for a workshop shared-task report, and it should be accepted on those terms. For a more general venue it would need baselines, ablations, and variance estimates.\n\nWho this is for: people working on radiology report generation or participating in RRG24-type shared tasks. The honest writing makes it a useful data point, and the external scoring gives it some weight. I would not cite it in my own work, but I would not mind seeing it in the literature.\n\nSerious referee: yes, if the editor asks the authors to confirm the hidden-test value and adds a request for one ablation on image count/order. Without that, it is a desk-acceptable workshop paper.","headline":"Competent shared-task system paper that deserves referee time, but the identical public/hidden F1-RadGraph score needs verification before trusting the 4th-place claim.","tokens_in":13810,"tokens_out":3041,"would_cite":false,"duration_ms":29917,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two-stage fine-tuned vision-language model generates competitive radiology reports from stitched chest X-rays.","keywords":["radiology report generation","visual instruction tuning","chest X-ray","vision-language model","LoRA","CLIP","Vicuna-7B","F1-RadGraph"],"falsifier":"Take a cohort of chest X-ray studies with five or more images whose reference report describes a finding visible only in the fifth or later image; run the Med-CXRGen-F model on the first four images stitched as in the paper, and compare its F1-RadGraph with the same model given all images or the same five images reordered to put the finding-bearing image first. If the truncated-input score drops materially while the all-images or reordered score holds, the four-image truncation assumption is falsified.","tokens_in":12743,"feed_emoji":"🩻","tokens_out":8069,"duration_ms":69171,"temperature":0.7,"pith_summary":"This paper claims that a general-purpose vision-language model becomes competitive at radiology report generation when it is fine-tuned in two stages on the RRG24 shared-task dataset: an MLP adapter is first trained to align CLIP chest X-ray features with the Vicuna-7B language model, then the LLM is LoRA-tuned for report generation with the encoder and adapter frozen. The authors report hidden-test F1-RadGraph scores of 24.13 for Findings and 22.10 for Impressions, placing 4th on the leaderboard at submission. They also claim that horizontally stitching up to four X-ray images into one input lets a single image encoder process multi-image studies without dedicated multi-image encoding. A sympathetic reader would care because the recipe suggests that modest reuse of open-source components, not a new architecture, can produce useful clinical text.","feed_headline":"Fine-tuned LLM on stitched X-rays places 4th at RRG24","feed_subtitle":"Hidden-test F1-RadGraph of 24.13 and 22.10 shows a general VLM can be tuned for radiology reporting.","key_machinery":"The mechanism is a two-stage visual instruction-tuning pipeline. Stage 1 freezes both the CLIP image encoder and the Vicuna-7B LLM and trains only a GELU-activated MLP adapter with hidden size 1024, aligning chest X-ray features to the LLM's text embedding space. Stage 2 keeps the encoder and adapter frozen and applies Low-Rank Adaptation (LoRA) to the LLM for three epochs on the section-generation task. Multi-image studies are handled by taking the first up to four images and horizontally concatenating them into a single input image before the encoder, so the model consumes a fixed-size stitched input instead of attending over separate images. This machinery is what lets a single-image encoder and a text-only LLM produce the reported results.","core_discovery":"The central discovery claimed is that visual instruction tuning, in the LLaVA-1.5 style, transfers to radiology report generation when applied in two stages on the provided dataset: in Stage 1 the CLIP encoder and Vicuna LLM stay frozen while the MLP adapter learns to align chest X-ray features with the LLM, and in Stage 2 the adapter and encoder remain frozen while LoRA updates the LLM for three epochs on the report-generation task. Separate models are trained for Findings and Impressions, each decoding up to 150 tokens from a prompt requesting a description of that section. With the first up to four study images stitched horizontally into a single input, the models score 24.13 and 22.10 F1-RadGraph on the hidden Findings and Impressions test sections, which the paper reports as 4th place among RRG24 submissions at the time of writing. The authors present this as demonstrating that a domain-specific fine-tuned VLM can handle multiple images and generate clinically relevant report sections, rather than as a new architectural contribution.","pith_inferences":["My inference: because no ablation varies the number or order of stitched images, the 'first four images' truncation probably loses findings that appear only in later images; on such studies the true score would be lower than the leaderboard number.","My inference: stitching discards spatial and temporal ordering among views; reordering-invariant or attention-based multi-image fusion would be a natural next step that this paper's setup cannot capture.","My inference: the authors' own discussion concedes that performance suffers when superfluous images are included, so an image-selection or relevance-weighting mechanism is a direct, testable follow-up.","My inference: because evaluation rests on F1-RadGraph, the clinical safety of the generated text is unmeasured; an error analysis on missed findings would be needed to judge whether the reports are usable rather than just score-competitive."],"forward_implications":["If the recipe is correct, a competitive radiology report generator can be assembled from an open-source VLM and public dataset without a new multimodal architecture, lowering the entry barrier for clinical NLP teams.","The horizontal-stitching trick implies that a single-image encoder suffices for studies with a small bounded number of images, which is a practical low-cost option for resource-limited settings.","Separate models per report section indicate that Findings and Impressions benefit from section-specific fine-tuning, so the design choice of one model per section is a safe default for similar shared tasks.","The reported hidden-test scores give later RRG24 participants a concrete baseline: with the same metric, matching or exceeding F1-RadGraph 24.13 on Findings would establish improvement over this system."],"supporting_citations":[{"why":"Provides the RRG24 shared task, dataset statistics, and the training/evaluation protocol that the models are fine-tuned on and scored against.","marker":"Xu et al., 2024"},{"why":"Supplies the LLaVA-Med observation that starting from a language-only pretrained LLM helps, which motivates the architecture choice.","marker":"Li et al., 2023"},{"why":"Defines the LLaVA-1.5 design of CLIP encoder plus MLP adapter and the joint tuning protocol the paper mirrors.","marker":"Liu et al., 2023"},{"why":"Provides Low-Rank Adaptation (LoRA), the efficient fine-tuning method used in Stage 2.","marker":"Hu et al., 2021"},{"why":"Defines F1-RadGraph, the primary metric used to report leaderboard results.","marker":"Delbrouck et al., 2022a"},{"why":"Supplies Vicuna-1.5, the 7B large language model that the report generators fine-tune.","marker":"Chiang et al., 2023"},{"why":"Supplies the CLIP vision encoder that produces the image patch features fed to the adapter.","marker":"Radford et al., 2021"},{"why":"Supplies the ViLMedic baseline pipeline used for evaluation and decoding settings.","marker":"Delbrouck et al., 2022b"}],"fun_headline_variants":["Stitched X-rays + fine-tuned Vicuna = 4th at RRG24","Two-stage tuning turns general VLM into radiology reporter, 4th at RRG24","Adapter then LoRA: 4th-place radiology reports from chest X-rays","Visual instruction tuning for radiology reports: 4th place at RRG24","General VLM, stitched inputs, and LoRA score 4th at RRG24"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The model sees only the first up to four images of a study, stitched side by side, and the paper offers no evidence that this truncation and ordering preserves the clinically relevant content of the full study.","fun_headline_variants_meta":{"raw":{"variants":["Stitched X-rays + fine-tuned Vicuna = 4th at RRG24","Two-stage tuning turns general VLM into radiology reporter, 4th at RRG24","Adapter then LoRA: 4th-place radiology reports from chest X-rays","Visual instruction tuning for radiology reports: 4th place at RRG24","General VLM, stitched inputs, and LoRA score 4th at RRG24"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000478,"raw_usage":{"total_tokens":2346,"prompt_tokens":898,"completion_tokens":1448,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":1334}},"tokens_in":514,"tokens_out":1448,"duration_ms":13219,"temperature":1.0,"reasoning_tokens":1334,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:06:03.688352+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a cohort of chest X-ray studies with five or more images whose reference report describes a finding visible only in the fifth or later image; run the Med-CXRGen-F model on the first four images stitched as in the paper, and compare its F1-RadGraph with the same model given all images or the same five images reordered to put the finding-bearing image first. If the truncated-input score drops materially while the all-images or reordered score holds, the four-image truncation assumption is falsified.","supporting_citations":[],"review_version":1}