{"id":"4cc087db-a004-47ce-a66a-5ed5d828e172","arxiv_id":"2501.02598","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Length-based curriculum learning improves an end-to-end GIT transformer for chest X-ray report generation, yielding high METEOR and clinical F1 scores; the state-of-the-art claim is however weakened by inconsistent evaluation setups across baselines.","lead":"An end-to-end transformer based on GIT is adapted to generate chest X-ray reports, with additions including multi-view images, patient context, and a report-length curriculum. The authors report top scores on METEOR and several clinical F1 metrics on MIMIC-CXR-JPG, but the comparisons to prior systems rely on different evaluation protocols.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA claim is built on cross-paper metrics with known protocol mismatches (target sections, labeler, splits, metric library); the headline holds only if a shared-pipeline re-run preserves the margins.","rationale":"I read the paper in good faith: it is a clean end-to-end GIT adaptation with a length-based curriculum, and the ablation study is internally consistent. The single most load-bearing step is the external 'state-of-the-art' comparison, which the reader correctly identifies. My concern is not a disagreement with consensus; it is that the claim's validity depends on cross-paper metric comparability, and the paper itself discloses at least four protocol mismatches (target section, labeler, splits, metric library) that are known to change scores by more than the margin of error in this literature. The most concrete test—re-running baselines under the paper's pipeline, or re-running the paper's model under baseline pipelines—directly settles the question, so a CONDITIONAL verdict is appropriate. I do not manufacture a second attack: the curriculum-learning contribution, the ablation ordering, and the report-length diagnosis are supported by Table 2 and Figure 3, though the 'clinically accurate and reliable for diagnosis' sentence in Section 5.2 goes beyond the per-pathology scores in Table 3 and the Limitations section. With code and the evaluation script released, the protocol comparisons could be made exact and the concern retired. Upholding CONDITIONAL is the honest call.","tokens_in":16359,"tokens_out":3186,"duration_ms":27434,"concrete_test":"Re-run at least the strongest baselines for which code/checkpoints exist (e.g., EKAG-EN 2024, RGRG 2023, ARR TR 2022B, or at minimum R2GEN/ANY with a reproduced protocol) through this paper's evaluation pipeline: official splits, 'findings'+'impression' targets, CheXbert labels, Huggingface BLEU/ROUGE-L/METEOR. If the re-computed METEOR and F1 values for those baselines move by more than ~2-3 points relative to Table 1, or if any baseline overtakes GIT-CXR (MV+C+CL), then the SOTA claims collapse and the abstract/§5 need protocol-matched comparisons. If baseline code is unavailable, an equivalent check is to recompute this paper's own scores using pycocoevalcap and CheXpert labels (the protocols of the main baselines) and show that the ranking vs. Table 1 is unchanged.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that GIT-CXR (MV+C+CL) sets a new state-of-the-art on METEOR and on F1-macro/F1-micro, reported in Table 1 (0.369 METEOR, 0.348 F1MA, 0.534 F1MI) and asserted in the Abstract and Section 1/5. This claim is only meaningful if the quoted baseline numbers are comparable to the authors' evaluation pipeline. The paper itself documents that they are not: (1) Section 4.1 and Appendix B state that this paper uses 'findings' + 'impression' as the target, while Nguyen et al. (2021), Lovelace and Mortazavi (2020), and Miura et al. (2020) use only 'findings'; longer targets change BLEU/ROUGE/METEOR distributions and also change what clinical labels are derived from. (2) Table 1 marks AGA (2021), LOVE (2020), MMTN (2023), CXR-R EPAIR (2021), and M2TR (2020) as not using the official splits; those numbers are therefore not measured on the same test set. (3) Appendix C says NLG metrics were computed with the Huggingface library; prior works typically use pycocoevalcap/coco-caption, and METEOR and BLEU have known implementation- and tokenizer-dependent differences. (4) The clinical F1 uses CheXbert labels here, while Chen et al. (2020), Lovelace and Mortazavi (2020), and Nguyen et al. (2021) use CheXpert labels; labeler choice alone can shift per-pathology F1 by several points. The reported margins are large (METEOR +14.7 pp over AGA, F1MI +12.3 pp over LOVE), so the SOTA claim would survive only if these protocol shifts do not change the ordering. No evidence for that is provided. The internal ablations (Table 2) and curriculum-learning analysis (Figure 3) are plausible and self-consistent, but they do not establish the external SOTA comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents GIT-CXR, an end-to-end transformer architecture for chest X-ray report generation, built by adapting the GIT model with a multi-label classification head, multi-view image inputs, patient context, and a length-based curriculum learning schedule. Experiments on the MIMIC-CXR-JPG dataset report that the best model (GIT-CXR MV+C+CL) achieves state-of-the-art results on METEOR and on F1-macro/F1-micro clinical accuracy metrics, while remaining comparable to prior work on BLEU and ROUGE-L. The ablation study shows that each added component improves the model on the same evaluation pipeline, and the paper argues that curriculum learning is particularly beneficial for generating long reports.","tokens_in":16726,"tokens_out":5243,"duration_ms":46834,"significance":"If the internal ablation is taken at face value, the paper makes a useful empirical contribution: a simple end-to-end transformer with a length-based curriculum consistently improves over its own baseline, and the aggregate curriculum gains are reported as statistically significant. The authors also provide a thorough ablation, per-pathology analysis, and use the official MIMIC-CXR splits, which is more rigorous than many prior works. However, the headline claims of new state-of-the-art on METEOR and F1-macro/F1-micro rest on cross-paper comparisons with documented protocol mismatches (target sections, labeler, splits, metric library). The significance of the contribution therefore hinges on whether those claims survive a controlled re-evaluation; the curriculum-learning idea is novel in this specific setting and could be of interest, but its claimed advantage for long reports needs stronger statistical support.","major_comments":[{"comment":"The headline SOTA claim is not supported by a controlled comparison. The paper uses 'findings' plus 'impression' as the target report, while the baselines of Nguyen et al. (2021), Lovelace and Mortazavi (2020), and Miura et al. (2020) use only 'findings'; Table 1 also marks AGA, LOVE, MMTN, CXR-R EPAIR, and M2TR as not using the official splits. These differences change the length and content distribution of targets, which directly affects BLEU/ROUGE/METEOR scores and the derived clinical labels. To substantiate the claim of new state-of-the-art, the authors should re-evaluate at least the most competitive baselines under the same target-section, labeler, split, and metric pipeline, or restrict the SOTA claim to methods evaluated under identical conditions.","section":"Section 4.1, Table 1, Appendix B"},{"comment":"The clinical and NLG metrics are not directly comparable across papers. The authors compute NLG metrics with the Huggingface library, whereas prior work predominantly uses pycocoevalcap; METEOR and BLEU are known to differ by implementation and tokenization. For clinical accuracy, the authors use CheXbert labels, while Chen et al. (2020), Lovelace and Mortazavi (2020), and Nguyen et al. (2021) use CheXpert. Labeler choice alone can shift per-pathology F1 by several points. The reported margins over baselines are large, but without a shared-pipeline comparison there is no evidence that the ordering holds; this undermines the F1-macro/F1-micro SOTA claims.","section":"Section 4.2, Appendix C"},{"comment":"The claim that curriculum learning is the main driver of improved performance on long reports is supported only by visual inspection of curves without error bars or per-bucket significance tests. The paper states that all curriculum improvements are statistically significant (p<0.01) for aggregate scores, but not for the length-stratified analysis in Figure 3. To support the load-bearing claim about long-report generation, the authors should report confidence intervals or significance tests on the per-length-bucket metrics, and clarify how the length bins are defined and how many samples fall in each bin.","section":"Section 5.1, Figure 3"}],"minor_comments":[{"comment":"There are several typos and grammatical errors: 'it's efficacy' should be 'its efficacy', 'weather' in Section 4.2 should be 'whether', 'descent accuracy' in Section 5.2 should be 'decent accuracy', and 'We address this on the fact' in Section 5.1 should be 'We attribute this to the fact'.","section":"Throughout"},{"comment":"The summation notation 'i≤DX' in Equation (1) is malformed; it should be written as a sum over i=1 to D.","section":"Section 3.3, Equation (1)"},{"comment":"The paper reports averages of three runs but does not provide standard deviations or confidence intervals; the claim of statistical significance (p<0.01) would be better supported by reporting variance alongside the mean values.","section":"Tables 1 and 2"},{"comment":"The curves in Figure 3 would be more informative with shaded confidence intervals or error bars, especially since the text states that short sequences are rare and the metrics are less reliable there.","section":"Figure 3"},{"comment":"The column 'SUPPORT' in Table 3 is not defined in the caption; it appears to be the number of positive samples per pathology, but this should be stated explicitly.","section":"Table 3"},{"comment":"The paper states 'We will make our code publicly available' in two places; consider including a URL or repository reference in the final version.","section":"Appendix A.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid engineering contribution with a coherent internal ablation and a transparent description of its evaluation setup. However, the SOTA claims are substantially overstated given the documented protocol mismatches with prior work. I would advise the editor to require either a shared-pipeline re-evaluation of competitive baselines or a careful rewording that restricts the claims to controlled comparisons. Additionally, the novelty claim of being 'first to introduce curriculum learning for end-to-end transformers in medical imaging' should be checked against the existing literature more thoroughly; Liu et al. (2022) already applied curriculum learning to medical report generation, albeit with a different architecture. The paper's use of the official MIMIC-CXR splits and its willingness to discuss limitations are strengths that should be preserved in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear Colleague,\n\nThe main thing you should know: this is a solid empirical paper that does one genuinely new thing—length-based curriculum learning for an end-to-end transformer in medical report generation—and shows it works. The curriculum helps long reports and, interestingly, hurts when combined with the auxiliary classification head. That interaction is the most valuable finding. The ablation is clean: every component improves under the same pipeline, and the curriculum gains are statistically significant across three runs. They also give per-pathology F1 and honest discussion of rare pathologies.\n\nThe soft spot is the SOTA claim, and the stress-test note is right. The headline numbers on METEOR and F1-macro/micro are taken from papers that use different target sections, different labelers, different metric libraries, and sometimes different splits. The paper itself documents most of these mismatches. The margins are large enough that the claim could survive a shared-pipeline re-run, but nothing in the paper shows that. So the abstract overstates what is established.\n\nA second issue: they conclude the model 'can reliably be used for diagnosis,' while their own Table 3 shows F1 below 0.15 for fracture, lung lesion, and pleural other. That is an overclaim, though the limitations section is more measured.\n\nThe paper promises code but doesn't ship it. That matters here because the central comparison cannot be audited without re-running baselines under their evaluation.\n\nOverall: the internal science is sound, the curriculum finding is worth publishing, and a serious referee should engage. A revision that tempers the SOTA language, re-evaluates the baselines under one protocol, and releases code would make the case.","headline":"Solid internal ablation of a simple GIT-based report generator, but the SOTA claims rest on cross-paper metric comparisons that the paper's own documentation shows are not apples-to-apples.","tokens_in":17334,"tokens_out":2526,"would_cite":true,"duration_ms":25066,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An end-to-end transformer with a report-length curriculum sets new state-of-the-art scores for chest X-ray report generation on METEOR and clinical F1 metrics.","keywords":["radiology report generation","chest X-ray","end-to-end transformer","curriculum learning","MIMIC-CXR-JPG","clinical accuracy","METEOR","image captioning"],"falsifier":"Take the strongest prior systems listed in Table 1 and run them through the preprocessing and evaluation code used here (concatenated impression plus findings, CheXbert labels, official splits), then re-measure METEOR, F1-macro, and F1-micro. If the gaps shrink to the noise level of the three-run averages, the state-of-the-art claim collapses; if they persist, it stands. A cheaper partial check is to ablate only the curriculum from GIT-CXR (MV+C+CL) and compare the length-binned metric curves.","tokens_in":16125,"feed_emoji":"🩻","tokens_out":8109,"duration_ms":71055,"temperature":0.7,"pith_summary":"This paper argues that a deliberately simple end-to-end transformer, built by fine-tuning the GIT image-to-text model, can generate chest X-ray reports that match or beat substantially more complex systems on the standard MIMIC-CXR-JPG benchmark. Its central innovation is a curriculum-learning schedule that sorts training reports by length and gradually shifts the model from short to long targets; the authors identify this schedule as the main driver of improved performance on long reports. If the claim is right, it matters because the best previous results came from systems with extra machinery such as object detectors, memory modules, and multi-stage classification, while GIT-CXR shows a plain transformer plus report-length scheduling can reach comparable or better scores. On top of this, the paper reports new top results on the NLG metric METEOR and on the clinical accuracy F1 metrics (macro, micro, and examples-averaged), with BLEU and ROUGE-L on par with earlier methods. The ablation also documents an incompatibility: adding an auxiliary classification head to the curriculum-trained model hurts performance, because length-based sampling distorts the pathology distribution the head sees.","feed_headline":"Length-based curriculum brings top chest X-ray report scores","feed_subtitle":"A plain transformer with short-to-long report training matches or beats more complex systems on MIMIC-CXR-JPG.","key_machinery":"The load-bearing mechanism is the length-based curriculum. Reports are split into $b=10$ equal-size bins by token length; during each epoch the model samples without replacement a fraction $f=0.25$ of the dataset with per-bin weights $1/(1+|i-i_e|)$, so the effective training distribution peaks at bin $i_e$ and advances from short to long reports across training. This prevents the decoder from collapsing toward short generic output and is what the paper credits for the flattened long-report performance curves. Supporting pieces are a temporal embedding that distinguishes the two views, concatenation of the 'indication' and 'history' fields as context, and an auxiliary multi-label classification head with weighted cross-entropy over the 14 CheXbert labels, which helps only when the curriculum is not active.","core_discovery":"The paper's central claim is that the configuration GIT-CXR (MV+C+CL) — a GIT transformer fine-tuned end-to-end on MIMIC-CXR-JPG using two-view images, concatenated indication/history context, and a length-based curriculum — sets a new state of the art on METEOR (0.369) and on the clinical accuracy metrics F1-macro (0.348), F1-micro (0.534), and F1 examples-averaged (0.458) on the full official test set, while matching the previous best on BLEU and ROUGE-L. The paper further claims that the curriculum is the main source of the gain: without it, generated reports run short, and METEOR, ROUGE-L, and F1-micro decline steeply as target reports exceed roughly 75 tokens, whereas with it the long-report curve flattens. The authors take this as evidence that simpler end-to-end transformer architectures can outperform more complex prior systems when paired with a training-order strategy that addresses the long-report problem directly.","pith_inferences":["One testable extension the paper does not run is a curriculum defined by clinical content, such as the number of pathologies mentioned, instead of raw report length; such a curriculum might avoid the pathology-distribution skew that makes the classification head incompatible with length-based training.","Because the paper truncates targets at 192 tokens and shows a performance decline with length, a natural follow-up is to raise the token cap and refine the bin schedule; the paper establishes the trend but does not test the boundary.","The clinical F1 numbers are computed by an automated labeler, so the paper does not establish that the improved F1 translates into expert-reader agreement; a reader study on the same test reports would be a meaningful next check.","The reported gaps over prior work rest on cross-paper metric comparability, which the paper itself notes is imperfect because prior work differs in report sections and labelers; re-running the leading baselines under this paper's pipeline would settle how much of the gap is real."],"forward_implications":["Future radiology report generators can drop extra modules such as object detectors or memory banks and instead pair an end-to-end transformer with a report-length curriculum, lowering training and inference complexity.","The paper's long-report analysis implies that any method evaluated on MIMIC-CXR-JPG should report performance broken down by target length, since aggregate scores hide the systematic shortfall on the longest reports.","Because the curriculum and the classification head fight each other, a direct next step suggested by the paper is to find a mixing scheme that lets a model benefit from both techniques simultaneously.","The reported figures — METEOR 0.369, F1-macro 0.348, F1-micro 0.534, and F1 examples-averaged 0.458 — become the numbers a new system must beat under the official splits with CheXbert labeling."],"supporting_citations":[{"why":"Supplies the GIT generative image-to-text transformer that all proposed variants fine-tune.","marker":"(Wang et al., 2022a)"},{"why":"Supplies the MIMIC-CXR-JPG dataset, official train/validation/test splits, and CheXpert-based labels used in every experiment.","marker":"(Johnson et al., 2019b)"},{"why":"Supplies the CheXbert labeler used to produce the 14-pathology labels for the auxiliary classifier and for clinical accuracy F1 evaluation.","marker":"(Smit et al., 2020)"},{"why":"Defines the METEOR metric on which the paper claims a new state of the art.","marker":"(Banerjee and Lavie, 2005)"},{"why":"Supplies the F1-macro and F1-micro evaluation protocol and the LOVE baseline values the paper compares against.","marker":"(Lovelace and Mortazavi, 2020)"},{"why":"Introduces curriculum learning, the training strategy generalized here to report length.","marker":"(Bengio et al., 2009)"},{"why":"Supplies the auxiliary multi-label classification idea and the AGA baseline compared in Table 1.","marker":"(Nguyen et al., 2021)"},{"why":"Supplies ARR TR, the only other pure end-to-end transformer baseline, used to claim state of the art among transformer-only approaches.","marker":"(Wang et al., 2022b)"},{"why":"Supplies the F1-micro on the five most frequent labels baseline and an earlier CheXbert-based evaluation.","marker":"(Miura et al., 2020)"},{"why":"Supplies the M2TR baseline whose F1-micro on five labels the paper trails by 0.2 points.","marker":"(Chen et al., 2020)"}],"fun_headline_variants":["Length-based curriculum sets new clinical accuracy SOTA for chest X-rays","Curriculum learning lifts chest X-ray reports to SOTA clinical scores","Simple transformer with curriculum tops chest X-ray report accuracy","Length-based training yields SOTA chest X-ray report metrics","End-to-end transformer + length curriculum: SOTA for chest X-rays"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The state-of-the-art claims assume that the scores quoted from earlier papers were measured under the same protocol — the same target report sections, the same CheXbert labeler, the same official test split, and compatible metric implementations — so the gaps in Table 1 are true performance differences rather than evaluation-pipeline artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Length-based curriculum sets new clinical accuracy SOTA for chest X-rays","Curriculum learning lifts chest X-ray reports to SOTA clinical scores","Simple transformer with curriculum tops chest X-ray report accuracy","Length-based training yields SOTA chest X-ray report metrics","End-to-end transformer + length curriculum: SOTA for chest X-rays"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001229,"raw_usage":{"total_tokens":5057,"prompt_tokens":959,"completion_tokens":4098,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":4012}},"tokens_in":575,"tokens_out":4098,"duration_ms":32426,"temperature":1.0,"reasoning_tokens":4012,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:09:11.805845+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the strongest prior systems listed in Table 1 and run them through the preprocessing and evaluation code used here (concatenated impression plus findings, CheXbert labels, official splits), then re-measure METEOR, F1-macro, and F1-micro. If the gaps shrink to the noise level of the three-run averages, the state-of-the-art claim collapses; if they persist, it stands. A cheaper partial check is to ablate only the curriculum from GIT-CXR (MV+C+CL) and compare the length-binned metric curves.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the F1-macro and F1-micro evaluation protocol and the LOVE baseline values the paper compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the M2TR baseline whose F1-micro on five labels the paper trails by 0.2 points."}],"review_version":1}