{"id":"ac5f6794-b6ec-40fb-8129-1aa37f415d70","arxiv_id":"2507.18433","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"DiagR1, a Qwen2.5-VL model post-trained with SFT and GRPO and prompted with classifier outputs, substantially improves BLEU and BERTScore for GI pathology report generation, though clinical-relevance claims lack supporting evaluation.","lead":"Researchers trained DiagR1, a vision-language model for gastrointestinal pathology, using supervised fine-tuning and reinforcement learning, plus prompts enriched with a classifier's lesion predictions. The model beats GPT-4o on text-similarity benchmarks, but the abstract's clinical-utility numbers are not defined anywhere in the experiments.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains rest on evaluation metrics that are also the RL rewards, so the clinical-relevance and diagnostic-error claims are not established.","rationale":"The reader's weakest assumption is correct and remains the most load-bearing: the evaluation uses BLEU and BERTScore computed against original reports as proxies for clinical relevance and diagnostic correctness. My pass does not change the CONDITIONAL verdict because the concern is addressable rather than fatal. I agree with the reader's identification, and the following observations strengthen it: (i) the rewards in Eqs. (5)-(6) are exactly the evaluation metrics, making the headline comparisons potentially circular; (ii) the abstract's clinical-relevance and diagnostic-error percentages are not defined or derivable from any table; (iii) LZ test data come from the same hospital as SFT and GRPO training data, so high BLEU may reflect template memorization rather than diagnostic skill. A blinded clinician-rated error study would directly settle whether DiagR1 genuinely reduces diagnostic errors. If the authors add such validation, or otherwise release data and code for independent checking, the claim could move toward acceptance; without it, CONDITIONAL is the appropriate verdict.","tokens_in":15576,"tokens_out":3335,"duration_ms":38978,"concrete_test":"Run a blinded clinician review on 100 LZ and 100 CQ test cases: two board-certified gastrointestinal pathologists independently classify each generated report's diagnosis as correct, minor error, or major management-changing error, and record hallucinated microscopic findings not supported by the image. Compare DiagR1, DiagSFT, GPT-4o, and Qwen2.5-VL under identical prompt augmentation, with report source masked. Report diagnostic error rates, completeness, and inter-rater agreement. If DiagR1's major-error rate is not significantly lower than GPT-4o's, the abstract's '41.2% fewer diagnostic errors' claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on evaluation metrics that measure lexical/semantic similarity to original reports. The diagnosis reward in Eq. (6) is BLEU and the caption reward in Eq. (5) is BERT embedding cosine similarity; Table I then evaluates the same two quantities. This is a circularity: DiagR1 is trained to maximize the exact numbers used to declare victory, so Table I can reflect reward overfitting rather than clinical quality. The problem is compounded by the internal LZ split: SFT, test, and GRPO samples are all drawn from the same hospital, and pathology reports are highly templated. Optimizing BLEU can exploit n-gram overlap with training templates; the low absolute BLEU values (e.g., 4.20 on CQ) mean small lexical overlaps drive large relative gains. The abstract's '18.7% higher clinical relevance, 32.4% improved structural completeness, and 41.2% fewer diagnostic errors' are not defined or derivable from any table; the only qualitative evidence is two selected examples in Fig. 4, which are not a systematic evaluation. Without clinician-scored outcomes, the paper does not establish clinical utility, even if the engineering pipeline is sound.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes DiagR1, a vision-language model for gastrointestinal pathology report generation. The authors construct a large dataset of pathology reports with microscopic findings and diagnoses, augment prompts with predictions from a UNIv2-based ROI classifier plus anatomical site information, and then train Qwen2.5-VL-7B through supervised fine-tuning followed by GRPO reinforcement learning. The RL reward is a weighted combination of a format reward, a caption reward based on BERT embedding cosine similarity (Eq. 5), and a diagnosis reward based on BLEU (Eq. 6). Experiments compare DiagR1 with GPT-4o, Gemini-2.5-pro, Llama-3.2-11B-Vision, DeepSeek-VL2, Qwen2.5-VL-7B, and DiagSFT on internal (LZ) and external (CQ) test sets using BERTScore and BLEU. The authors report consistent improvements and also provide ablation studies for prompt augmentation and chain-of-thought prompting. The abstract claims that DiagR1 achieves 18.7% higher clinical relevance, 32.4% improved structural completeness, and 41.2% fewer diagnostic errors compared to state-of-the-art models.","tokens_in":15805,"tokens_out":2787,"duration_ms":29024,"significance":"If the clinical utility claims were validated, the paper would address an important need in pathology report generation, and the combination of prompt augmentation with RL post-training is a sensible engineering direction. The internal/external split and the ablation studies are useful contributions, and the paper is clearly written. However, the evaluation is the central weakness: the two metrics used in Table I (BERTScore and BLEU) are exactly the caption and diagnosis rewards optimized in Eqs. (5) and (6). Consequently, the reported improvements partly reflect reward overfitting rather than clinical quality, and the abstract's headline numbers are not derivable from any reported table. The paper provides no clinician-based validation, no diagnostic-accuracy evaluation, and no statistical significance testing. Since the paper's stated contribution is clinical utility, this is a load-bearing gap. The paper also does not release code or data, which limits reproducibility, but that alone would not preclude publication if the evaluation were sound.","major_comments":[{"comment":"The evaluation protocol uses the same quantities that the model was trained to maximize. The caption reward R_cap in Eq. (5) is the cosine similarity of BERT embeddings, and the diagnosis reward R_ans in Eq. (6) is BLEU; Table I then reports BERTScore and BLEU on the test sets. Because GRPO optimizes these exact objectives on training samples, the observed gains in Table I could be a direct consequence of reward optimization rather than of improved clinical reasoning or diagnostic correctness. The claim that DiagR1 'outperforms' baselines in clinical relevance and diagnostic accuracy is therefore not established by the current evidence. An evaluation with clinician-scored outputs, diagnostic error rates verified by pathologists, or metrics distinct from the rewards (e.g., factual consistency against structured diagnostic categories) is required.","section":"Section IV.A and Eqs. (5)-(7)"},{"comment":"The abstract's headline numbers (18.7% higher clinical relevance, 32.4% improved structural completeness, 41.2% fewer diagnostic errors) do not appear anywhere in the manuscript, and their mapping to Table I is unclear. For example, on diagnosis BERTScore for LZ, DiagR1 improves over GPT-4o by 7.73 percentage points (from 85.07 to 92.80), which is a relative gain of about 9.1%, not 18.7%; BLEU increases from 2.69 to 20.93, which is far more than 18.7% in relative terms. The terms 'clinical relevance' and 'structural completeness' are not defined operationally. The authors should either remove these numbers or state precisely how they were computed, ideally with the corresponding table rows.","section":"Abstract and Section IV.C"},{"comment":"No error bars, confidence intervals, or significance tests are reported for the benchmark numbers. Given the low absolute BLEU values (e.g., 4.20 on the CQ diagnosis task), small differences in raw BLEU can translate into large relative differences, and the reported improvements over GPT-4o (e.g., 1.81 percentage points in BLEU on CQ) may not be statistically significant. The claim of 'significantly outperforms' in Section IV.C requires either repeated runs with variance estimates or an appropriate significance test across the test set.","section":"Table I"},{"comment":"The internal LZ split is at the level of reports, not patients, and all LZ training, test, and GRPO samples come from the same hospital. If a single patient contributed multiple biopsy reports across the SFT, GRPO, and test sets, the test results would be inflated by patient-level leakage. The authors should state whether the split was performed at the patient level and, if not, re-run the evaluation with a patient-exclusive split. This is relevant because pathology reports are highly templated and reports from the same patient are likely near-duplicates.","section":"Section IV.A (dataset split)"}],"minor_comments":[{"comment":"The text consistently uses 'prompt argumentation' where 'prompt augmentation' appears to be intended. Please correct this in the title of Section III.B and elsewhere to avoid confusion with formal argumentation theory.","section":"Throughout"},{"comment":"In the figure caption, 'DigeR1 Generation' should be 'DiagR1 Generation'.","section":"Fig. 4"},{"comment":"Reference [5] contains an incomplete citation: 'Official journal of the American College of Gastroenterology— ACG , pp. 10–14309, 2022' appears to be truncated. Please provide the full article title and volume/page details.","section":"References"},{"comment":"Section IV.C states that DiagR1 outperforms GPT-4o by 11.83 percentage points in BLEU on the microscopic findings task of LZ, but Table I shows a difference of 12.95 - 1.12 = 11.83, which is consistent; however, the sentence says '5.90 percentage points' for BERTScore while Table I gives 88.55 - 82.66 = 5.89, a minor rounding inconsistency. Please align these values.","section":"Table I and Section IV.C"},{"comment":"The relative improvement of GPT-4o BERTScore from 79.23 to 85.07 is described as approximately 7.40%, but the actual relative gain is (85.07 - 79.23)/79.23 = 7.37%; the source of the 7.40% figure is unclear. Please check the arithmetic.","section":"Section IV.E.1"}],"recommendation":"major_revision","confidential_remarks":"The paper would become eligible for acceptance if the authors (i) clarify or remove the indefensible abstract percentage claims, (ii) add a clinician-based or otherwise reward-independent evaluation of diagnostic correctness, and (iii) provide error bars or significance tests. The current evaluation is essentially a measure of reward overfitting, so the clinical claims are not yet supported. The scope of the journal (JBHI) is appropriate for the topic, but the gap between the reported numbers and the abstract's claims should be resolved before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"DiagR1 is a plausible engineering paper with a real dataset, but its headline clinical claims are not supported by the evidence it reports. Send it to peer review, but expect the authors to do substantial work on evaluation.\n\nThe genuinely new piece is the dataset: 18,627 GI pathology reports plus a curated ROI classification set, and the specific pipeline that feeds UNIv2 classifier probabilities into the prompt and then applies SFT+GRPO. That combination is not in the cited literature, and the ablations (location vs. classifier prompts, CoT vs. no CoT) are well structured. The authors also include an external test set from a second hospital, which is better than many medical ML papers.\n\nThe soft spots are real and load-bearing for the clinical claims. The diagnosis reward is BLEU (Eq. 6), the caption reward is BERT cosine similarity (Eq. 5), and Table I evaluates exactly those two quantities. So the reported gains tell us the model optimized the same numbers it is judged on. That is circularity, not necessarily fraud—it is common in text generation—but it means Table I cannot support the words \"clinical relevance\" or \"diagnostic errors.\" The abstract's 18.7%/32.4%/41.2% are not defined anywhere and do not come from any table. The only qualitative evidence is two examples in Fig. 4, which are cherry-picked by nature. No error bars, no significance tests, and no code or data release. The low absolute BLEU values on the external set (4.20) make the huge relative gains less reassuring.\n\nThese are addressable. The authors could define clinical metrics (e.g., clinician-rated accuracy of diagnosis, completeness of required elements), report error bars, do a small clinician study, and release at least feature-level data or a demo. The conclusion section already acknowledges offline evaluation and the need for pilot studies, which is honest but undercuts the abstract.\n\nWho is this for? Researchers working on pathology report generation and RL post-training for medical VLMs. They will get value from the dataset description and the ablation results. The paper deserves a serious referee; it is not a desk reject. But my recommendations to the editor would be: major revision, with the abstract rewritten to match evidence and clinical validation before publication.","headline":"A real dataset and a clear pipeline, but the clinical-relevance claims outrun the evidence; send to peer review with major revision.","tokens_in":16363,"tokens_out":2033,"would_cite":true,"duration_ms":21406,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A reinforcement-learning-trained vision-language model can write gastrointestinal pathology reports that outperform GPT-4o, by grounding generation in a pathology classifier's predictions and an explicit reasoning chain.","keywords":["gastrointestinal pathology","vision-language model","pathology report generation","reinforcement learning","GRPO","prompt augmentation","chain-of-thought","hallucination reduction"],"falsifier":"Have board-certified pathologists, blinded to model identity, score 200 DiagR1 and GPT-4o reports for clinically significant diagnostic errors and unsupported histologic claims. If DiagR1 does not show a lower error rate and higher acceptance, the central claim of clinical utility fails even though its BLEU and BERTScore are higher.","tokens_in":15383,"feed_emoji":"🔬","tokens_out":6773,"duration_ms":66137,"temperature":0.7,"pith_summary":"DiagR1 is a vision-language model for gastrointestinal pathology that the authors train to generate both the microscopic-findings section and the pathological diagnosis of a report. The paper's central claim is that grounding the model with predictions from a pathology-specific image classifier plus biopsy-site information, forcing an explicit reasoning trace, and then optimizing structure, caption semantics, and diagnosis wording through reinforcement learning reduces hallucination and produces reports that beat strong open and proprietary baselines. On their internal and external test sets, the authors report BERTScore and BLEU gains over GPT-4o, and the abstract summarizes these as 18.7% higher clinical relevance, 32.4% improved structural completeness, and 41.2% fewer diagnostic errors. A sympathetic reader should care because the pipeline is a concrete route from noisy, partially annotated hospital reports to a reviewable automated draft.","feed_headline":"RL-trained model beats GPT-4o at writing GI pathology reports","feed_subtitle":"Grounding prompts with a pathology classifier and explicit reasoning cuts hallucinations and improves report structure.","key_machinery":"The load-bearing mechanism is prompt argumentation plus a three-segment output scaffold optimized by GRPO. A frozen UNIv2 image encoder and two small linear heads supply per-image lesion-class probabilities; that classification output, together with the biopsy site, is inserted into the prompt so the language model knows what region and lesion type it is looking at. The output scaffold <think>...<caption>...<answer>... forces an auditable reasoning chain, and the GRPO reward is a composite of a strict 0/1 format check, a BERT-embedding cosine similarity for the caption, and BLEU for the diagnosis. The mechanism converts a general image-to-text model into a structured report writer by making the classifier's prior and the report template part of the optimization signal.","core_discovery":"The discovery the paper argues for is that a relatively small open-weight multimodal model, Qwen2.5-VL-7B, can be turned into a digestive-pathology report generator by a post-training recipe: supervised fine-tuning on 11,904 expert reports with microscopic findings, then GRPO reinforcement learning on 12,627 additional reports, with prompts enriched by an ROI classifier built on a frozen UNIv2 encoder and by the anatomical site. The model must emit its reasoning in <think>...</think>, the microscopic findings in <caption>...</caption>, and the diagnosis in <answer>...</answer>, and the reward is a weighted sum of a format check, BERT-cosine similarity of the caption, and BLEU of the diagnosis. On the Liuzhou test set, DiagR1 reaches 88.55 BERTScore and 12.95 BLEU for microscopic findings and 92.80 BERTScore and 20.93 BLEU for diagnoses, against 82.66 and 1.12 and 85.07 and 2.69 for GPT-4o on the same two tasks; on the external Chongqing set, the diagnosis gains are smaller but still positive. The paper takes these results to show that prompt argumentation plus RL post-training yields reports that are more structured, more semantically aligned with reference reports, and less prone to unsupported histologic claims than general-purpose vision-language models.","pith_inferences":["Because the rewards are n-gram and embedding similarity, the RL stage may be rewarding style and lexical mimicry of the original reports rather than diagnostic truth; a pathologist-adjudicated study of clinically significant errors is the real test.","If the ROI classifier's prior is wrong, the prompt could anchor the model toward an incorrect lesion class, so the method should be stress-tested on cases where the classifier is confident but wrong.","The headline 'clinical relevance' and 'fewer diagnostic errors' numbers are derived from text-similarity metrics; reading them as clinician-verified would be an extrapolation beyond the paper.","The same three-part scaffold and reward mix could transfer to endoscopy or radiology structured reporting, where templates and site labels are equally standardized."],"forward_implications":["If the reported gains hold, the same supervised-fine-tuning-plus-GRPO recipe can be applied to other structured medical reports whenever a decent organ-level classifier and site metadata are available.","Prompt argumentation can be a cheaper alternative to larger models: a high-accuracy classifier constraint appears to reduce VLM hallucination without retraining the visual encoder.","The <think> chain produces outputs a pathologist can audit, which is a prerequisite for real clinical workflow use rather than just a text-generation benchmark.","The larger gains on the in-distribution Liuzhou set and smaller gains on the external Chongqing set suggest the method is sensitive to hospital-specific report style and data completeness."],"supporting_citations":[{"why":"Supplies Qwen2.5-VL-7B-Instruct, the base vision-language model that is warm-started with SFT and then trained with GRPO.","marker":"[9]"},{"why":"Supplies the DeepSeek-R1 reasoning-through-reinforcement-learning recipe and the SFT warm-start motivation for the post-training pipeline.","marker":"[10]"},{"why":"Supplies the frozen UNIv2 pathology image encoder whose features feed the dual-branch ROI classifier used in prompt argumentation.","marker":"[22]"},{"why":"Supplies HistoGPT's BERTScore and BLEU evaluation protocol for pathology report generation, which the paper follows.","marker":"[12]"},{"why":"Supplies the idea of verifiable visual reward functions for reinforcement learning in VLM fine-tuning, one source of the reward design.","marker":"[41]"},{"why":"Supplies a prior application of GRPO to medical vision-language models across multiple imaging modalities, backing the choice of GRPO.","marker":"[44]"}],"fun_headline_variants":["RL-tuned DiagR1 beats GPT-4o at GI report writing","RL post-training slashes GI diagnosis errors by 41%","Small open VL model tops GPT-4o on GI pathology reports","DiagR1's RL reasoning cuts GI report errors by 41%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation hinges on BLEU and BERTScore computed against the original pathology reports being faithful proxies for clinical relevance and diagnostic correctness; if those text-similarity scores do not reflect clinical quality, the claimed superiority over GPT-4o does not establish clinical utility.","fun_headline_variants_meta":{"raw":{"variants":["RL-tuned DiagR1 beats GPT-4o at GI report writing","RL post-training slashes GI diagnosis errors by 41%","Small open VL model tops GPT-4o on GI pathology reports","DiagR1's RL reasoning cuts GI report errors by 41%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000839,"raw_usage":{"total_tokens":3714,"prompt_tokens":1058,"completion_tokens":2656,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":674,"completion_tokens_details":{"reasoning_tokens":2577}},"tokens_in":674,"tokens_out":2656,"duration_ms":19106,"temperature":1.0,"reasoning_tokens":2577,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:12:10.254594+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have board-certified pathologists, blinded to model identity, score 200 DiagR1 and GPT-4o reports for clinically significant diagnostic errors and unsupported histologic claims. If DiagR1 does not show a lower error rate and higher acceptance, the central claim of clinical utility fails even though its BLEU and BERTScore are higher.","supporting_citations":[{"cited_title":"Generating dermatopathology reports from gigapixel whole slide im- ages with histogpt,","cited_arxiv_id":null,"evidence_quote":"Supplies HistoGPT's BERTScore and BLEU evaluation protocol for pathology report generation, which the paper follows."}],"review_version":2}