{"id":"ec12e047-d07f-4a09-ab40-659785652f5e","arxiv_id":"2506.13363","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Using 100 medical images and an RLVR reward that balances precision and recall, the authors push Qwen2.5-VL-7B to a 77.8 field-level F1 on their private 203-image medical test set, above the compared baselines.","lead":"The paper fine-tunes a 7-billion-parameter vision-language model on only 100 medical report images using reinforcement learning with a precision-recall reward, and reports higher JSON extraction F1 than OCR pipelines and larger general models. For anyone building domain-specific document extraction with scarce annotations, this is a potentially cheap training recipe, though the private benchmark and undisclosed settings limit immediate confidence.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The undisclosed balancing weight α in Eq. (5) is load-bearing: the reported F1/precision/recall gains may reflect test-set tuning of the one knob that directly controls the metric being reported.","rationale":"The reader's weakest assumption is that the balancing weight α in Eq. (5) is load-bearing and undisclosed, and that test-set tuning could explain part of the reported gains. My reading agrees. The reward function is not a proxy for the evaluation metric; it is the same field-level precision and recall formula (Eq. 5 and Section 4.1), so α is a direct dial on the headline numbers. The paper's Figure 2 only illustrates the two extreme settings, not the value used, and no ablation over α is reported in Table 1 or elsewhere. This makes the central SOTA claim fragile in a way that a single disclosed number and a stability check could settle. I do not see a stronger objection: the method itself is internally consistent, the RLVR pipeline is plausible, and the direction of improvement is consistent with prior RLVR work. The private test set and lack of error bars are related weaknesses, but the α-selection issue is the most load-bearing because it directly undermines the quantitative comparison. The verdict remains CONDITIONAL pending disclosure and robustness evidence, so no change to the reader's verdict is needed.","tokens_in":15237,"tokens_out":5444,"duration_ms":62737,"concrete_test":"Run a controlled replication of RL-100 under α ∈ {0.3, 0.5, 0.7} with fixed seed, using the same 100 training images and a fresh held-out split of the 203-image set that is never inspected during development. Report α, F1, precision, recall, and TED for each run. If the best α is not chosen on the held-out split, or if the F1 spread across α is less than about 1.5 points, the disclosed-α concern is retired; if the spread is larger or α was selected on the final test set, the SOTA claim should be downgraded to conditional on the unreported selection protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that RLVR with 100 samples achieves SOTA F1, precision, and recall on medical VIE. The reward in Eq. (5) is a weighted sum of exactly the field-level precision and recall used for evaluation. The paper never reports the α value used for RL-100, RL-100(w/o sample), or OCR-SFT-17K-RL-100; Figure 2 only contrasts the α=0 and α=1 extremes. Because α controls the precision/recall trade-off in the identical metric family reported in Table 1, choosing α by inspecting the private 203-image test set (or a proxy for it) would amount to selecting the evaluation statistic itself. The baselines in Table 1 did not receive this tuning, so the 'significantly improving' comparison would not be apples-to-apples. This is not merely a reproducibility nit: without the α value and the selection protocol, the 77.81 F1 and the 10-point margins could be artifacts of picking the reward weight that maximizes the headline on the test set. The paper also reports no variance or multiple seeds, so a single favorable α run cannot be distinguished from a selected maximum. The concern would be resolved if the authors disclose α and show that performance is stable across a plausible range on a held-out split that was not used for any hyperparameter choice.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an RLVR-based method for medical Visual Information Extraction (VIE) using only 100 annotated images. The method fine-tunes Qwen2.5-VL-7B with GRPO and a rule-based reward that combines a format score and a weighted precision-recall matching score, along with a key-sampling strategy that varies queries during training. On a private medical VIE test set of 203 images, the RL-100 model is reported to achieve 77.81 F1, 79.85 precision, and 75.88 recall, exceeding pipeline, expert, commercial, and general MLLM baselines. The paper also evaluates the trained models on four general VIE benchmarks (CORD, FUNSD, SROIE, Ticket) and reports an ablation on the effect of the model's thinking process.","tokens_in":15515,"tokens_out":4051,"duration_ms":38547,"significance":"If the results hold, the paper's contribution is practical and relevant: a purely outcome-driven RL approach that improves domain-specific VIE with very few annotations, avoiding RLHF, and with a reward that directly controls hallucination versus coverage. The authors provide transparent reasoning for the reward design, compare RLVR with SFT, and show generalization behavior on dissimilar tasks. However, the central SOTA claim rests on a private test set with no error bars, an undisclosed value of the critical reward weight alpha, and an incomplete baseline description. The paper does not release code, data, or model weights, which limits reproducibility. These issues are fixable and do not invalidate the method's promise, but they currently prevent the reader from verifying that the reported gains are not artifacts of test-set hyperparameter selection.","major_comments":[{"comment":"The value of α in Eq. (5) is never disclosed for RL-100, RL-100(w/o sample), or OCR-SFT-17K-RL-100. Because Eq. (5) is literally a weighted combination of the field-level precision and recall that Table 1 reports as the headline metrics, choosing α by inspecting the private test set would be equivalent to selecting the evaluation statistic. Figure 2 only shows α=0 and α=1 extremes and does not indicate the value used in the final models. Please report the exact α used for each model, the protocol for selecting it (e.g., a held-out validation split), and ideally a sensitivity analysis of F1/precision/recall across a range of α values.","section":"§3.3 and §4.4, Eq. (5)"},{"comment":"Table 1 lists a commercial baseline 'TextIn' with an OCR output, but §4.2, which introduces all baselines (MinerU, Marker, GOT-OCR, GPT4o, Qwen2.5-VL, InternVL), does not describe TextIn or its configuration. Moreover, the entire medical VIE evaluation is on a private 203-image test set with no public medical VIE benchmark; the 'state-of-the-art' claim is therefore only relative to this private test set and the authors' own selection of fields and ground truth. Please add a proper description of TextIn and its settings, and, if no public medical VIE benchmark is used, state this limitation explicitly and justify why the private set is sufficient for the SOTA claim.","section":"§4.2 and Table 1"},{"comment":"All results in Tables 1 and 2 are single runs with no error bars, confidence intervals, or multiple seeds. Because RL training and rollout sampling are stochastic, the reader cannot distinguish a robust improvement from a single favorable run, especially given the undisclosed α and the private test set. Please report results over at least a few seeds (or provide the seed policy and the number of runs), and, if multiple seeds are too expensive, state this cost explicitly and report the variance over the evaluation set in some other way (e.g., bootstrap confidence intervals on the 203 test images).","section":"§4.4, Tables 1 and 2"}],"minor_comments":[{"comment":"In the description of JSON-SFT-100, 'model leans to extract' should be 'model learns to extract', and 'the learning rate starts from 1e-6 and decays to 0 following the liner schedule' should be 'linear schedule'.","section":"§4.3"},{"comment":"There are typos in the paragraph introducing general VIE benchmarks: 'widerly used' should be 'widely used', and 'comparision' should be 'comparison'.","section":"§4.6"},{"comment":"The evaluation protocol for CORD, FUNSD, SROIE, and Ticket is not described. Please specify how the field-level F1 and TED accuracy are computed for these public benchmarks, in particular how the ground-truth key-value trees are derived from the native annotations of each dataset, so that the results are reproducible.","section":"§4.6 and Table 3"},{"comment":"The citation '(Wang et al.; Li et al., 2024)' in the first paragraph of §2.1 lacks a year for the Wang et al. work; the corresponding reference list entry also appears incomplete. Please correct the citation.","section":"§2.1 and References"},{"comment":"The two case-study tables present model outputs entirely in Chinese without English translations. If the journal expects English, please provide translations or at least an English summary of what each case demonstrates, since the tables are intended to support the claim that thinking improves extraction.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for the journal, but the lack of public data, code, and model weights is a significant verification barrier. The undisclosed α is the most critical technical gap; the authors should be pushed to clarify the hyperparameter selection process. The paper also lists TextIn in the results table without introducing it, which is an editorial oversight that should be fixed. I see no grounds for rejection, but the load-bearing points need to be addressed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful core is applying a standard RLVR recipe to medical VIE with only 100 annotated images. That is a new application, not a new method, but it is a genuinely useful data point: the balanced precision/recall reward is a sensible way to handle the missing-field and hallucination trade-off, and the field-sampling strategy is a reasonable way to create query diversity. I also credit the authors for reporting that performance drops on dissimilar tasks, an honest negative result that many papers omit. The case studies are illustrative, not rigorous, but they do support the claim that the think step helps. The soft spots are in the evidence. The stress-test note is on target: alpha in Eq. (5) is the one knob that directly trades off the exact metrics reported in Table 1, and the paper never says what value was used for RL-100 or the other RL models. Figure 2 only shows alpha = 0 and alpha = 1, the degenerate extremes. If alpha was chosen by looking at the private 203-image test set, the reported F1, precision, and recall are partly selected statistics, and the comparison to baselines that did not get that tuning is not apples-to-apples. The absence of error bars or multiple seeds makes this worse: one favorable run cannot be distinguished from the expected best of several. The private test set and the undocumented TextIn baseline only add to the verification gap. These are not fatal flaws in the method; they are missing information that a thoughtful referee would demand. Overall, the central idea is sound and the direction is consistent with the broader RLVR literature, but the quantitative claims are not yet established. The paper deserves a serious referee, not a desk reject. The authors should be asked to disclose alpha, run multiple seeds with variance, evaluate on at least one public VIE benchmark or release the test set, and document all baselines. If they can do that, this becomes a useful contribution for anyone working on low-resource document extraction. I would not cite it in its current form, but I would put it on the reading list for the group working on multimodal RL.","headline":"A plausible RLVR-for-medical-VIE result with real gains, but the undisclosed reward weight and private test set make the SOTA claim hard to verify.","tokens_in":627,"tokens_out":688,"would_cite":false,"duration_ms":19640,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a 7B vision-language model with verifiable-reward reinforcement learning on just 100 annotated medical report images reaches 77.81 field-level F1, the best among compared systems on the medical VIE test set.","keywords":["visual information extraction","medical document understanding","reinforcement learning with verifiable rewards","GRPO","multimodal large language models","JSON extraction","precision-recall reward","few-shot learning"],"falsifier":"Train RLVR on the same 100 images with $\\alpha$ set to 0.2, 0.5, and 0.8 while holding everything else fixed, and evaluate each model on the 203-image medical test set; if the F1 spread across $\\alpha$ exceeds about two points, the headline 77.81 F1 is not robust to the undisclosed hyperparameter, and if the spread is negligible, the method's data-efficiency claim stands independently of $\\alpha$.","tokens_in":15078,"feed_emoji":"🩺","tokens_out":9694,"duration_ms":86480,"temperature":0.7,"pith_summary":"The paper tries to show that medical visual information extraction can be made data-efficient: 100 carefully chosen annotated report images, used with outcome-verifiable reinforcement learning, are enough to fine-tune a general 7B vision-language model into a top performer on medical JSON extraction. It argues that the key is a reward that weighs precision and recall separately, so the model is punished both for hallucinating fields and for missing real fields, together with query sampling that exposes the model to varied subsets of the schema. On a 203-image medical test set the resulting model reaches 77.81 F1, 79.85 precision, and 75.88 recall, ahead of pipeline OCR systems, larger multimodal models, and a supervised fine-tuned version. The same model also improves on receipt-like and form-like general benchmarks, but loses to the base model on dissimilar ticket and receipt tasks, which the paper reads as evidence that VIE rewards are strongly domain-specific.","feed_headline":"77.81 F1 from 100 medical images via RLVR","feed_subtitle":"A 7B vision-language model, fine-tuned on 100 annotated reports with a precision-recall reward, beats larger commercial pipelines at…","key_machinery":"The mechanism is a rule-based reward from Eq. (5): $r_{\\mathrm{matching}} = \\alpha\\, n_{\\mathrm{matched}}/|S_p| + (1-\\alpha)\\, n_{\\mathrm{matched}}/|S_g|$, where $S_p$ and $S_g$ are the flattened leaf key-value sets of predicted and ground-truth JSON, and $n_{\\mathrm{matched}}$ counts exactly matched pairs. The weight $\\alpha$ trades hallucination suppression against field coverage, and the paper combines this with a group-relative advantage estimator and the Clip-Higher and token-level policy-gradient loss modifications. Query sampling, which asks the model to extract random subsets of the schema keys rather than always the full schema, is the second load-bearing component because it shortens responses and speeds reward growth.","core_discovery":"The central claim is that RLVR with a rule-based matching reward, applied to only 100 diverse medical report images, outperforms supervised fine-tuning and much larger or commercial baselines on medical VIE. The reward flattens predicted and ground-truth JSON into leaf key-value pairs and scores them with a weighted combination of precision and recall; when the weight is set to emphasize both, the model is discouraged from emitting invented fields while still being pushed to cover the schema. The paper reports 77.81 F1, 79.85 precision, and 75.88 recall on its 203-image private medical test set, and shows the thinking-augmented model beats its no-thinking counterpart by 1.91 F1 points. On general benchmarks, the gains appear on CORD and FUNSD, which resemble the medical schema, but not on SROIE and Ticket.","pith_inferences":["The paper does not report the $\\alpha$ value used for the final models; readers cannot tell from the text whether 77.81 F1 is robust to $\\alpha$ or partly a product of tuning on the private test set.","A direct extension would apply the same 100-image RLVR recipe to other verticals such as legal, insurance, or scientific PDFs, to see whether the gain comes from RLVR generally or from the specific medical schema and reward design.","Because the reward matches flattened leaf pairs, it ignores nested structure; a prediction that places correct leaf values under the wrong parent keys could score well on F1 while failing TED accuracy, so production use should probably keep both metrics.","The 203-image private test set is small; reporting per-report-type F1 or confidence intervals would show whether the headline gain is spread across CT, ultrasound, lab, and pathology reports or driven by a few types."],"forward_implications":["Domain-specific VIE may no longer require thousands of labeled documents: 100 diverse images with verifiable JSON targets can produce a specialist extractor, provided the reward penalizes both missing and invented fields.","The precision-recall reward gives practitioners a direct dial: raising $\\alpha$ pushes toward hallucination-averse output, lowering it pushes toward full field coverage, and the paper shows the two extremes behave very differently.","Reasoning-chain supervision is not needed; the model learns to think during training because RLVR rewards the final JSON, and the think version outperforms the no-think version on F1.","Transfer of medical VIE models to other document types is limited: improvements are large on structurally similar benchmarks like CORD and FUNSD, and negative on dissimilar ones like SROIE and Ticket, implying VIE remains domain-bound.","RLVR handles unordered JSON targets better than SFT, because the reward is invariant to key ordering while cross-entropy on fixed JSON order can introduce ambiguity."],"supporting_citations":[{"why":"Supplies the group-relative advantage estimator and the R1-Zero style rule-based RLVR recipe that the training objective is built on.","marker":"(Guo et al., 2025)"},{"why":"Contributes the Clip-Higher and token-level policy-gradient loss modifications used in the final objective.","marker":"(Yu et al., 2025b)"},{"why":"Defines the Qwen2.5-VL-7B base model that all fine-tuned and RLVR models start from and compare against.","marker":"(Bai et al., 2025)"},{"why":"Used as the generator that converts OCR ground truth into initial JSON and as a commercial baseline in the medical VIE comparison.","marker":"(OpenAI, 2024a)"},{"why":"Provides the OmniDocBench protocol whose baseline settings are reused for MinerU and Marker and for the general OCR evaluation.","marker":"(Ouyang et al., 2025)"},{"why":"Source of the tree-edit-distance (TED) accuracy metric that complements field-level F1 in the evaluation tables.","marker":"(Kim et al., 2021)"},{"why":"Defines the CORD receipt benchmark, used as a similar-domain test of general VIE transfer.","marker":"(Park et al., 2019)"},{"why":"Defines the FUNSD form-understanding benchmark, another similar-domain transfer test.","marker":"(Jaume et al., 2019)"}],"fun_headline_variants":["RLVR fine-tunes 7B VL on 100 reports to 77.81 F1 for medical VIE","100 medical images, one RLVR reward: 77.81 F1 in medical VIE","Medical VIE: RLVR on 100 samples yields 77.81 F1, beating SFT","7B VL model trained on 100 reports with RLVR hits 77.81 F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reward-balancing weight $\\alpha$ is never reported for the final models, and the paper only illustrates the $\\alpha=0$ and $\\alpha=1$ extremes; if $\\alpha$ was chosen by looking at test-set performance, the reported gains over baselines could be partly an artifact of that selection.","fun_headline_variants_meta":{"raw":{"variants":["RLVR fine-tunes 7B VL on 100 reports to 77.81 F1 for medical VIE","100 medical images, one RLVR reward: 77.81 F1 in medical VIE","Medical VIE: RLVR on 100 samples yields 77.81 F1, beating SFT","7B VL model trained on 100 reports with RLVR hits 77.81 F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000777,"raw_usage":{"total_tokens":3412,"prompt_tokens":900,"completion_tokens":2512,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":2406}},"tokens_in":516,"tokens_out":2512,"duration_ms":18090,"temperature":1.0,"reasoning_tokens":2406,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:03:15.020534+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train RLVR on the same 100 images with $\\alpha$ set to 0.2, 0.5, and 0.8 while holding everything else fixed, and evaluate each model on the 203-image medical test set; if the F1 spread across $\\alpha$ exceeds about two points, the headline 77.81 F1 is not robust to the undisclosed hyperparameter, and if the spread is negligible, the method's data-efficiency claim stands independently of $\\alpha$.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the OmniDocBench protocol whose baseline settings are reused for MinerU and Marker and for the general OCR evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the CORD receipt benchmark, used as a similar-domain test of general VIE transfer."}],"review_version":2}