{"id":"d426b344-d3e1-4760-aa0a-73675445905f","arxiv_id":"2608.06758","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A Japanese document parsing model built from Nemotron-3-Nano-Omni, trained with mixed SFT and DAPO reinforcement learning, reaches 87.67 on a synthetic Japanese document parsing benchmark.","lead":"Researchers at Stockmark post-trained an existing multimodal AI model to turn images of Japanese documents into structured JSON, including tables and formulas, while trying not to erase its question answering skill. They show that mixing question-answer data into fine tuning limits forgetting, and that reinforcement learning with a custom reward pushes parsing quality higher.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed generalizable parsing improvement rests entirely on OmniDocBench-JASyn, a same-organization synthetic benchmark whose metric components are also the RL reward; without a real-document or distribution-shift test, 87.67 may only show in-distribution overfitting.","rationale":"The reader's weakest assumption correctly identifies the load-bearing vulnerability: the evaluation benchmark is synthetic, created by the same organization, and potentially aligned with the training data and reward. This is not an internal inconsistency—the experimental results are plausible and the authors give useful caveats about GSPO vs. DAPO and thinking vs. instruct modes. The data engine's ground-truth-by-construction approach is a genuine strength for training, but it also means the evaluation set may share the same generator biases, so the benchmark cannot independently validate generalizability. The reward–metric overlap makes the concern sharper: if RL directly optimizes the same similarity functions used in DocParse-Overall, then the observed improvement on the benchmark is expected under in-distribution training; the interesting claim is whether it transfers. The absence of error bars or multiple seeds further weakens the significance of the DAPO gain. I agree with the reader's CONDITIONAL verdict: the paper's internal comparisons support the direction of the claims, but the central claim of general Japanese document parsing capability requires an independent evaluation on real or at least distribution-shifted documents. The proposed real-document test would settle whether the improvement is genuine. I do not see grounds to reject the paper outright, but the condition is essential.","tokens_in":13518,"tokens_out":4906,"duration_ms":49415,"concrete_test":"Construct a held-out real-document benchmark: take 50–100 Japanese document pages from an external source such as JDocQA or public institution PDFs (not used in training), obtain manual document_structure JSON annotations with the seven classes, and compute DocParse-Overall using the same metric as Section 4.1.1. Report bootstrap confidence intervals. If the released model's real-document DocParse-Overall is not materially above the Mixed SFT checkpoint (86.60) and above Qwen3.6-27B (84.32), the claimed general parsing improvement is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that DAPO-based parsing-centric RL improves Japanese structured document parsing beyond the SFT ceiling and outperforms open-weight models (Section 4.5, Table 8)—is evaluated exclusively on OmniDocBench-JASyn, a synthetic benchmark created by the same organization (Section 4.1.1, reference [6]). The paper reports no holdout split, no distribution-shift analysis, and no real-document evaluation. The risk is compounded by reward–metric alignment: the RL reward in Table 3 uses S_text = 1−NED, S_table = TEDS, and S_formula = CDM, which are the same text, table, and formula similarity functions that define DocParse-Overall in Section 4.1.1. Training data come from the same programmatic rendering engine family (Section 2.2), so the model may learn to exploit generator-specific layout patterns, rendering artifacts, and content distributions rather than general document parsing skill. The 1.07-point improvement from DAPO over Mixed SFT (86.60 to 87.67) is also reported without error bars, seeds, or significance testing, making it impossible to distinguish real gains from noise. If OmniDocBench-JASyn is not representative of real Japanese documents, the headline improvement measures in-distribution fit to the authors' own synthetic pipeline, and the claimed general capability is unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper post-trains Nemotron-3-Nano-Omni-30B-A3B-Reasoning for Japanese structured document parsing using a synthetic data engine with two streams: a Japanese Document VQA stream and a programmatic structured document parsing stream. It compares parsing-centric SFT, mixed SFT, and parsing-centric RL with GSPO and DAPO, and reports that mixed SFT preserves most VQA ability while matching parsing gains, and that DAPO-based parsing-centric RL further improves DocParse-Overall to 87.67 on OmniDocBench-JASyn. The released model is claimed to outperform several open-weight models on this benchmark. The paper also discusses reward design and variance-based prompt filtering for continuous structured parsing rewards.","tokens_in":13850,"tokens_out":4488,"duration_ms":44321,"significance":"If the central claim survives independent evaluation, the paper would offer a practical and transferable recipe: mixed SFT followed by DAPO-based parsing-centric RL can inject structured document parsing into a reasoning-oriented VLM while limiting VQA forgetting. The manuscript is clearly written, reports training hyperparameters, releases the final model, and gives a detailed description of a synthetic data engine with ground truth by construction. However, the main evidence is currently confined to a synthetic benchmark created by the same organization, and the evaluation metric overlaps substantially with the RL reward. The generality of the claimed parsing improvement is therefore unverified; the reported 87.67 may reflect in-distribution fit to the authors' own data generation and reward rather than a general document parsing capability.","major_comments":[{"comment":"The evaluation metric DocParse-Overall is the average of (1 − Text Edit)×100, Table TEDS, and Formula CDM, while the RL reward in Table 3 uses S_text = 1−NED, S_table = TEDS, and S_formula = CDM with positive weights. The headline metric is therefore directly optimized by the training reward. Because both the training data (§2.2) and the evaluation benchmark (§4.1.1, reference [6]) are synthetic and produced by the same organization, the reported improvement from 70.43 to 87.67 can be explained as in-distribution overfitting to the reward and the generator. The central claim of generalizable parsing improvement requires evaluation on a benchmark whose metric is not a reward component and whose pages are not produced by the same data engine, for example an independently constructed real-document corpus or a clearly disjoint distribution-shift set.","section":"§4.1.1, Table 3"},{"comment":"No holdout split, contamination check, or provenance statement is reported. OmniDocBench-JASyn is released by the authors' organization and follows the same synthetic rendering approach as the training pipeline, but the paper never states that the 520 benchmark pages are disjoint from the training data generated by the engine in §2.2. Without explicit decontamination or an external evaluation, the comparison in Table 8 cannot rule out memorization of generator-specific layouts, rendering artifacts, or content distributions. The authors should provide a clear data-provenance table and evaluate on an independently created Japanese document benchmark.","section":"§4.1.1, §2.2"},{"comment":"All experiments are reported as single runs with no standard errors, no multiple seeds, and no significance testing. The key DAPO-over-Mixed-SFT gain is 1.07 DocParse-Overall points (86.60 to 87.67), and the Formula CDM gain is 3.39 points (89.48 to 92.87), while Text Edit is essentially unchanged. Given the variance of long-generation structured parsing outputs, these differences may be within noise. The authors should report multiple training runs or at least evaluation-level error bars and a paired significance test before claiming that DAPO improves beyond the SFT ceiling.","section":"§4.4, Tables 6–7"},{"comment":"The reward includes components S_schema, S_bbox, and S_picture that do not appear in DocParse-Overall, and the weights in Table 3 are hand-chosen with no sensitivity analysis. The paper calls the reward 'matched' to the evaluation, but the overlap is only partial: text, table, and formula similarities are shared, while bounding-box and picture-description quality are optimized but not measured in the headline metric. This weakens the claim that the reward is application-aware and task-aligned. A weight-sensitivity study or an ablation that removes the unmatched components would clarify whether the reported gains come from the matched content terms or from the additional structure terms.","section":"§3.2.1, §3.2.2, §4.1.1"}],"minor_comments":[{"comment":"The title and running header contain spacing typos such as 'F orgetting' and 'Official'; these should be corrected.","section":"Title and header"},{"comment":"The word 'difficult' appears repeatedly instead of 'difficult'; a global typo pass is needed.","section":"§2.1, §2.2, §3.2.3, §4.2"},{"comment":"The Japanese prompt in Appendix A appears as mojibake in the manuscript text; the appendix should be typeset with a correct Unicode font so the prompt is readable and reproducible.","section":"Appendix A"},{"comment":"The variance-filter thresholds σ_floor, μ_min, and μ_max are not reported, nor is the percentile-based calibration procedure described concretely; the authors should state the chosen values or the exact calibration rule.","section":"§3.2.3"},{"comment":"The GSPO-versus-DAPO comparison intentionally varies multiple factors, and the paper acknowledges this; a single-factor ablation would strengthen the causal reading that DAPO's token-level updates or lack of KL anchoring drive the improvement.","section":"§4.4"}],"recommendation":"major_revision","confidential_remarks":"The reader's stress-test concern is well founded. The paper's central result is plausible and the engineering effort is substantial, but the evaluation is too tightly coupled to the authors' own synthetic pipeline and to the training reward. A revision that adds an independently constructed or externally held-out benchmark, explicit contamination checks, and single-run noise quantification would materially change my assessment. I would be willing to review a revised version along these lines."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this as an honest system report rather than a claim of general Japanese document parsing capability. What is actually useful: a two-stream synthetic data engine with programmatic ground truth, a clear demonstration that mixed SFT recovers most VQA behavior lost under parsing-only SFT, and a thoughtful comparison of GSPO vs DAPO for this task. The thinking-mode vs instruct-mode SFT result (instruct mode hits 87.58 on parsing but crashes JA-Business-Doc-RQ-Bench to 0.384) is a genuine caution for anyone fine-tuning reasoning VLMs. The variance-based prompt filtering for continuous rewards also makes sense.\n\nThe soft spot is concentrated in the evaluation. OmniDocBench-JASyn is built by the same organization, uses synthetic images from a similar rendering pipeline, and the DocParse-Overall metric is a weighted average of text edit similarity, TEDS, and CDM—the same scoring functions used to build the RL reward in Table 3. The paper says the reward is \"matched\" to prevent proxy divergence, but it never addresses the fact that the benchmark is effectively a cousin of the training distribution. There is no holdout split, no real-document evaluation, no corruption or distribution-shift test. And the results are single runs: the 1.07-point DAPO improvement over mixed SFT could easily be noise, and the comparison in Table 8 has no error bars. The VQA benchmarks (JGraphQA-Refined, JDocQA-Refined, JA-Business-Doc-RQ-Bench) include some real-ish images, so the forgetting story is more convincing than the parsing capability story, but the central claim about parsing quality remains unverified outside the synthetic pipe.\n\nThe authors are careful about some things—they explicitly say the GSPO/DAPO comparison does not isolate a single design choice, and the reading-order result is labeled correlational. That honesty is real, and it makes the report more credible.\n\nFor a practitioner who wants a Japanese document parser and is willing to accept in-distribution performance, the released model is probably useful. For a researcher, the recipe is worth studying but the evaluation must be strengthened before the general-capability claim is taken seriously. I would send it to peer review, but with a request for external or at least independently held-out real-document evaluation, error bars or multiple seeds, and a sharper discussion of why reward/eval metric similarity is not itself harmless.","headline":"A credible post-training recipe for Japanese structured document parsing, but the headline scores all come from the authors' own synthetic benchmark whose metric is built from the same scoring functions as the RL reward; treat 87.67 as in-distribution until a real-document or distribution-shift test appears.","tokens_in":14354,"tokens_out":2956,"would_cite":true,"duration_ms":28571,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-stage recipe lifts Japanese document parsing to 87.67, without destroying document VQA.","keywords":["structured document parsing","Japanese document understanding","capability injection","forgetting control","DAPO reinforcement learning","synthetic data engine","document VQA","vision-language model"],"falsifier":"Evaluate the released model on a human-annotated corpus of real scanned Japanese documents, or on any independently built Japanese parsing benchmark with different rendering and degradation statistics; if DocParse-Overall on that set does not stay clearly above the base model and the Qwen baselines, the central claim of general capability injection fails. A cheaper check: compute image- or text-level overlap between the training data engine's outputs and OmniDocBench-JASyn; near-duplicate examples would indicate evaluation leakage.","tokens_in":13331,"feed_emoji":"📄","tokens_out":6264,"duration_ms":55878,"temperature":0.7,"pith_summary":"The paper reports a post-training recipe that gives a reasoning-oriented Japanese vision-language model the ability to convert a page image into a complete structured JSON representation—layout classes, bounding boxes, text, tables as HTML, formulas as LaTeX, and picture descriptions—while preserving most of its existing ability to answer questions about Japanese documents. The authors find that fine-tuning on parsing data alone improves parsing but measurably degrades document VQA; mixing VQA data back into SFT restores most of that ability with almost no parsing loss. They then show that a token-level reinforcement-learning pass (DAPO) with a task-matched, continuous reward pushes parsing further, scoring 87.67 on the OmniDocBench-JASyn benchmark, above the base model's 70.43 and above several open-weight comparison models. The cost is some additional VQA drift, which the paper treats as the central capability-injection versus forgetting-control trade-off.","feed_headline":"Two-stage recipe lifts Japanese doc parsing to 87.67","feed_subtitle":"Mixing VQA and parsing data, then RL with a matched reward, injects structured JSON extraction while preserving most question answering.","key_machinery":"The load-bearing machinery is a representation-aware reward coupled to a strict validity gate. The reward canonicalizes both prediction and reference into sets of document elements (class, bbox, contents, caption), matches predicted to ground-truth elements greedily over same-class bbox IoU, then scores schema validity, text similarity (1 minus normalized edit distance), table TEDS, formula CDM or LaTeX edit similarity, bbox IoU, and picture presence with weights 0.15, 0.45, 0.20, 0.05, 0.10, and 0.05. Any unparsable, truncated, or severely degenerate output receives zero total reward. This matching removes index-order noise so a single insertion or deletion does not misalign all later elements. Around it sit the mixed-SFT data blend (94k VQA plus 25k parsing examples) and a variance-based prompt filter that keeps only prompts whose rollout reward variance exceeds a floor and whose mean lies in a learnable band; DAPO's token-level policy-gradient update, without KL anchoring, then has precise gradients on schema, tables, and formulas.","core_discovery":"The paper's central claim is that the capability-injection/forgetting trade-off is not fixed. Parsing-centric SFT raises DocParse-Overall from 70.43 to 86.56 while dropping VQA-Overall from 0.863 to 0.826; mixing VQA examples into the same SFT run holds parsing at 86.60 while recovering VQA to 0.844. Removing the English reasoning trace entirely (instruct mode) is catastrophic for multi-step business-document reasoning (JA-Business-Doc-RQ-Bench falls from 0.895 to 0.384), so the final recipe keeps thinking-mode traces. On top of the mixed-SFT checkpoint, DAPO parsing-centric RL improves DocParse-Overall to 87.67 and Formula CDM from 89.48 to 92.87, outperforming the other open-weight models compared; GSPO, by contrast, falls to 84.01, which the authors attribute to its sequence-level importance ratio and KL anchoring being too conservative for fine-grained parsing improvement. The released model is the mixed-SFT plus DAPO checkpoint.","pith_inferences":["If the benchmark and training data share the same synthetic rendering pipeline, the recipe's transferability to real scanned Japanese documents is untested; a natural extension is evaluation on a human-annotated real-document parsing set.","Variance-based prompt filtering is a general principle for continuous-reward RL: selecting prompts by within-group reward variance rather than binary pass rate could benefit other dense-reward structured-generation tasks.","A controlled ablation separating DAPO's token-level importance ratios from its lack of KL anchoring would clarify whether the GSPO versus DAPO gap comes from update granularity or from the strength of regularization toward the SFT checkpoint.","The reported improvement in reading order is correlational; a direct ablation of bbox-based element matching against index-wise matching within the reward would test whether the matching mechanism drives that gain."],"forward_implications":["The released model achieves the best DocParse-Overall (87.67) and best Reading Order Edit (0.2366) among the compared open-weight models, so practitioners can expect stronger tables, formulas, and reading order than the Qwen baselines, at the cost of somewhat higher pure-text edit error.","Mixed SFT preserves nearly all of the parsing gain of parsing-centric SFT while recovering VQA-Overall from 0.826 to 0.844, so mixing VQA data back into SFT is a cheap forgetting-control measure.","DAPO-based RL improves parsing beyond the SFT ceiling, which means the SFT data alone does not bound achievable structured parsing quality when a matched reward and prompt filtering are used.","Dropping reasoning traces in favor of direct JSON output is harmful for multi-step document reasoning; the paper's instruct-mode experiment implies that preserving English reasoning traces during SFT is important for retaining VQA behavior."],"supporting_citations":[{"why":"Supplies the base reasoning-oriented VLM (Nemotron-3-Nano-Omni-30B-A3B-Reasoning) whose behavior the whole post-training recipe modifies.","marker":"[1]"},{"why":"Defines the OmniDocBench-JASyn evaluation set on which the central DocParse-Overall scores are measured.","marker":"[6]"},{"why":"Provides the DAPO token-level RL algorithm that the paper credits for pushing parsing beyond the SFT ceiling.","marker":"[5]"},{"why":"Defines the GSPO sequence-level RL baseline that the paper compares against and finds too conservative for fine-grained parsing.","marker":"[4]"},{"why":"Establishes the OmniDocBench format and the evaluation protocol that the DocParse-Overall metric follows.","marker":"[7]"},{"why":"Supplies the TEDS metric used for table structure similarity in both the reward and the evaluation.","marker":"[9]"},{"why":"Supplies the CDM metric used for formula recognition quality in both the reward and the evaluation.","marker":"[10]"}],"fun_headline_variants":["Mixed SFT plus DAPO beats SFT ceiling in doc parsing","Injection and forgetting control boosts parsing, saves VQA","VQA mix saves parsing, DAPO then pushes to 87.67","Two-stage recipe lifts Japanese doc parsing to 87.67","Capability injection with forgetting control yields 87.67"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation benchmark OmniDocBench-JASyn is a synthetic Japanese document set created by the same organization and generated from the same rendering pipeline family as the training data, and the paper gives no holdout split or real-document test, so the reported superiority could reflect in-distribution fit rather than general parsing skill.","fun_headline_variants_meta":{"raw":{"variants":["Mixed SFT plus DAPO beats SFT ceiling in doc parsing","Injection and forgetting control boosts parsing, saves VQA","VQA mix saves parsing, DAPO then pushes to 87.67","Two-stage recipe lifts Japanese doc parsing to 87.67","Capability injection with forgetting control yields 87.67"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1380,"prompt_tokens":1035,"completion_tokens":345,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":651,"completion_tokens_details":{"reasoning_tokens":255}},"tokens_in":651,"tokens_out":345,"duration_ms":3986,"temperature":1.0,"reasoning_tokens":255,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:13:13.666336+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the released model on a human-annotated corpus of real scanned Japanese documents, or on any independently built Japanese parsing benchmark with different rendering and degradation statistics; if DocParse-Overall on that set does not stay clearly above the base model and the Qwen baselines, the central claim of general capability injection fails. A cheaper check: compute image- or text-level overlap between the training data engine's outputs and OmniDocBench-JASyn; near-duplicate examples would indicate evaluation leakage.","supporting_citations":[{"cited_title":"OmniDocBench-JASyn","cited_arxiv_id":null,"evidence_quote":"Defines the OmniDocBench-JASyn evaluation set on which the central DocParse-Overall scores are measured."},{"cited_title":"DAPO: An open-source LLM reinforcement learning system at scale","cited_arxiv_id":null,"evidence_quote":"Provides the DAPO token-level RL algorithm that the paper credits for pushing parsing beyond the SFT ceiling."},{"cited_title":"OmniDocBench: Benchmarking diverse PDF document parsing with comprehensive annotations","cited_arxiv_id":null,"evidence_quote":"Establishes the OmniDocBench format and the evaluation protocol that the DocParse-Overall metric follows."},{"cited_title":"Image-based table recognition: data, model, and evaluation","cited_arxiv_id":null,"evidence_quote":"Supplies the TEDS metric used for table structure similarity in both the reward and the evaluation."}],"review_version":1}