{"id":"c4397c73-7a3c-4ed6-9463-b543c0ce0adb","arxiv_id":"2602.15257","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A large-scale empirical study finds that matching training context to evaluation, adding page indices, and using recursive distillation yields state-of-the-art long-document VQA on 24B and 32B models.","lead":"This paper trains open-weight vision-language models on long PDFs up to 344K context and reports recipes for continued pretraining, fine-tuning, and preference optimization that reach state-of-the-art document question-answering scores. It also releases a corrected version of MMLongBenchDoc and finds that simple interventions like page indices and matching training context to evaluation context matter.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MMLBD-C SOTA rests on a benchmark corrected with the same Qwen3-VL recursive pipeline used for training answers; the 0.4-pt original-benchmark gap is inside eval noise, so external re-annotation and official-protocol comparisons are required.","rationale":"Reader's weakest assumption matches my main concern. I looked for internal inconsistencies in the training results; page-indices, context-length matching, and CPT-vs-SFT ablations are coherent and supported by multiple tables and released artifacts. The transfer claim (+11.5 HELMET) is secondary; it has a direct controlled comparison (CPT without Prolong text) and is not the source of the SOTA assertion, although its HELMET evaluation uses 20 samples/task and would also benefit from error bars. The load-bearing issue is evaluative circularity. The authors do release annotations and use a non-family judge, which mitigates self-preference bias, but the flagging step is still produced by the same pipeline family and the manual review has no measured reliability. Because this is addressable with external annotation and official-protocol evaluation, I keep the reader's CONDITIONAL verdict rather than escalating.","tokens_in":25310,"tokens_out":6232,"duration_ms":54783,"concrete_test":"Pre-register an external re-annotation of all 251 modified and 16 removed MMLBD-C examples by two annotators blind to the recursive pipeline's flags; keep only unanimously endorsed corrections and recompute MMLBD-C for Qwen3-VL-235B, Qwen3-VL-32B plain-distillation, LongPO, and Mistral. In parallel, evaluate the same checkpoints on original MMLongBenchDoc with the official VLM EvalKit defaults (120-page cap, default resolution) and with the paper's modified settings, 3 seeds each, reporting per-benchmark CIs. If the plain-distillation 32B model still leads MMLBD-C and the 56.3-vs-56.7 gap persists outside noise under official settings, the SOTA claim is supported; otherwise it should be downgraded to conditional on MMLBD-C.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline SOTA on MMLBD-C (Table 6 footnote; Figure 1) is the main support for the central claim. The corrected benchmark is constructed in §4.1 by running the recursive pipeline (§5.2.1, App. A.6.2) — which is also used to generate training answers — to flag inconsistencies, then 251/1091 examples are manually revised. Since the same Qwen3-VL-family pipeline determines both what is a 'good' answer in the training set and what is a 'correct' label in the evaluation set, gains on MMLBD-C can partly reflect alignment to a self-consistent but biased standard. The reported 'match' on original MMLongBenchDoc (56.3 vs 56.7) is a 0.4-point margin, below the benchmark-level noise the paper itself reports for similar LC benchmarks (App. A.3: MMLongBench σ=1.66 with 20 samples/task; only aggregate σ reported for MMLBD-C). This is compounded by the nonstandard protocol in App. A.3 (max pages raised 120→336, resolution 1024×1024) when comparing to published baselines. No inter-annotator agreement is reported for the 251 manual corrections. The concern is not that the training recipe is wrong; it is that the evaluation target has been optimized in the same loop as the data, so the SOTA claim needs external validation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a large-scale empirical study of training long-context vision-language models (up to 344K tokens) for long-document visual question answering, using Mistral Small 3.1 24B and Qwen3 VL 32B as base models. The authors investigate continued pretraining (CPT), supervised finetuning (SFT), and LongPO, and release training recipes, checkpoints, and a corrected version of MMLongBenchDoc (MMLBD-C). The paper's central claims are: (i) state-of-the-art performance on MMLBD-C and matching Qwen3 VL 235B A22B on the original MMLongBenchDoc at the 32B scale; (ii) training at context lengths matching the evaluation benchmark outperforms training on longer contexts; (iii) explicit page indices provide a substantial and simple boost; (iv) the synthetic data pipelines enable self-improvement; and (v) visual long-context training transfers to long-context text performance, with a claimed +11.5 point gain on HELMET.","tokens_in":25710,"tokens_out":4787,"duration_ms":44256,"significance":"If the central claims hold, this is a valuable systems contribution: it provides reproducible, large-scale recipes for a practically important setting, and it quantifies trade-offs across CPT, SFT, and LongPO more thoroughly than most prior work. The release of checkpoints, data pipelines, training-length distributions, hardware configurations, and a large ablation suite is a genuine strength. The page-indices result and the context-length-matching finding are actionable and likely to be influential. The visual-to-text transfer result is interesting but rests on a noisier evaluation. The main caveat is that the headline SOTA claim is supported primarily by MMLBD-C, a benchmark corrected with the same recursive pipeline used to generate training data, while the original-benchmark margin is within the paper's own reported noise. The paper's value does not depend entirely on the SOTA claim, but the SOTA claim needs external validation before it can be accepted as stated.","major_comments":[{"comment":"MMLBD-C is constructed by flagging 342 examples with a version of the recursive pipeline (§5.2.1, App. A.6.2) and manually revising 251 examples and removing 16. The same recursive pipeline—including page ranking and evidence extraction—is used to generate SFT training answers. Evaluating on MMLBD-C therefore partially measures alignment with the authors' own data-generation loop rather than an independent ground truth. This circularity is load-bearing because MMLBD-C is the primary tiebreaker in the VA aggregate and supports the headline SOTA in Figure 1 and Table 6. No inter-annotator agreement is reported for the 251 manual revisions, and no external re-annotation is provided. I ask for independent re-annotation of the flagged examples with agreement statistics, and for the SOTA claim to be demonstrated on the original MMLongBenchDoc using the official protocol.","section":"§4.1, §5.2.1, App. A.6.2"},{"comment":"The SOTA claim on the original MMLongBenchDoc is not tabulated and the footnote is internally inconsistent: 'matches Qwen3 VL 235B A22B on MMLongBenchDoc with an accuracy of 56.3 (Qwen3 VL at 52.6) vs 56.7' does not identify which number is whose, and Table 6 contains only MMLBD-C. In addition, App. A.3 changes the evaluation protocol relative to published baselines: max pages is raised from 120 to 336 and resolution is set to 1024×1024. The 56.3 vs 56.7 margin is below the σ=1.66 reported for MMLongBench under the paper's own subsampling, and no per-benchmark error bars are given. The abstract's 'state-of-the-art performance on MMLongBenchDoc' is not supported unless the original benchmark is reported under the official protocol with error bars.","section":"Table 6 footnote, App. A.3"},{"comment":"The claim that 'visual long context training transfers to long-context text performance (+11.5 points on HELMET)' rests on HELMET scores from 37 to 48.5. HELMET is subsampled to 20 examples per task (App. A.3), and Table 11 reports only aggregate variance, not per-benchmark variance. No significance test is provided. The comparison baseline is also unclear: is the 37.0 score 'Mistral' as in Table 1, and is it Mistral Instruct or Mistral Base? Clarify the exact comparator and report HELMET variance or confidence intervals before this transfer result can be considered robust.","section":"§5.1.3, App. A.3"}],"minor_comments":[{"comment":"Typo: 'While several such strong are open-weight' should be 'While several such strong models are open-weight'.","section":"Abstract"},{"comment":"In the 'Document mismatch' example, 'increse' is a typo in the quoted benchmark question; if it is in the original benchmark, please note that it was corrected in MMLBD-C.","section":"§4.1"},{"comment":"The row labels are confusing: two rows are marked '× ×' but the second shows different values (84.9 etc.). Please relabel or explain the configuration of each row.","section":"Table 5"},{"comment":"Capitalization of 'HELMET' is inconsistent ('Helmet' in some places). Please standardize.","section":"Throughout"},{"comment":"The description of runs 'Eval#2' and 'Eval#3' is unclear; specify what varies across the three runs (seeds? sampling? judge?) and report per-benchmark variance.","section":"Table 11"},{"comment":"The +11.5 HELMET gain in §5.1.3 is measured without Prolong text data, while Table 1 shows +14.6 for the 100B CPT with all data. This is not inconsistent, but the difference should be stated explicitly.","section":"§5.1.3, Table 1"}],"recommendation":"major_revision","confidential_remarks":"This is a strong systems paper with unusually transparent reporting. The main risk is the benchmark-correction loop: the same recursive pipeline is used to generate training answers and to correct the evaluation benchmark, and the original-benchmark SOTA margin is within the paper's own reported noise. I would require an official-protocol comparison on MMLongBenchDoc with per-benchmark error bars, and independent re-annotation of the MMLBD-C flagged examples, before accepting the SOTA claim. The non-SOTA findings (page indices, context-length matching, compute curves) are likely solid and valuable on their own."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a serious referee. The training-recipe findings are credible and useful; the SOTA claim on MMLBD-C is shakier than the abstract implies.\n\nWhat's actually new: this is the first large-scale comparison of CPT/SFT/LongPO for long-document VQA at 24B/32B scale, with open checkpoints and recipes. The page-index trick (+2.8 points), context-length matching over longer training, and the reverse visual-to-text LC transfer (+11.5 HELMET) are all real, reproducible findings. The ablations are extensive and the authors are honest about limitations—CPT and SFT not being additive, and under-representation of extreme lengths. These insights are actionable and mostly don't depend on MMLBD-C.\n\nThe stress-test concern lands. MMLBD-C was corrected using a recursive pipeline adapted from the same one used to generate training answers, with the same Qwen3-VL model family. Human review of the 251 modified examples is a real check, but there's no inter-annotator agreement, and the same family's judgment shapes both training labels and eval labels. Gains on MMLBD-C therefore partly reflect alignment with a self-consistent standard. On the original MMLongBenchDoc, the best margin over Qwen3 VL 235B is 0.4 points—inside the eval noise the paper itself documents (MMLongBench σ=1.66 at 20 samples/task). The nonstandard eval protocol (max pages raised from 120 to 336, resolution 1024) further complicates direct comparison with published baselines. These are fixable: per-benchmark error bars, external re-annotation of a subset, and an official-protocol run.\n\nDon't overstate the damage. The central recipe findings—context matching, page indices, reverse transfer, self-improvement—are supported by multiple internal comparisons and do not hinge on MMLBD-C being perfectly corrected. The circularity mainly threatens the SOTA claim and MMLBD-C as an absolute benchmark, not the relative ablations.\n\nThis paper is for practitioners training long-context document VLMs and for anyone working on benchmark correction. Send it to peer review, but require the missing error bars, external validation of MMLBD-C, and official-protocol numbers before accepting. The recipes are worth citing now; the SOTA is not.","headline":"Solid, actionable training recipes for long-document VLMs, but the headline SOTA rests on a self-corrected benchmark and a sub-noise margin on the original; the recipe findings survive, the SOTA claim needs external validation.","tokens_in":26155,"tokens_out":2232,"would_cite":true,"duration_ms":23477,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training on context lengths matched to the target benchmark, plus explicit page indices, yields state-of-the-art long-document visual question answering at 24B and 32B scales.","keywords":["long-context vision language models","long-document visual question answering","continued pretraining","supervised finetuning","preference optimization","page indices","synthetic data pipelines","visual-to-text transfer"],"falsifier":"Run the released best checkpoints on a human-annotated, human-scored long-document VQA benchmark that no Qwen3 VL model was used to construct. If the 24B and 32B gains over baseline models shrink or vanish, the state-of-the-art claim is an artifact of the recursive correction loop rather than a genuine improvement in long-document understanding.","tokens_in":25192,"feed_emoji":"📄","tokens_out":3684,"duration_ms":33287,"temperature":0.7,"pith_summary":"This paper claims that a pragmatic, reproducible recipe produces state-of-the-art long-document visual question answering at 24B and 32B scales. The recipe covers continued pretraining, supervised finetuning, and preference optimization on synthetic data, with two simple twists: train on context lengths close to the evaluation benchmark rather than longer ones, and prepend explicit page indices to each image in the context. A 32B model trained with plain distillation matches a 235B teacher on the corrected MMLongBenchDoc variant, and the same visual training transfers to long-context text, raising HELMET by 11.5 points. The paper also releases MMLBD-C, a manually corrected and filtered version of MMLongBenchDoc, and finds that continued pretraining can be skipped when the base model's context length already suffices.","feed_headline":"Matching training length to the benchmark tops long-document VQA","feed_subtitle":"A 32B model matches a 235B teacher on the corrected benchmark; visual training also lifts text long-context by 11.5 points.","key_machinery":"The central mechanism is a recursive answer-generation pipeline for synthetic training data: it extracts evidence from each page individually, ranks pages by a relevance score, and passes the most relevant pages or extracted evidence to a strong teacher model to write answers. This pipeline enables both distillation from a larger teacher and self-improvement. Secondary load-bearing pieces are LongPO's short-to-long preference objective, model merging to prevent catastrophic forgetting, and MMLBD-C, the manually corrected benchmark used as the tiebreaker metric.","core_discovery":"The paper's central claim is that long-context visual document capability is not primarily a matter of raw context length or model scale, but of matching training data and auxiliary structure to the target task. It shows that training on context lengths similar to the evaluation benchmark outperforms training on longer contexts by 1.4–3.0 points on the visual long-context average, that explicit page indices give a +2.8 point boost on MMLBD-C, and that supervised finetuning alone, without continued pretraining, is competitive when the base context length is sufficient. It also claims the reverse of the known text-to-vision transfer: visual long-context training alone raises long-context text","pith_inferences":["The state-of-the-art claim on MMLBD-C is partly self-referential: the same recursive pipeline (relying on Qwen3 VL models) is used both to generate training answers and to flag and correct benchmark questions, so gains may reflect alignment to an internally consistent but biased standard rather than general long-document understanding.","The finding that SFT alone competes with CPT+SFT on MMLBD-C may be because most benchmark examples fall inside the base 128K context; the paper does not demonstrate gains at the advertised 344K extreme, only on under-128K benchmarks.","A testable extension: an independently human-annotated and human-scored long-document VQA benchmark would isolate whether the reported gains hold outside the paper's evaluation loop, which currently uses a VLM judge from a different model family.","The visual-to-text transfer result suggests long-context capabilities are partly modality-agnostic; a follow-up could quantify whether text-only long-context training equally benefits visual tasks, completing the symmetry."],"forward_implications":["If correct, practitioners can skip continued pretraining when the base model's context length already covers the target benchmark, saving substantial compute without sacrificing visual long-document performance.","Training context length should be matched to the target benchmark rather than increased arbitrarily; training on longer contexts can actively hurt performance.","Explicit page indices are a near-free intervention that improves long-document VQA by several points, and should be used consistently in both training and evaluation.","Visual long-context training is a viable route to improve long-context text performance, suggesting shared mechanisms across modalities that can be leveraged jointly.","A corrected benchmark like MMLBD-C changes apparent model rankings, with several checkpoints gaining more than 5 points after correction, so benchmark hygiene can materially alter conclusions."],"fun_headline_variants":["Match training length to eval for best long-doc VQA","Page indices give +2.8 points in long-doc VQA","Visual long-context training boosts text too","Long-doc VQA: training length matters more than scale","Best long-doc VQA: match context lengths, use page indices"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The evaluation and training data are corrected and generated by the same model family, so the headline benchmark gains may partly measure fit to a self-consistent but biased standard instead of general long-document ability.","fun_headline_variants_meta":{"raw":{"variants":["Match training length to eval for best long-doc VQA","Page indices give +2.8 points in long-doc VQA","Visual long-context training boosts text too","Long-doc VQA: training length matters more than scale","Best long-doc VQA: match context lengths, use page indices"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000656,"raw_usage":{"total_tokens":2855,"prompt_tokens":771,"completion_tokens":2084,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":1998}},"tokens_in":515,"tokens_out":2084,"duration_ms":12395,"temperature":1.0,"reasoning_tokens":1998,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T22:53:36.393856+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the released best checkpoints on a human-annotated, human-scored long-document VQA benchmark that no Qwen3 VL model was used to construct. If the 24B and 32B gains over baseline models shrink or vanish, the state-of-the-art claim is an artifact of the recursive correction loop rather than a genuine improvement in long-document understanding.","supporting_citations":[],"review_version":1}