Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

How to Train Your Long-Context Visual Document Model

T0 review · 3 major / 6 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Training on context lengths matched to the target benchmark, plus explicit page indices, yields state-of-the-art long-document visual question answering at 24B and 32B scales.

desk verdict Solid, actionable training recipes for long-document VLMs, but the headline SOTA rests on a self-corrected benchmark and a sub-noise margin on the original; the recipe findings survive, the SOTA claim needs external validation. read the letter →

arxiv 2602.15257 v3 pith:RLQYMBRA submitted 2026-02-16 cs.CV cs.AIcs.CL

Austin Veselka This is my paper · ORCID
classification cs.CVcs.AIcs.CL
keywords long-contextvisionlanguagemodelslong-documentvisualquestionansweringcontinuedpretrainingsupervisedfinetuningpreferenceoptimizationpageindicessyntheticdatapipelinesvisual-to-texttransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a pragmatic, reproducible recipe produces state-of-the-art long-document visual question answering at 24B and 32B scales. The recipe covers continued pretraining, supervised finetuning, and preference optimization on synthetic data, with two simple twists: train on context lengths close to the evaluation benchmark rather than longer ones, and prepend explicit page indices to each image in the context. A 32B model trained with plain distillation matches a 235B teacher on the corrected MMLongBenchDoc variant, and the same visual training transfers to long-context text, raising HELMET by 11.5 points. The paper also releases MMLBD-C, a manually corrected and filtered version of MMLongBenchDoc, and finds that continued pretraining can be skipped when the base model's context length already suffices.

What carries the argument

The central mechanism is a recursive answer-generation pipeline for synthetic training data: it extracts evidence from each page individually, ranks pages by a relevance score, and passes the most relevant pages or extracted evidence to a strong teacher model to write answers. This pipeline enables both distillation from a larger teacher and self-improvement. Secondary load-bearing pieces are LongPO's short-to-long preference objective, model merging to prevent catastrophic forgetting, and MMLBD-C, the manually corrected benchmark used as the tiebreaker metric.

What would settle it

Run the released best checkpoints on a human-annotated, human-scored long-document VQA benchmark that no Qwen3 VL model was used to construct. If the 24B and 32B gains over baseline models shrink or vanish, the state-of-the-art claim is an artifact of the recursive correction loop rather than a genuine improvement in long-document understanding.

Watch

Extended reading notes

Core claim

The paper's central claim is that long-context visual document capability is not primarily a matter of raw context length or model scale, but of matching training data and auxiliary structure to the target task. It shows that training on context lengths similar to the evaluation benchmark outperforms training on longer contexts by 1.4–3.0 points on the visual long-context average, that explicit page indices give a +2.8 point boost on MMLBD-C, and that supervised finetuning alone, without continued pretraining, is competitive when the base context length is sufficient. It also claims the reverse of the known text-to-vision transfer: visual long-context training alone raises long-context text

Load-bearing premise

The evaluation and training data are corrected and generated by the same model family, so the headline benchmark gains may partly measure fit to a self-consistent but biased standard instead of general long-document ability.

Editorial extensions

If this is right

  • If correct, practitioners can skip continued pretraining when the base model's context length already covers the target benchmark, saving substantial compute without sacrificing visual long-document performance.
  • Training context length should be matched to the target benchmark rather than increased arbitrarily; training on longer contexts can actively hurt performance.
  • Explicit page indices are a near-free intervention that improves long-document VQA by several points, and should be used consistently in both training and evaluation.
  • Visual long-context training is a viable route to improve long-context text performance, suggesting shared mechanisms across modalities that can be leveraged jointly.
  • A corrected benchmark like MMLBD-C changes apparent model rankings, with several checkpoints gaining more than 5 points after correction, so benchmark hygiene can materially alter conclusions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The state-of-the-art claim on MMLBD-C is partly self-referential: the same recursive pipeline (relying on Qwen3 VL models) is used both to generate training answers and to flag and correct benchmark questions, so gains may reflect alignment to an internally consistent but biased standard rather than general long-document understanding.
  • The finding that SFT alone competes with CPT+SFT on MMLBD-C may be because most benchmark examples fall inside the base 128K context; the paper does not demonstrate gains at the advertised 344K extreme, only on under-128K benchmarks.
  • A testable extension: an independently human-annotated and human-scored long-document VQA benchmark would isolate whether the reported gains hold outside the paper's evaluation loop, which currently uses a VLM judge from a different model family.
  • The visual-to-text transfer result suggests long-context capabilities are partly modality-agnostic; a follow-up could quantify whether text-only long-context training equally benefits visual tasks, completing the symmetry.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents a large-scale empirical study of training long-context vision-language models (up to 344K tokens) for long-document visual question answering, using Mistral Small 3.1 24B and Qwen3 VL 32B as base models. The authors investigate continued pretraining (CPT), supervised finetuning (SFT), and LongPO, and release training recipes, checkpoints, and a corrected version of MMLongBenchDoc (MMLBD-C). The paper's central claims are: (i) state-of-the-art performance on MMLBD-C and matching Qwen3 VL 235B A22B on the original MMLongBenchDoc at the 32B scale; (ii) training at context lengths matching the evaluation benchmark outperforms training on longer contexts; (iii) explicit page indices provide a substantial and simple boost; (iv) the synthetic data pipelines enable self-improvement; and (v) visual long-context training transfers to long-context text performance, with a claimed +11.5 point gain on HELMET.

Significance. If the central claims hold, this is a valuable systems contribution: it provides reproducible, large-scale recipes for a practically important setting, and it quantifies trade-offs across CPT, SFT, and LongPO more thoroughly than most prior work. The release of checkpoints, data pipelines, training-length distributions, hardware configurations, and a large ablation suite is a genuine strength. The page-indices result and the context-length-matching finding are actionable and likely to be influential. The visual-to-text transfer result is interesting but rests on a noisier evaluation. The main caveat is that the headline SOTA claim is supported primarily by MMLBD-C, a benchmark corrected with the same recursive pipeline used to generate training data, while the original-benchmark margin is within the paper's own reported noise. The paper's value does not depend entirely on the SOTA claim, but the SOTA claim needs external validation before it can be accepted as stated.

major comments (3)
  1. [§4.1, §5.2.1, App. A.6.2] MMLBD-C is constructed by flagging 342 examples with a version of the recursive pipeline (§5.2.1, App. A.6.2) and manually revising 251 examples and removing 16. The same recursive pipeline—including page ranking and evidence extraction—is used to generate SFT training answers. Evaluating on MMLBD-C therefore partially measures alignment with the authors' own data-generation loop rather than an independent ground truth. This circularity is load-bearing because MMLBD-C is the primary tiebreaker in the VA aggregate and supports the headline SOTA in Figure 1 and Table 6. No inter-annotator agreement is reported for the 251 manual revisions, and no external re-annotation is provided. I ask for independent re-annotation of the flagged examples with agreement statistics, and for the SOTA claim to be demonstrated on the original MMLongBenchDoc using the official protocol.
  2. [Table 6 footnote, App. A.3] The SOTA claim on the original MMLongBenchDoc is not tabulated and the footnote is internally inconsistent: 'matches Qwen3 VL 235B A22B on MMLongBenchDoc with an accuracy of 56.3 (Qwen3 VL at 52.6) vs 56.7' does not identify which number is whose, and Table 6 contains only MMLBD-C. In addition, App. A.3 changes the evaluation protocol relative to published baselines: max pages is raised from 120 to 336 and resolution is set to 1024×1024. The 56.3 vs 56.7 margin is below the σ=1.66 reported for MMLongBench under the paper's own subsampling, and no per-benchmark error bars are given. The abstract's 'state-of-the-art performance on MMLongBenchDoc' is not supported unless the original benchmark is reported under the official protocol with error bars.
  3. [§5.1.3, App. A.3] The claim that 'visual long context training transfers to long-context text performance (+11.5 points on HELMET)' rests on HELMET scores from 37 to 48.5. HELMET is subsampled to 20 examples per task (App. A.3), and Table 11 reports only aggregate variance, not per-benchmark variance. No significance test is provided. The comparison baseline is also unclear: is the 37.0 score 'Mistral' as in Table 1, and is it Mistral Instruct or Mistral Base? Clarify the exact comparator and report HELMET variance or confidence intervals before this transfer result can be considered robust.
minor comments (6)
  1. [Abstract] Typo: 'While several such strong are open-weight' should be 'While several such strong models are open-weight'.
  2. [§4.1] In the 'Document mismatch' example, 'increse' is a typo in the quoted benchmark question; if it is in the original benchmark, please note that it was corrected in MMLBD-C.
  3. [Table 5] The row labels are confusing: two rows are marked '× ×' but the second shows different values (84.9 etc.). Please relabel or explain the configuration of each row.
  4. [Throughout] Capitalization of 'HELMET' is inconsistent ('Helmet' in some places). Please standardize.
  5. [Table 11] The description of runs 'Eval#2' and 'Eval#3' is unclear; specify what varies across the three runs (seeds? sampling? judge?) and report per-benchmark variance.
  6. [§5.1.3, Table 1] The +11.5 HELMET gain in §5.1.3 is measured without Prolong text data, while Table 1 shows +14.6 for the 100B CPT with all data. This is not inconsistent, but the difference should be stated explicitly.

Circularity Check

2 steps flagged · score 5.0 of 10

MMLBD-C SOTA is partly self-referential: the same Qwen3-VL recursive pipeline both vets the corrected benchmark and generates the training answers.

  1. self definitional [§4.1 (MMLBD-C construction), §5.2.1 (answer generation), Table 6 footnote / Figure 1 (SOTA claim)]
    "We construct MMLBD-C by flagging and correcting issues in MMLongBenchDoc ... To do this, we apply a version of the recursive pipeline (see Section 5.2.1 and Appendix A.6.2) adapted to find inconsistencies between the source, question and answer. ... the first is a recursive pipeline which extracts evidence relevant to the given question from each page individually, uses a numerical score from the extraction model to rank the pages by relevance and passes either the most relevant pages or their extracted evidence to Qwen3 VL 235B A22B or Qwen3 235B respectively."

    The headline SOTA is on MMLBD-C, whose corrected labels were produced by applying the recursive Qwen3-VL pipeline to flag/revise examples. The same model family (Qwen3 VL 235B A22B) and the same page-ranking/evidence mechanism generate the SFT training answers ('plain distillation' or 'recursive'), and the best checkpoint is the Qwen3-VL 32B plain-distillation run. Consequently, MMLBD-C scores substantially measure agreement with the teacher/pipeline that defined the evaluation standard, not an externally fixed ground truth. The 251 manual edits and the original MMLongBenchDoc match provide partial independence, but the primary SOTA claim remains partly self-referential.

  2. other [§5.4 (Self-improvement), Table 7, Takeaway #5]
    "Our proposed recursive answer generation pipeline is not limited to the context available at the time of question generation and thus we distill an algorithm into the model that involves a non-trivial search over the full context. ... we find that CPT for 100B tokens achieves the strongest self-improvement performance with +3.8 VA and though SFT uses far less compute, we find that SFT alone is also effective, yielding +3.2 VA, while also surpassing CPT on MMLBD-C."

    The self-improvement gains are quantified on VA, with MMLBD-C as the tiebreaker and a major component. The training answers for self-improvement are generated by the same recursive pipeline, and MMLBD-C's corrections were produced by a version of that pipeline, so the +3.2/+3.8 VA result partly reflects the student learning the pipeline's own consistency criteria rather than an external capability gain. HELMET/LongBench v2 columns in Table 7 are external, but the headline takeaway is the VA number.

full rationale

The paper's recipe-level findings are largely self-contained and independently grounded: context-length matching and page indices are supported across multiple external benchmarks, and the visual-to-text transfer claim (+11.5 on HELMET) is measured against a fixed text benchmark with no training/eval overlap. The circularity is concentrated in the MMLBD-C-centered claims. MMLBD-C is not an untouched public benchmark: it is a corrected variant whose corrections were generated by the same recursive Qwen3-VL pipeline used to create training answers, so SOTA and self-improvement numbers on it partly quantify alignment to the authors' own teacher/pipeline rather than to independent ground truth. The 251 manual revisions and the reported match on original MMLongBenchDoc provide partial external anchoring, but the primary SOTA headline is on MMLBD-C. There is no load-bearing self-citation chain or imported uniqueness theorem, and no fitted parameter is renamed as a prediction. Score 5 reflects partial circularity in the headline evaluation target while acknowledging that the central training-recipe contributions retain independent content.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the quality of the teacher model, the validity of the benchmarks (including the self-corrected MMLBD-C), and hand-tuned choices for model merging scale, resolution scaling, and evaluation subsampling. No new physical entities are introduced.

free parameters (4)
  • Model merging scaling factor = 0.5 for Mistral CPT; 0.25 for SFT/LongPO and Qwen3 VL
    Chosen by hand (Appendix A.1); no ablation reported; directly affects whether merging prevents forgetting and final scores.
  • Recursive pipeline page-selection threshold = Not specified
    The pipeline 'uses a numerical score from the extraction model to rank pages' (Section 5.2.1) but the cut-off used to select pages is never given; this choice shapes training data and MMLBD-C corrections.
  • Dynamic resolution scaling range = 616-840 (CPT), 728-1400 (SFT/LongPO)
    Chosen to fit context (Appendix A.1); changes tokens per image and thus the training distribution.
  • Evaluation subsample size for HELMET/MMLongBench = 20 samples per task
    Section A.3: 'we limit expensive benchmarks (HELMET and MMLongBench) to 20 samples per task'; increases noise (σ=1.66 for MMLongBench) and may bias aggregates.
assumptions (5)
  • domain assumption The teacher model (Qwen3 VL 235B A22B) provides answers accurate enough for distillation, self-improvement, and benchmark correction.
    Used throughout Sections 5.2-5.4; if the teacher is biased, training and the corrected benchmark share that bias.
  • domain assumption The benchmark suite (MMLongBenchDoc, MMLBD-C, MMLongBench, DUDE, SlideVQA, HELMET, LongBench v2) faithfully measures long-document understanding.
    Section 4; the results are aggregated into VA/LCA and treated as ground truth.
  • ad hoc to paper The recursive pipeline's evidence extraction and page ranking correctly identify the pages that support an answer.
    Section 5.2.1/A.6.2; no independent validation of the page-selection mechanism; it is used both to generate SFT answers and to flag MMLBD-C corrections.
  • domain assumption LongPO (Chen et al., 2025) and its short-to-long constraint transfer to the multimodal setting with β=0.1, λ=0.01.
    Section 3, Eq. 1; the paper adopts the method without re-deriving or validating the constraint for VLMs.
  • domain assumption The base models (Mistral Small 3.1 Instruct, Qwen3 VL 32B Instruct) are appropriate starting points and their instruct behavior is preserved via model merging.
    Section 3/A.1; all conclusions are tied to these two model families.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How to Train Your Long-Context Visual Document Model." pith.science (2026). https://pith.science/paper/RLQYMBRA

@misc{pith2026260215257,
  author       = {Pith},
  title        = {Pith review of: How to Train Your Long-Context Visual Document Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RLQYMBRA}},
  note         = {Machine review of arXiv:2602.15257}
}
read the original abstract

We present the first comprehensive, large-scale study of training long-context vision language models up to 344K context, targeting long-document visual question answering with measured transfer to long-context text. While several such strong are open-weight, namely Qwen3 VL and GLM 4.5/6V, their training recipes and data pipelines are not reproducible. We systematically study continued pretraining, supervised finetuning, and preference optimization for 24B and 32B parameter models, backed by extensive LC evaluations and ablations to bridge this gap, and achieve state-of-the-art performance on MMLongBenchDoc for both parameter scales. In addition to this, our key findings include: (i) training on context lengths that match evaluation context lengths outperforms training on longer contexts, (ii) training and evaluating with page indices provides a simple, high-impact boost to long-document performance, (iii) our synthetic data pipelines enable self-improvement via continued pretraining and supervised finetuning, and (iv) we extend the known text-to-visual long context transfer to the reverse, showing that visual long context training transfers to long-context text performance. We also release MMLBD-C, a manually corrected version of MMLongBenchDoc to reduce erroneous and low quality examples in the benchmark.

Figures

Figures reproduced from arXiv: 2602.15257 by the authors.

Figure 1
Figure 1. Performance for our best training recipes compared to the base models we train [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of the scraped PDF corpus: (left) total pages by top-level category [PITH_FULL_IMAGE:figures/full_fig_p017_2.png] view at source ↗
Figure 3
Figure 3. Length distributions of training examples. (Left) CPT example length (tokens): [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Distribution of number of pages per PDF in the PDFA English split. [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Top subcategories by total pages within the scraped PDF corpus (grouped by [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: Examples of issues in MMLongBenchDoc: (a) question paired with wrong docu [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]
Figure 7
Figure 7. Figure 7: Compute vs. Average Visual LC (VA) performance across key training runs in [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]
Figure 8
Figure 8. Figure 8: PDF page-length distribution after upsampling long documents for CPT (compared [PITH_FULL_IMAGE:figures/full_fig_p024_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Internalized Reasoning for Long-Context Visual Document Understanding

    cs.CV 2026-03 unverdicted novelty 7.0 of 10

    Synthetic page-ranked reasoning traces plus low-strength model merging give a 32B VLM 58.3 on MMLongBenchDoc, beating a 235B teacher while cutting output tokens ~12× versus explicit reasoning.

Reference graph

Works this paper leans on

6 extracted references · 5 linked inside Pith · cited by 1 Pith paper

  1. [4]

    Hongyuan Tao, Bencheng Liao, Shaoyu Chen, Haoran Yin, Qian Zhang, Wenyu Liu, and Xinggang Wang

    URLhttps://arxiv.org/abs/2301.04883. Hongyuan Tao, Bencheng Liao, Shaoyu Chen, Haoran Yin, Qian Zhang, Wenyu Liu, and Xinggang Wang. Infinitevl: Synergizing linear and sparse attention for highly- efficient, unlimited-input vision-language models, 2025. URL https://arxiv.org/abs/ 2512.08829. Liang Wang, Nan Yang, Xingxing Zhang, Xiaolong Huang, and Furu W...

    arXiv 2025

  2. [5]

    13 Preprint

    URLhttps://openreview.net/forum?id=Ns8zGZ0lmM. 13 Preprint. Under review. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large lan- guage models, 2023. URLhttps://arxiv.org/abs/2201.11903. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, J...

    arXiv 2023

  3. [2023]

    Greg Kamradt

    URLhttps://arxiv.org/abs/2212.04089. Greg Kamradt. Pressure testing gpt-4-128k with long context recall. https://x.com/ GregKamradt/status/1722386725635580292?lang=en, 2023. Accessed 2026-01-23. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franc ¸ois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention, 2020. ...

    arXiv 2023

  4. [2024]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt

    URLhttps://arxiv.org/abs/2312.00752. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL https://arxiv.org/abs/2009.03300. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Rule...

    arXiv 2021

  5. [2025]

    URLhttps://arxiv.org/abs/2502.13922. Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, Ethan He, Hongxu Yin, Pavlo Molchanov, Jan Kautz, Linxi Fan, Yuke Zhu, Yao Lu, and Song Han. Longvila: Scaling long-context visual language models for long videos, 2024. URLhttps://arxiv.org/abs/24...

    arXiv 2024

  6. [2026]

    least”→“lease

    URLhttps://arxiv.org/abs/2502.04194. Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision, 2024a. URLhttps://arxiv.org/abs/2406.16852. Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Le...

    arXiv 2024

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.