Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

MM-R5: MultiModal Reasoning-Enhanced ReRanker via Reinforcement Learning for Document Retrieval

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 7B reranker explicitly trained to reason about page relevance reaches the top MMDocIR results, beating retrieval-only baselines and larger 32B models.

desk verdict Solid empirical paper on RL+CoT for multimodal reranking, but the SOTA claim is undercut by the missing MonoQwen baseline. read the letter →

arxiv 2506.12364 v2 pith:Y2TAAOIB submitted 2025-06-14 cs.AI cs.CLcs.CV

Mingjun Xu , Jinhan Dong , Jue Hou , Zehui Wang , Sihang Li , Zhifeng Gao , Renxin Zhong , Hengxing Cai This is my paper · ORCID
classification cs.AIcs.CLcs.CV
keywords multimodalrerankingchain-of-thoughtreasoningreinforcementlearningGRPOvision-languagemodelsdocumentretrievalMMDocIRretrieval-augmentedgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MM-R5 is a reranker for multimodal document retrieval: given a query and a set of candidate pages, it looks at every page, writes a short chain-of-thought about each page's relevance, and outputs a ranked list. The paper's central claim is that this reasoning behavior can be trained in two stages—supervised fine-tuning on per-image reasoning traces, then reinforcement learning with a reward that favors correct pages near the top—and that this training is what makes the reranker accurate. On the MMDocIR benchmark, MM-R5 reaches macro Recall@1 of 0.6951 and micro Recall@1 of 0.6759, improving on the best retriever-only baseline by 4.70 and 4.05 absolute points and beating much larger 32B vision-language models. The paper concludes that a relatively small model can outperform larger general-purpose models on reranking when it is explicitly taught to reason, and that the visible reasoning chains make the reranker's decisions interpretable.

What carries the argument

The load-bearing mechanism is the two-stage training recipe plus the reward shape. For SFT, the data construction decomposes each multi-page ranking question into single-image sub-tasks; a strong vision-language model (GPT-4o) explains why each page is or is not relevant, those explanations are concatenated into a <think>...</think> reasoning chain, and the ground-truth label order is wrapped in <answer>...</answer>. This sidesteps the multi-image reasoning weakness of existing VLMs and yields 7,200 training instances. For RL, the paper applies GRPO with two rewards: a result reward $R_{\text{result}} = \frac{\sum_{j} s_j / j^3}{\sum_{j=1}^{|G|} 1/j^3}$, where $s_j=1$ if the item predicted at rank $j$ is in the golden set, which weights correct pages by the inverse cube of their predicted rank and so emphasizes the top position far more than logarithmic discounting; and a format reward $R_{\text{format}} = R_{\text{valid}} \times R_{\text{len}} \times R_{\text{range}}$, which hard-gates on the <think>/<answer> structure and smoothly penalizes wrong list lengths and out-of-range indices. The RL data is drawn with a resolution-balanced sampling strategy so that documents with very different image sizes contribute evenly.

What would settle it

Train the same two-stage pipeline but replace the GPT-4o reasoning traces with (a) labels-only SFT without any reasoning chains or (b) reasoning chains whose per-page relevance judgments are randomly shuffled, keeping the RL stage identical; if macro Recall@1 on MMDocIR does not drop substantially from 0.6951, the reasoning data is not the carrier of the gain. A more direct check is to compare GPT-4o's per-image relevance judgments with the page-level ground-truth labels on a held-out MMDocIR subset: low agreement would predict the SFT stage cannot teach correct relevance.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a multimodal reranker can be made both more accurate and more interpretable by treating chain-of-thought reasoning as a trainable behavior rather than a prompted style. The evidence is on MMDocIR: the full MM-R5 model reaches macro Recall@1 of 0.6951, versus 0.6481 for ColQwen retriever-only, 0.6768 for Qwen2.5-VL-32B-cot, 0.6673 for the SFT-only variant, and 0.6586 for the RL-only variant. The same model reranking five different retrievers improves Recall@1 in every configuration, with the largest relative gains on the weakest retrievers (CLIP macro Recall@1 rises from 0.3334 to 0.5942). From these results the paper concludes that the two stages are complementary, that CoT improves ranking, and that the gains are not tied to a particular retrieval backbone.

Load-bearing premise

The load-bearing premise is that the automatically generated reasoning traces from GPT-4o are accurate and complete enough to teach correct page-level relevance; no human verification or quality metric is reported for those chains, so any bias or noise in them could propagate through both training stages into the final rankings.

Editorial extensions

If this is right

  • On MMDocIR, reranking with MM-R5 raises macro Recall@1 by 4.70 absolute points over ColQwen alone, so reasoning-trained reranking can be a drop-in improvement on top of a strong retriever.
  • The full two-stage model beats both single-stage variants (0.6951 vs 0.6673 SFT-only and 0.6586 RL-only), so the paper's recipe says SFT and RL contribute complementary gains rather than being redundant.
  • MM-R5 improves Recall@1 for all five tested retrievers, with the largest relative gains for the weakest ones, meaning the method can partially compensate for retriever weakness.
  • A 7B model with task-specific reasoning training surpasses the 32B general-purpose VLM with CoT prompting (macro Recall@1 0.6951 vs 0.6768), suggesting task-specific training can substitute for model scale in reranking.
  • Because the model emits explicit per-page reasoning inside <think> tags, reranker outputs become inspectable, which the paper presents as a step toward controllable and trustworthy multimodal retrieval.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable consequence the paper does not run: swapping GPT-4o for a cheaper or open-weight teacher in the SFT data pipeline would isolate how much of the gain depends on teacher quality; the paper's design implies teacher quality matters but never varies it.
  • The inverse-cube reward is motivated as stronger top-rank emphasis than NDCG's logarithmic discount, but the paper does not ablate reward shape; comparing $j^{-3}$ against $1/\log_2(j+1)$ on the same RL pipeline would show whether the cube itself is load-bearing.
  • The format-reward hard gate on <think>/<answer> tags suggests the pipeline is also a structured-output trainer; the same SFT+GRPO recipe may transfer to other list-generation tasks with strict output schemas, such as layout ranking or multi-step tool selection, though the paper only tests document pages.
  • The resolution-balanced sampling addresses a distributional shift that likely affects any VLM reranker trained on mixed-resolution documents; applying this sampling to other multimodal training regimes could reduce RL instability beyond reranking.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents MM-R5, a multimodal reranker built on Qwen2.5-VL-7B, trained in two stages: supervised fine-tuning on GPT-4o-generated reasoning chains and reinforcement learning with GRPO using a result reward (normalized inverse-cube DCG) plus a format reward. Evaluation on MMDocIR reports macro Recall@1 of 0.6951 when reranking ColQwen candidates, an absolute gain of 4.7 points over ColQwen, and consistent gains when applied to four other retrievers. The paper claims state-of-the-art performance on MMDocIR and attributes the gains to the reasoning-enhanced two-stage pipeline.

Significance. The work is potentially useful: it provides a concrete recipe for injecting explicit CoT reasoning into a multimodal reranker, with ablations showing that both SFT and RL contribute, and a generalization study across five retrievers. The public code and use of a public benchmark are strengths, and the reward design based on ground-truth relevance labels avoids circularity. The main quantitative claims, however, rest on a comparison set that omits MonoQwen, the most directly comparable prior training-based reranker, and on differences of 1–3 points that are reported without variance or significance. The core idea is interesting and the experiments are extensive, but the SOTA label is not yet established.

major comments (3)
  1. [§4.1.4, Table 1] The claim that MM-R5 achieves state-of-the-art performance on MMDocIR is not verifiable from the reported comparisons because MonoQwen (Chaffin and Lac 2024), cited in §2.2 as a training-based multimodal reranker, is not included in Table 1 or §4.1.4. Since MonoQwen is a Qwen-based visual document reranker specifically designed for this task, it is the most directly comparable prior system; without its numbers on the same ColQwen top-10 candidates, the SOTA assertion for macro Recall@1 (0.6951) is unsupported. Please add MonoQwen and, if possible, other reranking systems from the WWW 2025 Multimodal RAG Challenge leaderboard, since the Introduction reports second place without identifying the winner.
  2. [Table 1, §4.2.1] The reported margins over the strongest baselines are small (e.g., 0.6951 vs. 0.6768 for Qwen2.5-VL-32B-cot and 0.6951 vs. 0.6673 for the SFT-only variant), and no variance estimates, multiple runs, or significance tests are provided. Without these, it is difficult to judge whether the claimed improvements are stable or within noise, particularly for the macro Recall@1 differences of about 1–3 points. Please report standard deviations over at least three seeds or provide a significance analysis for the main comparisons.
  3. [§3.2.1] The SFT stage depends entirely on 7,200 automatically generated reasoning chains produced by GPT-4o via single-image sub-tasks, followed by a GPT-4o-based refinement. The paper asserts that these are high-quality but provides no human evaluation, no inter-annotator agreement, and no automatic quality metrics for the reasoning chains. Since the quality of this data is the foundation for the claimed contribution of the SFT stage, the paper should include at least a small-scale human validation or a comparison of the generated reasoning against an alternative construction method, such as full multi-image GPT-4o reasoning.
minor comments (6)
  1. [§3.3.2] The sentence defining the length accuracy reward uses "donate" where "denote" is intended; please correct the typo.
  2. [§3.3.2, Eq. (4)] The text describes the three format-reward components as "independent and differentiable," but the structure validity reward is a hard binary indicator and is not differentiable; please rephrase to avoid the inaccurate claim.
  3. [Figure 1] The figure uses closing tags written as "<\think>" and "<\answer>" while the text uses "</think>" and "</answer>"; please unify the notation.
  4. [Table 1] The baseline is labeled "RagVL" in the table but "RAG-VL" in the text; please make the naming consistent.
  5. [§4.1.3, Table 1 caption] Please clarify the initialization of the RL-only variant (Qwen2.5-VL-7B-rl): does it start from the base Qwen2.5-VL-7B or from the SFT checkpoint? This affects the interpretation of the ablation.
  6. [References] The reference to MonoQwen (Chaffin and Lac 2024) gives only a title and no venue or technical report identifier; since the paper discusses it in §2.2 and it is directly relevant to the experiments, please provide a complete citation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MM-R5's gains are measured against fixed ground-truth labels and a held-out benchmark, with training signals from an external model and ground-truth relevance, so the central results are not forced by construction.

full rationale

The paper's derivation chain is self-contained and non-circular. The SFT reasoning data are generated by an external model (GPT-4o) from the query, each candidate image, and the ground-truth relevance label; the final ranked list is sorted from those labels, not from MM-R5's own outputs (Sec. 3.2.1). The RL result reward in Eq. (1) uses the golden set G of relevant indices from the benchmark, and the format reward checks structural properties of the output; neither defines success in terms of the model's own predictions. Evaluation is performed on the fixed MMDocIR evaluation split with expert-annotated page-level labels, separately from the training data (Sec. 4.1.1). Reporting Recall@k after training with a DCG-style reward is an empirical outcome, not a tautology, because optimizing a surrogate ranking reward does not guarantee a particular held-out Recall@k value. The omission of MonoQwen from Table 1 is a legitimate concern about the breadth of the SOTA comparison, but it is a baseline-coverage/correctness issue rather than evidence that any result reduces to its inputs by construction. No self-citation chain or imported uniqueness theorem is load-bearing in the method or the evaluation. Therefore no circular step can be exhibited, and the appropriate score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on several hand-chosen training hyperparameters and on the reliability of external data sources (MMDocIR labels, GPT-4o reasoning chains). These are all reasonable for an empirical systems paper, but they are not derived from first principles.

free parameters (4)
  • Reward exponent in Rresult = 3
    Chosen to emphasize top-ranked items; no sensitivity analysis or tuning against validation is reported.
  • Number of SFT examples = 7,200
    Set based on budget; no scaling study.
  • Number of RL samples = 3,000
    Selected proportionally from resolution buckets; no ablation of this size choice.
  • Number of resolution subsets = 10
    A heuristic bucket count; the effect of this choice is not tested.
assumptions (4)
  • domain assumption MMDocIR ground-truth page labels are correct and exhaustive.
    The benchmark is used for both training and evaluation; if labels are noisy, the reward and metrics are unreliable.
  • domain assumption GPT-4o generated reasoning data is high quality and accurately reflects page relevance.
    The SFT stage relies entirely on this synthetic data; no human verification or quality metrics are provided.
  • domain assumption GRPO with one epoch and LoRA is sufficient to optimize the reranking policy.
    The paper does not study convergence or the effect of longer training; this is a practical assumption about the RL setup.
  • domain assumption The inverse-cube reward is a valid surrogate for retrieval quality.
    The reward is a hand-designed weighting and is not validated against user preferences or other ranking metrics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MM-R5: MultiModal Reasoning-Enhanced ReRanker via Reinforcement Learning for Document Retrieval." pith.science (2026). https://pith.science/paper/Y2TAAOIB

@misc{pith2026250612364,
  author       = {Pith},
  title        = {Pith review of: MM-R5: MultiModal Reasoning-Enhanced ReRanker via Reinforcement Learning for Document Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y2TAAOIB}},
  note         = {Machine review of arXiv:2506.12364}
}
read the original abstract

Multimodal document retrieval systems enable information access across text, images, and layouts, benefiting various domains like document-based question answering, report analysis, and interactive content summarization. Rerankers improve retrieval precision by reordering retrieved candidates. However, current multimodal reranking methods remain underexplored, with significant room for improvement in both training strategies and overall effectiveness. Moreover, the lack of explicit reasoning makes it difficult to analyze and optimize these methods further. In this paper, We propose MM-R5, a MultiModal Reasoning-Enhanced ReRanker via Reinforcement Learning for Document Retrieval, aiming to provide a more effective and reliable solution for multimodal reranking tasks. MM-R5 is trained in two stages: supervised fine-tuning (SFT) and reinforcement learning (RL). In the SFT stage, we focus on improving instruction-following and guiding the model to generate complete and high-quality reasoning chains. To support this, we introduce a novel data construction strategy that produces rich, high-quality reasoning data. In the RL stage, we design a task-specific reward framework, including a reranking reward tailored for multimodal candidates and a composite template-based reward to further refine reasoning quality. We conduct extensive experiments on MMDocIR, a challenging public benchmark spanning multiple domains. MM-R5 achieves state-of-the-art performance on most metrics and delivers comparable results to much larger models on the remaining ones. Moreover, compared to the best retrieval-only method, MM-R5 improves recall@1 by over 4%. These results validate the effectiveness of our reasoning-enhanced training pipeline. Our code is available at https://github.com/i2vec/MM-R5 .

Figures

Figures reproduced from arXiv: 2506.12364 by the authors.

Figure 1
Figure 1. Workflow of MM-R5. It takes all candidate pages [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of our two-stage training pipeline. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Prompt template for Chain-of-Thought based Reranking. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Comparison between our model and multimodal rerankers without training. All page candidates are retrieved by [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Learning from Failures: Retrieval-Centric CoT via Hard Negatives for Unified Multimodal Retrieval

    cs.CV 2026-08 conditional novelty 7.0 of 10

    UniME-R1 uses a failure-aware adviser to diagnose embedding mistakes from initial retrieval results and then either reranks candidates or re-retrieves with a feedback-based query rewrite.

  2. UniRank: End-to-End Domain-Specific Reranking of Hybrid Text-Image Candidates

    cs.IR 2026-02 conditional novelty 5.0 of 10

    UniRank natively scores hybrid text-image candidates with a VLM, then SFT plus hard-negative RLHF adapts it to a domain; it outperforms baselines on scientific literature and design patent retrieval.

Reference graph

Works this paper leans on

53 extracted references · 13 canonical work pages · cited by 2 Pith papers

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Ahmadian, A.; Cremer, C.; Gall \'e , M.; Fadaee, M.; Kreutzer, J.; Pietquin, O.; \"U st \"u n, A.; and Hooker, S. 2024. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740

    arXiv 2024

  4. [4]

    Alayrac, J.-B.; Donahue, J.; Luc, P.; Miech, A.; Barr, I.; Hasson, Y.; Lenc, K.; Mensch, A.; Millican, K.; Reynolds, M.; et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35: 23716--23736

    2022

  5. [5]

    Bai, S.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Song, S.; Dang, K.; Wang, P.; Wang, S.; Tang, J.; et al. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923

    arXiv 2025

  6. [6]

    Chaffin, A.; and Lac, A. 2024. MonoQwen: Visual Document Reranking

    work page 2024

  7. [7]

    Chen, L.; Li, L.; Zhao, H.; Song, Y.; and Vinci. 2025. R1-V: Reinforcing Super Generalization Ability in Vision-Language Models with Less Than \ 3. https://github.com/Deep-Agent/R1-V. Accessed: 2025-02-02

    work page 2025

  8. [8]

    Chen, Z.; Xu, C.; Qi, Y.; and Guo, J. 2024. Mllm is a strong reranker: Advancing multimodal retrieval-augmented generation via knowledge-enhanced reranking and noise-injected training. arXiv preprint arXiv:2407.21439

    arXiv 2024

Show all 53 references
  1. [9]

    Cui, L.; Xu, Y.; Lv, T.; and Wei, F. 2021. Document ai: Benchmarks, models and applications. arXiv preprint arXiv:2111.08609

    2021 arXiv

  2. [10]

    Deng, H.; Zou, D.; Ma, R.; Luo, H.; Cao, Y.; and Kang, Y. 2025. Boosting the generalization and reasoning of vision language models with curriculum reinforcement learning. arXiv preprint arXiv:2503.07065

    2025 arXiv

  3. [11]

    D.; Li, D.; Tang, R.; and Liu, Y

    Dong, K.; Chang, Y.; Goh, X. D.; Li, D.; Tang, R.; and Liu, Y. 2025. MMDocIR: Benchmarking Multi-Modal Retrieval for Long Documents. arXiv preprint arXiv:2501.08828

    2025

  4. [12]

    Faysse, M.; Sibille, H.; Wu, T.; Omrani, B.; Viaud, G.; Hudelot, C.; and Colombo, P. 2024. Colpali: Efficient document retrieval with vision language models. In The Thirteenth International Conference on Learning Representations

    2024

  5. [13]

    Guo, D.; Yang, D.; Zhang, H.; Song, J.; Zhang, R.; Xu, R.; Zhu, Q.; Ma, S.; Wang, P.; Bi, X.; et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  6. [14]

    Guo, F.; Li, W.; Zhuang, H.; Luo, Y.; Li, Y.; Yan, L.; and Zhang, Y. 2024. Generating diverse criteria on-the-fly to improve point-wise LLM rankers. CoRR

    2024

  7. [15]

    Hendrycks, D.; Burns, C.; Chen, A.; and Ball, S. 2021. CUAD: an expert-annotated NLP dataset for legal contract review. arXiv preprint arXiv:2103.06268

    2021 arXiv

  8. [16]

    Huang, W.; Jia, B.; Zhai, Z.; Cao, S.; Ye, Z.; Zhao, F.; Xu, Z.; Hu, Y.; and Lin, S. 2025. Vision-r1: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06749

    2025 arXiv

  9. [17]

    A.; and Manning, C

    Hudson, D. A.; and Manning, C. D. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 6700--6709

    2019

  10. [18]

    Jiang, T.; Song, M.; Zhang, Z.; Huang, H.; Deng, W.; Sun, F.; Zhang, Q.; Wang, D.; and Zhuang, F. 2024. E5-v: Universal embeddings with multimodal large language models. arXiv preprint arXiv:2407.12580

    2024 arXiv

  11. [19]

    K.; Mart \' nez, J

    Koukounas, A.; Mastrapas, G.; G \"u nther, M.; Wang, B.; Martens, S.; Mohr, I.; Sturua, S.; Akram, M. K.; Mart \' nez, J. F.; Ognawala, S.; et al. 2024. Jina clip: Your clip model is also your text retriever. arXiv preprint arXiv:2405.20204

    2024 arXiv

  12. [20]

    Lee, J.; Ko, J.; Baek, J.; Jeong, S.; and Hwang, S. J. 2024. Unified Multi-Modal Interleaved Document Representation for Information Retrieval. arXiv preprint arXiv:2410.02729

    2024

  13. [21]

    Liu, Z.; Sun, Z.; Zang, Y.; Dong, X.; Cao, Y.; Duan, H.; Lin, D.; and Wang, J. 2025. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785

    2025 arXiv

  14. [22]

    Lu, P.; Bansal, H.; Xia, T.; Liu, J.; Li, C.; Hajishirzi, H.; Cheng, H.; Chang, K.-W.; Galley, M.; and Gao, J. 2023. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255

    2023 arXiv

  15. [23]

    Luo, J.; Chen, X.; He, B.; and Sun, L. 2024. Prp-graph: Pairwise ranking prompting to llms with graph aggregation for effective text re-ranking. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 5766--5776

    2024

  16. [24]

    Ma, X.; Lin, S.-C.; Li, M.; Chen, W.; and Lin, J. 2024. Unifying multimodal retrieval via document screenshot embedding. arXiv preprint arXiv:2406.11251

    2024 arXiv

  17. [25]

    Ma, X.; Zhang, X.; Pradeep, R.; and Lin, J. 2023. Zero-shot listwise document reranking with a large language model. arXiv preprint arXiv:2305.02156

    2023 arXiv

  18. [26]

    Marino, K.; Rastegari, M.; Farhadi, A.; and Mottaghi, R. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, 3195--3204

    2019

  19. [27]

    Meng, F.; Du, L.; Liu, Z.; Zhou, Z.; Lu, Q.; Fu, D.; Han, T.; Shi, B.; Wang, W.; He, J.; et al. 2025. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2503.07365

    2025 arXiv

  20. [28]

    Moreira, G. d. S. P.; Ak, R.; Schifferer, B.; Xu, M.; Osmulski, R.; and Oldridge, E. 2024. Enhancing Q&A Text Retrieval with Ranking Models: Benchmarking, fine-tuning and deploying Rerankers for RAG. arXiv preprint arXiv:2409.07691

    2024 arXiv

  21. [29]

    Nogueira, R.; and Cho, K. 2019. Passage Re-ranking with BERT. arXiv preprint arXiv:1901.04085

    2019 arXiv

  22. [30]

    Peng, Y.; Zhang, G.; Zhang, M.; You, Z.; Liu, J.; Zhu, Q.; Yang, K.; Xu, X.; Geng, X.; and Yang, X. 2025. Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.07536

    2025 arXiv

  23. [31]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748--8763. PmLR

    2021

  24. [32]

    Sassioui, A.; Benouini, R.; El Ouargui, Y.; El Kamili, M.; Chergui, M.; and Ouzzif, M. 2023. Visually-Rich Document Understanding: Concepts, Taxonomy and Challenges. In 2023 10th International Conference on Wireless Networks and Mobile Communications (WINCOM), 1--7

    2023

  25. [33]

    Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y.; Wu, Y.; et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    2024 arXiv

  26. [34]

    Shen, H.; Liu, P.; Li, J.; Fang, C.; Ma, Y.; Liao, J.; Shen, Q.; Zhang, Z.; Zhao, K.; Zhang, Q.; Xu, R.; and Zhao, T. 2025. Vlm-r1: A stable and generalizable r1-style large vision-language model. arXiv preprint arXiv:2504.07615

    2025 arXiv

  27. [35]

    Sun, S.; Zhuang, S.; Wang, S.; and Zuccon, G. 2025. An investigation of prompt variations for zero-shot llm-based rankers. In European Conference on Information Retrieval, 185--201. Springer

    2025

  28. [36]

    Sun, W.; Yan, L.; Ma, X.; Wang, S.; Ren, P.; Chen, Z.; Yin, D.; and Ren, Z. 2023. Is ChatGPT good at search? investigating large language models as re-ranking agents. arXiv preprint arXiv:2304.09542

    2023 arXiv

  29. [37]

    Tanaka, R.; Nishida, K.; Nishida, K.; Hasegawa, T.; Saito, I.; and Saito, K. 2023. Slidevqa: A dataset for document visual question answering on multiple images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 13636--13645

    2023

  30. [38]

    Team, G.; Kamath, A.; Ferret, J.; Pathak, S.; Vieillard, N.; Merhej, R.; Perrin, S.; Matejovicova, T.; Ram \'e , A.; Rivi \`e re, M.; et al. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786

    2025 arXiv

  31. [39]

    Tito, R.; Karatzas, D.; and Valveny, E. 2023. Hierarchical multimodal transformers for multipage docvqa. Pattern Recognition, 144: 109834

    2023

  32. [40]

    Van Landeghem, J.; Tito, R.; Borchmann, .; Pietruszka, M.; Joziak, P.; Powalski, R.; Jurkiewicz, D.; Coustaty, M.; Anckaert, B.; Valveny, E.; et al. 2023. Document understanding dataset and evaluation (dude). In Proceedings of the IEEE/CVF International Conference on Computer ...

    2023

  33. [41]

    Wan, Y.; Liu, Y.; Ajith, A.; Grazian, C.; Hoex, B.; Zhang, W.; Kit, C.; Xie, T.; and Foster, I. 2024. SciQAG: A Framework for Auto-Generated Science Question Answering Dataset with Fine-grained Evaluation. arXiv preprint arXiv:2405.09939

    2024 arXiv

  34. [42]

    Wang, K.; Pan, J.; Shi, W.; Lu, Z.; Ren, H.; Zhou, A.; Zhan, M.; and Li, H. 2024. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37: 95095--95169

    2024

  35. [43]

    Wei, C.; Chen, Y.; Chen, H.; Hu, H.; Zhang, G.; Fu, J.; Ritter, A.; and Chen, W. 2024. Uniir: Training and benchmarking universal multimodal information retrievers. In European Conference on Computer Vision, 387--404. Springer

    2024

  36. [44]

    Xu, Z. 2024. RankMamba: Benchmarking Mamba's Document Ranking Performance in the Era of Transformers. arXiv preprint arXiv:2403.18276

    2024

  37. [45]

    Yang, Y.; He, X.; Pan, H.; Jiang, X.; Deng, Y.; Yang, X.; Lu, H.; Yin, D.; Rao, F.; Zhu, M.; et al. 2025. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503.10615

    2025 arXiv

  38. [46]

    Zhang, R.; Jiang, D.; Zhang, Y.; Lin, H.; Guo, Z.; Qiu, P.; Zhou, A.; Lu, P.; Chang, K.-W.; Qiao, Y.; et al. 2024 a . Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, 169--186. Springer

    2024

  39. [47]

    Zhang, X.; Zhang, Y.; Xie, W.; Li, M.; Dai, Z.; Long, D.; Xie, P.; Zhang, M.; Li, W.; and Zhang, M. 2024 b . GME: Improving Universal Multimodal Retrieval by Multimodal LLMs. arXiv preprint arXiv:2412.16855

    2024 arXiv

  40. [48]

    Zhao, Y.; Huang, J.; Hu, J.; Wang, X.; Mao, Y.; Zhang, D.; Jiang, Z.; Wu, Z.; Ai, B.; Wang, A.; Zhou, W.; and Chen, Y. 2024. SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning. arXiv:2408.05517

    2024 arXiv

  41. [49]

    Aha Moment

    Zhou, H.; Li, X.; Wang, R.; Cheng, M.; Zhou, T.; and Hsieh, C.-J. 2025. R1-Zero's" Aha Moment" in Visual Reasoning on a 2B Non-SFT Model. arXiv preprint arXiv:2503.05132

    2025 arXiv

  42. [50]

    Zhu, F.; Lei, W.; Feng, F.; Wang, C.; Zhang, H.; and Chua, T.-S. 2022. Towards complex document understanding by discrete reasoning. In Proceedings of the 30th ACM International Conference on Multimedia, 4857--4866

    2022

  43. [51]

    Zhuang, H.; Qin, Z.; Hui, K.; Wu, J.; Yan, L.; Wang, X.; and Bendersky, M. 2023 a . Beyond yes and no: Improving zero-shot llm rankers via scoring fine-grained relevance labels. arXiv preprint arXiv:2310.14122

    2023 arXiv

  44. [52]

    Zhuang, S.; Liu, B.; Koopman, B.; and Zuccon, G. 2023 b . Open-source large language models are strong zero-shot query likelihood models for document ranking. arXiv preprint arXiv:2310.13243

    2023 arXiv

  45. [53]

    Zhuang, S.; Ma, X.; Koopman, B.; Lin, J.; and Zuccon, G. 2025. Rank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learning. arXiv:2503.06034

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.