{"id":"74dc7cab-2660-410b-aba2-df44a86825a3","arxiv_id":"2509.07666","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MoLoRAG boosts multi-page document QA by traversing a page graph and scoring pages with both semantic and VLM-judged logical relevance, improving retrieval and answer accuracy on four benchmarks.","lead":"This paper introduces MoLoRAG, a retrieval system that combines page-level embeddings with a vision-language model to find relevant pages in long multi-page documents for question answering. It reports accuracy gains over direct LVLM inference and earlier retrieval baselines on four document QA benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MoLoRAG+ is fine-tuned on page snapshots sampled from the same MMLongBench and LongDocURL datasets used for retrieval evaluation; without a document-disjoint split, the fine-tuned gains in Tables 2-3 may reflect page memorization rather than logic-aware retrieval.","rationale":"The reader's verdict was CONDITIONAL and I agree, but for a different reason than the reader's weakest_assumption. The reader focused on graph connectivity, asking whether evidence pages can be reached within w=3 and nhop=4 from the semantic seed set. That is a legitimate risk, and the ablation in Table 8 already shows MoLoRAGFull, which replaces graph traversal with exhaustive VLM scoring, slightly outperforms MoLoRAG+ in recall, suggesting the graph is an efficiency tradeoff rather than an accuracy contributor. However, the more directly load-bearing issue is the apparent train/eval overlap for the fine-tuned variant. The paper never states that training snapshots are drawn from documents excluded from the evaluation sets. Since the data are sampled from exactly the two benchmarks used for retrieval evaluation in Section 4.3, the natural reading is that pages from evaluation documents can appear in the training set. This is not an ad hominem point; it is a missing control that is standard in benchmark reporting. The concrete overlap check is straightforward because the dataset is released. If overlap exists, the fine-tuned variant's claimed further performance gains in Section 4.2 and its retrieval improvements in Table 3 are inflated, and the paper's central claim should be re-scoped to the training-free MoLoRAG. If overlap is absent, the concern is resolved and only the reader's graph-connectivity and error-bar issues remain. I therefore keep the CONDITIONAL verdict, conditional on either demonstrating zero overlap or re-running the fine-tuned experiments on a document-disjoint split.","tokens_in":28495,"tokens_out":7365,"duration_ms":65546,"concrete_test":"Compare the released training set (HuggingFace dataset xxwu/MoLoRAG, 3,519 snapshots) with the MMLongBench and LongDocURL evaluation splits. For each training image, check whether it is a page of a document that appears among the test queries, or is itself an evidence page. If any overlap exists, retrain MoLoRAG+ on a document-disjoint split, for example 50% of documents for training and held-out documents for evaluation, and recompute Table 3. If MoLoRAG+'s top-3 MMLongBench Recall drops from 68.87 to at or below MoLoRAG's 67.22, with a similar drop on LongDocURL, the fine-tuning advantage over the training-free variant is not substantiated and should be reported as such. If the overlap check is empty, the concern is resolved and only the reader's graph-connectivity and error-bar issues remain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix C states that the 3,519 training snapshots for MoLoRAG+ are 'randomly selected from MMLongBench and LongDocURL.' Section 4.3 then reports retrieval performance on the same two benchmarks. No document-level or page-level split between training and evaluation is described anywhere in the paper. If the sampled snapshots come from the same documents or pages as the evaluation queries, the fine-tuned retriever has been trained on the very pages it is later asked to score, so the 1-5 logical relevance predictions can encode memorized visual content. This directly threatens the stronger rows of Table 3, for example MoLoRAG+ top-3 Recall 68.87 versus MoLoRAG 67.22 on MMLongBench and top-1 Recall 51.32 versus 45.46, and the corresponding QA gains in Table 2. It likely also contributes to the abstract's claimed average improvements. The training-free MoLoRAG numbers are not affected by this particular flaw, but the paper's two-variant story and its 'logic-aware' fine-tuning contribution are not cleanly supported until the overlap is ruled out. The graph-connectivity issue raised by the reader is real, but it is secondary: even if graph traversal is sound, the fine-tuned results remain uninterpretable under this overlap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MoLoRAG, a two-stage retrieval-augmented framework for multi-page document QA. In the retrieval stage, a page graph is built from ColPali embedding similarities; a lightweight VLM assigns a 1–5 'logical relevance' score to each visited page while traversing the graph, and these scores are combined with semantic scores to re-rank pages. The top-K pages are then passed to an arbitrary LVLM for answer generation. A fine-tuned variant, MoLoRAG+, is trained on GPT-4o-generated triplets to improve logical-relevance scoring. Experiments on MMLongBench, LongDocURL, PaperTab, and FetaTab report consistent QA and retrieval gains over M3DocRAG, MDocAgent, and direct LVLM inference, with additional ablations for the logical-only variant and the full-traversal variant.","tokens_in":28806,"tokens_out":2900,"duration_ms":26126,"significance":"If the claims hold, MoLoRAG is a useful contribution: it extends graph-RAG ideas to multimodal page retrieval, offers a training-free variant that is easy to deploy, and releases code, model weights, and a training dataset. The central idea of combining semantic similarity with VLM-assigned logical relevance is plausible and the paper provides a concrete instantiation with ablations. The empirical scope is broad (four datasets, multiple LVLMs) and the paper includes useful analyses of efficiency and fine-grained modality performance. However, the significance is currently conditional on resolving the train/evaluation overlap for MoLoRAG+, the truncated direct baseline, and the lack of uncertainty quantification.","major_comments":[{"comment":"The fine-tuned variant MoLoRAG+ is trained on 3,519 page snapshots 'randomly selected from MMLongBench and LongDocURL' and then evaluated on the same two benchmarks, but the paper does not describe any document-level or page-level split between training and evaluation. If the sampled snapshots come from the same documents or pages as the evaluation queries, the reported gains (e.g., MoLoRAG+ top-3 Recall 68.87 versus MoLoRAG 67.22 on MMLongBench, and top-1 Recall 51.32 versus 45.46) may reflect memorized visual content rather than improved logical-relevance reasoning. This directly affects the strongest rows of Table 3 and the corresponding QA improvements in Table 2. The authors should either use a document-disjoint split for fine-tuning and evaluation, or explicitly demonstrate that no overlap exists between the training snapshots and the evaluation pages.","section":"Appendix D.2 and Table 2"},{"comment":"The 'LVLM Direct' baseline truncates documents to the first 30 pages, while LongDocURL has an average of 85.6 pages per document and the case studies in Figures 5 and 8 place evidence pages at indices 49 and 51. Truncation therefore handicaps the direct baseline on long documents, making the 'Direct' comparisons in Table 2 and the claimed average improvements over direct inference difficult to interpret. The authors should compare against direct inference that processes the full document where the LVLM context allows it, or clearly report the percentage of LongDocURL questions whose evidence pages fall outside the 30-page window.","section":"Tables 2 and 3"},{"comment":"All reported numbers are single-run point estimates without error bars, multiple seeds, or significance tests. Some differences that support the paper's claims are small—for example Qwen2.5-VL-3B on MMLongBench: MoLoRAG 32.11 versus MoLoRAG+ 32.47 in Table 2, and LongDocURL top-3 Recall 70.04 versus 68.92 in Table 3. Without variance estimates, it is not possible to tell whether these differences are meaningful. The authors should report at least three independent runs with standard deviations, or a paired significance test, for the main retrieval and QA results.","section":"Appendix D.6 and Table 8"},{"comment":"The ablation comparing MoLoRAG+ with MoLoRAGFull shows that traversing all pages (instead of the graph-constrained candidate set) improves Recall@3 from 68.87 to 73.64 on MMLongBench and from 68.92 to 72.30 on LongDocURL. The paper attributes this to a 'marginal performance difference' and emphasizes the time saving, but a 4–5 point recall gain is substantial and suggests that the graph construction is dropping relevant pages. The claim that the graph provides a 'high-quality candidate set' is therefore not fully supported. The authors should provide a cost-benefit analysis (e.g., recall versus number of queried pages) that shows the graph-based trade-off is actually favorable, or qualify the claim that the graph preserves accuracy.","section":"Section 3.2 and Appendix D.2"},{"comment":"Several load-bearing design choices are fixed without any sensitivity analysis: the edge threshold theta = 0.4, exploration set size w = 3, hop limit nhop = 4, and the equal weighting in the Combine(·) function. The graph-connectivity assumption is central to the method: if an evidence page is not within a few hops of the initial semantic set, traversal cannot find it. The authors should at least report retrieval performance over a range of theta and w/nhop values on MMLongBench, and justify the chosen weights rather than simply using the average.","section":"Section 4.3 and Table 3"}],"minor_comments":[{"comment":"The abstract reports average improvements of 9.68% in accuracy and 7.44% in retrieval precision, while Section 4.3 reports 9.94% on MMLongBench and 7.16% on LongDocURL. These numbers should be reconciled or the abstract should cite the specific table.","section":"Abstract and Section 4.3"},{"comment":"In Table 1, FetaTab's average tokens are printed as '16,524.5' but the formatting suggests a possible typo ('16.524.5'); please verify.","section":"Table 1"},{"comment":"The naming of the logical-only variant is inconsistent: 'MoLoRAGLogi' in Table 7, 'MoLoRAGlogi' in the text of D.6. Please unify.","section":"Section D.6"},{"comment":"The MRR definition is unclear: the formula as written states MRR@K = 1 if p_i is the first relevant page and 0 otherwise, but the text then says 'where i denotes the position of the first relevant page'. The formula should be MRR@K = 1/i with i being that position, otherwise the metric is mis-specified.","section":"Appendix D.1"},{"comment":"The example in Figure 1 contains repeated identical thumbnails of '24 Adams Street Combined Sewer System Performance for a Typical Year', which is visually confusing; please clarify which page is the ground-truth evidence page and which is the retrieved page.","section":"Figure 1"},{"comment":"The notation for the logical relevance score uses both s_logi and s^logi_i; please use a single consistent notation throughout.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First: this is a genuinely useful retrieval framework for multi-page DocQA, and the training-free variant MoLoRAG is well supported. The fine-tuned MoLoRAG+ is not, and the stress-test note is right: Appendix C randomly selects training snapshots from the same MMLongBench and LongDocURL documents that Table 3 evaluates on, with no document-level split. That makes the MoLoRAG+ rows in Tables 2–3 (e.g., top-1 Recall 51.32 vs 45.46 on MMLongBench) uninterpretable until overlap is ruled out. This is a load-bearing problem for the fine-tuning story.\n\nWhat is new: the page graph plus VLM traversal for logical relevance is a clean combination not present in M3DocRAG or MDocAgent. The paper shows it works: training-free MoLoRAG beats those baselines on retrieval and downstream accuracy for several LVLMs, and the ablations (MoLoRAGlogi vs combined, MoLoRAGFull vs graph) support the design choices. The compatibility across weak and strong LVLMs, and the efficiency numbers, add practical value.\n\nSoft spots beyond the overlap: no error bars anywhere; hyperparameters (theta=0.4, w=3, nhop=4, equal weights) are set without sensitivity analysis; the 'logical relevance' is an unvalidated 1–5 VLM score, so we don't know how well it reflects true relevance; and the direct baseline is truncated at 30 pages, which favors retrieval methods on longer documents. The graph-connectivity assumption the reader flagged is real but secondary—Table 8 suggests full traversal barely beats graph traversal, so the graph is not the weak point.\n\nVerdict: the training-free contribution deserves a serious referee and likely publication after revision. The fine-tuned variant needs a document-disjoint split, or the MoLoRAG+ claims should be withdrawn. I'd send this to peer review and ask the authors to fix the overlap and add variance/robustness checks. The paper is honest and well-written; this is a fixable flaw, not a fraudulent one.","headline":"A useful training-free retrieval module for multi-page DocQA, but the fine-tuned variant's gains are compromised by training/eval overlap.","tokens_in":29310,"tokens_out":2150,"would_cite":true,"duration_ms":19194,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Graph-walking VLM finds evidence pages that semantic search misses","keywords":["document question answering","multi-modal retrieval","page graph","logical relevance","graph traversal","vision-language model","retrieval-augmented generation","multi-page documents"],"falsifier":"Construct a split of MMLongBench and LongDocURL where every ground-truth evidence page is more than n_hop=4 graph hops away from the top-3 semantic pages at θ=0.4. If MoLoRAG still retrieves those evidence pages, graph traversal is not what supplies them; if it fails, the load-bearing assumption is confirmed. A complementary check is to measure whether the VLM's 1-5 score discriminates ground-truth evidence pages from non-evidence pages at better than chance.","tokens_in":28314,"feed_emoji":"📄","tokens_out":8016,"duration_ms":61561,"temperature":0.7,"pith_summary":"The paper argues that document question answering over long, image-heavy documents fails when retrieval looks only at semantic similarity between the query and each page. Its proposed answer, MoLoRAG, builds a graph of pages whose embeddings are similar and lets a lightweight vision-language model walk that graph, giving each visited page a logical-relevance score from 1 to 5. The final ranking combines semantic and logical scores, and the top pages are passed to any LVLM for answering. On four DocQA datasets the paper reports that this consistently beats semantic-only retrievers, with average retrieval improvements of 9.94 percent on MMLongBench and 7.16 percent on LongDocURL, and that retrieval gains carry through to answer accuracy. The underlying point is that reasoning questions often need evidence pages that are logically connected to the question rather than merely keyword-similar.","feed_headline":"Graph-walking VLM finds evidence pages semantic search misses","feed_subtitle":"MoLoRAG's page graph and 1-5 logical relevance scores lift retrieval and answer accuracy on long documents.","key_machinery":"The load-bearing mechanism is the page graph G(V,E) constructed from ColPali page embeddings, with an edge when the inner product exceeds threshold θ. Graph traversal, seeded by the top-w semantic pages and limited by a hop count, lets the retrieval engine reach pages that are not semantically similar to the query but are connected to similar pages; the VLM's 1-5 logical-relevance score, combined with the semantic score, decides which pages survive the re-ranking. The graph supplies the candidate paths, the VLM supplies the new logical signal, and the two hyperparameters w and n_hop keep the search from visiting every page.","core_discovery":"MoLoRAG's central claim is that a page graph plus a VLM-assigned logical relevance score retrieves the evidence pages needed to answer multi-page document questions, where semantic retrieval alone returns pages containing query keywords but lacking the information. The method encodes each page with ColPali, adds an edge when embedding similarity exceeds a threshold, initializes an exploration set from the top semantically similar pages, and then iteratively scores neighbors with a VLM prompt (1-5), combines that score with the semantic score, and expands the set until the hop limit is reached. All visited pages are re-ranked by the combined score, and the top-K snapshots are sent to an LVLM. The paper reports that this consistently outperforms baselines, with an average improvement of 9.94 percent on MMLongBench and 7.16 percent on LongDocURL in retrieval metrics, and that a fine-tuned variant MoLoRAG+ improves further by making the logical-relevance judgments more accurate.","pith_inferences":["Beyond the paper: if logical relevance is the bottleneck, then the page graph's edge construction deserves at least as much attention as the scorer; replacing the similarity threshold with structural edges from tables of contents, hyperlinks, or repeated entities should raise recall without retraining.","The discrete 1-5 VLM score creates ties, and the paper itself notes that equal scores make ranking hard; a continuous regression head or pairwise preference model trained on the released triplets is a natural extension.","A testable extension is threshold sensitivity: recall as a function of θ should show a knee, and per-document adaptive thresholds could preserve logical edges at the cost of more VLM calls.","The paper's limitation section notes it targets closed-domain documents; carrying the same idea to open-domain corpora would require cross-document graphs, which the current traversal does not model."],"forward_implications":["MoLoRAG is compatible with arbitrary LVLMs: once retrieval is done, only the top-K page snapshots are passed in, so single-image models can be used on multi-page documents.","Retrieval gains are largest on questions whose evidence page is logically, not lexically, related to the query; the paper's case studies show such questions are precisely where semantic-only baselines fail.","The training-free variant is deployable without extra data, while the fine-tuned MoLoRAG+ variant provides a further accuracy boost by improving logical relevance scoring.","On long documents the traversal queries fewer than half the pages on average, so the method scales to documents too large for direct LVLM input."],"supporting_citations":[{"why":"Supplies the ColPali document encoder that produces the page embeddings used for graph edges and semantic relevance scores.","marker":"Faysse et al., 2024"},{"why":"Defines M3DocRAG, the semantic-only page retrieval baseline that MoLoRAG is compared against.","marker":"Cho et al., 2024"},{"why":"Defines MDocAgent, the multi-agent baseline whose semantic-only retrieval fails on logically relevant evidence pages.","marker":"Han et al., 2025"},{"why":"Provides the MMLongBench benchmark and its rule-based accuracy evaluation protocol, and motivates concatenating images for single-image LVLMs.","marker":"Ma et al., 2024b"},{"why":"Provides LongDocURL, the benchmark with ground-truth evidence pages and cross-page reasoning questions used for retrieval metrics.","marker":"Deng et al., 2024"},{"why":"Provides the UDA benchmark that supplies the PaperTab and FetaTab datasets for QA evaluation.","marker":"Hui et al., 2024"},{"why":"Supplies Qwen2.5-VL, the lightweight VLM used as the retrieval engine and fine-tuned in MoLoRAG+.","marker":"Bai et al., 2025"},{"why":"Provides GPT-4o, the model that generates and quality-checks the logical-relevance training triplets for MoLoRAG+.","marker":"OpenAI, 2024"}],"fun_headline_variants":["Page-graph retrieval finds evidence semantic search skips","VLM-guided page traversal beats keyword-only retrieval","MoLoRAG adds logical links to multi-page document retrieval","Multi-modal RAG that follows document logic, not just text","Page graph + VLM logic scores answer multi-page questions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the page graph, built from embedding similarity above a fixed threshold, links semantically similar pages to the logically relevant evidence pages within a few hops, and that a VLM's one-to-five relevance score recognizes those evidence pages when it sees them.","fun_headline_variants_meta":{"raw":{"variants":["Page-graph retrieval finds evidence semantic search skips","VLM-guided page traversal beats keyword-only retrieval","MoLoRAG adds logical links to multi-page document retrieval","Multi-modal RAG that follows document logic, not just text","Page graph + VLM logic scores answer multi-page questions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000418,"raw_usage":{"total_tokens":2193,"prompt_tokens":1021,"completion_tokens":1172,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":637,"completion_tokens_details":{"reasoning_tokens":1092}},"tokens_in":637,"tokens_out":1172,"duration_ms":8706,"temperature":1.0,"reasoning_tokens":1092,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:22:32.673244+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a split of MMLongBench and LongDocURL where every ground-truth evidence page is more than n_hop=4 graph hops away from the top-3 semantic pages at θ=0.4. If MoLoRAG still retrieves those evidence pages, graph traversal is not what supplies them; if it fails, the load-bearing assumption is confirmed. A complementary check is to measure whether the VLM's 1-5 score discriminates ground-truth evidence pages from non-evidence pages at better than chance.","supporting_citations":[],"review_version":2}