{"id":"f8d45ce2-fc8d-47b9-9676-a786e05a1a49","arxiv_id":"2506.17589","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A multimodal knowledge graph benchmark for Monster Hunter: World, plus a training-free multi-agent graph retriever, improves MLLM accuracy on rare-domain questions from about 0.31 to 0.51 for GPT-4o.","lead":"This paper builds a multimodal knowledge graph for the game Monster Hunter: World and a 238-question benchmark, then shows that a multi-agent, training-free retrieval pipeline helps multimodal LLMs answer rare-domain questions. The testbed is designed to measure whether external structured knowledge can compensate for what LLMs do not know.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Knowledge leakage is acknowledged but never quantified across sub-tasks; a text-only control would determine if retrieval gains are causal.","rationale":"The reader's weakest assumption identifies benchmark knowledge leakage, and the paper's own text supports this concern: footnote 2 concedes built-in knowledge about Monster Hunter, and Section 5.2 reports sub-task VI answers being correct while path precision/recall are low, meaning those answers rely on inherent knowledge. This is load-bearing because the central claim is that graph retrieval enables MLLMs to answer questions they otherwise cannot; if leakage is substantial, the measured gains partly reflect memorized knowledge rather than retrieval capability. I considered alternative concerns such as the lack of mRAG baselines and the path-count hyperparameter tuned on the test set, but those are secondary: the paper positions its method as a baseline for a new benchmark, and the hyperparameter issue affects the magnitude of the gain, not the core attribution. The proposed concrete tests directly settle the attribution question: a text-only control isolates built-in knowledge from perception and retrieval, and a random-path control tests whether the specific retrieved subgraph content causes the accuracy improvement rather than the mere presence of additional text or prompt structure. My read does not change the reader's conditional verdict; the concern is real but addressable, and no additional adjustment is needed.","tokens_in":22541,"tokens_out":3206,"duration_ms":33019,"concrete_test":"Run two controls on all 238 queries with GPT-4o and Claude 3.7: (1) Text-only vanilla: feed q and auxiliary information z (monster name and extra hints) with no images and no KG, measuring accuracy; compare to vanilla and vanilla+ numbers in Table 3. If text-only accuracy approaches the perceptive or unaided accuracy, built-in knowledge leakage dominates. (2) Random-path control: in the unaided-online pipeline, replace the retrieved paths with an equal number of randomly sampled paths from MH-MMKG (or from a different monster's subgraph), keeping the same summarizer prompt; if accuracy stays near 0.51 rather than dropping toward vanilla 0.31, the specific retrieved content is not causal. Report results per sub-task, especially sub-task VI.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that training-free graph retrieval significantly enhances MLLM performance on rare-domain questions, which presumes the benchmark isolates external knowledge. That premise is insecure: the authors concede in footnote 2 that 'current MLLMs have already learned some knowledge about it' and report in Section 5.2 that sub-task VI answers are correct while retrieved-path precision/recall are low, i.e., answers come from built-in memory. No control quantifies this leakage for the other five sub-tasks, so the accuracy gap between vanilla (0.31 for GPT-4o) and unaided-online (0.51) may mix retrieval benefit with memorized game knowledge activated by the query's auxiliary information z (monster name plus extra hints). The pipeline itself is also leakage-prone: the expansion prompt instructs the model to 'always choose' attack-action neighbors, and the summarizer receives candidate paths, which can cue the model's own knowledge rather than demonstrate retrieval. Without a text-only baseline (q and z, no images, no KG) or a random-path control, the 'significantly enhances' claim is not uniquely attributable to graph-based retrieval.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MH-MMKG, a multimodal knowledge graph for the game Monster Hunter: World, containing entities for monsters, attacks, phases, elements, weapons, props, and effects, along with text, images, video (keyframes and captions) attributes. The authors also build a benchmark of 238 query-answer pairs across six sub-tasks (individual information, attack recognition, combo premonition, condition awareness, proc effect insight, cross-monster analysis), with ground-truth subgraphs for retrieval evaluation. They propose a training-free multi-agent retriever baseline: a perceiver converts visual input to text, a topic selection agent finds the root entity, expansion and validation agents conduct a BFS-like search over the KG, and a summarizer generates the final answer from the top α paths. Experiments on 13 MLLMs compare vanilla, vanilla+ (human captions), knowledgeable (perfect retrieval), perceptive, and unaided-offline/online settings, reporting accuracy and knowledge-consistency precision/recall. The main empirical finding is that the multi-agent retrieval improves accuracy (e.g., GPT-4o from 0.31 to 0.51 in unaided-online), and the knowledgeable setting reaches ~0.9, suggesting the KG covers the needed knowledge. The paper also includes ablations on keyframes, number of paths, BFS/DFS, and cross-model agents, plus a GPT-4o-as-judge human validation.","tokens_in":22714,"tokens_out":7575,"duration_ms":68822,"significance":"The paper's strongest contributions are the benchmark itself and the reproducible infrastructure: the dataset/code is released, the MH-MMKG combines multiple modalities and rich relation types, and the six sub-tasks are more fine-grained than typical VQA benchmarks. The knowledgeable-setting results (≈0.9 across models) demonstrate that the KG contains the information needed for the benchmark, and the multi-agent retriever is a reasonable, training-free baseline that improves over vanilla in several models. The ablations in the supplementary (cross-model agent swaps, BFS/DFS, path count) are useful for the community. However, the central quantitative claim that the proposed retrieval 'significantly enhances performance' is not yet established because the benchmark does not isolate external knowledge from built-in memory, the path-count hyperparameter is tuned on the test set, and no statistical significance tests are provided. If the attribution issue is resolved with appropriate controls, this would be a solid contribution to multimodal knowledge-augmented reasoning; in its current form the evaluation overclaims.","major_comments":[{"comment":"The central claim that graph retrieval causes the accuracy gains is not established because built-in knowledge leakage is acknowledged but never quantified. Footnote 2 states 'current MLLMs have already learned some knowledge about it,' and Section 5.2 reports for sub-task VI that models answer correctly while retrieved-path precision/recall are low, indicating reliance on inherent knowledge. The 0.31 to 0.51 improvement for GPT-4o may therefore partly reflect the model's own game knowledge activated by the auxiliary information z rather than the retrieved knowledge. To support the causal claim, the paper should report a text-only baseline (q and z, no images, no KG) and a random-path control for all sub-tasks; without these, the 'significantly enhances' conclusion is not uniquely attributable to graph-based retrieval.","section":"Section 5.2, Table 3, footnote 2"},{"comment":"The number of retrieval paths α is tuned on the test set. The paper states 'All evaluations so far used 5 paths' and Figure 6a shows that GPT-4o gives optimal performance when 5 paths are used on the benchmark; this test-set selection may overstate the reported accuracies and the cross-model comparisons. Please fix α on a validation split (or report results across a range of α for all models) and state the selection procedure explicitly.","section":"Section 5.3, Figure 6a"},{"comment":"No error bars or statistical significance tests are reported, despite the stochasticity of API and open-source MLLMs and the modest benchmark size (238 samples). Moreover, for several models the unaided-online setting is worse than unaided-offline (e.g., MiniCPM-o-2.6 accuracy 0.1687 vs 0.3459; DeepSeek-VL2 recall 0.0042 vs 0.0759), which is in tension with the claim that online captioning improves performance. Please provide variance over multiple runs, significance tests for the key comparisons, and qualify the 'significantly enhances' phrasing accordingly.","section":"Section 5.2, Table 3"},{"comment":"The expansion-agent prompt biases retrieval by instructing the model to 'always choose' attack-action neighbors unless highly irrelevant. Since most benchmark questions concern attacks, this instruction trivially boosts knowledge-consistency precision/recall on those sub-tasks and may inflate the measured retrieval quality. Please ablate this instruction or report its effect on both consistency and accuracy.","section":"Section 4.1, Supplementary Table 10"}],"minor_comments":[{"comment":"The notation G=(E,V,R) uses V for the edge set and R for the relation set, conflicting with the common (V,E) vertex/edge convention; consider renaming to avoid confusion (e.g., N for nodes, E for edges).","section":"Section 3.1"},{"comment":"There is a missing space in the GPT-4o online row: '.5625.2948' should be '.5625 .2948'.","section":"Table 4"},{"comment":"Reference [45] is duplicated in the citation list '[45, 45, 48]'.","section":"Section 2.3"},{"comment":"The sentence 'all methods hardly predicted correct answers' is inaccurate given GPT-4o's vanilla accuracy of 0.3122; consider 'well below the knowledgeable human (0.5252)' instead.","section":"Section 5.2"},{"comment":"Typo: 'Gemeni' should be 'Gemini'.","section":"Supplementary Table 18"},{"comment":"The legend labels 'Pre.' and 'Re.' are abbreviated and could be confused with 'Recall'; spell out 'Precision' and 'Recall'.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and code release are valuable; the main issue is the attribution of the gains to retrieval. A text-only control and a proper validation split for α would substantially strengthen the paper. The paper also has a few table/label inconsistencies (e.g., the human row in Table 3 appears to have fewer entries than the column headers) that the authors should check."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing you should know: this is primarily a dataset paper. They assembled MH-MMKG, a multimodal KG for Monster Hunter: World with 22 monster subgraphs, video-derived keyframes, captions, and 158 relation types, plus a 238-question benchmark with six sub-tasks. That is a real artifact. No existing MMKG benchmark covers a domain this far from web-text, and the visual gap is genuinely large. They also include a training-free multi-agent retriever as a baseline. It is an assembly of known components—topic selection, expansion, validation, summarization—but the application to a video-derived MMKG is new, and the knowledgeable variant reaching ~0.9 accuracy shows the KG contains enough information to answer the queries. That is useful.\n\nThe paper is also honest in spots. It admits in footnote 2 that MLLMs already know some Monster Hunter, and in Section 5.2 it acknowledges sub-task VI is answered from inherent knowledge when retrieval fails. The supplementary includes human evaluation of the GPT-4o judge, ablations on path count, keyframe selection, BFS versus DFS, and prompt robustness. That is more than many benchmark papers do.\n\nThe soft spots are real but not fatal. First, the abstract claims the approach 'significantly enhances' performance, but there are no error bars on the 238-question benchmark and no significance tests. A 0.31-to-0.51 jump on GPT-4o looks meaningful, but per-subtask samples are as small as 13, and the reported precision/recall swings are large. Second, the leakage concern is acknowledged but never quantified. Sub-task VI is the obvious case, and the stress-test note is right that a text-only control—q and auxiliary info, no images, no KG—would clarify how much of the gain is retrieval versus memorized knowledge triggered by the query. Third, alpha, the number of paths, is tuned on GPT-4o's test-set performance and then used for all models; they even show the curve. It is minor but should be a held-out selection. Fourth, there is no comparison to existing mRAG baselines, so the 'significantly enhances' claim is really 'the pipeline helps', not 'the pipeline beats alternatives'. The expansion prompt also biases toward attack-action neighbors, which helps recall on a benchmark dominated by attack questions—that is a fair design choice, but it should be stated as a bias.\n\nWho is this for? People building multimodal knowledge-augmented benchmarks, and teams working on game-VQA or rare-domain retrieval. It deserves a serious referee—the artifacts are reusable and the issues are fixable. I would ask for a leakage-control experiment and error bars before accepting, but I would not desk-reject it.","headline":"A real new benchmark with a useful baseline, but the headline claim needs leakage controls and significance tests before it can be taken at face value.","tokens_in":23280,"tokens_out":2572,"would_cite":true,"duration_ms":26748,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multimodal language models can answer rare, domain-specific questions by autonomously searching a structured multimodal knowledge graph; this paper builds a Monster Hunter testbed that demonstrates the approach.","keywords":["multimodal knowledge graphs","retrieval-augmented generation","multimodal large language models","multi-agent retrieval","visual question answering","Monster Hunter: World","knowledge graph reasoning","training-free retrieval"],"falsifier":"Replace the retrieved subgraph in the reasoning step with a same-sized subgraph taken from a different monster or from random paths, holding everything else fixed; if accuracy stays near the augmented level, the retrieval step is not what carries the answer. A cheaper observation is to compute accuracy on the subset of questions where the model retrieved zero correct paths: the paper already reports this happens in sub-task VI with high accuracy, and if that subset is large across all sub-tasks, the graph is adding less than the headline comparison suggests.","tokens_in":22328,"feed_emoji":"🎮","tokens_out":7658,"duration_ms":75506,"temperature":0.7,"pith_summary":"This paper argues that multimodal large language models (MLLMs) can handle rare, domain-specific questions when they are given a structured external knowledge graph to search, and it builds a complete testbed to demonstrate that claim. The testbed is Monster Hunter: World, a video game whose imaginary world and computer-generated visuals sit far outside typical training data. The authors construct MH-MMKG, a multimodal knowledge graph of monsters, attacks, phases, combos, and conditions with text, images, video clips, and captions curated by expert players, plus 238 question-answer pairs across six sub-tasks. They then show that a multi-agent retriever, using the same MLLM as all agents and no additional training, raises accuracy well above the model's unaided baseline by finding relevant paths in the graph. A careful reader should care because the work offers a training-free route to grounding MLLMs in specialized knowledge and a benchmark for studying knowledge retrieval apart from built-in knowledge.","feed_headline":"AI models answer rare game questions by searching a knowledge graph","feed_subtitle":"A Monster Hunter graph lifts GPT-4o from 31% to 51% accuracy without fine-tuning.","key_machinery":"The load-bearing mechanism is the multi-agent retriever operating over MH-MMKG. MH-MMKG is an attribute-based multimodal knowledge graph in which monsters and attack actions are entities linked by relation types such as 'has attack action of' and 'continues with attack action of,' many of them carrying conditions like 'when angry,' and each entity can carry video, human-selected keyframes, a human-written caption, and textual context. The retriever starts with a topic-selection agent that picks the root monster, then alternates an expansion agent, which proposes neighboring entities that could help answer the query, and a validation agent, which decides whether the knowledge accumulated along the current path is sufficient. The same MLLM plays the perceiver, the three agents, and the summarizer through different prompts, so no training or retriever embedding is needed; the summarizer converts the retrieved subgraph paths into text and answers the question.","core_discovery":"The paper's central claim is that an MLLM can answer questions about a rarely seen domain by autonomously walking a structured multimodal knowledge graph, and that this retrieval-based route outperforms relying on the model's built-in knowledge. It demonstrates the claim on Monster Hunter: World through MH-MMKG, an attribute-based knowledge graph built by experienced players, in which monsters, attacks, phases, elements, and effects are entities connected by relation edges with conditions, and entities carry text, images, video, and human-written captions. On the 238-question benchmark, the proposed training-free multi-agent retriever—a topic-selection agent, an expansion agent, and a validation agent—finds relevant subgraphs and lifts answer accuracy over the vanilla MLLM baselines; when models are given the ground-truth subgraph, accuracy reaches roughly 0.9, showing the graph itself contains enough knowledge to answer. The paper also reports that generating captions online during retrieval helps, that BFS outperforms DFS for these queries, and that retrieval quality matters more than summarizer strength in the ablation.","pith_inferences":[],"forward_implications":["Models that can search a structured multimodal graph should handle rare, domain-specific questions in other low-resource domains without fine-tuning, as long as an expert-curated graph exists.","Because the ablation shows retrieval quality, not summarizer strength, drives the gains, improving the expansion and validation agents should transfer to better answers across model families.","The BFS-over-DFS result implies many domain questions are answerable in a few graph hops, so shallow graph traversal may be enough for practical retrieval-augmented systems.","Online captioning, where the model sees the question and current path while describing visual evidence, raises both caption quality and final accuracy, pointing to retrieval-aware perception as a useful direction.","The dataset and baseline can serve as a reusable testbed for comparing knowledge retrieval methods on a genuinely unfamiliar visual and knowledge domain.","Editorial inference: the recipe likely transfers to other closed-world domains—other games, synthetic environments, or specialized manuals—where models lack built-in knowledge but an expert-curated multimodal graph is feasible; the limiting cost is graph construction, not retrieval.","Editorial inference: the paper's own sub-task VI observation, where models answer correctly while retrieving no correct paths, implies the headline accuracy numbers mix retrieval and memorization; a cleaner evaluation would report retrieval-conditioned accuracy, and a causal ablation that deletes correct graph paths would isolate the retrieval contribution.","Editorial inference: the supplementary finding that the validation agent is prompt-sensitive and that replacing it with a weaker model hurts accuracy suggests future work should focus on making stopping decisions robust, perhaps with calibrated confidence instead of a binary 'sufficient' judgment."],"supporting_citations":[{"why":"Defines knowledge graphs as structured knowledge storage, motivating graph-based retrieval over unstructured text.","marker":"[24]"},{"why":"Provides the attribute-based multimodal knowledge graph formulation that MH-MMKG instantiates.","marker":"[60]"},{"why":"Supplies the self-searching and route-planning idea over knowledge graphs that the multi-agent retriever extends to multimodal data.","marker":"[45]"},{"why":"Shows a training-free way to adapt knowledge graphs as references for LLM reasoning, the basis for the no-training retriever.","marker":"[16]"},{"why":"GPT-4o is the main agent backbone and judge; its vanilla-to-augmented accuracy gap is the paper's headline demonstration.","marker":"[1]"},{"why":"Claude models serve as backbone agents and baselines, showing the retrieval benefit across a second model family.","marker":"[2]"},{"why":"Gemini models serve as additional closed-source baselines for the retrieval experiments.","marker":"[47]"},{"why":"Provides the LLM-as-judge evaluation protocol used to score open-ended answers and caption similarity.","marker":"[59]"},{"why":"Motivates the game domain as a testbed where MLLMs' built-in perception and knowledge are weak.","marker":"[9]"}],"fun_headline_variants":["Graph search lifts MLLM accuracy from 31% to 51% on rare queries","Autonomous knowledge graph retrieval boosts MLLM answers in rare domains","Training-free graph walk helps MLLMs answer unfamiliar questions","Monster Hunter graph guides MLLMs from 31% to 51% without fine-tuning","Multimodal graph retrieval enhances MLLM reasoning on unknown tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the questions cannot be answered from the models' built-in knowledge, so any accuracy gain can be credited to graph retrieval; the paper concedes that current MLLMs already know some Monster Hunter content and that some correct answers occur without any correctly retrieved path.","fun_headline_variants_meta":{"raw":{"variants":["Graph search lifts MLLM accuracy from 31% to 51% on rare queries","Autonomous knowledge graph retrieval boosts MLLM answers in rare domains","Training-free graph walk helps MLLMs answer unfamiliar questions","Monster Hunter graph guides MLLMs from 31% to 51% without fine-tuning","Multimodal graph retrieval enhances MLLM reasoning on unknown tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000384,"raw_usage":{"total_tokens":2024,"prompt_tokens":928,"completion_tokens":1096,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":996}},"tokens_in":544,"tokens_out":1096,"duration_ms":10950,"temperature":1.0,"reasoning_tokens":996,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:06:02.703670+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the retrieved subgraph in the reasoning step with a same-sized subgraph taken from a different monster or from random paths, holding everything else fixed; if accuracy stays near the augmented level, the retrieval step is not what carries the answer. A cheaper observation is to compute accuracy on the subset of questions where the model retrieved zero correct paths: the paper already reports this happens in sub-task VI with high accuracy, and if that subset is large across all sub-tasks, the graph is adding less than the headline comparison suggests.","supporting_citations":[{"cited_title":"A survey on knowledge graphs: Represen- tation, acquisition, and applications.TNNLS, 33(2):494–514,","cited_arxiv_id":null,"evidence_quote":"Defines knowledge graphs as structured knowledge storage, motivating graph-based retrieval over unstructured text."},{"cited_title":"Multi-modal knowledge graph construction and ap- plication: A survey","cited_arxiv_id":null,"evidence_quote":"Provides the attribute-based multimodal knowledge graph formulation that MH-MMKG instantiates."},{"cited_title":"Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph","cited_arxiv_id":null,"evidence_quote":"Supplies the self-searching and route-planning idea over knowledge graphs that the multi-agent retriever extends to multimodal data."}],"review_version":2}