{"id":"7fd6721b-56d9-4e4b-b3e0-73896f2d5da7","arxiv_id":"2505.03654","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A personalized multimodal assistant trained with knowledge graphs and chain-of-thought QA can reason about relations between a user's concepts, beating prior recognition-only personalization methods.","lead":"ReGraP-LLaVA adds knowledge graphs and step-by-step question answers to personalized vision assistants, letting them answer relational queries such as \"who can help Bocchi with lyrics\" rather than only recognizing faces or objects. The paper releases a 120-set dataset and benchmark for this task, and reports large gains over previous personalization models and strong GPT-4o-based competitors.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GPT-4o-generated KGs and benchmark answers form a closed loop; if graph facts are not independently verified, the claimed reasoning is ungrounded.","rationale":"I read the paper as a systems contribution proposing a new dataset, training framework, and benchmark for personalized relational reasoning. For the central claim to hold, the model must actually learn and reason over the personalized knowledge contained in the images and text, and the benchmark must measure that knowledge. The weakest point is the provenance of both training and evaluation data: GPT-4o constructs the KGs (Section 3), generates the CoT QA pairs, generates reference answers, and scores open-ended responses. The paper tries to control for fabrication via prompt instructions and manual review, but no quantitative verification of KG triplet accuracy is reported. This is more than a label-noise issue: if the graphs are inaccurate, training and evaluation share the same systematic error source, so high accuracy could merely indicate that the model is good at imitating GPT-4o's synthetic knowledge. The reader's verdict of CONDITIONAL is appropriate because the concern is concrete and addressable: independent human verification of a sample of KGs and a human-built gold benchmark would either substantiate or refute the claim. I also considered omitting KG-augmented baselines like GraphVis/KAM-CoT; while that weakens the 'SoTA' comparison, it is secondary because the paper's main claim is about the method's capability, not exclusivity. The tiny hard-vs-combination ablation gap (Table 5) does not threaten the central claim. Therefore I agree with the reader's weakest-assumption identification and recommend no change to the verdict.","tokens_in":517,"tokens_out":3500,"duration_ms":131108,"concrete_test":"Select 10 of the 120 knowledge sets. For each, have two human annotators independently verify every KG triplet (entity, relation, entity/attribute) against the source images and textual descriptions, labeling each as correct, partially correct, or fabricated. Compute triplet-level precision and Cohen's kappa. Then construct a new 'gold' benchmark for those sets with questions written by humans (not GPT-4o) based only on verified triplets, ensuring no overlap with training CoT QA pairs. Re-run ReGraP-LLaVA against LLaVA (CoT) and GPT-4o (Prompt) on this gold benchmark. If triplet precision is below ~95% or if the accuracy gap on difficult questions shrinks below the reported weighted improvement, the central reasoning claim is an artifact of the synthetic data pipeline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that ReGraP-LLaVA performs relational reasoning over personalized knowledge rests on the correctness of the KGs and CoT QA pairs generated by GPT-4o (Section 3) and on the benchmark questions/reference answers also derived from the same pipeline. The prompts in Tables 13-16 instruct GPT-4o to avoid 'make-up information,' but no independent verification of the generated graph triplets is reported. Section C only evaluates QA-pair quality, not graph fidelity. The benchmark's reference answers are generated by GPT-4o (Table 18), and GPT-4o also scores open-ended outputs (Table 19), creating a closed evaluation loop. The paper's own Section J admits the model 'may occasionally generate fabricated information.' Section E says responses are manually reviewed, but that is not a systematic verification of KG facts. If the KGs contain invented attributes or relations, the model is trained to reproduce, and evaluated against, the generator's fiction rather than genuine personalized knowledge. This affects training and evaluation simultaneously, so the reported gains on 'difficult' reasoning questions (e.g., fill-in-the-blank 0.858 vs 0.817 for LLaVA (CoT), Table 2) could reflect memorization of the generator's synthetic consistency instead of human-grounded relational reasoning.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ReGraP, a dataset of 120 personalized-knowledge sets, each containing images, a GPT-4o-constructed knowledge graph (KG), and chain-of-thought QA pairs derived from that graph. On top of this, the authors propose ReGraP-LLaVA, a LLaVA-based MLLM trained with soft and/or hard graph prompts, and a benchmark with multiple-choice, fill-in-the-blank, true/false, and descriptive questions in closed- and open-ended settings. Experiments report that ReGraP-LLaVA outperforms prompt-based and finetuning-based baselines on most relational-reasoning tasks, with additional ablations, human-preference evaluation, and qualitative evidence.","tokens_in":33862,"tokens_out":4417,"duration_ms":48281,"significance":"If the central claims hold, the paper makes a useful contribution to personalized MLLMs: it is among the first to move beyond single-concept recognition to multi-concept relational reasoning, it releases code and data, and it includes careful comparisons against both prompt-based and finetuning-based baselines. The proposed benchmark, despite the concerns below, is a plausible template for evaluating personalized relational reasoning. However, the empirical case rests on GPT-4o-generated training and evaluation content, and the paper currently does not establish that the model is reasoning over human-grounded personalized knowledge rather than reproducing the generator's synthetic graph; this is the load-bearing issue for the claimed significance.","major_comments":[{"comment":"The training and evaluation are closed-loop with respect to GPT-4o-generated content. The KGs and CoT QA pairs are produced by GPT-4o (Section 3), and the benchmark's reference answers for open-ended questions are also GPT-4o-generated (Table 18) and scored by GPT-4o (Table 19). Section C evaluates the quality of CoT QA pairs, but not the fidelity of the graph facts to the real objects or human-provided knowledge. Section E states that responses are manually reviewed, but this is not a systematic fact-level verification of the triplets. Section J itself admits that the model may generate fabricated information. As a result, the gains on 'difficult' reasoning questions (e.g., Table 2, Fill-in-the-Blank Difficult 0.858 vs 0.817) could reflect the model learning the generator's synthetic consistency rather than performing human-grounded relational reasoning. The authors should either independently verify graph triplets (e.g., by human judges or cross-referencing the images and user-provided text), reserve a held-out set of human-written benchmark questions, or analyze which errors are attributable to graph hallucinations.","section":"Section 3, Eq. (3); Section 5 (Benchmark); Tables 18-19; Section J"},{"comment":"No error bars, confidence intervals, or significance tests are reported for any of the main quantitative results. The differences between top methods are often small; for example, in Table 2, GPT-4o (Prompt) and Qwen2.5-VL-72B (Prompt) differ by 0.021 on difficult multiple-choice, and in Table 5 the three graph-prompting variants differ by at most about 0.4%. Section I checks semantic consistency of outputs across repeated runs for five sets, but it does not report the variance of the benchmark metrics used in Tables 2-5. Without this information, the 'SoTA' claim and the head-to-head comparisons are not statistically grounded. Reporting repeated-seed standard deviations or bootstrap confidence intervals would make the claims commensurate with the evidence.","section":"Section 6, Tables 2-5; Section I"},{"comment":"The benchmark questions appear to be near-duplicates of the training QA facts. For instance, the training example for girls_band_cry in Table 25 states 'Rupa is the bassist of Togenashi Togeari,' and the benchmark in Table 31 asks 'Rupa plays the ___ in TOGENASHI' with the answer 'Bass.' Similarly, several ROG_mouse benchmark questions mirror the attribute questions in the training data (brand, connection type, RGB lighting). If the benchmark questions are generated from the same KGs used for training, then the closed-ended tasks largely measure memorization of the training graph rather than the model's ability to reason over genuinely new personalized queries. The authors should clarify the exact construction process of the benchmark and, ideally, add held-out facts or paraphrased human-written questions that require composing known relations in new ways.","section":"Section 5 (Benchmark); Tables 25, 26-31"}],"minor_comments":[{"comment":"Equation (4) writes \\H = MLP(H_g) ∈ R^d, but X_emb in Eq. (5) is in R^{m×d}; the addition X_new = X_emb + \\H requires either broadcasting or a sequence-alignment explanation. Please clarify the intended shapes.","section":"Section 4.1, Eq. (4)"},{"comment":"Several placeholder glyphs appear in the figure text (e.g., '????????0', 'dram', and '????????1'), which makes the graph-prompting mechanism hard to read. The figures should be regenerated with actual token labels.","section":"Figures 2 and 3"},{"comment":"The training description reports 10 training images and about 20 CoT QA pairs per set, but does not state how the per-set models are trained and evaluated (e.g., one model per set, batch size, number of steps, or how the 10 epochs are scheduled across sets). This information is needed for reproducibility.","section":"Section 5 (Training)"},{"comment":"The QA-pair quality evaluation reports that all LLM evaluators except GPT-4o answer 'Yes' for nearly all criteria; this near-perfect agreement may indicate evaluator leniency or prompt-induced bias rather than genuine quality. Reporting per-criterion disagreement statistics and including more negative control items would strengthen the evaluation.","section":"Section C, Table 7"},{"comment":"The robustness analysis checks semantic consistency of generated descriptions, but not the stability of the numerical metrics in Tables 2-5. The claim 'confirming the robustness of our training pipeline' is therefore narrower than the robustness evidence needed for the main results.","section":"Section I"}],"recommendation":"major_revision","confidential_remarks":"The code/data release and the benchmark design are strengths, and the paper is clearly written. The main risk is the circularity between GPT-4o-generated training graphs, the benchmark, and the GPT-4o-based evaluation; I would like to see the graph facts independently verified or the benchmark grounded in human-written questions before this is accepted as a demonstration of relational reasoning. The lack of significance testing is a second, relatively standard but important, revision request."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know about this paper is that it's a solid resource contribution with a real validity problem in the evaluation. The ReGraP dataset (120 sets, multi-object, KGs, CoT QA) and the benchmark are genuinely new and useful, and the graph-prompting idea is worth having. But the KG facts, the benchmark questions, and the reference answers are all generated by GPT-4o, and GPT-4o also scores the open-ended outputs. That is a closed loop. The paper's claims of relational reasoning are only as good as the generator's fiction, and there is no independent verification that the graph triplets are true of the images or the user's knowledge.\n\nWhat is good: the dataset fills a real gap; existing personalized MLLM sets are single-object with no relations. The method is clearly described; they try both soft and hard prompting. The experiments include ablations, human preference, robustness, and comparison with reasonable baselines. The paper is honest about limitations (Section J admits hallucination). That part deserves credit.\n\nWeaknesses, in proportion: The main one is the closed loop. I don't think it is a fake paper; I think it is an unverified dependence. If the KGs contain invented attributes, both training and evaluation measure fidelity to the generator's fiction. Section C only checks QA-pair quality, not KG fidelity. Section E says responses are manually reviewed, but that is not a systematic verification of graph facts. The stress-test note holds up. Minor: no error bars or significance tests; the SoTA claim omits the closest KG-augmented baselines (GraphVis, KAM-CoT) from Table 2, which makes the comparison less convincing.\n\nVerification of the central claim: The qualitative examples look impressive, and ReGraP-LLaVA's answers do seem to use relations. But without independent grounding, I cannot call the reasoning claim proven. It is plausible. The dataset is still useful even if the absolute numbers are suspect.\n\nWho this is for: researchers working in personalized MLLMs, knowledge graph prompting, and benchmark design. They will get value from the resource and the method, and they should be the ones to pressure-test the KG fidelity.\n\nMy recommendation: send it to peer review, not desk reject. The resource is real, the method is sensible, and the problems are fixable. The referee report should require (1) independent verification of a sample of KG triplets against images/user knowledge, (2) error bars, (3) KG-augmented baselines, (4) human-checked reference answers or at least a second LLM for grading. If the authors can show the graph facts hold up, this becomes a genuinely useful paper.","headline":"Useful new dataset and graph-prompting method for personalized MLLMs, but the GPT-4o closed evaluation loop undermines the reasoning claim until the graph facts are independently checked.","tokens_in":34497,"tokens_out":2101,"would_cite":true,"duration_ms":21259,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A personalized multimodal assistant trained on knowledge graphs and chain-of-thought QA pairs can reason over relations among a user's concepts, not just recognize or caption them.","keywords":["personalized multimodal large language models","knowledge graph reasoning","chain-of-thought QA","graph prompting","relational reasoning in MLLMs","personalization benchmark","visual instruction tuning","knowledge graph alignment"],"falsifier":"Show the model a counterfactual image in which a learned relation is visually reversed (e.g., two band members' roles swapped) and ask the same relational question; if the model answers according to the stored graph rather than the visible evidence, then the reported reasoning is retrieval of generated relations, not reasoning grounded in the image.","tokens_in":33413,"feed_emoji":"🧠","tokens_out":7507,"duration_ms":67933,"temperature":0.7,"pith_summary":"ReGraP-LLaVA tries to establish that a personalized multimodal assistant can reason over the relations among a user's concepts, not merely recognize or caption them. The paper builds the ReGraP dataset of 120 personalized-knowledge sets, each combining images, a knowledge graph, and chain-of-thought QA pairs, then trains a 7B LLaVA model on these triples with soft and hard graph prompting. On a new benchmark spanning multiple-choice, fill-in-the-blank, true/false, and descriptive questions, the model reports the highest accuracy among prompt-based and fine-tuned baselines, including larger models. If correct, user-facing assistants could answer multi-step contextual queries such as \"Who can help Bocchi with lyric writing?\" by connecting attributes and relations across several concepts.","feed_headline":"Graph-prompted AI assistant reasons over personal knowledge in images","feed_subtitle":"Graphs plus chain-of-thought let a small assistant answer relational personal questions, not just recognize objects.","key_machinery":"The carrying object is the personalized knowledge graph $G$: nodes are personalized concepts and their attributes, edges are relations, and each CoT QA pair is attached to the reasoning subgraph $R \\subseteq G$ that justifies the answer. Soft prompting encodes $R$ with a graph neural network and projects it with an MLP into the LLM's embedding space as $\\hat{H} = \\mathrm{MLP}(\\mathrm{GNN}(R))$, then concatenates it with instruction embeddings. Hard prompting serializes $R$ into triples, introduces new entity and relation tokens into the tokenizer, and extends the classification head by $2N+1$ tokens. Both routes make the graph structure part of the model's input, and the CoT QA pairs act as a natural-language extraction of that same structure; the hard-prompt route is the configuration with the best accuracy in the paper.","core_discovery":"On the paper's own terms, the discovery is that a 7B multimodal LLM can internalize a user's knowledge graph and chain-of-thought reasoning from QA pairs, and then use that structure to answer questions that join facts across two or more concepts. The authors construct 120 personalized-knowledge sets, each with images, a knowledge graph, and CoT QA pairs whose answers are tied to the reasoning subgraph, and train LLaVA on them with soft and/or hard graph prompting. Across close-ended tasks, ReGraP-LLaVA reaches 0.942 simple and 0.892 difficult multiple-choice accuracy, and improves open-ended descriptive answers, while also transferring to the Yo'LLaVA and MyVLM recognition/captioning benchmarks. The conclusion is that graph-structured personal knowledge plus CoT supervision is sufficient for an MLLM to move from recognizing concepts to reasoning over their relations.","pith_inferences":["Editorial inference: the same soft/hard graph-prompting recipe may transfer to any structured knowledge injection into MLLMs, such as temporal or causal graphs, since the mechanism only assumes a graph can be serialized or embedded.","Editorial inference: the benchmark's difficult questions are all derivable from the same generated graphs; a stronger generalization test would hold out entire relation types or ask questions requiring graph knowledge to be combined with visual details not present in the graph.","Editorial inference: because the paper reports no negative or counterfactual training examples, the model's confidence in denying an object's presence may degrade; adding such examples could improve both negative recognition and robustness.","Editorial inference: replacing the GPT-4o-generated graphs with human-verified graphs would clarify whether the reported relational reasoning transfers beyond the generator's internal consistency; this is a testable extension the paper does not report."],"forward_implications":["Personalized assistants can answer multi-step relational queries (e.g., \"Who can help Bocchi with lyrics?\") instead of only recognizing or captioning a user's concepts.","Training MLLMs on image-KG-CoT triples with graph prompting can yield large gains on relation-based questions without scaling the base model beyond 7B.","The ReGraP benchmark provides a reusable test for evaluating knowledge connection and relational reasoning in personalized MLLMs across MC, fill-in-the-blank, true/false, and descriptive tasks.","The hard graph-prompting variant, which adds entity and relation tokens, is the highest-accuracy configuration in the paper and is the recommended default.","The model also transfers to prior personalization datasets (Yo'LLaVA and MyVLM), preserving recognition and captioning while adding relational ability."],"supporting_citations":[{"why":"Provides Yo'LLaVA, the personalized-MLLM baseline and the new-token concept approach that hard prompting extends.","marker":"[5]"},{"why":"Supplies LLaVA, the base vision-language architecture and training recipe used for ReGraP-LLaVA.","marker":"[2]"},{"why":"Motivates the chain-of-thought QA construction as a way to elicit step-by-step reasoning.","marker":"[14]"},{"why":"Serves as GPT-4o, the LLM used to build relation graphs, enrich KGs, generate CoT QA pairs, and score open-ended answers.","marker":"[29]"},{"why":"Provides MyVLM and its dataset, used in transfer experiments for recognition and captioning.","marker":"[9]"},{"why":"Is the LLaVA-v1.6-Vicuna-7B base model and a main fine-tuning baseline.","marker":"[32]"},{"why":"Is the 13B LLaVA prompt-based baseline that is the strongest competitor on simple descriptive QA.","marker":"[35]"},{"why":"Offers GraphVis, a knowledge-graph-to-visual-prompt method that motivates injecting KG structure into vision-language models.","marker":"[42]"}],"fun_headline_variants":["Graph reasoning makes personal AI assistants smarter","Personal AI now reasons over your knowledge graph","Graph-prompted AI links personal concepts with reasoning","AI assistant uses knowledge graphs to answer relational queries","ReGraP-LLaVA: AI that reasons over personalized knowledge"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the GPT-4o-constructed knowledge graphs and CoT QA pairs, together with the benchmark questions and reference answers derived from them, correctly and completely capture true personalized knowledge and the answers a human would give; if the generated graphs contain invented attributes or relations, both training and evaluation are measuring fidelity to the generator's fiction rather than genuine reasoning about the world.","fun_headline_variants_meta":{"raw":{"variants":["Graph reasoning makes personal AI assistants smarter","Personal AI now reasons over your knowledge graph","Graph-prompted AI links personal concepts with reasoning","AI assistant uses knowledge graphs to answer relational queries","ReGraP-LLaVA: AI that reasons over personalized knowledge"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1404,"prompt_tokens":1056,"completion_tokens":348,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":672,"completion_tokens_details":{"reasoning_tokens":274}},"tokens_in":672,"tokens_out":348,"duration_ms":3759,"temperature":1.0,"reasoning_tokens":274,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:46:09.421705+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Show the model a counterfactual image in which a learned relation is visually reversed (e.g., two band members' roles swapped) and ask the same relational question; if the model answers according to the stored graph rather than the visible evidence, then the reported reasoning is retrieval of generated relations, not reasoning grounded in the image.","supporting_citations":[{"cited_title":"Yo'llava: Your personalized language and vision assistant","cited_arxiv_id":null,"evidence_quote":"Provides Yo'LLaVA, the personalized-MLLM baseline and the new-token concept approach that hard prompting extends."},{"cited_title":"Myvlm: Personalizing vlms for user-specific queries","cited_arxiv_id":null,"evidence_quote":"Provides MyVLM and its dataset, used in transfer experiments for recognition and captioning."},{"cited_title":"Llava-v1.6-vicuna-7b","cited_arxiv_id":null,"evidence_quote":"Is the LLaVA-v1.6-Vicuna-7B base model and a main fine-tuning baseline."},{"cited_title":"response 1 is better","cited_arxiv_id":null,"evidence_quote":"Offers GraphVis, a knowledge-graph-to-visual-prompt method that motivates injecting KG structure into vision-language models."}],"review_version":1}