REAL: Resolving Knowledge Conflicts in Knowledge-Intensive Visual Question Answering via Reasoning-Pivot Alignment

Bin Li; Haikuan Huang; Jiajun Bu; Kai Ye; Liangliang Liu; Sheng Zhou; Xianwei Mao; Ye Mo; Zirui Shao

REVIEW 5 major objections 5 minor 1 cited by

The paper defines knowledge conflicts in VQA at the granularity of reasoning-pivots — atomic steps external evidence must supply — and shows that pivot-aware training plus pivot-guided decoding makes conflict detection generalizable.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 23:18 UTC pith:RJ4QE274

load-bearing objection Solid empirical framework for pivot-level conflict resolution in KI-VQA; main VQA gains are plausible, but the conflict-discrimination generalization claim is undercut by same-pipeline synthetic test labels and missing release of code/data. the 5 major comments →

arxiv 2602.14065 v2 pith:RJ4QE274 submitted 2026-02-15 cs.AI

REAL: Resolving Knowledge Conflicts in Knowledge-Intensive Visual Question Answering via Reasoning-Pivot Alignment

Kai Ye , Xianwei Mao , Sheng Zhou , Zirui Shao , Ye Mo , Liangliang Liu , Haikuan Huang , Bin Li

show 1 more author

Jiajun Bu

This is my paper

classification cs.AI

keywords knowledge conflictsknowledge-intensive visual question answeringretrieval-augmented generationreasoning-pivotconflict discriminationcontrastive decodingmultimodal language modelsreasoning chain

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When a retrieval-augmented vision-language model is fed mutually contradictory paragraphs, the paper argues, standard conflict detectors look at the wrong granularity: they compare entities or keywords and raise false alarms over differences that are irrelevant to the answer. The proposed alternative defines a conflict only where mutually exclusive assertions attach to the same reasoning-pivot, an atomic node or edge in the reasoning chain that depends on external evidence. Around that definition, the authors build a pivot-annotated dataset, fine-tune a discriminator to first extract pivots and then judge logical consistency within each pivot, and add a training-free decoding strategy that suppresses the pivot-specific conflicting direction during generation. If the framework works as claimed, conflict detection becomes a transferable reasoning task rather than a keyword-matching heuristic, and answer accuracy on knowledge-intensive VQA improves with it.

Core claim

The central claim is that a knowledge conflict in KI-VQA should be formalized as logical incompatibility within the information set of a single reasoning-pivot, not as a mismatch between retrieved text and visual input. For a reasoning chain, the pivots are the minimal set of nodes and edges whose removal would make the answer unreachable under zero prior knowledge; a conflict exists exactly when some pivot has two assertions that cannot both be true, written as a_i ∧ a_j → ⊥. The paper further claims that training a model to output question pivots, paragraph pivots, and then the conflict label — Reasoning-Pivot Aware SFT — transfers across domains and to unseen conflict benchmarks, and that

What carries the argument

The load-bearing object is the reasoning-pivot, defined as an atomic node or edge in the reasoning chain that anchors knowledge linkage and requires external evidence to complete the inference. Two devices carry the argument: RPA-SFT supervises the model to extract pivots from the question and paragraphs before producing a conflict label, making the discriminator reason about logical consistency instead of superficial entity or keyword differences; and RPGD decodes by comparing normal logits with logits from a patch-shuffled image pathway, gating suppression onto pivot tokens and removing only the Gram-Schmidt projection of the conflict-aligned component, so valid reasoning structure survive

Load-bearing premise

The framework rests on the premise that every question has a well-defined minimal set of reasoning-pivots — units whose removal genuinely makes the answer unreachable — and that automated annotation marks those pivots consistently enough to serve as training supervision.

What would settle it

Have independent human annotators mark reasoning pivots on a random sample of the authors' test items and measure inter-annotator agreement; if annotators cannot agree on which units are indispensable, the definition is not operational. Also, a zero-knowledge model should fail when a rated pivot is removed but succeed when a non-pivot unit is removed, and RPGD's gain should vanish when the gate is applied to random tokens rather than pivot tokens.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Pivot-first training transfers conflict detection beyond its training distribution, including to a held-out multimodal conflict benchmark.
Conflict resolution can work at decoding time without additional training once pivots are known, adding accuracy on top of supervised tuning.
The discriminator can localize the contradiction, not just label it, because it is trained to output which pivot carries the conflict.
Removing only the projected conflict direction preserves language quality better than plain contrastive subtraction.
The method applies across model families and parameter scales, making the pivot signal useful without a custom retriever.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

A testable extension beyond the paper: the same pivot criterion could label document-level conflicts in text-only retrieval-augmented generation, where there is no image but the notion of an indispensable reasoning unit still applies.
If the definition is right, retrieval quality could be scored pivot-wise — the retriever's job becomes covering every pivot rather than returning topically similar passages — connecting retrieval and conflict resolution into one objective.
The paper does not test how robust the discriminator is to bad pivot extraction; one could perturb or omit pivots at inference and measure the drop in discrimination accuracy, isolating how much of the gain is truly pivot-driven.
A minimal probe of the visual pathway: apply the same gated orthogonal subtraction with the image removed or unshuffled; if the gain persists, the conflict signal comes mostly from the textual pivots rather than the patch-shuffle mechanism.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Solid empirical framework for pivot-level conflict resolution in KI-VQA; main VQA gains are plausible, but the conflict-discrimination generalization claim is undercut by same-pipeline synthetic test labels and missing release of code/data.

read the letter

Things to know up front: the paper is a genuine attempt to define knowledge conflicts at the granularity of reasoning pivots, and the main VQA gains are consistent across three model families. The soft spot is exactly where the stress-test points: the headline generalization numbers for conflict discrimination mostly come from test sets synthesized with the same GPT-4o pipeline used to build the training data, so the "generalizable discriminator" claim is not yet empirically established.

What is new and what works: the reasoning-pivot formalism (Eq. 1) is a clean departure from entity- or keyword-mismatch conflict definitions, and the paper makes a plausible case that pivot-level supervision is more semantically meaningful. REAL-VQA is a real dataset contribution, even if small (4,149 train / 629 test), and the construction pipeline is described in enough detail to be reproduced. RPA-SFT with special pivot tokens and staged reasoning targets is a reasonable way to avoid shortcut learning. RPGD is a clever combination of patch shuffle, adaptive gating, and Gram-Schmidt orthogonalization; the ablations show each component contributes, and the decoding gains over VCD/CAD are consistent. The scale experiments in Table 10 are a nice addition, and the limitations section is honest rather than perfunctory.

The soft spots are real. Most importantly, the conflict-discrimination evaluation on E-VQA and ScienceQA uses labels synthesized with the same Sec. 4.1 pipeline and the same GPT-4o annotator as REAL-VQA. The large MCC gains there may reflect distribution matching rather than transfer to independently annotated conflicts. The only external set, MMKC, is evaluated on 892 of 1,573 samples, and the RPA-SFT advantage over SFT there is modest (e.g., Qwen3-VL-8B MCC 52.9 vs 38.2; F1 +1.6). The headline "14.68% average improvement" is mostly measured against zero-shot, not a strong SFT baseline. There are also no error bars or seeded repeats, and no code or data release, which makes it impossible to tell how much variance is in these numbers. The pivot definition itself assumes a well-defined minimal pivot set that GPT-4o can annotate consistently; the paper does not independently validate that assumption. None of these flaws is fatal, but they do mean the central claim is conditional.

Who this is for: researchers working on retrieval-augmented VQA, knowledge conflicts, or contrastive decoding will find the formalism and the decoding recipe worth their time. The paper deserves a serious referee, but the verdict should be major revision until the authors release the dataset and code, evaluate on at least one independently annotated conflict benchmark in full, report variance, and compare against a stronger SFT-only baseline.

My read: send it to review, but expect the reviewers to press hard on the evaluation circularity. If the authors can show transfer on genuinely external annotations, this becomes a solid contribution.

Referee Report

5 major / 5 minor

Summary. The paper proposes REAL, a framework for resolving knowledge conflicts in knowledge-intensive VQA. It introduces a new notion of 'Reasoning-Pivot'—purportedly the minimal set of nodes/edges in a reasoning chain that require external evidence—and defines a conflict as mutually exclusive assertions about the same pivot. The authors construct a GPT-4o-generated dataset, REAL-VQA, with pivot and conflict annotations; train a discriminator via Reasoning-Pivot Aware SFT (RPA-SFT); and introduce Reasoning-Pivot Guided Decoding (RPGD), a contrastive decoding method that suppresses conflict-aligned logits at pivot tokens using patch shuffle, adaptive gating, and Gram-Schmidt projection. Experiments report state-of-the-art results on E-VQA, InfoSeek, and A-OKVQA, plus conflict-discrimination gains on REAL-VQA, E-VQA, ScienceQA, and MMKC.

Significance. The reasoning-pivot formalism is a conceptually interesting way to localize knowledge conflicts, and the idea of combining pivot-level supervision with contrastive decoding is worth pursuing. The paper has several concrete strengths: consistent accuracy gains on three KI-VQA benchmarks across three model families, a component ablation (Table 6) that points to all three RPGD mechanisms being useful, and a parameter-scaling analysis in Appendix C.1. However, the central claim that RPA-SFT yields a generalizable conflict discriminator is not yet empirically established: part of the generalization evidence is generated by the same GPT-4o pipeline used to create the training set, and the only external conflict benchmark is evaluated on a subset with small, inconsistent gains over the SFT control. If the authors add independent conflict annotations, clarify the decoding-time use of pivot extraction, and fix several reporting inconsistencies, the contribution could become solid. As it stands, the paper needs substantial additional validation.

major comments (5)

[§5.1, Table 3] The main generalization evidence for the conflict discriminator is partly circular. The paper states that 'we synthesize conflict samples for E-VQA and ScienceQA following the pipeline described in Sec. 4.1,' i.e., the same GPT-4o rewriting and annotation protocol used to build REAL-VQA. The large RPA-SFT gains on these sets (e.g., Qwen3-VL-8B MCC 93.4 vs. 82.6 on E-VQA) may therefore reflect distribution matching with the training generator rather than transfer to naturally occurring or independently annotated conflicts. The only fully external set, MMKC, is evaluated on a subset (892 of 1,573 samples, §B.1), and gains over SFT are small or absent (Qwen3-VL-8B F1 74.8 vs. 73.2; InternVL3.5 F1 85.8 vs. 85.8). Please add independently annotated or human-verified test sets, and report full-MMKC results.
[§1, Table 3] The headline claim that 'RPA-SFT improves conflict discrimination by 14.68% on average over Qwen3-VL-8B' is computed against zero-shot/few-shot baselines, not against the appropriate SFT control. Against SFT, the gains are much smaller and inconsistent: for Qwen3-VL-8B, ScienceQA MCC is 87.9 vs. 87.0 and MMKC F1 is 74.8 vs. 73.2; for InternVL3.5, MMKC F1 is 85.8 vs. 85.8. The claim of a generalizable discriminator should be quantified and discussed relative to SFT without pivot supervision, not only against zero-shot.
[§4.3, Algorithm 1] RPGD's adaptive gating depends on 'identified pivot spans' mapped to the token set K, but the paper does not specify how K is obtained at decoding time. RPA-SFT produces a three-stage textual output (question pivots, paragraph pivots, conflict label), yet no parser, confidence threshold, or fallback is described for converting these outputs into vocabulary indices. It is also unclear whether K comes from the model's own (possibly imperfect) extraction or from oracle annotations. This makes the adaptive-gating ablation in Table 6 difficult to interpret and the method hard to reproduce. Please specify the extraction protocol and, ideally, report results with predicted vs. oracle pivots.
[§3.2, §4.1] The Reasoning-Pivot definition is not operational. 'Minimal essential set' and 'zero-knowledge model' are informal, and no inter-annotator agreement, human verification counts, or per-type pivot error analysis are reported for the GPT-4o annotations. Since pivot quality is the substrate for both RPA-SFT supervision and RPGD gating, the paper should report annotation consistency and include representative annotation failures.
[Table 5 vs. Table 1] The LLaVA-1.5-7B 'REAL(w/o RPGD)' row in Table 5 (27.3/26.3/26.8/31.1/28.5) is numerically identical to the EchoSight row in Table 1, although the text identifies this row as RPA-SFT without RPGD. If this is not an error, the coincidence requires explanation; if this is EchoSight rather than RPA-SFT, the ablation conclusion that RPGD adds +3.2/+5.4 over RPA-SFT is not supported. Please correct the table or the description and rerun/restate the ablation accordingly.

minor comments (5)

[§5.1] The ScienceQA reference appears to be Saikh et al. 2022 (scholarly article QA), but the described multimodal science benchmark is the ScienceQA of Lu et al. 2022. Please correct the reference and clarify how the 1,000-example conflict split was sampled.
[Table 1] Please define Un-Q, Un-E, Single-Hop, and All in the caption. Also, 'EV A-CLIP-8B' is likely a formatting/typo issue.
[Algorithm 1] The Cutoff function and the elementwise multiplication symbol ⊙ are used without definition. Please add formal definitions or a pointer to the appendix.
[Appendix D.2] The RPA-SFT prompt shown is abbreviated. Include the exact three-stage output template and the special-token grammar so that the target format is reproducible.
[General] No code or dataset release is mentioned. Given that REAL-VQA is a central contribution and the paper does not provide data or code, a public release statement and a reproducibility checklist would be important.

Circularity Check

1 steps flagged

Conflict-discrimination 'generalization' is evaluated on labels synthesized by the same GPT-4o pipeline that generated the REAL-VQA training data, so the headline discriminator gains are partly a same-generator distribution match rather than fully independent transfer.

specific steps

fitted input called prediction [Section 5.1 (Datasets, Conflict Discrimination) and Table 3; contrasted with Sec. 4.1]
"Crucially, to rigorously assess generalization, we synthesize conflict samples for E-VQA and ScienceQA following the pipeline described in Sec. 4.1, annotating external conflicts and ground truths based on available fields containing rationales."

The REAL-VQA training labels are produced by the Sec. 4.1 GPT-4o rewrite/annotation pipeline, and RPA-SFT is trained on those labels. The E-VQA and ScienceQA conflict test sets are then created 'following the pipeline described in Sec. 4.1' with the same GPT-4o procedure. Therefore Table 3's 'cross-scenario adaptation' gains (e.g., Qwen3-VL-8B MCC 82.6->93.4 on E-VQA, 87.0->87.9 on ScienceQA) measure agreement with the same generator distribution used for training, not transfer to naturally occurring or independently annotated conflicts. The external MMKC benchmark shows a much smaller RPA-SFT-over-SFT gain (F1 74.8 vs 73.2 for Qwen3-VL-8B), consistent with partial circularity.

full rationale

The paper's central derivation chain is: define reasoning-pivots; construct REAL-VQA with GPT-4o; train RPA-SFT; evaluate the discriminator on REAL-VQA, E-VQA, ScienceQA, and MMKC; then use the discriminator to gate RPGD decoding for KI-VQA accuracy. The load-bearing circular step is in the generalization evidence: the conflict labels for E-VQA and ScienceQA are synthesized with the same Sec. 4.1 GPT-4o pipeline that created the REAL-VQA training set. Thus the discriminator's apparent cross-scenario adaptation is partly a self-consistency check against the training generator, not an independent test. This is not full circularity: the KI-VQA accuracy results on E-VQA, InfoSeek, and A-OKVQA use external human-annotated labels, and MMKC provides a (smaller-gain) external conflict benchmark. The paper does not rely on load-bearing self-citation, uniqueness theorems, or ansatz smuggling; the 'reasoning-pivot' definition itself is not circular by construction. The inflated 14.68% average improvement is mostly versus zero-shot and is dominated by same-pipeline sets, but that is an evaluation-strength issue rather than a definitional equivalence. Overall, one significant partial evaluation circularity is present, so the score is moderate.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 2 invented entities

The central machinery rests on the pivot conflict ontology and on GPT-4o-generated supervision; external grounding comes from standard VQA benchmarks and MMKC. RPGD's epsilon, beta, kappa, tau, and delta are hand-set hyperparameters.

free parameters (5)

RPGD baseline suppression epsilon = 0.1
Global suppression baseline in the token-wise gate (Eq. 2, App. B.3); hand-set, not derived.
RPGD suppression strength beta = 0.2
Controls overall suppression strength on conflict-prone pivot tokens (Eq. 3, App. B.3).
RPGD sigmoid temperature kappa = 0.1
Temperature to avoid saturation in adaptive gating (Eq. 3, App. B.3).
RPGD stability constant delta = 1e-6
Numerical stability in Gram-Schmidt projection (Eq. 4, App. B.3).
RPGD cutoff tau = not reported numerically
Cutoff threshold used during sampling in Algorithm 1; mentioned but no value given in main text or appendix.

axioms (4)

domain assumption Reasoning-pivots are the minimal indispensable set of nodes/edges for answering a query under zero prior knowledge.
Sec. 3.2 defines pivots by this counterfactual; no evidence that human or MLLM reasoning decomposes this way or that GPT-4o annotates it consistently.
domain assumption A valid knowledge conflict exists iff some reasoning-pivot has two mutually exclusive assertions (Eq. 1).
This is the paper's redefinition of conflict; it replaces entity/keyword mismatch but is asserted, not derived.
domain assumption Patch shuffling creates a conflict-dominant pathway in which the model over-relies on conflicting text rather than merely degrading.
Sec. 4.3 and Algorithm 1; only component ablations support it, with no direct measurement of pathway semantics.
domain assumption Gram-Schmidt projection of standard logits onto shuffled logits isolates conflict-induced noise.
Sec. 4.3; the linear algebra is standard, but the semantic claim that the projected component is conflict noise is an assumption.

invented entities (2)

Reasoning-Pivot no independent evidence
purpose: Atomic node/edge in the reasoning chain used to define and localize knowledge conflicts.
Defined by the authors and annotated via GPT-4o; no falsifiable prediction outside the framework.
Conflict-dominant pathway no independent evidence
purpose: Reference distribution for contrastive decoding, created by patch shuffle.
Operational construct; its interpretation as 'dominance of conflicting text' is not independently verified.

pith-pipeline@v1.3.0-alltime-deepseek · 24580 in / 12523 out tokens · 115805 ms · 2026-08-02T23:18:14.137208+00:00 · methodology

0 comments

read the original abstract

Knowledge-intensive Visual Question Answering (KI-VQA) frequently suffers from severe knowledge conflicts caused by the inherent limitations of open-domain retrieval. However, existing paradigms face critical limitations due to the lack of generalizable conflict detection and intra-model constraint mechanisms to handle conflicting evidence. To address these challenges, we propose the REAL (Reasoning-Pivot Alignment) framework centered on the novel concept of the Reasoning-Pivot. Distinct from reasoning steps that prioritize internal self-derivation, a reasoning-pivot serves as an atomic unit (node or edge) in the reasoning chain that emphasizes knowledge linkage, and it typically relies on external evidence to complete the reasoning. Supported by our constructed REAL-VQA dataset, our approach integrates Reasoning-Pivot Aware SFT (RPA-SFT) to train a generalizable discriminator by aligning conflicts with pivot extraction, and employs Reasoning-Pivot Guided Decoding (RPGD), an intra-model decoding strategy that leverages these pivots for targeted conflict mitigation. Extensive experiments on diverse datasets demonstrate that REAL significantly enhances discrimination accuracy and achieves superior performance, validating our pivot-driven resolution paradigm.

Figures

Figures reproduced from arXiv: 2602.14065 by Bin Li, Haikuan Huang, Jiajun Bu, Kai Ye, Liangliang Liu, Sheng Zhou, Xianwei Mao, Ye Mo, Zirui Shao.

**Figure 1.** Figure 1: Comparison of Conflict Definitions.Conventional methods (Left) incorrectly flag irrelevant entity or keyword variations as conflicts, while the Reasoning-Pivot definition (Right) correctly distinguishes irrelevant location information from nationality information and only detects conflicts within the nationality pivot, treating unrelated locations as non-conflicting noise. Therefore, the research focus sho… view at source ↗

**Figure 2.** Figure 2: The proposed framework for KIVQA. (1) Data Processing augments the REAL-VQA training set by inserting special tokens, denoted as <RPivot> and </RPivot> (2) RPA-SFT fine-tunes the model with explicit reasoning-pivot awareness to guide the reasoning process. (3) RPGD employs a conflict-based contrastive decoding strategy to resolve ambiguities and ensure accurate reasoning. 4.1. REAL-VQA Dataset Construction… view at source ↗

**Figure 3.** Figure 3: Overview of the REAL-VQA data construction. Quality Assurance. Following the experimental settings of (Wang et al., 2022), we implement a vote-of-confidence filter where each sample undergoes ten stochastic GPT-4o scorings, retaining only those with a cumulative sum ≥ 80 and no single score < 6. After expert verification, the final dataset comprises 4,149 training and 629 test samples. 4.2. Reasoning-Pivot… view at source ↗

**Figure 5.** Figure 5: Case study on E-VQA, comparing logits under RPGD and greedy decoding, showing that RPGD better focuses on conflict knowledge and yields the correct answer. strategies inevitably incur additional inference latency due to dual-stream processing, [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

**Figure 4.** Figure 4: (Left) Accuracy vs. relative per-token latency for different decoding methods; (Right) Improvements of REAL on pivot based QA accuracy on E-VQA. 5.5. Inference Latency We investigate the computational cost across LLaVA-1.5-7B and Qwen3-VL (2B and 8B). While contrastive decoding Q: In which city around the world is this stadium located? Greedy： The city is Valencia RPGD：The city is Barcelona Q: Which write… view at source ↗

**Figure 6.** Figure 6: Data distribution comparison of REAL-VQA, E-VQA, and InfoSeek [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗

**Figure 7.** Figure 7: Representative data samples from the REAL-VQA dataset across different conflict labels. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗

**Figure 8.** Figure 8: Representative success and failure cases from the E-VQA dataset, illustrating decoding methods across different scenarios. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_8.png] view at source ↗

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answering
cs.CV 2026-06 unverdicted novelty 6.0

ProMSA is a progressive multimodal search agent for KB-VQA that iteratively selects search tools under budgets, trained via rejection-sampling SFT then TN-GSPO RL, reporting gains on E-VQA and InfoSeek over RAG baselines.

Reference graph

Works this paper leans on

9 extracted references · cited by 1 Pith paper

[1]

We then select a conflicting coun- terpart pneg (e.g., Entity:Statue of Liberty ) from the same ontological category within Wikidata

Counterfactual Pivot Selection.For each sample, we first identify the ground-truth reasoning-pivot pgt (e.g., Entity:Eiffel Tower ). We then select a conflicting coun- terpart pneg (e.g., Entity:Statue of Liberty ) from the same ontological category within Wikidata. This ensures that the conflict is logically plausible—i.e., the objectcould theoretically ...
[2]

Instead, we retrieve the actual Wikipedia entry corresponding to pneg

Reference Context Retrieval.Crucially, we do not ask the GPT-4o to hallucinate details about pneg. Instead, we retrieve the actual Wikipedia entry corresponding to pneg. This retrieves aReference Context( Cref ) containing accu- rate attributes, history, and descriptions of the conflicting entity
[3]

Hybrid Hallucination

Few-Shot Rewrite Instruction.We employ anIn- Context Learning (Few-Shot)paradigm to guide GPT-4o in the rewriting task. The prompt is structured with three components: • Task Definition:Instructing the model to rewrite the original paragraph describing the visual object (Ovis) so that it describes the conflicting object ( Oneg) in- stead, while maintainin...
[4]

the person in the photo

Synthesis Output.The final output is a coherent paragraph that accurately describes pneg (based on real Wikipedia data) but creates a preciseReasoning-Pivot Con- flictwhen paired with the original image (which depictspgt). This forces the VQA model to trust the visual evidence over the textual description to answer correctly. A.2. Data Statistics and Dist...

2023
[5]

Rewrite the given question into a shared-attribute question
[6]

Generate the corresponding answer to this rewritten question
[7]

shared feature

Generate five evidence sentences (≈50 words each) for the rewritten question and answer. following the rules below, especiallyrule8: ----------------------------------- Rule1:A shared attribute means a general element or property from the semantic list (avoid specific names/values). Rule2:Do not use phrases like "shared feature" explicitly-let it be infer...
[8]

Decide whether there are meaningful conflicts among these external knowledge passages
[9]

Identify the key semantic information units that appear in the question and in the external knowledge passages, and that are directly involved in the reasoning process needed to answer the question. D.3. Prompts for Decoding Prompt for RPGD Decoding You are an expert visual question answering model that jointly uses the image and textual evidence to answe...

1920

[1] [1]

We then select a conflicting coun- terpart pneg (e.g., Entity:Statue of Liberty ) from the same ontological category within Wikidata

Counterfactual Pivot Selection.For each sample, we first identify the ground-truth reasoning-pivot pgt (e.g., Entity:Eiffel Tower ). We then select a conflicting coun- terpart pneg (e.g., Entity:Statue of Liberty ) from the same ontological category within Wikidata. This ensures that the conflict is logically plausible—i.e., the objectcould theoretically ...

[2] [2]

Instead, we retrieve the actual Wikipedia entry corresponding to pneg

Reference Context Retrieval.Crucially, we do not ask the GPT-4o to hallucinate details about pneg. Instead, we retrieve the actual Wikipedia entry corresponding to pneg. This retrieves aReference Context( Cref ) containing accu- rate attributes, history, and descriptions of the conflicting entity

[3] [3]

Hybrid Hallucination

Few-Shot Rewrite Instruction.We employ anIn- Context Learning (Few-Shot)paradigm to guide GPT-4o in the rewriting task. The prompt is structured with three components: • Task Definition:Instructing the model to rewrite the original paragraph describing the visual object (Ovis) so that it describes the conflicting object ( Oneg) in- stead, while maintainin...

[4] [4]

the person in the photo

Synthesis Output.The final output is a coherent paragraph that accurately describes pneg (based on real Wikipedia data) but creates a preciseReasoning-Pivot Con- flictwhen paired with the original image (which depictspgt). This forces the VQA model to trust the visual evidence over the textual description to answer correctly. A.2. Data Statistics and Dist...

2023

[5] [5]

Rewrite the given question into a shared-attribute question

[6] [6]

Generate the corresponding answer to this rewritten question

[7] [7]

shared feature

Generate five evidence sentences (≈50 words each) for the rewritten question and answer. following the rules below, especiallyrule8: ----------------------------------- Rule1:A shared attribute means a general element or property from the semantic list (avoid specific names/values). Rule2:Do not use phrases like "shared feature" explicitly-let it be infer...

[8] [8]

Decide whether there are meaningful conflicts among these external knowledge passages

[9] [9]

Identify the key semantic information units that appear in the question and in the external knowledge passages, and that are directly involved in the reasoning process needed to answer the question. D.3. Prompts for Decoding Prompt for RPGD Decoding You are an expert visual question answering model that jointly uses the image and textual evidence to answe...

1920