Pith. sign in

REVIEW 3 major objections 5 minor 55 references

A 7B multimodal model, trained only on its own answer-conditioned reasoning, beats retrieval-augmented and API-based KBVQA baselines.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 22:15 UTC pith:RGHNV7UQ

load-bearing objection Neat self-distillation recipe, but the 'elicited reasoning' claim is undercut by the answer-containment training loop and the confusion matrix. the 3 major comments →

arxiv 2511.11132 v4 pith:RGHNV7UQ submitted 2025-11-14 cs.CV

From Hindsight to Foresight: Self-Encouraged Hindsight Distillation for Knowledge-based Visual Question Answering

Yu Zhao , Ying Zhang , Xuhui Sui , Baohang Zhou , Xinying Qian , Li Shen , Dacheng Tao This is my paper
classification cs.CV
keywords Knowledge-based Visual Question AnsweringHindsight DistillationChain-of-Thought ReasoningPreference OptimizationSelf-DistillationMultimodal Large Language ModelsOK-VQAA-OKVQA
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that a 7-8B vision-language model can match or beat much larger or externally augmented systems on knowledge-based visual question answering by learning to reason from its own hindsight. The authors first prompt the model with the ground-truth answer to produce structured step-by-step and knowledge traces, then distill those traces back into the model so it can produce them without seeing the answer. A final preference-optimization step nudges the model to emit low-confidence but answer-relevant facts over high-confidence irrelevant ones. If right, it means the knowledge bottleneck for KBVQA is not access to external corpora but the ability to elicit and calibrate what the model already knows.

Core claim

The paper's central claim, on its own terms, is that hindsight self-distillation fully elicits the internal reasoning ability of a 7B-scale multimodal LLM: given a question, the model can generate its own chain-of-thought and discrete knowledge pieces, and these self-generated contexts are sufficient to answer knowledge-based questions without retrieved documents or commercial API teachers. The framework, HinD, constructs Hindsight-Zero by prompting the frozen model with both question and ground-truth answer to complete the reasoning process, filters to trajectories containing the answer, fine-tunes separate CoT and knowledge generators on those traces, and applies Knowledge Encouragement Pr

What carries the argument

The load-bearing object is the hindsight teacher: a mapping f(X, Y) -> R that completes the reasoning path between a question-image pair and its ground-truth answer. Because the same 7B model generates these traces and later learns them, the authors call it self-distillation; the crucial sub-mechanism is Knowledge Encouragement Preference Optimization, which uses normalized perplexity as a confidence score and, via a preference-optimization objective, rewards low-confidence knowledge that contains the answer over high-confidence knowledge that does not. This re-calibrates the knowledge generator toward vision-language-independent facts.

Load-bearing premise

The entire training signal is filtered through the ground-truth answer, so the central premise is that answer-conditioned hindsight teaches a transferable reasoning skill and not just a repetition of answer-containing phrasing—if that premise fails, the test-time gains will vanish on questions with answers the model cannot produce.

What would settle it

Take a held-out set of OK-VQA questions whose ground-truth answers appear rarely in the training set; if HinD's knowledge generator achieves PRR@K and VQA scores comparable to the overall 94.3/68.3 on that subset, the internal-knowledge claim survives; if scores collapse to near-zero-shot levels, the method is reproducing answer-surrounding surface patterns rather than reasoning.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • The knowledge a 7B model already holds is sufficient for strong performance on OK-VQA and A-OKVQA; external retrieval is not required for these benchmarks.
  • Answer-conditioned, self-generated reasoning traces are a nearly free source of reasoning supervision, replacing expensive human annotations or larger teacher models.
  • Aligning generation confidence with answer relevance via preference optimization improves both knowledge recall and final VQA accuracy.
  • In an out-of-domain check, the framework trained on A-OKVQA transfers to ScienceQA, suggesting the elicited reasoning skill generalizes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the filtering step (keeping only answer-containing traces) could inflate PRR@K and final scores by training the generator to imitate answer-surrounding text; the paper's own limitation section concedes that knowledge absent from the model's parameters would still fail, which is a concrete bound on how far the claim extends.
  • Editorial inference: the confidence-encouragement signal could be reapplied at inference time—for example, by sampling more knowledge and selecting by confidence rather than by majority vote—which might improve the knowledge-shot trade-off shown in the paper.
  • Editorial inference: if the method is genuinely eliciting latent knowledge, its failure mode should be sharp on long-tail answer distributions; a test split stratified by answer frequency would separate real reasoning from memorized phrasing.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper addresses knowledge-based VQA with a 7B MLLM (Qwen2.5-VL-7B) without retrieval or commercial APIs. The proposed HinD framework first prompts the frozen MLLM with the ground-truth answer to produce structured reasoning traces (Hindsight-Zero), filters traces that contain the answer, and distills them into separate CoT and Knowledge generators via HDFT. A DPO-style KEPO step biases the Knowledge Generator toward low-confidence answer-containing knowledge and away from high-confidence non-answer knowledge. An Answer Generator is fine-tuned to output the answer given the generated CoT and sampled knowledge, with self-consistency at inference. On OK-VQA and A-OKVQA, the authors report state-of-the-art or competitive scores, e.g., 68.3 VQA score on OK-VQA and 87.2 MC / 69.0 DA on A-OKVQA, and include ablations, OOD transfer to ScienceQA, and hyper-parameter analyses.

Significance. If the effect were demonstrated, this would be a practically valuable result: it would show that a 7B MLLM's internal knowledge, elicited by answer-conditioned hindsight distillation and preference optimization, can match or beat much larger API models and retrieval-augmented systems on prominent KBVQA benchmarks. The paper includes useful diagnostics (confidence-correctness joint distributions, confusion matrices, OOD transfer, hyper-parameter sensitivity) that go beyond a single accuracy number. However, the absence of code/data and the lack of variance estimates in the main tables, together with the answer-containment filtering in training, leave the central claim — that the gains reflect transferable elicited reasoning rather than answer-copying — insufficiently supported. The OOD ScienceQA result is the strongest evidence for transfer, but it uses the same answer-conditioned generators and does not isolate the mechanism.

major comments (3)
  1. [§3.3–3.4, Eq. (2), Eq. (4), Table 1] PRR@K, used to argue that HinD generates 'more valuable prompting contexts' (§4.2), is partially constructed by the training procedure. The Knowledge Generator is fine-tuned only on knowledge pieces sampled from K_hit (Eq. 2), i.e., pieces that contain the ground-truth answer, and KEPO explicitly selects winners from K_hit and losers from K_miss (Eq. 4). Consequently, the 94.3% PRR@K in Table 1 largely measures how well the model reproduces the answer-containment filter used to build its own training set, not an independent assessment of knowledge quality. Retrieval-augmented baselines such as ReAuSE (92.6 PRR@K) were not trained with this filter, so the PRR@K comparison is not on equal footing. Please report PRR@K for a variant trained without answer-containment filtering, and separate the effect of the filter from the effect of HDFT/KEPO.
  2. [§4.6, Fig. 5; §3.5 Eq. (6)] The confusion matrix in Fig. 5 shows that when the generated context does not contain any ground-truth answer, accuracy falls to ~10.6% (HinD-Know) and ~11.8% (HinD-CoT-Know), while it is ~76–77% when the answer is contained. This is consistent with the Answer Generator learning to locate and copy the answer phrase from the generated knowledge/CoT, rather than performing multi-step reasoning. Since the Answer Generator is trained on contexts that include answer-containing hindsight data (Eq. 6), the method may simply inherit the answer from the context. The paper's claim of 'elicited reasoning' (Abstract, §4.2) requires a control: e.g., a span-extraction baseline that selects the answer-like phrase from generated knowledge, or an Answer Generator trained/evaluated only on contexts that never contain the answer. Without such controls, the VQA gains can be attributed to improved answer rec
  3. [Tables 1–3; §4.2] Main results are reported as point estimates without error bars or multiple seeds. The only variance reported is for the w/o SC row in Table 3 (±0.2 over 5 runs). Because the method involves stochastic sampling (temperature 0.7, top-p, self-consistency) and LoRA fine-tuning, the claimed improvements of 1.3–2.6% over ReAuSE/QACap and 2.2% over PaLM-E may be within run-to-run noise. Please report mean and std over at least 3 seeds for Tables 1 and 2, or a paired significance test against the strongest baselines. This is necessary to support the headline 'superior performance' claim.
minor comments (5)
  1. [§3.3, Figure 2] The word 'Hindsignt' appears to be a typo for 'Hindsight' in multiple places, including the Section 3.3 heading and Figure 2.
  2. [Tables 1–2] Model names are inconsistently spaced: 'LLaV A' should be 'LLaVA', and 'Minigpt4' should be 'MiniGPT-4'. Please standardize.
  3. [Appendix G] The Limitations section already acknowledges that the framework fails for knowledge absent from the MLLM parameters and calls for a hybrid RAG approach. This is in tension with the abstract's 'without outside knowledge' phrasing and should be reconciled in the main text.
  4. [§4.1, PRR@K] The definition of 'contain' for PRR@K is not stated precisely (substring match? token match? normalized answer forms?). Please specify the matching criterion used to determine whether a knowledge piece or CoT contains the ground-truth answer.
  5. [Reproducibility] No link to code or data is provided. Given that the method depends on the generated Hindsight-Zero datasets and filtering logic, releasing code, prompts, and filtered data would be important for reproducibility.

Circularity Check

2 steps flagged

PRR@K is partly circular (training selects answer-containing knowledge; metric counts answer-containing knowledge), but the held-out VQA scores are external and not reduced.

specific steps
  1. fitted input called prediction [Sec. 3.3, Eq. (2); Sec. 4.1, PRR@K definition]
    "After obtaining the knowledge set K0 in Hindsight-Zero, we split the knowledge pieces into those hitting the ground truth answer and those not, as K0 = K0_hit ∪ K0_miss. For fine-tuning, we randomly select one of the hit knowledge ∗R0 Know ∼ U(Khit) as the training objective. ... we report Pseudo-relevance Recall@K (PRR@K) [30] to evaluate whether K pieces of knowledge ... contain any human-annotated answers."

    The Knowledge Generator is trained only on pieces already filtered to contain the ground-truth answer, i.e., K_hit. PRR@K then measures exactly whether sampled knowledge contains human-annotated answers — the same criterion used to build the training labels. High PRR@K (94.3) is therefore largely a report of how well the model matched its training filter, not independent evidence that HinD 'generates more valuable prompting contexts' via elicited reasoning. The CoT Generator is analogous: only trajectories that incorporate the ground-truth answer are kept (Sec. 3.3), so CoT PRR@K is also partly a training-fit metric.

  2. self definitional [Sec. 1; Sec. 3.4, Eq. (4)]
    "we optimize the Knowledge Generator with KEPO to prefer the less-confident but helpful knowledge, i.e., ground-truth answer containing, over the over-confident but unhelpful ones. ... The Kw is the winning knowledge with the minimum confidence in Khit, while the Kl is the losing knowledge with the maximum confidence in Kmiss."

    'Helpful' is explicitly defined as containing the ground-truth answer, and KEPO's preference pairs are exactly answer-containing (K_hit) versus non-answer-containing (K_miss) samples. The DPO objective (Eq. 5) therefore directly optimizes the property that PRR@K measures. Reporting the KEPO-induced PRR@K gain (85.0 -> 94.3 in Table 3) as evidence of better knowledge is circular: the optimization target and the evaluation metric are the same set-membership condition.

full rationale

This is not a derivation paper, and the headline result — 68.3 VQA score on OK-VQA, 69.0 DA / 87.2 MC on A-OKVQA — is an external held-out benchmark evaluation, so it is not circular. The training signal uses ground-truth answers only on the training set to create Hindsight-Zero supervision, then distills a student that must generate reasoning at test time without the answer. The main circularity is confined to the PRR@K reporting: the paper filters and optimizes the generators to emit answer-containing knowledge, then evaluates them with PRR@K, which is defined as the fraction of generated contexts containing human-annotated answers. That is a by-construction reduction of the metric to the training objective, so the claim that high PRR@K demonstrates superior reasoning is only partially supported. The paper's own confusion matrix (Fig. 5) and limitation statement — 'when HinD generates contexts without answers, the answer generator cannot infer the answer' — flag a genuine generalization risk (answer copying from hit contexts), but that is a correctness/robustness concern, not an additional circularity. No load-bearing self-citations or imported uniqueness theorems are present. Score 4 reflects one partial, metric-level circularity while the central VQA claim remains independently testable.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The central claim rests on four unevidenced premises: answer-conditioned prompting produces transferable reasoning; answer-containment labels helpfulness; token-probability confidence aligns with usefulness; the backbone already contains the needed facts. None are derived; the paper gives empirical support via ablations but no external validation of these assumptions.

free parameters (6)
  • KEPO coefficient beta = 0.1
    Tuned on OK-VQA (Supplementary Table 5); controls KL constraint in DPO loss (Eq. 5).
  • Knowledge sampling temperature = 0.7
    Tuned on OK-VQA (Table 5); temperature=1.0 gave higher PRR@K but 0.7 chosen as default.
  • Number of sampled knowledge pieces |K_hat| = 5
    Chosen to mimic 5 retrieved documents in retrieval baselines (Sec 4.1); Table 4 shows score varies with |K_hat|.
  • Self-consistency runs = 5
    Chosen for major-vote decoding; ablation w/o SC reports 66.0 +/- 0.2 (Table 3).
  • Answer generation temperature = 1e-5
    Hand-chosen low temperature for answer certainty (Sec 4.1).
  • Decoding hyperparameters top-p/top-k = 0.95 / 50
    Default knowledge sampling parameters (Sec 4.1); chosen by hand, not tuned.
axioms (5)
  • domain assumption Answer-conditioned prompting of a 7B MLLM produces high-quality reasoning trajectories that can be distilled into a model that reasons without the answer.
    Core premise of Hindsight-Zero (Section 3.2); if false, HDFT teaches imitation of answer-laden text rather than transferable reasoning.
  • ad hoc to paper Containing the ground-truth answer is a valid proxy for a helpful CoT/knowledge piece.
    Used to filter CoT and knowledge HDFT data (Sec 3.3) and to define K_hit/K_miss for KEPO (Eq 4); ignores cases where answer-containing text is verbose/noisy and non-containing text is useful.
  • ad hoc to paper Confidence, measured as mean token probability (Eq 3), is a meaningful signal of whether generated knowledge is under-/over-confident relative to helpfulness.
    Underlies the choice of Kw/Kl in KEPO (Eq 4); heuristic with no theoretical guarantee.
  • domain assumption The backbone Qwen2.5-VL-7B contains enough world knowledge for OK-VQA/A-OKVQA answers; failures are due to elicitation/calibration, not absent knowledge.
    The paper's own limitation (Supplementary G) admits failure when knowledge is absent; for the main claim, that the knowledge is in the parameters is assumed by 'without external knowledge'.
  • standard math DPO and LoRA behave as specified in [16,33] and self-distillation reduces distribution shift [49].
    Background method assumptions used without proof; reasonable but not verified in this paper.

pith-pipeline@v1.3.0-alltime-deepseek · 19388 in / 15292 out tokens · 126565 ms · 2026-08-03T22:15:02.177931+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of From Hindsight to Foresight: Self-Encouraged Hindsight Distillation for Knowledge-based Visual Question Answering." pith.science (2026). https://pith.science/paper/RGHNV7UQ

@misc{pith2026251111132,
  author       = {Pith},
  title        = {Pith review of: From Hindsight to Foresight: Self-Encouraged Hindsight Distillation for Knowledge-based Visual Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RGHNV7UQ}},
  note         = {Machine review of arXiv:2511.11132}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Knowledge-based Visual Question Answering (KBVQA) necessitates external knowledge incorporation beyond cross-modal understanding. Existing KBVQA methods either utilize implicit knowledge in multimodal large language models via in-context learning or explicit knowledge via retrieval augmented generation. However, their reasoning processes remain implicit, without explicit multi-step trajectories. To address this gap, we propose a Self-Encouraged Hindsight Distillation Reasoning (HinD) framework, aiming at eliciting reasoning ability inside the MLLM by constructing a Hindsight Teacher with privileged information to teach the Foresight Student. First, we construct the Hindsight Teacher by prompting the MLLM with the reasoning target as privileged information to complete the reasoning process, obtaining Hindsight-Zero training data. Then, the Foresight Student, without knowing the answer, learns the golden trajectories from Hindsight in two ways: (1) Hindsight Distillation Fine-Tuning to self-distill the Hindsight-Zero into a modularized Chain-of-Thought Generator and a Knowledge Generator for sequential steps and discrete facts generation, respectively; (2) Knowledge Encouragement Preference Optimization to encourage the under-confident but relevant knowledge inside the MLLM and suppress the over-confident but irrelevant one. Experiments on OK-VQA and A-OKVQA validate the effectiveness of HinD, showing that HinD with 7-8B MLLM achieves superior performance without commercial model APIs or retrieved knowledge.

Figures

Figures reproduced from arXiv: 2511.11132 by Baohang Zhou, Dacheng Tao, Li Shen, Xinying Qian, Xuhui Sui, Ying Zhang, Yu Zhao.

Figure 1
Figure 1. Figure 1: KBVQA requires necessary knowledge incorpora [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Our Hindsight Distilled Reasoning (HinD) framework for KBVQA: (1) Constructing training data Hindsight-Zero by [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The joint distribution of Knowledge Generator’s confidence [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Case study on OK-VQA test set. The answer heuristics candidates are from the smaller VQA model MCAN in Prophet [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Confusion matrix on OK-VQA test set. The ver [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 11 linked inside Pith

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023. 1, 2, 3, 5

    Pith/arXiv arXiv 2023

  2. [2]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. InPro- ceedings of the IEEE international conference on com- puter vision, pages 2425–2433, 2015. 1, 2

    2015

  3. [3]

    Language models are few-shot learners.Ad- vances in neural information processing systems, 33: 1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Ad- vances in neural information processing systems, 33: 1877–1901, 2020. 1, 3

    1901

  4. [4]

    Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming-Wei Chang. Can pre-trained vision and language models answer visual information-seeking questions? InProceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, pages 14948–14968, 2023. 2

    2023

  5. [5]

    Visual chain-of-thought prompting for knowledge-based visual reasoning

    Zhenfang Chen, Qinhong Zhou, Yikang Shen, Yining Hong, Zhiqing Sun, Dan Gutfreund, and Chuang Gan. Visual chain-of-thought prompting for knowledge-based visual reasoning. InProceedings of the AAAI Conference on Artificial Intelligence, pages 1254–1262, 2024. 2, 3, 5

    2024

  6. [6]

    Instructblip: Towards general- purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. Advances in neural information processing systems, 36: 49250–49267, 2023. 3

    2023

  7. [7]

    Modality-aware inte- gration with large language models for knowledge-based visual question answering

    Junnan Dong, Qinggang Zhang, Huachi Zhou, Daochen Zha, Pai Zheng, and Xiao Huang. Modality-aware inte- gration with large language models for knowledge-based visual question answering. In62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024, pages 2417–2429. Association for Computational Linguistics (ACL), 2024. 3, 5

    2024

  8. [8]

    Palm- e: an embodied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm- e: an embodied multimodal language model. InProceed- ings of the 40th International Conference on Machine Learning, pages 8469–8488, 2023. 5

    2023

  9. [9]

    Notes-guided mllm reasoning: Enhancing mllm with knowledge and visual notes for visual question answer- ing

    Wenlong Fang, Qiaofeng Wu, Jing Chen, and Yun Xue. Notes-guided mllm reasoning: Enhancing mllm with knowledge and visual notes for visual question answer- ing. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19597–19607, 2025. 3

    2025

  10. [10]

    Transform- retrieve-generate: Natural language-centric outside- knowledge visual question answering

    Feng Gao, Qing Ping, Govind Thattai, Aishwarya Re- ganti, Ying Nian Wu, and Prem Natarajan. Transform- retrieve-generate: Natural language-centric outside- knowledge visual question answering. InProceedings of the IEEE/CVF conference on computer vision and pat- tern recognition, pages 5067–5077, 2022. 3, 5

    2022

  11. [11]

    Conceptbert: Concept-aware repre- sentation for visual question answering

    Franc ¸ois Gard`eres, Maryam Ziaeefard, Baptiste Abeloos, and Freddy Lecue. Conceptbert: Concept-aware repre- sentation for visual question answering. InFindings of the Association for Computational Linguistics: EMNLP 2020, pages 489–498, 2020. 3

    2020

  12. [12]

    Making the v in vqa matter: El- evating the role of image understanding in visual ques- tion answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: El- evating the role of image understanding in visual ques- tion answering. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 6904– 6913, 2017. 1, 2

    2017

  13. [13]

    Kat: A knowledge augmented transformer for vision- and-language

    Liangke Gui, Borui Wang, Qiuyuan Huang, Alexan- der G Hauptmann, Yonatan Bisk, and Jianfeng Gao. Kat: A knowledge augmented transformer for vision- and-language. InProceedings of the 2022 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, pages 956–968, 2022. 1, 3, 5

    2022

  14. [14]

    Deepseek-r1: Incentivizing reason- ing capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reason- ing capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025. 3

    Pith/arXiv arXiv 2025

  15. [15]

    Self-bootstrapped visual-language model for knowledge selection and question answering

    Dongze Hao, Qunbo Wang, Longteng Guo, Jie Jiang, and Jing Liu. Self-bootstrapped visual-language model for knowledge selection and question answering. InPro- ceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, pages 1857–1868,

    2024

  16. [16]

    Lora: Low-rank adaptation of large language mod- els.ICLR, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language mod- els.ICLR, 1(2):3, 2022. 5

    2022

  17. [17]

    Promptcap: Prompt- guided image captioning for vqa with gpt-3

    Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo. Promptcap: Prompt- guided image captioning for vqa with gpt-3. InProceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 2963–2975, 2023. 1, 2, 5

    2023

  18. [18]

    Reveal: Retrieval-augmented visual-language pre-training with multi-source multi- modal knowledge memory

    Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai- Wei Chang, Yizhou Sun, Cordelia Schmid, David A Ross, and Alireza Fathi. Reveal: Retrieval-augmented visual-language pre-training with multi-source multi- modal knowledge memory. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 23369–23379, 2023. 5

    2023

  19. [19]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Ak- ila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024. 1, 2, 3

    Pith/arXiv arXiv 2024

  20. [20]

    Large language models know what is key visual entity: An llm-assisted multimodal retrieval for vqa

    Pu Jian, Donglei Yu, and Jiajun Zhang. Large language models know what is key visual entity: An llm-assisted multimodal retrieval for vqa. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10939–10956, 2024. 5 9

    2024

  21. [21]

    Corvid: Improving multimodal large lan- guage models towards chain-of-thought reasoning.arXiv preprint arXiv:2507.07424, 2025

    Jingjing Jiang, Chao Ma, Xurui Song, Hanwang Zhang, and Jun Luo. Corvid: Improving multimodal large lan- guage models towards chain-of-thought reasoning.arXiv preprint arXiv:2507.07424, 2025. 2, 3

    Pith/arXiv arXiv 2025

  22. [22]

    Mm-reasoner: A multi-modal knowledge-aware framework for knowledge-based vi- sual question answering

    Mahmoud Khademi, Ziyi Yang, Felipe Frujeri, and Chenguang Zhu. Mm-reasoner: A multi-modal knowledge-aware framework for knowledge-based vi- sual question answering. InFindings of the Associa- tion for Computational Linguistics: EMNLP 2023, pages 6571–6581, 2023. 1, 2, 5

    2023

  23. [23]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR, 2023. 3

    2023

  24. [24]

    Retrieval augmented vi- sual question answering with outside knowledge

    Weizhe Lin and Bill Byrne. Retrieval augmented vi- sual question answering with outside knowledge. InPro- ceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11238–11254,

    2022

  25. [25]

    Fine-grained late-interaction multi-modal retrieval for retrieval augmented visual question answering.Advances in Neural Information Processing Systems, 36:22820–22840, 2023

    Weizhe Lin, Jinghong Chen, Jingbiao Mei, Alexandru Coca, and Bill Byrne. Fine-grained late-interaction multi-modal retrieval for retrieval augmented visual question answering.Advances in Neural Information Processing Systems, 36:22820–22840, 2023. 1, 3, 5, 7

    2023

  26. [26]

    Revive: Regional visual representation matters in knowledge-based visual ques- tion answering.Advances in neural information process- ing systems, 35:10560–10571, 2022

    Yuanze Lin, Yujia Xie, Dongdong Chen, Yichong Xu, Chenguang Zhu, and Lu Yuan. Revive: Regional visual representation matters in knowledge-based visual ques- tion answering.Advances in neural information process- ing systems, 35:10560–10571, 2022. 3, 5

    2022

  27. [27]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 5

    2023

  28. [28]

    Retrieval-augmented visual question answering via built-in autoregressive search en- gines

    Xinwei Long, Zhiyuan Ma, Ermo Hua, Kaiyan Zhang, Biqing Qi, and Bowen Zhou. Retrieval-augmented visual question answering via built-in autoregressive search en- gines. InProceedings of the AAAI Conference on Artifi- cial Intelligence, pages 24723–24731, 2025. 1, 3, 5, 7, 4, 6

    2025

  29. [29]

    Learn to explain: Multi- modal reasoning via thought chains for science question answering.Advances in Neural Information Processing Systems, 35:2507–2521, 2022

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multi- modal reasoning via thought chains for science question answering.Advances in Neural Information Processing Systems, 35:2507–2521, 2022. 2, 3, 4

    2022

  30. [30]

    Weakly-supervised visual-retriever-reader for knowledge-based question answering.arXiv preprint arXiv:2109.04014, 2021

    Man Luo, Yankai Zeng, Pratyay Banerjee, and Chitta Baral. Weakly-supervised visual-retriever-reader for knowledge-based question answering.arXiv preprint arXiv:2109.04014, 2021. 3, 5, 7

    Pith/arXiv arXiv 2021

  31. [31]

    Ok-vqa: A visual question answer- ing benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answer- ing benchmark requiring external knowledge. InPro- ceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019. 1, 2, 3, 5

    2019

  32. [32]

    Krisp: Integrating implicit and symbolic knowledge for open-domain knowledge- based vqa

    Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, and Marcus Rohrbach. Krisp: Integrating implicit and symbolic knowledge for open-domain knowledge- based vqa. InProceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pages 14111–14121, 2021. 3

    2021

  33. [33]

    Di- rect preference optimization: Your language model is se- cretly a reward model.Advances in neural information processing systems, 36:53728–53741, 2023

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. Di- rect preference optimization: Your language model is se- cretly a reward model.Advances in neural information processing systems, 36:53728–53741, 2023. 5

    2023

  34. [34]

    Exploring the limits of transfer learn- ing with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learn- ing with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020. 3

    2020

  35. [35]

    A- okvqa: A benchmark for visual question answering using world knowledge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A- okvqa: A benchmark for visual question answering using world knowledge. InEuropean conference on computer vision, pages 146–162. Springer, 2022. 2, 3, 5, 8

    2022

  36. [36]

    Kvqa: Knowledge-aware visual question answering

    Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar. Kvqa: Knowledge-aware visual question answering. InProceedings of the AAAI confer- ence on artificial intelligence, pages 8876–8884, 2019. 1

    2019

  37. [37]

    Combo of thinking and observing for outside-knowledge vqa

    Qingyi Si, Yuchen Mo, Zheng Lin, Huishan Ji, and Weiping Wang. Combo of thinking and observing for outside-knowledge vqa. InProceedings of the 61st An- nual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 10959–10975,

  38. [38]

    Con- ceptnet 5.5: An open multilingual graph of general knowledge

    Robyn Speer, Joshua Chin, and Catherine Havasi. Con- ceptnet 5.5: An open multilingual graph of general knowledge. InProceedings of the AAAI conference on artificial intelligence, 2017. 3

    2017

  39. [39]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    Qwen Team. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024. 5

    Pith/arXiv arXiv 2024

  40. [40]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 1, 2

    Pith/arXiv arXiv 2023

  41. [41]

    Wikidata: a free collaborative knowledgebase.Communications of the ACM, 57(10):78–85, 2014

    Denny Vrande ˇci´c and Markus Kr ¨otzsch. Wikidata: a free collaborative knowledgebase.Communications of the ACM, 57(10):78–85, 2014. 3

    2014

  42. [42]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. InThe Eleventh Interna- tional Conference on Learning Representations, 2022. 5

    2022

  43. [43]

    Mul- timodal chain-of-thought reasoning: A comprehensive survey.arXiv preprint arXiv:2503.12605, 2025

    Yaoting Wang, Shengqiong Wu, Yuecheng Zhang, Shuicheng Yan, Ziwei Liu, Jiebo Luo, and Hao Fei. Mul- timodal chain-of-thought reasoning: A comprehensive survey.arXiv preprint arXiv:2503.12605, 2025. 2, 3

    Pith/arXiv arXiv 2025

  44. [44]

    Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022. 2, 3

    2022

  45. [45]

    Multi-modal answer validation for 10 knowledge-based vqa

    Jialin Wu, Jiasen Lu, Ashish Sabharwal, and Roozbeh Mottaghi. Multi-modal answer validation for 10 knowledge-based vqa. InProceedings of the AAAI conference on artificial intelligence, pages 2712–2721,

  46. [46]

    A simple baseline for knowledge-based visual question answering

    Alexandros Xenos, Themos Stafylakis, Ioannis Patras, and Georgios Tzimiropoulos. A simple baseline for knowledge-based visual question answering. InPro- ceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14871–14877,

    2023

  47. [47]

    Llava-cot: Let vision language models reason step-by-step.arXiv preprint arXiv:2411.10440,

    Guowei Xu, Peng Jin, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by-step.arXiv preprint arXiv:2411.10440,

    Pith/arXiv arXiv

  48. [48]

    An empiri- cal study of gpt-3 for few-shot knowledge-based vqa

    Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empiri- cal study of gpt-3 for few-shot knowledge-based vqa. In Proceedings of the AAAI conference on artificial intelli- gence, pages 3081–3089, 2022. 2

    2022

  49. [49]

    Self-distillation bridges distribution gap in language model fine-tuning

    Zhaorui Yang, Tianyu Pang, Haozhe Feng, Han Wang, Wei Chen, Minfeng Zhu, and Qian Liu. Self-distillation bridges distribution gap in language model fine-tuning. InProceedings of the 62nd Annual Meeting of the Asso- ciation for Computational Linguistics (Volume 1: Long Papers), pages 1028–1043, 2024. 3, 4

    2024

  50. [50]

    Separation of powers: On segregating knowledge from observation in llm-enabled knowledge-based visual question answer- ing

    Zhen Yang, Zhuo Tao, Qi Chen, Liang Li, Yuankai Qi, Anton van den Hengel, and Qingming Huang. Separation of powers: On segregating knowledge from observation in llm-enabled knowledge-based visual question answer- ing. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 24753–24762, 2025. 1, 5

    2025

  51. [51]

    Mulberry: Empow- ering mllm with o1-like reasoning and reflection via collective monte carlo tree search.arXiv preprint arXiv:2412.18319, 2024

    Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang, Yuxin Song, Haocheng Feng, Li Shen, et al. Mulberry: Empow- ering mllm with o1-like reasoning and reflection via collective monte carlo tree search.arXiv preprint arXiv:2412.18319, 2024. 2, 3

    Pith/arXiv arXiv 2024

  52. [52]

    Zhou Yu, Xuecheng Ouyang, Zhenwei Shao, Meng Wang, and Jun Yu. Prophet: Prompting large lan- guage models with complementary answer heuristics for knowledge-based visual question answering.IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 2025. 1, 2, 3, 5, 7, 8

    2025

  53. [53]

    Multimodal chain-of- thought reasoning in language models.Transactions on Machine Learning Research, 2024, 2024

    Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of- thought reasoning in language models.Transactions on Machine Learning Research, 2024, 2024. 3, 5

    2024

  54. [54]

    Llamafactory: Unified efficient fine-tuning of 100+ language models

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yan- han Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguis- tics (Volume 3: System Demonstrations), Bangkok, Thai- land, 2024. Association for Computational Lin...

    2024

  55. [55]

    Chosen” and “Rejected

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023. 5 11 Hindsight Distillation Reasoning with Knowledge Encouragement Preference for Knowledge-based Visual Question Answering Supplementary Material A. Hyper-param...

    Pith/arXiv arXiv 2023