Pith. sign in

REVIEW 3 major objections 5 minor 15 references

Solution for Meta KDD Cup'25: A Comprehensive Three-Step Framework for Vision Question Answering

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A three-step RAG framework with reranking ranks first on CRAG-MM Task3.

desk verdict Credible competition results with real leaderboard evidence, but the ablation claims are single-shot and the GPT-4o oracle is the main reliability question. read the letter →

arxiv 2507.21520 v1 pith:AWHMKWFP submitted 2025-07-29 cs.IR

Zijian Zhang , Xiaocheng Zhang , Yang Zhou , Zhimin Lin , Peng Yan This is my paper
classification cs.IR
keywords multimodalretrieval-augmentedgenerationvisualquestionansweringCRAG-MMKDDCup2025hallucinationmitigationrerankingmulti-taskfine-tuningLoRA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports a competition solution for Meta's CRAG-MM Challenge, a multimodal retrieval-augmented question answering benchmark built around 5,000 images and multi-turn evaluation. The authors argue that a single Llama-3.2-11B-Vision-Instruct model, fine-tuned with LoRA and trained on three sub-tasks, is sufficient to reach the top automatic leaderboard positions: 3rd on Task1, 3rd on Task2, and 1st on Task3, with a second-place finish in Task3 after human evaluation. The load-bearing components are image retrieval with extracted text context, a data-augmentation loop that verifies answers with GPT-4o mini and converts apparent hallucinations to "I don't know", multi-query web retrieval, and a listwise reranking step. If correct, the paper shows that the largest single improvement on Task3 came from constructing refusal data, moving the score from 0.0700 to 0.1322, with later additions reaching 0.1755.

What carries the argument

The central object is a single Llama-3.2-11B-Vision-Instruct model adapted with LoRA (low-rank adaptation) to perform three sub-tasks in one pass: generating diverse retrieval queries, reranking retrieval results listwise into an ordered list of relevant identifiers, and producing the final answer. Around this model, the pipeline builds a retrieval step that keeps the result group with the most hits from several sampled queries, a reranking prompt that asks for "[x, xx, xxx, ...]" or "[]", and a multi-task fine-tuning set where the QA label is replaced by the model's own answer when GPT-4o mini confirms consistency with the ground truth, or by "I don't know" when it does not. For Task1, image retrieval is converted to extracted text context, and data augmentation generates 20 paraphrased labels per sample after hallucination filtering.

What would settle it

Take a random sample of the marked samples from the paper's augmentation loop and have two human annotators re-label them as hallucinated or consistent; if agreement with GPT-4o mini is low, the "I don't know" conversions are not as clean as the ablation implies. A cheaper check is to rerun the pipeline with a different judge model (or no judge) and see whether the Task3 gap from 0.0700 to 0.1755 reproduces.

Watch

Extended reading notes

Core claim

The paper's central claim is that its three-step framework—query generation, retrieval-result reranking, and answer generation carried out by one multi-task fine-tuned model—is a competitive recipe for multimodal RAG QA under the CRAG-MM evaluation rules. On the task with multi-turn conversations, the paper reports that fine-tuning on the plain RAG data yields 0.0700, adding refusal-data construction raises it to 0.1322, multi-query retrieval to 0.1471, reranking to 0.1505, and reducing the refusal-data proportion to 0.1755, enough to rank first in automatic evaluation. It also claims automatic third place on the single-source and multi-source tasks and second place in Task3 after human correction of automatic scores.

Load-bearing premise

Everything rests on GPT-4o mini correctly deciding which model answers are hallucinations and which generated labels agree with the ground truth; if that oracle is unreliable, the training labels become noisy and the reported leaderboard gains may not transfer.

Editorial extensions

If this is right

  • Refusal-data construction is worth a 0.0622-point jump on Task3, from 0.0700 to 0.1322, making abstention a trainable behavior under the CRAG scoring rubric.
  • One 11B model can serve query generation, reranking, and answering, so the whole recipe needs a single deployment within the 30-second limit.
  • Keeping the retrieval-result group with the most hits and reranking to the top 10 fits the 8192-token input cap while preserving recall.
  • Multi-turn Task3 re-enters the retrieval step with history, so the framework extends directly to conversational QA.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to swap the GPT-4o mini oracle for a deterministic or self-consistency check, which would show whether the refusals can be produced without a proprietary judge.
  • If abstention generalizes, there is an optimal abstention rate: converting a hallucination from -1 to 0 helps, but giving up too many answers sacrifices valid correct answers, which may explain the final reduction in refusal-data proportion.
  • The listwise reranking module is a transferable piece, since any retrieval-augmented system with more passages than fit in the context window needs a selection mechanism.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper describes the BlackPearl team's solution for the three tasks of the CRAG-MM challenge at KDD Cup 2025. For Task 1, the solution combines official image retrieval with extraction of associated textual information, data augmentation guided by a GPT-4o mini judge, and LoRA fine-tuning of Llama-3.2-11B-Vision-Instruct. For Tasks 2 and 3, the pipeline generates multiple retrieval queries from the Llama model, performs web retrieval, applies a listwise reranking module, and uses multi-task fine-tuning on query-generation, reranking, and QA data, with GPT-4o mini also used to decide whether model-generated answers can replace ground-truth labels. The authors report automatic evaluation rankings of 3rd, 3rd, and 1st on the three tasks, and 2nd in Task 3 after human evaluation. Ablation results in Tables 5 and 6 attribute score improvements to data augmentation, refusal-data construction, multi-query retrieval, and reranking, with the Task 3 score rising from 0.0700 to 0.1755.

Significance. If the reported rankings and ablations hold, this is a competitive and reproducible pipeline for multimodal retrieval-augmented question answering, with practical insights into refusal-data construction, multi-query retrieval, and reranking. The paper's main strengths are its use of externally measured leaderboard evaluations, clear descriptions of the engineering components, and publicly available code. The main limitation is that the central internal claims about component effectiveness rest on unvalidated GPT-4o mini oracle decisions and on single-score ablations without uncertainty quantification. The work is therefore more a well-documented competition solution than a fully supported research contribution, but the externally measured results give it value.

major comments (3)
  1. [Sections 2.2 and 3.3; Tables 5 and 6] The paper reports no agreement analysis between GPT-4o mini and the official evaluator, and no sensitivity analysis with respect to the oracle's leniency. Because the ablation gains in Table 5 (0.01 to 0.043) and Table 6 (0.0700 to 0.1755) depend directly on labels produced or accepted by this oracle, the reported improvements may reflect the oracle's biases rather than a generalizable method.
  2. [Section 4.3; Tables 5 and 6] The authors should report multiple runs, cross-validation, or at least a variance estimate over the validation split before claiming that each component 'significantly improved the scores.'
  3. [Section 3.3] The paper should report a retrieval-success rate (for example, the fraction of generated queries that return nonempty or relevant results) and, ideally, compare against a baseline of fixed or human-written queries.
minor comments (5)
  1. [Abstract and Section 1] Grammar issues such as 'Our solution achieve' and 'win second place' should be corrected to 'achieves' and 'won second place.'
  2. [Table 4] The scores in Table 4 (e.g., 30.9 for BlackPearl in Task 3) are not explained relative to the 0-1 scores in Tables 1-3; if they are percentages, the relationship should be stated explicitly.
  3. [Section 2.2 and Appendix A.2] Section 2.2 states that GPT-4o mini generates 10 similar labels (n=10), but the prompt in Figure 5 says '20 similar standard answers'; the number should be made consistent.
  4. [Section 3.3] The phrase 'mLLama's LoRA weights' is unclear; the base model is referred to elsewhere as Llama-3.2-11B-Vision-Instruct, and the terminology should be unified.
  5. [Appendix A.6] The sentence 'And the correct answer is satisfied following rules' in the VQA prompt is ungrammatical and should read 'the correct answer should satisfy the following rules.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported rankings and ablation improvements are grounded in external official evaluation, not in fitted or self-referential quantities.

full rationale

The paper contains no mathematical derivation or first-principles claim whose output is equivalent to its input. The central claims are the automatic-evaluation rankings (3rd, 3rd, 1st) and the post-human-evaluation second place in Task3, plus the ablation progression in Table 6 (0.0700 to 0.1755). These are measured by the competition's official rubric described in Section 4.1 (Perfect/Acceptable/Missing/Incorrect scoring) and by the organizers' human evaluation, so they are not defined in terms of the model's own outputs or GPT-4o mini judgments. The closest candidate for circularity is the self-labeling pipeline in Sections 2.2 and 3.3: Llama proposes answers, GPT-4o mini judges hallucination/consistency, and consistent model-generated answers replace ground-truth labels in training data. That is self-training, but it does not make the evaluation scores equivalent to the oracle by construction; the oracle is not used to score the test set, and the paper's claims do not reduce to a fitted parameter. There is no load-bearing self-citation chain: the cited related work ([8], [9], etc.) is external, and no uniqueness theorem is imported from the authors. The unvalidated GPT-4o mini oracle is a legitimate correctness/robustness concern, not a circularity, and should be weighed in a correctness review rather than raising the circularity score. Accordingly, the appropriate finding is no significant circularity.

Assumptions & free parameters 10 free parameters · 4 assumptions · 0 invented entities

The paper makes no derivation; its central claim is external benchmark performance. The free parameters are hand-tuned hyperparameters, several of which directly change the final scores. The axioms are domain assumptions that the benchmark, retrieval tools, and GPT-4o mini oracle behave as the authors assume. No invented entities are introduced.

free parameters (10)
  • learning_rate_task1 = 5e-5
    Hand-chosen in Section 4.1 for Task1 LoRA fine-tuning.
  • learning_rate_task2_3 = 5e-6
    Hand-chosen in Section 4.1 for Task2/3 fine-tuning.
  • training_epochs_task1 = 2
    Number of fine-tuning epochs for Task1.
  • training_epochs_task2_3 = 10
    Number of fine-tuning epochs for Task2/3.
  • LoRA_rank = 64
    Rank for LoRA adaptation; chosen to balance memory and capacity.
  • LoRA_alpha = 128
    Alpha scale for LoRA.
  • LoRA_dropout = 0.05
    Dropout factor in LoRA layers.
  • max_retrieval_results = 30
    Maximum retrieval results fed to the model in Task2/3; affects context length and answer quality.
  • query_sampling_temperature = 0.8
    Temperature for sampling multiple retrieval queries; answer generation uses temperature 0.0.
  • refusal_data_proportion = reduced
    The share of 'I don't know' labels in the training set is tuned downward in the final step, changing the Task3 score from 0.1505 to 0.1755 (Table 6). This is a hand-tuned calibration to the validation/leaderboard.
assumptions (4)
  • domain assumption The CRAG-MM benchmark and its official retrieval tools provide reliable ground-truth answers and useful retrieved content.
    The reported scores assume the official image retrieval (Section 2.1) and web retrieval (Section 3.1) tools return correct and relevant results as described.
  • domain assumption An 8:2 image-based split of the provided training data is representative of the official evaluation set.
    The paper fine-tunes on 80% and reports ablations without a separate test set; generalization to the hidden evaluation is assumed (Section 2.3, 3.3).
  • domain assumption GPT-4o mini's judgments about hallucinations and label consistency match the competition scoring rubric.
    Used as the oracle in data augmentation (Section 2.2) and refusal-data construction (Section 3.3); no validation is provided.
  • domain assumption Self-generated labels for retrieval-query generation and reranking are of sufficient quality for multi-task fine-tuning.
    The fine-tuning data for the auxiliary tasks is created by the model itself plus GPT-4o (prompts in Appendix A.4), without checking against ground truth.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Solution for Meta KDD Cup'25: A Comprehensive Three-Step Framework for Vision Question Answering." pith.science (2026). https://pith.science/paper/AWHMKWFP

@misc{pith2026250721520,
  author       = {Pith},
  title        = {Pith review of: Solution for Meta KDD Cup'25: A Comprehensive Three-Step Framework for Vision Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AWHMKWFP}},
  note         = {Machine review of arXiv:2507.21520}
}
read the original abstract

Vision Large Language Models (VLLMs) have improved multi-modal understanding and visual question answering (VQA), but still suffer from hallucinated answers. Multi-modal Retrieval-Augmented Generation (RAG) helps address these issues by incorporating external information, yet challenges remain in visual context comprehension, multi-source retrieval, and multi-turn interactions. To address these challenges, Meta constructed the CRAG-MM benchmark and launched the CRAG-MM Challenge at KDD Cup 2025, which consists of three tasks. This paper describes the solutions of all tasks in Meta KDD Cup'25 from BlackPearl team. We use a single model for each task, with key methods including data augmentation, RAG, reranking, and multi-task fine-tuning. Our solution achieve automatic evaluation rankings of 3rd, 3rd, and 1st on the three tasks, and win second place in Task3 after human evaluation.

Figures

Figures reproduced from arXiv: 2507.21520 by the authors.

Figure 1
Figure 1. Illustration of Task1 framework. 2.1 Image Retrieval We utilize the official retrieval tool for image retrieval, representing the image as a vector, and retrieving the top 10 images most similar to the current input image. Instead of directly adding the retrieved images to the model input, we first extract the textual information from the structured data associated with these images. This textual information is then… view at source ↗
Figure 3
Figure 3. Illustration of Task2 and Task3 framework. The entire process consists of three steps. ’Q’ and ’I’ represent Question [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. QA prompt for Task1. Given a question and a standard answer, please help me create 20 similar standard answers. The answers should either be simplified or made more complex, but must not exceed 50 words. Output format: 1. xx (Rule: Simplified|Complexified) 2. xx (Rule: Simplified|Complexified) 3. xx ... Question: {query} Standard answer: {ans_full} Data Augmentation Prompt for Task1 [PITH_FULL_IMAGE:figures/full_fi… view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Data augmentation prompt for Task1. You are a web retrieval and query reformulation agent. Based on the history dialog, the current original question, and the provided image, please generate a search phrase for retrieval. Image: {image} The History Dialog is: {history_…
Figure 6
Figure 6. Figure 6: Retrieval query generation prompt for Task2/3. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Prompt for reranking data construction in Task2/3. [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Rerank prompt for Task2/3. You are a helpful assistant that truthfully answers user questions about the provided image and the retrieval information. The retrieval information may not related to the provided query and image. Please pay attention to identifying that inf…
Figure 9
Figure 9. Figure 9: VQA prompt for Task2/3 [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 9 canonical work pages

  1. [1]

    Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, et al. 2024. Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. arXiv preprint arXiv:2402.14762 (2024)

    arXiv 2024

  2. [2]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 2, 1 (2023)

    arXiv 2023

  3. [3]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. ICLR 1, 2 (2022), 3

    2022

  4. [4]

    Jules King, L Burleigh, Simon Woodhead, Panagiota Kon, Perpetual Baffour, Scott Crossley, Walter Reade, and Maggie Demkin. 2024. Eedi - Mining Misconceptions in Mathematics. https://kaggle.com/competitions/eedi-mining-misconceptions- in-mathematics. Kaggle

    work page 2024

  5. [5]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAtten- tion. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles

    work page 2023

  6. [6]

    Gonzalez, Ion Stoica, Sohier Dane, Maggie Demkin, and Nate Keating

    Wei lin Chiang, Evan Frick, Lisa Dunlap, Anastasios Angelopoulos, Joseph E. Gonzalez, Ion Stoica, Sohier Dane, Maggie Demkin, and Nate Keating. 2024. WSDM Cup - Multilingual Chatbot Arena. https://kaggle.com/competitions/ wsdm-cup-multilingual-chatbot-arena. Kaggle

    work page 2024

  7. [7]

    Jielin Qiu, Andrea Madotto, Zhaojiang Lin, Paul A Crook, Yifan Ethan Xu, Xin Luna Dong, Christos Faloutsos, Lei Li, Babak Damavandi, and Seungwhan Moon. 2024. Snapntell: Enhancing entity-centric visual question answering with retrieval augmented multimodal llm. arXiv preprint arXiv:2403.04735 (2024)

    arXiv 2024

  8. [8]

    Yikuan Xia, Jiazun Chen, and Jun Gao. 2024. Winning Solution For Meta KDD Cup’24. arXiv preprint arXiv:2410.00005 (2024)

    arXiv 2024

Show all 15 references
  1. [9]

    Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Gui, Ziran Jiang, Ziyu Jiang, et al. 2024. Crag-comprehensive rag benchmark. Advances in Neural Information Processing Systems 37 (2024), 10470–10490

    2024

  2. [10]

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490 (2023). A Prompts Used in the Competition A.1 VQA Prompt fo...

    2023 arXiv

  3. [12]

    xx (Rule: Simplified|Complexified)

  4. [13]

    Question: {query} Standard answer: {ans_full} Data Augmentation Prompt for Task1 Figure 5: Data augmentation prompt for Task1

    xx ... Question: {query} Standard answer: {ans_full} Data Augmentation Prompt for Task1 Figure 5: Data augmentation prompt for Task1. You are a web retrieval and query reformulation agent. Based on the history dialog, the current original question, and the provided image, plea...

  5. [14]

    The answer is correct if it captures all the key information

  6. [15]

    The answer is correct even if phrased differently as long as the meaning is the same

  7. [16]

    For example, when answer a question about time, it's better to answer with day, month and year

    The answer is incorrect if it contains incorrect information or is missing essential details. For example, when answer a question about time, it's better to answer with day, month and year. Remeber the above rules and keep your response concise and to the point. Note that the ...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.