Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

NoisyEQA: Benchmarking Embodied Question Answering Against Noisy Queries

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Noisy human questions are a real failure mode for embodied question-answering agents, and a simple 'self-correction' prompt wrapper—either a generic noise-awareness instruction or a step-by-step chain-of-thought—substantially improves how…

desk verdict A useful first benchmark for noisy user questions in EQA, but the headline prompting gains may be partly an artifact of an unvalidated LLM judge. read the letter →

arxiv 2412.10726 v1 pith:IBROWF3S submitted 2024-12-14 cs.CV cs.AIcs.LGcs.RO

Tao Wu , Chuhao Zhou , Yen Heng Wong , Lin Gu , Jianfei Yang This is my paper · ORCID
classification cs.CVcs.AIcs.LGcs.RO
keywords embodiedquestionansweringnoisyqueriesvision-languagemodelsself-correctionpromptingnoisetaxonomybenchmarkdatasetdetectionratecorrection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that in real-world embodied question answering, users ask questions with hidden assumptions and mistakes, and that such noisy questions cause current vision-language agents to give wrong or fabricated answers. To make this measurable, it introduces NoisyEQA, a 500-question benchmark with four types of noise—latent hallucination, memory, perception, and semantic—generated automatically from real scenes. The paper then shows that a simple 'self-correction' prompt wrapper, either a generic noise-awareness prompt (NAP) or a step-by-step chain-of-thought version (NACoT), substantially improves both detection of noise and answer correctness on the benchmark. The point of the work is that this kind of noise is a practical failure mode that can be partly mitigated without retraining, and that detection and correction should be measured separately.

What carries the argument

The load-bearing tool is the four-way noise taxonomy plus the automated generation pipeline: an LLM writes noisy questions from scene facts and templates, with perception noise generated by corrupting images and having a VLM describe the distorted scene. The second mechanism is the Self-Correction prompt: NAP appends a general instruction to check all question information against the image; NACoT decomposes the question into object-presence, attribute, and function/semantic checks and asks the agent to explain the detected noise before answering. A confidence-based view selector picks the most reliable camera view by the probability of a 'yes' token to a confidence question. The evaluation scheme is a 1-to-5 rubric scored by an LLM judge, with Detection Rate (scores ≥3) and Correction Rate (scores ≥4) as the noise-specific metrics.

What would settle it

Have two or more human annotators score a sample of NoisyEQA agent responses with the paper's 1–5 rubric, then compare the human scores to the GPT-4 judge scores. If the judge's threshold agreement (score ≥3 for detection, ≥4 for correction) with humans is low, the headline DR/CR improvements would not survive a human evaluation.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that noisy user questions are a real and measurable failure mode for VLM-based EQA agents and that a detection-then-correction prompting mechanism materially improves answer quality. The central evidence is the NoisyEQA benchmark: 500 questions built from real scenes, covering latent hallucination noise (non-existent objects), memory noise (wrong attributes: position, color, shape, count, material), perception noise (visually confused objects), and semantic noise (semantically related substitutions). Adding NAP or NACoT raises the detection rate (DR) for GPT-4o-EQA from 46.2% to 79.0% and the correction rate (CR) from 17.4% to 42.0%, while also raising accuracy on every noise subcategory. The paper also proposes a five-point LLM-judge scale, with DR/CR defined as score thresholds, to separate the ability to notice noise from the ability to fix it.

Load-bearing premise

The results assume that the GPT-4-based judge's scores match what a human would say about whether an agent detected and corrected the noise; the paper does not measure that agreement on NoisyEQA, so the reported detection and correction gains are only as trustworthy as the judge.

Editorial extensions

If this is right

  • Agents that answer without checking will regularly fabricate objects or repeat the user's wrong assumptions in real deployments; a pre-answer verification step is a cheap partial remedy.
  • Noise awareness can be added as a plug-in prompt to existing EQA systems, so the benefit does not require architecture changes or retraining.
  • Detection is substantially easier than correction: even with NAP/NACoT, correction rates stay well behind human performance, especially for perception and semantic noise.
  • The generation framework can be scaled to more scenes and more noise types, providing a controlled way to stress-test robustness of future EQA agents.
  • The separation of detection and correction via DR/CR gives a clearer diagnostic than a single accuracy number for where agents fail.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same detect-then-correct prompting pattern could transfer to other instruction-following tasks, such as vision-language navigation or household instruction following, where user commands carry implicit assumptions.
  • Because the benchmark's generation pipeline is automated and LLM-driven, it could be inverted to produce training pairs (noisy question, corrected question), potentially making agents inherently more robust rather than dependent on prompt wrappers.
  • The paper's observation that noise lowers response confidence suggests that confidence scores could be used as a zero-shot signal to flag likely-noisy questions before answering.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces NoisyEQA, a benchmark of 500 noisy questions for embodied question answering, with a four-type taxonomy of noise (Latent Hallucination, Memory, Perception, and Semantic, with Memory further subdivided into five attributes). A generation framework powered by templates and LLMs creates the questions. The authors also propose a Self-Correction prompting mechanism (NAP and NACoT) that prompts agents to detect and correct noise before answering, and a new evaluation framework with a 1-5 LLM-judge rubric and two metrics, Detection Rate (DR) and Correction Rate (CR). The central empirical finding is that two VLM-based agents (Llama2-EQA and GPT-4o-EQA) perform substantially worse on noisy than on clean questions, and that NAP/NACoT improve their scores on the proposed metrics. The paper also reports human performance on the same benchmark for comparison.

Significance. If the results hold, the paper addresses a real and under-studied failure mode: humans often pose questions with incorrect presumptions, and EQA agents are not robust to them. The taxonomy is reasonable and the automated generation framework is a useful contribution. The DR/CR metrics are a natural way to separate noise detection from correction. The paper is honest in reporting that the improved agents still fall well short of human performance. However, the headline improvements of NAP/NACoT rest on an LLM-judge rubric that is authored by the same team and is not validated against human ratings on NoisyEQA itself; this is a load-bearing gap. The baseline finding of degraded performance on noisy questions is consistent across both agents and all noise types and is likely robust.

major comments (4)
  1. [Section 5, Figure 8, Eqs. (2)-(3), Table 3] The headline DR/CR improvements rest entirely on the GPT-4-based LLM judge introduced in Section 5. The 1-5 rubric explicitly rewards detecting and correcting noise (scores 4 and 5), and DR/CR are defined as thresholded counts on that same rubric (Eqs. 2-3). The only validation offered (supplementary Section 3.2) is OpenEQA's 0.909 Spearman correlation, measured on clean open-vocabulary answers under a different rubric; no human-scoring agreement study is run on NoisyEQA itself. Because NAP and NACoT instruct the agent to check for noise, state the detected noise, and then answer, their outputs are tailored to the rubric's reward structure. The measured jumps (e.g., GPT-4o-EQA DR from 46.2% to 79.0% with NAP, and CR from 17.4% to 42.0%, Table 3) could therefore be partly scoring artifacts. The authors should either (a) run a human-correlation study on a sample of NoisyEQA responses (ideally at least 100 responses, with multiple annotators and inter-annotator agreement), or (b) report results with an independent metric such as exact-match or a second judge with a different rubric, and show that the conclusions are stable.
  2. [Section 5, Figure 8 (rubric examples)] The rubric conflates answer correctness with a particular style of 'correction' on yes/no questions. Score 3 is assigned to a response like 'Sorry, there is no square mirror on the wall', which is a factually correct answer to 'Is the square mirror on the wall?', while higher scores require also stating the actual attribute (e.g., 'the round mirror is on the wall'). This means DR and CR reward a corrective style rather than pure answer accuracy, and the NAP/NACoT prompts explicitly elicit that style. The authors should clarify whether the benchmark's ground truth requires the corrected object/attribute, and report how much of the NAP/NACoT gain survives when scoring is based only on the truth/falsity of the agent's answer to the literal question.
  3. [Section 3.3, Figure 5] Perception Noise is generated by adding Gaussian noise to the scene image, using a VLM to describe the disturbed image, and constructing questions from the misidentified objects. This operationalizes 'human perceptual error' with a specific noise model, but no evidence is given that Gaussian perturbation produces errors similar to human misperception (e.g., due to myopia, occlusion, or low lighting). This is a construct-validity assumption that is load-bearing for the benchmark's taxonomy. The paper should include a human study where annotators judge whether the generated Perception-Noise questions are plausible misperceptions, or at least report agreement between VLM misidentifications and human misidentifications on the same disturbed images.
  4. [Section 3.1, Eqs. (2)-(3)] The composition of the 500-question benchmark is not fully specified: the distribution over the four noise types and the five memory-attribute subtypes is only shown in Figure 4, and the selection procedure (e.g., random sampling, balanced design, or frequency matching to the source datasets) is not described. Likewise, the DR and CR thresholds (score >= 3 and >= 4) are introduced without justification or sensitivity analysis. For a benchmark whose central metrics are DR and CR, the authors should state the intended distribution and report DR/CR under alternative thresholds (e.g., >=2 and >=4, or a continuous score) to show that the conclusions are not threshold artifacts.
minor comments (6)
  1. [Section 5, Eq. (1)] Equation (1) is ambiguous as typeset: 'C = 1/N X_i sigma_i - 1/4 × 100%' should be written as C = (1/N) * sum_i (sigma_i - 1) / 4 * 100%. Please clarify the formula and the range of C.
  2. [Throughout] There are several typos and grammatical errors, including 'adpoted' (Section 5), 'Inpsired' (Supplementary 3.2), 'evluation' (Supplementary 3.1), 'tow noise types' (Supplementary 4.1), 'lined in detection' (Supplementary 4.2), and 'real-word application' (Conclusion). A careful proofread is needed.
  3. [Figure 4] The text of Figure 4 in the provided manuscript appears garbled with unicode escape sequences; please ensure the final PDF shows the actual distribution plots clearly and that the figure is legible.
  4. [Section 6.1] Only two agent variants are evaluated, both built on the same Prismatic-VLM vision encoder, and the authors note that agents without token-probability access are excluded. The abstract's claim of a 'comprehensive evaluation' is stronger than the evidence; consider tempering the language or adding at least one additional agent family.
  5. [Section 1 and Supplementary Section 1] The claim that the Self-Correction mechanism 'meets the AI Act's transparency requirements' is asserted rather than argued. The connection between prompting an agent to verbalize detected noise and legal transparency obligations is not immediate and should be either substantiated or softened.
  6. [Dataset Availability] No link to the NoisyEQA dataset, generation code, or evaluation scripts is provided. For a benchmark paper, releasing the dataset and the LLM-judge prompting code is essential for reproducibility and for the community to verify the judge's behavior.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the rubric–prompt alignment reflects the intended construct, and the missing human-judge validation is a validity gap, not a definitional reduction.

full rationale

The paper's claim that NAP/NACoT improve accuracy on NoisyEQA is an empirical measurement rather than a construction. The 1–5 rubric (Figure 8) rewards noise detection and correction, and NAP/NACoT prompt the agent to check for inaccuracies; however, the method is not defined in terms of the DR/CR thresholds (Eqs. 2–3), and the scores are not fitted to the method's outputs. The benchmark was generated via templates and LLM-based frameworks with volunteer review, independent of the self-correction prompts. The reported DR/CR gains are therefore not forced by definition. The main weakness is that the LLM judge's agreement with human judgment is only cited from OpenEQA (Spearman 0.909) and not re-validated on NoisyEQA, so the rubric's reward structure may inflate measured gains; this is an evaluation-validity concern, not a circular self-citation or fitted-input problem. No load-bearing self-citations, imported uniqueness theorems, or ansatz-by-citation steps are present. Consequently, the derivation chain is self-contained apart from the external judge validation, and no circular step can be exhibited.

Assumptions & free parameters 3 free parameters · 4 assumptions · 4 invented entities

The central empirical claims rest on the validity of the benchmark (can these 500 questions stand in for real noisy human questions?) and the validity of the LLM-judge metric (does the 1 to 5 score reflect human judgment on this dataset?). The paper provides human volunteer review but no inter-annotator agreement, and it borrows OpenEQA's correlation figure without reproducing it here. The noise taxonomy and its five memory sub-attributes are author-defined constructs without external validation.

free parameters (3)
  • DR threshold = score >= 3
    Detection Rate is defined as the fraction of responses with LLM-judge score at least 3 (Eq. 2); this cutoff is chosen by the authors and directly determines the reported DR values, with no sensitivity analysis.
  • CR threshold = score >= 4
    Correction Rate uses cutoff at least 4 (Eq. 3); the choice is arbitrary and not justified by data, yet it drives the headline correction numbers.
  • Noise type distribution in 500 questions = not stated (Figure 4)
    The number of questions per noise type was set by the authors to form the 500-question benchmark; aggregate results depend on this composition, but no explicit counts are given.
assumptions (4)
  • domain assumption The four noise types (latent hallucination, memory, perception, semantic) adequately cover noisy human questions in real EQA interactions.
    The paper states this taxonomy as a modeling contribution (Section 3.2) without empirical evidence on the distribution of real user questions.
  • domain assumption GPT-4 LLM-judge scores on NoisyEQA agree with human judgment on this benchmark.
    Section 3.2 of the supplement cites OpenEQA's 0.909 correlation, but no human-correlation experiment is reported on NoisyEQA itself; the rubric (Figure 8) is authored by the same team that proposed the self-correction method.
  • ad hoc to paper Gaussian perturbation of scene images produces misperceptions similar to human perceptual errors.
    Perception-noise generation (Section 3.3) assumes that adding Gaussian noise and re-describing with a VLM mimics how a nearsighted or distracted person misidentifies objects.
  • domain assumption Human volunteer review ensures the 500 questions contain the intended noise and have correct ground-truth answers.
    Section 3.3 states two volunteers validated the dataset but no agreement statistics or qualification details are given.
invented entities (4)
  • Latent Hallucination Noise
    purpose: A category of question error where the question refers to an object absent from the scene.
    Introduced by this paper as a benchmark construct; no external evidence that real questions follow this pattern.
  • Memory Noise (with position/color/shape/count/material subtypes)
    purpose: Question errors where a recalled object attribute mismatches the scene.
    Defined by the authors; the five attribute subtypes are a design choice without external validation.
  • Perception Noise
    purpose: Question errors caused by visual misidentification.
    Constructed via Gaussian noise plus VLM description; the operationalization is specific to this paper.
  • Semantic Noise
    purpose: Question errors replacing an object with a semantically related incorrect one.
    Defined and operationalized via BERT similarity plus human selection; no independent benchmark exists.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NoisyEQA: Benchmarking Embodied Question Answering Against Noisy Queries." pith.science (2026). https://pith.science/paper/IBROWF3S

@misc{pith2026241210726,
  author       = {Pith},
  title        = {Pith review of: NoisyEQA: Benchmarking Embodied Question Answering Against Noisy Queries},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IBROWF3S}},
  note         = {Machine review of arXiv:2412.10726}
}
read the original abstract

The rapid advancement of Vision-Language Models (VLMs) has significantly advanced the development of Embodied Question Answering (EQA), enhancing agents' abilities in language understanding and reasoning within complex and realistic scenarios. However, EQA in real-world scenarios remains challenging, as human-posed questions often contain noise that can interfere with an agent's exploration and response, bringing challenges especially for language beginners and non-expert users. To address this, we introduce a NoisyEQA benchmark designed to evaluate an agent's ability to recognize and correct noisy questions. This benchmark introduces four common types of noise found in real-world applications: Latent Hallucination Noise, Memory Noise, Perception Noise, and Semantic Noise generated through an automated dataset creation framework. Additionally, we also propose a 'Self-Correction' prompting mechanism and a new evaluation metric to enhance and measure both noise detection capability and answer quality. Our comprehensive evaluation reveals that current EQA agents often struggle to detect noise in questions, leading to responses that frequently contain erroneous information. Through our Self-Correct Prompting mechanism, we can effectively improve the accuracy of agent answers.

Figures

Figures reproduced from arXiv: 2412.10726 by the authors.

Figure 1
Figure 1. (a) A dining table locates in the dining room, a drunk person mistakenly recalls its location, causing the Agent incorrectly makes [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The accuracy of Explore-EQA decreases significantly [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Noise types in NoisyEQA benchmark: Latent Hallucination Noise, Memory Noise, Perception Noise, and Semantic Noise. Memory Noise is further subdivided into Memory Position Noise, Memory Shape Noise, Memory Color Noise, Memory Counting Noise, and Memory Material Noise. ing the information inconsistency in EQA tasks. 3. Dataset 3.1. Overview When querying about agents, humans often unconsciously introduce various presu… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: (a) Distribution of various types of noisy questions. (b) [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: (a) original image and description and (b) disturbed im [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 7
Figure 7. Figure 7: (a) Selection of reliable view. (b) Insights for our Self [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: LLM-powered evaluation criteria to noisy questions. [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: The proportion of different Noise types in (a) OpenEQA [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: illustrates the decline in the agent’s answer confidence across different categories of noise. [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: illustrates the detection and correction rates under different types of EQA Noise. [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ActiveFly-Bench: Aligning Embodied Question Answering with Vision-Language-Action for Aerial Embodied Perception

    cs.RO 2026-07 conditional novelty 7.0 of 10

    ActiveFly-Bench defines Air-EQA, Observation Behavior Planning, and 7-DoF FLUC tasks on 10k real/sim trajectories so UAV agents must plan, fly, and answer questions they cannot solve from the start view.

Reference graph

Works this paper leans on

48 extracted references · 35 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pages 2425– 2433, 2015. 2

    work page 2015

  3. [3]

    Embodied question answer- ing

    Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answer- ing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1–10, 2018. 2, 3

    work page 2018

  4. [4]

    Neural modular control for embodied question answering

    Abhishek Das, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Neural modular control for embodied question answering. InConference on Robot Learning, pages 53–62. PMLR, 2018. 3

    work page 2018

  5. [5]

    S-eqa: Tackling situational queries in embodied question answering

    Vishnu Sashank Dorbala, Prasoon Goyal, Robinson Pira- muthu, Michael Johnston, Reza Ghanadhan, and Dinesh Manocha. S-eqa: Tackling situational queries in embodied question answering. arXiv preprint arXiv:2405.04732, 2024. 3

    arXiv 2024

  6. [6]

    The eu ai act: a summary of its significance and scope

    Lilian Edwards. The eu ai act: a summary of its significance and scope. Artificial Intelligence (the EU AI Act) , 1, 2021. 2, 1

    work page 2021

  7. [7]

    Dissecting dissonance: Benchmarking large mul- timodal models against self-contradictory instructions

    Jin Gao, Lei Gan, Yuankai Li, Yixin Ye, and Dequan Wang. Dissecting dissonance: Benchmarking large mul- timodal models against self-contradictory instructions. In European Conference on Computer Vision, pages 404–420. Springer, 2025. 3

    work page 2025

  8. [8]

    Activelab: Active learn- ing with re-labeling by multiple annotators

    Hui Wen Goh and Jonas Mueller. Activelab: Active learn- ing with re-labeling by multiple annotators. arXiv preprint arXiv:2301.11856, 2023. 3

    arXiv 2023

Show all 48 references
  1. [9]

    Hallusionbench: an advanced diagnos- tic suite for entangled language hallucination and visual il- lusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnos- tic suite for entangled language hallucination and visual il- lusion in large vision-language models. In P...

    2024

  2. [10]

    Detecting and preventing hallucinations in large vision language models

    Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 18135–18143, 2024. 3

    2024

  3. [11]

    Quantifying the uncertainty of llm hallucination spreading in complex adaptive social networks

    Guozhi Hao, Jun Wu, Qianqian Pan, and Rosario Morello. Quantifying the uncertainty of llm hallucination spreading in complex adaptive social networks. Scientific Reports, 14 (1):16375, 2024. 3

    2024

  4. [12]

    Hal-eval: A uni- versal and fine-grained hallucination evaluation framework for large vision language models

    Chaoya Jiang, Hongrui Jia, Mengfan Dong, Wei Ye, Haiyang Xu, Ming Yan, Ji Zhang, and Shikun Zhang. Hal-eval: A uni- versal and fine-grained hallucination evaluation framework for large vision language models. InProceedings of the 32nd ACM International Conference on Multimedia...

    2024

  5. [13]

    Hallucination detection in llm-enriched prod- uct listings

    Ling Jiang, Keer Jiang, Xiaoyu Chu, Saaransh Gulati, and Pulkit Garg. Hallucination detection in llm-enriched prod- uct listings. In Proceedings of the Seventh Workshop on e- Commerce and NLP@ LREC-COLING 2024 , pages 29–39,

    2024

  6. [14]

    Prismatic vlms: Investigating the design space of visually-conditioned language models

    Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: Investigating the design space of visually-conditioned language models. arXiv preprint arXiv:2402.07865, 2024. 7

    2024 arXiv

  7. [15]

    Mitigating object hal- lucinations in large vision-language models through visual contrastive decoding

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hal- lucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recogn...

    2024

  8. [16]

    How to configure good in-context sequence for visual question answering

    Li Li, Jiawei Peng, Huiyi Chen, Chongyang Gao, and Xu Yang. How to configure good in-context sequence for visual question answering. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 26710–26720, 2024. 2

    2024

  9. [17]

    Evaluating object hallucina- tion in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models. arXiv preprint arXiv:2305.10355, 2023. 3

    2023 arXiv

  10. [18]

    A survey on hallucination in large vision-language models

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiu- tian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253, 2024. 3

    2024 arXiv

  11. [19]

    Aligning cyber space with physi- cal world: A comprehensive survey on embodied ai

    Yang Liu, Weixing Chen, Yongjie Bai, Jingzhou Luo, Xin- shuai Song, Kaixuan Jiang, Zhida Li, Ganlong Zhao, Junyi Lin, Guanbin Li, et al. Aligning cyber space with physi- cal world: A comprehensive survey on embodied ai. arXiv preprint arXiv:2407.06886, 2024. 2, 3

    2024 arXiv

  12. [20]

    Robust-eqa: robust learning for embodied question answering with noisy labels

    Haonan Luo, Guosheng Lin, Fumin Shen, Xingguo Huang, Yazhou Yao, and Hengtao Shen. Robust-eqa: robust learning for embodied question answering with noisy labels. IEEE Transactions on Neural Networks and Learning Systems ,

  13. [21]

    Sqa3d: Situated question answering in 3d scenes

    Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yi- tao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07474, 2022. 2

    2022 arXiv

  14. [22]

    Openeqa: Embodied question answering in the era of foun- dation models

    Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the era of foun- dation models. In Proceedings of the IEEE/CVF Conference on...

    2024

  15. [23]

    Confident learning: Estimating uncertainty in dataset labels

    Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373–1411, 2021. 3

    2021

  16. [24]

    Per- vasive label errors in test sets destabilize machine learning benchmarks

    Curtis G Northcutt, Anish Athalye, and Jonas Mueller. Per- vasive label errors in test sets destabilize machine learning benchmarks. arXiv preprint arXiv:2103.14749, 2021. 3

    2021 arXiv

  17. [25]

    Dataset shift in ma- chine learning

    Joaquin Qui ˜nonero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in ma- chine learning. Mit Press, 2022. 3

    2022

  18. [26]

    Explore until confi- dent: Efficient exploration for embodied question answering

    Allen Z Ren, Jaden Clark, Anushri Dixit, Masha Itkina, Anirudha Majumdar, and Dorsa Sadigh. Explore until confi- dent: Efficient exploration for embodied question answering. arXiv preprint arXiv:2403.15941, 2024. 2, 3, 5, 1

    2024 arXiv

  19. [27]

    Prompt- ing large language models with answer heuristics for knowledge-based visual question answering

    Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. Prompt- ing large language models with answer heuristics for knowledge-based visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 14974–14983, 2023. 2

    2023

  20. [28]

    Adaptive integration of par- tial label learning and negative learning for enhanced noisy label learning

    Mengmeng Sheng, Zeren Sun, Zhenhuang Cai, Tao Chen, Yichao Zhou, and Yazhou Yao. Adaptive integration of par- tial label learning and negative learning for enhanced noisy label learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4820–4828, 2024. 3

    2024

  21. [29]

    Mind the error! detection and localization of instruction errors in vision-and-language nav- igation

    Francesco Taioli, Stefano Rosa, Alberto Castellini, Lorenzo Natale, Alessio Del Bue, Alessandro Farinelli, Marco Cristani, and Yiming Wang. Mind the error! detection and localization of instruction errors in vision-and-language nav- igation. arXiv preprint arXiv:2403.10700, 2024. 3

    2024 arXiv

  22. [30]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text. arXiv preprint arXiv:2403.05530, 2024. 7

    2024 arXiv

  23. [31]

    Cabo Verde, Equatorial Guinea, North Macedonia, Sierra Leone, Costa Rica, Marshall Islands, Papua New Guinea, Saudi Arabia, South Sudan, Sri Lanka, et al. Seizing the opportunities of safe, secure and trustworthy artificial intel- ligence systems for sustainable development: d...

  24. [32]

    Unlocking the power of open set: A new perspective for open-set noisy label learn- ing

    Wenhai Wan, Xinrui Wang, Ming-Kun Xie, Shao-Yuan Li, Sheng-Jun Huang, and Songcan Chen. Unlocking the power of open set: A new perspective for open-set noisy label learn- ing. In Proceedings of the AAAI conference on artificial in- telligence, pages 15438–15446, 2024. 3

    2024

  25. [33]

    3d-aware visual question answering about parts, poses and occlusions

    Xingrui Wang, Wufei Ma, Zhuowan Li, Adam Kortylewski, and Alan L Yuille. 3d-aware visual question answering about parts, poses and occlusions. Advances in Neural Information Processing Systems, 36, 2024. 2

    2024

  26. [34]

    Mitigating hallucinations in large vision-language models with instruction contrastive decoding

    Xintong Wang, Jingheng Pan, Liang Ding, and Chris Bie- mann. Mitigating hallucinations in large vision-language models with instruction contrastive decoding. arXiv preprint arXiv:2403.18715, 2024. 3

    2024 arXiv

  27. [35]

    Multi-target embodied question answering

    Licheng Yu, Xinlei Chen, Georgia Gkioxari, Mohit Bansal, Tamara L Berg, and Dhruv Batra. Multi-target embodied question answering. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 6309–6318, 2019. 3

    2019

  28. [36]

    Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data

    Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wen- tao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio...

    2024

  29. [37]

    Early stopping against label noise without validation data

    Suqin Yuan, Lei Feng, and Tongliang Liu. Early stopping against label noise without validation data. InThe Twelfth In- ternational Conference on Learning Representations , 2024. 3

    2024

  30. [38]

    Learning visual question an- swering on controlled semantic noisy labels

    Haonan Zhang, Pengpeng Zeng, Yuxuan Hu, Jin Qian, Jingkuan Song, and Lianli Gao. Learning visual question an- swering on controlled semantic noisy labels. Pattern Recog- nition, 138:109339, 2023. 2

    2023

  31. [39]

    Badlabel: A robust perspective on evaluating and enhancing label-noise learn- ing

    Jingfeng Zhang, Bo Song, Haohan Wang, Bo Han, Tongliang Liu, Lei Liu, and Masashi Sugiyama. Badlabel: A robust perspective on evaluating and enhancing label-noise learn- ing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 3 NoisyEQA: Benchmarking Embodie...

    2024

  32. [40]

    AI Transparency in NoisyEQA Transparency in AI systems is essential for reliable [6] and interpretable decision-making, especially in noisy scenarios addressed in NoisyEQA

    Transparency and Accountability 1.1. AI Transparency in NoisyEQA Transparency in AI systems is essential for reliable [6] and interpretable decision-making, especially in noisy scenarios addressed in NoisyEQA. In this work, we propose the Self- Correction mechanism to tackle t...

  33. [41]

    Posi- tion

    More Details of Existing Datasets As shown in Figure 9, we observed that in existing OpenEQA [22] and ExploreEQA [26] datasets, “Posi- tion” constitutes the largest proportion of all attributes in the questions. This aligns with the natural tendency of hu- mans that we usually...

  34. [42]

    Evaluation Details 3.1. Evaluation Scale To comprehensively evaluate the quality of responses from question-answering agents in the presence of noisy ques- tions, we elaborately design a scoring scale ranging from 1 to 5. The evluation scale assesses three key aspects: noise d...

  35. [43]

    This indicates a fun- damental misunderstanding for both the noise and the question’s core intent

    Score of 1: The question’s noise remains uncorrected, resulting in a response that is completely irrelevant or different from the correct answer. This indicates a fun- damental misunderstanding for both the noise and the question’s core intent

  36. [44]

    This suggests that the agent achieves the correct answer by coincidence, but fails to handle the noise in the question

    Score of 2: Although the noise in the question is not ad- dressed or corrected, the response aligns with the correct answer in meaning. This suggests that the agent achieves the correct answer by coincidence, but fails to handle the noise in the question

  37. [45]

    No, it’s not mentioned

    Score of 3: The agent recognizes the question is unan- swerable due to excessive noise and responds with a statement such as “No, it’s not mentioned.” This demon- strates the agent is able to detect unanswerable ques- tions, rather than forging an inaccurate or irrelevant an- swer

  38. [46]

    It shows that the agent fully understands the question but not achieves a completely correct answer

    Score of 4: The agent effectively corrects the noise in the question, but the response is only partially aligned with the correct answer. It shows that the agent fully understands the question but not achieves a completely correct answer

  39. [47]

    It demonstrates that the agent has a full understanding of the question and can provide a high-quality answer

    Score of 5: The agent both corrects the noise in the question accurately and generates a response that per- fectly matches the correct answer. It demonstrates that the agent has a full understanding of the question and can provide a high-quality answer. 3.2. Effectiveness of L...

  40. [48]

    Impact of Noise on Response Confidence Figure 2 in the manuscript shows that the noise in the ques- tion will significantly decrease the generation accuracy

    More Experiment Results 4.1. Impact of Noise on Response Confidence Figure 2 in the manuscript shows that the noise in the ques- tion will significantly decrease the generation accuracy. Ex- cept for it, we further show that the impact of noise can deteriorate the response con...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.