{"id":"d89e50eb-5caa-46e5-a48c-6da97419927b","arxiv_id":"2412.10726","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"NoisyEQA benchmarks embodied QA agents against four noise types and claims a self-correcting prompt (NACoT) markedly improves noisy-question accuracy as scored by GPT-4.","lead":"The paper introduces NoisyEQA, a 500-question benchmark that injects four types of human-like noise into embodied question answering, and shows that current vision-language agents answer such noisy questions poorly. It also proposes a self-correction prompting method and new metrics, reporting big accuracy gains, though the gains are measured only by the authors' own GPT-4-based scoring rubric.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline DR/CR gains may be an artifact of an unvalidated GPT-4 judge whose rubric rewards the very behaviors NAP/NACoT are prompted to produce.","rationale":"The evaluation protocol is the load-bearing component of the paper's central empirical claim. Unlike the benchmark construction, which is relatively transparent (template-based generation, volunteer validation), the metric is bespoke and unvalidated on this dataset. The paper's only support is a citation to OpenEQA's correlation, but that was measured on clean, open-ended answers, not on noisy questions with a rubric that explicitly rewards the behaviors the proposed method is designed to produce. Without a human-agreement check, the reported DR/CR improvements could reflect the judge rewarding the presence of noise-detection language rather than genuinely better answers. This is a correctness risk, not a stylistic preference, because it directly affects the headline numbers that support the abstract's claim of 'effectively improve the accuracy of agent answers.' The reader's weakest assumption identifies the same concern, so I agree with it. My recommended verdict remains CONDITIONAL, since the issue is addressable with a human-correlation experiment and does not require rejecting the benchmark contribution.","tokens_in":19024,"tokens_out":3603,"duration_ms":35009,"concrete_test":"Sample 100 agent responses stratified across the six conditions and have two annotators, blind to condition, score each with the same 1-5 rubric. Compute Spearman correlation between the GPT-4 judge and the pooled human scores, and recompute DR/CR from human scores. If the correlation is below roughly 0.7, or if the NAP/NACoT DR/CR gains shrink by more than half when human scores replace LLM scores, then the headline improvement is not established; if gains survive with high correlation, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that NAP/NACoT improve answer accuracy rests entirely on the LLM-judge scores in Section 5. The 1-5 rubric (Figure 8) explicitly rewards detection (score 3), correction (4), and perfect match (5), and DR/CR are defined by thresholds on those scores (Eqs. 2-3). The only validation offered is OpenEQA's 0.909 Spearman correlation, which was measured on clean open-vocabulary answers under a different rubric. No human-scoring agreement study is run on NoisyEQA. Because NAP and NACoT instruct the agent to check for noise, state the detected noise, and then answer, their outputs are tailored to match the rubric's reward structure even if the final answer is not substantively better. The measured jumps (e.g., GPT-4o-EQA DR 46.2% to 79.0% with NAP, CR 17.4% to 42.0%; Table 3) could therefore be partly scoring artifacts. The benchmark's headline conclusion that self-correction improves answer quality is unsupported unless the judge is shown to track human judgment on this dataset.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces NoisyEQA, a benchmark of 500 noisy questions for embodied question answering, with a four-type taxonomy of noise (Latent Hallucination, Memory, Perception, and Semantic, with Memory further subdivided into five attributes). A generation framework powered by templates and LLMs creates the questions. The authors also propose a Self-Correction prompting mechanism (NAP and NACoT) that prompts agents to detect and correct noise before answering, and a new evaluation framework with a 1-5 LLM-judge rubric and two metrics, Detection Rate (DR) and Correction Rate (CR). The central empirical finding is that two VLM-based agents (Llama2-EQA and GPT-4o-EQA) perform substantially worse on noisy than on clean questions, and that NAP/NACoT improve their scores on the proposed metrics. The paper also reports human performance on the same benchmark for comparison.","tokens_in":19294,"tokens_out":5125,"duration_ms":46024,"significance":"If the results hold, the paper addresses a real and under-studied failure mode: humans often pose questions with incorrect presumptions, and EQA agents are not robust to them. The taxonomy is reasonable and the automated generation framework is a useful contribution. The DR/CR metrics are a natural way to separate noise detection from correction. The paper is honest in reporting that the improved agents still fall well short of human performance. However, the headline improvements of NAP/NACoT rest on an LLM-judge rubric that is authored by the same team and is not validated against human ratings on NoisyEQA itself; this is a load-bearing gap. The baseline finding of degraded performance on noisy questions is consistent across both agents and all noise types and is likely robust.","major_comments":[{"comment":"The headline DR/CR improvements rest entirely on the GPT-4-based LLM judge introduced in Section 5. The 1-5 rubric explicitly rewards detecting and correcting noise (scores 4 and 5), and DR/CR are defined as thresholded counts on that same rubric (Eqs. 2-3). The only validation offered (supplementary Section 3.2) is OpenEQA's 0.909 Spearman correlation, measured on clean open-vocabulary answers under a different rubric; no human-scoring agreement study is run on NoisyEQA itself. Because NAP and NACoT instruct the agent to check for noise, state the detected noise, and then answer, their outputs are tailored to the rubric's reward structure. The measured jumps (e.g., GPT-4o-EQA DR from 46.2% to 79.0% with NAP, and CR from 17.4% to 42.0%, Table 3) could therefore be partly scoring artifacts. The authors should either (a) run a human-correlation study on a sample of NoisyEQA responses (ideally at least 100 responses, with multiple annotators and inter-annotator agreement), or (b) report results with an independent metric such as exact-match or a second judge with a different rubric, and show that the conclusions are stable.","section":"Section 5, Figure 8, Eqs. (2)-(3), Table 3"},{"comment":"The rubric conflates answer correctness with a particular style of 'correction' on yes/no questions. Score 3 is assigned to a response like 'Sorry, there is no square mirror on the wall', which is a factually correct answer to 'Is the square mirror on the wall?', while higher scores require also stating the actual attribute (e.g., 'the round mirror is on the wall'). This means DR and CR reward a corrective style rather than pure answer accuracy, and the NAP/NACoT prompts explicitly elicit that style. The authors should clarify whether the benchmark's ground truth requires the corrected object/attribute, and report how much of the NAP/NACoT gain survives when scoring is based only on the truth/falsity of the agent's answer to the literal question.","section":"Section 5, Figure 8 (rubric examples)"},{"comment":"Perception Noise is generated by adding Gaussian noise to the scene image, using a VLM to describe the disturbed image, and constructing questions from the misidentified objects. This operationalizes 'human perceptual error' with a specific noise model, but no evidence is given that Gaussian perturbation produces errors similar to human misperception (e.g., due to myopia, occlusion, or low lighting). This is a construct-validity assumption that is load-bearing for the benchmark's taxonomy. The paper should include a human study where annotators judge whether the generated Perception-Noise questions are plausible misperceptions, or at least report agreement between VLM misidentifications and human misidentifications on the same disturbed images.","section":"Section 3.3, Figure 5"},{"comment":"The composition of the 500-question benchmark is not fully specified: the distribution over the four noise types and the five memory-attribute subtypes is only shown in Figure 4, and the selection procedure (e.g., random sampling, balanced design, or frequency matching to the source datasets) is not described. Likewise, the DR and CR thresholds (score >= 3 and >= 4) are introduced without justification or sensitivity analysis. For a benchmark whose central metrics are DR and CR, the authors should state the intended distribution and report DR/CR under alternative thresholds (e.g., >=2 and >=4, or a continuous score) to show that the conclusions are not threshold artifacts.","section":"Section 3.1, Eqs. (2)-(3)"}],"minor_comments":[{"comment":"Equation (1) is ambiguous as typeset: 'C = 1/N X_i sigma_i - 1/4 × 100%' should be written as C = (1/N) * sum_i (sigma_i - 1) / 4 * 100%. Please clarify the formula and the range of C.","section":"Section 5, Eq. (1)"},{"comment":"There are several typos and grammatical errors, including 'adpoted' (Section 5), 'Inpsired' (Supplementary 3.2), 'evluation' (Supplementary 3.1), 'tow noise types' (Supplementary 4.1), 'lined in detection' (Supplementary 4.2), and 'real-word application' (Conclusion). A careful proofread is needed.","section":"Throughout"},{"comment":"The text of Figure 4 in the provided manuscript appears garbled with unicode escape sequences; please ensure the final PDF shows the actual distribution plots clearly and that the figure is legible.","section":"Figure 4"},{"comment":"Only two agent variants are evaluated, both built on the same Prismatic-VLM vision encoder, and the authors note that agents without token-probability access are excluded. The abstract's claim of a 'comprehensive evaluation' is stronger than the evidence; consider tempering the language or adding at least one additional agent family.","section":"Section 6.1"},{"comment":"The claim that the Self-Correction mechanism 'meets the AI Act's transparency requirements' is asserted rather than argued. The connection between prompting an agent to verbalize detected noise and legal transparency obligations is not immediate and should be either substantiated or softened.","section":"Section 1 and Supplementary Section 1"},{"comment":"No link to the NoisyEQA dataset, generation code, or evaluation scripts is provided. For a benchmark paper, releasing the dataset and the LLM-judge prompting code is essential for reproducibility and for the community to verify the judge's behavior.","section":"Dataset Availability"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is timely and the dataset construction is systematic, but the evaluation protocol is the main risk. The authors use their own LLM-judge rubric to validate their own prompting method, with no human agreement study on NoisyEQA. I would encourage the editor to require a human-scoring experiment (or an independent metric) before acceptance. Also, the 'comprehensive evaluation' phrasing overstates the two-agent pilot. If the authors add the validation study, the paper could be a solid contribution to the EQA benchmarking literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on NoisyEQA.\n\nThe paper does something genuinely new: it builds the first benchmark for embodied question answering where the user's question contains noise—false premises, misremembered attributes, misperceived objects, or semantic substitutions. Four types of noise, 500 questions generated from OpenEQA/HMEQA scenes with a template-plus-LLM pipeline and two volunteers checking each entry. The taxonomy is sensible, and the examples are clear. The baseline finding is solid: both Llama2-EQA and GPT-4o-EQA lose a lot of accuracy on noisy questions, and the loss is consistent across every noise type. That alone is a useful result for the field.\n\nThe paper also proposes DR and CR (detection rate and correction rate) as metrics and a simple prompting mechanism (NAP and NACoT) to make agents check the question against the visual scene before answering. These are reasonable ideas. The authors are honest that even with the prompts, correction rates are far below human performance.\n\nThe soft spot is the evaluation. The headline gains—e.g., GPT-4o-EQA DR jumping from 46.2% to 79.0% with NAP—come from a GPT-4 judge using a 1–5 rubric authored by the same team. The rubric explicitly gives 4 and 5 for detecting and correcting the noise, which is precisely what the prompts tell the agent to do. The only validation offered is OpenEQA's reported 0.909 Spearman correlation, measured on clean open-vocabulary answers. That's not evidence that the judge tracks human judgments on this dataset. So the improvements could be partly scoring artifacts. This is the main reason I can't fully trust the numbers as reported.\n\nMinor issues: no data or code released, no error bars or significance tests, and 500 questions is a modest size. The perception noise generation via adding Gaussian noise to an image and using a VLM to misdescribe objects is a bit contrived, though the human review helps.\n\nOverall: the benchmark is worth having, and the central qualitative finding—noisy queries break current EQA agents—is unlikely to be an artifact. The magnitude of the prompting gains needs a human-correlation study on NoisyEQA before I'd cite those numbers. I'd send this to peer review with a request for that study and release of the data. It's the kind of paper that will be useful to embodied AI and VLM evaluation folks, and it opens a fair direction for future work.","headline":"A useful first benchmark for noisy user questions in EQA, but the headline prompting gains may be partly an artifact of an unvalidated LLM judge.","tokens_in":19824,"tokens_out":2973,"would_cite":false,"duration_ms":25512,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Noisy human questions are a real failure mode for embodied question-answering agents, and a simple 'self-correction' prompt wrapper—either a generic noise-awareness instruction or a step-by-step chain-of-thought—substantially improves how…","keywords":["embodied question answering","noisy queries","vision-language models","self-correction prompting","noise taxonomy","benchmark dataset","detection rate","correction rate"],"falsifier":"Have two or more human annotators score a sample of NoisyEQA agent responses with the paper's 1–5 rubric, then compare the human scores to the GPT-4 judge scores. If the judge's threshold agreement (score ≥3 for detection, ≥4 for correction) with humans is low, the headline DR/CR improvements would not survive a human evaluation.","tokens_in":18824,"feed_emoji":"❓","tokens_out":6281,"duration_ms":51119,"temperature":0.7,"pith_summary":"The paper argues that in real-world embodied question answering, users ask questions with hidden assumptions and mistakes, and that such noisy questions cause current vision-language agents to give wrong or fabricated answers. To make this measurable, it introduces NoisyEQA, a 500-question benchmark with four types of noise—latent hallucination, memory, perception, and semantic—generated automatically from real scenes. The paper then shows that a simple 'self-correction' prompt wrapper, either a generic noise-awareness prompt (NAP) or a step-by-step chain-of-thought version (NACoT), substantially improves both detection of noise and answer correctness on the benchmark. The point of the work is that this kind of noise is a practical failure mode that can be partly mitigated without retraining, and that detection and correction should be measured separately.","feed_headline":"Check-first prompt lifts noisy-question detection to 79%","feed_subtitle":"A 500-question benchmark shows vision-language agents answer wrong when users make wrong assumptions.","key_machinery":"The load-bearing tool is the four-way noise taxonomy plus the automated generation pipeline: an LLM writes noisy questions from scene facts and templates, with perception noise generated by corrupting images and having a VLM describe the distorted scene. The second mechanism is the Self-Correction prompt: NAP appends a general instruction to check all question information against the image; NACoT decomposes the question into object-presence, attribute, and function/semantic checks and asks the agent to explain the detected noise before answering. A confidence-based view selector picks the most reliable camera view by the probability of a 'yes' token to a confidence question. The evaluation scheme is a 1-to-5 rubric scored by an LLM judge, with Detection Rate (scores ≥3) and Correction Rate (scores ≥4) as the noise-specific metrics.","core_discovery":"On its own terms, the paper establishes that noisy user questions are a real and measurable failure mode for VLM-based EQA agents and that a detection-then-correction prompting mechanism materially improves answer quality. The central evidence is the NoisyEQA benchmark: 500 questions built from real scenes, covering latent hallucination noise (non-existent objects), memory noise (wrong attributes: position, color, shape, count, material), perception noise (visually confused objects), and semantic noise (semantically related substitutions). Adding NAP or NACoT raises the detection rate (DR) for GPT-4o-EQA from 46.2% to 79.0% and the correction rate (CR) from 17.4% to 42.0%, while also raising accuracy on every noise subcategory. The paper also proposes a five-point LLM-judge scale, with DR/CR defined as score thresholds, to separate the ability to notice noise from the ability to fix it.","pith_inferences":["The same detect-then-correct prompting pattern could transfer to other instruction-following tasks, such as vision-language navigation or household instruction following, where user commands carry implicit assumptions.","Because the benchmark's generation pipeline is automated and LLM-driven, it could be inverted to produce training pairs (noisy question, corrected question), potentially making agents inherently more robust rather than dependent on prompt wrappers.","The paper's observation that noise lowers response confidence suggests that confidence scores could be used as a zero-shot signal to flag likely-noisy questions before answering."],"forward_implications":["Agents that answer without checking will regularly fabricate objects or repeat the user's wrong assumptions in real deployments; a pre-answer verification step is a cheap partial remedy.","Noise awareness can be added as a plug-in prompt to existing EQA systems, so the benefit does not require architecture changes or retraining.","Detection is substantially easier than correction: even with NAP/NACoT, correction rates stay well behind human performance, especially for perception and semantic noise.","The generation framework can be scaled to more scenes and more noise types, providing a controlled way to stress-test robustness of future EQA agents.","The separation of detection and correction via DR/CR gives a clearer diagnostic than a single accuracy number for where agents fail."],"supporting_citations":[{"why":"Supplies the base question-answer dataset and the LLM-Match metric whose 0.909 human-correlation is cited as validation for the evaluation framework.","marker":"[22]"},{"why":"Provides the Explore-EQA baseline agent and the HMEQA dataset source for object candidates used in semantic-noise generation.","marker":"[26]"},{"why":"Defines the DINO+SigLIP vision encoder and the Prismatic-VLM design used to build the Llama2-EQA and GPT-4o-EQA baselines.","marker":"[14]"},{"why":"The GPT-4 model is used both to generate descriptions of perturbed images for perception noise and to act as the LLM judge in the evaluation framework.","marker":"[1]"},{"why":"Supplies the argument that test-time noise is an important, neglected source of error, motivating the benchmark's focus on noisy queries.","marker":"[24]"}],"fun_headline_variants":["NoisyEQA: 500 questions reveal embodied AI's noise blind spot","Self-correction prompt lifts noisy-EQA detection to 79%","Benchmark: Noisy queries in embodied QA get self-corrected","From 46% to 79%: Catching noise in embodied question answering","Noisy user questions? Embodied agents learn to self-correct"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results assume that the GPT-4-based judge's scores match what a human would say about whether an agent detected and corrected the noise; the paper does not measure that agreement on NoisyEQA, so the reported detection and correction gains are only as trustworthy as the judge.","fun_headline_variants_meta":{"raw":{"variants":["NoisyEQA: 500 questions reveal embodied AI's noise blind spot","Self-correction prompt lifts noisy-EQA detection to 79%","Benchmark: Noisy queries in embodied QA get self-corrected","From 46% to 79%: Catching noise in embodied question answering","Noisy user questions? Embodied agents learn to self-correct"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000908,"raw_usage":{"total_tokens":3897,"prompt_tokens":934,"completion_tokens":2963,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":2866}},"tokens_in":550,"tokens_out":2963,"duration_ms":21849,"temperature":1.0,"reasoning_tokens":2866,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:39:54.174690+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two or more human annotators score a sample of NoisyEQA agent responses with the paper's 1–5 rubric, then compare the human scores to the GPT-4 judge scores. If the judge's threshold agreement (score ≥3 for detection, ≥4 for correction) with humans is low, the headline DR/CR improvements would not survive a human evaluation.","supporting_citations":[{"cited_title":"Openeqa: Embodied question answering in the era of foun- dation models","cited_arxiv_id":null,"evidence_quote":"Supplies the base question-answer dataset and the LLM-Match metric whose 0.909 human-correlation is cited as validation for the evaluation framework."}],"review_version":1}