{"id":"f07e3a9d-e547-48d4-a6e8-cf737b0ce6ff","arxiv_id":"2412.01250","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"In collaborative instance navigation, an uncertainty-aware training-free method (AIUTA) lets an agent ask concise questions to find a specific object instance with minimal human input.","lead":"This paper introduces a new robot navigation task where a robot can ask a human questions during its search instead of needing a full description upfront. The authors' method, AIUTA, uses vision-language models to decide when to ask, and they release a benchmark, CoIN-Bench, plus a small human study.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"All headline numbers come from a VLM-simulated user validated on only 40 easy episodes; if real human answer patterns differ, claimed success rates and question counts could be substantially off.","rationale":"The reader's weakest_assumption identifies the VLM-simulated user as the load-bearing premise, and I agree. All large-scale quantitative results—the main comparison in Table 2, the ablations in Table 4, and the NQ numbers—are generated by a simulated user that has never been validated on the actual CoIN-Bench distribution. The human evaluation is the only evidence for simulation fidelity, but it uses just 40 episodes, explicitly selects episodes with detectable targets, and shows SR values (42.5%) far above the full-benchmark numbers (6.67% on Val Unseen), confirming that the validation subset is not representative. No statistical test is reported for the 'no significant differences' claim, and the sample size is far too small to detect meaningful differences in SR, SPL, or NQ. If human answer distributions differ from the VLM's—for example, if humans say 'I don't know' more often or give less precise attribute values—AIUTA's stop/ask decisions could change substantially, altering success rates and question counts. This directly threatens the central claim that AIUTA outperforms trained methods and significantly surpasses VLFM. The paper has genuine strengths: a new task, a benchmark with many episodes, a training-free method with clear design, and a good-faith (if small) human study. These make the work worth considering, but the core empirical claims are only as strong as the user simulation, which is not yet adequately validated. The threshold-tuning issue is real but secondary: even with slightly different thresholds, the large gap on Val Unseen would likely persist; the simulated-user issue could in principle erase the advantage entirely. The IDKVQA result is also a supporting claim with a small margin and no significance test, but the main headline is the navigation performance. Therefore the reader's CONDITIONAL verdict is appropriate: the paper should be accepted only if the simulated-user fidelity is demonstrated on a larger, unbiased human study.","tokens_in":25200,"tokens_out":7210,"duration_ms":64992,"concrete_test":"Run a human study on a random sample of at least 100 episodes drawn uniformly from the full CoIN-Bench (no 'detectable target' filter), with at least three participants per episode. Compute SR, SPL, and NQ for humans and for the VLM-simulated user, and report bootstrap 95% CIs for the differences plus a two-one-sided t-test for equivalence. Also compute per-question answer agreement (exact or semantic) between humans and the simulated user on the same episodes. If the CI for the SR difference exceeds ±5 points, or if human-VLM agreement on question answers falls below 85%, the simulated-user results in Table 2 cannot be treated as reliable evidence for the central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—AIUTA outperforms trained InstanceObjectNav methods on Val Unseen and surpasses zero-shot VLFM—rests entirely on Table 2, where the user is simulated by a VLM with access to a 1024x1024 target image (Sec. 5, Supp. B). The human validation (Sec. 6, 'Validation with real human'; Supp. E) uses only 40 episodes 'with detectable target instances', a biased, easy subset: SR on this subset is 42.5% for both simulated and human users, whereas Val Unseen SR in Table 2 is 6.67%. The paper claims 'no significant differences' without reporting any statistical test, and the sample is far too small to detect meaningful differences in SR, SPL, or NQ. If real humans answer open-ended questions less precisely, hedge with 'I don't know', or provide different attribute values, AIUTA's fact-update loop could yield different stop/ask decisions, changing success rates and NQ. Since all ablations (Table 4) and the main comparison (Table 2) rely on this simulated user, the quantitative validity of the central claim is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Collaborative Instance object Navigation (CoIN), a task in which an embodied agent receives only a minimal category-level instruction and must resolve instance-level ambiguity through open-ended, template-free dialogue with a human user. The authors propose AIUTA, a training-free method that combines a VLFM navigation policy with two LLM/VLM-based modules: a Self-Questioner that refines observations through self-dialogue and normalized-entropy uncertainty estimation, and an Interaction Trigger that decides whether to ask the user, continue exploring, or stop. They also introduce CoIN-Bench, a benchmark of 1,649 episodes derived from GOAT-Bench with multiple distractors, and an IDKVQA dataset for evaluating VLM uncertainty. Experiments with a VLM-simulated user report that AIUTA outperforms trained InstanceObjectNav methods on Val Unseen and surpasses the zero-shot VLFM baseline, while a 40-episode human study is claimed to validate the simulation.","tokens_in":25454,"tokens_out":4128,"duration_ms":39402,"significance":"If the central claims are substantiated, the paper makes a useful contribution: it defines a practical interactive navigation task, shows that a training-free pipeline of pretrained VLMs and LLMs can meaningfully disambiguate object instances with a small number of questions, and provides a reproducible benchmark with a simulated user. The IDKVQA dataset and the proposed normalized-entropy uncertainty estimator are also potentially valuable. However, the current evidence is not yet sufficient to support the quantitative claims, because the human validation is small and biased, the main results rely entirely on a simulated user whose fidelity is not established on the actual evaluation distribution, and no significance testing or confidence intervals are reported.","major_comments":[{"comment":"The human validation that underpins the simulated-user setup uses only 40 episodes, all explicitly selected to have 'detectable target instances.' This is an easy subset, as shown by the SR of 42.5% on this set versus 6.67% on Val Unseen in Table 2. The paper states 'we observe no significant differences' but reports no statistical test, no confidence intervals, and no effect size. With 40 episodes, the study cannot detect all but extremely large differences in SR, SPL, or NQ. Since every headline result in Table 2 and every ablation in Table 4 uses the VLM-simulated user, the quantitative validity of the central claim is not established. Please provide a proper statistical comparison (e.g., bootstrap confidence intervals or a hypothesis test) and validate on a more representative sample, or explicitly frame the simulated-user results as preliminary.","section":"Sec. 6 'Validation with real human' and Supp. E"},{"comment":"The thresholds tau = 0.75, tau_stop = 7, and tau_skip = 5 are described only as set 'as they yield the best result.' If these values were selected on the CoIN-Bench evaluation splits, all reported SR, SPL, and NQ numbers are optimistically biased. The paper does not state that tuning was performed on a separate validation split (such as the Ablation split used in Ablation I). Please clarify the tuning procedure, and either report results across a range of thresholds or use a nested validation to assess the sensitivity of the headline comparisons.","section":"Sec. 6 'Implementation Details'"},{"comment":"The principal evidence for the claimed superiority of Normalized-Entropy over Energy Score is a difference of 0.67 points (21.12 vs. 20.45) on the Effective Reliability metric in Table 5. No variance, confidence intervals, or significance tests are provided, and the gap is small relative to the scale of the metric. The sensitivity analysis in Fig. 3 uses normalized values and does not by itself demonstrate that the difference in the main metric is meaningful. Please report standard errors or statistical tests, and consider additional datasets or question distributions to support the claim of 'more reliable uncertainty measure.'","section":"Ablation II: 'VLM uncertainty estimation on IDKVQA'"},{"comment":"All comparisons in Table 2 are reported as point estimates without any measure of variability. Given the stochastic components (VLFM, Grounding-DINO, LLaVA, GPT-4o), the differences between AIUTA and trained baselines on Val Unseen (SR 6.67% vs. 2.61%) and on other splits may be within run-to-run noise. The paper's central claim that AIUTA 'outperforms training-based methods on Val Unseen' would be strengthened by reporting standard errors across multiple runs or episodes, or by a statistical test for the SR differences.","section":"Table 2 and Sec. 6 'Results with simulated user-agent interaction'"}],"minor_comments":[{"comment":"The sentence 'the agent actively resolve uncertainties' should be 'the agent actively resolves uncertainties.'","section":"Abstract"},{"comment":"The paper reports NQ = 1.29 for real human versus 1.10 for simulated in Table 3, but no standard deviations or per-episode distributions are provided, making it impossible to judge the variability of this key metric.","section":"Sec. 6 'Validation with real human'"},{"comment":"The 'Distractor Success' metric is a useful diagnostic, but the text would benefit from explaining how it relates to the main SR metric, particularly whether the agent stops at a distractor because of the interaction policy or because of the navigation policy.","section":"Supp. F"},{"comment":"The definition of NQ as 'average Number of Questions asked in successful episodes' should be clarified: does it count questions across all interactions in an episode, and is it averaged only over successful episodes or over all episodes?","section":"Sec. 5 'Metrics'"}],"recommendation":"major_revision","confidential_remarks":"The paper proposes a timely interactive navigation task and a practical training-free baseline; the benchmark itself may be of value to the community. However, the evaluation protocol is the main weakness. The simulated user is validated on only 40 easy episodes with no statistical testing, and the thresholds appear to be tuned directly on the benchmark. I would like the authors to either substantially strengthen the human validation (larger sample, more representative episodes, proper statistical comparison) or soften the claims to clearly state that the simulated-user results are preliminary. The IDKVQA comparison also needs variance estimates. With these revisions, the paper could become acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this paper defines a genuinely new task and ships a reasonable baseline. CoIN lets the agent ask the user open-ended questions during navigation instead of requiring a full target description up front, and AIUTA decides when to ask by combining VLM self-questioning, a normalized-entropy uncertainty filter, and an LLM alignment score. That is a real departure from the templated-QA interactive nav work and from map-based personalization. The benchmark filters GOAT-Bench to multi-instance scenes with ~5 distractors per episode, and the IDKVQA dataset with human annotations for 'Yes/No/I don't know' is a useful small addition. The method is training-free, which is a clean property, and the ablations show both components of AIUTA earn their keep.\n\nThe soft spot is exactly where the stress-test points. Table 2, the main comparison, is generated with a VLM-simulated user that sees a 1024x1024 image of the target. The human validation is 40 episodes chosen because targets were detectable. SR on that subset is 42.5% for both simulated and human users, but the full Val Unseen SR is only 6.67%. So the validation says nothing about how the simulation behaves on the hard cases that make up the benchmark. The paper says 'no significant differences' with no test reported, and with 40 episodes that claim is not believable. If real humans say 'I don't know' more often, or describe attributes differently, AIUTA's question-asking and stop decisions change, and the reported success rates could shift. That doesn't sink the paper, but it means the quantitative headline is not yet established.\n\nTwo smaller issues: the three thresholds (tau, tau_stop, tau_skip) are chosen on the benchmark itself, so the numbers are tuned; and the IDKVQA gain over Energy Score (21.12 vs 20.45) is small with no significance testing. Neither is fatal, but they should be acknowledged more carefully. The paper is honest about inference cost and the simulated-user setup, which I appreciate.\n\nWho is this for: people working on interactive embodied navigation, VLM uncertainty, or benchmark design. It deserves a serious referee; the task is new and the method is a reasonable first baseline. I would engage with it, but I'd ask for either a larger human study or a more careful argument that the simulation is unbiased. Would I cite it? Probably yes, for the task and benchmark. I'd bring it to reading group to discuss the simulation methodology.\n\nRecommendation: accept for peer review, with major revision expectations around the human-validation claim.","headline":"New interactive instance-nav task and a sensible training-free baseline, but the headline numbers rest on a simulated user validated on only 40 easy episodes.","tokens_in":25994,"tokens_out":2551,"would_cite":true,"duration_ms":21778,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training-free method that asks the user only when uncertain beats trained instance-navigation policies.","keywords":["collaborative instance object navigation","human-agent dialogue","embodied AI","zero-shot navigation","vision-language models","large language models","uncertainty estimation","visual question answering"],"falsifier":"Run the full 1,649-episode CoIN-Bench with human participants who are given only the target image and compare their answers with the VLM-simulated user on the exact same episodes; if success rate or questions-asked diverges beyond the variation seen in the 40-episode pilot, the simulated-user results that drive the headline comparison would not transfer to real users.","tokens_in":25011,"feed_emoji":"🤖","tokens_out":8440,"duration_ms":75590,"temperature":0.7,"pith_summary":"This paper introduces a new task, Collaborative Instance object Navigation (CoIN), in which an embodied agent must find a specific object instance after receiving only a category phrase such as \"Find the picture,\" resolving ambiguity by asking the human user a few natural-language questions during navigation. To solve it, the authors propose AIUTA, a training-free module that wraps an existing navigation policy with two LLM/VLM components: a Self-Questioner that refines what the agent sees while filtering out uncertain details using a normalized-entropy estimate, and an Interaction Trigger that decides when to ask, keep exploring, or stop. The paper claims AIUTA outperforms trained instance-navigation methods on unseen object categories on the new CoIN-Bench (1,649 episodes with about five distractors each), and that asking the user fewer than two questions per successful episode suffices. This matters because it shifts the burden of describing the target from the human to the agent, and suggests that task-specific training may not be needed for interactive instance disambiguation.","feed_headline":"Ask only when unsure: training-free agent beats trained navigators","feed_subtitle":"On the new CoIN benchmark, it outdoes trained policies on unseen object categories while asking fewer than two questions per success.","key_machinery":"The carriage of the argument is a two-module loop wrapped around any navigation policy. The Self-Questioner obtains a coarse VLM description of each detected candidate, asks an LLM to generate attribute-specific yes/no questions, and restricts VLM answers to \"Yes\", \"No\", or \"I don't know\" so that the Shannon entropy of the three-token output distribution, normalized by its maximum, gives a per-attribute uncertainty in [0,1]. Attributes above a threshold τ are dropped when the LLM recomposes the refined description S_refined. The Interaction Trigger then asks the LLM for a single alignment score s between S_refined and the accumulated facts F_t about the target, along with one question to the user; s ≥ τ_stop halts, s < τ_skip continues silently, and the middle band triggers the user question, whose answer updates F_t. The Normalized-Entropy estimator is the component that lets the agent trust its own perception, and the paper evaluates it separately on IDKVQA, a 502-question dataset with triple human annotations, where it scores 21.12 on the Effective Reliability metric versus 20.45 for the best baseline.","core_discovery":"The central claim is that a training-free agent can solve instance object navigation by actively managing its own perceptual uncertainty and asking the human user only when a computed alignment score says a question will help. On CoIN-Bench, AIUTA achieves 6.67% success on the Val Unseen split, above the best trained baseline (PSL, 4.58%), while the zero-shot navigator it wraps, VLFM, scores near zero because it cannot tell the target apart from the roughly five same-category distractors per episode. On the seen splits AIUTA is competitive with trained baselines, and across all splits it succeeds with fewer than two user questions per successful episode (NQ between 1.13 and 1.67). The authors also report a 40-episode human study in which real humans and the VLM-simulated user give the same success rate (42.5%), which they take as evidence that the simulated protocol is a reliable stand-in for large-scale evaluation.","pith_inferences":["If the simulated-user result transfers, the evaluation design generalizes: any dialogic embodied task with a visual referent could be benchmarked by giving a VLM a high-resolution image of the goal, making human studies a verification step rather than the main data source.","The normalized-entropy abstention rule is not specific to navigation; it could serve as a generic hallucination filter for any constrained VQA pipeline, including caption-refinement systems.","A natural extension the paper does not test is making the two thresholds τ_skip and τ_stop adaptive, since a fixed threshold that works on GOAT-Bench scenes may need recalibration in more cluttered or occluded environments.","The benchmark filters out episodes where the target is hard to see, so the reported question counts likely understate what real deployment would require; testing on unfiltered, occluded episodes is an open empirical question."],"forward_implications":["A user can start an instance-navigation task with a bare category phrase, and the agent, not the user, supplies the descriptive effort through targeted questions.","On unseen object categories, a training-free pipeline can beat policies trained for the task, suggesting instance disambiguation is more about perception-and-query reasoning than about navigation-policy training.","The NQ metric plus the simulated-user protocol make large dialogue-based navigation benchmarks reproducible without recruiting hundreds of human annotators.","Restricting VLM answers to a three-way vocabulary turns open-ended hallucination into a measurable entropy signal, giving a reusable abstention rule for visual question answering.","Because AIUTA operates independently of the navigation policy, the same interaction reasoning can be plugged into any detector-equipped navigator."],"supporting_citations":[{"why":"Supplies the GOAT-Bench episodes and splits whose multi-instance cases define CoIN-Bench, so the comparison set and the trained baselines come from here.","marker":"[17]"},{"why":"The zero-shot navigation policy and value-map exploration that AIUTA wraps; its near-zero success is the baseline AIUTA must beat.","marker":"[53]"},{"why":"Open-set object detector that generates candidate detections which trigger the Self-Questioner pipeline.","marker":"[24]"},{"why":"Vision-language model used for initial descriptions, self-question answers, and the simulated user; the uncertainty estimator operates on its token distribution.","marker":"[23]"},{"why":"Large language model that generates self-questions, refinement descriptions, alignment scores, and user-directed questions.","marker":"[15]"},{"why":"Energy-score uncertainty baseline that Normalized-Entropy is compared against on IDKVQA and in the threshold-sensitivity study.","marker":"[26]"},{"why":"Logit-probe uncertainty baseline used as a second comparison on IDKVQA.","marker":"[59]"},{"why":"Supplies the Effective Reliability metric Φc that scores abstention quality on IDKVQA.","marker":"[50]"},{"why":"Simulator that executes episodes and computes success and path-length metrics.","marker":"[42]"}],"fun_headline_variants":["Training-free agent asks less, beats trained navigators on CoIN-Bench","Uncertainty-aware AI cuts user input, outperforms trained policies","Ask only when unsure: training-free navigation beats trained peers","Fewer questions, better navigation: training-free agent outperforms","Agent's uncertainty guidance slashes human input, boosts success"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that a vision-language model shown a high-resolution image of the target answers the agent's questions like a real human, a check the paper runs on only 40 episodes with detectable targets.","fun_headline_variants_meta":{"raw":{"variants":["Training-free agent asks less, beats trained navigators on CoIN-Bench","Uncertainty-aware AI cuts user input, outperforms trained policies","Ask only when unsure: training-free navigation beats trained peers","Fewer questions, better navigation: training-free agent outperforms","Agent's uncertainty guidance slashes human input, boosts success"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000325,"raw_usage":{"total_tokens":1866,"prompt_tokens":1034,"completion_tokens":832,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":650,"completion_tokens_details":{"reasoning_tokens":744}},"tokens_in":650,"tokens_out":832,"duration_ms":8281,"temperature":1.0,"reasoning_tokens":744,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:31:21.012877+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full 1,649-episode CoIN-Bench with human participants who are given only the target image and compare their answers with the VLM-simulated user on the exact same episodes; if success rate or questions-asked diverges beyond the variation seen in the 40-episode pilot, the simulated-user results that drive the headline comparison would not transfer to real users.","supporting_citations":[{"cited_title":"GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigation","cited_arxiv_id":null,"evidence_quote":"Supplies the GOAT-Bench episodes and splits whose multi-instance cases define CoIN-Bench, so the comparison set and the trained baselines come from here."},{"cited_title":"VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation","cited_arxiv_id":null,"evidence_quote":"The zero-shot navigation policy and value-map exploration that AIUTA wraps; its near-zero success is the baseline AIUTA must beat."},{"cited_title":"LLaV A-NeXT: Improved reasoning, OCR, and world knowledge, 2024","cited_arxiv_id":null,"evidence_quote":"Vision-language model used for initial descriptions, self-question answers, and the simulated user; the uncertainty estimator operates on its token distribution."},{"cited_title":"Energy-based Out-of-distribution Detection","cited_arxiv_id":null,"evidence_quote":"Energy-score uncertainty baseline that Normalized-Entropy is compared against on IDKVQA and in the threshold-sensitivity study."},{"cited_title":"The First to Know: How Token Distributions Reveal Hidden Knowledge in Large Vision- Language Models? In Computer Vision - ECCV 2024","cited_arxiv_id":null,"evidence_quote":"Logit-probe uncertainty baseline used as a second comparison on IDKVQA."},{"cited_title":"Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectly","cited_arxiv_id":null,"evidence_quote":"Supplies the Effective Reliability metric Φc that scores abstention quality on IDKVQA."},{"cited_title":"Habitat: A Platform for Embodied AI Research","cited_arxiv_id":null,"evidence_quote":"Simulator that executes episodes and computes success and path-length metrics."}],"review_version":1}