{"id":"5f5d6850-c1e7-43e9-9c4e-00641c431126","arxiv_id":"2412.17288","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"FLARE, a few-shot planner that retrieves examples using visual context and repairs missing objects via semantic similarity, sets new ALFRED results with only 100 training pairs.","lead":"A robot-planning system called FLARE uses both language and the robot's view of the room, and swaps missing objects for similar ones it can see, nearly tripling success on a standard household benchmark with only 0.5% of the training data. A smart generalist should read it because it suggests cheaper, few-example training for household robots, but the gains are partly due to system-level choices beyond the two new components.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline gain versus LLM-Planner is mostly produced by FLARE's base system, not by the proposed components, because Table 3(d) already exceeds the cited baseline by 12.16% before adding MMP or EAR.","rationale":"The reader's weakest assumption identifies precisely the same load-bearing concern: the comparison to LLM-Planner is not controlled for differences in the base system, and Table 3(d) shows the base already surpasses the cited baseline by a wide margin. My independent reading of the paper confirms this and adds that Table 2 suffers from the same issue with plan-format-sensitive exact-match accuracy. This concern is load-bearing because the Introduction and abstract attribute the improvement to the proposed FLARE system, while the data actually attribute most of the improvement to unspecified base-system choices (low-level policy, retrieval, prompt) that are not part of the contribution. The concern does not, however, invalidate the internal ablations or the qualitative demonstrations; the components do add a few points, and the code is provided. The appropriate response is to require the authors to re-run the comparison under matched conditions and to re-state the contribution as the component-wise gain rather than the total margin above a differently configured baseline. That is a condition for acceptance, not a rejection, so I leave the reader's CONDITIONAL verdict unchanged.","tokens_in":18075,"tokens_out":7929,"duration_ms":78525,"concrete_test":"Re-run the released LLM-Planner code (or a faithful reimplementation) on the identical FLARE evaluation harness: same low-level action policy, same prompt template, same plan parsing, and same LLM version (e.g., GPT-3.5-turbo-instruct at temperature 0), then measure unseen SR. If the re-run reproduces the cited 16.42%, the headline comparison is at least fair; if it rises toward FLARE's row (d) value of 28.58%, the gap is a base-system artifact. Additionally, compute the contribution of MMP and EAR as (row a - row d) with GPT-4 under the same matched baseline; if that contribution stays near 3% SR, the paper's emphasis on the proposed components as the source of the headline gain is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is the +24.46% absolute improvement over LLM-Planner on unseen test success (Table 1: 40.88% vs. 16.42%). The paper's own ablation with GPT-3.5 (Table 3) shows that the system without either proposed component (row d) already reaches 28.58% unseen SR, which is +12.16% over LLM-Planner's cited 16.42%. Adding MMP and EAR (row a) yields 31.79%, so the two proposed components together contribute only +3.21% SR. In other words, roughly 80% of the advertised gap to LLM-Planner is present in the base system before the paper's contributions are applied. This does not show the components are ineffective, but it does mean the headline 'outperforming state-of-the-art' comparison is not informative about the value of the proposed visual grounding and replanning. Furthermore, the LLM-Planner numbers are excerpted from the published paper rather than re-run under matched conditions; FLARE uses a different low-level controller (deterministic FMM navigation and a learned semantic-mapping perception pipeline), a different subgoal representation (action-object-receptacle triplets), and a different prompt structure, and the LLM version behind the cited 16.42% is not controlled. The same issue extends to Table 2: plan accuracy is measured by exact match to ground-truth plans, which is highly sensitive to output formatting, yet FLARE and LLM-Planner use different plan formats. Because the ablations never hold the base system constant against the comparison method, the paper has not established that the proposed components are the source of the claimed state-of-the-art result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents FLARE, an embodied instruction-following agent for the ALFRED benchmark in a few-shot (0.5%) setting. It contributes a Multi-Modal Planner (MMP) that retrieves k in-context demonstrations using a weighted combination of language and visual environment similarity (Eq. 1), and an Environment Adaptive Replanning (EAR) module that replaces an undetected target object with the most semantically similar detected object using Eq. (3). The system is evaluated against LLM-Planner and other baselines, reporting a test-unseen success rate of 40.88% with GPT-4 versus LLM-Planner's cited 16.42%, with additional ablations, planner-accuracy comparisons, and a tabletop robotic application.","tokens_in":18393,"tokens_out":3512,"duration_ms":32280,"significance":"The work addresses a real and timely problem: grounding LLM-based planning in observed environment state while using very few annotated demonstrations. Strengths include a clearly described and modular architecture, publicly released code, evaluation across four different LLMs, a component-level ablation, and a qualitative robotic demonstration. If the component-level results are considered in isolation, MMP and EAR each give a consistent but modest improvement over the base system, and the static-planner accuracy gains in Table 2 are large. The principal weakness is that the headline state-of-the-art comparison is not cleanly attributable to the proposed components, because the base system without either component already exceeds the cited LLM-Planner baseline by a wide margin.","major_comments":[{"comment":"The headline claim in the Introduction of an up to +24.46% absolute gain over LLM-Planner is not attributable to the proposed components. Table 3(d) shows that with GPT-3.5, the base system without MMP or EAR already reaches 28.58% unseen success rate, which is +12.16 points over the cited LLM-Planner value of 16.42% in Table 1, while adding MMP and EAR together (row a) yields 31.79%, an additional +3.21 points. The paper should separate the gain due to the base system from the gain due to MMP and EAR, or re-run LLM-Planner under matched conditions, before claiming that visual grounding and replanning produce the reported margin.","section":"Table 1 vs. Table 3"},{"comment":"The comparison with LLM-Planner is not controlled. The published LLM-Planner numbers are excerpted rather than re-run with the same low-level action policy, subgoal representation, prompt format, and LLM version; the paper itself uses deterministic FMM navigation, a different triplet representation (Eq. (2)), and a different prompt structure (Fig. 8). This also affects Table 2, where plan accuracy is judged by exact match to ground-truth plans and the two systems use different plan formats. Please provide a matched re-run or explicitly qualify the comparison as a system-level benchmark difference rather than evidence about the proposed components.","section":"Table 1 and Table 2"},{"comment":"The title and contributions describe 'efficient replanning,' and Section 3.2 contrasts EAR with LLM-based replanning, but the paper reports no runtime, token cost, or LLM query-count measurements. Without such measurements, the computational-efficiency claim is not empirically supported. Please add at least an LLM-call count or token-cost comparison with and without EAR, or revise the efficiency claim accordingly.","section":"Section 3.2 and title"}],"minor_comments":[{"comment":"The word 'path-length-weighteed' should be 'path-length-weighted.'","section":"Table 1 caption"},{"comment":"The sentence 'Despite with a relatively less capable language models' is ungrammatical; consider 'Despite using a relatively less capable language model...'","section":"Section 4.3"},{"comment":"The phrase 'unrounded plan' should be 'ungrounded plan.'","section":"Section 4.5 and Figure 6 caption"},{"comment":"In the supplementary text for Figure 9, 'CoffeMaker' is a typo for 'CoffeeMaker.'","section":"Supplementary Figure 9"},{"comment":"The values of w_l and w_e in Eq. (1) are said to be equal, but the actual values are not reported; please state them for reproducibility.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nFLARE is a few-shot LLM planner for ALFRED with two new pieces: MMP (multimodal retrieval for in-context examples using egocentric views plus language) and EAR (LLM-free replanning that replaces an absent object with the most semantically similar detected object). The empirical work is genuine: FLARE with GPT-4 gets 40.88% unseen SR, up from the cited LLM-Planner 16.42%, and the code is public.\n\nWhat the paper does well: the ablations are internally consistent and show each component helps; the qualitative examples are illustrative; the choice to do replanning without calling the LLM is sensible for cost; and the effort to ground retrieval in visual state is a reasonable extension beyond LLM-Planner. The planner-accuracy comparison across four LLMs is a nice check that the gains are not tied to one model.\n\nThe biggest soft spot is that the paper's central claim—that the SOTA gain comes from the proposed components—is not actually supported by its own Table 3. FLARE without either MMP or EAR already reaches 28.58% unseen SR, about 12 points above the cited LLM-Planner. Adding both components yields 31.79%, so the components together contribute roughly 3.2 points. That means the advertised +24.46% gap is mostly explained by the base system: the deterministic FMM navigation, the triplet subgoal representation, the prompt structure, and the language-only retrieval. The paper never re-runs LLM-Planner with the same low-level controller and format, so the Table 1 comparison is confounded. This is fixable—re-run the baseline under matched conditions or at least re-frame the contribution as the full system—but as written the attribution is overstated.\n\nMinor issues: tau (the EAR uncertainty threshold) is never reported even though it is a free parameter in Algorithm 1; no error bars appear anywhere, which matters for a 1.5–2 point ablation difference; and the plan-accuracy metric is exact match, which is sensitive to output format when the two systems use different formats.\n\nShould we take the result seriously? Yes. The direction is sound, the code is out, and the full-system improvement is large enough that even if the components only add a few points, the system is a strong few-shot baseline. With the attribution problem addressed, it would be a useful contribution.\n\nRecommendation: send it to peer review, but the reviewers should press on baseline comparability and the missing hyperparameters.","headline":"Useful few-shot ALFRED system with released code, but the headline gain over LLM-Planner is mostly from the base system, not the two proposed components.","tokens_in":18963,"tokens_out":2211,"would_cite":true,"duration_ms":21351,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An embodied agent can plan household tasks from about 100 language–demonstration pairs when the LLM planner is grounded in visual observations and missing objects are replaced by similar ones in the scene.","keywords":["embodied instruction following","few-shot learning","large language models","task planning","visual grounding","replanning","ALFRED benchmark","multi-modal retrieval"],"falsifier":"Run LLM-Planner with the same low-level action policy, same prompt format, and same LLM version as FLARE on ALFRED test unseen. If the difference in success rate drops below the reported gap, the claimed advantages of multi-modal grounding and visual replanning are not supported.","tokens_in":17853,"feed_emoji":"🤖","tokens_out":8784,"duration_ms":72472,"temperature":0.7,"pith_summary":"This paper claims that a few-shot embodied agent can plan long-horizon household tasks from about 100 language-and-demonstration pairs if the LLM's in-context examples are selected using both the instruction text and the agent's initial visual view, and if a lightweight replanner later substitutes missing objects with semantically similar ones seen in the scene. On the ALFRED benchmark, the resulting system, FLARE, reports the highest few-shot success rates among the compared methods, with up to +24.46 absolute percentage points over the previous best on the test-unseen split when using GPT-4. The authors argue this matters because collecting free-form language annotations is costly, and prior LLM planners rely mostly on linguistic common sense and ignore the environment's current state, producing plans that send the agent searching for objects that are not there.","feed_headline":"Visual grounding lifts LLM planner by 24 points on household tasks","feed_subtitle":"FLARE, with 100 instruction pairs and visual replanning, reports 40.9% ALFRED unseen success, up from 16.4%.","key_machinery":"Key machinery: a multi-modal similarity retrieval that scores each candidate demonstration by the normalized sum of cosine similarities between (a) the current instruction and the demonstration's instruction, both embedded with BERT, and (b) the agent's initial panoramic view and the demonstration's view, both embedded with CLIP; the top-k demonstrations are then represented as compact (Action, Object, Receptacle) triplets for the LLM prompt. This is paired with an environment adaptive replanning rule that, upon failure to find the target object, replaces it with the detected object maximizing cosine similarity of text embeddings. The two equations are Eq. (1) for retrieval and Eq. (3) for replacement; together they convert a purely linguistic plan into one tied to objects actually present.","core_discovery":"FLARE's central claim is that grounding matters at both ends of planning: before the LLM writes a plan, and after it fails. Before, the Multi-Modal Planner retrieves the top-k training demonstrations by a normalized sum of language and visual similarity, embedding the instruction with BERT and the agent's surrounding views with CLIP, then formats them as (Action, Object, Receptacle) triplets in the prompt. After, the Environment Adaptive Replanning module tracks a list of detected objects and, when the agent cannot find the target object, replaces it with the observed object whose text embedding is maximally similar to the missing one—without another LLM call. On ALFRED with 100 examples, the full system reaches 40.88% success (and 51.72% goal-condition) on the test-unseen split with GPT-4 and sequential instructions, versus 16.42% success reported for LLM-Planner; isolating the planner, FLARE without EAR attains 67.48% unseen plan accuracy versus 30.12% for LLM-Planner static with GPT-4. Ablations with GPT-3.5 show each component contributes, and a version with neither still reaches 28.58% unseen success, so the margin over LLM-Planner is not solely due to the proposed components.","pith_inferences":["The paper does not isolate how much of the headline margin comes from multi-modal retrieval versus the base system's different prompt format and low-level policy; a clean test would compare language-only retrieval and multi-modal retrieval under the same LLM and action policy.","EAR's object replacement is an unconstrained lexical lookup; adding a similarity threshold or an action-compatibility check (e.g., a Put target must be a receptacle) would likely reduce false replacements, which the paper does not evaluate.","The same EAR mechanism could be reused for other vocabulary mismatch problems, such as zero-shot navigation with novel object names, by treating any absent target as a candidate for semantic substitution."],"forward_implications":["With 100 training pairs, an agent can reach 40.88% success and 51.72% goal-condition success on ALFRED test-unseen when using GPT-4 with step-by-step instructions, compared with 16.42% for the cited LLM-Planner baseline.","The replanning step is an LLM-free lookup, so correcting a wrong subgoal costs one text-embedding computation rather than another full LLM generation.","The (Action, Object, Receptacle) representation reduces prompt token usage by about 25% relative to LLM-Planner's format, leaving more room for in-context examples.","Using a better backbone LLM produces consistent gains across all four tested models (LLaMA2, Vicuna, GPT-3.5, GPT-4), so the method is not tied to a single model."],"supporting_citations":[{"why":"Defines the few-shot LLM-planner baseline and the 100-example setting; supplies the comparison numbers FLARE claims to beat, and the retrieval-by-language approach that MMP extends.","marker":"(Song et al. 2023)"},{"why":"Provides the ALFRED benchmark, metrics (SR, GC, PLW), data splits, and the expert demonstrations used as retrieval candidates.","marker":"(Shridhar et al. 2020)"},{"why":"Establishes in-context learning with a few examples, the mechanism the multi-modal planner relies on when prompting the LLM.","marker":"(Brown et al. 2020)"},{"why":"Supplies the frozen BERT text encoder used for language similarity in MMP and for computing semantic similarity in EAR.","marker":"(Devlin et al. 2018)"},{"why":"Supplies the frozen CLIP image encoder used to embed the agent's surrounding views for the environment-similarity term of MMP.","marker":"(Radford et al. 2021)"},{"why":"Supplies the fast-marching-method navigation used by the low-level action policy, which FLARE keeps fixed while varying the planner.","marker":"(Sethian 1996)"}],"fun_headline_variants":["Visual grounding lifts LLM planner to 40.9% on ALFRED","FLARE: few-shot visual replanning boosts ALFRED success 2.5x","Visual replanning lifts household task success from 16.4% to 40.9%","Few-shot grounded planner with visual cues hits 40.9% ALFRED","Environmental adaptation corrects LLM plans, ALFRED success 40.9%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparison treats the published LLM-Planner numbers as directly comparable to FLARE's, but the two systems differ in low-level action policy, plan representation, prompt structure, and possibly LLM version, and the paper's own ablation shows most of the headline gap comes from the base system rather than the two proposed components.","fun_headline_variants_meta":{"raw":{"variants":["Visual grounding lifts LLM planner to 40.9% on ALFRED","FLARE: few-shot visual replanning boosts ALFRED success 2.5x","Visual replanning lifts household task success from 16.4% to 40.9%","Few-shot grounded planner with visual cues hits 40.9% ALFRED","Environmental adaptation corrects LLM plans, ALFRED success 40.9%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000503,"raw_usage":{"total_tokens":2488,"prompt_tokens":1010,"completion_tokens":1478,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":626,"completion_tokens_details":{"reasoning_tokens":1364}},"tokens_in":626,"tokens_out":1478,"duration_ms":10995,"temperature":1.0,"reasoning_tokens":1364,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:37:43.512729+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run LLM-Planner with the same low-level action policy, same prompt format, and same LLM version as FLARE on ALFRED test unseen. If the difference in success rate drops below the reported gap, the claimed advantages of multi-modal grounding and visual replanning are not supported.","supporting_citations":[{"cited_title":"H.; Wu, J.; Washington, C.; Sadler, B","cited_arxiv_id":null,"evidence_quote":"Defines the few-shot LLM-planner baseline and the 100-example setting; supplies the comparison numbers FLARE claims to beat, and the retrieval-by-language approach that MMP extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ALFRED benchmark, metrics (SR, GC, PLW), data splits, and the expert demonstrations used as retrieval candidates."},{"cited_title":"W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I","cited_arxiv_id":null,"evidence_quote":"Supplies the frozen CLIP image encoder used to embed the agent's surrounding views for the environment-similarity term of MMP."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the fast-marching-method navigation used by the low-level action policy, which FLARE keeps fixed while varying the planner."}],"review_version":1}