Pith. sign in

REVIEW 2 major objections 4 minor 57 references

A general-purpose model, given only a monocular camera and four movement commands, can sustain the full vision-and-language navigation loop on its own, without a navigation-specific policy, map, depth, or memory.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 03:18 UTC pith:RGCY2RZ2

load-bearing objection Solid zero-shot VLN evaluation with unusually honest reporting; the core minimal-interface result is believable, but the title/abstract overreach the trained-policy comparison because it is cross-split on an unvalidated 100-episode subset. the 2 major comments →

arxiv 2607.26148 v2 pith:RGCY2RZ2 submitted 2026-07-28 cs.RO

Embodied Agents Take Control: Minimal-Interface Zero-Shot Agents Rival Industrial-Scale Policies in Vision-and-Language Navigation

Jian Zhou , Xunyi Zhao , Gengze Zhou , Zerui Li , Sihao Lin , Jiajun Liu , Qi Wu This is my paper
classification cs.RO
keywords embodied agentsvision-and-language navigationzero-shot navigationagentic controlminimal interfacewaypoint toolR2R-CElong-horizon autonomy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Navigation systems for household-style robots usually come with maps, depth sensors, trained policies, or hand-built planning loops. This paper asks what happens if all of that is stripped away: a general-purpose reasoning model is given a single forward-facing camera and four discrete commands—move forward, turn left, turn right, stop—and left to run the whole perception-action loop itself. On a fixed 100-episode sample of the R2R-CE vision-and-language navigation benchmark, this minimal agent reaches 70.7±3.5% success on average at default effort, 78% at maximum effort, and 76.7±0.6% when a trained waypoint module is offered as an optional tool rather than forced. The paper's central claim is that, for frontier models, navigation-specific scaffolding is not necessary: the model itself carries the capability, with model choice dominating performance far more than harness or interface changes. If true, this reframes embodied-agent design from 'what scaffolding to add' to 'what control the model should hold.'

Core claim

The paper claims that a general-purpose reasoning model, equipped with only a monocular RGB camera feed and four discrete movement primitives, can sustain the perceive-act-verify-stop loop of vision-and-language navigation by itself—no learned navigation policy, no map, no depth, no explicit memory, and no navigation-specific training. On a shared 100-episode sample of the R2R-CE val-unseen benchmark, replicated default-effort runs of one frontier model average 70.7±3.5% success; a second frontier model reaches 78% at maximum reasoning effort; and when a trained waypoint module is offered as an optional tool alongside the primitives, the same model hits 76.7±0.6% while using about half the e

What carries the argument

The load-bearing object is the minimal-interface probe: a two-tool coding-agent harness—observe(), which returns a 512×512 front-facing RGB frame, and step(), which executes an ordered list of four discrete primitives (forward 0.25 m, turn ±15°, STOP)—with no depth, panorama, pose, map, waypoint candidates, collision feedback, or cross-episode memory. The mechanism is what the paper calls agentic embodied control: the reasoning model directly decides when to observe, move, recover from a mistake, and stop, rather than serving as one stage in a fixed workflow. The harness maintains the session and executes tools; the interface bounds what the model can perceive and do; the model decides the r

Load-bearing premise

The load-bearing premise is that the fixed 100-episode subset used for the headline numbers fairly represents the full R2R-CE val-unseen split; if those episodes are easier or better conditioned than the full distribution, the claim that the agent rivals trained policies loses its foundation, even though the same-subset comparison with another zero-shot agent would still hold.

What would settle it

Run the identical frozen configuration on the full R2R-CE val-unseen split (or on a fresh random 100-episode draw) and measure success rate; a substantial drop from the 70.7–78% range would falsify the cross-split claim of rivaling trained policies. A second, sharper test: collect multiple independent human baselines on the same 100 episodes—if their success consistently clusters near the single reported 94%, the difficulty calibration of the subset remains unresolved.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If correct, zero-shot vision-and-language navigation does not require navigation-specific scaffolding; a frontier general-purpose model with a camera and primitive actions lands in the performance range of recent trained industrial-scale policies.
  • Changing only the model—while holding the harness and interface fixed—moves success from 5% to 72–78% SR, so capability sits primarily in the model rather than in the surrounding loop.
  • A trained waypoint module is most useful as an optional tool: the hybrid agent adopts a coarse-to-fine strategy, matches near-peak success with roughly half the environment steps and under a quarter of the wall time.
  • Imposing waypoints helps the weakest models (gains of 19–38 SR) but can slightly hurt the strongest, suggesting interface design should compensate for model weaknesses rather than uniformly replace primitive control.
  • The same frozen loop drops to 26–39% SR on longer-horizon RxR-CE, so the headline result is episode-scale; sustained autonomy would require bounded context, verified motion feedback, and persistent spatial state.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: if the fixed 100-episode sample is not representative of the full R2R-CE val-unseen distribution, the abstract-level claim of rivaling trained policies (which were evaluated on the full split) would not be supported; the same-subset comparison against another zero-shot method would still stand.
  • Editorial extension: the failure audit suggests a cheap, testable intervention—give the agent a backtracking or return-to-last-branch tool and measure whether the 20 of 30 route-level failures shrink; the paper does not test this.
  • Editorial extension: the context-growth bottleneck (final-turn contexts reaching 169k tokens) implies that context-windowing or selective memory could be evaluated as a minimal interface change; the paper only conjectures that history dilution is a likely mechanism.
  • Editorial extension: the physical-robot results—reasoning transfers but body awareness, motion verification, and spatial memory do not—point to interface design (collision and pose feedback) as a higher-leverage fix for real-world deployment than further model scaling.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes "agentic embodied control" and evaluates it through a minimal-interface probe: an off-the-shelf coding-agent harness controls a robotic simulator through two tools, observe() (monocular RGB) and step() (four discrete primitives), with no map, depth, memory, or navigation training. On the fixed 100-episode rand100 subset of R2R-CE val-unseen, replicated default-effort runs reach 70.7±3.5 SR (opus-5) and 68.3±1.5 SR (fable-5); a single max-effort fable-5 run reaches 78 SR; and a hybrid interface with an optional trained waypoint tool reaches 76.7±0.6 SR. The paper compares these numbers with AgenticNav (55 SR on the same subset) and with trained policies on the full val-unseen split, and concludes that navigation-specific scaffolding is not necessary for competitive zero-shot VLN. It also reports effort ablations, long-horizon RxR-CE stress, a two-pass audit of all 30 failures in one run, and a physical Unitree Go2 deployment.

Significance. If the result holds, this is a noteworthy empirical finding: a generic agent loop with only a monocular RGB camera and discrete actions can reach performance levels previously associated with navigation-specific architectures or large-scale training. The paper has real methodological strengths: a frozen-configuration protocol with an anti-override run registry (Table 8), three-run replication for the key default-effort cells, complete episode archives, a two-pass audit of all failures, and a physical-robot deployment. The failure taxonomy in §4.6 and the control-authority taxonomy in Appendix F are also useful. The main weakness is the population validity of the 100-episode subset: the headline comparison to trained industrial policies rests on an unvalidated subset, and the abstract's language outruns the paper's own careful caveats in Table 1.

major comments (2)
  1. [§4.1, Table 1, Abstract] The headline claim that the minimal-interface probe "rivals industrial-scale policies" rests on comparing rand100 results (68.3–78 SR) with trained-policy results on the full val-unseen split (Qwen-RobotNav 72, ABot-N1 71). Table 1's note correctly says the trained rows are reference-only, but the abstract and title still assert the comparison. The paper never shows that rand100 is representative of full val-unseen. The only difficulty calibration is a single human tester (94 SR, n=1, Table 1), which, if anything, suggests the subset may be easy. Appendix A.1 also states that the rand100 episodes differ from the release in spawn-heading and instruction-token fields; a changed spawn heading alters the initial observation and can change difficulty, so even same-episode comparisons are not automatically transportable. Without distributional statistics (route length, instruction length, goal
  2. [§4.3, Table 3, Abstract] The abstract's "fable-5 reaches 78% at maximum effort" is based on a single run. Given that the same cell at default effort has a sample standard deviation of 1.5 SR over three runs, and the paper itself cautions that "differences of only a few points are descriptive," the 78% value could be a favorable draw. The max-effort result is not load-bearing for the weaker same-subset claim, but it is used in the abstract as a headline result. Please report replications for the max-effort cell or clearly label it as a single-run exploratory observation and avoid giving it equal weight to the replicated default-effort cells.
minor comments (4)
  1. [Table 1] The "Human" row should state the number of trials and ideally some variance; a single tester with 94 SR is anecdotal, not a benchmark calibration.
  2. [Table 4 / §4.3] The phrasing "forced waypoints help weaker models but can hinder stronger ones" is supported by the VLNVerse reversal for sonnet-5 and fable-5, but on R2R-CE the two strongest models still gain slightly with waypoints (+0.7 and +1.3 SR). Consider clarifying that the "hinder" effect is benchmark-specific and mostly visible on VLNVerse.
  3. [§A.6] The Codex CLI version drift is disclosed, and the paper says no conclusion rests on GPT runs; this is good. To avoid readers over-weighting those rows, a one-sentence repetition of this caveat near Table 2 would help.
  4. [§4.5] The RxR-CE comparison is appropriately hedged as not isolating cause. Please report the number of RxR-CE episodes used (stated as 100 English episodes) in the main text rather than only in the table caption.

Circularity Check

0 steps flagged

No significant circularity: the paper is an externally benchmarked evaluation with native metrics, explicit split caveats, and no fitted parameter renamed as a prediction.

full rationale

This is an empirical evaluation paper rather than a derivation, so the circularity patterns mainly reduce to whether any comparison is forced by construction or by self-citation. The headline success rates are measured with Habitat-native metrics computed driver-side, and the agent is explicitly barred from observing its own score: per A.1, 'the shortest-path and oracle sensors present in the raw observation dictionary reach the tool bridge but are discarded there, never forwarded.' The central comparison to trained industrial policies is not hidden: Table 1 states, 'Trained rows use the full val-unseen split, so they serve only as a reference for the performance range,' and section 4.2 says 'The comparison is not controlled because the systems use different models and serving paths.' The same-subset comparison to AgenticNav (55 SR on rand100) is an external result, not a self-citation. The rand100 subset originates from Open-Nav, which shares an author with the present paper, but it is used only as a fixed episode sample; the paper does not invoke Open-Nav's authority to justify the outcome, and the metric itself is standard VLN-CE SR. The waypoint predictor used in the hybrid/waypoint cells comes from the SmartWay lineage with overlapping authorship, but the paper explicitly flags that these cells are not zero-shot ('the waypoint cells are therefore not zero-shot in the strict sense of the main board'), so no zero-shot capability is being claimed through that module. There is no fitted parameter that is later reported as a prediction, no uniqueness theorem imported from prior work, and no ansatz smuggled in by self-citation. The cross-split comparison to Qwen-RobotNav and ABot-N1 on full val-unseen is a genuine validity limitation: if rand100 is easier than the full split, the 'rivals industrial-scale policies' phrasing is unsupported. That is a benchmarking/representativeness concern, not circularity, and the paper's own Table 1 note already downgrades those rows to a 'reference for the performance range.' Under the stated rules, circularity requires quoting a specific reduction to the paper's inputs, and no such reduction is present.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

Empirical evaluation rather than derivation; the ledger therefore enumerates hand-chosen evaluation settings that the conclusions load on, rather than fitted constants. The model/harness/interface choices are independent variables, but the episode subset, prompt, budgets/effort, and waypoint predictor are human decisions that shape every reported number. No new physical entities are postulated; the two conceptual labels are organizational terms.

free parameters (4)
  • rand100 episode subset (episodes 0-99, with modified spawn-heading and instruction-token fields) = n=100 of R2R-CE val-unseen
    All headline SR figures (68-78) are measured on this 100-episode subset introduced by Open-Nav, a same-group citation (Qi Wu et al.); episodes also deviate from the release in spawn-heading/instruction-token fields (A.1). Trained-policy comparisons use full val-unseen; subset representativeness is assumed, not established.
  • Frozen task briefing (system prompt) = Verbatim text in A.3
    A hand-authored prompt is part of the measured system. It is frozen and disclosed byte-identically across harnesses, but prompt wording is a human choice bundled into the 'agentic control' result.
  • Budget/effort envelope = 200 LLM calls, 500 primitives, 2400s timeout; vendor-default effort (Claude resolves to 'high')
    Defines the operating envelope. The abstract's 78 SR is a single max-effort run of fable-5, not replicated; the default-effort numbers are replicated 3x.
  • Waypoint predictor (SmartWay-lineage RGB-D, depth-driven deployment) = Depth-driven; RGB branch unwired (zeroed)
    Interface and hybrid conclusions are specific to this trained module; the paper states 'This result is specific to the tested predictor' (§4.3). Its quality is a hand-chosen independent variable, not a benchmark property.
axioms (4)
  • domain assumption R2R-CE val-unseen with the 3 m geodesic STOP criterion is a meaningful proxy for embodied navigation competence
    Standard benchmark and metric; invoked throughout §4 and used for all comparative claims.
  • domain assumption The closed proprietary models (fable-5, opus-5, gpt-5.5, etc.) exist as described and their serving behavior is stable
    Required for replication. The paper itself documents Codex CLI version drift (A.6) and 'unmatched serving paths and defaults' (§4.3), so the assumption is soft even on the authors' own account.
  • domain assumption External self-reported numbers (AgenticNav 55, Qwen-RobotNav 72/76.7, FAST-EQA 69.2) are accurately transcribed
    Tables 1 and 10 rely on other papers' self-reported results for the comparative claims; no independent verification is possible from the text.
  • domain assumption The 100-episode rand100 subset resembles the full val-unseen distribution
    Load-bearing for the 'rivals industrial-scale policies' framing; see weakest_assumption. The only difficulty calibration offered is one human tester (94 SR, n=1).
invented entities (2)
  • agentic embodied control no independent evidence
    purpose: Organizing concept: the reasoning model directly steers every action, keeping reasoning and control aligned (§1, §3.1)
    A label for the evaluated design, not a new physical entity; its 'evidence' is the benchmark results themselves, so it does not carry an independent falsifiable handle.
  • Autonomous Embodied Agent (AutoEA) no independent evidence
    purpose: Operational definition of the long-horizon target system (Appendix E): continuous multi-task service, persistent situated state, closed-loop monitoring, resource-bounded self-maintenance, safe adaptation
    Explicitly a target definition, not a realized system; the paper states 'Our experiments remain episodic and do not claim to realize an AutoEA.' No independent evidence is claimed or needed.

pith-pipeline@v1.3.0-alltime-deepseek · 5443 in / 5426 out tokens · 503478 ms · 2026-08-04T03:18:31.866957+00:00 · methodology

0 comments
read the original abstract

Autonomous embodied agents must sustain a long decision-making loop that involves perceiving, acting, verifying, and self-correcting over many steps. Current systems sustain this loop through task-specific workflows or embodied policies. However, these fixed workflows and policies offer limited flexibility across environments and often lack effective recovery strategies when execution goes wrong. We find that a general-purpose agent can instead sustain the loop on its own. We term this organization agentic embodied control: the reasoning model directly steers every action, keeping reasoning and control aligned. Using zero-shot navigation as a controlled testbed, we equip three coding-agent harnesses with only a monocular RGB camera and discrete actions. At default effort, replicated opus-5 runs average $70.7\pm3.5$% success, while fable-5 reaches 78% at maximum effort. When a trained waypoint tool is offered alongside primitives, the hybrid fable-5 agent reaches $76.7\pm0.6$% at default effort, using half the environment steps and under a quarter of the wall time. Across the ablations, model choice dominates performance variation. Observed harness differences are modest, and forced waypoints help weaker models but can hinder stronger ones. Although longer horizons, latency, and context growth remain barriers to sustained autonomy, these results show that a general-purpose model can already achieve competitive embodied control without a navigation policy.

Figures

Figures reproduced from arXiv: 2607.26148 by Gengze Zhou, Jiajun Liu, Jian Zhou, Qi Wu, Sihao Lin, Xunyi Zhao, Zerui Li.

Figure 1
Figure 1. Figure 1: Who holds the interaction loop, what we hand the model, and how far that gets it. Left: at inference the loop is directed by an external environment loop (policy), human-authored pipeline code (workflow), or the model itself (agentic), with systems placed by the loop they execute, trained (T) or zero-shot (ZS). Middle: our minimal-interface probe, a software-engineering harness with one monocular RGB view … view at source ↗
Figure 2
Figure 2. Figure 2: One successful R2R-CE episode, reconstructed from its raw log. Six of 20 archived observations, each with the step() call issued from that view and the recorded reasoning verbatim ([. . . ] = elision). The bottom tape lists all 111 primitives in order (F = 0.25 m forward, L/R = 15◦ turn). Amber marks segments the model itself describes as blocked or off-route. No map, pose, depth, panorama, or waypoint is … view at source ↗
Figure 2
Figure 2. Figure 2: One successful R2R-CE episode, reconstructed from its raw log. Six of 20 archived observations, each with its step() call and the recorded reasoning verbatim. The bottom tape lists all 111 primitives. Amber marks segments the model calls blocked or off-route. mini-swe-agent, fable-5, default effort. Agentic control remains rare in embodied navigation. Yet this is where the zero-shot line began (NavGPT’s Re… view at source ↗
Figure 3
Figure 3. Figure 3: Locating capability across model, harness, and interface (shared SR scale). (A) With harness and interface fixed, the model spans 5–72 SR. (B) The open and vendor harness differ by 2–7 SR across the six shared pairs—mini-swe-agent leads in five, the vendor harness in one (descriptive; serving paths differ). (C) A waypoint interface rescues the weakest models on R2R-CE but reverses for strong models on VLNV… view at source ↗
Figure 3
Figure 3. Figure 3: Locating capability across model, harness, and interface (shared SR scale). (A) The model spans 5–72 SR. (B) Open and vendor harnesses differ by 2–7 SR. (C) A waypoint interface rescues the weakest models on R2R-CE but reverses for strong models on VLNVerse (section 4.3) [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: ep2, category A (wrong referent / wrong branch). The kitchen has two sinks. Obs 1, 3 and 6: the route itself is right. The agent turns off the spawn wall, skirts the dining table, and enters the kitchen aisle. Obs 9 (amber): the annotated far sink is visible through the doorway at the end of the aisle, and the agent even resolves to “see if there’s another sink ahead”, a check it never executes. Obs 12: in… view at source ↗
Figure 5
Figure 5. Figure 5: ep91, category B (stop-decision failure). The cleanest exhibit of object-anchored stopping. Obs 1: spawn facing a wall. Obs 3: after turning, the geometry is parsed correctly: pool right, bar ahead-left. Obs 5: “roughly 4–5 meters away”, and it is. The estimate chain stays arithmetically consistent to the end. Obs 8: STOP with the bar corner ∼2 m away and 467 primitives unused. The failure is entirely in t… view at source ↗
Figure 5
Figure 5. Figure 5: ep56, category B (stop-decision failure). The overshoot variant of object-anchored stopping. Obs 4: the correct branch, the exit-sign door left of the red canopy bed. Obs 12: standing at the annotated endpoint’s doorway, reasoning “well within the 3-meter limit”. The episode is, at this moment, a success (oracle success). Obs 25 (amber): a mid-hall survey re-parses the instruction as two further legs. Obs … view at source ↗
Figure 6
Figure 6. Figure 6: ep59, category C (runaway search). No exercise room is ever found, so one is invented. Obs 1 and 6: the exit from the kitchen into the tiled hall is correct. The agent then clears the wrong wing room by room. Obs 56: the surrogate: a home-gym rack standing in the master bedroom, the bed visible in the same frame. Obs 62 (amber): wedged in a curtain gap (the next frame is byte-identical). Obs 96: it walks b… view at source ↗
Figure 6
Figure 6. Figure 6: ep7, category C (runaway search). Obs 10: the correct left turn into the hallway. The endpoint lies dead ahead. Obs 13 (amber): “the end of the hallway” binds to the gym entrance and the agent turns aside. Obs 26: closest approach, goal 4.4 m dead ahead, unrecognised. Obs 39: clipped into the treadmill mesh (pixel-identical frames). Obs 86: after sweeping the house it stops at torn geometry read as “the fo… view at source ↗
Figure 7
Figure 7. Figure 7: ep98, category D (geometry / simulator trap). What the missing collision signal costs. Obs 12 and 26 (amber): the hallway chokepoint, the same view 67 env steps apart. ∼100 primitives spent with zero progress, every one reported as executed. Obs 41: the re-route succeeds (the dining room is reached, the kitchen doorway visible beyond the table), but chair meshes seal every path to it. Obs 55 and 72 (amber)… view at source ↗
Figure 8
Figure 8. Figure 8: Condition and reasoning navigation, success. go2 ep16. The instruction hides a logic test. The agent turns around, evaluates that the stated arithmetic is false, and drives to the yellow tape rather than the purple decoy. The branch is settled before the first step. INTERACT multi-person interaction nav · go2 ep25 “Go to the remote control that fell on the floor behind you. Assuming you could pick it up, b… view at source ↗
Figure 8
Figure 8. Figure 8: Conditional navigation, success. go2 ep16. The agent evaluates the arithmetic premise as false before moving and selects the yellow tape rather than the purple decoy. EMBODIMENT exit-door embodiment · go2 ep7 (control) “Exit through the door, turn left, and walk to the cardboard cutout of a person.” The control for ep6 — same start, same instruction. Here it drives a full F×6 straight out before turning, s… view at source ↗
Figure 9
Figure 9. Figure 9: Multi person interaction, success. go2 ep25. Told to bring a remote to the person wearing white shoes, the agent separates the seated target from a standing bystander in gray from the first frame and stops in front of the correct one. The one weak point is that the small remote falls below the low camera, so the pickup is assumed rather than confirmed. EMBODIMENT enter a room, recover at the door · real ep… view at source ↗
Figure 9
Figure 9. Figure 9: Doorway control, success. go2 ep7. The robot clears the opening before turning and reaches the target. The start and instruction are identical to fig. 10. D.3 Where Hardware Exposes the Gap Body awareness. The designed pair in figs. 9 and 10 holds the start, instruction, doorway, and target constant. In the successful run, the agent drives fully through the opening before turning. In the failed run, it tur… view at source ↗
Figure 10
Figure 10. Figure 10: Entering a room with a doorway recovery, success. real ep4. The agent finds the kitchen on the right and, at the narrow doorway, reads a body catch correctly as its own unseen rear, frees itself with small pivots, and stops inside. This is the positive counterpart to the exit failure in fig. 12. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_10.png] view at source ↗
Figure 10
Figure 10. Figure 10: Doorway body catch, failure. go2 ep6. The agent turns before the body behind the lens has cleared the opening. The rear catches the wall edge, after which the agent misreads its pose and never reaches the target. COUNT counting-gate (ordinal) · real ep3 “Walk forward and pass the white pillar on your right. Once you have completely passed two pillars, turn right, go forward a bit, and stop.” 'Pass two pil… view at source ↗
Figure 11
Figure 11. Figure 11: Exit through a doorway, control, success. go2 ep7. Same start and instruction as fig. 12. The agent drives fully out of the door before turning, so the turn is clean and it walks straight to the cutout. EMBODIMENT exit-door embodiment · go2 ep6 “Exit through the door, turn left, and walk to the cardboard cutout of a person.” Turns left the moment the camera clears the doorway — but the ~50 cm body behind … view at source ↗
Figure 11
Figure 11. Figure 11: Cross-view counting, failure. real ep3. The agent reads the corridor correctly but does not retain a stable pillar count, turning after the third pillar rather than the second. It relabels the second pillar and turns only after the third. The failure is therefore not elementary counting but maintenance of grounded state through motion. Longer routes expose the same weakness in a spatial frame. Real ep2 su… view at source ↗
Figure 12
Figure 12. Figure 12: Exit through a doorway, body catch, failure. go2 ep6. The agent turns while still in the opening, before the body behind the lens has cleared it. The rear jams the wall edge, it misreads its pose as facing back into the room, and it never reaches the target. It does not leave room for its own body to turn. jams the wall edge, the agent then misreads its own pose as facing back into the room it just left, … view at source ↗
Figure 13
Figure 13. Figure 13: Counting gate, failure. real ep3. Asked to pass two pillars and turn, the agent reads the corridor correctly but never fixes the count. It relabels the second pillar across frames and turns only at the end wall, one pillar too far. MULTI-GOAL multi-goal (long-horizon) · go2 ep17 “Walk to the yellow tape on the floor, then turn around and walk to the robot arm on the table.” Leg 1 (yellow tape) is clean. A… view at source ↗
Figure 14
Figure 14. Figure 14: Long route with a reference flip, failure. go2 ep17. The first leg to the tape is clean. After the agent turns around, its memory of the target seen earlier decays, and it stops at a monitor cart that it reports as the robot arm. The clearest terminal error of the set. on a heuristic that a floor target which drops below the low camera must lie underfoot, which stops it early or late whenever the geometry… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

57 extracted references · 13 linked inside Pith

  1. [1]

    Zhang, K

    J. Zhang, K. Wang, R. Xu, G. Zhou, Y . Hong, X. Fang, Q. Wu, Z. Zhang, and H. Wang. Navid: Video-based vlm plans the next step for vision-and-language navigation.arXiv preprint arXiv:2402.15852, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    M. Wei, C. Wan, X. Yu, T. Wang, Y . Yang, X. Mao, C. Zhu, W. Cai, H. Wang, Y . Chen, et al. Streamvln: Streaming vision-and-language navigation via slowfast context modeling.arXiv preprint arXiv:2507.05240, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    G. Zhou, Y . Hong, and Q. Wu. Navgpt: Explicit reasoning in vision-and-language navigation with large language models. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7641–7649, 2024

    2024

  4. [4]

    J. Chen, B. Lin, R. Xu, Z. Chai, X. Liang, and K.-Y . Wong. Mapgpt: Map-guided prompting with adaptive path planning for vision-and-language navigation. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9796–9810, 2024

    2024

  5. [5]

    B. Lin, Y . Nie, Z. Wei, J. Chen, S. Ma, J. Han, H. Xu, X. Chang, and X. Liang. Navcot: Boosting llm-based vision-and-language navigation via learning disentangled reasoning.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    2025

  6. [6]

    G. Zhao, G. Li, and Y . Yu. Navgemini: a multi-modal llm agent for vision-and-language navigation.Visual Intelligence, 4(1), 2026

    2026

  7. [7]

    R. Gong, Y . Guo, J. Hu, J. Kong, X. Leng, T. Li, W. Li, F. Liu, Z. Liu, J. Lu, M. Luo, C. Ming, Y . Shen, J. Tao, Z. Wang, M. Yin, et al. ABot-N1: Toward a general visual language navigation foundation model, 2026. URL https://arxiv.org/abs/2607.10383

    Pith/arXiv arXiv 2026

  8. [8]

    M. Wei, C. Wan, J. Peng, X. Yu, Y . Yang, D. Feng, W. Cai, C. Zhu, T. Wang, J. Pang, and X. Liu. Ground slow, move fast: A dual-system foundation model for generalizable vision-and-language navigation, 2025. URL https://arxiv.org/abs/2512.08186

    arXiv 2025

  9. [9]

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao. ReAct: Synergizing reasoning and acting in language models. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  10. [10]

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan. SWE-bench: Can language models resolve real-world github issues? InThe Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VTF8yNQM66

    2024

  11. [11]

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press. SWE-agent: Agent-computer interfaces enable automated software engineering. InNeurIPS, 2024

    2024

  12. [12]

    Anderson, Q

    P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. Van Den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 3674–3683, 2018

    2018

  13. [13]

    Krantz, E

    J. Krantz, E. Wijmans, A. Majumdar, D. Batra, and S. Lee. Beyond the nav-graph: Vision-and- language navigation in continuous environments. InEuropean Conference on Computer Vision, pages 104–120. Springer, 2020

    2020

  14. [14]

    Zhang, G

    J. Zhang, G. Zhou, H. Yin, Y . Huang, Z. Lei, Q. Peng, H. Yuan, J. Zhang, X. Guo, X. Chen, A. Yang, F. Huang, J. Lin, D. Liu, J. Zhou, Z. Yu, J. Fan, Z. Liang, P. Lin, Y . Wang, A. Chen, K. Yan, X. Xu, J. Li, L. Hu, M. Zhang, S. Li, W. Xiao, S. Bai, X. Ren, C. Lv, C. Wu, and X.-H. Chen. Qwen-RobotNav technical report: A scalable navigation model designed ...

    Pith/arXiv arXiv 2026

  15. [15]

    Zhang, A

    J. Zhang, A. Li, Y . Qi, M. Li, J. Liu, S. Wang, H. Liu, G. Zhou, Y . Wu, X. Li, Y . Fan, W. Li, Z. Chen, F. Gao, Q. Wu, Z. Zhang, and H. Wang. Embodied navigation foundation model, 2025. URL https://arxiv.org/abs/2509.12129

    arXiv 2025

  16. [16]

    Driess, F

    D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y . Chebotar, P. Sermanet, D. Duckworth, S. Levine, V . Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. Florence. PaLM-E: An embodied multimodal language model. InProceedings of the 40th International Confere...

    2023

  17. [17]

    Brohan, N

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y . Kuang, I. Leal, L. Lee, T.-W. E. Lee, S. Levine, Y . Lu, H. Michalewski, I. Mordatch, K. Pe...

    2023

  18. [18]

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn. OpenVLA: An open-source vision-language-action model, 2024

    2024

  19. [19]

    Black, N

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky.π0: A vision-language- action flow model for general robot control, 2024

    2024

  20. [20]

    Y . Qiao, W. Lyu, H. Wang, Z. Wang, Z. Li, Y . Zhang, M. Tan, and Q. Wu. Open-nav: Exploring zero-shot vision-and-language navigation in continuous environment with open-source llms. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 6710–6717. IEEE, 2025

    2025

  21. [21]

    X. Shi, Z. Li, W. Lyu, J. Xia, F. Dayoub, Y . Qiao, and Q. Wu. Smartway: Enhanced waypoint prediction and backtracking for zero-shot vision-and-language navigation. In2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 16923–16930. IEEE, 2025

    2025

  22. [22]

    Y . Long, X. Li, W. Cai, and H. Dong. Discuss before moving: Visual language navigation via multi-expert discussions. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 17380–17387. IEEE, 2024

    2024

  23. [23]

    Y . Long, W. Cai, H. Wang, G. Zhan, and H. Dong. Instructnav: Zero-shot system for generic instruction navigation in unexplored environment. InConference on Robot Learning, pages 2049–2060. PMLR, 2025

    2049

  24. [24]

    D. Li, W. Chen, and X. Lin. TINA: Think, interaction, and action framework for zero-shot vision language navigation, 2024. URL https://arxiv.org/abs/2403.08833

    Pith/arXiv arXiv 2024

  25. [25]

    Jeong, G.-C

    S. Jeong, G.-C. Kang, J. Kim, and B.-T. Zhang. Zero-shot vision-and-language navigation with collision mitigation in continuous environment, 2024. URL https://arxiv.org/abs/2410.17267

    Pith/arXiv arXiv 2024

  26. [26]

    H. Li, X. Dong, H. Jiang, Y . Zhou, and X. Ma. CMMR-VLN: Vision-and-language navigation via continual multimodal memory retrieval, 2026. URL https://arxiv.org/abs/2603.07997. 12

    arXiv 2026

  27. [27]

    Z. Li, S. Li, Z. Zhang, B. Li, and S. Zhou. DV-VLN: Dual verification for reliable LLM-based vision-and-language navigation, 2026. URL https://arxiv.org/abs/2601.18492

    arXiv 2026

  28. [28]

    Bjorck, Z

    J. Bjorck, Z. Li, Y . Man, J. Wang, A.-C. Cheng, S. Liu, S. Wang, Z. Yu, A. Badki, S. Birchfield, V . Blukis, Y . Chebotar, S. Chen, S. Leng, Y .-C. Chou, T. Ding, B. Li, Z. Luo, H. Su, J. Tremblay, T. Wang, B. Wen, J. Wu, X. Xie, H. Ye, H. Yin, K. R. Zentner, L. Gui, Y .-X. Wang, Y . Zhu, L. Fan, and J. Kautz. Vesta: A generalist embodied reasoning model...

    Pith/arXiv arXiv 2026

  29. [29]

    J. Zhou, S. Lin, J. Li, S. Fu, G. Zhou, and Q. Wu. Automating the design of embodied agent architectures, 2026. URL https://arxiv.org/abs/2606.30111

    Pith/arXiv arXiv 2026

  30. [30]

    mini-swe-agent: The minimal AI software engineering agent

    The SWE-agent team. mini-swe-agent: The minimal AI software engineering agent. https: //github.com/SWE-agent/mini-swe-agent, 2025

    2025

  31. [31]

    G. Wang, Y . Xie, Y . Jiang, A. Mandlekar, C. Xiao, Y . Zhu, L. Fan, and A. Anandkumar. V oyager: An open-ended embodied agent with large language models.Transactions on Machine Learning Research, 2024

    2024

  32. [32]

    Y . Li, C. Li, H. Shi, J. Luo, J. Cai, M. Yang, and T. Qin. AgenticNav: Zero-shot vision-and- language navigation as a tool-calling harness, 2026. URL https://arxiv.org/abs/2606.10577

    Pith/arXiv arXiv 2026

  33. [33]

    Savva, A

    M. Savva, A. Kadian, O. Maksymets, Y . Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V . Koltun, J. Malik, et al. Habitat: A platform for embodied ai research. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 9339–9347, 2019

    2019

  34. [34]

    Cheng, Y

    A.-C. Cheng, Y . Ji, Z. Yang, Z. Gongye, X. Zou, J. Kautz, E. Bıyık, H. Yin, S. Liu, and X. Wang. NaVILA: Legged robot vision-language-action model for navigation, 2024. URL https://arxiv.org/abs/2412.04453

    Pith/arXiv arXiv 2024

  35. [35]

    Z. Wang, X. Yu, Y . Rao, Y . Ling, Y . Li, O. Wang, M. Gao, Y . Zhou, Y . Liang, Z. Liu, Y . Zhang, R. Huang, X. Xu, B. Yuan, Y . Yuan, X. Tan, H. Zhang, Y . Huang, S. Zhang, H. Wu, H. Hu, and Z. Zhang. Hy-Embodied-VLM-1.0: Efficient physical-world agents, 2026. URL https: //arxiv.org/abs/2607.12894

    Pith/arXiv arXiv 2026

  36. [36]

    R. Dang, J. Guo, B. Hou, S. Leng, K. Li, X. Li, J. Liu, Y . Mao, Z. Wang, Y . Yuan, M. Zhu, X. Lin, Y . Bai, Q. Jiang, Y . Zhao, M. Zeng, J. Gao, Y . Jiang, J. Cen, S. Huang, L. Wang, W. Zhang, C. Liu, J. Yang, S. Lu, and D. Zhao. RynnBrain: Open embodied foundation models,

  37. [37]

    X. Xue, J. Hu, M. Luo, S. Xie, J. Chen, Z. Xie, K. Quan, W. Guo, M. Xu, and Z. Chu. OmniNav: A unified framework for prospective exploration and visual-language navigation, 2025. URL https://arxiv.org/abs/2509.25687

    arXiv 2025

  38. [38]

    Claude agent SDK

    Anthropic. Claude agent SDK. https://docs.anthropic.com/en/api/agent-sdk, 2025

    2025

  39. [39]

    Unrolling the Codex agent loop

    OpenAI. Unrolling the Codex agent loop. https://openai.com/index/ unrolling-the-codex-agent-loop/, 2026

    2026

  40. [40]

    S. Lin, Z. Li, X. Zhao, G. Zhou, L. Wang, R. Wei, R. Tang, J. Li, H. Wang, J. Pang, A. van den Hengel, J. Liu, and Q. Wu. VLNVerse: A benchmark for vision-language navigation with versatile, embodied, realistic simulation and evaluation, 2025. URL https://arxiv.org/abs/2512. 19021

    2025

  41. [41]

    Y . Hong, Z. Wang, Q. Wu, and S. Gould. Bridging the gap between learning in discrete and continuous environments for vision-and-language navigation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15439–15449, 2022. 13

    2022

  42. [42]

    A. Ku, P. Anderson, R. Patel, E. Ie, and J. Baldridge. Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4392–4412, 2020

    2020

  43. [43]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022

    2022

  44. [44]

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . K. Li, Y . Wu, and D. Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  45. [45]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  46. [46]

    Chang, A

    A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner, M. Savva, S. Song, A. Zeng, and Y . Zhang. Matterport3D: Learning from RGB-D data in indoor environments. InInternational Conference on 3D Vision, pages 667–676, 2017. doi:10.1109/3DV .2017.00081

    arXiv 2017

  47. [47]

    Model context protocol

    Anthropic. Model context protocol. https://modelcontextprotocol.io, 2024

    2024

  48. [48]

    LiteLLM documentation

    BerriAI. LiteLLM documentation. https://docs.litellm.ai/, 2026

    2026

  49. [49]

    A. Z. Ren, J. Clark, A. Dixit, M. Itkina, A. Majumdar, and D. Sadigh. Explore until confident: Efficient exploration for embodied question answering. InRobotics: Science and Systems (RSS), 2024

    2024

  50. [50]

    {instruction}

    H. Zhang, N. Savaliya, F. Siddiqui, and E. Sachdeva. FAST-EQA: Efficient embodied ques- tion answering with global and local region relevancy. InIEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2026. 14 A Experimental Settings This appendix records the complete experimental configuration. The authoritative source is the released run r...

    2026

  51. [53]

    New goals may interrupt, revise, or depend on earlier ones

    Continuous multi-task service.The system accepts successive goals at run time and completes feasible tasks without routine process, session, or environment reset. New goals may interrupt, revise, or depend on earlier ones. Task boundaries do not erase the robot’s operational context

  52. [54]

    It manages what to retain, forget, or consolidate rather than replaying the complete raw interaction history

    Persistent situated state.The system maintains task-relevant state across goals, such as household layout, object locations, user preferences, unfinished work, and prior failures. It manages what to retain, forget, or consolidate rather than replaying the complete raw interaction history. The state may reside in maps, external stores, context, or learned ...

  53. [55]

    Closed-loop monitoring and recovery.The system checks the effects of its actions rather than assuming successful execution. It detects loss of progress, localization or tool failures, blocked motion, and relevant environmental change, then re-observes, backtracks, or re-plans to restore goal-directed operation. Human rescue is an exceptional response to c...

  54. [56]

    Resource-bounded self-maintenance.The system operates under explicit budgets for decision latency, compute, model-call rate, retained state and storage, and energy use. Context and retained state are compacted or offloaded before they grow without bound, and the robot manages physical needs such as charging rather than relying on a reset to restore resources

  55. [57]

    dual-system,

    Safe adaptation and escalation.A household is shared with people and changes over time. The system updates its state and plans as users, object locations, and routines change, while respecting operational and safety constraints. It recognizes uncertainty or conditions it cannot safely resolve, stops when necessary, and requests help selectively rather tha...

  56. [200]

    the route should be around 3 hops with a path length of about 10 meters, but I’ve already traveled 15+

    The plain gpt-5.6 identifier is not served to ChatGPT-subscription accounts, so these runs use the account’sgpt-5.6-solvariant. Effort tiers are given in section A.4. A.7 Waypoint interface The waypoint arm of table 4 replaces the primitive action space with three tools. observe() renders a 12-view RGB-D panorama, feeds it to a trained candidate-waypoint ...

    2026

  57. [2026]

    URL https://arxiv.org/abs/2602.14979

    arXiv