{"id":"778e0f21-9051-449b-86f5-65709392cd2b","arxiv_id":"2411.13262","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Fine-tuned small language models, coached by a GPT-4 teacher through iterative prompting, can perform multi-point robot navigation on edge devices with success rates approaching larger models.","lead":"The paper proposes FASTNav, a way to fine-tune small language models so robots can follow multi-point navigation commands on cheap edge hardware. It combines LoRA fine-tuning with a GPT-4 teacher that iteratively coaches the small model, and it reports success rates close to GPT-4 on simulated and real navigation tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Teacher–student iteration may tune prompts on the test set itself: Algorithm 1 takes expected goals G as inputs and Figure 4 reports test-set accuracy during iteration, so the reported gains may reflect label leakage rather than model improvement.","rationale":"The central contribution is not the coordinate-output formulation (that is a task simplification) but the claim that LoRA fine-tuning plus teacher–student iteration lets a 0.5–3B model reach GPT-4-level navigation accuracy. For that claim to hold, the iteration phase must be a training procedure, not a test-time optimization loop. Algorithm 1's explicit 'expected goals G' input and 'UpdateFeedback(result, success)' make the latter possible. The Figure 4 caption reports test-set accuracy during iteration, which is exactly the situation in which leakage would be visible. If the teacher is told, for each test task, whether the previous prediction matched the goal coordinates, it can craft prompts that encode the correct answer or nudge the student toward it; after several iterations the test accuracy can rise without any real generalization. This would explain the large improvements in Figure 6b and would invalidate both the '4-fold increase' statement and the 'close to GPT-4' comparison. The fix is straightforward and standard: restrict iteration to the training split and evaluate once, on a frozen held-out split. The paper does not report such a protocol. I therefore cannot treat the headline numbers as trustworthy evidence. I disagree with the reader's choice of weakest assumption: the coordinate-only representation is a stated limitation with explicit assumptions, whereas the evaluation protocol is a potential validity flaw that threatens the measured results directly. A conditional acceptance would be appropriate only with a clean re-evaluation showing the effect survives holdout.","tokens_in":10407,"tokens_out":5834,"duration_ms":60256,"concrete_test":"Re-run the full FASTNav pipeline with an explicit split: (i) fine-tune on the training tasks; (ii) run teacher–student iteration using only training tasks, withholding all 100 test tasks and their goal coordinates from the teacher and from feedback; (iii) after iteration, freeze the student and evaluate on the untouched test set. Report success rates and the iteration curve for this held-out split. If openllama-3B still reaches ~70% and tau-0.5B ~63%, the leakage concern is resolved. If the rates drop substantially, the reported gains are test-set artifacts.","verdict_should_be":"REJECT","load_bearing_attack":"The central empirical claim is that the teacher–student iteration module raises SLMs to near-GPT-4 success rates. That claim requires that iteration improve the model or its prompting on training tasks and that the reported success rates measure held-out generalization. As written, the paper does not establish this separation. Algorithm 1 (Section V-B) lists 'expected goals G' as an input, and inside the loop it computes success = (result[positions] == Gi) and calls UpdateFeedback(result, success). The teacher then uses that feedback to generate the next prompt for the same task. Figure 4's caption says the plotted curve is 'accuracy ... on the test set during fine-tuning and iteration processes.' Taken together, this means the teacher can receive per-task ground-truth success signals for what is supposed to be the evaluation set, and can adjust prompts accordingly. A GPT-4-class teacher performing per-instance prompt optimization with access to correct goal coordinates could plausibly produce high success rates without the student ever learning a general navigation policy. The 30–40% SR improvement attributed to iteration in the ablation (Figure 6b) may therefore be an artifact of optimizing on the test data. The paper never states that the iteration phase is restricted to a training split and that test tasks are never shown to the teacher. This is a load-bearing validity threat: if it lands, the headline '4-fold increase' and 'close to GPT-4' are not supported.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"FASTNav proposes a pipeline for boosting small language models (SLMs) for multi-point robot navigation, consisting of (1) LoRA fine-tuning on a human-in-the-loop generated dataset, (2) a teacher-student iteration where GPT-4 acts as a prompt engineer that receives success feedback, and (3) a Navigation2-based controller that executes the model's output as an ordered list of coordinates. The paper reports simulation experiments on a hospital environment, comparisons against quantization/distillation baselines and LLM-based planning methods, ablations of the fine-tuning and iteration modules, and real-robot deployments on a DIABLO platform with a Jetson Orin NX. The central claim is that fine-tuned and iterated SLMs (e.g., openllama-3B, tau-0.5B) achieve success rates close to or above those of much larger models—a 4-fold accuracy increase over the base SLMs—while remaining lightweight and locally deployable.","tokens_in":10688,"tokens_out":3631,"duration_ms":37642,"significance":"If the reported results hold, FASTNav would be a practically useful contribution: it demonstrates that a sub-3B model can carry out multi-point navigation with high success after task-specific fine-tuning and iterative prompt refinement, and it validates the approach on real hardware. The paper ships a concrete pipeline with code-level descriptions (via PEFT and Navigation2), real-robot evaluations, and ablations that decompose the contributions of fine-tuning and iteration. These are strengths: the work addresses an actual deployment problem (privacy, latency, edge compute) and provides falsifiable comparisons. However, the significance is currently conditional on resolving evaluation-validity concerns, because the teacher-student loop as described may optimize on the test set and the headline comparisons against compression baselines are not apples-to-apples.","major_comments":[{"comment":"The teacher-student iteration procedure as written may leak test-set information into prompt optimization. Algorithm 1 takes expected goals G as input and, inside the loop, sets success = 1 if result[positions] == Gi, then calls UpdateFeedback(result, success). If this feedback is computed for test tasks, the teacher (GPT-4) receives per-task ground-truth success signals for the evaluation set and can adjust prompts accordingly. Figure 4's caption explicitly states that the plotted curves show accuracy on the test set during fine-tuning and iteration. The paper never states that the iteration phase is restricted to a training split and that test tasks are never shown to the teacher. If the reported 30–40% SR improvement from iteration (Figure 6b) is obtained by per-instance prompt optimization on the test set, the central claim of student generalization is not supported. Please clarify whether the iteration curves and Table II/Figure 6 results reflect held-out tasks, and if so, report the exact data-splitting protocol. If iteration was performed on the test set, the experiments must be redone on a separate validation split.","section":"§V-B, Algorithm 1, Figure 4"},{"comment":"The comparison in Table II is not a controlled comparison of methods: FASTNav models receive task-specific LoRA fine-tuning and teacher-student iteration, whereas the compression baselines (Llama3-8B-GPTQ, Mistral-7B-GPTQ, Qwen-7B-Chat-Int4, distilled-MiniMA-3B) are used off-the-shelf with no task adaptation. The reported ≥30% success-rate gap therefore conflates the proposed method with the fact that the baselines were not adapted to the navigation task. A fair comparison would fine-tune the compressed models on the same dataset, or evaluate FASTNav without the fine-tuning/iteration modules, and then compare. As written, the claim that FASTNav 'improves on other model compression methods' is overstated because it compares an adapted system against unadapted checkpoints.","section":"§VI-A, Table II"},{"comment":"The evaluation of 'close to GPT-4' is partly self-referential: GPT-4 is used to generate the fine-tuning dataset (Section V-A), to act as the teacher that produces prompts and receives feedback (Section V-B), and as the reference model in Figure 4. Because the test set is human-designed and success is judged by navigation outcomes, this is not a fatal circularity, but the 'close to GPT-4' claim measures how well the student mimics a teacher on tasks that the same model family helped shape. The paper should state what independent evidence supports the absolute accuracy claim, and should report whether the test tasks were never seen by the teacher during iteration.","section":"§V-A, §V-B, Figure 4"}],"minor_comments":[{"comment":"The success-rate values in Table II (e.g., 36.67, 23.33, 6.67) appear to be based on only 30 test tasks; no confidence intervals or statistical tests are reported. Please add error bars or exact counts (e.g., 11/30) for the main comparisons.","section":"§VI-A, Table II"},{"comment":"The column 'Performance (%)' is undefined; specify which benchmark or metric it refers to, and add the evaluation condition (e.g., zero-shot, few-shot).","section":"Table I"},{"comment":"The pseudo-code does not define the functions ReadFeedback, GeneratePrompt, Completion, and UpdateFeedback in terms of inputs and outputs, and it is unclear how the teacher selects prompts across iterations. Providing a formal specification would help reproducibility.","section":"§V-B, Algorithm 1"},{"comment":"The dataset sizes are given as 'approximately 1400' and 'about 100' tasks; please report exact numbers, the train/test split, and the number of tasks per complexity bin (single-, two-, three-, multi-goal).","section":"§VI-A"},{"comment":"There is a typographical spacing issue in 'F ASTNav' in the abstract; please fix to 'FASTNav'.","section":"Abstract"},{"comment":"The Navigation Error metric as written averages distances over goals, but the normalization (e.g., per task, per goal) is not specified; clarify how d(ĝ_i, g_i) is aggregated when tasks have different numbers of goals.","section":"§VI-B, Equation (4)"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about test-set leakage is the core validity threat, and the manuscript as written does not rule it out. The authors should be asked to (a) confirm whether the teacher-student iteration ever saw the test tasks, and (b) re-run the reported experiments with a properly separated validation set if needed. The comparison against compression baselines also needs a controlled setup. If the authors can provide those clarifications and rerun the affected experiments, the paper could be publishable; as it stands, the central empirical claim is not yet supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the core idea is genuinely useful: take a small language model, LoRA-fine-tune it on navigation tasks, then run a GPT-4 teacher that iteratively rewrites prompts based on the student's mistakes. That is a sensible way to squeeze task-specific performance out of sub-3B models on edge hardware, and the real-robot experiments on a Jetson Orin NX are a plus. Second, the paper has a load-bearing ambiguity that could sink the whole empirical claim: it never states whether teacher-student iteration runs on a training split or on the test set.\n\nAlgorithm 1 takes 'expected goals G' as input and computes success = (result[positions] == Gi), using that to update the teacher's feedback. Figure 4 plots 'accuracy on the test set during fine-tuning and iteration processes.' If the iteration loop is running on the same 100 test tasks and feeding per-task success back to the teacher, then the teacher is effectively optimizing prompts on the test set. That would explain the 30–40% SR gains from iteration without the student learning any general navigation policy. The paper simply does not tell us which split T refers to. This is not a minor omission; it determines whether the headline 'close to GPT-4' result means anything.\n\nThe comparison table has a separate but related problem. Table II pits off-the-shelf quantization/distillation baselines (Llama3-8B-GPTQ, Mistral-7B-GPTQ, Qwen-7B-Chat-Int4, MiniMA-3B) against FASTNav models that receive task-specific LoRA fine-tuning and teacher feedback. That conflates method with training data. A fair comparison would give the baselines the same fine-tuning or at least the same prompt-optimization budget.\n\nCredit where it is due: the human-in-the-loop dataset generation is well described, the ablation separating fine-tuning and iteration is the right kind of experiment, and the limitations section is honest about the static-map assumption and the need for task-specific adaptation. The novelty is incremental — combining known pieces — but that is fine for a systems paper.\n\nFor whom: robotics practitioners who want to deploy language-guided navigation on edge devices will find this useful if the validity threats are addressed. It deserves a serious referee, but only a conditional accept. The authors should be required to (1) state explicitly that iteration is performed on a training split, or re-run with a held-out test set that the teacher never sees; (2) report error bars or confidence intervals; (3) fine-tune the compression baselines on the same data, or at least justify why off-the-shelf is the right comparison; and (4) release code, data, and hyperparameters. If the test-set leakage turns out to be real, the paper reduces to a fine-tuning recipe with modest incremental value. If it is not, the numbers are interesting and the edge-deployment angle is worth publishing.","headline":"Plausible recipe for edge LLM navigation, but the paper must prove its iteration loop isn't tuning prompts on the test set before the headline numbers mean anything.","tokens_in":11203,"tokens_out":2200,"would_cite":false,"duration_ms":26175,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FASTNav claims that fine-tuned sub-3B language models, boosted by teacher-student iteration, reach near-GPT-4 accuracy on multi-point robot navigation while staying lightweight enough to run on edge devices.","keywords":["small language models","robot navigation","LoRA fine-tuning","teacher-student iteration","edge deployment","multi-point navigation","model compression","JSON output format"],"falsifier":"Shift 20% of landmark coordinates in the hospital simulation between training and testing, and measure the success rate; if the model relied on memorized static positions rather than reasoning from the map, success should collapse. Alternatively, issue a command whose target is described by a visual attribute absent from the text map (e.g., 'go to the red chair') and check whether the model can produce a coordinate at all.","tokens_in":10192,"feed_emoji":"🤖","tokens_out":4898,"duration_ms":44722,"temperature":0.7,"pith_summary":"The paper claims that small language models (SLMs) with under three billion parameters can be boosted through LoRA fine-tuning plus a teacher-student iteration loop to perform multi-point robot navigation almost as accurately as much larger models such as GPT-4, while remaining fast enough to run locally on an edge computer. The motivation is that relying on cloud API calls for robot navigation raises latency, privacy, and network-availability problems, so a locally deployable model that still solves complex navigation instructions would be practically valuable. The paper reports that in a hospital simulation, a 3B model with FASTNav reached a 70% success rate, outperforming quantized 7-8B models, and a 0.5B model reached 63.33%, with inference times around 3 seconds. It also demonstrates the same trend on a real wheel-footed robot in two indoor environments.","feed_headline":"Small language models nearly match GPT-4 at robot wayfinding","feed_subtitle":"FASTNav fine-tunes sub-3B models on edge devices, reaching 70% success in simulations with 2.87s inference.","key_machinery":"The machinery is a three-module pipeline: (1) LoRA fine-tuning of a sub-3B model on a human-in-the-loop generated dataset, with outputs constrained to JSON containing an explanation and an ordered list of x–y goal coordinates; (2) a teacher-student iteration in which a large model (GPT-4) acts as prompt engineer and feedback receiver, generating prompts for the student and feeding the student's mistakes back into the next round of prompts; and (3) a Navigation2 controller that executes the coordinate list. The load-bearing identity is the mapping f(W, M) = {p1, ..., pn} from a natural-language command and a static text map to an ordered waypoint sequence.","core_discovery":"The central discovery is that a small language model fine-tuned on a domain-specific dataset of navigation tasks and then iteratively guided by a larger teacher model can close most of the performance gap with the teacher on the specific task of translating natural-language instructions into ordered coordinates. In the paper's experiments, fine-tuning constrains the output into a JSON format and encodes map knowledge, raising accuracy sharply, while the teacher-student iteration adds a further 30–40% success-rate improvement by having the teacher rewrite prompts and provide feedback on previous failures. As a result, openllama-3B reaches 70% success and tau-0.5B reaches 63.33% in simulation, compared with 36.67% for Llama3-8B-GPTQ, and the small models keep memory usage at 2–8 GB and inference below three seconds.","pith_inferences":["The coordinate-list abstraction implicitly assumes that navigation goals are fully describable by static map coordinates; a testable extension would be a multimodal version that takes camera images as additional input, which the paper's limitations section names as future work.","The method's edge-deployment advantage applies to inference, not training, since both the human-in-the-loop dataset generation and the GPT-4 teacher require access to a strong external model during the build phase.","One could probe the transferability of FASTNav by training in the hospital simulation and fine-tuning in the lab hallway with only 100 tasks; if the performance gain persists, the method generalizes beyond a single memorized map, and if not, it is tightly coupled to the training environment."],"forward_implications":["If FASTNav is correct, sub-3B models deployed on edge devices can handle multi-point navigation tasks that currently require API calls to GPT-4-scale models, cutting both response time and privacy exposure.","The same fine-tuning-plus-iteration recipe may transfer to other structured output tasks where a small model must produce a constrained format (JSON, PDDL, STL) from natural language, making local deployment feasible beyond navigation.","Quantization and distillation baselines (GPTQ on Llama3-8B and Mistral-7B, distilled MiniMA-3B) are outperformed by a smaller model fine-tuned for the task, suggesting that task-specific adaptation matters more than raw parameter count or generic compression.","Because the teacher is only needed during training, the deployed system has no dependency on the cloud at inference time, so network failure does not disable navigation."],"supporting_citations":[{"why":"LoRA is the parameter-efficient fine-tuning method that keeps the small model lightweight while adapting it to the navigation task.","marker":"[4]"},{"why":"GPTQ is the quantization baseline that FASTNav is compared against in success rate and memory usage.","marker":"[18]"},{"why":"LLM+P is a PDDL-based planning baseline that slightly outperforms FASTNav with GPT-4 but fails completely when run with an unmodified openllama-3B.","marker":"[11]"},{"why":"AutoTAMP and LLM-As-Task Planner are STL- and subtask-based baselines that both fail with the raw small model, motivating the need for fine-tuning.","marker":"[13]"},{"why":"Navigation2 is the low-level controller that executes the ordered coordinate list produced by the language model.","marker":"[27]"},{"why":"The hospital world provides the simulation environment used to generate the approximately 1400-task training dataset and the test set.","marker":"[28]"},{"why":"Cited as evidence that the teacher-student paradigm improves language model training, grounding the iteration module's design.","marker":"[24]"},{"why":"Orca 2 is cited to support the claim that small language models can reach performance close to large models when properly fine-tuned.","marker":"[21]"}],"fun_headline_variants":["FASTNav boosts small AI models to 70% success at robot navigation","Edge SLMs with teacher tuning beat compressed large model in navigation","Sub-3B model outperforms 8B compressed version in robot wayfinding","Small language models get 30-40% accuracy boost from teacher feedback","FASTNav enables 2GB-memory AI to guide robots with high accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that multi-point navigation can be fully captured by outputting an ordered list of coordinates read from a static text map of landmarks, so if landmarks move, the map is incomplete, or the robot needs to see a target to identify it, the approach fails regardless of the language model's quality.","fun_headline_variants_meta":{"raw":{"variants":["FASTNav boosts small AI models to 70% success at robot navigation","Edge SLMs with teacher tuning beat compressed large model in navigation","Sub-3B model outperforms 8B compressed version in robot wayfinding","Small language models get 30-40% accuracy boost from teacher feedback","FASTNav enables 2GB-memory AI to guide robots with high accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000346,"raw_usage":{"total_tokens":1874,"prompt_tokens":902,"completion_tokens":972,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":873}},"tokens_in":518,"tokens_out":972,"duration_ms":9434,"temperature":1.0,"reasoning_tokens":873,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:38:35.833530+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Shift 20% of landmark coordinates in the hospital simulation between training and testing, and measure the success rate; if the model relied on memorized static positions rather than reasoning from the map, success should collapse. Alternatively, issue a command whose target is described by a visual attribute absent from the text map (e.g., 'go to the red chair') and check whether the model can produce a coordinate at all.","supporting_citations":[{"cited_title":"Autotamp: Autoregressive task and motion planning with llms as translators and checkers,","cited_arxiv_id":null,"evidence_quote":"AutoTAMP and LLM-As-Task Planner are STL- and subtask-based baselines that both fail with the raw small model, motivating the need for fine-tuning."},{"cited_title":"The marathon 2: A navigation system,","cited_arxiv_id":null,"evidence_quote":"Navigation2 is the low-level controller that executes the ordered coordinate list produced by the language model."},{"cited_title":"aws-robomaker-hospital-world,","cited_arxiv_id":null,"evidence_quote":"The hospital world provides the simulation environment used to generate the approximately 1400-task training dataset and the test set."}],"review_version":1}