{"id":"b0d79b6e-48dc-49a5-af46-57699f7cec2c","arxiv_id":"2505.02216","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LLM-guided probabilistic program induction can learn low-complexity POMDP models from ten demonstrations and outperform tabular learning, behavior cloning, and direct LLM planning in simulated and real robot domains.","lead":"This paper presents POMDP Coder, a method that uses a large language model to propose and refine short probabilistic programs representing a POMDP's transitions, observations, rewards, and initial state from a few demonstrations. It reports that this approach beats tabular model learning, behavior cloning, and direct LLM planning on classical, MiniGrid, and two real Spot robot search tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Coverage-based model selection does not establish that the learned POMDP model drives planning; Unlock initial-test coverage of 0.24 shows the model can be badly wrong while the method still succeeds, so the central attribution is unsupported.","rationale":"The central claim is causal: LLM-guided construction of a low-complexity POMDP model is what yields the reported advantage. The only model-quality evidence presented is the coverage metric, and that metric is too weak to carry the claim. Coverage is a support test, not a distributional test; an over-broad program that places positive mass everywhere can score perfectly while being useless for belief updating. The paper's own Section 7 admits this. The Unlock Initial Test coverage of 0.24 is the clearest concrete symptom: after offline learning, the selected initial-state model declares most held-out initial states impossible, yet the full system still performs well. That pattern points to the planner or the online repair loop compensating for model error, so the comparison against baselines does not isolate the contribution of LLM-guided model construction. The selection-on-test procedure in Algorithm 2 compounds the problem: using Dtest coverage to pick the final model means the reported test coverage numbers are optimistic and not a valid generalization measure. This does not make the experimental results fraudulent or even necessarily wrong; the method may genuinely work. But the evidence as presented does not establish the specific claim that the learned model is the effective component. A conditional verdict is appropriate: the paper should either release code, add an ablation that varies model fidelity while holding planner and repair fixed, or report held-out likelihoods and planning performance as a function of model accuracy. The reader's conditional verdict is therefore unchanged.","tokens_in":15026,"tokens_out":9006,"duration_ms":117269,"concrete_test":"Run the Unlock (and, if budget allows, Lava) domain with online repair disabled and compare two conditions using the same planner: (a) the learned initial-state program selected by coverage, and (b) the true initial-state distribution while keeping the learned transition, observation, and reward programs fixed. If task reward under (b) is substantially higher than under (a), the learned initial model is a bottleneck and the coverage metric is not a reliable proxy for planning quality; if the rewards are similar, the planner is robust to the model error and the method's success is not attributable to the learned model's fidelity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 2 selects the returned program by maximizing coverage computed on Dtrain and Dtest (Lines 5, 11, 15), so the 'test coverage' reported in Table 3 is not an independent generalization estimate. More fundamentally, coverage as defined in Section 4.1 only checks whether each empirical transition has nonzero probability under the model; it does not constrain the distribution's shape, and Section 7 concedes that the metric can reward overly broad programs. Table 3 shows Initial Test coverage of only 0.24 for Unlock, meaning the learned initial-state program assigns zero probability to about three quarters of test initial states, yet Figure 3 reports strong task performance. This decoupling undermines the paper's central claim that LLM-guided construction of the low-complexity POMDP model is what makes POMDP Coder effective: the successes could instead be carried by the determinized belief-space planner in Appendix C or by the online repair loop in Algorithm 1. To support the claim, the paper needs evidence that task reward is sensitive to the accuracy of the learned model, not just to its coverage.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces POMDP Coder, a method that learns low-complexity POMDP models from data by using an LLM to propose and iteratively repair short probabilistic programs for the initial-state, transition, observation, and reward components. Candidate programs are selected according to a coverage metric that measures the fraction of empirical transitions with nonzero probability under the model. The learned model is then used by a determinized belief-space planner with particle-filter belief updates. Experiments are reported on two classical POMDPs (Tiger, RockSample), five modified MiniGrid domains, and two real Spot robot search tasks, comparing against an oracle model, direct LLM planning, behavior cloning, tabular model learning, a random baseline, and offline/online ablations. The central claim is that LLM-guided construction of a low-complexity POMDP model can be more effective than tabular POMDP learning, behavior cloning, or direct LLM planning.","tokens_in":15253,"tokens_out":3337,"duration_ms":45576,"significance":"If the attribution holds, this is a valuable step toward data-efficient, interpretable POMDP model learning, combining the LLM's prior knowledge with an empirical coverage check to produce executable models. The paper's strengths include the use of multiple seeds and error bars in simulation, ablations that separate offline and online learning, real-robot evaluation, and a promised public code release. The interpretable probabilistic-program representation is a genuine advantage over tabular approaches. However, the current evidence does not fully establish that the learned model, rather than the planner or the online repair loop, is what drives the reported gains; the coverage-based selection metric is weak and its reported test values are not independent generalization estimates. The significance is therefore contingent on additional analysis demonstrating sensitivity of planning performance to learned-model quality.","major_comments":[{"comment":"The 'test' coverage reported in Table 3 is not an independent generalization estimate. Algorithm 2 evaluates coverage on both Dtrain and Dtest (Lines 5 and 11) and returns the model maximizing coverage over this combined evaluation (Line 15). Thus the 'Test' column in Table 3 measures the selection objective itself, not holdout generalization, and the paper's claim of avoiding overfitting through a train/test split is not supported by the reported numbers. I recommend using a three-way split (train/validation/test), selecting on train and validation, and reporting coverage only on the truly held-out test set, or otherwise clarifying that the reported test coverage is part of the selection criterion.","section":"§4.1 and Appendix B, Algorithm 2"},{"comment":"The paper's central attribution is undermined by the decoupling between coverage and task reward. Section 7 concedes that the coverage metric can reward overly broad distributions, and Table 3 shows Initial Test coverage of only 0.24 for Unlock, meaning the learned initial-state program assigns zero probability to roughly three-quarters of test initial states; yet Figure 3 reports strong task performance in Unlock. This indicates that planning success is not sensitive to the accuracy of the learned model in the way the central claim requires, and the gains could be carried by the determinized belief-space planner (Appendix C) or by the online repair loop of Algorithm 1. To support the claim that LLM-guided construction of the low-complexity POMDP model is what makes the method effective, the paper should provide evidence that task reward is sensitive to learned-model quality, for example by substituting oracle components for learned ones one at a time, or by showing that across candidate models, higher coverage is associated with higher planning reward.","section":"§7 and Table 3"},{"comment":"The real-robot results are based on only ten runs per condition with high variance and no significance tests. For example, in Large-Tables, the Direct LLM baseline has 2 successes out of 3 runs, and the standard deviations (e.g., 0.42 vs. 0.08 for Ours) are large relative to the means. The claim that POMDP Coder achieves 'more efficient and accurate exploration' is not statistically supported. I request confidence intervals or a significance test (e.g., a permutation or bootstrap test), or additional runs, and a discussion of the low effective sample size for baselines with fewer than ten runs.","section":"§5.4 and Table 1"},{"comment":"The planning hyperparameters (λ, α, H, and others) are tuned to work best with the oracle ground-truth models. Because the same planner is used with the learned and tabular models, this tuning may advantage methods whose learned models resemble the oracle; for instance, the information-gain weight λ was set to 0.1 for all domains without checking whether this choice favors certain model classes. A sensitivity analysis over λ and α, or at least an explicit argument that the comparison is fair across baselines, is needed to rule out planner-tuning bias as an explanation of the results.","section":"Appendix D"}],"minor_comments":[{"comment":"The text in Appendix B refers to 'Line 26' when returning the candidate with the highest coverage, but the pseudocode shown has only 15 lines; this cross-reference should be corrected.","section":"Appendix B, Algorithm 2"},{"comment":"The Initial Test coverage of 0.24 for Unlock is a striking number that deserves explicit discussion in the main text, not only in a table, since it bears directly on the paper's attribution claim.","section":"Table 3"},{"comment":"The direct LLM baseline prompt contains typos ('aciton', 'Explan') that should be fixed for reproducibility since the prompt is part of the baseline definition.","section":"Appendix E.4"},{"comment":"References [38] and [43] are the same paper (Smith and Simmons, 'Heuristic Search Value Iteration for POMDPs') and should be merged or renumbered.","section":"References"},{"comment":"Normalization of discounted rewards by the oracle's performance should be described more precisely, including how domains are handled if the oracle's expected reward is near zero.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"This is a promising paper with a novel combination of LLM-based program induction and POMDP planning, and the experimental scope is appropriate. The main barrier is the attribution problem: the coverage metric used for model selection is weak, the reported test coverage is not a held-out estimate, and the Unlock result shows a large gap between coverage and task performance. These issues are fixable with additional experiments or re-analysis, rather than being fundamental. I also note that the paper promises code release 'upon publication' but no code or supplementary material is currently available, so reproducibility could not be verified; the editor may want to request code as part of the revision. A minor concern is the potential circularity of the LLM prior on classic domains (Tiger, RockSample), though the modified MiniGrid environments partially mitigate this."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the POMDP Coder paper. My take: the method is a real contribution, but the paper's central attribution is not yet proven.\n\nWhat's new: they apply LLM-guided probabilistic program induction to the full POMDP model—transition, observation, reward, initial state—rather than just reward as in [37], and they show it works on classical, MiniGrid, and two real Spot robot search tasks. That's a useful step, and the sample efficiency story is credible: programs generalize where tabular models can't, and the baselines are reasonable. The authors are also unusually honest: Section 7 admits the coverage metric rewards overly broad distributions, the post-hoc observability assumption, and particle filter limits.\n\nThe main soft spot is exactly what the stress-test note says. Coverage only checks whether each empirical sample has nonzero probability under the model. It says nothing about distribution shape, and model selection in Algorithm 2 uses both train and test coverage, so the reported \"test\" numbers are not a clean generalization estimate. The Unlock domain is the smoking gun: initial test coverage 0.24 means three-quarters of test initial states are assigned zero probability, yet the method still solves the task well. That decoupling means the learned initial-state model is not what's driving performance in that domain—something else (the planner's robustness, the online repair loop, or the other model components) is compensating. The paper never tests whether task reward is sensitive to model accuracy; they only report coverage. So the claim that \"using an LLM to guide construction of the low-complexity POMDP model\" is what makes it work is unsupported. It could be the online repair, or the planner. They do show the full method beats offline-only and online-only ablations, which helps, but they don't isolate the contribution of the learned model.\n\nTwo smaller points: the planner hyperparameters (λ, α, etc.) are explicitly tuned to the oracle model, which gives their method an informational advantage over the baselines. And the real-robot results have no significance tests with n=10; the effect sizes are large, but they should report confidence intervals or at least per-run data.\n\nThat said, the paper is well-executed and honest. The coverage metric is crude but interpretable, and the ablation suite (offline/online, Table 2/3, Appendix prompts) is above the usual bar. This deserves refereeing. I'd want the authors to add a sensitivity analysis: degrade the learned model (e.g., use the 0.24 initial-state program) and show task performance drops, or demonstrate that the planner can't compensate. Without that, I'd treat the central claim as a hypothesis rather than a conclusion.\n\nFor the reading group: yes—good for discussing what it means to \"validate\" a learned world model.\n\nRecommendation: send to peer review with the expectation of heavy revision, mainly on the attribution question.","headline":"A genuinely useful method for learning POMDP models from few demos, but the coverage metric obscures how much the learned model deserves the credit.","tokens_in":15794,"tokens_out":4483,"would_cite":true,"duration_ms":59854,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a POMDP's transition, observation, reward, and initial-state functions can be learned as short probabilistic programs proposed and repaired by an LLM, and that the resulting model plans better than tabular POMDP…","keywords":["POMDP model learning","probabilistic program induction","LLM-guided program synthesis","coverage-based model selection","partially observable planning","mobile robot search","MiniGrid","sample-efficient learning"],"falsifier":"A decisive check is to re-run the Unlock and Lava domains while selecting candidate models by their planned expected return instead of by coverage; if a model with low initial-state test coverage still drives successful search, coverage is not the binding constraint and the reported gains should be attributed to the planner and online repair, whereas if planning collapses, the induction step is doing the work attributed to it.","tokens_in":14825,"feed_emoji":"🤖","tokens_out":6978,"duration_ms":84801,"temperature":0.7,"pith_summary":"This paper tries to establish that a Partially Observable Markov Decision Process (POMDP) model can be learned from just ten demonstrations when an LLM is used as a prior: the LLM proposes short probabilistic programs for the transition, observation, reward, and initial-state components, the programs are scored against observed data, and failures are fed back to the LLM for repair. A sympathetic reader would care because POMDPs are powerful but famously hard to specify by hand, so a data-driven way to write interpretable models could make partial observability practical in robotics and other sequential decision problems. The paper tests this on classical POMDP benchmarks, five simulated MiniGrid environments, and two real mobile-robot search tasks, reporting that the resulting models match or outperform tabular model learning, behavior cloning, and direct LLM action selection in every simulated domain, and outperform them on the real robot. If the claim is right, the bottleneck in applying POMDPs shifts from hand-engineering world models to writing a small code-level API plus collecting a handful of trajectories.","feed_headline":"LLM-written POMDP models beat tabular learning and direct LLM planning","feed_subtitle":"An LLM writes and repairs probabilistic programs that outperform tabular POMDP models and direct LLM planning.","key_machinery":"The central object is the short probabilistic program written in Pyro: executable Python code that samples stochastic structure, one program per POMDP component. The mechanism that carries the argument is the LLM proposal-and-repair loop paired with coverage-based selection. Coverage is the fraction of empirical transitions that receive nonzero probability under the candidate model, and it gives the LLM a concrete, interpretable training signal: a repair prompt lists conditions and outcomes that the current program cannot produce, alongside samples the program does produce. Thompson sampling over Beta distributions on each candidate's coverage decides which program to refine next, and the highest-coverage program is eventually returned. Because the program uses symbolic ranges and conditions rather than enumerating states, it can generalize to initial states and situations that never appeared in the demonstration data, which is precisely where tabular models and behavior cloning fail.","core_discovery":"The central discovery is that the components of a POMDP—the initial-state distribution, transition function, observation function, and reward function—can be treated as low-complexity probabilistic programs, and that an LLM-guided proposal-and-repair loop can find those programs from data. The method, POMDP Coder, gives the LLM a code template, a state/action/observation API, and a small set of demonstration transitions; the LLM writes an initial program for each component, and the program is kept only if it gives nonzero probability to the observed data under a coverage metric. Candidate programs form a tree, Thompson sampling decides which candidate to refine next, and the LLM repairs a program using the specific empirical outcomes it failed to cover. The final programs are coupled to a belief-space planner with particle-filter belief updates. Across Tiger, RockSample, five MiniGrid tasks, and two real Boston Dynamics Spot search scenarios, POMDP Coder matched or outperformed all baselines in the simulated domains and produced higher expected discounted reward and more successful episodes on the robot, with the learned initial-state distribution providing more efficient exploration than a uniform one.","pith_inferences":["The paper leaves implicit that the coverage metric actively biases the search toward broader distributions, so a promising cheap extension is to add a model-complexity or distribution-width penalty to the selection objective and test whether planning quality improves on the Lava, Rooms, and Unlock domains where initial-state coverage was lowest.","The success on the real robot suggests a testable generalization: if the learned model is replaced by an equally good hand-written one, planning performance should be nearly identical; comparing those two cases would isolate how much of the reported gain comes from the induction procedure itself versus from the belief-space planner and online repair.","A further consequence not explored in the paper is that the same LLM-guided induction could learn belief updates or abstraction layers, not just the four POMDP components, since the code-API interface seems to be the main bottleneck; richer scene graphs or learned predicates would likely extend the approach to more complex manipulation tasks.","The post-hoc full-observability assumption means the method is most directly applicable where episode states can be recovered after the fact, such as object-search tasks; extending the repair loop to actively ask for missing state information during execution is a natural next step."],"forward_implications":["Ten demonstrations can be enough to obtain a usable POMDP model in domains with structured state spaces, because probabilistic programs generalize far beyond the enumerated training examples.","Learning a model rather than a policy separates the task of understanding the world from the task of acting in it, so the same learned model can be replanned with any POMDP solver and does not inherit the quality limitations of the demonstrations.","The online repair loop lets the agent extend and correct its model from its own experience, fixing gaps that offline-only learning misses, such as the consequence of stepping into lava or the effect of using a key.","Because the learned model is code, it is inspectable and debuggable in a way that learned tables or neural policies are not, which matters when the model must be validated for safe robot deployment.","Direct LLM action selection is not enough in partially observable settings: it can loop or ignore obstacle constraints, whereas the LLM only needs to write local generative code within POMDP Coder, leaving global reasoning to the planner."],"supporting_citations":[{"why":"Supplies Pyro, the probabilistic programming language in which all POMDP components are written.","marker":"[12]"},{"why":"Establishes the pattern of using language models for statistical model discovery, which this paper adapts to POMDPs.","marker":"[17]"},{"why":"Shows that LLMs can build world models as code in fully observable settings, the approach here extended to partial observability.","marker":"[21]"},{"why":"Provides the post-hoc full-observability assumption that lets the method label demonstration states after episodes end.","marker":"[36]"},{"why":"Supplies the LLM code-proposal-and-repair loop and Thompson-sampling exploration strategy that the learning procedure is built on.","marker":"[37]"},{"why":"Defines the MiniGrid environments used to test how well the learned programs generalize beyond LLM-pretraining experience.","marker":"[39]"}],"fun_headline_variants":["LLM-created POMDP programs beat tabular learning and direct planning","POMDP Coder: LLM-guided programs outperform tabular and LLM planning","LLM writes and repairs POMDP models: superior to tabular learning and LLM planning","From LLM priors to POMDP solutions: better than behavior cloning and direct planning","LLM-coded POMDPs: empirical win over tabular models and direct LLM planning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the coverage metric—the fraction of recorded transitions the model can produce with nonzero probability—reliably tracks how well the model supports good planning; the paper itself concedes that coverage can reward overly broad distributions and reports initial-state test coverage as low as 0.24 in the Unlock domain.","fun_headline_variants_meta":{"raw":{"variants":["LLM-created POMDP programs beat tabular learning and direct planning","POMDP Coder: LLM-guided programs outperform tabular and LLM planning","LLM writes and repairs POMDP models: superior to tabular learning and LLM planning","From LLM priors to POMDP solutions: better than behavior cloning and direct planning","LLM-coded POMDPs: empirical win over tabular models and direct LLM planning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000841,"raw_usage":{"total_tokens":3669,"prompt_tokens":957,"completion_tokens":2712,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":2597}},"tokens_in":573,"tokens_out":2712,"duration_ms":21070,"temperature":1.0,"reasoning_tokens":2597,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:57:44.628051+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check is to re-run the Unlock and Lava domains while selecting candidate models by their planned expected return instead of by coverage; if a model with low initial-state test coverage still drives successful search, coverage is not the binding constraint and the reported gains should be attributed to the planner and online repair, whereas if planning collapses, the induction step is doing the work attributed to it.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LLM code-proposal-and-repair loop and Thompson-sampling exploration strategy that the learning procedure is built on."}],"review_version":1}