Pith. sign in

REVIEW 5 major objections 6 minor 55 references

A Model-free Biomimetics Algorithm for Deterministic Partially Observable Markov Decision Process

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By tracking its own action vectors, a model-free agent recovers optimal policies in deterministic POMDPs without knowing the environment's transition or observation model.

desk verdict The paper's central claim fails because BIOMAP reconstructs states from cumulative action vectors, which only works when the environment is translation-embedded—an unstated and nontrivial restriction. read the letter →

arxiv 2412.14614 v1 pith:CDJS537S submitted 2024-12-19 eess.SY cs.SY

Yide Yu , Yue Liu , Xiaochen Yuan , Dennis Wong , Huijie Li , Yan Ma This is my paper · ORCID
classification eess.SYcs.SY MSC 90C4068Q4568T05
keywords DET-POMDPmodel-freePOMDPsolverbiomimeticsalgorithmpathintegrationCompactVectorGraphMarkovautomatonMDP-Graph-Automatoncliffwalking
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a class of partially observable decision problems—deterministic POMDPs with many-to-one state-observation mappings—can be solved without any model of the environment. The authors propose BIOMAP, an algorithm that tracks the cumulative vector of the agent's own actions, much as a desert ant performs path integration, and builds a graph whose vertices are these dead-reckoned positions. They claim this graph is isomorphic to the true state space, turning the partially observable problem into a fully observable MDP that can be solved with a shortest-path algorithm. Experiments on a masked Cliff Walking task show BIOMAP reaching the same optimal policy as model-based solvers while using no transition or observation model. The paper also quantifies the 'Cognitive Fog' that aliased observations create, measuring the resulting bias in Q-values.

What carries the argument

The Compact Vector Graph: a multigraph whose edges are action unit vectors and whose vertices are cumulative sums of those vectors, i.e., the agent's path-integrated position. Together with the Deterministic Finite Markov Automaton into which the graph is mapped, it lets the agent test whether the observed history is deterministic; if it is, the graph is taken as a full MDP. The Boundary Arbiter decides when repeated observations indicate a wall rather than a new hidden state, so the graph does not expand forever.

What would settle it

Build a deterministic POMDP with four states arranged in a cycle where two adjacent states emit the same observation and actions are 'go clockwise' / 'go counterclockwise'; BIOMAP's action vectors will assign the same coordinate to states reached by different histories, so the Compact Vector Graph will not be isomorphic to the true state space, and the returned policy will be suboptimal. If BIOMAP still solves it optimally, the hidden uniqueness condition is not necessary.

Watch

Extended reading notes

Core claim

The central claim is that in a deterministic POMDP, an agent that remembers only its own action history—encoded as a cumulative action vector—can reconstruct the hidden state space and recover the optimal policy. BIOMAP implements this by abstracting actions into unit vectors, recording each visited cumulative vector as a vertex of a 'Compact Vector Graph', and using a Markov automaton to check whether the resulting graph has deterministic transitions. If the check passes, the graph is declared a fully observable MDP and the optimal path is found by a shortest-path algorithm. On the Mask Cliff Walking benchmark, BIOMAP attains the same maximum reward as the best model-based solvers, with zero variance across masking configurations, while the model-free agent never sees the transition or observation function.

Load-bearing premise

Each hidden state is uniquely identified by the cumulative sum of the agent's action vectors, so no two different states can share both an observation and an action history, which is only true when states behave like positions in a vector space.

Editorial extensions

If this is right

  • If BIOMAP's reconstruction is correct, any deterministic POMDP whose state space is a translation-invariant grid can be solved optimally without a model, matching model-based solvers.
  • The algorithm supplies a practical test for 'environmental deceptiveness': if the graph fails the determinism check, the environment is not reducible to a fully observable MDP by this method.
  • BIOMAP's zero-variance performance across masking settings in the experiments suggests the recovered policy is independent of which observations are aliased, as long as the dead-reckoned positions stay distinct.
  • The framework offers a finite-state representation of DET-POMDP trajectories, so planning reduces to a shortest-path problem on a graph of size at most the number of distinct cumulative action vectors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The hidden-state uniqueness condition is not stated explicitly by the paper: BIOMAP works when distinct states have distinct action-vector coordinates AND same observation only for states that are genuinely equivalent in transitions and rewards. A counterexample is a deterministic POMDP with a cycle in state space, where two different states share the same observation and the same incoming action
  • Testable extension: run BIOMAP on a non-grid DET-POMDP where actions do not act as translations (e.g., a graph with a branching structure where two paths arrive at the same observation from different directions); the graph would collapse states and the policy would degrade.
  • The Q-value variance theorem quantifies Cognitive Fog, but the paper does not use it to predict when BIOMAP will fail; that threshold could be derived from the variance formula.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper defines DET-POMDP, a deterministic subclass of POMDPs with many-to-one observation mappings, and proposes a model-free algorithm called BIOMAP. The algorithm represents the agent's action history as a cumulative vector, builds a 'Compact Vector Graph,' checks the graph with a finite automaton, and then solves a shortest-path problem to obtain a policy. The authors claim this converts a general DET-POMDP into a fully observable MDP. The experimental section introduces a Mask Cliff Walking simulator with several masking settings and reports that BIOMAP matches the best model-based solvers while outperforming some online POMDP solvers.

Significance. If the conversion claim were correct, the paper would make a useful contribution to model-free planning in deterministic POMDPs. The path-integration analogy from desert ants is appealing, and the Mask Cliff Walking environment is a reasonable testbed for studying observation aliasing. However, the central theoretical claim is not established: the algorithm's state encoding requires a geometric translation structure that Definition 1 does not provide, the automaton-based determinism test is logically vacuous in light of NDFA/DFA equivalence, and the proofs of Lemmas 2 and 3 are not valid as written. The experimental success is essentially forced by the grid-world geometry. The paper does not provide machine-checked proofs, formal verification, or public code, and the central derivation depends on unstated assumptions.

major comments (5)
  1. [Sec. 3.3.2, Defs. 3-4; Alg. 2 line 15] The core state-encoding step o' = o + a (Alg. 2 line 15) requires the state space to be embedded in a vector space and each action to act as a fixed translation. Definition 1 imposes no such structure: S, T, and Omega are arbitrary finite sets and functions. Consider S={A,B}, O={o}, T(A,a)=B, T(B,a)=A, R(A,a)=+1, and R(B,a)=-1. From initial state A, after one action the cumulative action vector is the same for both states, the observation is o in both cases, and the Compact Vector Graph has one vertex for two states with different rewards. The bijection V <-> S required by Definition 5 therefore fails, and the graph is not an MDP representation of the DET-POMDP. The paper never states the translation-embedding assumption, although the grid-world experiments satisfy it.
  2. [Sec. 3.4.4 and Alg. 2 lines 23-26] The determinism test is not a valid test of full observability. Theorem 4 correctly states the standard equivalence of NDFA and DFA, so every language generated by the NDFMA has a DFMA recognizing it. The subsequent claim that 'if a language is not recognizable by DFMA, then the corresponding MDP for that language is partially observable' is inconsistent with Theorem 4. Consequently, the check 'if M_Markov is not deterministic then Break' (Alg. 2 lines 24-25) cannot detect whether the graph corresponds to a fully observable MDP. Rejecting the graph after construction does not provide a conversion procedure for the general DET-POMDP.
  3. [Appendices E and F (Lemmas 2 and 3)] The proofs of Lemmas 2 and 3 are not valid as written. In Appendix E, the proof asserts that the action history trajectory set is a subset of the alphabet Sigma, but the argument is circular because the claimed equality H = L_Markov is used in the definition of L_Markov, the statement 'forall a in exists circle ha' is incomplete, and the transition step 'w1 = a0w0' has the word concatenation backwards. In Appendix F, Lemma 3 is not proved: setting A=A' and S=S' and observing P(s|o) <= 1 only shows the transition relation is not a function; it does not construct a nondeterministic automaton whose accepted language is H', nor does it connect the automaton's acceptance to the POMDP trajectory semantics. Since Lemmas 2 and 3 are the stated basis for the MDP-Graph-Automaton conversion, this is a load-bearing gap.
  4. [Alg. 2, lines 11-14] Line 11 tests 'o = o_vector', comparing the environment observation o with the action-vector label o_vector; these live in different spaces and the condition is not meaningful as written. If the condition is false, b is never assigned before being used in line 14; if the condition is true, Algorithm 1 may have inserted a self-loop while Algorithm 2 does not update the observation on the boundary branch. The boundary-detection mechanism therefore needs to be specified precisely; as written, the pseudocode does not define a valid algorithm.
  5. [Sec. 4, Tables 1-2 and Fig. 5] The experiment only tests the translation-embedding case. In Mask Cliff Walking, grid states are identified by coordinates, and the cumulative sum of the four unit action vectors from the start recovers the grid cell exactly; masking changes only the observation labels and does not disturb this dead-reckoned encoding. Thus the observed success is a direct consequence of the geometric assumption identified in the first major comment, not evidence for the general DET-POMDP conversion claimed in the abstract. In addition, Table 1 shows BIOMAP tied with QMDP, SARSOP, and POMDPSolver at -2; the claim of 'superior capabilities' over existing POMDP solvers is not supported by the tabulated comparison.
minor comments (6)
  1. [Abstract and Sec. 3.1.1] The phrase 'states and observations are many-to-one relationships' is imprecise; the paper means that the observation function is many-to-one, not that the state-observation relation is itself many-to-one in both directions.
  2. [Def. 3] The expression 'not exists a_vector_prime, 0 < a_vector_prime < a_vector' uses a partial order on vectors that is never defined; the definition of an action unit vector should be stated in terms of an explicit embedding of actions into a normed vector space.
  3. [Fig. 5] The labels '[5,2]=28' and '[5,3]=16' in Fig. 5(b) do not match any obvious coordinate-to-index convention (e.g., row-major indexing would give different values); the coordinate convention should be stated clearly.
  4. [Sec. 2] The term 'value episode' is used repeatedly where the standard term is 'value iteration' or 'value-function update'; this should be corrected throughout.
  5. [Sec. 3.2, Theorem 1] The statement that in an 'extreme situation' the variance 'tends towards infinity' is not implied by the displayed finite formula for the variance of the Q-value error; this claim should either be removed or proved.
  6. [Related work] The most relevant baseline for deterministic POMDPs, namely Bonet's deterministic POMDP algorithms cited as reference [1], is not compared or discussed in the related-work section; some discussion of why these algorithms are not benchmarks would strengthen the paper.

Circularity Check

2 steps flagged · score 8.0 of 10

BIOMAP's state space is defined, not derived: the Compact Vector Graph's vertices are cumulative action vectors, so for the grid-world experiments the claimed conversion to a fully observable MDP is built into the representation.

  1. self definitional [Sec. 3.3.2, Defs. 3–4; Sec. 3.6, Alg. 2 line 15]
    "In this graph, each vertex represents a cumulative action vector. ... Let a multigraph ⃗G be a compact Vector Graph if ∀e ∈ E(⃗G), |W(e)| = ⃗a, where ⃗a is an action unit vector. ... Calculate the next action vector ⃗o′ = ⃗o + ⃗a."

    The graph's state label is the cumulative action vector by definition, and the update rule makes the 'state' equal to the action history. The paper's claim that this graph converts DET-POMDP into a fully observable MDP (Sec. 3.4) and Def. 5's bijection V↔S are therefore not derived from the DET-POMDP tuple (S,T,O,R) of Def. 1; they hold only when the hidden state is already a vector position reached by translation actions, as in the grid-world experiments. Masking then changes nothing: the agent's dead-reckoned coordinate is the hidden grid cell, so the optimal policy is read off the coordinate graph. This is representation-level circularity: the converted MDP is the input encoding, not a general derived result.

  2. other [Sec. 3.5 (after Alg. 1)]
    "It is important to note that an observation has two distinct identities. One identity is obtained from the environment and is represented by a numeric value o. The other identity is derived from the action vector and is denoted by ⃗o."

    This assigns each observation a second, action-derived identity. Since Alg. 2 updates it as ⃗o′ = ⃗o + ⃗a, the action-derived identity is exactly the agent's dead-reckoned position. The later determinism check (Alg. 2 lines 23–26) verifies only that this action-vector graph has a deterministic transition relation; it does not verify that the action-derived identity is in bijection with the hidden states of a general DET-POMDP. For the grid-world experiments that bijection is supplied by the environment geometry, so the determinism check is checking the representation against itself rather than converting the POMDP.

full rationale

The central circularity is localized to the state-identification mechanism. Defs. 3–4 and Alg. 2 line 15 define graph vertices as cumulative sums of action unit vectors; this is a modeling choice, and it is exactly what makes Mask Cliff Walking solvable, because the hidden grid cell equals the accumulated move vector. The paper's general abstract claim that DET-POMDP is converted to a fully observable MDP is therefore true only for environments equipped with a translation-vector embedding, a condition never stated in Def. 1. The automaton lemmas and determinism check do not repair this: they test the action-vector graph's internal determinism, not its isomorphism to the original state space. Self-citation [27] is present but not load-bearing, and the experimental comparisons against POMDPs.jl solvers are external, so no separate circularity arises there. Because the main conversion claim reduces, for the tested environments, to the definition of the representation, the score is 8 rather than lower.

Assumptions & free parameters 3 free parameters · 6 assumptions · 3 invented entities

The central claim rests on two strong inputs not paid for by the paper: the geometric embedding of states in a vector space (action unit vectors) and the finiteness and detectability of boundaries. The free parameters delta, N, M are tuning knobs, and the action unit vector set encodes domain geometry that a truly model-free agent should not have. The automata results are standard mathematics; the invented terms are repackagings of known objects rather than new entities.

free parameters (3)
  • boundary tolerance delta = not reported
    User-set parameter in Alg. 1 controlling how many repeated transitions confirm a boundary; affects whether a self-loop is added and hence the shape of the graph.
  • maximum episodes N and maximum steps M = not reported
    Stopping criteria for the exploration loops in Alg. 2; the algorithm assumes exploration completes, so these must be large enough to cover the environment.
  • action unit vector set A~ = e.g., up=[0,1], right=[1,0] in experiments
    Chosen by human expertise (Sec. 3.3.2); this encodes the effect of actions on the hidden state and is the main source of model knowledge in a supposedly model-free algorithm.
assumptions (6)
  • domain assumption State transitions and observations are deterministic (Def. 1)
    Defines the problem class. Stated in Sec. 3.1.1.
  • ad hoc to paper The hidden state is uniquely determined by the cumulative action vector
    Introduced in Def. 3-4 and used in Alg. 2 line 15. Not true for general DET-POMDPs and never stated as a limitation.
  • domain assumption The environment is finite and has detectable boundaries
    The Boundary Arbiter (Alg. 1) assumes that repeated identical observations indicate a boundary and that the graph becomes finite; the paper notes the unbounded problem in Sec. 3.5 but does not prove the arbiter detects all boundaries.
  • standard math Every finite set of finite action sequences is a regular language
    Used implicitly in Lem. 2 and 3 to say trajectory sets are recognizable by finite automata.
  • standard math NDFAs and DFAs recognize the same class of languages
    Used in Thm. 4. This is textbook; however the paper draws an invalid conclusion that a language not recognizable by a DFMA indicates partial observability.
  • domain assumption The Compact Vector Graph is isomorphic to the true MDP state graph
    Required for the Dijkstra shortest path in phase 4 to produce the optimal policy. Only holds under the geometric embedding assumption.
invented entities (3)
  • Cognitive Fog
    purpose: Metaphor for the decision bias caused by aliased Q-values in DET-POMDPs.
    Introduced in Sec. 3.2; it is a descriptive label for the variance identity in Thm. 1, not an independently testable quantity.
  • Compact Vector Graph
    purpose: Graph whose vertices are cumulative action vectors; used as the algorithm's internal state representation.
    Defined in Def. 4; a new name for an action-history graph, not a new physical or mathematical entity.
  • Markov Automaton (DFMA/NDFMA)
    purpose: Automata that recognize the trajectory languages, used to decide determinism of the graph.
    Defined in Sec. 3.4; a relabeling of standard DFA/NFA with MDP-style transitions, contributing no new capability beyond standard automata.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Model-free Biomimetics Algorithm for Deterministic Partially Observable Markov Decision Process." pith.science (2026). https://pith.science/paper/CDJS537S

@misc{pith2026241214614,
  author       = {Pith},
  title        = {Pith review of: A Model-free Biomimetics Algorithm for Deterministic Partially Observable Markov Decision Process},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CDJS537S}},
  note         = {Machine review of arXiv:2412.14614}
}
read the original abstract

Partially Observable Markov Decision Process (POMDP) is a mathematical framework for modeling decision-making under uncertainty, where the agent's observations are incomplete and the underlying system dynamics are probabilistic. Solving the POMDP problem within the model-free paradigm is challenging for agents due to the inherent difficulty in accurately identifying and distinguishing between states and observations. We define such a difficult problem as a DETerministic Partially Observable Markov Decision Process (DET-POMDP) problem, which is a specific setting of POMDP. In this problem, states and observations are in a many-to-one relationship. The state is obscured, and its relationship is less apparent to the agent. This creates obstacles for the agent to infer the state through observations. To effectively address this problem, we convert DET-POMDP into a fully observable MDP using a model-free biomimetics algorithm called BIOMAP. BIOMAP is based on the MDP Graph Automaton framework to distinguish authentic environmental information from fraudulent data. Thus, it enhances the agent's ability to develop stable policies against DET-POMDP. The experimental results highlight the superior capabilities of BIOMAP in maintaining operational effectiveness and environmental reparability in the presence of environmental deceptions when compared with existing POMDP solvers. This research opens up new avenues for the deployment of reliable POMDP-based systems in fields that are particularly susceptible to DET-POMDP problems.

Figures

Figures reproduced from arXiv: 2412.14614 by the authors.

Figure 1
Figure 1. General POMDP vs. DET-POMDP example. In DET-POMDP with environment-free [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Schematic Diagrams and Flowchart of the Desert Ant’s Navigational Biometrics and [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Structure of MDP-Graph-Automaton. 3.4.1 MDP-Graph Relation Def. 5 conveys the relation between graph and MDP. Definition 5 (MDP with graph representation). A Markov Decision Process with graph representation MG is a multigraph. It is formalized by the tuple ⟨V, E⟩. MG is constructed through deterministic Markov Decision Process M = ⟨S, A, T , R, γ⟩. Corresponding to M, MG satisfied the bijective map￾ping relations a… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: This figure illustrates the design of the Masking Cliff Walking experiment, showcasing [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Visualization of results. In (a), ○1 is the process of experimental settings, ○2 − ○6 are BIOMAP’s working process on the masking cliff walking; (b) is a recovered Action Vector graph for Cliff Walking with masking (direction: row, number = 12, continuity = True, layer…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 43 canonical work pages

  1. [1]

    Deterministic pomdps revisited

    Blai Bonet. Deterministic pomdps revisited. arXiv preprint arXiv:1205.2659 , 2012

    arXiv 2012

  2. [2]

    Planning and control in stochastic domains with imperfect information

    Milos Hauskrecht. Planning and control in stochastic domains with imperfect information . PhD thesis, Massachusetts Institute of Technology, 1997

    work page 1997

  3. [3]

    Planning and acting in partially observable stochastic domains

    Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99–134, 1998

    1998

  4. [4]

    Belief space planning assuming maximum likelihood observations

    Robert Platt Jr, Russ Tedrake, Leslie Pack Kaelbling, and Tomas Lozano-Perez. Belief space planning assuming maximum likelihood observations. In Robotics: Science and Systems, volume 2, 2010

    work page 2010

  5. [5]

    A survey of pomdp applications

    Anthony R Cassandra. A survey of pomdp applications. In Working notes of AAAI 1998 fall symposium on planning with partially observable Markov decision processes , volume 1724, 1998

    work page 1998

  6. [6]

    Monte-carlo planning in large pomdps

    David Silver and Joel Veness. Monte-carlo planning in large pomdps. Advances in neural infor- mation processing systems, 23, 2010

    work page 2010

  7. [7]

    Human-level control through deep reinforcement learning

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015. 18

    work page 2015

  8. [8]

    Playing atari with deep reinforcement learning

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013

    arXiv 2013

Show all 55 references
  1. [9]

    Maxim Egorov, Zachary N Sunberg, Edward Balaban, Tim A Wheeler, Jayesh K Gupta, and Mykel J Kochenderfer. Pomdps. jl: A framework for sequential decision making under uncertainty. Journal of Machine Learning Research , 18(26):1–5, 2017

    2017

  2. [10]

    Wainwright, and Michael I

    Eric Xia, Koulik Khamaru, Martin J. Wainwright, and Michael I. Jordan. Instance-optimality in optimal value estimation: Adaptivity via variance-reduced q-learning. IEEE Transactions on Information Theory, pages 1–1, 2024

    2024

  3. [11]

    Learning policies for partially observable environments: Scaling up

    Michael L Littman, Anthony R Cassandra, and Leslie Pack Kaelbling. Learning policies for partially observable environments: Scaling up. In Machine Learning Proceedings 1995 , pages 362–370. Elsevier, 1995

    1995

  4. [12]

    Computationally feasible bounds for partially observed markov decision pro- cesses

    William S Lovejoy. Computationally feasible bounds for partially observed markov decision pro- cesses. Operations research, 39(1):162–175, 1991

    1991

  5. [13]

    Point-based value iteration: An anytime algorithm for pomdps

    Joelle Pineau, Geoff Gordon, Sebastian Thrun, et al. Point-based value iteration: An anytime algorithm for pomdps. In Ijcai, volume 3, pages 1025–1032, 2003

    2003

  6. [14]

    A survey of point-based pomdp solvers

    Guy Shani, Joelle Pineau, and Robert Kaplow. A survey of point-based pomdp solvers. Au- tonomous Agents and Multi-Agent Systems , 27:1–51, 2013

    2013

  7. [15]

    Incremental pruning: A simple, fast, exact method for partially observable markov decision processes

    Anthony R Cassandra, Michael L Littman, and Nevin Lianwen Zhang. Incremental pruning: A simple, fast, exact method for partially observable markov decision processes. arXiv preprint arXiv:1302.1525, 2013

    2013 arXiv

  8. [16]

    Bandit based monte-carlo planning

    Levente Kocsis and Csaba Szepesv´ ari. Bandit based monte-carlo planning. InEuropean conference on machine learning , pages 282–293. Springer, 2006

    2006

  9. [17]

    Monte-carlo tree search: A new framework for game ai

    Guillaume Chaslot, Sander Bakkes, Istvan Szita, and Pieter Spronck. Monte-carlo tree search: A new framework for game ai. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment , volume 4, pages 216–217, 2008

    2008

  10. [18]

    Online algorithms for pomdps with continuous state, action, and observation spaces

    Zachary Sunberg and Mykel Kochenderfer. Online algorithms for pomdps with continuous state, action, and observation spaces. In Proceedings of the International Conference on Automated Planning and Scheduling , volume 28, pages 259–263, 2018

    2018

  11. [19]

    Integrated perception and planning in the continuous space: A pomdp approach

    Haoyu Bai, David Hsu, and Wee Sun Lee. Integrated perception and planning in the continuous space: A pomdp approach. The International Journal of Robotics Research , 33(9):1288–1302, 2014

    2014

  12. [20]

    Shared information for a markov chain on a tree

    Sagnik Bhattacharya and Prakash Narayan. Shared information for a markov chain on a tree. IEEE Transactions on Information Theory , pages 1–1, 2024

    2024

  13. [21]

    Algorithms for decision making

    Mykel J Kochenderfer, Tim A Wheeler, and Kyle H Wray. Algorithms for decision making . MIT press, 2022

    2022

  14. [22]

    Sarsop: Efficient point-based pomdp planning by approximating optimally reachable belief spaces

    Hanna Kurniawati, David Hsu, and Wee Sun Lee. Sarsop: Efficient point-based pomdp planning by approximating optimally reachable belief spaces. 2009

    2009

  15. [23]

    Despot: Online pomdp planning with regularization

    Adhiraj Somani, Nan Ye, David Hsu, and Wee Sun Lee. Despot: Online pomdp planning with regularization. Advances in neural information processing systems , 26, 2013

    2013

  16. [24]

    Adaptive online packing-guided search for pomdps

    Chenyang Wu, Guoyu Yang, Zongzhang Zhang, Yang Yu, Dong Li, Wulong Liu, and Jianye Hao. Adaptive online packing-guided search for pomdps. Advances in Neural Information Processing Systems, 34:28419–28430, 2021

    2021

  17. [25]

    Aems: An anytime online search algorithm for approx- imate policy refinement in large pomdps

    St´ ephane Ross, Brahim Chaib-Draa, et al. Aems: An anytime online search algorithm for approx- imate policy refinement in large pomdps. In IJCAI, pages 2592–2598, 2007. 19

    2007

  18. [26]

    Markov decision processes: discrete stochastic dynamic programming

    Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons, 2014

    2014

  19. [27]

    Measuring the state-observation-gap in pomdps: An exploration of observation confidence and weighting algo- rithms

    Yide Yu, Yan Ma, Yue Liu, Dennis Wong, Kin Lei, and Jos´ e Vicente Egas-L´ opez. Measuring the state-observation-gap in pomdps: An exploration of observation confidence and weighting algo- rithms. In IFIP International Conference on Artificial Intelligence Applications and Inn...

    2023

  20. [28]

    Individually different foraging methods in the desert ant cataglyphis bicolor (hymenoptera, formicidae)

    Paul Schmid-Hempel. Individually different foraging methods in the desert ant cataglyphis bicolor (hymenoptera, formicidae). Behavioral Ecology and Sociobiology, 14:263–271, 1984

    1984

  21. [29]

    Path integration in desert ants, cataglyphis fortis.Proceedings of the National Academy of Sciences , 85(14):5287–5290, 1988

    Martin M¨ uller and R¨ udiger Wehner. Path integration in desert ants, cataglyphis fortis.Proceedings of the National Academy of Sciences , 85(14):5287–5290, 1988

    1988

  22. [30]

    The desert ant odometer: a stride integrator that accounts for stride length and walking speed

    Matthias Wittlinger, Ru¨ udiger Wehner, and Harald Wolf. The desert ant odometer: a stride integrator that accounts for stride length and walking speed. Journal of experimental Biology , 210(2):198–207, 2007

    2007

  23. [31]

    The desert ant’s navigational toolkit: procedural rather than positional knowledge

    R ¨UDIGER WEHNER. The desert ant’s navigational toolkit: procedural rather than positional knowledge. Navigation, 55(2):101–114, 2008

    2008

  24. [32]

    Desert ant navigation: how miniature brains solve complex tasks

    Rudiger Wehner. Desert ant navigation: how miniature brains solve complex tasks. Journal of Comparative Physiology A, 189:579–588, 2003

    2003

  25. [33]

    Animal cognition in nature: the convergence of psychology and biology in laboratory and field

    Russell P Balda, Irene M Pepperberg, and Alan C Kamil. Animal cognition in nature: the convergence of psychology and biology in laboratory and field . Academic Press, 1998

    1998

  26. [34]

    Loft features reveal the functioning of the young pigeon’s navigational system

    Paulo Jorge, Inˆ es Silva, and Luis Vicente. Loft features reveal the functioning of the young pigeon’s navigational system. Naturwissenschaften, 95(3):223–231, 2008

    2008

  27. [35]

    What strategies do homing pigeons use during ontogeny? Behaviour, 143(1):105–122, 2006

    P Jorge, L Vicente, and W Wiltschko. What strategies do homing pigeons use during ontogeny? Behaviour, 143(1):105–122, 2006

    2006

  28. [36]

    An anatomically constrained model for path integration in the bee brain

    Thomas Stone, Barbara Webb, Andrea Adden, Nicolai Ben Weddig, Anna Honkanen, Rachel Templin, William Wcislo, Luca Scimeca, Eric Warrant, and Stanley Heinze. An anatomically constrained model for path integration in the bee brain. Current Biology, 27(20):3069–3085, 2017

    2017

  29. [37]

    Path integration by swimming rats

    Simon Benhamou. Path integration by swimming rats. Animal Behaviour, 54(2):321–327, 1997

    1997

  30. [38]

    Reducibility among combinatorial problems

    Richard M Karp. Reducibility among combinatorial problems . Springer, 2010

    2010

  31. [39]

    Introduction to algorithms

    Thomas H Cormen, Charles E Leiserson, Ronald L Rivest, and Clifford Stein. Introduction to algorithms. MIT press, 2022

    2022

  32. [40]

    A note on two problems in connexion with graphs

    Edsger W Dijkstra. A note on two problems in connexion with graphs. In Edsger Wybe Dijkstra: His Life, Work, and Legacy , pages 287–290. 2022

    2022

  33. [41]

    Reinforcement learning: An introduction

    Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018

    2018

  34. [42]

    Learning to play the game of chess

    Sebastian Thrun. Learning to play the game of chess. Advances in neural information processing systems, 7, 1994

    1994

  35. [43]

    Multi-agent reinforcement learning for traffic light control

    Marco A Wiering et al. Multi-agent reinforcement learning for traffic light control. In Machine Learning: Proceedings of the Seventeenth International Conference (ICML’2000) , pages 1151– 1158, 2000

    2000

  36. [44]

    Agricultural robots for field operations: Concepts and components

    Avital Bechar and Cl´ ement Vigneault. Agricultural robots for field operations: Concepts and components. Biosystems Engineering, 149:94–111, 2016

    2016

  37. [45]

    Julia: A fresh approach to numerical computing

    Jeff Bezanson, Alan Edelman, Stefan Karpinski, and Viral B Shah. Julia: A fresh approach to numerical computing. SIAM review, 59(1):65–98, 2017. 20

    2017

  38. [46]

    R: A language and environment for statistical computing

    R Core Team. R: A language and environment for statistical computing. r foundation for statistical computing. (No Title) , 2013

    2013

  39. [47]

    The optimal control of partially observable Markov processes

    Edward Jay Sondik. The optimal control of partially observable Markov processes . Stanford University, 1971

    1971

  40. [48]

    Scaling theory of fragmentation

    Z Cheng and S Redner. Scaling theory of fragmentation. Physical review letters , 60(24):2450, 1988

    1988

  41. [49]

    The witness algorithm: Solving partially observable markov decision processes

    Michael L Littman. The witness algorithm: Solving partially observable markov decision processes. Brown University, Providence, RI , 1994

    1994

  42. [50]

    Littman, and Nevin L

    Anthony Cassandra, Michael L. Littman, and Nevin L. Zhang. Incremental pruning: a simple, fast, exact method for partially observable markov decision processes. In Proceedings of the Thirteenth Conference on Uncertainty in Artificial Intelligence , UAI’97, page 54–61, San Fran...

    1997

  43. [51]

    Cassandra

    Anthony R. Cassandra. Tony’s pomdp file repository page, 2004

    2004

  44. [52]

    Analysis of variance (anova)

    Lars St, Svante Wold, et al. Analysis of variance (anova). Chemometrics and intelligent laboratory systems, 6(4):259–272, 1989

    1989

  45. [53]

    Oliver and Boyd Edinburgh, 1966

    Ronald Aylmer Fisher, Ronald Aylmer Fisher, Statistiker Genetiker, Ronald Aylmer Fisher, Statistician Genetician, Great Britain, Ronald Aylmer Fisher, and Statisticien G´ en´ eticien.The design of experiments , volume 21. Oliver and Boyd Edinburgh, 1966

    1966

  46. [54]

    On the state minimization of nondeterministic finite au- tomata

    Tsunehiko Kameda and Peter Weiner. On the state minimization of nondeterministic finite au- tomata. IEEE Transactions on Computers , 100(7):617–627, 1970

    1970

  47. [55]

    Introduction to automata theory, languages, and computation

    John E Hopcroft, Rajeev Motwani, and Jeffrey D Ullman. Introduction to automata theory, languages, and computation. Acm Sigact News , 32(1):60–65, 2001. 21 A Summary Table for Sec. 2 Table 3: POMDP Solver Classification. Algorithm Classification Advantage Disadvantage QMDP [11...

    1995

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.