Pith. sign in

REVIEW 5 major objections 4 minor 1 cited by

Inverse Reinforcement Learning with Switching Rewards and History Dependency for Characterizing Animal Behaviors

T0 review · 5 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A switching, history-dependent inverse reinforcement learning model recovers hidden motivational modes—water seeking, home seeking, exploring—from long animal trajectories without knowing where the goals are.

desk verdict SWIRL is a solid, honestly-reported switching-IRL extension that deserves review, but the universal-improvement and 'first' claims need revision. read the letter →

arxiv 2501.12633 v3 pith:XLN36F6L submitted 2025-01-22 cs.LG cs.AI

Jingyang Ke , Feiyang Wu , Jiyi Wang , Jeffrey Markowitz , Anqi Wu This is my paper · ORCID
classification cs.LGcs.AI
keywords inversereinforcementlearninghidden-modeMarkovdecisionprocesshistory-dependentrewardsswitchinganimalbehaviormodelingexpectation-maximizationmaximumentropyRLnon-Markovian
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SWIRL is a framework for inverse reinforcement learning that treats a long recording of animal behavior not as one fixed reward-seeking process but as a sequence of hidden decision modes, each with its own reward, with the switch between modes depending on where the animal is and the reward inside a mode depending on recent locations as well as the current one. The paper argues that this two-level history dependence—at the level of decision switching and at the level of within-mode action choice—is what previous time-varying IRL models lack, and that adding it yields better held-out prediction and more interpretable recovered reward maps than the baselines. On simulated gridworlds the full model recovers the true switching rewards only when both dependencies are present, and on seven-hour mouse labyrinth recordings it segments behavior into water, home, and explore modes and finds that leaving the water port is valued more than staying, matching a 90-second water-delivery rule. If the method is right, researchers can infer changing internal motivations from raw, non-trial-based trajectories and can use the framework to test whether a behavior is better described as Markovian or history-dependent.

What carries the argument

The central object is a hidden-mode Markov decision process (HM-MDP) augmented by two history dependencies. Mode transitions are state-conditioned, so the animal's current location influences which goal it next pursues; within a mode, reward and policy depend on the last L states, implemented by state augmentation, meaning the recent state history is appended to the current state to make a non-Markovian process Markovian in an enlarged state space. Inference is an EM loop: the E-step runs forward-backward message passing over hidden modes using the current policies, and the M-step updates rewards and transitions by gradient descent on an auxiliary likelihood, with the reward objective solved through soft-Q iteration inside the maximum-entropy IRL dual. The mechanism works because the EM objective separates into a term for mode transitions and a term for the mode-conditioned policy, so any MaxEnt-compatible IRL solver can be plugged in; the model-based soft-Q version is used in the main experiments.

What would settle it

Run S-2 on a simulated gridworld where the ground-truth policy is deliberately suboptimal (larger temperature or action noise) and see whether the reward maps and segment accuracy collapse, or run it on labyrinth data from non-water-restricted mice and check whether the recovered water-mode reward still assigns higher value to leaving the port than staying; either failure would show the inferred history dependence is an artifact of the soft-optimality assumption rather than a recovered motivation.

Watch

Extended reading notes

Core claim

The paper claims that SWIRL is the first IRL model to incorporate history-dependent policies and rewards. On its own terms, its central discovery is that unobserved switches in animal motivation and memory of recent locations can be jointly recovered from long behavioral trajectories by embedding IRL in a hidden-mode Markov decision process. The generative model assumes each hidden mode carries its own reward function over the last L states and a corresponding Boltzmann-softmax policy, while modes transition in a way that depends on the animal's current state. Inference alternates expectation-maximization with maximum-entropy IRL solved by soft-Q iteration; the paper shows empirically that this full variant recovers the true non-Markovian water and home reward maps in simulation and yields interpretable water, home, and explore segments from the labyrinth, while on a syllable-level spontaneous-behavior dataset it finds the data are better explained by Markovian rewards with state-dependent mode switching.

Load-bearing premise

The load-bearing premise is that the animals act near-optimally with respect to an internal reward function of the assumed Boltzmann-softmax form; if their choices are substantially suboptimal or that policy family is misspecified, the inferred rewards need not correspond to the animals' actual motivations.

Editorial extensions

If this is right

  • On the labyrinth data, SWIRL segments seven-hour trajectories into water, home, and explore modes without being told where the water port or home is, and water-port visits land at the end of water-mode segments; this is the paper's demonstration that long non-stereotyped behavior can be carved into goal-directed processes.
  • The recovered water-mode reward is history-dependent: arriving at the port from elsewhere is worth 0.7, but after already being at the port, leaving is worth 1.0 while staying is worth 0.4, matching the 90-second water-delivery constraint.
  • SWIRL generalizes the autoregressive HMM class: in a deterministic MDP where each action uniquely determines the next state, SWIRL reduces to an ARHMM, and adding state-conditioned mode transitions reproduces the recurrent ARHMM; unlike those models, SWIRL also outputs a reward function.
  • On the spontaneous-behavior data, the best model is S-1, with no action-level history dependence, so the framework acts as a hypothesis test: this dataset is better explained by Markovian rewards plus state-dependent mode switching than by longer non-Markovian reward memory.
  • SWIRL can be combined with model-free IRL (IQ-Learn) in place of soft-Q iteration, recovering the high-reward regions of the gridworld with a reward-map correlation of 0.737, indicating the framework does not depend on having a perfect environment model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 'explore' mode in the labyrinth is a residual category by construction: any behavior not well explained by water or home seeking is absorbed into it, so its interpretation as a genuine exploratory motivation should be tested against independent measures such as head-direction or novel-object exposure.
  • SWIRL's history length L can be read as an estimated memory span: in the labyrinth, larger L gives higher held-out likelihood, consistent with the animal coping with partial observability; this suggests SWIRL could serve as a behavioral assay for memory, with the prediction that manipulations impairing spatial memory should lower the effective L or degrade S-2 segmentation.
  • Because the EM objective only touches the policy and transition terms, the same switching-history machinery could be transplanted to other domains with context-dependent goals, such as robots or language agents, though the paper does not demonstrate this.
  • The dopamine correlations (0.40 and 0.48) in the spontaneous-behavior experiment are suggestive but not causal; a testable implication is that transiently perturbing dopamine should shift the inferred mode-specific reward maps in the directions the paper associates with each hidden mode.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. SWIRL (SWitching IRL) extends time-varying inverse reinforcement learning by modeling long behavioral sequences as hidden-mode Markov decision processes with two additional dependencies: decision-level switching that depends on the current state, and action-level reward and policy dependence on the previous L states. The inference procedure is an EM algorithm in which the E-step uses forward-backward message passing over hidden modes and the M-step updates the reward and transition parameters, with the policy computed by soft-Q iteration. The paper reports three experiments: a 5x5 gridworld simulation where the full model S-2 best recovers the true reward maps and segmentation, a water-restricted 127-node labyrinth mouse dataset where S-2 yields interpretable water/home/explore reward maps and segments, and a spontaneous-behavior dataset where the best model is S-1 rather than S-2. The paper also claims a formal connection between SWIRL and autoregressive hidden Markov models and presents a model-free IQ-Learn variant in the appendix.

Significance. If the central claims hold, SWIRL provides a useful methodological bridge between time-varying IRL and history-dependent decision-making in naturalistic animal behavior, and the model-free extension (Appendix C.1) suggests broader applicability. The paper has concrete strengths: the EM derivation in Appendix A is standard and checkable, the comparisons use held-out test log-likelihood over 10 runs, code is publicly available, and the spontaneous-behavior experiment honestly reports a dataset where action-level history dependency does not help. The main risks are the untested policy-optimality assumption underlying the real-data reward interpretations, the selective presentation of L=2 in the labyrinth when L=3 and L=4 obtain higher test log-likelihood, and the abstract's universal claim that SWIRL outperforms models without history dependency.

major comments (5)
  1. [Abstract and §4.3] The abstract and §1 state that SWIRL 'outperforms models lacking history dependency' and that adding history dependency improves accuracy, but Fig. 4B shows S-1 and I-1 (L=1) beating S-2 and I-2 (L=2) on the spontaneous-behavior dataset, and the text acknowledges this. The universal claim should be replaced by a claim scoped to datasets where non-Markovian reward structure is expected or empirically supported.
  2. [§4.2 and Appendix B.6.2] The main text presents S-2 (L=2) as the full SWIRL model, but Appendix B.6.2 reports that S-3 and S-4 achieve higher held-out test log-likelihood in the labyrinth dataset. Selecting L=2 after observing S-4's higher test LL and presenting only S-2 in the main results is post hoc; either report the L=3/L=4 results as primary or justify L=2 with a model-selection criterion that does not depend on the main evaluation metric.
  3. [§4.2 and §3.3] The labyrinth's water port delivers water at most once every 90 seconds, so the reward-relevant history is time since last water. SWIRL's action-level history is a window of L previous states (L=2 in the main text), and the manuscript does not state the temporal resolution of the 500 time points or show that two state transitions cover the 90-second constraint. If the window is shorter than the constraint, the recovered 'history-dependent' water reward in Fig. 3C may be an approximation of a longer time dependency, and the interpretability claim is not supported as stated.
  4. [§4.2 and Appendix B.7] The only robustness experiment permutes random fractions of states and actions in the training data. This tests tolerance to observation noise, not the correctness of the soft-Q/Boltzmann policy class or the fixed temperature alpha. Since the real-data reward maps are interpreted mechanistically, the manuscript should include a misspecification experiment, e.g., fitting SWIRL to trajectories generated by a policy with a different rationality level or by a non-Boltzmann policy, and report how reward correlation and segmentation accuracy degrade.
  5. [§3.5] The claim that SWIRL 'is a more generalized version of ARHMM' relies on the step 'st+1 directly implies at', but the stated condition 'each action at uniquely determines st+1' does not make the action-to-state map injective; multiple distinct actions can lead to the same next state. Unless injectivity is assumed, the reduction pi(at|st, zt) = p(st+1|st, zt) does not follow, and the generality claim needs qualification or an additional assumption.
minor comments (4)
  1. [Fig. 3B and Fig. 4B] The test log-likelihood differences between models are reported as box plots over 10 runs, but no significance tests or confidence intervals are given; the text says state dependency and history dependency 'contributed to improved performance' without quantifying the uncertainty in these differences.
  2. [Fig. 3C and Fig. 3D] The labyrinth reward heatmaps appear to be from a single run; reporting stability across the 10 runs (e.g., bootstrap intervals or example variability) would strengthen the interpretability claim.
  3. [§3.3 and Fig. 2] The notation rz: S^L x A -> R suggests rewards depend on actions, but the experiments and figures primarily describe state-only rewards r(st, st-1); this discrepancy should be clarified in the notation or in the text.
  4. [Fig. 4A and §4.3] The 'DA corr' values are Pearson correlations between inferred reward maps and averaged dopamine fluctuation levels, but the averaging procedure for the dopamine signal is not specified in the text or caption.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the SWIRL derivation is fit-based but validated on held-out data and synthetic ground truth, and its stated optimality assumption is an assumption, not a definitional loop.

full rationale

The paper's derivation chain is self-contained: it defines an HM-MDP with history-dependent rewards and policies, fits the model by EM/MaxEnt IRL, validates reward recovery against ground-truth reward maps in simulation, and selects models using held-out test log-likelihood. No equation defines the fitted reward in terms of the quantity it is claimed to predict. In the gridworld experiment, recovered rewards are compared with the true reward maps (Fig. 2A), and test segmentation accuracy is evaluated on held-out trajectories, so the reported success is not forced by the training objective. The connection to ARHMM is an explicit mathematical reduction: in a deterministic MDP, marginalizing the policy over actions collapses to p(s_t+1 | s_t, z_t), which is an equivalence, not a renaming that smuggles in the conclusion. The optimality and Boltzmann-form assumptions for real data are stated openly (Sec. 4.2) rather than hidden in a self-citation, and the temperature parameter is searched rather than fixed to the answer. Citations to prior IRL methods (Ziebart, Zeng, Garg) are external and load-bearing only as standard results; the one self-citation (Wu et al. 2024) is contextual and not used to justify the core claim. The paper also honestly reports cases where its preferred model does not win, including the spontaneous-behavior dataset where history dependency lowers held-out test LL (Sec. 4.3) and the labyrinth result where longer history L=4 achieves higher test LL than the presented S-2 (Appendix B.6.2). These admissions further show the model-selection procedure is not constructed to force the paper's narrative. Any concern about whether mice are truly softmax-optimal is an identifiability and assumption-validity concern, not a circularity of the derivation.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on a small set of model-selection parameters (Z, L, alpha, gamma) and on behavioral assumptions (optimality, known MDP, first-order switching). These are standard for IRL but make the recovered rewards depend on the soft-max temperature and on the choice of mode count and history length, which the paper handles through held-out model comparison.

free parameters (4)
  • Number of hidden modes Z = Z=3 (labyrinth), Z=5 (spontaneous)
    Chosen by held-out test LL (Appendix B.6, Fig 5A, Fig 4B). Z is a model-selection knob, not a physical constant.
  • History length L = L=2 (labyrinth primary), L=1 (spontaneous), L=4 higher test LL
    Chosen by model comparison; L=2 presented as primary despite S-4 higher test LL (Appendix B.6.2).
  • Soft-Q temperature alpha = searched over {0.01,0.1,0.5,1}
    Optimal alpha varies by dataset (Sec B.5); reward recovery is sensitive to this choice.
  • Discount factor gamma = gamma=0.95 (0.7 tested for spontaneous)
    Standard RL value; authors report low sensitivity (Sec B.5).
assumptions (4)
  • domain assumption Animals act optimally (soft-Q Boltzmann policy) w.r.t. an internal reward
    Assumed in Sec 4.2 for the labyrinth and in the gridworld; makes IRL identifiable but is untestable from behavior alone.
  • domain assumption MDP transition kernel P(s'|s,a) is known and finite
    Used in the objective (Eq. 2) and in soft-Q iteration; the labyrinth transitions are known from the binary-tree structure.
  • domain assumption Hidden modes switch via first-order Markov chain conditioned on current state, Pz(z'|z,s)
    Defines decision-level dependency (Sec 3.3); longer cross-mode dependencies are not modeled.
  • domain assumption Reward within a mode depends only on the last L states and the action
    Action-level history dependency (Sec 3.3); excludes dependence on earlier states or actions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Inverse Reinforcement Learning with Switching Rewards and History Dependency for Characterizing Animal Behaviors." pith.science (2026). https://pith.science/paper/XLN36F6L

@misc{pith2026250112633,
  author       = {Pith},
  title        = {Pith review of: Inverse Reinforcement Learning with Switching Rewards and History Dependency for Characterizing Animal Behaviors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XLN36F6L}},
  note         = {Machine review of arXiv:2501.12633}
}
read the original abstract

Traditional approaches to studying decision-making in neuroscience focus on simplified behavioral tasks where animals perform repetitive, stereotyped actions to receive explicit rewards. While informative, these methods constrain our understanding of decision-making to short timescale behaviors driven by explicit goals. In natural environments, animals exhibit more complex, long-term behaviors driven by intrinsic motivations that are often unobservable. Recent works in time-varying inverse reinforcement learning (IRL) aim to capture shifting motivations in long-term, freely moving behaviors. However, a crucial challenge remains: animals make decisions based on their history, not just their current state. To address this, we introduce SWIRL (SWitching IRL), a novel framework that extends traditional IRL by incorporating time-varying, history-dependent reward functions. SWIRL models long behavioral sequences as transitions between short-term decision-making processes, each governed by a unique reward function. SWIRL incorporates biologically plausible history dependency to capture how past decisions and environmental contexts shape behavior, offering a more accurate description of animal decision-making. We apply SWIRL to simulated and real-world animal behavior datasets and show that it outperforms models lacking history dependency, both quantitatively and qualitatively. This work presents the first IRL model to incorporate history-dependent policies and rewards to advance our understanding of complex, naturalistic decision-making in animals.

Figures

Figures reproduced from arXiv: 2501.12633 by the authors.

Figure 1
Figure 1. SWIRL graphical model. Green dotted lines repre￾sent transitions of the hidden modes depend on the previous state (decision-level dependency). Blue dotted lines represent that po￾lices depend on past states (action-level dependency). from an expert agent π ∗ , comprising a set of state-action pairs {(s ∗ t , a∗ t )}. The goal is to estimate the policy and reward that generated these state-action pairs, often referre… view at source ↗
Figure 2
Figure 2. Simulation experiment on a 5 × 5 gridworld. (A) Comparison between the true and discovered reward maps. The color scale represents reward values ranging from 0 to 1. The home reward is defined as r(st), while the water reward depends on both the current and previous locations, r(st, st−1). To present the water reward, each location is divided into five groups, as detailed in the table on the far right. For example, … view at source ↗
Figure 3
Figure 3. Water-restricted labyrinth experiment. (A) Setup for the labyrinth experiment. (B) Boxplot showing held-out test LL over 10 runs, with the x-axis representing the five different models. Outlier selection method is described in Appendix B.8. (C) History dependency inferred by SWIRL (S-2), as reflected in reward maps of the water mode and the explore mode. (D) Inferred reward maps from SWIRL (S-2) under three hidden m… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Mouse spontaneous behavior experiment. (A) SWIRL (S-1) inferred reward maps for five hidden modes. h1 denotes hidden mode 1, and so on. For better visualization, reward values are normalized to a range of (0, 1). DA corr represents the Pearson correlation between the i…
Figure 5
Figure 5. Figure 5: Water-restricted labyrinth experiment with different number of hidden modes Z. (A) Held-out test LL for each model across different number of hidden modes over 10 runs. The shaded area represents the total area that falls between one standard deviation above and below …
Figure 6
Figure 6. Figure 6: Water-restricted labyrinth experiment with different action-level history length L. (A) Boxplot showing held-out test LL over 10 runs, with the x-axis representing the four different models from L = 1 to L = 4. Outlier selection method is described in Appendix B.8. (B)…
Figure 7
Figure 7. Figure 7: SWIRL (S-2) experiment on 5 × 5 gridworld dataset with ten random permutations. Box plots illustrating the Pearson correlation between the true and recovered reward maps, test log-likelihood, and test segmentation accuracy. The x-axis represents the percentage of state…
Figure 8
Figure 8. Figure 8: True and discovered reward maps in model-free SWIRL (S-2) simulation experiment. The color scale represents reward values ranging from 0 to 1. The home reward is defined as r(st), while the water reward depends on both the current and previous locations, r(st, st−1). T…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Distributional Inverse Reinforcement Learning

    cs.LG 2025-10 unverdicted novelty 6.0 of 10

    DistIRL recovers reward distributions and risk-aware policies from offline demonstrations by minimizing first-order stochastic dominance violations between agent and expert returns.

Reference graph

Works this paper leans on

42 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the Twenty-First International Conference on Machine Learning, ICML '04, pp.\ 1, New York, NY, USA, 2004. Association for Computing Machinery. ISBN 1581138385. doi:10.1145/1015330.1015430. URL https://doi.org/10.1145/1015330.1015430

    arXiv 2004

  2. [2]

    Zoe Ashwood, Aditi Jha, and Jonathan W. Pillow. Dynamic inverse reinforcement learning for characterizing animal behavior. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022 a . URL https://openreview.net/forum?id=nosngu5XwY9

    work page 2022

  3. [3]

    Ashwood, Nicholas A

    Zoe C. Ashwood, Nicholas A. Roy, Iris R. Stone, Anne E. Urai, Anne K. Churchland, Alexandre Pouget, Jonathan W. Pillow, and The International Brain Laboratory. Mice alternate between discrete strategies during perceptual decision-making. Nature Neuroscience, 25 0 (2): 0 201--212, Feb 2022 b . ISSN 1546-1726. doi:10.1038/s41593-021-01007-z. URL https://doi...

    doi:10.1038/s41593-021-01007-z 2022

  4. [4]

    Monica Babes-Vroman, Vukosi Marivate, Kaushik Subramanian, and Michael L. Littman. Apprenticeship learning about multiple intentions. In International Conference on Machine Learning, 2011. URL https://api.semanticscholar.org/CorpusID:9737656

    work page 2011

  5. [5]

    Reinforcement learning with lstm in non-markovian tasks with longterm dependencies

    Bram Bakker. Reinforcement learning with lstm in non-markovian tasks with longterm dependencies. Memory, pp.\ 1--18, 2001

    work page 2001

  6. [6]

    Option-aware adversarial inverse reinforcement learning for robotic control

    Jiayu Chen, Tian Lan, and Vaneet Aggarwal. Option-aware adversarial inverse reinforcement learning for robotic control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp.\ 5902--5908, 2023. doi:10.1109/ICRA48891.2023.10160374

    arXiv 2023

  7. [7]

    Learning robust rewards with adverserial inverse reinforcement learning

    Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=rkHywl-A-

    2018

  8. [8]

    Iq-learn: Inverse soft-q learning for imitation

    Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp.\ 4028--4039. Curran Associates, Inc., 2021. URL https://proceedings.neurips...

    work page 2021

Show all 42 references
  1. [9]

    Reinforcement learning with deep energy-based policies

    Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pp.\ 1352--1361. PMLR, 2017

    2017

  2. [10]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, vo...

    2018

  3. [11]

    Area-specificity and plasticity of history-dependent value coding during learning

    Ryoma Hattori, Bethanny Danskin, Zeljana Babic, Nicole Mlynaryk, and Takaki Komiyama. Area-specificity and plasticity of history-dependent value coding during learning. Cell, 177 0 (7): 0 1858--1872.e15, 2019. ISSN 0092-8674. doi:https://doi.org/10.1016/j.cell.2019.04.027. URL...

    2019 doi

  4. [12]

    Deep recurrent q-learning for partially observable mdps

    Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In AAAI 2015 Fall Symposium, 2015

    2015

  5. [13]

    Can ai predict animal movements? filling gaps in animal trajectories using inverse reinforcement learning

    Tsubasa Hirakawa, Takayoshi Yamashita, Toru Tamaki, Hironobu Fujiyoshi, Yuta Umezu, Ichiro Takeuchi, Sakiko Matsumoto, and Ken Yoda. Can ai predict animal movements? filling gaps in animal trajectories using inverse reinforcement learning. Ecosphere, 9 0 (10): 0 e02447, 2018

    2018

  6. [14]

    Vime: Variational information maximizing exploration

    Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. Advances in neural information processing systems, 29, 2016

    2016

  7. [15]

    The what, how, and why of naturalistic behavior

    Ann Kennedy. The what, how, and why of naturalistic behavior. Current Opinion in Neurobiology, 74: 0 102549, 2022. ISSN 0959-4388. doi:https://doi.org/10.1016/j.conb.2022.102549. URL https://www.sciencedirect.com/science/article/pii/S0959438822000435

    2022

  8. [16]

    Recurrent switching linear dynamical systems

    Scott W Linderman, Andrew C Miller, Ryan P Adams, David M Blei, Liam Paninski, and Matthew J Johnson. Recurrent switching linear dynamical systems. arXiv preprint arXiv:1610.08466, 2016

    2016 arXiv

  9. [17]

    Spontaneous behaviour is structured by reinforcement without explicit reward

    Jeffrey E Markowitz, Winthrop F Gillis, Maya Jay, Jeffrey Wood, Ryley W Harris, Robert Cieszkowski, Rebecca Scott, David Brann, Dorothy Koveal, Tomasz Kula, Caleb Weinreb, Mohammed Abdal Monium Osman, Sandra Romero Pinto, Naoshige Uchida, Scott W Linderman, Bernardo L Sabatini...

    2023

  10. [18]

    Neural mechanisms underlying the temporal organization of naturalistic animal behavior

    Luca Mazzucato. Neural mechanisms underlying the temporal organization of naturalistic animal behavior. Elife, 11: 0 e76577, 2022

    2022

  11. [19]

    Ng and Stuart J

    Andrew Y. Ng and Stuart J. Russell. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning, ICML '00, pp.\ 663–670, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc. ISBN 1558607072

    2000

  12. [20]

    Inverse reinforcement learning with locally consistent reward functions

    Quoc Phong Nguyen, Kian Hsiang Low, and Patrick Jaillet. Inverse reinforcement learning with locally consistent reward functions. In Neural Information Processing Systems, 2015. URL https://api.semanticscholar.org/CorpusID:182075

    2015

  13. [21]

    Neural map: Structured memory for deep reinforcement learning

    Emilio Parisotto and Ruslan Salakhutdinov. Neural map: Structured memory for deep reinforcement learning. arXiv preprint arXiv:1702.08360, 2017

    2017 arXiv

  14. [22]

    Inverse reinforcement learning of bird flocking behavior

    Robert Pinsler, Max Maag, Oleg Arenz, and Gerhard Neumann. Inverse reinforcement learning of bird flocking behavior. In ICRA Swarms Workshop, 2018

    2018

  15. [23]

    Mice in a labyrinth show rapid learning, sudden insight, and efficient exploration

    Matthew Rosenberg, Tony Zhang, Pietro Perona, and Markus Meister. Mice in a labyrinth show rapid learning, sudden insight, and efficient exploration. eLife, 10: 0 e66175, jul 2021. ISSN 2050-084X. doi:10.7554/eLife.66175. URL https://doi.org/10.7554/eLife.66175

    2021 doi

  16. [24]

    Obtaining reward functions of rats using inverse reinforcement learning

    Can Eren Sezener, Eiji Uchibe, and Kenji Doya. Obtaining reward functions of rats using inverse reinforcement learning. In Türkiye Autonomous Robots Conference, 2014

    2014

  17. [25]

    Active sensing with predictive coding and uncertainty minimization

    Abdelrahman Sharafeldin, Nabil Imam, and Hannah Choi. Active sensing with predictive coding and uncertainty minimization. Patterns, 5 0 (6), Jun 2024. ISSN 2666-3899. doi:10.1016/j.patter.2024.100983. URL https://doi.org/10.1016/j.patter.2024.100983

    2024

  18. [26]

    Latent Variable Models for Characterizing the Dynamic Structure Underlying Complex Behaviors

    Iris Reid Stone. Latent Variable Models for Characterizing the Dynamic Structure Underlying Complex Behaviors. PhD thesis, Princeton University, 2023

    2023

  19. [27]

    Bayesian nonparametric inverse reinforcement learning for switched markov decision processes

    Amit Surana and Kunal Srivastava. Bayesian nonparametric inverse reinforcement learning for switched markov decision processes. In 2014 13th International Conference on Machine Learning and Applications, pp.\ 47--54. IEEE, 2014

    2014

  20. [28]

    Dyna, an integrated architecture for learning, planning, and reacting

    Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 2 0 (4): 0 160--163, 1991

    1991

  21. [29]

    Keypoint-moseq: parsing behavior by linking point tracking to pose dynamics

    Caleb Weinreb, Jonah E Pearl, Sherry Lin, Mohammed Abdal Monium Osman, Libby Zhang, Sidharth Annapragada, Eli Conlin, Red Hoffmann, Sofia Makowska, Winthrop F Gillis, et al. Keypoint-moseq: parsing behavior by linking point tracking to pose dynamics. Nature Methods, 21 0 (7): ...

    2024

  22. [30]

    Mapping sub-second structure in mouse behavior

    Alexander B Wiltschko, Matthew J Johnson, Giuliano Iurilli, Ralph E Peterson, Jesse M Katon, Stan L Pashkovski, Victoria E Abraira, Ryan P Adams, and Sandeep Robert Datta. Mapping sub-second structure in mouse behavior. Neuron, 88 0 (6): 0 1121--1135, 2015

    2015

  23. [31]

    C. F. Jeff Wu. On the convergence properties of the em algorithm. The Annals of Statistics, 11 0 (1): 0 95--103, 1983. ISSN 00905364, 21688966. URL http://www.jstor.org/stable/2240463

    1983

  24. [32]

    Inverse reinforcement learning with the average reward criterion

    Feiyang Wu, Jingyang Ke, and Anqi Wu. Inverse reinforcement learning with the average reward criterion. Advances in Neural Information Processing Systems, 36, 2024

    2024

  25. [33]

    Imitating language via scalable inverse reinforcement learning, 2024

    Markus Wulfmeier, Michael Bloesch, Nino Vieillard, Arun Ahuja, Jorg Bornschein, Sandy Huang, Artem Sokolov, Matt Barnes, Guillaume Desjardins, Alex Bewley, Sarah Maria Elisabeth Bechtle, Jost Tobias Springenberg, Nikola Momchev, Olivier Bachem, Matthieu Geist, and Martin Riedm...

    2024 arXiv

  26. [34]

    Identification of animal behavioral strategies by inverse reinforcement learning

    Shoichiro Yamaguchi, Honda Naoki, Muneki Ikeda, Yuki Tsukada, Shunji Nakano, Ikue Mori, and Shin Ishii. Identification of animal behavioral strategies by inverse reinforcement learning. PLoS computational biology, 14 0 (5): 0 e1006122, 2018

    2018

  27. [35]

    Maximum-likelihood inverse reinforcement learning with finite-time guarantees

    Siliang Zeng, Chenliang Li, Alfredo Garcia, and Mingyi Hong. Maximum-likelihood inverse reinforcement learning with finite-time guarantees. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35...

    2022

  28. [36]

    Multi-intention inverse q-learning for interpretable behavior representation

    Hao Zhu, Brice De La Crompe, Gabriel Kalweit, Artur Schneider, Maria Kalweit, Ilka Diester, and Joschka Boedecker. Multi-intention inverse q-learning for interpretable behavior representation. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openrev...

    2024

  29. [37]

    Ziebart, Andrew Maas, J

    Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 3, AAAI'08, pp.\ 1433–1438. AAAI Press, 2008. ISBN 9781577353683

    2008

  30. [38]

    Ziebart, J

    Brian D. Ziebart, J. Andrew Bagnell, and Anind K. Dey. Modeling interaction via the principle of maximum causal entropy. In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML'10, pp.\ 1255–1262, Madison, WI, USA, 2010. Omnipr...

    2010

  31. [39]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  32. [40]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  33. [41]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  34. [42]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.