Pith. sign in

REVIEW 2 major objections 4 minor 79 references

This paper proves that, once von Neumann–Morgenstern independence is restricted to environmental randomness, the combination of independence of irrelevant alternatives and monotonicity uniquely forces the softmax (Boltzmann) policy, the ent

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 18:21 UTC pith:WP6OPLHG

load-bearing objection Novel axiom-to-softmax path with a solid core, but the soft Bellman equation rides on an explicit, unproven extension assumption that the abstract glosses over. the 2 major comments →

arxiv 2607.17316 v1 pith:WP6OPLHG submitted 2026-07-19 cs.LG econ.TH

Rationalizing Boltzmann Rationality: An Axiomatic Characterization of Entropy-Regularized Policies

Silviu Pitis This is my paper
classification cs.LG econ.TH MSC 91B1690C4091A10
keywords Axiomatic RLDecision TheoryStochastic PoliciesIndependence of Irrelevant AlternativesEntropy RegularizationBoltzmann RationalitySoft Bellman Equation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to prove that the softmax (Boltzmann) policy, the backbone of modern reinforcement learning, is not just a convenient or robust heuristic but the unique rational response to a precise set of axioms. The move is to split randomness into two kinds: environmental chance, which is evaluated by von Neumann–Morgenstern expected utility from deterministic plans ('base prospects'), and the agent's own choice, which is governed by independence of irrelevant alternatives and monotonicity. If those axioms are accepted, the paper's main theorem derives the softmax policy, the entropy-regularized objective, and the soft Bellman equation from scratch, giving a first-principles justification for why the entropy bonus appears. The paper also shows what changes if the axioms are rejected—correlated action sets can break IIA, producing the red-bus/blue-bus failure mode—and draws concrete reinforcement-learning consequences like a minimum rationality temperature under amplifying discounting. A sympathetic reader should care because the paper converts a default engineering choice into a testable normative commitment, and makes explicit which axis of the problem (chance vs. choice) is doing the moral work.

Core claim

The paper's central claim is Theorem 3.1: in a finite MDP with at least three actions and contraction discounting, an agent whose preferences over environmental lotteries satisfy VNM independence, whose Bellman relation is extended to successor choice nodes, and whose choice rule and value function satisfy IIA (both policy-level and value-level) and monotonicity must have π(a|s) ∝ exp(βQ(s,a)), must choose π to maximize Σ_a π_a Q(s,a) + (1/β)H(π), and must have V(s) = (1/β) LSE(βQ(s,·)) with Q(s,a) = R(s,a) + γ(s,a) E[V(s′)]. In other words, the entropy bonus is not an ad hoc regularizer but the exact 'option premium' an agent pays for being able to choose among alternatives rather than faci

What carries the argument

The load-bearing object is the chance/choice separation. A base prospect is a deterministic plan with only environmental randomness; VNM Independence applies there. At choice nodes, two axioms are imposed: Policy IIA requires the ratio π(a|q)/π(b|q) to depend only on q_a−q_b, and Value IIA requires each action's selection probability to be a function of the loss in menu value when that action is removed. Monotonicity makes those functions invertible. The proof's engine is the functional equation that emerges from evaluating a three-action menu by adding actions in different orders; its unique solution is f(p)=-(1/β)log(1-p), which converts the value of any menu into (1/β)LSE(βQ). Marginal co

Load-bearing premise

The paper explicitly assumes, rather than derives, that the Bellman relation extends from deterministic continuation policies to successor choice nodes (Section 3.1), so that Q(s,a)=R(s,a)+γ(s,a)E[V(s′)] with V evaluated at the choice-node value; without this extension, the derivation of the soft Bellman equation does not follow from Axioms 1 and 2 alone, and the theorem's conclusion (iii) lacks support from the stated axioms.

What would settle it

Numerically solve the two functional equations from Step 2 of the proof—f(p)-f(1-p)=(1/β)log(p/(1-p)) and f(p/(1-t))+f(t)=f(t/(1-p))+f(p) for p+t<1—for continuous, monotone f with f(0)=0. If a solution other than f(p)=-(1/β)log(1-p) exists, the axiomatic system admits non-Boltzmann policies and Theorem 3.1 is false; a reader can test this directly with a simple solver.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the axioms are accepted, any agent whose action preferences are menu-independent must be a Boltzmann agent; the only free parameter is the inverse temperature β.
  • Shannon entropy is the unique entropy that satisfies Value IIA; Tsallis, sparsemax, and other regularizers are excluded by the axioms, not just by convenience.
  • The choice between the soft and the hard Bellman equation is a normative choice: whether the agent values its own ability to choose.
  • With state-action dependent discount factors larger than one, the soft Bellman equation can fail to have a finite fixed point below a threshold β0; horizon continuity, which keeps all discounted transition matrices contractive, removes the threshold and guarantees convergence for all β.
  • Actual unregularized return is non-decreasing in β, so increasing deliberateness (higher β) never hurts true performance in finite discounted MDPs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The axiomatization yields a practical audit: check whether log(π(a)/π(b)) changes when other actions are added or removed; violations signal whether IIA is appropriate for the domain, which is a testable property of deployed policies.
  • The same chance/choice split could justify hierarchical choice models such as nested logit when actions are correlated; the paper points to this as open, and it would be the natural fix for the near-duplicate problem in language-model decoding.
  • If the Gumbel-noise story and the information-cost story are equivalent to this axiomatic system under the stated conditions, then those practical justifications are not independent heuristics but consequences of the same axioms—suggesting the Boltzmann form is overdetermined.
  • Because only the product β·reward-scale is identified, any claim about 'the right temperature' for stochastic policies is empty without fixing the reward scale; hyperparameter searches can be reduced to one dimension.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper presents an axiomatic derivation of entropy-regularized (Boltzmann) policies and the soft Bellman equation in finite MDPs. The key modeling move is to restrict VNM Independence to environmental lotteries over deterministic base prospects, reserving IIA and monotonicity (in both Policy and Value versions) for the agent's choice rule at decision nodes. Theorem 3.1 claims these axioms uniquely imply the softmax policy, the entropy-regularized policy objective, and the soft Bellman equation; the proof proceeds in three steps (Cauchy functional equation, menu-value functional equations, then Bellman recursion). The paper then studies consequences: divergence of the soft Bellman equation under γ>1 and its resolution by horizon continuity, monotonicity of actual return in β, and a numerical demonstration that Shannon entropy is the only IIA-consistent member of the Tsallis family. It closes with a cross-disciplinary synthesis and a normative discussion of when IIA is appropriate.

Significance. If the characterization were fully rigorous, it would be a valuable contribution: it would supply a missing first-principles justification for softmax in RL, clarify the normative content of the entropy bonus as an 'option premium,' and connect RL to a large body of choice-theoretic and econometric results. The paper is unusually transparent about the strength of its axioms, especially Value IIA, and the Step 2 derivation of the Shannon entropy bonus from menu-level functional equations is a genuine contribution. The discussion of generalized discounting and the β0 threshold is also suggestive. However, the central theorem depends on an extra structural assumption about the Bellman relation that is not derived from the stated axioms, which currently limits the force of the uniqueness claim.

major comments (2)
  1. [Section 3.1 / Step 3 of Theorem 3.1] The assumption that 'the Bellman relation extends from deterministic continuation policies to successor choice nodes' is stated in prose and is not derived from Axioms 1–4. Step 3 uses it to write Q(s,a)=R(s,a)+γ(s,a)E[V(s′)] with V(s′) the choice-node value, and then to substitute V=(1/β)LSE(βQ). Without this extension, Axioms 1–2 only determine Q for deterministic continuations, and Axioms 3–4 only determine V as a function of an arbitrary Q-vector at a single menu. The theorem's 'if and only if' is therefore conditional on an unproved structural assumption, and the abstract's claim that IIA and monotonicity uniquely determine the soft Bellman equation is not supported. Please either prove the extension from Axioms 1–2 or state it as an explicit numbered axiom with its own normative justification.
  2. [Footnote 1 / Step 2 of Theorem 3.1] The proof of Step 2 assumes that submenu limits exist and are finite, and specifically that f(p)→0 as p→0, in order to conclude h(p)→0 at the boundary and hence h≡0. This assumption is not a consequence of Axioms 3 and 4 as stated (monotonicity alone does not give continuity at the boundary of the probability simplex). It is load-bearing for the uniqueness of the Shannon entropy among solutions of the functional equations (1)–(3). If this regularity is needed, it should be elevated to an explicit axiom or proved from finiteness of the action set plus the other axioms; otherwise non-Shannon solutions may survive.
minor comments (4)
  1. [Section 1] Typo: 'V on Neumann-Morgenstern' should read 'von Neumann–Morgenstern.'
  2. [Table 1] The footnote markers (∗, †) are not fully defined in the caption; in particular, the Thompson-sampling row uses '✓∗' in two columns but the footnote only explains the conditions partially. Please clarify.
  3. [Figure 4.3] The y-axis label 'std of log (a)/(b)' should be rendered as 'std of log(π(a)/π(b))' and the simulation details (number of draws, treatment of zero-probability draws) should be stated in the figure caption.
  4. [Proof of Proposition 4.3] The proof suppresses the start state and asserts that πα is optimal from every start state at once. This is true for the regularized MDP but should be stated explicitly, since the reader might otherwise wonder whether the envelope argument requires a single start state.

Circularity Check

0 steps flagged

No significant circularity: Boltzmann policy and entropy bonus are derived from stated axioms via functional equations, not assumed; the explicit Bellman-extension assumption is a scope limitation, not a circular step.

full rationale

The paper's central derivation is not circular. Step 1 derives the softmax form from Policy IIA (Axiom 3a) and Policy Monotonicity (Axiom 4a) using the standard Cauchy functional-equation argument, an independent classical result (Luce 1959; Aczél 1966). Step 2 derives the entropy bonus from Value IIA (Axiom 3b) and Monotonicity by solving functional equations (1)–(3); Shannon entropy is not assumed in the axioms. Value IIA is admittedly strong and does much of the work, but it is a coherence axiom linking choice probabilities to menu-value differences, not the target representation itself. Step 3 substitutes the derived V=(1/β)LSE(βQ) into the Bellman relation Q=R+γE[V], which is algebra rather than a circular reduction. The most significant caveat is the explicit assumption in Section 3.1 that the Bellman relation extends from deterministic continuation policies to successor choice nodes; this is stated in prose and is needed for conclusion (iii). This makes the theorem conditional on an additional structural assumption, but it is not a case of assuming the conclusion: the extension does not specify the LSE form. The self-citations to Pitis (2019) provide R and γ from prior work with stated assumptions (VNM and dynamic consistency) that do not include entropy regularization; those are independent support rather than load-bearing circularity. No fitted parameter is relabeled as a prediction, and no external benchmark is used as the derivation's input. Thus the paper does not exhibit a forbidden circular chain; at most it has a scope/assumption concern that belongs under correctness risk, not circularity. Score 1 reflects the absence of circular steps and the presence of an explicit, non-derived structural assumption plus self-citations that are not themselves challenged.

Axiom & Free-Parameter Ledger

1 free parameters · 9 axioms · 1 invented entities

The central theorem rests on the chance/choice domain restriction, the new Value IIA axiom, and the Bellman-extension assumption. The only free parameter in the representation is β. The paper is transparent about all of these, but the Bellman extension and Value IIA are strong assumptions doing most of the work.

free parameters (1)
  • Inverse temperature β = >0, unspecified
    Emerges from the multiplicative Cauchy equation in Step 1; any positive β satisfies the axioms. The paper notes (Section 5.2) that no axiomatization determines β and that choice data identify only β times reward scale.
axioms (9)
  • domain assumption VNM Expected Utility over lotteries of base prospects (Axiom 1)
    Restricts Independence to environmental randomness and excludes choice-node lotteries, which is the paper's key modeling move.
  • domain assumption Dynamic Consistency (Axiom 2)
    Orders deterministic continuation policies by continuation value; used with Pitis (2019) to derive R(s,a) and γ(s,a).
  • ad hoc to paper Bellman relation extends to successor choice nodes (Section 3.1)
    Assumes affine aggregation of P(s'|s,a) with choice-node values V(s'); necessary for Step 3 and not derived from Axioms 1–4.
  • domain assumption Policy IIA, translation-invariant (Axiom 3a)
    Requires π(a)/π(b) to depend only on q_a − q_b; a strengthened Luce IIA that directly yields the Cauchy equation in Step 1.
  • ad hoc to paper Value IIA (Axiom 3b)
    Requires choice probability to depend only on the menu-value loss from removing the action; new and load-bearing, drives Step 2.
  • domain assumption Monotonicity (Axiom 4)
    Better actions receive higher probability and improving options cannot reduce menu value; used for the exponential solution and invertibility in Step 2.
  • domain assumption Finite MDP, |A| ≥ 3, ¯γ < 1
    |A| ≥ 3 is needed for the functional-equation uniqueness; ¯γ < 1 gives contraction. Binary actions leave softmax unforced, as the paper admits.
  • domain assumption Submenu limits exist and are finite (footnote 1)
    Assumes V({a}) = q_a and limits as q_n → −∞ exist; used in the boundary argument that pins down h.
  • domain assumption Horizon continuity and spanning condition from Pitis (2019) for Proposition 4.2
    Invoked to eliminate the β0 threshold by bounding the spectral radius of every policy discount-transition matrix.
invented entities (1)
  • Option premium (choice-value interpretation of the entropy bonus) no independent evidence
    purpose: To interpret (1/β)H(π) as the value of being able to choose rather than having an outcome externally imposed.
    A normative interpretation layered on the derived entropy term; it has no falsifiable handle outside the paper.

pith-pipeline@v1.3.0-alltime-deepseek · 14936 in / 19242 out tokens · 186563 ms · 2026-08-01T18:21:37.885713+00:00 · methodology

0 comments
read the original abstract

The softmax policy $\pi(a \mid s) \propto \exp(\beta Q(s,a))$ is the default model of stochastic choice in reinforcement learning (RL). Various justifications based on robustness, exploration, and optimization have been offered in the RL literature, but none uniquely derives the softmax form from first principles. This leaves a basic tension unresolved: the entropy bonus in the soft Bellman equation violates the Independence axiom that underwrites the Markov decision process (MDP) reward structure. We dissolve this tension by distinguishing two kinds of randomness: chance and choice. By restricting von Neumann-Morgenstern (VNM) Independence to environmental lotteries over base prospects, we show that imposing independence of irrelevant alternatives (IIA) and monotonicity on the policy and value functions at choice nodes uniquely determines the Boltzmann policy, the entropy-regularized representation, and the soft Bellman equation. The choice between the soft and hard Bellman equations thus reduces to a design decision: whether the agent values its own ability to choose. We develop RL-specific consequences, including return monotonicity and convergence under generalized discounting, and synthesize the independent lines from economics and information theory that arrive at the same structure, offering a normative assessment of when IIA is appropriate for agent design.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

79 extracted references · 5 linked inside Pith

  1. [1]

    Adams, Ernest and Messick, Samuel , title =

  2. [2]

    , title =

    Kreps, David M. , title =. 1988 , series =

    1988

  3. [3]

    and de Palma, Andr\'

    Anderson, Simon P. and de Palma, Andr\'. Discrete Choice Theory of Product Differentiation , publisher =

  4. [4]

    Economic Theory , year =

    Breitmoser, Yves , title =. Economic Theory , year =

  5. [5]

    Proceedings of the 34th International Conference on Machine Learning , pages=

    Unifying Task Specification in Reinforcement Learning , author=. Proceedings of the 34th International Conference on Machine Learning , pages=

  6. [6]

    Econometrica , volume=

    Envelope Theorems for Arbitrary Choice Sets , author=. Econometrica , volume=

  7. [7]

    Lectures on Functional Equations and Their Applications , author=

  8. [8]

    Proceedings of the 33rd International Conference on Machine Learning , pages=

    Asynchronous Methods for Deep Reinforcement Learning , author=. Proceedings of the 33rd International Conference on Machine Learning , pages=

  9. [9]

    arXiv preprint arXiv:1707.06347 , year=

    Proximal Policy Optimization Algorithms , author=. arXiv preprint arXiv:1707.06347 , year=

    Pith/arXiv arXiv

  10. [10]

    arXiv preprint arXiv:2602.03003 , year=

    Methods and open problems in differentiable social choice , author=. arXiv preprint arXiv:2602.03003 , year=

    arXiv

  11. [11]

    International Conference on Machine Learning , pages=

    An alternative softmax operator for reinforcement learning , author=. International Conference on Machine Learning , pages=

  12. [12]

    International Conference on Learning Representations , year=

    Extreme Q-Learning: MaxEnt RL without Entropy , author=. International Conference on Learning Representations , year=

  13. [13]

    Advances in Neural Information Processing Systems , year=

    Double Gumbel Q-Learning , author=. Advances in Neural Information Processing Systems , year=

  14. [14]

    SIAM Journal on Computing , volume=

    The nonstochastic multiarmed bandit problem , author=. SIAM Journal on Computing , volume=

  15. [15]

    Rank analysis of incomplete block designs:

    Bradley, Ralph Allan and Terry, Milton E , journal=. Rank analysis of incomplete block designs:

  16. [16]

    Discrete dynamic choice: An extension of the choice models of

    Dagsvik, John K , journal=. Discrete dynamic choice: An extension of the choice models of

  17. [17]

    Econometrica , volume=

    Stochastic Choice and Cardinal Utility , author=. Econometrica , volume=. 1958 , publisher=

    1958

  18. [18]

    Maximum entropy

    Eysenbach, Benjamin and Levine, Sergey , booktitle=. Maximum entropy

  19. [19]

    Econometrica , volume=

    Stochastic choice and revealed perturbed utility , author=. Econometrica , volume=

  20. [20]

    Econometrica , volume=

    Dynamic logit with choice aversion , author=. Econometrica , volume=

  21. [21]

    A theory of regularized

    Geist, Matthieu and Scherrer, Bruno and Pietquin, Olivier , booktitle=. A theory of regularized

  22. [22]

    Experimental Economics , volume=

    Regular quantal response equilibrium , author=. Experimental Economics , volume=

  23. [23]

    International Conference on Machine Learning , pages=

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor , author=. International Conference on Machine Learning , pages=

  24. [24]

    American Economic Review , volume=

    On the empirical content of quantal response equilibrium , author=. American Economic Review , volume=

  25. [25]

    Review of Economic Studies , volume=

    Conditional choice probabilities and the estimation of dynamic models , author=. Review of Economic Studies , volume=

  26. [26]

    Laidlaw, Cassidy and Dragan, Anca , booktitle=. The

  27. [27]

    arXiv preprint arXiv:1805.00909 , year=

    Reinforcement learning and control as probabilistic inference: Tutorial and review , author=. arXiv preprint arXiv:1805.00909 , year=

    Pith/arXiv arXiv

  28. [28]

    Individual Choice Behavior: A Theoretical Analysis , author=

  29. [29]

    Jackpot!

    Maura-Rivero, Roberto-Rafael and Lanctot, Marc and Visin, Francesco and Larson, Kate , journal=. Jackpot!

  30. [30]

    Frontiers in Econometrics , pages=

    Conditional logit analysis of qualitative choice behavior , author=. Frontiers in Econometrics , pages=. 1973 , publisher=

    1973

  31. [31]

    Games and Economic Behavior , volume=

    Quantal response equilibria for normal form games , author=. Games and Economic Behavior , volume=

  32. [32]

    Essentially Sharp Estimates on the Entropy Regularization Error in Discrete Discounted

    M. Essentially Sharp Estimates on the Entropy Regularization Error in Discrete Discounted. arXiv preprint arXiv:2406.04163 , year=

    arXiv

  33. [33]

    Advances in Neural Information Processing Systems , year=

    Bridging the gap between value and policy based reinforcement learning , author=. Advances in Neural Information Processing Systems , year=

  34. [34]

    Advances in Neural Information Processing Systems , year=

    A* sampling , author=. Advances in Neural Information Processing Systems , year=

  35. [35]

    A unified view of entropy-regularized

    Neu, Gergely and Jonsson, Anders and G. A unified view of entropy-regularized. arXiv preprint arXiv:1705.07798 , year=

    Pith/arXiv arXiv

  36. [36]

    Proceedings of the Royal Society A , volume=

    Thermodynamics as a theory of decision-making with information-processing costs , author=. Proceedings of the Royal Society A , volume=

  37. [37]

    AAAI Conference on Artificial Intelligence , volume=

    Rethinking the discount factor in reinforcement learning: A decision theoretic approach , author=. AAAI Conference on Artificial Intelligence , volume=

  38. [38]

    Consistent aggregation of objectives with diverse time preferences requires non-

    Pitis, Silviu , booktitle=. Consistent aggregation of objectives with diverse time preferences requires non-

  39. [39]

    Optimal replacement of

    Rust, John , journal=. Optimal replacement of

  40. [40]

    IEEE Transactions on Information Theory , volume=

    Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy , author=. IEEE Transactions on Information Theory , volume=

  41. [41]

    Quarterly Journal of Economics , volume=

    A behavioral model of rational choice , author=. Quarterly Journal of Economics , volume=

  42. [42]

    Journal of Monetary Economics , volume=

    Implications of rational inattention , author=. Journal of Monetary Economics , volume=

  43. [43]

    Linearly-solvable

    Todorov, Emanuel , booktitle=. Linearly-solvable

  44. [44]

    arXiv preprint arXiv:2509.21172 , year=

    Inverse reinforcement learning using just classification and a few regressions , author=. arXiv preprint arXiv:2509.21172 , year=

    Pith/arXiv arXiv

  45. [45]

    The relationship between

    Yellott, John I , journal=. The relationship between

  46. [46]

    Bell System Technical Journal , volume=

    A mathematical theory of communication , author=. Bell System Technical Journal , volume=

  47. [47]

    Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy , author=

  48. [48]

    On the Expressivity of

    Abel, David and Dabney, Will and Harutyunyan, Anna and Ho, Mark K and Littman, Michael and Precup, Doina and Singh, Satinder , booktitle=. On the Expressivity of

  49. [49]

    International Conference on Machine Learning , pages=

    Settling the reward hypothesis , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  50. [50]

    International Conference on Machine Learning , pages=

    Utility Theory for Sequential Decision Making , author=. International Conference on Machine Learning , pages=

  51. [51]

    Advances in Neural Information Processing Systems , volume=

    Direct Preference Optimization: Your Language Model is Secretly a Reward Model , author=. Advances in Neural Information Processing Systems , volume=

  52. [52]

    Advances in Neural Information Processing Systems , volume=

    Deep Reinforcement Learning from Human Preferences , author=. Advances in Neural Information Processing Systems , volume=

  53. [53]

    IEEE International Conference on Robotics and Automation , year=

    Composable Deep Reinforcement Learning for Robotic Manipulation , author=. IEEE International Conference on Robotics and Automation , year=

  54. [54]

    Xu, Wanqiao and Dong, Shi and Lu, Xiuyuan and Lam, Grace and Wen, Zheng and Van Roy, Benjamin , journal=

  55. [55]

    Advances in Neural Information Processing Systems , volume=

    Boltzmann Exploration Done Right , author=. Advances in Neural Information Processing Systems , volume=

  56. [56]

    International Conference on Machine Learning , year=

    Understanding the Impact of Entropy on Policy Optimization , author=. International Conference on Machine Learning , year=

  57. [57]

    arXiv preprint arXiv:2506.07085 , year=

    State Entropy Regularization for Robust Reinforcement Learning , author=. arXiv preprint arXiv:2506.07085 , year=

    arXiv

  58. [58]

    American Economic Review , volume=

    Rational Inattention to Discrete Choices: A New Foundation for the Multinomial Logit Model , author=. American Economic Review , volume=

  59. [59]

    International Conference on Machine Learning , year=

    When Maximum Entropy Misleads Policy Optimization , author=. International Conference on Machine Learning , year=

  60. [60]

    Chen, Anthony GX and Prakash, Jatin and Guo, Jeff and Fergus, Rob and Ranganath, Rajesh , journal=

  61. [61]

    Lee, Kyungjae and Choi, Sungjoon and Oh, Songhwai , journal=. Sparse

  62. [62]

    Path Consistency Learning in

    Chow, Yinlam and Nachum, Ofir and Ghavamzadeh, Mohammad , booktitle=. Path Consistency Learning in

  63. [63]

    Proceedings of EMNLP , year=

    Surface Form Competition: Why the Highest Probability Answer Isn't Always Right , author=. Proceedings of EMNLP , year=

  64. [64]

    arXiv preprint arXiv:1609.08144 , year=

    Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation , author=. arXiv preprint arXiv:1609.08144 , year=

    Pith/arXiv arXiv

  65. [65]

    Meng, Yu and Xia, Mengzhou and Chen, Danqi , booktitle=

  66. [66]

    2018 , publisher=

    Reinforcement Learning: An Introduction , author=. 2018 , publisher=

    2018

  67. [67]

    Contributions to Probability and Statistics: Essays in Honor of Harold Hotelling , editor=

    Random orderings and stochastic theories of responses , author=. Contributions to Probability and Statistics: Essays in Honor of Harold Hotelling , editor=. 1960 , publisher=

    1960

  68. [68]

    Journal of Mathematical Psychology , volume=

    A representation theorem for finite random scale systems , author=. Journal of Mathematical Psychology , volume=. 1978 , publisher=

    1978

  69. [69]

    and Jaakkola, Tommi and Jordan, Michael I

    Singh, Satinder P. and Jaakkola, Tommi and Jordan, Michael I. , booktitle=. Learning Without State-Estimation in Partially Observable

  70. [70]

    1947 , edition=

    Theory of Games and Economic Behavior , author=. 1947 , edition=

    1947

  71. [71]

    Econometrica , volume=

    A Representation Theorem for ``Preference for Flexibility'' , author=. Econometrica , volume=. 1979 , publisher=

    1979

  72. [72]

    Advances in Neural Information Processing Systems , volume=

    A definition of continual reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=

  73. [73]

    Spatial Interaction Theory and Planning Models , pages=

    Modeling the choice of residential location , author=. Spatial Interaction Theory and Planning Models , pages=. 1978 , publisher=

    1978

  74. [74]

    Advances in Neural Information Processing Systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in Neural Information Processing Systems , volume=

  75. [75]

    American Economic Review , year=

    Debreu, Gerard , title=. American Economic Review , year=

  76. [76]

    Williams, H. C. W. L. , title =. Environment and Planning A , year =

  77. [77]

    and Zachary, Stan , title =

    Daly, Andrew J. and Zachary, Stan , title =. Determinants of Travel Choice , editor =. 1978 , pages =

    1978

  78. [78]

    Structural Analysis of Discrete Data with Econometric Applications , editor =

    McFadden, Daniel , title =. Structural Analysis of Discrete Data with Econometric Applications , editor =. 1981 , pages =

    1981

  79. [79]

    and Sarver, Todd , title =

    Ahn, David S. and Sarver, Todd , title =. Econometrica , year =