{"id":"dc100581-1d63-4d2d-ad41-8c6b5c17e764","arxiv_id":"2412.10692","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"For entropy-regularized reinforcement learning in a continuous-time Merton market, the optimal exploratory policy is Gaussian, becoming truncated Gaussian under interval portfolio constraints, with closed-form log and quadratic utility solutions.","lead":"The paper derives explicit optimal randomized investment policies for continuous-time portfolio choice under reinforcement learning with exploration. It shows the optimal exploration distribution is Gaussian, or truncated Gaussian when holdings are constrained, and it tests these formulas with an actor-critic algorithm.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Value functions in Thms 3.1, 4.1 and 6.1 carry a systematic +m(T-t)/2 error caused by an extra factor e in the log of Eq. (2.12); correcting it changes the closed-form values and the numerical learning targets, though the Gaussian/truncated-Gaussian policy claims survive.","rationale":"The reader's overall conditional verdict is sound, and I agree with it, but I would rank the +m(T-t)/2 value-function error as the more load-bearing concern. It is an internal algebraic inconsistency: Eq. (2.12) applies the Donsker-Varadhan variational formula incorrectly, introducing a factor e in the logarithm, and the error propagates unchanged into the closed-form value functions in Theorems 3.1, 4.1 and 6.1, as well as into the 'true' parameter values used as learning targets in Sections 5 and 6. This is not a gap that can be patched by adding assumptions; as written, those quantitative theorem statements are false. The policy improvement issue identified by the reader is real but different in character: Theorem 3.4 is an explicitly conditional statement, and the authors themselves flag in footnote 3 that well-posedness of the average PDE is an open problem, so that issue is an acknowledged limitation rather than a hidden contradiction. The policy improvement results are also supported for the Gaussian initialization actually used in the logarithmic-utility algorithm. Correcting Eq. (2.12) leaves the Gaussian and truncated-Gaussian optimal policy structure intact, preserves m→0 convergence, and likely leaves the exploration cost at mT/2 once the entropy calculations are fixed. Thus the appropriate disposition remains conditional: the paper should be revised with corrected value formulas and rerun numerical comparisons before the quantitative claims are relied upon.","tokens_in":39494,"tokens_out":19038,"duration_ms":165780,"concrete_test":"Re-derive Eq. (2.12) from Eq. (2.10) by evaluating sup_λ = m ln∫ exp(H/m)dπ; if the logarithmic term contains 2πem instead of 2πm, the value functions in Theorems 3.1, 4.1 and 6.1 each shift by m(T-t)/2. As a numerical check, simulate 10^6 paths of (3.3) with T = 1, σ = 0.3, m = 0.01 and compute the sample average of ln X_T + m∫_0^T (1/2) ln(2πe m/σ^2) dt; the corrected value ln x + (r + (1/2)(µ-r)^2/σ^2)T + (mT/2) ln(2πm/σ^2) should match, while (3.1) is larger by 0.005. Re-run Table 2 with the corrected target θ = r + (1/2)(µ-r)^2/σ^2 - m ln σ - m/2.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In (2.9) the optimization is over λ of ∫ H λ dπ - m∫ λ lnλ dπ, where H = rxv_x + (µ-r)xπv_x + (1/2)σ^2x^2π^2v_xx. The Donsker-Varadhan substitution gives sup_λ = m ln∫ exp(H/m)dπ. Completing the square with q = -σ^2x^2v_xx > 0 yields m ln∫ exp(H/m)dπ = rxv_x - (µ-r)^2v_x^2/(2σ^2v_xx) + (m/2) ln(2πm/q), so the logarithmic term in the HJB should be (m/2) ln(-2πm/(σ^2x^2v_xx)), not (m/2) ln(-2πem/(σ^2x^2v_xx)) as written in (2.12). The extra factor e adds m/2 to the HJB. For logarithmic utility, direct computation confirms the error: under λ* = N(π_Merton, m/σ^2), E[ln X_T] = ln x + (r + (1/2)(µ-r)^2/σ^2 - m/2)T and the entropy contribution is (mT/2) ln(2πem/σ^2), so the true regularized value is ln x + (r + (1/2)(µ-r)^2/σ^2)T + (mT/2) ln(2πm/σ^2). This differs from Theorem 3.1 by +mT/2. The same factor-e error propagates to (4.9) and (6.13), and to the 'true' θ values used as targets in Sections 5 and 6. The Gaussian and truncated-Gaussian policy statements are unaffected because the additive l(t) term cancels in the ratio v_x/v_xx, and the m→0 convergence is also unaffected, but the closed-form value functions stated in the theorems are not the value functions of problem (2.8).","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies an entropy-regularized exploratory optimal investment problem in a Black-Scholes market, with logarithmic and quadratic utility, in both unconstrained and interval-constrained settings. It claims that the optimal feedback policy is Gaussian in the unconstrained case and truncated Gaussian under an interval constraint, derives closed-form value functions, proves convergence to the classical Merton values as the exploration weight m goes to zero, establishes a policy improvement theorem, and proposes a martingale-based actor-critic algorithm with numerical examples. The derivations rely on the Donsker-Varadhan variational formula and an HJB reduction; several proofs are omitted with reference to prior work by Wang-Zhou and Jia-Zhou.","tokens_in":39989,"tokens_out":28716,"duration_ms":237026,"significance":"If the quantitative statements were correct, the paper would make a useful contribution to continuous-time RL portfolio choice: it gives explicit, parameter-free Gaussian policies whose mean is the Merton strategy and whose variance is m/sigma^2, it handles portfolio constraints in a clean way, and it provides an interpretable learning target rather than a black-box neural network. The policy improvement theorem and the martingale formulation are also well aligned with the modern continuous-time RL literature. However, the central closed-form value functions contain a systematic constant error, and the quadratic-utility value function has an additional sign error. These errors directly affect the numerical learning targets and the quantitative interpretation of exploration, so the main quantitative claims are not correct in their current form.","major_comments":[{"comment":"The logarithmic term in the HJB equation (2.12) contains an extra factor e: the Donsker-Varadhan supremum of ∫Hλ dπ - m∫λ log λ dπ is m log∫exp(H/m)dπ, and completing the square gives (m/2)log(-2πm/(σ^2 x^2 v_xx)), not (m/2)log(-2πem/(σ^2 x^2 v_xx)). Consequently the stated value functions in Theorem 3.1, Remark 3.1, Theorem 4.1, Theorem 6.1, Corollary 6.1, and the extension in Eq. (7.7) are off by a term of the form (m/2)(T-t). A direct check for logarithmic utility confirms this: under λ* = N(π_Merton, m/σ^2), one has E[ln X_T] = ln x + (r + 1/2(μ-r)^2/σ^2 - m/2)T, and the entropy contribution is (mT/2)log(2πem/σ^2), so the true regularized value is ln x + (r + 1/2(μ-r)^2/σ^2)T + (mT/2)log(2πm/σ^2), which differs from Eq. (3.1) by mT/2. The Gaussian and truncated-Gaussian policy statements survive because the additive l(t) term cancels in v_x/v_xx, and the m→0 convergence is unaffected, but the closed-form value functions and the numerical learning targets in (5.4), (5.7), and Section 6.1 must be corrected.","section":"Eq. (2.12); Theorems 3.1, 4.1, 6.1; Eqs. (5.4), (5.7), (7.7)"},{"comment":"The quadratic-utility value function also has a sign error. With m=0, the constant term in Eq. (6.13) is written as -K^2/(2ε)(1-e^{-ρ^2(T-t)}), but the ODE (C.1) with m=0 gives q(t)=+K^2/(2ε)(1-e^{-ρ^2(T-t)}). For example, when r=0, x=0, and K=ε, the proposed formula gives a negative value, although the zero-investment strategy yields utility 0. In addition, the exploration term m/4(ρ^2-2r)(T-t)^2 in (6.13) has the wrong sign: substituting the stated q(t) into (C.1) leaves a residual of -m(ρ^2-2r)(T-t), not zero. After also correcting the factor-e error of (2.12), the correct temporal exploration term is -(m/4)(ρ^2-2r)(T-t)^2. The learning target θ2 = m/4(ρ^2-2r) in Section 6.1 and the numerical targets in Section 6.2 therefore have the wrong sign.","section":"Theorem 6.1, Corollary 6.2, Eq. (C.1), Section 6.1"},{"comment":"The policy improvement theorem is conditional on v^λ ∈ C^{1,2}, v^λ_xx < 0, and admissibility of the Gaussian (or truncated-Gaussian) improvement, but no conditions are established under which these hypotheses hold for the policies generated by the iteration. The authors note in footnote 3 that well-posedness of the average PDE (3.18) is open. Since Theorems 3.5 and 4.4 assert convergence of the policy-iteration sequence, this is a load-bearing gap: the convergence claims are not unconditional. The theorems should either be stated as purely conditional improvement results or be supplemented with verification results for the logarithmic and quadratic cases where the paper claims convergence.","section":"Theorem 3.4, Theorem 4.3, footnote 3"}],"minor_comments":[{"comment":"The Gaussian entropy identities contain an extra 1/2: the standard differential entropy is log(b√(2πe)) = 1/2 log(2πeb^2), whereas (5.5) gives 1/2 + 1/2(φ2 + log(2πem)), and (3.8) has the analogous discrepancy. The final exploration cost mT/2 in Proposition 3.1 is nevertheless correct because the errors cancel in that particular calculation, but the intermediate identities should be fixed.","section":"Eq. (3.8), Eq. (5.5), Appendix A.3"},{"comment":"The proof of Theorem 4.2 refers to \"ψ[a,b](t,x;m) given by (4.14)\", but the definition of ψ[a,b] appears in Lemma 4.2, not in Eq. (4.14). Please correct the cross-reference.","section":"Theorem 4.2 proof"},{"comment":"The title in the text is typeset as \"POR TFOLIO\" with a spurious space, and the paper uses π for both the portfolio weight and the mathematical constant; the footnote on πe helps, but the notation remains awkward in formulas such as (3.1) where \"2πem\" mixes both uses.","section":"Title and Notation"},{"comment":"The numerical \"true\" parameter values in Table 2 and the target θ* in Section 5.3 are computed from the uncorrected closed-form value functions and must be recomputed after the corrections described in the major comments.","section":"Section 5.3"}],"recommendation":"major_revision","confidential_remarks":"The paper has multiple independent constant and sign errors in the closed-form value functions, but the policy characterizations (Gaussian and truncated Gaussian) appear to be correct and are the most valuable part of the paper. The errors are fixable within the manuscript's scope, but they require reworking the HJB reduction, the stated value functions, the numerical targets, and the associated interpretations. The policy improvement theorems are conditional in a way that should be stated more carefully; the well-posedness caveat in footnote 3 is acknowledged but not reflected in the convergence statements."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Minx, quick take. This paper extends entropy-regularized exploratory control to continuous-time expected utility with portfolio constraints. The genuinely new piece is the constrained case: when the risky weight lies in [a,b], the optimal exploratory policy is a truncated Gaussian, and they give explicit formulas for the mean, variance, and the wealth dynamics. That is a real, useful contribution for RL portfolio work. The unconstrained Gaussian policy is an application of the Wang-Zhou framework, which they acknowledge. The numerical experiments are straightforward but support the qualitative claims: exploration widens the wealth distribution, and constraints narrow the exploration cost.\n\nThe problem is a systematic error in the HJB reduction. In (2.12) they write the log term as (m/2) ln(-2*pi*e*m / (sigma^2 x^2 v_xx)). The correct Donsker-Varadhan completion gives (m/2) ln(-2*pi*m / (sigma^2 x^2 v_xx)). The extra factor e adds m/2 to the equation, and it propagates: the value functions in Theorem 3.1, Theorem 4.1, and Theorem 6.1 are all too high by +m(T-t)/2, and the 'true' theta targets in Section 5 are computed from the wrong formula. The policy statements survive, because the Gaussian mean and variance come from the ratio v_x/v_xx and the additive l(t) term cancels. The m->0 convergence also survives. But the closed-form values in the theorems are not the values of problem (2.8), and the numerical targets in Tables 1-2 and Figure 5 are off. This is a concrete bug, not a matter of taste.\n\nThe other soft spot is Theorem 3.4. It assumes the value function of a given policy is C^{1,2} with v_xx<0 and that the constructed Gaussian policy is admissible. No conditions establish that; the authors themselves note in footnote 3 that well-posedness of the average PDE is open. That is a genuine gap in the policy improvement claim. I would not call it fatal, because the policy improvement might be provable under reasonable assumptions, but it is load-bearing for the convergence results.\n\nThe citation pattern is appropriate; the framework is credited to Wang et al. (2020) and Jia and Zhou (2022a,b). Several proofs are omitted as similar to those papers, which is acceptable for a paper that is mostly an application, though the truncated-Gaussian computations deserve more detail.\n\nBottom line: this deserves a serious referee. The constrained truncated-Gaussian result is new and the paper is well-written, but the factor-e error must be fixed and the regularity assumption either proved or weakened.","headline":"Genuine new constrained truncated-Gaussian result, but the value functions carry a systematic factor-e error that needs fixing before the closed forms can be trusted.","tokens_in":40467,"tokens_out":1983,"would_cite":false,"duration_ms":18343,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["34D20","60H10","92D25","93D05","93D20"],"pacs":[],"model":"deepseek-v4-flash","headline":"In an entropy-regularized exploratory investment problem, the optimal feedback policy is Gaussian, with mean equal to the Merton strategy and variance equal to the exploration rate divided by squared volatility, and it becomes truncated…","keywords":["optimal investment","entropy-regularized reinforcement learning","exploratory stochastic control","Gaussian policy","truncated Gaussian","portfolio constraints","policy improvement theorem","martingale policy gradient"],"falsifier":"A concrete test would be to take a non-logarithmic, non-quadratic utility such as a power utility U(x) = x^(1−γ)/(1−γ) with γ ≠ 1 and numerically solve the exploratory HJB (or run the actor-critic algorithm) in an unconstrained Black–Scholes market; if the resulting optimal policy is not Gaussian with mean (µ−r)/σ² and variance m/σ², then the Gaussian-optimality claim is specific to the two utility classes rather than general. More directly, one could search for an admissible policy λ whose value function is known to be non-smooth or for which the Gaussian improvement is not admissible, which would violate the premise of Theorem 3.4.","tokens_in":39265,"feed_emoji":"📈","tokens_out":3315,"duration_ms":32655,"temperature":0.7,"pith_summary":"This paper studies a continuous-time reinforcement learning version of expected utility maximization, where the investor samples portfolio weights from a distribution rather than committing to a single strategy. The central claim is that the optimal exploration policy for logarithmic and quadratic utility is a Gaussian, with mean given by the classical Merton fraction and variance proportional to the exploration parameter m divided by σ². When the portfolio weight is constrained to lie in [a, b], the same Gaussian is simply truncated to that interval. The paper further derives closed-form value functions, shows that they converge to the non-exploratory Merton values as m goes to zero, and provides a policy improvement theorem that underpins an implementable reinforcement learning algorithm.","feed_headline":"Optimal RL investing policy is Gaussian—truncated under constraints","feed_subtitle":"Closed-form log and quadratic utility solutions converge to Merton as exploration fades.","key_machinery":"The central object is the entropy-regularized exploratory control problem, where the agent chooses at each time a probability density λ over portfolio weights and pays a Shannon differential entropy penalty scaled by m. The optimal distribution is obtained by applying the Donsker–Varadhan variational formula to the HJB equation, which yields a Gaussian with mean α = −(µ−r)x v_x / (σ²x² v_xx) and variance −m/(σ²x² v_xx) whenever v_xx < 0. For logarithmic utility the value function is linear in ln x, making the Gaussian parameters independent of wealth, and for quadratic utility the ansatz is quadratic in x, again reducing the PDE to solvable form. In the constrained case the same variational argument gives a truncated Gaussian; its moments and entropy are computed explicitly, which directly yields the exploration cost. The martingale characterization of the value function (Theorem 3.6) and the policy gradient representation (Theorem 3.7) transform the learning problem into a parametric optimization that can be implemented with a standard actor-critic scheme using sampled wealth trajectories.","core_discovery":"For the entropy-regularized expected utility problem in a Black–Scholes market, the paper establishes that the optimal feedback control distribution is Gaussian for logarithmic utility (Theorem 3.1) and for quadratic utility (Corollary 6.1), with mean equal to the Merton proportion (µ−r)/σ² and variance m/σ². If the portfolio weight is restricted to an interval [a, b], the optimal policy becomes the same Gaussian truncated to that interval (Theorem 4.1). The associated optimal value functions are obtained in closed form, and the paper proves that as the exploration weight m tends to zero, the optimal policies converge to Dirac measures at the corresponding constrained or unconstrained Merton strategies and the value functions converge to the classical expected utility values. A policy improvement theorem (Theorems 3.4 and 4.3) guarantees that iterating the Gaussian (or truncated Gaussian) improvement step monotonically improves the value function, and the martingale-based policy evaluation and gradient formulas yield a concrete actor-critic algorithm.","pith_inferences":["The closed-form Gaussian structure likely extends beyond the two utility classes only under strong regularity conditions; for general utilities the optimal exploratory policy would still be Gaussian if the value function is smooth and concave, but verifying those conditions is exactly the open problem the paper identifies for the average PDE.","The result that the constrained exploration cost is smaller than the unconstrained one suggests a broader design principle: deliberate exploration is cheaper when the admissible action set is bounded, which could inform how regulators or firms choose portfolio restrictions in learning environments.","The convergence as m→0 is a local statement; one could test whether for fixed m the truncated Gaussian remains optimal for other utility functions such as power utility with relative risk aversion not equal to one, but the paper does not address that case.","The martingale policy evaluation scheme is likely applicable to other continuous-time stochastic control problems with entropy regularization, especially where the value function can be parametrized in a closed family, but the paper only demonstrates it for logarithmic and quadratic utilities."],"forward_implications":["If the central claim is correct, then for logarithmic and quadratic utility the full trade-off between exploration and exploitation is described by a single scalar parameter m, and the optimal policy is always a Gaussian (or truncated Gaussian) centered at the Merton fraction.","In the unconstrained case the exploration cost is exactly mT/2, and in the constrained case it is smaller whenever the Merton fraction lies inside the constraint interval, confirming that restricting the action space lowers the cost of exploration.","As m goes to zero, the learned policies and value functions recover the classical Merton solution, so any small-exploration RL implementation is guaranteed to approach the standard optimal investment strategy.","The policy improvement theorem implies that starting from any admissible—not necessarily Gaussian—policy, a single Gaussian update improves the value, and repeated updates converge to the optimal Gaussian policy; this provides a theoretical basis for restricting attention to Gaussian policies in practice.","For quadratic utility, the exploratory optimal portfolio remains on the mean-variance efficient frontier under a mild condition, which connects the RL solution to classical Markowitz analysis."],"supporting_citations":[{"why":"Provides the exploratory formulation of continuous-time RL with a Shannon entropy penalty, which is the framework adopted throughout the paper.","marker":"Wang et al. (2020)"},{"why":"In the mean-variance setting it was shown that the optimal exploratory policy is Gaussian; the present paper extends that observation to expected utility and proves the Gaussian is truncated under constraints.","marker":"Wang and Zhou (2020)"},{"why":"Supplies the martingale approach to policy evaluation and policy gradient that the paper adapts to obtain its implementable actor-critic algorithm.","marker":"Jia and Zhou (2022b)"},{"why":"Gives the variational formula used to derive the optimal Boltzmann–Gaussian distribution in the HJB step.","marker":"Donsker and Varadhan (2006)"},{"why":"Addresses the well-posedness and convergence of exploratory HJB equations, which is the related theoretical foundation for the policy improvement and convergence results.","marker":"Tang et al. (2022)"}],"fun_headline_variants":["Exploratory RL portfolio policy: Gaussian, truncated under constraints","RL optimal investing: Gaussian policy, truncated by portfolio limits","Entropy-regularized RL: Gaussian optimal policy with constrained truncation","As exploration fades, RL investing policy converges to Merton"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The policy improvement theorem assumes that for any admissible policy the corresponding value function is smooth with a negative second derivative and that the Gaussian (or truncated Gaussian) policy built from that value function is admissible, but the paper does not establish general conditions under which the average PDE admits such a classical solution.","fun_headline_variants_meta":{"raw":{"variants":["Exploratory RL portfolio policy: Gaussian, truncated under constraints","RL optimal investing: Gaussian policy, truncated by portfolio limits","Entropy-regularized RL: Gaussian optimal policy with constrained truncation","As exploration fades, RL investing policy converges to Merton"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000689,"raw_usage":{"total_tokens":3130,"prompt_tokens":965,"completion_tokens":2165,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":2093}},"tokens_in":581,"tokens_out":2165,"duration_ms":15769,"temperature":1.0,"reasoning_tokens":2093,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:44:11.403317+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test would be to take a non-logarithmic, non-quadratic utility such as a power utility U(x) = x^(1−γ)/(1−γ) with γ ≠ 1 and numerically solve the exploratory HJB (or run the actor-critic algorithm) in an unconstrained Black–Scholes market; if the resulting optimal policy is not Gaussian with mean (µ−r)/σ² and variance m/σ², then the Gaussian-optimality claim is specific to the two utility classes rather than general. More directly, one could search for an admissible policy λ whose value function is known to be non-smooth or for which the Gaussian improvement is not admissible, which would violate the premise of Theorem 3.4.","supporting_citations":[{"cited_title":"and Varadhan, S","cited_arxiv_id":null,"evidence_quote":"Gives the variational formula used to derive the optimal Boltzmann–Gaussian distribution in the HJB step."},{"cited_title":"P., and Zhou, X","cited_arxiv_id":null,"evidence_quote":"Addresses the well-posedness and convergence of exploratory HJB equations, which is the related theoretical foundation for the policy improvement and convergence results."}],"review_version":1}