{"id":"091d4675-bc8f-4d31-bcb7-a28922f8fc04","arxiv_id":"2507.04396","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A three-chapter monograph that uses Afriat's theorem and Bayesian revealed preference tests for inverse reinforcement learning, plus a passive Langevin dynamics algorithm for real-time reward reconstruction.","lead":"This monograph applies revealed preference theory from economics to inverse reinforcement learning, showing how to test and reconstruct utility functions from observed actions, including for cognitive radar and Bayesian stopping problems. It also proposes a passive Langevin dynamics algorithm that reconstructs a reward function in real time from noisy gradients observed from reinforcement learning agents.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gibbs-measure claim (3.4) depends on RL agents resetting after bounded runs; the formal averaging proof is cited to [KY21] and not in the manuscript.","rationale":"The reader and I converge on the same load-bearing point: the Ch. 3 central claim is not self-contained. My refinement is that the failure mode is not generic 'slow mixing' but the specific bounded-run/reset structure that makes theta nearly pi-distributed. The informal derivation (Eq. 3.11) averages over theta ~ pi; in model (3.1) this approximation is controlled by tau_{n+1}-tau_n < M and by epsilon = beta*mu -> 0. If an agent runs long enough to approach a local maximum, the effective distribution of theta becomes the invariant measure of noisy gradient ascent rather than pi, and the kernel-weighted drift no longer converges to pi(alpha) grad R(alpha). Thus the nonparametric reconstruction of R from the log-density of IRL samples is only guaranteed in a regime where the RL agents do not actually learn. This does not contradict the reader's verdict. Chapters 1 and 2 are well supported: Afriat's theorem is stated with a proof sketch, Theorem 2.1 has its proof in Appendix 2.B, and the numerical illustrations are plausible. The conditional verdict remains appropriate; no change is needed.","tokens_in":61800,"tokens_out":21228,"duration_ms":230541,"concrete_test":"Re-run the Sec. 3.4.1 example with all settings fixed except the per-agent RL run length: set tau_{n+1}-tau_n = 1e5 instead of 100, so each agent approaches a local maximum before reset. If the empirical density of {alpha_k} from IRL algorithm (3.2) no longer matches exp(beta R) at the reported variational distance (about 0.01-0.03), the bounded-run assumption is load-bearing. In parallel, inspect Theorem 3.1 in Sec. 3.5 or [KY21]: it must explicitly assume and use tau_{n+1}-tau_n < M; if it instead assumes ergodic mixing of the RL process, the proof does not apply to the stated model (3.1).","verdict_should_be":"UNCHANGED","load_bearing_attack":"Chapter 3's central claim (3.4) rests on the stochastic-averaging step in Sec. 3.3.1 (Eq. 3.11), which replaces the kernel-weighted noisy gradient at theta_k by E_{theta~pi}[K((theta-alpha)/Delta) grad R(theta)]. For this replacement to be valid, theta_k must be a fast variable whose empirical distribution is pi. In the model (3.1), this is not a mixing property of the RL dynamics; it is enforced solely by the explicit assumption tau_{n+1}-tau_n < M, i.e., each RL agent runs a bounded number of iterations and is reinitialized from pi. Since epsilon = beta*mu -> 0, theta_k stays within O(epsilon) of its initial draw, so the average over theta is approximately an average over pi only under this bounded-run assumption. The informal statement (3.4) does not state the bounded-run condition, and the formal theorem in Sec. 3.5 is not included in the manuscript, being cited to the author's own paper [KY21]. If the RL agents instead run until convergence (M large relative to 1/epsilon), the observed theta's concentrate near local maxima of R, the averaged drift is no longer the one in (3.12), and the stationary measure of the IRL algorithm is not exp(beta R). The numerics in Sec. 3.4.1 use only 100 iterations per agent with epsilon = 1e-3, so they do not probe the failure regime.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript is a three-chapter monograph on inverse reinforcement learning (IRL). Chapter 1 recasts IRL as revealed-preference analysis: it restates Afriat's theorem, extends it to nonlinear budgets, applies it to cognitive-radar waveform and beam allocation, and develops statistical detection and utility-masking procedures. Chapter 2 develops Bayesian IRL for rationally inattentive agents, presenting NIAS/NIAC feasibility tests for one-step decisions and for Bayesian stopping-time problems, with applications to inverse sequential detection, search, and quickest detection. Chapter 3 proposes adaptive IRL by passive Langevin dynamics: observing noisy gradients produced by multiple stochastic-gradient RL agents, the IRL algorithm (3.2) is claimed to generate samples asymptotically from the Gibbs measure p(alpha) proportional to exp(beta R(alpha)), so that R can be reconstructed nonparametrically from the empirical density of the samples. The chapter also presents multi-kernel, active, and non-reversible variants and gives numerical comparisons against classical Langevin dynamics and Metropolis-Hastings.","tokens_in":62124,"tokens_out":11924,"duration_ms":120776,"significance":"If the main convergence claim (3.4) holds, the paper offers a genuinely new capability: nonparametric reconstruction of a reward function from passively observed, misspecified noisy gradients, in real time and with a constant step size that permits tracking of time-varying rewards. This would be useful in adversarial controlled sensing, adaptive Bayesian learning, and inverse stochastic control. The first two chapters are careful, largely correct restatements of published revealed-preference results, and the manuscript is explicit about the set-valued and ordinal nature of reconstructed utilities, which is a strength. The numerical work in Chapter 3 is honest in comparing against classical Langevin sampling and Metropolis-Hastings, and reports quantitative variational distances. The central contribution, however, is conditional on a formal weak-convergence theorem that is not present in the submitted text: the proof is deferred to Section 3.5, which is not included, and is cited to the author's own prior work [KY21]. The load-bearing stochastic-averaging assumption is stated only informally and is not reflected in the main claim as written.","major_comments":[{"comment":"The Gibbs-measure claim (3.4) rests on the stochastic-averaging step in Eq. (3.11), which replaces the kernel-weighted noisy gradient at theta_k by its expectation over theta~pi. This replacement is valid only if the RL iterates theta_k are a fast variable with stationary distribution pi. The model (3.1) does not provide such mixing: with epsilon = beta mu -> 0, a bounded-length run keeps theta_k within O(epsilon) of its pi-initialization, whereas an unbounded run drives theta_k toward local maxima of R. The bounded-run condition tau_{n+1}-tau_n < M is introduced only in the informal proof (Sec. 3.3.1, item 2) and is absent from the statement of (3.4) and from the algorithm description. The formal theorem in Sec. 3.5 is not included in the manuscript and is cited to [KY21]. The formal statement must be included and must state the bounded-run or reset conditions; otherwise (3.4) is not a theorem about algorithm (3.2) with the RL protocol (3.1) as described.","section":"Sec. 3.3.1, Eqs. (3.11)-(3.13), and claim (3.4)"},{"comment":"The numerical validation of (3.4) operates entirely in the favorable regime of the bounded-run assumption: each RL agent runs only 100 iterations with epsilon = 10^{-3}, so theta_k remains close to its initialization and the empirical distribution of theta is close to pi. This experiment does not probe the regime where the RL agents run until convergence, which is precisely the regime where the averaging in (3.11) fails and the stationary measure of the IRL algorithm is not exp(beta R). Please add experiments with large or unbounded run lengths, or alternatively restrict the formal claim explicitly to the bounded-run case, since this assumption is load-bearing for the central result.","section":"Sec. 3.4.1"},{"comment":"The multi-kernel IRL algorithm (3.20) is claimed to converge to the same Gibbs measure, with the formal proof again cited to [KY21] rather than included. The informal argument uses the self-normalized importance-sampling limit (3.23), which requires L -> infinity for fixed alpha, followed by a small-variance Bernstein-von Mises limit so that the conditional density acts as a Dirac measure. The relative rates of L, mu, and sigma^2 are not specified, so the order of limits in (3.24)-(3.25) is not justified in the manuscript. If the multi-kernel algorithm is advertised as a contribution, the missing theorem and rate conditions need to be supplied.","section":"Sec. 3.3.2, Eq. (3.20)"}],"minor_comments":[{"comment":"The parameters state epsilon = 10^{-3} and mu = 5 x 10^{-4}, which gives beta = epsilon/mu = 2, but the text says 'beta = 1'; please correct this inconsistency or state the actual values used in the simulations.","section":"Sec. 3.4.1"},{"comment":"The observer's reconstruction of the gradient as (theta_{k+1} - theta_k)/mu yields beta grad r_k(theta_k), not grad r_k(theta_k), since theta_{k+1} - theta_k = epsilon grad r_k(theta_k); although the scale factor beta is later absorbed, the presentation should make this explicit so the reader can see how the algorithm is implemented without knowing epsilon.","section":"Sec. 3.1, after Eq. (3.1)"},{"comment":"The informal proof takes the double limit mu -> 0 for fixed Delta and then Delta -> 0; the claim (3.4) should state this order of limits explicitly, since for any fixed Delta > 0 the stationary distribution of the averaged diffusion (3.12) is not exactly exp(beta R).","section":"Sec. 3.3.1, Eqs. (3.12)-(3.13)"},{"comment":"The table of contents lists Sec. 3.5 (Weak Convergence Analysis) and Sec. 3.6 (Tracking Analysis), but the submitted text breaks off inside Sec. 3.4.2; these sections contain the formal statements that underlie the main claims of Chapter 3 and must be supplied.","section":"Contents and Chapter 3"},{"comment":"The proof uses the fact that the complementary CDF ar F_M(M) is uniform in [0,1]; this holds only when the random variable M has a continuous distribution, and this regularity condition on the noise sequence should be stated explicitly.","section":"Sec. 1.8, Theorem 1.11"},{"comment":"The Summary states that the monograph is work in progress and that Chapter 3 draws heavily on [KY21]; for a journal submission, the relationship to [KY21] should be clarified so that the reader can identify which results are new in this manuscript.","section":"Summary and Sec. 3.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads as a work-in-progress monograph rather than a self-contained journal article. Chapters 1 and 2 are largely restatements of published Afriat and NIAS/NIAC results with radar applications, and the main new claim in Chapter 3 depends on a formal theorem that is cited to the author's own prior work [KY21] and to a missing Section 3.5. If the journal is open to monograph-style submissions, the main fix is to include the complete formal statement and proof, or at least a detailed statement with all assumptions. The editor may also wish to consider whether the first two chapters should be condensed, since the paper's new contribution is concentrated in Chapter 3."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nThe genuinely new piece here is Chapter 3: a passive Langevin algorithm that reconstructs a reward function from noisy gradients observed from RL agents, plus a multi-kernel variance-reduced variant. Chapters 1 and 2 are a clean, correct restatement of Afriat’s theorem and the Caplin–Dean NIAS/NIAC machinery, applied to cognitive radar and Bayesian stopping problems. That material is not new, but it is well organized and useful.\n\nThe passive Langevin idea is worth attention. The algorithm (3.2) uses a kernel to weight gradients evaluated at θ_k when the IRL iterate is α_k, and the heuristic in Sec. 3.3.1 shows why the stationary measure should be exp(βR(α)). The numerics comparing against classical Langevin and Metropolis–Hastings are reassuring, and the multi-kernel version (3.20) that avoids knowing the initialization density π is a genuine practical addition.\n\nThe soft spot is exactly where the reader put it: the central claim (3.4) is not proved in this manuscript. The informal proof does state the bounded-run assumption τ_{n+1}−τ_n < M, which is the load-bearing condition for the stochastic averaging step, but the actual theorem is said to be in Sec. 3.5 and then cited to the author’s earlier paper [KY21]. A reader cannot verify the claim from the monograph alone. The numerics also use short runs (100 iterations, ε = 1e−3), so they stay close to the initialization and do not probe the regime where the bounded-run condition fails. I don’t read the self-citation as a problem — the proof belongs in the earlier paper — but it does make this document non-self-contained on its main new result.\n\nIf the formal proof in [KY21] is sound, this is a solid contribution to IRL and controlled sensing. Who gets value: researchers working on inverse reinforcement learning from passive observations, and anyone wanting a compact summary of revealed-preference tests with radar examples. It deserves a serious referee; the referee should insist that the main theorem be stated in the paper with its hypotheses, or at least that the bounded-run condition be explicit in the statement of (3.4).","headline":"The passive Langevin IRL algorithm in Chapter 3 is genuinely new and worth engaging, but its main convergence claim is cited to the author's prior paper rather than proved here; Chapters 1-2 are a clean restatement of known revealed-preference results.","tokens_in":62652,"tokens_out":2878,"would_cite":true,"duration_ms":28759,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An inverse learner can recover a hidden reward function just by watching a learner's noisy gradient updates.","keywords":["inverse reinforcement learning","revealed preferences","Afriat's theorem","passive Langevin dynamics","stochastic gradient algorithms","Bayesian IRL","cognitive radar","Gibbs measure"],"falsifier":"For a reward $R(\\theta)=-\\frac12\\|\\theta\\|^2$ the claimed stationary measure is a Gaussian $\\mathcal{N}(0,I/\\beta)$; run algorithm (3.2) while several learners run gradient ascent on this reward, collect a long sample path $\\{\\alpha_k\\}$, and test whether the empirical log-density is quadratic with the predicted curvature $-(\\beta/2)\\|\\alpha\\|^2$ plus a constant. A persistent mismatch that does not shrink as $\\mu$ and $\\Delta$ go to zero would refute the Gibbs-measure claim.","tokens_in":61566,"feed_emoji":"🎯","tokens_out":13036,"duration_ms":124105,"temperature":0.7,"pith_summary":"Inverse reinforcement learning (IRL) aims to recover an agent's reward function from its observed decisions; this monograph argues that two passive, nonparametric routes can do it. The first route, taken from revealed-preference theory, checks whether a finite dataset of probe–response pairs could have been produced by a constrained utility maximizer, and if so returns the whole family of utility functions that rationalize the data, not a single point estimate. The second route is an adaptive algorithm that runs while reinforcement learners perform stochastic gradient ascent on a possibly time-varying reward: an inverse learner passively observes their noisy gradient evaluations and runs a kernel-weighted Langevin recursion whose samples asymptotically follow the Gibbs measure $p(\\alpha)\\propto \\exp(\\beta R(\\alpha))$. Since the reward is then the logarithm of the sample density (up to the scale factor $\\beta$), the hidden reward is reconstructed in real time without the inverse learner ever choosing where gradients are evaluated. The same machinery is applied to detect cognitive radar, to test whether a sensor is a Bayes-optimal sequential detector, and to mask a utility from an adversary's IRL.","feed_headline":"Passive Langevin algorithm recovers rewards from noisy gradients","feed_subtitle":"Watching a learner's noisy gradient stream yields samples from the reward's Gibbs measure; the log-density is the reward.","key_machinery":"The load-bearing identity is the reversible-diffusion stationary measure: a diffusion with drift $(\\beta/2)\\pi(\\alpha)\\nabla R(\\alpha)+\\nabla\\pi(\\alpha)$ multiplied by $\\pi(\\alpha)$ and unit diffusion coefficient has stationary density $p(\\alpha)\\propto \\exp(\\beta R(\\alpha))$. The paper's algorithm (3.2) is an Euler–Maruyama discretization of this diffusion, made passive by inserting the kernel $K_\\Delta(\\theta-\\alpha)$ so that a noisy gradient evaluated at the wrong point $\\theta_k$ contributes to the update at $\\alpha_k$ with weight proportional to its proximity. Two averaging steps then complete the mechanism: stochastic averaging replaces the fast RL iterates by their stationary distribution $\\pi$, and the kernel-width limit $\\Delta\\to 0$ turns the kernel into a Dirac delta, delivering the reversible diffusion above. On the revealed-preference side, the corresponding identity is Afriat's theorem—GARP (1.4) is equivalent to feasibility of the linear inequalities (1.2), whose solutions parametrize the concave piecewise-linear utilities (1.3) that rationalize the dataset—and its Bayesian analogue NIAS/NIAC (2.11)–(2.12) for rationally inattentive agents.","core_discovery":"The central claim of Chapter 3 is that the passive Langevin IRL algorithm (3.2) asymptotically generates samples $\\{\\alpha_k\\}$ from the Gibbs measure $p(\\alpha)\\propto \\exp(\\beta R(\\alpha))$ with $\\beta=\\varepsilon/\\mu$, where $\\varepsilon$ is the step size of the reinforcement learners' gradient algorithm (3.1) and $\\mu$ is the step size of the inverse learner's recursion. The recursion weights each observed noisy gradient $\\nabla_\\theta r_k(\\theta_k)$ by a kernel $K_\\Delta(\\theta_k-\\alpha_k)$ that decays with the distance between the learner's evaluation point and the inverse learner's current estimate, incorporates the learners' initialization density $\\pi$, and injects Gaussian noise; under stochastic averaging and the $\\Delta\\to 0$ kernel limit, the interpolated trajectory converges weakly to a reversible diffusion whose stationary measure is exactly $\\exp(\\beta R)$. Consequently the expected reward $R(\\theta)=\\mathbb{E}\\{r_k(\\theta)\\}$ can be recovered nonparametrically as the logarithm of the empirical density of $\\{\\alpha_k\\}$. Chapters 1 and 2 make the complementary claim that a dataset of probes and responses is consistent with (Bayesian) utility maximization if and only if a set of linear inequalities—Afriat's inequalities (1.2) for the classical case, NIAS and NIAC (2.11)–(2.12) for rationally inattentive Bayesian agents—is feasible, and every feasible solution defines a set of utility functions that rationalize the data.","pith_inferences":["If the passive-Langevin claim extends to arbitrary stochastic-gradient logs, reward models could be extracted from recorded SGD training histories of deep networks, a setting the paper does not discuss but whose ingredients (random evaluation points, noisy gradients) match its assumptions.","The multikernel variant (3.20) is offered as a high-dimensional remedy; a natural follow-up is to measure empirically how its self-normalized importance-weighting bias trades off against variance reduction, and how the optimal pool size $L$ scales with dimension $N$.","The paper's Type-I error bound for the noisy-detection test suggests that the choice of probe vectors in active IRL could be formulated as a Bayesian design problem; the paper optimizes probes via SPSA, but a closed-form characterization of the optimal probe remains open.","The tracking analysis predicts that when the environment changes at the same rate as the IRL updates ($\\eta=O(\\mu)$), the reconstructed reward is a mixture over recent rewards; an explicit simulation test of this mixing behaviour would cleanly separate the paper's contribution from classical stochastic-gradient tracking."],"forward_implications":["The inverse learner can reconstruct the entire reward function from passively observed gradient streams, without ever specifying where gradients should be evaluated; the log-empirical-density estimator recovers $R$ up to the scale $\\beta$.","Revealed-preference IRL yields set-valued utility estimates: the feasible polytope of Afriat's inequalities (or NIAS/NIAC) contains every utility that rationalizes the dataset, so predictions for future probes come as intervals or confidence regions rather than single points.","The same linear-inequality test applies to Bayesian stopping-time problems, so an analyst can decide whether a sequential detector's decisions are Bayes-optimal and reconstruct its stopping and continue costs from observed action frequencies alone.","Because the inverse recursion uses a constant step size, it can track time-varying rewards; when the reward jumps according to a Markov chain, the interpolated IRL trajectory converges to a Markov-switched diffusion whose limit behavior is analyzed in three time-scale regimes.","A cognitive sensor that knows its own utility can deliberately shrink the feasibility margin of Afriat's test (by solving (1.27)), hiding its cognition from an adversary's IRL at a small performance cost."],"supporting_citations":[{"why":"Supplies Afriat's theorem, the necessary-and-sufficient GARP test that underlies Chapter 1's set-valued utility reconstruction.","marker":"[Afr67]"},{"why":"Extends Afriat's theorem to nonlinear budget constraints, used for cognitive radar and multiple simultaneous constraints.","marker":"[FM09]"},{"why":"Provides the Bayesian revealed-preference (NIAS/NIAC) conditions that Chapter 2 uses for rationally inattentive agents and stopping-time problems.","marker":"[CD15]"},{"why":"The classical IRL linear-inequality formulation that the revealed-preference approach generalizes and that motivates the passive-gradient setting.","marker":"[NR00]"},{"why":"Gives the reversible-diffusion framework with Gibbs stationary measure that the passive Langevin algorithm (3.2) is designed to emulate.","marker":"[ST99]"},{"why":"Cited as the formal weak-convergence analysis of the passive Langevin IRL algorithm, the proof of the main result (3.4).","marker":"[KY21]"}],"fun_headline_variants":["Passive Langevin IRL extracts rewards from noisy gradients","Noisy gradient stream yields Gibbs measure for reward","Adaptive IRL tracks utility via passive stochastic optimization","Revealed preferences detect utility maximizers from actions","Bayesian IRL uses Afriat and NIAS for rational agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the points where the reinforcement learners evaluate their gradients move around on a much faster time scale than the inverse learner's own estimate updates, so the kernel-weighted noisy gradient can be replaced by its expectation over the learners' typical behavior; if the learners evolve too slowly, the samples generated by the inverse algorithm will not follow $\\exp(\\beta R)$.","fun_headline_variants_meta":{"raw":{"variants":["Passive Langevin IRL extracts rewards from noisy gradients","Noisy gradient stream yields Gibbs measure for reward","Adaptive IRL tracks utility via passive stochastic optimization","Revealed preferences detect utility maximizers from actions","Bayesian IRL uses Afriat and NIAS for rational agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000359,"raw_usage":{"total_tokens":2054,"prompt_tokens":1169,"completion_tokens":885,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":785,"completion_tokens_details":{"reasoning_tokens":805}},"tokens_in":785,"tokens_out":885,"duration_ms":10289,"temperature":1.0,"reasoning_tokens":805,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:48:03.765687+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For a reward $R(\\theta)=-\\frac12\\|\\theta\\|^2$ the claimed stationary measure is a Gaussian $\\mathcal{N}(0,I/\\beta)$; run algorithm (3.2) while several learners run gradient ascent on this reward, collect a long sample path $\\{\\alpha_k\\}$, and test whether the empirical log-density is quadratic with the predicted curvature $-(\\beta/2)\\|\\alpha\\|^2$ plus a constant. A persistent mismatch that does not shrink as $\\mu$ and $\\Delta$ go to zero would refute the Gibbs-measure claim.","supporting_citations":[],"review_version":1}