REVIEW 6 major objections 6 minor 5 references
Dynamic Decision-Making under Model Misspecification
T0 review · 6 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Under model misspecification, Thompson Sampling still drives the posterior exponentially toward a pseudo-truth set, and average regret per period remains bounded by a constant.
desk verdict A promising pseudo-truth set framing for Thompson Sampling under misspecification, but the main theorem is unproven due to load-bearing KL-orientation and boundedness errors. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the pseudo-truth set $\Theta^\dagger$, defined in Assumption 2 as the parameters that cannot be uniformly dominated in KL divergence from the true data-generating distribution, together with the decomposition of the log-likelihood-ratio process $Z_t^\gamma = \sum_{s=0}^t \log \Lambda_s^\gamma$ into a martingale and a predictable drift. The posterior on the pseudo-truth set is rewritten as $1/(1 + \sum_{\gamma \notin \Theta^\dagger} c_\gamma \exp(-Z_t^\gamma))$, and the proof bounds the martingale by Azuma-Hoeffding while Assumption 2 provides the positive drift $\epsilon$ that makes the exponential decay rate. The overshadowing graph, with edges $\theta \to \gamma$ when $\gamma$ fits better on $\theta$'s preferred action, is the structure that decides which subsets of the pseudo-truth set can persist as pathwise attractors.
What would settle it
A concrete falsifier is to take the paper's Example 2, where the true reward function is piecewise linear and the model is quadratic, and compute $\inf_{a \in A} [K(v^a_\theta | v_*) - K(v^a_{\theta^\dagger} | v_*)]$ for a parameter $\theta$ whose quadratic curve agrees with the true function on an interval; if that infimum is zero, Assumption 2 fails, and the predicted exponential concentration would have to be tested directly by running Thompson Sampling and measuring whether $\pi_t(\Theta^\dagger)$ indeed converges to 1 at an exponential rate.
Extended reading notes
Core claim
The central discovery is an exponential-concentration theorem for Thompson Sampling under misspecification. Theorem 1 states that, under the paper's Assumptions 1 and 2, there exist constants $a_{\Theta^\dagger}, b_{\Theta^\dagger} > 0$ such that $E_{\theta^*}[1 - \pi_t(\Theta^\dagger)] \leq a_{\Theta^\dagger} e^{-b_{\Theta^\dagger} t}$; Corollary 1 then bounds average per-period regret by an absolute constant, making total regret linear in $T$. The companion characterization says that a subset $S \subset \Theta^\dagger$ can sustain posterior mass in the long run only if it is jointly closed under overshadowing and strongly connected in the graph of likelihood-ratio domination. The simulations illustrate the consequence: the MAP estimate of the parameters does not converge to a single point under misspecification, multiple limits can coexist, and yet the algorithm's average regret stays bounded at a level comparable to the correctly specified setting.
Load-bearing premise
The load-bearing premise is Assumption 2, which says that for every parameter outside the pseudo-truth set there is a strictly positive gap in KL divergence from the true distribution that holds uniformly over all actions and covariate values, because the proof uses exactly that gap as the drift that forces exponential posterior decay.
Editorial extensions
If this is right
- Posterior mass on the pseudo-truth set grows exponentially fast regardless of the action sequence chosen, so exploration noise does not slow the learning rate under misspecification.
- The per-period regret is bounded by a constant, meaning total regret grows linearly; misspecified models cannot achieve the sublinear regret of correctly specified bandits unless the pseudo-truth set is a single point.
- Individual parameter recovery can fail completely inside the pseudo-truth set: the MAP estimate can cycle among multiple limits, and only jointly closed, strongly connected subsets can attract posterior mass pathwise.
- The necessary conditions give a diagnostic recipe: inspect the overshadowing graph and check for jointly closed, strongly connected components to predict which parameters will survive in the long run.
Reading between the lines
- The paper's wording calls Assumption 2 'mild,' but the uniform KL gap over the entire action space is a strong condition; in settings like the paper's own Example 2 where infinitely many parameters agree with the true function at a fixed point, the uniform gap can vanish, so the exponential-rate result may not cover those examples as stated.
- A testable extension is to replace the uniform gap in Assumption 2 by a gap that holds only for the actions the algorithm actually plays; this would connect the concentration rate to the exploration schedule and to existing notions of misspecified Bayesian learning such as Berk-Nash equilibrium.
- If the constant per-period regret holds in applied settings, it suggests reporting a set of pseudo-true parameters with posterior weights, rather than a point estimate, because the MAP can be non-convergent while the policy remains nearly optimal.
- One can use the overshadowing graph as an empirical diagnostic: estimate KL gaps between candidate parameters from data, draw the graph, and check which subsets are jointly closed and strongly connected to anticipate the set of possible posterior limits before running the algorithm.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies Thompson Sampling in a finite-parameter dynamic decision problem when the reward model is misspecified. The main theoretical claim (Theorem 1, Eq. 12) is that, under Assumptions 1 and 2, the posterior probability assigned to a 'pseudo-truth set' Θ† converges to 1 exponentially fast at rate e^{-bt}, regardless of the actions chosen; Corollary 1 then asserts constant per-period regret. Section 5 develops necessary conditions for further posterior concentration inside Θ† using an overshadowing graph and a notion of jointly closed, strongly connected subsets. The paper also presents two numerical examples with quadratic and piecewise reward functions, reporting that MAP estimates do not converge while average regret remains relatively stable.
Significance. If the main theorem were correct, the paper would make a useful conceptual contribution by extending White's (1982) pseudo-true parameter idea to dynamic decision problems and by showing that Thompson Sampling can still be robust under misspecification. The graphical language of 'overshadowing' and 'pseudo-truth set' is intuitive and could inform future work. However, the central proof contains load-bearing errors: the drift calculation uses the wrong expectation, the bounded-increment lemma fails for the paper's own Gaussian examples, and the final concentration inequality has inconsistent time exponents. The constant-regret corollary is asserted without proof, and the Section 5 characterization is partly tautological. Because these issues affect the main theorem, the paper's central claims are not established as they stand.
major comments (6)
- [Appendix, Lemma 2 (Eq. 21)] The claimed drift lower bound is not implied by Assumption 2, and the KL orientation is wrong. The increment of the log-likelihood ratio in Eq. (16)-(18) is E_{θ*}[log Λ_s^γ | H_{s-1}] = E_{θ*}[log L_{Θ†}(R_s | A_s, X_s) - log L_γ(R_s | A_s, X_s) | H_{s-1}], with the expectation taken under the true DGP. Lemma 2 instead rewrites this as K(v_{Θ†} | v_γ) = E_{v_{Θ†}}[log(v_{Θ†}/v_γ)], i.e. an expectation under the mixture v_{Θ†}, which is neither the true-data expectation nor generally nonnegative under misspecification. Assumption 2 (Eq. 11) bounds a difference of KL divergences of the form K(v_θ^a | v_*) - K(v_{θ†}^a | v_*), and the paper never shows how this quantity controls E_{θ*}[log(L_{Θ†}/L_γ)]. Consequently the step I_t^γ ≥ ϵ t is unproven, and Theorem 1 (Eq. 12) does not follow.
- [Appendix, Lemma 1 (Eq. 19)] The bounded-increment condition fails for the paper's own Gaussian examples. When f_θ is Gaussian, log Λ_s^γ is an affine function of the observed reward R_s; since R_s is unbounded (normal noise), no finite constant d can satisfy |log Λ_s^γ - E[log Λ_s^γ | H_{s-1}]| ≤ d almost surely. The proof's appeal to compactness of the action space A is insufficient, because the likelihood is evaluated at the reward R_s, not merely at A_s. Therefore the Azuma-Hoeffding tail bound in Eq. (23) cannot be applied, and the exponential rate in Eq. (25) is unsupported.
- [Eqs. (24)-(25)] The final concentration inequality contains inconsistent time exponents. The denominator in Eq. (24) has exp(-(ε-δ)t²), whereas the preceding display and the final bound use exp(-(ε-δ)t); the numerator in Eq. (25) mixes exp(-εt/2) and exp(-ε²t²/(8d²)). These expressions are not algebraically equivalent for t ≥ 1, and the proof does not establish the stated a_{Θ†} e^{-b_{Θ†} t} bound. The drift and martingale tail terms need to be handled with the same time exponent.
- [Corollary 1 (Eq. 14)] The constant-average-regret claim is asserted without a proof and does not follow from Theorem 1. Posterior concentration on Θ† does not by itself bound regret: parameters in Θ† need not select the oracle action, and under misspecification the oracle is ambiguous because the regret definition in Section 2 compares policies under θ rather than under the true θ*. A proof must show that actions induced by Θ† are near-optimal in expectation under the true DGP; no such argument appears anywhere in the paper.
- [Assumption 2 (Eq. 11) and Section 3 examples] Assumption 2 is a strong uniform-gap condition and the paper never verifies it for the numerical examples. In Example 2 (Eq. 9), the paper acknowledges in Remark 2 that infinitely many parameter vectors agree at a fixed point; for actions near that point, the required uniform KL gap ϵ(θ) > 0 can vanish, so the condition may fail. Since Assumption 2 is the engine of Theorem 1, the simulations do not demonstrate the theory, and the scope of applicability of the main result is unclear.
- [Section 5.5, Proposition 2 and Definition 6] The paper's characterization of pathwise collective concentration is largely tautological. Definition 6 defines 'jointly closed under overshadowing' as the existence of a stable posterior distribution p supported on S such that every outsider γ has non-positive expected log-likelihood ratio against the mixture p on every action; Proposition 2 then states, as a necessary condition, that S is jointly closed in precisely this sense. No proof is given that pathwise concentration implies the existence of such a stable p, and the sketch only rephrases the definition. The second condition, internal cohesion, is also asserted without a formal argument.
minor comments (6)
- [Throughout] The manuscript contains numerous typos and grammatical errors ('psuedo', 'convergnce', 'algoirthms', 'exisits', 'expectaion'), which should be corrected in any revision.
- [Theorem 1 statement] The theorem statement writes 1 - π(Θ†) without a time subscript, while the proof and the rest of the paper use π_t(Θ†); the notation should be made consistent.
- [Section 5.2] The measure of fit Δ(θ, a) is introduced with 'for example' and is never formally defined, yet Proposition 1 and the overshadowing definitions rely on its properties. A precise definition is needed.
- [Section 2 and Assumption 2] The paper inconsistently uses Θ, P, and Θ† for the parameter space and pseudo-truth set; for example, Assumption 2 refers to θ ∉ Θ† but the surrounding text uses P, and Θ† is described as a subset of P even though the model was defined with finite Θ. The notation should be unified.
- [References] Several references are incomplete or missing details, e.g. 'Andrews et al. (2023)' and 'Müller and Norets (2024)' lack publication venues or full information; the reference list should be checked against a standard style.
- [Section 3 simulations] The simulation section does not report the number of repetitions, parameter grid, or seed, and no code is provided; the quantile bands in Figures 2-4 are also not precisely defined. These details matter for reproducibility.
Circularity Check
One self-definitional necessary condition in Section 5; central Theorem 1 is conditional, not circular.
-
self definitional
[Section 5.4–5.5, Definition 6 and Proposition 2(i)]
"Definition 6 (Jointly Closed Under Overshadowing). A subset S ⊆ Θ is called jointly closed under overshadowing if there exists a stable posterior distribution p(·) supported on S such that, for every γ /∈ S, γ cannot achieve a strictly positive log-likelihood ratio against p(·) on any action a ∈ Ao(S). ... Proposition 2 ... 1. Jointly Closed: For every parameter γ /∈ S, given the distribution of actions which are played infinitely often on ΩS, γ is overshadowed by the collective mixture of parameters in S."
Proposition 2's first necessary condition is a verbal restatement of Definition 6. 'Overshadowed by the collective mixture' is defined as failing to achieve a strictly positive expected log-likelihood ratio against the mixture p(·), and the proposition's proof sketch only repeats that an outsider which could beat all members of S would eventually receive posterior mass. The condition is thus imported by the definition rather than derived from a more primitive requirement; no independent argument establishes the existence of a fixed stable distribution p(·) satisfying the defining inequality. This makes the first part of the characterization close to a tautology.
full rationale
The paper's central Theorem 1 is a conditional statement: under the uniform KL-dominance condition of Assumption 2, the posterior mass on the pseudo-truth set decays exponentially. Assumption 2 is not merely a restatement of the conclusion; it posits a specific gap in KL divergences, and the exponential rate in Theorem 1 requires the additional martingale/Azuma argument. Whether that argument is valid is a separate correctness question, not a circularity one. The paper contains no load-bearing self-citations; the references to Kim (2017) and White (1982) are independent prior work. However, the Section 5 characterization is self-definitional: 'jointly closed under overshadowing' is defined as the property that outside parameters cannot achieve positive log-likelihood ratios against a mixture, and Proposition 2 then returns that same property as a necessary condition. This is a genuine, if localized, circular step, so the overall score is elevated above the no-circularity range.
Assumptions & free parameters
free parameters (2)
- ε(θ), the KL gap in Assumption 2 =
unspecified, assumed positive
- d, the uniform bound on log-likelihood increments in Lemma 1 =
unspecified positive constant
assumptions (5)
- domain assumption Assumption 1: the true parameter θ* lies outside the finite parameter space Θ.
- ad hoc to paper Assumption 2: there exists a pseudo-truth set Θ† such that parameters outside it are uniformly dominated in KL distance for every action and covariate.
- ad hoc to paper The KL divergence in Eq (5) is used with orientation E_θ log(fθ/fγ), while the proof's drift uses E_true log(L_Θ†/L_γ); these are different quantities.
- ad hoc to paper Lemma 1: log-likelihood increments are bounded by a constant d.
- ad hoc to paper Definition 6 assumes a stable posterior distribution p supported on S exists and makes outsiders' expected log-likelihood ratios non-positive.
invented entities (2)
-
Pseudo-truth set Θ†
-
Overshadowing edge and jointly closed strongly connected subgraph
Cite this review
Pith. "Pith review of Dynamic Decision-Making under Model Misspecification." pith.science (2026). https://pith.science/paper/NDLMZLHG
@misc{pith2026250514913,
author = {Pith},
title = {Pith review of: Dynamic Decision-Making under Model Misspecification},
year = {2026},
howpublished = {\url{https://pith.science/paper/NDLMZLHG}},
note = {Machine review of arXiv:2505.14913}
}
abstract
In this study, I investigate the dynamic decision problem with a finite parameter space when the functional form of conditional expected rewards is misspecified. Traditional algorithms, such as Thompson Sampling, guarantee neither an $O(e^{-T})$ rate of posterior parameter concentration nor an $O(T^{-1})$ rate of average regret. However, under mild conditions, we can still achieve an exponential convergence rate of the parameter to a pseudo truth set, an extension of the pseudo truth parameter concept introduced by White (1982). I further characterize the necessary conditions for the convergence of the expected posterior within this pseudo-truth set. Simulations demonstrate that while the maximum a posteriori (MAP) estimate of the parameters fails to converge under misspecification, the algorithm's average regret remains relatively robust compared to the correctly specified case. These findings suggest opportunities to design simple yet robust algorithms that achieve desirable outcomes even in the presence of model misspecifications.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Risk and optimal policies in bandit experiments
Adusumilli, Karun (2021). “Risk and optimal policies in bandit experiments”. In:arXiv preprint arXiv:2112.06363. Andrews, Isaiah et al. (2023). “Structural estimation under misspecification: theory and impli- cations for practice”. In. Armstrong, Timothy, Patrick M Kline, and Liyang Sun (2024). Adapting to Misspecification . Tech. rep. National Bureau of ...
arXiv 2021
-
[2]
1 1 − P γ /∈Θ† cγ(M γ t + I γ t ) # ≥ Eτ θ∗
Hence we can write the expectation of the posterior of the pseudo-truth set as: Eτ θ∗ πt(Θ†) = Eτ θ∗ " 1 1 − P γ /∈Θ† cγ(M γ t + I γ t ) # ≥ Eτ θ∗ " 1 1 − P γ /∈Θ† cγ(M γ t + ϵt) # (22) Then following the same methods by Kim (2017), for any δ >0 and γ ∈ Pwe can define the event: Bγ t (δ) = {|M γ t | ≤δt} then, for any choice of 0 < δ < ϵ 22 Eτ θ∗ πt(Θ†) ≥...
work page 2017
-
[33]
Which misspecifications persist?
Curran Associates, Inc., pp. 11478–11489. Fudenberg, Drew and Giacomo Lanzani (2023). “Which misspecifications persist?” In: Theo- retical Economics 18.3, pp. 1271–1315. Kim, Michael Jong (2017). “Thompson sampling for stochastic control: The finite parameter case”. In: IEEE Transactions on Automatic Control 62.12, pp. 6415–6422. Lattimore, Tor, Csaba Sze...
work page 2023
-
[34]
Minimizing sensitivity to model misspeci- fication
Curran Associates, Inc., pp. 3004–3015. Bonhomme, St´ ephane and Martin Weidner (2022). “Minimizing sensitivity to model misspeci- fication”. In: Quantitative Economics 13.3, pp. 907–954. Chung, Fan and Linyuan Lu (2006). “Concentration inequalities and martingale inequalities: a survey”. In: Internet mathematics 3.1, pp. 79–127. Esponda, Ignacio and Demi...
arXiv 2022
-
[119]
Dynamic Selection in Algorithmic Decision-making
Proceedings of Machine Learning Research. PMLR, pp. 5662–5670. Lattimore, Tor and Csaba Szepesv´ ari (2020).Bandit algorithms . Cambridge University Press. Li, Jin, Ye Luo, and Xiaowei Zhang (2023).Dynamic Selection in Algorithmic Decision-making. arXiv: 2108.12547 [econ.EM]. Liang, Annie (2019). “Games of incomplete information played by statisticians”. ...
work page Pith review arXiv 2020
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.