REVIEW 3 major objections 5 minor 20 references
The Limits of Predicting Agents from Behaviour
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Behavioral data alone can guarantee which actions an AI will not take in a novel environment exactly when a data-computable inequality is positive.
desk verdict A genuinely new and correct conditional characterization of when behavioral data can rule out an agent's decisions under intervention, with the caveat that everything rests on the grounding assumption. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Structural causal models (SCMs) serve as the common language for the AI's internal world model and the observer's data, with grounding as the bridge between them: the AI's internal model probabilities P_hat_d(V) equal the observed behavior probabilities P_d(V) for every decision d. The carrying object is the preference gap Δ_{d≻d*} = E_{P_hat}[Y | do(σ,d), c] − E_{P_hat}[Y | do(σ,d*), c]; weak predictability of d* holds exactly when some alternative d has a positive lower bound on Δ. The lower bound is derived by manipulating counterfactual probabilities using the axioms of counterfactuals and the σ-calculus, replacing the AI's internal expectations with observable P_d terms through grounding, and tightness is shown by explicit SCM constructions that attain the bound.
What would settle it
Construct a grounded AI whose internal structural causal model is fully known, record its behavior P_d(V), compute the Theorem 1 expression for a candidate unsafe decision d* under an intervention do(z), and then actually deploy the AI under do(z). If the expression is positive and the AI nevertheless chooses d*, the theorem's sufficiency claim is false; if the expression is negative but d* is never chosen across many runs with different internal models, the necessity claim needs qualification. The cleanest single check is to build two SCMs with identical P_d(V) but opposite optimal decisions under do(z) while the Theorem 1 expression is positive, which the theorem says is impossible.
Extended reading notes
Core claim
An AI that is grounded in a domain M, meaning its internal world model assigns the same intervention probabilities as the observed environment for every decision, is weakly predictable under a shift do(z) in a context c if and only if there is a decision d* such that for some alternative d the lower bound on the preference gap Δ_{d≻d*} is strictly positive. The preference gap is the difference in the AI's expected utility between the two decisions under the shift. The theorem gives a closed-form expression in terms of the observable behavior distribution P_d(V) and proves the bound is tight: whenever the expression is not positive, one can construct two structural causal models generating exactly the same behavior but making d and d* respectively optimal, so behavior alone cannot rule out either action. The same machinery yields tighter bounds when behavior is observed in multiple interventional domains, a negative result when the shift is entirely unspecified, partial predictability when covariate data from the shifted environment is available, and tight bounds on the AI's perceived counterfactual fairness gap and counterfactual harm gap.
Load-bearing premise
The whole argument rests on grounding: the AI's internal world model must assign exactly the same intervention probabilities as the real training environment for every decision; if the AI learned a distorted causal model, the observed-data inequalities no longer constrain its out-of-distribution choices.
Editorial extensions
If this is right
- Behavioral audits can certify that an AI will avoid specific unsafe actions under well-defined interventions, without inspecting its internal weights, whenever the Theorem 1 inequality is positive.
- Because the characterization is if-and-only-if, a failed inequality is a genuine epistemic limit: more behavior data from the same domain cannot rule out any action.
- Observations from several interventional domains tighten the bounds, so multi-environment evaluations can strictly expand the set of provably avoided actions.
- If the deployment shift is completely unspecified, behavior alone provides no action guarantees at all; knowing the shifted environment's covariate distribution restores partial predictability.
- Counterfactual-fairness claims cannot be certified from behavior alone — the fairness gap is compatible with any observed behavior — while the counterfactual harm gap has tight, informative bounds.
- The paper states its guarantees in the infinite-sample limit, so finite-sample application requires additional statistical treatment that the paper does not provide.
Reading between the lines
- A practical audit protocol follows directly: before deploying under an intervention, compute the inequality on logged behavior; a negative result should push evaluators toward model inspection or interventional data collection rather than more observational data.
- For large language models, the grounding assumption could be tested by comparing predicted optimal decisions under prompt-level interventions against the bounds computed from a behavior corpus, though assigning variables and decisions to text is nontrivial.
- The paper's own limitation notes point to natural extensions: relaxing acyclicity, incorporating verbal self-reports from the AI, and moving from exact to high-probability bounds by parameterizing the SCM family.
- When fairness and harm are defined counterfactually, audits that rely only on behavioral logs can certify harm bounds but not fairness intentions, shifting the burden of evidence toward model-based or interventional sources.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies how much of an AI agent's future out-of-distribution decisions can be predicted from its observed behavior, under the assumptions that the agent is an expected-utility maximizer over an internal structural causal model (SCM) and that the agent is 'grounded' in its training domain, i.e. its internal interventional distributions coincide with the real environment's for every decision. The central result (Theorem 1) is a necessary-and-sufficient inequality, computable from observed interventional distributions P_d(V), for the existence of a decision that can be ruled out as suboptimal under a hard intervention do(z). Section 4 also derives bounds for multiple environments, general shifts, and counterfactual fairness and harm gaps; Section 5 relaxes exact grounding, exact expected-utility maximization, and exact observation of the utility; the appendices contain proofs, canonical-model constructions, and a limitations discussion.
Significance. If correct, Theorem 1 is a clean, parameter-free characterization: either behavioral data certifies that a grounded agent will not take a given action in a novel environment, or there exist two equally data-compatible internal models with different optimal actions. The paper is explicit that the grounding assumption is load-bearing, and Section 5.1 extends it approximately; the limitations appendix (B.5) honestly acknowledges the finite-sample, acyclicity, and verbal-behavior caveats. The bounds follow the Balke-Pearl style and are derived rather than assumed, and the multiple-environments and fairness/harm results are natural and falsifiable. However, the tightness proofs as printed are flawed, which affects the central if-and-only-if claim; the issues appear repairable with the piecewise diagonal construction, so the contribution remains of high potential value.
major comments (3)
- [Appendix C, Eq. (79)] The SCM displayed for the lower-bound tightness construction branches on 'f_Z(u)=z' and sets Y=0 for d1 and Y=1 for d0 whenever f_Z(u) is not equal to the intervention value z. But in the observed regime the actual value of Z is f_Z(u), so this branch forces P_d1(Z≠z,Y=1)=0 and P_d0(Z≠z,Y=0)=0 irrespective of the observed distribution. For Example 1, Table 1 gives P_d1(Z=0,Y=1)=0.2, so the displayed construction cannot generate the observed P_d1. The correct construction should branch on the equality between the actual value of Z and f_Z(u): define f_Y(d,c,Z,u) to use the observed response when Z=f_Z(u) and to take the extreme values only on the off-diagonal Z≠f_Z(u) in the intervened regime. Because this construction is the only demonstration of tightness for Theorem 1, the if-and-only-if claim is not proved as written.
- [Theorem 1] Weak predictability is defined by the existence of d* such that for every valid SCM some alternative d (possibly depending on the SCM) is preferred to d*. Theorem 1, however, characterizes weak predictability by the existence of a single pair d,d* with min_{cM} Δ_{d≻d*} > 0. This is a quantifier exchange: from 'for every cM there exists d with Δ_d(cM)>0' it does not in general follow that 'there exists d with Δ_d(cM)>0 for every cM'. The proof only constructs tightness examples separately for each pair, so no argument is given that when all pairwise minima are non-positive, a single SCM exists in which d* is optimal against all alternatives simultaneously. A simultaneous construction is needed, or the theorem should be restated as a sufficiency result.
- [Theorem 2] The tightness construction for the multiple-environments bound suffers from the same off-diagonal branching problem as Theorem 1: the branch on f_S(u)=s forces outcomes in the observed regime that need not match the data. In addition, the proof of tightness is only given for the special case of two environments with Z=R1∪R2; the general statement for arbitrary R_i and k>2 is asserted without a construction. The theorem should either be proved in full generality or restricted to the case for which the tightness argument is supplied.
minor comments (5)
- [Theorem 1] The numerator of the second term in the displayed inequality is typeset as a separate line without a clear fraction; it should read [E[Y|c,z]P(c,z)+1-P(z)] / [P(c,z)+1-P(z)], as in Eq. (77) of Appendix C.
- [Theorem 4] The displayed condition is split across two fractions with denominator P_{σ,d*}(c); it would be much clearer to write the numerator as a single bracketed term, e.g. 1 - [2 + E[Y|c]P(c) - E[Y|c]P(c) - 2P(z) + P(c)] / P_{σ,d*}(c) > 0.
- [Corollary 1] The statement and proof use a single 'min over bP' for expressions involving both bP_d and bP_d*, which is ambiguous about whether the two decision-indexed distributions are optimized independently or jointly as marginals of the same internal model; the proof's denominators are also inconsistent (P_d versus bP_d). Please clarify the feasible set and the optimization.
- [Appendix C] In the displayed SCMs, the branch for C sets C to '1 otherwise'; the subsequent denominator computations require P(C=c | f_Z(u)≠z)=1, so the constant should be c (the context value) rather than the number 1.
- [Section 3] The expression 'argmax_π E[Y|do(π)]' is used without defining the intervened SCM induced by a policy π; the paper should define do(π) or replace it by an explicit policy-intervention notation.
Circularity Check
No significant circularity: the paper's bounds are conditional derivations from observed behavior under explicit grounding assumptions, with tightness constructions as independent verification.
full rationale
The central results are conditional characterization theorems. Theorem 1 states that an AI grounded in M (Definition 3, bP_d(V)=P_d(V)) is weakly predictable under do(z) iff a particular observable inequality holds; the inequality is expressed entirely in terms of the training-domain distributions P_d, and the target out-of-domain behavior enters only as the quantity to be bounded, never as an input. The proof derives lower and upper bounds by replacing bP expectations with P via grounding, then constructs SCMs (Appendix C, e.g., M1_d and M2_d) that match the observed P_d while attaining the bounds, which is a tightness check rather than a fit. The same structure holds for Theorems 2, 4, 5, 6, 9, and 10. Theorem 6 is explicitly acknowledged as an extension of Pearl (1999) and Tian and Pearl (2000), so it is not a renamed known result dressed up as novel. The only self-citations (Richens and Everitt 2024 for world-model learning; Jalaldoust et al. 2024 for canonical SCM construction) are motivational or auxiliary; the main derivations do not reduce to them, and the paper's own assumptions (grounding, SCM acyclicity, infinite sample) are stated openly in Definition 3 and Appendix B.5. No fitted parameter is later renamed as a prediction, and no uniqueness theorem is imported from the authors' prior work to force a choice. The paper is self-contained given its explicitly scoped assumptions.
Assumptions & free parameters
assumptions (6)
- domain assumption The AI's behaviour is generated by a recursive structural causal model cM over V, including decision D, context C, and utility Y.
- domain assumption Grounding: for every decision d, P_hat_d(V) = P_d(V).
- domain assumption The AI chooses a policy maximizing expected utility under its internal model.
- domain assumption Observer has access to interventional distributions P_d(V) for all decisions d, or can recover them under positivity of the policy.
- standard math The counterfactual axioms (composition, effectiveness, reversibility) and the sigma-calculus rules for soft interventions are valid.
- domain assumption Utility Y is bounded in [0,1].
Cite this review
Pith. "Pith review of The Limits of Predicting Agents from Behaviour." pith.science (2026). https://pith.science/paper/GKZ5JXN7
@misc{pith2026250602923,
author = {Pith},
title = {Pith review of: The Limits of Predicting Agents from Behaviour},
year = {2026},
howpublished = {\url{https://pith.science/paper/GKZ5JXN7}},
note = {Machine review of arXiv:2506.02923}
}
read the original abstract
As the complexity of AI systems and their interactions with the world increases, generating explanations for their behaviour is important for safely deploying AI. For agents, the most natural abstractions for predicting behaviour attribute beliefs, intentions and goals to the system. If an agent behaves as if it has a certain goal or belief, then we can make reasonable predictions about how it will behave in novel situations, including those where comprehensive safety evaluations are untenable. How well can we infer an agent's beliefs from their behaviour, and how reliably can these inferred beliefs predict the agent's behaviour in novel situations? We provide a precise answer to this question under the assumption that the agent's behaviour is guided by a world model. Our contribution is the derivation of novel bounds on the agent's behaviour in new (unseen) deployment environments, which represent a theoretical limit for predicting intentional agents from behavioural data alone. We discuss the implications of these results for several research areas including fairness and safety.
Reference graph
Works this paper leans on
-
[1]
as𝑃𝑑(𝑌∗ = 1|𝑍=𝑧) is no longer arbitrarily defined. The bounds derived in Example 2 on the AI’s belief on optimal decisions under an intervention 𝜎:={𝑍←𝑧}continue to hold: Δ𝑑1≻𝑑 0≥𝑃 𝑑1(𝑧,𝑦 ∗)−𝑃 𝑑0(𝑧,𝑦 ∗)+𝑃 𝑑0(𝑧)−1(29) Δ𝑑0≻𝑑 1≥𝑃 𝑑0(𝑧,𝑦 ∗)−𝑃 𝑑1(𝑧,𝑦 ∗)+𝑃 𝑑1(𝑧)−1,(30) where we have used the shorthand𝑃𝑑(𝑧,𝑦 ∗)=𝑃 𝑑(𝑍=𝑧,𝑌 ∗ = 1). But the distributions{𝑃𝑑(𝑧,𝑦 ∗)}𝑑...
work page 2024
-
[3]
Yoshua Bengio, Michael Cohen, Damiano Fornasiere, Joumana Ghosn, Pietro Greiner, Matt Mac- Dermott, Sören Mindermann, Adam Oberman, Jesse Richardson, Oliver Richardson, et al. Su- perintelligent agents pose catastrophic risks: Can scientist ai offer a safer path?arXiv preprint arXiv:2502.15657,
-
[5]
Does chatgpt have a mind?arXiv preprint arXiv:2407.11015,
Simon Goldstein and Benjamin A Levinstein. Does chatgpt have a mind?arXiv preprint arXiv:2407.11015,
-
[6]
Language models represent space and time.arXiv preprint arXiv:2310.02207,
Wes Gurnee and Max Tegmark. Language models represent space and time.arXiv preprint arXiv:2310.02207,
-
[9]
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg
Accessed: 2025-01-24. Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task.arXiv preprint arXiv:2210.13382,
arXiv 2025
-
[10]
Robust agents learn causal world models.arXiv preprint arXiv:2402.10877,
Jonathan Richens and Tom Everitt. Robust agents learn causal world models.arXiv preprint arXiv:2402.10877,
-
[14]
Steve Yadlowsky, Hongseok Namkoong, Sanjay Basu, John Duchi, and Lu Tian. Bounds on the conditional and average treatment effect with unobserved confounding factors.arXiv preprint arXiv:1808.09521,
-
[17]
Were I to intervene in the environment, what action do you believe is optimal?
by parameterizing the AI’s underlying model and making inference on the corresponding latent variable model to get high-probability bounds. An example parameterization of SCMs and probabilistic inference for decision-making across environments is given in (Bellot et al., 2024; Jalaldoust et al., 2024). We expect that similar techniques could be applied in...
work page 2009
Show all 20 references
-
[18]
𝐴 is a difference of two terms written𝐴(𝒓)=𝐴 1(𝒓)−𝐴 2(𝒓)
Proof.We claim here that for any𝑹⊂𝒁, 𝐴(∅)≤𝐴(𝒓)(135) This means that the bounds onΔ that we can obtain from an AI system grounded inM𝒓 are more informative than the bounds obtained from an AI system grounded inM. 𝐴 is a difference of two terms written𝐴(𝒓)=𝐴 1(𝒓)−𝐴 2(𝒓). 𝐴1(𝒓):=...
2024
-
[19]
The nature of the modification is unknown but we are told that after modification, the expected probability of𝑪 is given by𝑃𝜎,𝑑(𝑪), assumed to be known and internalised by the A
in a situation𝑪=𝒄 is defined as: Δ𝑑1≻𝑑 0 :=𝔼 b𝑃𝜎,𝑑1 [𝑌|𝑪=𝒄 ]−𝔼 b𝑃𝜎,𝑑0 [𝑌|𝑪=𝒄 ] (160) Here we know that𝜎 potentially modifies the mechanisms of the set of variables𝒁. The nature of the modification is unknown but we are told that after modification, the expected probability of𝑪...
2000
-
[20]
good” or “beneficial
Here, we consider this notion of fairness to illustrate the kind of inference that is possible to obtain from an AI’s external behaviour with one alternative account. The following theorem shows that, contrary to the counterfactual fairness gap,Ψ can be bounded given the AI’s ...
2022
-
[1967]
Towards resolving unidentifiability in inverse reinforcement learning.arXiv preprint arXiv:1601.06569,
Kareem Amin and Satinder Singh. Towards resolving unidentifiability in inverse reinforcement learning.arXiv preprint arXiv:1601.06569,
-
[1972]
Eric Schwitzgebel. Belief. In Edward N. Zalta and Uri Nodelman, editors,The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Spring 2024 edition,
2024
-
[1996]
A calculus for stochastic interventions: Causal effect identification and surrogate experiments
Juan Correa and Elias Bareinboim. A calculus for stochastic interventions: Causal effect identification and surrogate experiments. InProceedings of the AAAI conference on artificial intelligence, volume 34, pages 10093–10100, 2020a. Juan Correa and Elias Bareinboim. General tr...
-
[2008]
Risks from learned optimization in advanced machine learning systems.arXiv preprint arXiv:1906.01820,
Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems.arXiv preprint arXiv:1906.01820,
1906 arXiv
-
[2010]
Preference elicitation and inverse reinforcement learning
Constantin A Rothkopf and Christos Dimitrakakis. Preference elicitation and inverse reinforcement learning. InMachine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2011, Athens, Greece, September 5-9, 2011, Proceedings, Part III 22, pages 34–48....
2011
-
[2021]
Partial counterfactual identification from observational and experimental data.arXiv preprint arXiv:2110.05690,
Junzhe Zhang, Jin Tian, and Elias Bareinboim. Partial counterfactual identification from observational and experimental data.arXiv preprint arXiv:2110.05690,
-
[2022]
Evaluating the world model implicit in a generative model.arXiv preprint arXiv:2406.03689,
Keyon Vafa, Justin Y Chen, Jon Kleinberg, Sendhil Mullainathan, and Ashesh Rambachan. Evaluating the world model implicit in a generative model.arXiv preprint arXiv:2406.03689,
-
[2023]
Subjective causality.arXiv preprint arXiv:2401.10937,
Joseph Y Halpern and Evan Piermont. Subjective causality.arXiv preprint arXiv:2401.10937,
-
[2024]
Can a bayesian oracle prevent harm from an agent?arXiv preprint arXiv:2408.05284,
Yoshua Bengio, Michael K Cohen, Nikolay Malkin, Matt MacDermott, Damiano Fornasiere, Pietro Greiner, and Younesse Kaddar. Can a bayesian oracle prevent harm from an agent?arXiv preprint arXiv:2408.05284,
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.