Pith. sign in

REVIEW 4 major objections 3 minor 36 references

Goal-Directedness is in the Eye of the Beholder

T0 review · 4 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Every goal-directedness score measures the fit between a chosen formalism and the modeled system, not an objective property of the agent.

desk verdict A readable critique that names real problems in goal-directedness measures, but the impossibility claim is stronger than the evidence; worth refereeing, not worth accepting as is. read the letter →

arxiv 2508.13247 v1 pith:C6HR2PTX submitted 2025-08-18 cs.MA cs.AI

Nina Rajcic , Anders S{o}gaard This is my paper
classification cs.MAcs.AI
keywords goal-directednessmeasurementmechanisticprobingbehavioraldefinitioncausalinfluencediagramsmulti-agentsimulationAIalignmentintentionalstance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that goal-directedness is not an objective, measurable property of an agent. It shows that the leading behavioral definition—a decision variable is goal-directed when its behavior is well-predicted by the hypothesis that it optimizes a utility—collapses in cases where the goal is absent, unreachable, or where all actions lead to the same outcome, and it cannot handle the granularity of real goal descriptions. It also targets mechanistic probing, reporting that classifiers trained on the raw weights of networks optimized for different tasks perform at chance, undermining the idea that goals are encoded in internal states. The authors conclude that every goal-directedness score measures only the fit between a modeling framework and the system being modeled, and they propose studying goal-directed behavior as something that emerges in multi-agent simulation rather than something to be detected inside an agent. If correct, this removes the hope of a neutral, objective goal-detection test for AI safety.

What carries the argument

The load-bearing object is the formal notion of goal-directedness from Definition 2.1: a decision variable D is goal-directed with respect to utility U to the extent that D's conditional distribution is well-predicted by the hypothesis that D optimizes U. The paper runs this definition through three failure modes: syntactic pathologies where all behaviors score equally (No Cheese, Cheese-Craving Stone, Black Hole Collector); granularity and entailment problems in goal specification; and computational intractability when agent interactions introduce cycles, requiring game-theoretic equilibrium reasoning. On the mechanistic side, the key machinery is a probing experiment: 1,000 trained network

What would settle it

Re-run the probing experiment with a positive control: add a known linear marker of the training goal to the weights and ask whether the probe can detect it. If the probe detects the marker but still cannot distinguish goals, the paper's conclusion is supported; if it cannot detect even the marker, the experiment is too insensitive to license a claim about goal encoding. A stronger test would use a permutation-invariant representation or activation-based probe; above-chance classification of the training goal would refute the claim that goals are not internally encodable.

Watch

Extended reading notes

Core claim

The central claim is that goal-directedness cannot be measured objectively. Working from the definition of goal-directedness as a variable's behavior being well-predicted by the hypothesis that it optimizes a utility function, the paper shows that this measure assigns maximal goal-directedness to random behavior in cases where the goal is absent or unreachable (a mouse with no cheese, a stone that cannot move, a black hole that makes all paths equivalent). Goal specification runs into granularity problems—eating this cheese versus any cheese—and once a second agent is introduced, causal influence diagrams contain cycles that make joint inference computationally intractable. Turning to mechan

Load-bearing premise

The empirical case rests on the premise that feeding raw parameter vectors to a static classifier is a fair test of whether goals are encoded in a network's internals; if permutation symmetries of the weights or the probe's own inductive bias explain the null result, the mechanistic critique is left without its experimental anchor.

Editorial extensions

If this is right

  • No single objective score of goal-directedness can serve as an alarm for emergent goals; any reported goal-directedness value should be read as conditional on an explicit modeling framework.
  • Behavioral measures inheriting Definition 2.1 will be uninformative in multi-agent or partially observable settings, because modeling the interaction becomes computationally intractable.
  • Mechanistic probing of raw weights, at least in the tested linear-task setting, cannot retrieve which goal a network was trained toward, so internal-state goal detection needs new representations or controls.
  • Research should move to multi-agent simulation where goal-directedness is studied as patterns of persistence, norm-sensitivity, and causal intervention over time, without assuming explicit internal goals.
  • Biological evidence suggests goal-directed behavior can arise from task-structure progress signals without a goal representation, so optimal-policy-based definitions may be too narrow for both animals and AI.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension of the mechanistic critique is to probe activations rather than weights, or to apply a permutation-canonicalized weight representation with a positive control; a positive result would localize the null effect to the probing substrate rather than to the reality of goals.
  • If goal-directedness is observer-relative, then two conflicting goal attributions can both be valid; the useful comparison is which formalism better predicts behavior in a specific deployment context.
  • The simulation agenda implies goal-directedness could be quantified by causal intervention—for example, asking whether removing the goal object changes behavior in context—turning it into an empirical effect size rather than a hidden property.
  • The relational view also bears on animal cognition debates: claims that an animal 'really' has a goal become model-dependent, so neuroethological evidence should be framed as supporting one predictive model over another.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The manuscript argues that goal-directedness cannot be measured objectively. It critiques behavioral definitions, exemplified by MacDermott et al. [3], and mechanistic probing approaches, exemplified by Xu and Rivera [4]. For behavioral accounts, it identifies syntactic pathology cases (No Cheese, Cheese-Craving Stone, Black Hole Collector), granularity and uncertainty problems, and computational intractability in multi-agent settings. For mechanistic accounts, it raises multiple realizability and externalism, and reports a null experiment probing raw weights of networks trained on two tasks. The paper concludes that goal-directedness is a relation between an observer's formalism and the system, and proposes multi-agent simulation as an alternative.

Significance. The paper is a useful critical survey and a thought-provoking proposal. The granularity problem, the externalism challenge, and the emphasis on multi-agent interaction are real and often underappreciated issues. The proposed turn to multi-agent simulation is plausible as a research direction. However, the paper's central universal negative is not supported: the evidence is a set of contingent limitations plus a null result with no positive control. The paper's main contribution, if it held, would be significant for AI alignment, but as it stands the conclusion overreaches.

major comments (4)
  1. [Abstract and §6] The paper's own concessions contradict the universal negative claim. §3.1 says the syntactic problems 'could easily be addressed'; §4.1 says mechanistic accounts 'do seem to resolve some of the problems of behavioral accounts'; §6 says the authors are 'amenable' to the practical value of existing measures. These admissions are compatible with the claim that objective measurement is difficult, not impossible. A universal negative ('cannot be measured objectively') requires showing that no possible formalization can yield objective measurement; the paper only critiques two specific approaches. This is a load-bearing logical gap.
  2. [§4.2 Measurement Problems] The null result is uninformative without a positive control. The probe's input is raw weight vectors, but networks have permutation symmetries: functionally identical networks correspond to many weight orderings. A static classifier on raw weights can fail even for trivial properties like task identity, so chance performance does not imply 'goals are not directly encoded.' The paper itself concedes that the null result 'may of course be due to the inductive bias of the learning classifier.' To draw the claimed conclusion, the authors need to show the probe can recover some signal from the weights (e.g., task identity) and then fails only for goal labels.
  3. [§3.3 Measurement Problems] The paper states 'Absence of measurement isn't evidence of absence' when discussing computational intractability, yet uses both the null result and intractability as evidence of non-objectivity. The two-mice example shows that a particular formalism (CIDs) becomes intractable in interactive settings; this does not establish that no other formalism could represent goal-directedness objectively. The tractability issue is a property of the model class, not of the target phenomenon.
  4. [§3.1 and §3.2] The pathological examples (No Cheese, Black Hole Collector) are edge cases in existing definitions, and the paper acknowledges that Proposition 3.3 in [3] may cover them (§3.1). Without proving that no amended definition can avoid these pathologies, they do not support an impossibility claim. The granularity problem similarly shows that specifying goals requires choices, but the existence of multiple possible formalizations does not imply that all are equally valid or that none can be objective.
minor comments (3)
  1. [Figure 3] The caption omits experimental details: number of seeds, hyperparameters, data-set sizes, and error bars. Reporting variance across runs is necessary to interpret chance-level performance.
  2. [§2] 'It is customary to train linear and non-linear probes to probe for their capabilities' is awkward phrasing; suggest 'to probe for their capabilities' once.
  3. [§4.2] 'We trained up to 1,000 linear feed-forward neural networks' is vague; specify the exact number and how the probing classifiers were trained on them.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's argument is a case-based critique plus a novel null experiment, and its conclusions do not reduce to fitted inputs or self-citations.

full rationale

The paper does not derive goal-directedness from a fitted parameter or from a self-citation chain. Its main claim—that goal-directedness cannot be measured objectively—is supported by conceptual arguments against behavioral definitions (Section 3), mechanistic definitions (Section 4.1), and a new experiment (Section 4.2). The experimental inference is from a null result, not from a prediction forced by construction: the authors train networks on two tasks and then probe their raw weights, finding chance-level performance. They explicitly concede the null result 'may of course be due to the inductive bias of the learning classifier,' which is an honest limitation weakening the empirical anchor, but it is not circularity. The conclusion that goals are 'not directly encoded' is an interpretation of the null result, not a restatement of the experimental setup. No equations are shown to be equivalent by construction, no fitted parameter is renamed as a prediction, and no load-bearing premise is justified solely by a self-citation. The paper also explicitly concedes in Sections 3.1, 4.1, and 6 that the problems it raises 'could easily be addressed,' that mechanistic probes 'do seem to resolve some of the problems of behavioral accounts,' and that existing measures have practical value—concessions that make the argument more moderate than a purely circular position. The central thesis may overreach, but that is a correctness risk, not a circularity concern.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper's central claim rests on philosophical (intentional stance), empirical (weight-space probe), and biological (El-Gaby et al., Hill et al.) premises. No free parameters are fitted; the main unexamined assumption is the validity of raw-weight probing.

assumptions (4)
  • domain assumption Intentional attribution is pluralistic and context-dependent (Dennett's intentional stance).
    The conclusion that measurement is observer-relative presupposes Dennett's instrumentalism, cited in Section 5, rather than being derived.
  • ad hoc to paper A static classifier on raw weight vectors can detect goal encoding, if goals are encoded at all.
    Section 4.2: 'their input was the raw model weights'; no account is given of weight-space symmetries or of a positive control, so the null result depends on this premise.
  • domain assumption The biological findings (El-Gaby et al. 2024, Hill et al. 2025) show that goal-directed behavior need not involve explicit goal representation.
    Section 5 uses these results to support the proposal that goals need not be encoded internally; this is an unproved background claim carried from cited biology.
  • domain assumption Observable features such as persistence, norm-sensitivity, and causal intervention in simulation are legitimate indicators of goal-directedness.
    Section 5 proposes these features as indicators; the paper provides no formal definition, so the proposal rests on an assumed equivalence between these features and goal-directedness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Goal-Directedness is in the Eye of the Beholder." pith.science (2026). https://pith.science/paper/C6HR2PTX

@misc{pith2026250813247,
  author       = {Pith},
  title        = {Pith review of: Goal-Directedness is in the Eye of the Beholder},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C6HR2PTX}},
  note         = {Machine review of arXiv:2508.13247}
}
read the original abstract

Our ability to predict the behavior of complex agents turns on the attribution of goals. Probing for goal-directed behavior comes in two flavors: Behavioral and mechanistic. The former proposes that goal-directedness can be estimated through behavioral observation, whereas the latter attempts to probe for goals in internal model states. We work through the assumptions behind both approaches, identifying technical and conceptual problems that arise from formalizing goals in agent systems. We arrive at the perhaps surprising position that goal-directedness cannot be measured objectively. We outline new directions for modeling goal-directedness as an emergent property of dynamic, multi-agent systems.

Figures

Figures reproduced from arXiv: 2508.13247 by the authors.

Figure 1
Figure 1. Example 3.4 modeled as a Causal Bayesian Network [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Example 3.4 represented as a CID for a cooperative (left) and non-cooperative (right) game, [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Goal-directedness is not learnable for linear (left) or non-linear (right) probing classifiers. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

36 extracted references · 24 canonical work pages

  1. [3]

    Measuring goal- directedness

    Matt MacDermott, James Fox, Francesco Belardinelli, and Tom Everitt. Measuring goal- directedness. Advances in Neural Information Processing Systems, 37:11412–11431, 2024

    work page 2024

  2. [4]

    Towards measuring goal-directedness in AI systems, 2024

    Dylan Xu and Juan-Pablo Rivera. Towards measuring goal-directedness in AI systems, 2024. URL https://arxiv.org/abs/2410.04683

    arXiv 2024

  3. [1]

    A behavioral model of rational choice

    Herbert A Simon. A behavioral model of rational choice. The quarterly journal of economics, pages 99–118, 1955

    work page 1955

  4. [2]

    Agents and devices: A relative definition of agency, 2018

    Laurent Orseau, Simon McGregor McGill, and Shane Legg. Agents and devices: A relative definition of agency, 2018. URL https://arxiv.org/abs/1805.12387

    arXiv 2018

  5. [5]

    Modeling AGI Safety Frameworks with Causal Influence Diagrams

    Tom Everitt, Ramana Kumar, Victoria Krakovna, and Shane Legg. Modeling agi safety frameworks with causal influence diagrams. arXiv preprint arXiv:1906.08663, 2019

    work page Pith review arXiv 1906

  6. [6]

    Agent incentives: A causal perspective

    Tom Everitt, Ryan Carey, Eric D Langlois, Pedro A Ortega, and Shane Legg. Agent incentives: A causal perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11487–11495, 2021

    work page 2021

  7. [7]

    Causality

    Judea Pearl. Causality. Cambridge university press, 2009

    2009

  8. [8]

    Bayesianism and causality, or, why I am only a half-Bayesian

    Judea Pearl. Bayesianism and causality, or, why I am only a half-Bayesian. In Foundations of Bayesianism, pages 19–36. Springer, 2001

    work page 2001

Show all 36 references
  1. [9]

    Zachary C. Lipton. The mythos of model interpretability, 2017. URL https://arxiv.org/ abs/1606.03490

    2017 arXiv

  2. [10]

    Evaluating the goal-directedness of large language models, 2025

    Tom Everitt, Cristina Garbacea, Alexis Bellot, Jonathan Richens, Henry Papadatos, Siméon Campos, and Rohin Shah. Evaluating the goal-directedness of large language models, 2025. URL https://arxiv.org/abs/2504.11844

    2025 arXiv

  3. [11]

    Function vectors in large language models

    Eric Todd, Millicent L Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. arXiv preprint arXiv:2310.15213, 2023

    2023 arXiv

  4. [12]

    Thorn, Christian Eichhorn, Gabriele Kern-Isberner, and Gerhard Schurz

    Paul D. Thorn, Christian Eichhorn, Gabriele Kern-Isberner, and Gerhard Schurz. Qualitative probabilistic inference with default inheritance. In Christoph Beierle, Gabriele Kern-Isberner, Marco Ragni, and Frieder Stolzenburg, editors,Proceedings of the KI 2015 Workshop on Forma...

    2015

  5. [13]

    Towards bidirectional human-ai alignment: A systematic review for clarifications, framework, and future directions

    Hua Shen, Tiffany Knearem, Reshmi Ghosh, Kenan Alkiek, Kundan Krishna, Yachuan Liu, Ziqiao Ma, Savvas Petridis, Yi-Hao Peng, Li Qiwei, et al. Towards bidirectional human-ai alignment: A systematic review for clarifications, framework, and future directions. arXiv preprint arXi...

    2024

  6. [14]

    Multi- agent risks from advanced ai

    Lewis Hammond, Alan Chan, Jesse Clifton, Jason Hoelscher-Obermaier, Akbir Khan, Euan McLean, Chandler Smith, Wolfram Barfuss, Jakob Foerster, Tomáš Gavenˇciak, et al. Multi- agent risks from advanced ai. arXiv preprint arXiv:2502.14143, 2025

    2025 arXiv

  7. [15]

    Collective intentions and actions

    John R Searle. Collective intentions and actions. Intentions in communication, 401(4):401, 1990

    1990

  8. [16]

    Multi-agent influence diagrams for representing and solving games

    Daphne Koller and Brian Milch. Multi-agent influence diagrams for representing and solving games. Games and economic behavior, 45(1):181–221, 2003

    2003

  9. [17]

    On imperfect recall in multi-agent influence diagrams

    James Fox, Matt MacDermott, Lewis Hammond, Paul Harrenstein, Alessandro Abate, and Michael Wooldridge. On imperfect recall in multi-agent influence diagrams. arXiv preprint arXiv:2307.05059, 2023

    2023 arXiv

  10. [18]

    Equi- librium refinements for multi-agent influence diagrams: theory and practice

    Lewis Hammond, James Fox, Tom Everitt, Alessandro Abate, and Michael Wooldridge. Equi- librium refinements for multi-agent influence diagrams: theory and practice. arXiv preprint arXiv:2102.05008, 2021. 10

    2021 arXiv

  11. [19]

    Graphical models for interactive pomdps: representations and solutions

    Prashant Doshi, Yifeng Zeng, and Qiongyu Chen. Graphical models for interactive pomdps: representations and solutions. Autonomous agents and multi-agent systems, 18:376–416, 2009

    2009

  12. [20]

    Recursively modeling other agents for decision making: A research perspective

    Prashant Doshi, Piotr Gmytrasiewicz, and Edmund Durfee. Recursively modeling other agents for decision making: A research perspective. Artificial Intelligence, 279:103202, 2020

    2020

  13. [21]

    A cognitive hierarchy model of games

    Colin F Camerer, Teck-Hua Ho, and Juin-Kuan Chong. A cognitive hierarchy model of games. The Quarterly Journal of Economics, 119(3):861–898, 2004

    2004

  14. [22]

    Bounded rationality

    Herbert A Simon. Bounded rationality. Utility and probability, pages 15–18, 1990

    1990

  15. [23]

    The Intentional Stance

    Daniel Clement Dennett. The Intentional Stance. MIT Press, 1981

    1981

  16. [24]

    Actions, reasons, and causes

    Donald Davidson. Actions, reasons, and causes. The Journal of Philosophy, 60(23):685–700,

  17. [25]

    Précis of the intentional stance

    Daniel C Dennett. Précis of the intentional stance. Behavioral and brain sciences , 11(3): 495–505, 1988

    1988

  18. [27]

    Cognitive ethology and the intentionality of animal behavior

    Colin Allen and Marc Bekoff. Cognitive ethology and the intentionality of animal behavior. Mind and Language, 10(4):313–328, 1995. doi: 10.1111/j.1468-0017.1995.tb00017.x

    1995

  19. [28]

    Technological approach to mind everywhere: A framework for conceptualizing goal-directedness in biology and other domains

    Michael Levin and David Resnik. Technological approach to mind everywhere: A framework for conceptualizing goal-directedness in biology and other domains. 2025

    2025

  20. [29]

    Planarian regeneration: achievements and future directions after 20 years of research

    Emili Saló, Josep F Abril, Teresa Adell, Francesc Cebrià Sánchez, Kay Eckelt, Enrique Fernández-Taboada, Mette Handberg-Thorsager, Marta Iglesias, M Dolores Molina Jiménez, and Gustavo Rodríguez-Esteban. Planarian regeneration: achievements and future directions after 20 years...

    2009

  21. [30]

    Mohamady El-Gaby, Adam Loyd Harris, James C. R. Whittington, William Dorrell, Arya Bhomick, Mark E. Walton, Thomas Akam, and Timothy E. J. Behrens. A cellular basis for mapping behavioural structure. Nature, 636(8043):671–680, 2024

    2024

  22. [31]

    Mistakes in action: on clarifying the phenomenon of goal-directedness

    Jonathan Hill, David S Oderberg, Christopher Austin, François Cinotti, Ingo Bojak, and Jonathan M Gibbins. Mistakes in action: on clarifying the phenomenon of goal-directedness. Biological Theory, pages 1–14, 2025

    2025

  23. [32]

    Cooperate or collapse: Emergence of sustainable cooperation in a society of llm agents

    Giorgio Piatti, Zhijing Jin, Max Kleiman-Weiner, Bernhard Schölkopf, Mrinmaya Sachan, and Rada Mihalcea. Cooperate or collapse: Emergence of sustainable cooperation in a society of llm agents. Advances in Neural Information Processing Systems, 37:111715–111759, 2024

    2024

  24. [33]

    Understanding from machine learning models

    Emily Sullivan. Understanding from machine learning models. British Journal for the Philoso- phy of Science, 73(1):109–133, 2022. doi: 10.1093/bjps/axz035

    2022 doi

  25. [34]

    Bowman, and Evan Hubinger

    Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam Marks, Johannes Treutlein, Tim Belonax, Jack Chen, David Duvenaud, Akbir Khan, Julian Michael, Sören Mindermann, Ethan Perez, Linda Petrini, Jonathan Uesato, Jared Kaplan, Buck Shlegeris, Sam...

  26. [35]

    Frontier models are capable of in-context scheming, 2025

    Alexander Meinke, Bronson Schoen, Jérémy Scheurer, Mikita Balesni, Rusheb Shah, and Marius Hobbhahn. Frontier models are capable of in-context scheming, 2025. URL https: //arxiv.org/abs/2412.04984. 11

    2025 arXiv

  27. [1963]

    URL http://www.jstor.org/stable/2023177

    ISSN 0022362X. URL http://www.jstor.org/stable/2023177

  28. [2024]

    URL https://arxiv.org/abs/2412.14093

    arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.