Recognition: unknown
Information Aggregation with AI Agents
Pith reviewed 2026-05-10 00:19 UTC · model grok-4.3
The pith
AI agents in prediction markets aggregate private signals well in simple cases but lose effectiveness as reasoning about others' knowledge grows more complex.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
AI agents aggregate dispersed private information effectively through trading when the information structure is simple, but their performance declines significantly as the structure increases in complexity, particularly when agents must reason about the knowledge and actions of others. This pattern persists even when cheap talk is allowed, market duration or initial price is altered, or strategic prompting is used. Smarter agents achieve better aggregation and higher profits, yet feedback on past performance leaves aggregation unchanged.
What carries the argument
Prediction markets in which AI agents receive private signals about an event and trade repeatedly, with aggregation success measured by the log error between the final price and the true probability.
If this is right
- Prediction markets continue to aggregate information reliably even when participants are AI agents and design features such as communication or timing are varied.
- Higher-capability AI agents produce more accurate final prices and earn greater trading profits.
- Feedback on past trading performance does not improve subsequent information aggregation by AI agents.
- Cheap talk among agents does not change the degree of information aggregation achieved.
Where Pith is reading between the lines
- Real-world markets populated by AI agents may need deliberately simple information environments to avoid the performance drop observed here.
- The robustness to prompting and communication suggests prediction markets could serve as a stable coordination device for groups of AI agents even as models advance.
- Testing whether mixing human and AI traders mitigates the complexity penalty would reveal whether the limitation is specific to current AI reasoning patterns.
Load-bearing premise
That log error of the final price fully measures information aggregation and that the tested AI agents and signal structures represent how such agents behave more generally.
What would settle it
An experiment in which log error stays low and stable as signal complexity increases, or in which providing feedback on past performance measurably lowers error.
Figures
read the original abstract
Can Large Language Models (AI agents) aggregate dispersed private information through trading and reason about the knowledge of others by observing price movements? We conduct a controlled experiment where AI agents trade in a prediction market after receiving private signals, measuring information aggregation by the log error of the last price. We find that although the median market is effective at aggregating information in the easy information structures, increasing the complexity has a significant and negative impact, suggesting that AI agents may suffer from similar limitations as humans when reasoning about others. Consistent with our theoretical predictions, information aggregation remains unaffected by allowing cheap talk communication, changing the duration of the market or initial price, and strategic prompting, thus demonstrating that prediction markets are robust. We establish that "smarter" AI agents perform better at aggregation and they are more profitable. Surprisingly, giving them feedback about past performance has no impact on aggregation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript reports a controlled experiment in which LLM-based AI agents trade in prediction markets after receiving private signals about an asset's value. Information aggregation is measured by the log error between the final market price and the realized fundamental value. The authors find that the median market aggregates information effectively under simple information structures but that performance deteriorates significantly as the complexity of the signal structure increases. They report that these outcomes are robust to cheap talk, market duration, initial price, and strategic prompting, that higher-capability agents aggregate better and earn higher profits, and that performance feedback has no effect.
Significance. If the central empirical pattern holds after proper statistical controls and benchmarking, the work would contribute to the growing literature on AI agents in strategic economic environments by documenting a complexity-dependent limitation in higher-order belief formation that parallels human behavior. The robustness claims, if substantiated, would also support the use of prediction markets as a relatively stable aggregation mechanism even when participants are artificial agents. The absence of sample sizes, formal statistical tests, and an explicit comparison to the full-information Bayesian posterior in the current text, however, leaves the magnitude and interpretation of the complexity effect uncertain.
major comments (3)
- [Abstract and Results] The abstract and results section report a 'significant and negative impact' of increasing information-structure complexity on aggregation, yet supply no sample sizes, number of independent markets, statistical tests, or controls for prompt stochasticity. Without these, the claimed effect size and its attribution to reasoning limitations cannot be evaluated.
- [Measurement and Results] Log error of the last price is used as the sole measure of aggregation success. This metric is informative only if the market design makes the rational-expectations price coincide with the posterior mean conditional on the union of all private signals; the manuscript does not report a comparison of observed prices to this full-information benchmark, so deviations cannot be isolated to failures of recursive reasoning about others rather than to prompt sensitivity or non-equilibrium heuristics.
- [Introduction and Results] The claim that results are 'consistent with theoretical predictions' is stated without an explicit statement of the model, the derived predictions for each information structure, or a quantitative test of those predictions against the experimental outcomes.
minor comments (2)
- [Abstract] The abstract refers to the 'median market' without defining how the median is computed across repeated runs or how outliers are handled.
- [Experimental Design] Details on the precise prompting templates, token limits, and temperature settings used for the different AI agents are not provided, limiting replicability.
Simulated Author's Rebuttal
We thank the referee for the constructive and detailed comments. We address each major point below, agreeing where revisions are needed to improve clarity and rigor.
read point-by-point responses
-
Referee: [Abstract and Results] The abstract and results section report a 'significant and negative impact' of increasing information-structure complexity on aggregation, yet supply no sample sizes, number of independent markets, statistical tests, or controls for prompt stochasticity. Without these, the claimed effect size and its attribution to reasoning limitations cannot be evaluated.
Authors: We agree that these details are essential for evaluating the results. The experiment consisted of 100 independent markets per information structure (four structures total). In the revised manuscript we will report the exact sample sizes, present median log errors together with interquartile ranges, and include formal non-parametric tests (Mann-Whitney U tests) comparing complexity levels. We will also document our controls for prompt stochasticity, which consisted of averaging across five random seeds and two temperature values per agent. These additions will make the effect sizes and robustness transparent. revision: yes
-
Referee: [Measurement and Results] Log error of the last price is used as the sole measure of aggregation success. This metric is informative only if the market design makes the rational-expectations price coincide with the posterior mean conditional on the union of all private signals; the manuscript does not report a comparison of observed prices to this full-information benchmark, so deviations cannot be isolated to failures of recursive reasoning about others rather than to prompt sensitivity or non-equilibrium heuristics.
Authors: The referee correctly notes that interpretation would be strengthened by an explicit benchmark. Our market mechanism is designed so that the rational-expectations price equals the full-information posterior mean given common knowledge of the signal structure. In the revision we will add a new subsection that computes this Bayesian benchmark for each information structure and reports the gap between observed closing prices and the benchmark. This comparison will help isolate the contribution of higher-order belief failures from other sources of error. revision: yes
-
Referee: [Introduction and Results] The claim that results are 'consistent with theoretical predictions' is stated without an explicit statement of the model, the derived predictions for each information structure, or a quantitative test of those predictions against the experimental outcomes.
Authors: We accept that the theoretical link should be stated more explicitly. The predictions follow from a model of iterative belief updating in which the depth of recursion required rises with signal complexity. In the revised version we will add a concise theoretical subsection that states the model, lists the predicted ordering of aggregation performance across the four structures, and includes a direct quantitative comparison (table or figure) of theoretical predictions versus observed median errors. This will substantiate the consistency claim with the necessary detail. revision: yes
Circularity Check
Empirical experiment reports direct outcomes with no circular derivations
full rationale
The paper describes a controlled experiment in which AI agents trade in prediction markets after receiving private signals, with information aggregation measured directly by the log error of the last price. Main findings (effect of complexity, robustness to cheap talk/duration/prompting, effect of agent intelligence) are reported as experimental results rather than outputs of any derivation chain. No equations, fitted parameters, or self-citations are invoked to generate the reported statistics by construction; the abstract's reference to consistency with theoretical predictions does not reduce the empirical claims to inputs. This is a standard non-circular empirical design.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Prediction market prices aggregate dispersed private information
Reference graph
Works this paper leans on
-
[1]
Unawareness and Partitional Information Structures
Salvatore Modica and Aldo Rustichini. Unawareness and Partitional Information Structures. Games and Economic Behavior. 1999
1999
-
[2]
Unawareness
Jing Li. Unawareness. Mimeo, University of Wisconsin-Madison. 2003
2003
-
[3]
Unawareness
Jing Li. Unawareness. Mimeo, University of Pennsylvania. 2004
2004
-
[4]
Information Structures with Unawareness
Jing Li. Information Structures with Unawareness. Journal of Economic Theory. 2009
2009
-
[5]
Information Structures with Unawareness
Jing Li. Information Structures with Unawareness. Journal of Economic Theory. 2008
2008
-
[6]
Standard state spaces preclude unawareness
Eddie Dekel and Bart Lipman and Aldo Rustichini. Standard state spaces preclude unawareness. Econometrica. 1998
1998
-
[7]
Schipper
Aviad Heifetz and Martin Meier and Burkhard C. Schipper. Interactive unawareness and speculative trade. Bonn Econ Discussion Papers. 2003
2003
-
[8]
Schipper
Aviad Heifetz and Martin Meier and Burkhard C. Schipper. Interactive unawareness. Journal of Economic Theory. 2006
2006
-
[9]
Schipper
Aviad Heifetz and Martin Meier and Burkhard C. Schipper. A canonical model of interactive unawareness. Games and Economic Behavior. 2008
2008
-
[10]
Schipper , title =
Aviad Heifetz and Martin Meier and Burkhard C. Schipper , title =. The University of California, Davis, Mimeo , year =
-
[11]
Schipper , title =
Aviad Heifetz and Martin Meier and Burkhard C. Schipper , title =. Games and Economic Behavior , volume =
-
[12]
Halpern , title =
Ronald Fagin and Joseph Y. Halpern , title =. Artificial Intelligence , year =
-
[13]
Halpern , title =
Joseph Y. Halpern , title =. Games and Economic Behavior , year =
-
[14]
Awareness and Partitional Information Structures
Salvatore Modica and Aldo Rustichini. Awareness and Partitional Information Structures. Theory and Decision. 1994
1994
-
[15]
Modeling Bounded Rationality
Ariel Rubinstein. Modeling Bounded Rationality. 1998
1998
-
[16]
Knowledge and Belief
Jaako Hintikka. Knowledge and Belief. 1962
1962
-
[17]
On Strategy Proofness and Single Peakedness
H.Moulin. On Strategy Proofness and Single Peakedness. Public Choice. 1980
1980
-
[18]
Straightforward elections, unanimity and phantom voters
K.Border and J.Jordan. Straightforward elections, unanimity and phantom voters. Review of Economic Studies. 1983
1983
-
[19]
Strategy-proofnes and median voters
S.Ching. Strategy-proofnes and median voters. International Journal of Game Theory. 1997
1997
-
[20]
Strategy-proof Location on a Network
James Schummer and Rakesh V.Vohra. Strategy-proof Location on a Network. Journal of Economic Theory. 2002
2002
-
[21]
Annals of Statistics , year =
Robert Aumann , title =. Annals of Statistics , year =
-
[22]
Cowles Foundation Discussion Paper , year =
John Geanakoplos , title =. Cowles Foundation Discussion Paper , year =
-
[23]
The Complete Sherlock Holmes , OPTcrossref =
Arthur Conan Doyle , title =. The Complete Sherlock Holmes , OPTcrossref =. 1930 , OPTeditor =
1930
-
[24]
International Journal of Game Theory , year =
Aviad Heifetz , title =. International Journal of Game Theory , year =
-
[25]
International Journal of Game Theory , year =
Jean-Francois Mertens and Shmuel Zamir , title =. International Journal of Game Theory , year =
-
[26]
Discussion Paper \#1875, Stanford University , year =
Yossi Feinberg , title =. Discussion Paper \#1875, Stanford University , year =
-
[27]
Discussion Paper \#1894, Stanford University , year =
Yossi Feinberg , title =. Discussion Paper \#1894, Stanford University , year =
-
[28]
Halpern and Yoram Moses and Moshe Y
Ronald Fagin and Joseph Y. Halpern and Yoram Moses and Moshe Y. Vardi , ALTeditor =. Reasoning About Knowledge , publisher =. 1995 , OPTkey =
1995
-
[29]
Journal of Economic Theory , year =
John Geanakoplos and Heraklis Polemarchakis , title =. Journal of Economic Theory , year =
-
[30]
American Economic Review , year =
Enriqueta Aragones and Itzhak Gilboa and Andrew Postlewaite and David Schmeidler , title =. American Economic Review , year =
-
[31]
University of Southampton, Discussion Papers in Economics and Econometrics , year =
Spyros Galanis , title =. University of Southampton, Discussion Papers in Economics and Econometrics , year =
-
[32]
Economic Theory , year =
Spyros Galanis , title =. Economic Theory , year =
-
[33]
1998 , OPTannote =
Ely, Jeffrey, Yi-Chun Chen and Xiao Luo , title =. 1998 , OPTannote =
1998
-
[34]
Ely and and Xiao Luo , title =
Yi-Chun Chen and Jeffrey C. Ely and and Xiao Luo , title =. International Journal of Game Theory , year =
-
[35]
and Ely, Jeffrey and Luo, X
Chen, Y.C. and Ely, Jeffrey and Luo, X. , title =. 2009 , OPTannote =
2009
-
[36]
Awareness Equilibrium , journal =
Jernej. Awareness Equilibrium , journal =. 2007 , OPTkey =
2007
-
[37]
Mimeo, University of Pennsylvania , year =
Jing Li , title =. Mimeo, University of Pennsylvania , year =
-
[38]
Games and Economic Behavior , year =
Emel Filiz-Ozbay , title =. Games and Economic Behavior , year =
-
[39]
Mimeo, University of Maryland , year =
Emel Filiz-Ozbay , title =. Mimeo, University of Maryland , year =
-
[40]
Mimeo, University of Maryland , year =
Erkut Ozbay , title =. Mimeo, University of Maryland , year =
-
[41]
Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability , pages =
David Blackwell , title =. Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability , pages =. 1951 , OPTeditor =
1951
-
[42]
Halpern and Leandro Chaves R\^
Joseph Y. Halpern and Leandro Chaves R\^. Interactive unawareness revisited , OPTcrossref =. Theoretical Aspects of Rationality and Knowledge: Proc. Tenth Conference , pages =. 2005 , OPTeditor =
2005
-
[43]
Halpern and Leandro Chaves R\^
Joseph Y. Halpern and Leandro Chaves R\^. Interactive unawareness revisited , journal =. 2008 , OPTkey =
2008
-
[44]
Halpern and Leandro Chaves R\^
Joseph Y. Halpern and Leandro Chaves R\^. Extensive games with possibly unaware players , OPTcrossref =. Proc. Fifth International Joint Conference on Autonomous Agents and Multiagent Systems , pages =. 2006 , OPTeditor =
2006
-
[45]
Halpern and Leandro C
Joseph Y. Halpern and Leandro C. R\^ e go. Extensive games with possibly unaware players. Mathematical Social Sciences. 2014
2014
-
[46]
2007 , OPTannote =
Siyang Xiong , title =. 2007 , OPTannote =
2007
-
[47]
The Journal of Economic Perspectives , year =
John Geanakoplos , title =. The Journal of Economic Perspectives , year =
-
[48]
Journal of Economic Theory , year =
Paul Milgrom and Nancy Stokey , title =. Journal of Economic Theory , year =
-
[49]
1988 , OPTannote =
Adam Brandenburger and Eddie Dekel and John Geanakoplos , title =. 1988 , OPTannote =
1988
-
[50]
Journal of the American Statistical Association , year =
James Sebenius and John Geanakoplos , title =. Journal of the American Statistical Association , year =
-
[51]
2005 , OPTannote =
Itai Sher , title =. 2005 , OPTannote =
2005
-
[52]
Journal of Risk and Uncertainty , year =
Zvi Safra and Eyal Sulganik , title =. Journal of Risk and Uncertainty , year =
-
[53]
Theory and Decision , year =
Edward Schlee , title =. Theory and Decision , year =
-
[54]
Journal of Behavioral Decision Making , year =
Peter Wakker , title =. Journal of Behavioral Decision Making , year =
-
[55]
Logic and the Foundations of Game and Decision Theory, Proceedings of the Seventh Conference , OPTpages =
Oliver Board and Kim-Sau Chung , title =. Logic and the Foundations of Game and Decision Theory, Proceedings of the Seventh Conference , OPTpages =. 2007 , editor =
2007
-
[56]
Econometrica , year =
Stephen Morris , title =. Econometrica , year =
-
[57]
Games and Economic Behavior , year =
Dov Samet , title =. Games and Economic Behavior , year =
-
[58]
Journal of Economic Theory , year =
Man-Chung Ng , title =. Journal of Economic Theory , year =
-
[59]
1996 , OPTannote =
Giacomo Bonanno and Klaus Nehring , title =. 1996 , OPTannote =
1996
-
[60]
1996 , OPTkey =
Yossi Feinberg , title =. 1996 , OPTkey =
1996
-
[61]
1995 , OPTannote =
Yossi Feinberg , title =. 1995 , OPTannote =
1995
-
[62]
The Economics of Uncertainty and Information , publisher =
Jean-Jacques Laffont , ALTeditor =. The Economics of Uncertainty and Information , publisher =. 1989 , OPTkey =
1989
-
[63]
Review of Economic Studies , year=2012, volume=
Ernst-Ludwig Von Thadden and Xiaojian Zhao , title=. Review of Economic Studies , year=2012, volume=
2012
-
[64]
Mimeo, University of Mannheim , year=
Ernst-Ludwig Von Thadden and Xiaojian Zhao , title=. Mimeo, University of Mannheim , year=
-
[65]
B. A. Davey and H. A. Priestley , ALTeditor =. Introduction to Lattices and Order , publisher =. 1990 , OPTkey =
1990
-
[66]
Chellas , ALTeditor =
Brian F. Chellas , ALTeditor =. Modal logic: An intorduction , publisher =. 1980 , OPTkey =
1980
-
[67]
Aumann , title =
Robert J. Aumann , title =. International Journal of Game Theory , year =
-
[68]
Journal of Economic Theory , year =
Spyros Galanis , title =. Journal of Economic Theory , year =
-
[69]
2006 , OPTannote =
Tomasz Sadzik , title =. 2006 , OPTannote =
2006
-
[70]
Schipper , title =
Aviad Heifetz and Martin Meier and Burkhard C. Schipper , title =. Games and Economic Behavior , year =
-
[71]
Schipper , title =
Aviad Heifetz and Martin Meier and Burkhard C. Schipper , title =. Mimeo , year =
-
[72]
Zhao , title =
Xiaojian J. Zhao , title =. Rationality and Society , year =
-
[73]
2001 , OPTannote =
Christian Ewerhart , title =. 2001 , OPTannote =
2001
-
[74]
Theory and Decision , year =
Spyros Galanis , title =. Theory and Decision , year =
-
[75]
Economic Theory , year =
Salvatore Modica and Aldo Rustichini and Jean-Marc Tallon , title =. Economic Theory , year =
-
[76]
2008 , OPTannote =
Jing Li , title =. 2008 , OPTannote =
2008
-
[77]
Jerome Keisler , title =
Adam Brandenburger and Amanda Friedenberg and H. Jerome Keisler , title =. Econometrica , year =
-
[78]
Games and Economic Behavior , year =
Larry Samuelson , title =. Games and Economic Behavior , year =
-
[79]
Journal of Economic Theory , year =
Eddie Dekel and Drew Fudenberg , title =. Journal of Economic Theory , year =
-
[80]
Economic Analysis of Markets and Games , OPTcrossref =
Adam Brandenburger , title =. Economic Analysis of Markets and Games , OPTcrossref =. 1992 , editor =
1992
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.