REVIEW 4 major objections 5 minor 14 references
Cross-Entropy Games for Language Models: From Implicit Knowledge to General Capability Measures
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper claims that an LLM's implicit knowledge can be elicited by Cross-Entropy Games, and that a minimal, axiomatically defined family of such games yields capability measures and a path to general capability measures.
desk verdict A serious and novel formal framework for LLM evaluation games, but the general-capability measure it advertises rests on unproven connectivity assumptions. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a signed cross-entropy sum: an expression $\sum_i \sigma_i \, \mathrm{xent}(s_i|t_i)$ with $\sigma_i \in \{\pm 1\}$, where $\mathrm{xent}(s|t)$ is the judge model's negative log-probability of string $s$ following string $t$. These sums are the only reward and constraint primitives of Xent Games, and the characterization theorem shows that they are exactly what the three stability axioms, applied to the reverse-prompt game, generate. The other moving parts are cat/cut string updates, the transfer value $V_{G_1}(G_2)$ defined through fine-tuning gains, and the normalization of raw scores by a base model's play-iteration counts.
What would settle it
Compute the directed graph whose edges are $G \to G'$ when $V_G(G') \ge \alpha$ for a fixed base model, judge, and fine-tuning budget, over a large corpus of XGL games; if the graph has more than one nontrivial connected component, or if some playable game lies in no component reachable from a small finite seed set, the connectivity assumption is false and the scope-growth algorithm's coverage guarantee does not hold.
Extended reading notes
Core claim
The central claim is a characterization theorem: perfect-information Cross-Entropy Games form the smallest family of string-space games that contains the reverse-prompt game, supports cat/cut string updates, and is stable under sequential composition, adversarial rescaling, and zero-summing. The reverse-prompt game asks a player to produce a short string $t$ that makes a target string $s$ likely, i.e. to minimize the judge model's cross-entropy $\mathrm{xent}(s|t)$; from this single game the four operations generate all signed cross-entropy rewards and constraints, and hence games as varied as summarization, anomaly detection, chess-like play, and mathematical proof debate. The paper then builds a benchmarking structure on this game space: playable games are those where models improve with repeated attempts or fine-tuning, transfer value $V_{G_1}(G_2)$ measures how much skill gained in one game helps another, and Xent Measures are histograms of normalized scores over a minimal covering family of a chosen scope. For the unbounded scope of general ability, a greedy scope-growth algorithm is proposed to explore the game space in an evolution-inspired way.
Load-bearing premise
The load-bearing premise is that every playable Xent Game can be reached from a finite starter set by a chain of games in which each step transfers a nontrivial amount of skill ($\alpha$-connectivity); this is assumed, not established, and the scope-growth algorithm's coverage guarantee fails if it is false.
Editorial extensions
If this is right
- If the characterization theorem is correct, every perfect-information Xent Game can be synthesized from the reverse-prompt game by the four allowed operations, so benchmark designers do not need an ad hoc list of tasks.
- A Xent Measure over a scope is a histogram of normalized scores, so benchmark output can be read as the fraction of a task family a model can handle at each skill level rather than one opaque scalar.
- With scores normalized by a base model's iteration counts, a Xent Game score is expressed as attempts saved relative to the base model, making scores meaningfully comparable across games.
- Under the stated connectivity and finite-escaping-directions assumptions, the scope-growth algorithm asymptotically covers the space of playable games, offering a concrete route out of the unbounded-scope problem for general capability measures.
Reading between the lines
- Beyond the paper: the judge model is the real bottleneck, because all rewards and constraints pass through one cross-entropy measure; a judge that is weak or miscalibrated on a region of text can silently make games ill-posed, so benchmark validity is conditional on judge quality in a way the paper does not quantify.
- Beyond the paper: the connectivity assumption could be probed empirically before building benchmarks by measuring transfer values over a corpus of games; the paper treats it as a postulate, not a measured fact.
- Beyond the paper: if the meta-sampling process is itself an LLM, the framework closes into a self-improvement loop in which games generated by a model are used to fine-tune it; the paper mentions this as a perspective, not an established result.
- Beyond the paper: a natural test of universality is to run the scope-growth algorithm with two different base models and check whether the resulting rankings of a fixed panel of LLMs agree; the paper expects this but does not prove it.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Cross-Entropy (Xent) Games, a framework in which LLM-based tasks are formulated as finite, turn-based games whose rewards and constraints are signed sums of cross-entropy terms computed by a judge model. The central structural result, Theorem 25, characterizes perfect-information Xent Games as the smallest family of string-space games containing the reverse-prompt game and closed under cat/cut string updates, sequential composition, adversarial rescaling, and zero-summing. Sections 3–4 present examples and define evaluation notions: well-posedness, few-shot and fine-tuning playability, and transfer value. Sections 5–6 propose using Xent Games to build capability benchmarks: a finite family of games (a Xent Measure) is extracted from a scope, and a Scope Growth Algorithm is proposed as a way to grow scopes toward a general capability measure. Numerical experiments report few-shot improvement on a single-text game by four models and transfer-value measurements among three games for Qwen3 8B.
Significance. If the general-capability claims were established, the paper would offer a principled alternative to ad hoc LLM benchmarking, connecting implicit knowledge, compositional game theory, transfer learning, and open-ended evolution. The formal game definitions and Theorem 25 are coherent and provide a clean closure statement; the paper also ships reproducible code in the xega repository and makes explicit falsifiable assumptions, which is a strength. The examples in Section 3 illustrate a genuinely rich design space. However, the paper's headline claim—that the Scope Growth Algorithm yields general capability measures—rests on unproven and nontrivial assumptions, and the numerical support is narrow and does not directly test those assumptions. The contribution is therefore better assessed as a promising conceptual framework than as an established measurement methodology.
major comments (4)
- [Section 6.1, Assumption 55; Claim 66 in Section 6.3.2] Assumption 55 (α-connectivity of the space of playable Xent Games) is load-bearing for the scope-growth coverage claim, but it is neither derived nor empirically supported. The paper itself labels it a postulate, and Remark 56 concedes that the choice of α is subtle. The toy model in Section 6.1.2 shows the assumption is not automatic: if skills are independent and infinite, a game whose only large skill coordinate lies outside the finite base set has zero transfer from every base game, so the space is not α-connected for any finite Σ0. The authors should either provide evidence for Assumption 55 on a nontrivial family of games, or explicitly restrict the claims to an α-connected component and describe what would be lost.
- [Claim 66 in Section 6.3.2 and the meta-sampling process in Section 6.3.3] Claim 66 asserts that, under Assumptions 55 and 62, the Scope Growth Algorithm covers the space of playable Xent Games as N → ∞, but no proof is given and the claim is not a direct consequence of the stated assumptions. In particular, the algorithm's behavior depends on the meta-sampling process Λ, yet no condition on Λ is stated. A degenerate Λ that always returns games from a small neighborhood of the current scope, or that samples from a fixed finite pool, would cause the algorithm to terminate while leaving most of the α-connected component unreached. A formal coverage guarantee needs a precise condition on Λ (e.g., that the sampled games are dense in the relevant metric, or that each 'escaping direction' is sampled with positive probability).
- [Assumption 62 in Section 6.2.3] Assumption 62, bounding by K the number of 'escaping directions' from any game, is used to guarantee finite termination of the inner loop of Algorithm 1, but it is unproven and its plausibility is not established for the actual Xent Game space. Remark 63 shows that a version of the bound holds in the toy model, but the toy model is not shown to be predictive of the real space of Xent Games, which is much richer and includes compositions and zero-sum transformations. At minimum, the authors should state what K would correspond to in concrete Xent Games and provide an argument or experiment, rather than leaving this as a purely existential assumption.
- [Section 5.4 and Definition 47; Sections 4.2.2 and 4.3.2] The numerical evidence does not yet substantiate the benchmark-measurement claims. The few-shot experiment in Section 4.2.2 uses one simple game and four models, and the transfer-value experiment in Section 4.3.2 uses three games and one model. These results support the modest claim that some Xent Games are playable and exhibit transfer, but they do not test the key mechanisms behind Xent Measures—such as the existence of small optimal covers, the behavior of the normalized score narms, or the convergence of the scope-growth algorithm. The authors should either add experiments on larger families of games that exercise Definitions 42, 45, and 47, or clearly mark Sections 5–6 as a proposal whose validation is deferred.
minor comments (5)
- [Definition 42, Section 5.2.2] The text defines crossing times 'when j exceeds b + jα', but α is not defined in that definition; it should likely be b + jθ, using the increment θ from Definition 33. Please correct the notation.
- [Example 16, Section 2.2.9] The phrase 'shows that shows the (theoretical) impossibility' contains a duplicated 'shows'; please remove the repetition.
- [Section 2.3 and Section 6.1] The informal statement in Section 2.3 that 'the space of Xent Games is connected' is later replaced by the much weaker α-connectedness assumption in Section 6.1. The paper should explicitly note this distinction earlier to avoid giving the impression that Theorem 25 establishes the connectivity used by the scope-growth algorithm.
- [Remark 39 and Section 6.2.3] The universality postulate (independence of the final measures from M, F, T and exploration details) is stated as an expectation in Remark 39 and as a postulate in Section 6.2.3, but it is not derived or tested. Since it is essential for the claim that a single general ability measure is meaningful, it should be listed alongside Assumptions 55 and 62 as an explicit assumption with a discussion of how it could be falsified.
- [Figure 4.1 and Section 4.2.2] The text says the arms 'keeps improving for all models up to k = 17', but the individual game-map results are described as mixed and sometimes flat or negative. Clarify whether the claimed improvement is in the averaged arms across game maps or in every individual map, and make the figure caption state the aggregation procedure.
Circularity Check
No significant circularity: Theorem 25 is a genuine closure theorem; the general-capability claims rest on explicitly stated but unproven assumptions, not on circular reasoning.
full rationale
I reviewed the derivation chain from the definition of Xent Games (Section 2.2) through the characterization theorem (Theorem 25) to the scope-growth algorithm (Section 6). Theorem 25 is a mathematical closure statement, not a circular prediction: the axioms (Definitions 20, 21, 23 plus the reverse-prompt game) are stated independently of the Xent Game definition, and the proof supplies both inclusions by explicit construction (rewards assembled from repeated reverse-prompt games via composition, signed rewards via zero-summing, constraints via adversarial rescaling, and cat/cut by assumption). The axioms were clearly chosen to generate the already-defined space, but the theorem's content—that Xent Games are the smallest such family—is proved by construction rather than assumed. The capability-measure claims rest on explicitly stated assumptions: Assumption 55 (α-connectivity of playable Xent Games) and Assumption 62 (bounded escaping directions), and the coverage guarantee Claim 66 is asserted rather than proved. These are unsupported premises and would be correctness risks if the framework were deployed, but they are not circular because they are presented as assumptions and claims, not derived from the conclusions they support. The numerical experiments (Sections 4.2.2 and 4.3.2) use external code and models (GPT-4.1, Claude, Gemini, Grok, Qwen3) and demonstrate playability and transfer value; they do not fit a parameter and then rename it as a prediction. Self-citations: [PWH24] and [CGHCL21, CGHCL24] include one of the present authors, but they are used only as examples or inspiration (time-reversal and the Sprig protocol), not as load-bearing uniqueness theorems or as substitutes for the paper's own derivations. No circular step was found.
Assumptions & free parameters
free parameters (5)
- epsilon (well-posedness threshold) =
1/|V| (recommended default)
- playability parameters (N, epsilon, theta, m) =
N=20, m=30 in experiments; θ not fixed globally
- scope growth parameters (alpha, alpha1, alpha2, rho) =
alpha1=0.9, alpha2=0.05, rho=0.1 in Figure 6.1
- fine-tuning hyperparameters (REINFORCE lr, token budget T, corpus sizes) =
lr=1e-6, corpus 20k stories train / 600 eval, judge GPT2-XL
- base model M_B and judge model J =
GPT-2 or GPT-2-XL judge; Qwen3 8B base for fine-tuning
assumptions (7)
- standard math Next-token probabilities define an LLM measure via the product rule P_M{x}=exp(-S_M(x)).
- domain assumption Signed cross-entropy sums are sufficient to express interesting rewards and constraints.
- domain assumption Adversarial rescaling turns any reward into a hard constraint, following a Lagrange-multiplier analogy.
- domain assumption The reverse-prompt game plus cat/cut updates and closure properties generate all Xent Games.
- ad hoc to paper Assumption 55: the space of playable Xent Games is α-connected for some α>0.
- ad hoc to paper Assumption 62: each game has at most K escaping directions.
- ad hoc to paper Universality: good choices of M, F, T and exploration details yield roughly the same general ability measures.
Cite this review
Pith. "Pith review of Cross-Entropy Games for Language Models: From Implicit Knowledge to General Capability Measures." pith.science (2026). https://pith.science/paper/RRDKONYN
@misc{pith2026250606832,
author = {Pith},
title = {Pith review of: Cross-Entropy Games for Language Models: From Implicit Knowledge to General Capability Measures},
year = {2026},
howpublished = {\url{https://pith.science/paper/RRDKONYN}},
note = {Machine review of arXiv:2506.06832}
}
read the original abstract
Large Language Models (LLMs) define probability measures on text. By considering the implicit knowledge question of what it means for an LLM to know such a measure and what it entails algorithmically, we are naturally led to formulate a series of tasks that go beyond generative sampling, involving forms of summarization, counterfactual thinking, anomaly detection, originality search, reverse prompting, debating, creative solving, etc. These tasks can be formulated as games based on LLM measures, which we call Cross-Entropy (Xent) Games. Xent Games can be single-player or multi-player. They involve cross-entropy scores and cross-entropy constraints, and can be expressed as simple computational graphs and programs. We show the Xent Game space is large enough to contain a wealth of interesting examples, while being constructible from basic game-theoretic consistency axioms. We then discuss how the Xent Game space can be used to measure the abilities of LLMs. This leads to the construction of Xent Game measures: finite families of Xent Games that can be used as capability benchmarks, built from a given scope, by extracting a covering measure. To address the unbounded scope problem associated with the challenge of measuring general abilities, we propose to explore the space of Xent Games in a coherent fashion, using ideas inspired by evolutionary dynamics.
Figures
Figures from the paper (15 more)
Reference graph
Works this paper leans on
-
[1]
[Ada23] A.A. Team, J. Bauer, . Baumli, S. Baveja, F. Behbahani, A. Bhoopchand, N. Bradley-Schmieg, M. Chang, N. Clay, A. Collister, V. Dasagi, L. Gonzalez, K. Gregor, E. Hughes, S. Kashem, M. Loks-Thompson, H. Openshaw, J. Parker-Holder, S. Pathak, N. Perez-Nieves, N. Rakicevic, T. Rocktäschel, Y. Schroecker, J. Sygnowski, K. Tuyls, S. York, A. Zacherl, a...
arXiv 2000
-
[5]
https://arxiv.org/abs/2406.04268 [KGRMI22] T. Kojima, S.S. Gu, M. Reid, Y. Matsuo, Y. Iwasawa, Large Language Models are Zero-Shot Reasoners, https://arxiv.org/abs/2205.11916 [Lal2015] K.N. Laland, T. Uller, M.W. Feldman, K. Sterelny, B.B. Müller, A. Moczek, E. Jablonka, and J. Odling-Smee, The extended evolutionary synthesis: its structure, assumptions a...
-
[8]
https://arxiv.org/abs/2401.17505,
-
[9]
41 [PBS16] E. Parisotto, J.L. Ba, R. Salakhutdinov, Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning, Interna- tional Conference on Learning Representations, 2016, https://arxiv.org/abs/1511.06342. [PCDSC24] Ji-Lun Peng, Sijia Cheng, Egil Diau, Yung-Yu Shih, Po-Heng Chen, Yen-Ting Lin, Yun-Nung Chen, A Survey of Useful LLM Evaluation, https...
arXiv 2016
-
[10]
https://sferics.idsia.ch/pub/juergen/gmAGI.pdf [Schmi10] J. Schmidhuber. Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990-2010). IEEE Transactions on Au- tonomous Mental Development, 2(3):230-247,
work page 1990
-
[1950]
[WXSLD22] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, D. Zhou, Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, Proceedings of NeurIPS 2022, https://arxiv.org/abs/2201.11903. [WWZ24] Z. Wang, P. Wang, K. Liu, P. Wang, Y. Fu, C.-T. Lu, C.C. Aggarwal, J. Pei, Y. Zhou, A Comprehensive Survey on Data Augme...
arXiv 2022
-
[1993]
https://dl.acm.org/doi/10.1145/168304.168306 [Hug24] E. Hughes, M.D. Dennis, J. Parker-Holder, F. Behbahani, A. Mavalankar, Y. Shi, T. Schaul, T. Rocktäschel, Open- Endedness is Essential for Artificial Superhuman Intelligence,Proceedings of the 41st International Conference on Ma- chine Learning, PMLR 235:20597-20616,
-
[2007]
https://www.cs.utexas.edu/~ai-lab/pubs/IJCAI07-bikram.pdf [BCK23] J. Blüm, J. Czech, and K. Kersting, AlphaZe ∗∗: AlphaZero-like baselines for imperfect information games are sur- prisingly strong, Front. Artif. Intell., 12 May 2023, Sec. Machine Learning and Artificial Intelligence, Volume 6, https://doi.org/10.3389/frai.2023.1014561, 2023 [CGHCL21] S.Ca...
arXiv 2023
Show all 14 references
-
[2010]
IEEE, https://people.idsia.ch/~juergen/ieeecreative.pdf. [SYCYL24] Chufan Shi, Haoran Yang, Deng Cai, Zhisong Zhang, Yifan Wang, Yujiu Yang, Wai Lam, A Thorough Examination of Decoding Methods in the Era of LLMs, https://arxiv.org/abs/2402.06925 [SYSKZ24] SanchitSinha, Yuguang...
2024 arXiv
-
[2011]
Liang, R
[LBL23] P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cosgrove, C. D. Manning, C. Ré, D. Acosta-Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Ya...
2015 arXiv
-
[2015]
Lehman and K.O
[LeSt10] J. Lehman and K.O. Stanley, Efficiently evolving programs through the search for novelty,GECCO ’10: Proceedings of the 12th annual conference on Genetic and evolutionary computation, 2010, https://doi.org/10.1145/1830483.1830638. [LeSt11a] J. Lehman and K.O. Stanley. ...
2010
-
[2018]
[SWLL24] Y. Song, G. Wang, S. Li, B.Y. Lin, The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism, https://arxiv.org/abs/2407.10457 [StBy19] Felix Stahlberg and Bill Byrne. On NMT search errors and model errors: Cat got your tongue? Proceeding...
2019 arXiv
-
[2019]
URL https://aclanthology.org/D19-1331
doi: 10.18653/v1/D19-1331. URL https://aclanthology.org/D19-1331. [SSSJ25] Z. Stojanovski, O. Stanley, J. Sharratt, R. Jones, A. Adefioye, Jean Kaddour, Andreas Köpf, REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards, https://arxiv.org/ab...
-
[2024]
40 [CWWWX23] Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, Xing Xie, A Survey on Evaluation of Large Language Models, https://arxiv.org/abs/2307.03...
1911 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.