REVIEW 4 major objections 4 minor 29 references
Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read IIT consciousness metrics fail on LLM hidden states
desk verdict A large, transparent, but methodologically over-constructed attempt to apply IIT to LLM hidden states; the negative finding is uninterpretable until the Representation Network is validated against surrogate nulls. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Representation Network (RN), a hypothetical four-node network derived from each LLM's hidden states by PCA-reducing token representations to four dimensions, z-scoring and binarizing each node, and concatenating response representations until the binary time series satisfies Markov and conditional-independence assumptions. IIT 3.0 and 4.0 metrics are computed from the transition probability matrix of this RN, while Span Representations, built by concatenating boundary vectors, differences, and element-wise products, serve as a consciousness-independent baseline. The comparison between IIT estimates and Span Representations determines the paper's verdict on whether Theory of Mind differences reflect integrated information or ordinary representational geometry.
What would settle it
Compute $\Phi$ for the same Layer-32 Mixtral sequence after randomly permuting the 4,096 embedding dimensions before PCA: if the reported positive case persists unchanged, then it is an artifact of dimension ordering rather than a property of the representations; if it vanishes, the negative conclusion is supported.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is a qualified negative result: sequences of LLM representations do not carry robust, statistically significant IIT-based consciousness indicators. The study computes weighted averages of $\Phi^{\max}$ and Conceptual Information from IIT 3.0, and $\Phi$ and $\Phi$-structure from IIT 4.0, for each score category of each Theory of Mind stimulus across layers, linguistic spans, and permutation controls. Under temporal permutation controls, no case met all three criteria; under spatial permutation controls, a small set did, including Layer 32 (the last layer) of Mixtral-8x7B on Strange Stories under IIT 4.0 for both the Entire and Complement linguistic spans. The paper claims that variations in Theory of Mind score categories are more likely attributed to span-level information of the LLM representation sequence than to a consciousness phenomenon suggested by IIT estimates.
Load-bearing premise
The load-bearing premise is that a simplified four-node network obtained by compressing, thresholding, and concatenating LLM hidden states is a valid substrate for measuring integrated information; if this preprocessing destroys or creates structure, the $\Phi$ values measure the construction procedure rather than the representations.
Editorial extensions
If this is right
- If the paper is right, IIT-based $\Phi$ estimates computed from LLM hidden states should not be cited as evidence of machine consciousness.
- Theory of Mind performance differences in LLM representations are, under temporal permutation controls, better explained by span-level representational geometry than by integrated-information metrics.
- The spatial-permutation results indicate that apparent positive cases are fragile and may depend on the arbitrary ordering of embedding dimensions, so any positive signal must survive permutation controls.
- The absence of significant $\Phi$ differences across score categories suggests that fixed-parameter next-token-prediction models do not encode integration-like structure in the way IIT would require.
Reading between the lines
- A direct extension would be to apply the same pipeline to synthetic binary time series with known integration structure: if the pipeline fails to rank known-high-$\Phi$ sequences above known-low-$\Phi$ sequences, the negative result is an artifact of the construction.
- The fact that spatial permutation, which randomly reorders embedding dimensions before PCA, produces the only fully qualifying case suggests the $\Phi$ estimates are not invariant under node relabeling; this could be checked directly by computing $\Phi$ after deterministic dimension shuffles.
- If the Theory of Mind dataset were replaced by stimuli with controlled linguistic complexity, one could separate the contribution of stimulus features from any candidate consciousness signal.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript applies Integrated Information Theory (IIT 3.0 and 4.0) to sequences of Transformer-based LLM hidden states derived from Theory of Mind (ToM) test responses. It constructs a "Representation Network" (RN) by attending response representations to stimulus representations, reducing the dimensionality to four via PCA, binarizing each node, concatenating and augmenting responses to reach at least 1,000 words, and then selecting the concatenation that best satisfies Markov and conditional-independence assumptions. The authors compute IIT metrics (Phi_max, Phi, Conceptual Information, and Phi-structure) and compare them with span representations, which are independent of IIT. The main finding is that under temporal permutation controls no case satisfies all three criteria for a potential"consciousness" phenomenon, while under spatial permutation controls a small number of cases do, including Layer 32 of Mixtral-8x7B on Strange Stories (2 scores) with IIT 4.0.
Significance. The paper is unusually candid about its limitations and provides a large-scale empirical effort (165,365 valid samples) with code and processed data. The independent span-representation comparator is a commendable design choice that partially insulates the headline negative result from the specific choice of IIT metrics. If the negative conclusion were robust, it would be a useful piece of evidence against simple claims of "consciousness" in LLM representations. However, the IIT estimates are computed from a heavily constructed statistic, and the paper does not supply a surrogate-null test to rule out pipeline artifacts. The significance is therefore conditional: the result is interesting and reproducible in principle, but the current pipeline does not establish that the computed quantities measure representation content rather than the construction procedure.
major comments (4)
- [Sec. 2.2.7, Eq. (7)] Equation (7) identifies Phi_max (IIT 3.0) with the sum of per-mechanism Conceptual Information values. In IIT 3.0 as implemented in PyPhi, the system-level Phi is the distance between the cause-effect structure and its partitioned counterpart, not the sum of mechanism-level CI values. Because the paper's headline comparisons are labeled Phi_max (IIT 3.0), this is a load-bearing mischaracterization of the computed quantity. The manuscript should either correct the equation or clearly state that a different, non-standard quantity is being used.
- [Sec. 2.2.7] The text states that "for the sake of computational efficiency, we evaluated only the full-network subset of each RN at specific states." This means the reported Phi_max was never maximized over subsystems, and for IIT 3.0 not over all partitions. The quantity is therefore not Phi_max in the IIT sense; it is at best the Phi of a full 4-node network in a particular state. All cross-score comparisons in Sec. 3.2 and Sec. 3.4 should be re-framed accordingly, since the abstract and conclusions use the term Phi_max without this caveat.
- [Secs. 2.2.4–2.2.6] The 4-node binary time series is assembled through a sequence of free choices: PCA dimensionality, mask values, the arbitrary 1,000-word augmentation target, node-specific binarization thresholds, the token-count grid, and selection of the concatenation that best passes Markov and conditional-independence tests. The paper does not provide a surrogate-null test that applies the whole construction to permuted or noise input. Without such a control, both the absence of effects under temporal permutation and the presence of effects under spatial permutation could be artifacts of the construction procedure, so the central negative result is not yet established.
- [Sec. 2.2.8] The spatial permutation control, as described, appears to be a no-op given the PCA step. If the embedding dimension is permuted before PCA, the resulting PCA scores are unchanged up to a reordering of the principal axes; if the four PCA components are permuted after reduction, the transformation merely relabels nodes, and IIT measures are invariant under such relabeling. The control therefore cannot support the claim that the spatial-permutation positives reflect latent node structure. The authors should either describe a permutation that actually disrupts the data or provide a control that directly randomizes the node time series after PCA.
minor comments (4)
- [Sec. 2.2.2] The claim that the mask values (1.0, 0.6, 0.2) provide "greater distinguishability" than alternatives is asserted without a quantitative criterion; the selection should be justified by a concrete measure or sensitivity analysis.
- [Sec. 3.4 and Sec. 3.6] The layer indexing is confusing: the paper refers to "Layer 32 (indexed at 11)" and also to the "2/3 layer"; the mapping from sampled indices to actual layer numbers should be made consistent and explicit for each model.
- [Abstract and Sec. 3.6] The manuscript uses both "spatio-permutational" and "spatio permutation" inconsistently; a single consistent term should be used throughout.
- [References] Several citations are informal or incomplete (e.g., "LBC, 2025" and "Yann, 2024"); these should be converted to standard journal-style references.
Circularity Check
Spatio-permutation control is identity under PCA, so the reported 'spatio-permutational' consciousness indicators are the unpermuted results by construction.
-
other
[Sec. 2.2.8 (spatio control) combined with Sec. 2.2.5 (PCA reduction)]
"For the spatio control procedure, we randomly permuted the time series along the embedding dimension for each LLM to randomize the arrangements of the embedding order. ... we reduced the embedding dimension D to4 for the concatenated (C)ARR within each iteration i using Principal Component Analysis (PCA)."
Permuting embedding dimensions is a column permutation of the token-by-dimension matrix. PCA scores are invariant to column permutation up to sign: if X' = X P, then X' = U S (P^T V)^T, so the scores X' V' equal X V. Thus every 'spatio' seed produces the same 4-node binarized time series, the same TPM, and identical Phi, CI, and Phi-structure values as the unpermuted data. The 'spatio-permutational' cases, including the Layer-32 Mixtral result, are therefore the original unpermuted results by construction; the control is the identity operation, and the claimed contrast between spatio and temporal controls is actually a contrast between original-order and concatenation-permuted data, mislabeled as probing embedding-dimension order.
full rationale
The paper's central negative claim is not circular: IIT estimates are computed from an explicitly constructed Representation Network and compared across ToM score categories against an external dataset and PyPhi, with a span-representation benchmark derived from the same reduced data as a non-consciousness comparator. The expectation of higher Phi for higher ToM scores is a theory-derived hypothesis, not a fitted parameter, and the temporal-permutation null is a genuine reordering of the concatenated responses. No load-bearing self-citations are present, and the decomposition of Phi into CI and Phi-structure is transparent rather than used as an independent derivation. However, the spatio-permutation control is a no-op by construction because PCA is column-permutation invariant, so the 'intriguing patterns under spatio-permutational analyses' highlighted in the abstract are the unpermuted data patterns relabeled as a control result. This makes one part of the positive finding equivalent to its input by construction, while the main negative result retains independent content; hence a moderate circularity score.
Assumptions & free parameters
free parameters (8)
- PCA dimensionality =
4 nodes
- CARR mask values =
1.0 (target span), 0.6 (context), 0.2 (non-relevant)
- Augmentation target length =
1000 words
- Token-count grid =
50 to 1000 in steps of 50
- Criterion 1 'good' threshold =
80%
- Number of sampled transformer layers =
12 plus the 2/3 layer
- Node-specific binarization thresholds =
per-node time series mean
- Permutation replicates =
10 per control
assumptions (7)
- domain assumption IIT axioms: existence, composition, information, integration, exclusion
- domain assumption Higher Φ implies a higher degree of consciousness under comparable conditions
- ad hoc to paper ToM performance score monotonically tracks the degree of consciousness in the RN
- domain assumption PCA-reduced, binarized LLM hidden states form a Markovian binary causal system
- domain assumption Concatenating responses from different subjects plus LLM-augmented text yields a meaningful per-score time series
- ad hoc to paper The Representation Network is a valid substrate for IIT estimates
- standard math Standard statistical machinery
invented entities (1)
-
Representation Network (RN)
Cite this review
Pith. "Pith review of Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis." pith.science (2026). https://pith.science/paper/FY2VQR3J
@misc{pith2026250622516,
author = {Pith},
title = {Pith review of: Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis},
year = {2026},
howpublished = {\url{https://pith.science/paper/FY2VQR3J}},
note = {Machine review of arXiv:2506.22516}
}
abstract
Integrated Information Theory (IIT) provides a quantitative framework for explaining consciousness phenomenon, positing that conscious systems comprise elements integrated through causal properties. We apply IIT 3.0 and 4.0 -- the latest iterations of this framework -- to sequences of Large Language Model (LLM) representations, analyzing data derived from existing Theory of Mind (ToM) test results. Our study systematically investigates whether the differences of ToM test performances, when presented in the LLM representations, can be revealed by IIT estimates, i.e., $\Phi^{\max}$ (IIT 3.0), $\Phi$ (IIT 4.0), Conceptual Information (IIT 3.0), and $\Phi$-structure (IIT 4.0). Furthermore, we compare these metrics with the Span Representations independent of any estimate for consciousness. This additional effort aims to differentiate between potential "consciousness" phenomena and inherent separations within LLM representational space. We conduct comprehensive experiments examining variations across LLM transformer layers and linguistic spans from stimuli. Our results suggest that sequences of contemporary Transformer-based LLM representations lack statistically significant indicators of observed "consciousness" phenomena but exhibit intriguing patterns under $\textit{spatio}$-permutational analyses. The Appendix and code are available as Supplementary Materials at: https://doi.org/10.1016/j.nlp.2025.100163.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[4]
URL:https://www.eneuro.org/content/4/5/ENEURO.0085-17.2017, doi:10.1523/ENEURO.0085-17.2017, arXiv:https://www.eneuro.org/content/4/5/ENEURO.0085-17.2017.full.pdf. Hofstadter, D.R.,
-
[8]
Could a large language model be conscious? arXiv preprint arXiv:2303.07103 . Chalmers, D.J., et al.,
-
[11]
arXiv preprint arXiv:2412.04571
Dissociating artificial intelligence from artificial consciousness. arXiv preprint arXiv:2412.04571 . Fleming,S.,Frith,C.,Goodale,M.,Lau,H.,LeDoux,J.E.,Lee,A.L.,Michel,M.,Owen,A.,Peters,M.A.,Slagter,H.A.,. Theintegratedinformation theory of consciousness as pseudoscience . Fletcher, P.C., Happe, F., Frith, U., Baker, S.C., Dolan, R.J., Frackowiak, R.S., F...
-
[12]
arXiv preprint arXiv:1410.5401
Neural turing machines. arXiv preprint arXiv:1410.5401 . Griffin, R., Friedman, O., Ween, J., Winner, E., Happé, F., Brownell, H.,
-
[16]
arXiv preprint arXiv:2401.04088
Mixtral of experts. arXiv preprint arXiv:2401.04088 . Jolliffe, I.T., Cadima, J.,
- [17]
-
[20]
URL: https://arxiv.org/abs/2402.06196, arXiv:2402.06196
Large language models: A survey. URL: https://arxiv.org/abs/2402.06196, arXiv:2402.06196. Müller, R.A., Fishman, I.,
- [22]
Show all 29 references
-
[23]
An analysis of encoder representations in transformer-based machine translation, in: Proceedings of the 2018 EMNLP workshop BlackboxNLP: analyzing and interpreting neural networks for NLP, pp. 287–297. Rubner, Y., Tomasi, C., Guibas, L.J.,
2018
-
[26]
arXiv preprint arXiv:2405.18870
Llms achieve adult human performance on higher-order theory of mind tasks. arXiv preprint arXiv:2405.18870 . Tenney, I., Xia, P., Chen, B., Wang, A., Poliak, A., McCoy, R.T., Kim, N., Van Durme, B., Bowman, S.R., Das, D., et al.,
-
[27]
arXiv preprint arXiv:1905.06316
What do you learn from context? probing for sentence structure in contextualized word representations. arXiv preprint arXiv:1905.06316 . Tononi, G.,
1905 arXiv
-
[28]
Brain research 1124, 100–110
Neural substrates of sarcasm: A functional magnetic- resonance imaging study. Brain research 1124, 100–110. Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser,L.u.,Polosukhin,I.,2017. Attentionisallyouneed,in:Guyon, I.,Luxburg,U.V.,Bengio,S.,Wallach,H.,Fer...
2017
-
[31]
URL:https://x.com/ylecun/status/1804834054954459539
The problem isn’t that it is a transformer. URL:https://x.com/ylecun/status/1804834054954459539. J. Li: Published as a journal paper at: https://doi.org/10.1016/j.nlp.2025.100163 Page 37 of 37
2025
-
[288]
Languagemodelsarebetterthanhumansatnext-tokenprediction
Shlegeris,B.,Roger,F.,Chan,L.,McLean,E.,2024. Languagemodelsarebetterthanhumansatnext-tokenprediction. Trans.Mach.Learn.Res.. Smith,S.M.,Nichols,T.E.,Vidaurre,D.,Winkler,A.M.,Behrens,T.E.,Glasser,M.F.,Ugurbil,K.,Barch,D.M.,VanEssen,D.C.,Miller,K.L.,2015. A positive-negative mo...
2024
-
[692]
Thetadynamicsrevealdomain-specificcontroloverstimulusandresponseconflict
Nigbur,R.,Cohen,M.X.,Ridderinkhof,K.R.,Stürmer,B.,2012. Thetadynamicsrevealdomain-specificcontroloverstimulusandresponseconflict. Journal of Cognitive Neuroscience 24, 1264–1274. Oizumi, M., Albantakis, L., Tononi, G.,
2012
-
[2001]
Consciousness
Mind reading: neural mechanisms of theory of mind and self-perspective. Neuroimage 14, 170–181. J. Li: Published as a journal paper at: https://doi.org/10.1016/j.nlp.2025.100163 Page 36 of 37 Observing Potential "Consciousness" from LLM Representations Wang, A.T., Lee, S.S., S...
2025
-
[2006]
Laterality 11, 195–225
Theory of mind and the right cerebral hemisphere: Refining the scope of impairment. Laterality 11, 195–225. Haun,A.M.,Oizumi,M.,Kovach,C.K.,Kawasaki,H.,Oya,H.,Howard,M.A.,Adolphs,R.,Tsuchiya,N.,2017. Consciousperceptionasintegrated information patterns in human electrocorticog...
2017
-
[2007]
Basic books
I am a strange loop. Basic books. Jawahar,G.,Sagot,B.,Seddah,D.,2019. Whatdoesbertlearnaboutthestructureoflanguage?,in:Proceedingsofthe57thAnnualMeetingofthe Association for Computational Linguistics, pp. 3651–3657. Jiang, A.Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, ...
2019
-
[2009]
Behavioral and Brain Sciences 32, 121–138
How we know our own minds: The relationship between mindreading and metacognition. Behavioral and Brain Sciences 32, 121–138. doi:10.1017/S0140525X09000545. Caucheteux, C., Gramfort, A., King, J.R.,
-
[2014]
arXiv preprint arXiv:1409.0473
Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 . Baron-Cohen, S.,
-
[2016]
Distill 1, e2
How to use t-sne effectively. Distill 1, e2. Wei,J.,Tay,Y.,Bommasani,R.,Raffel,C.,Zoph,B.,Borgeaud,S.,Yogatama,D.,Bosma,M.,Zhou,D.,Metzler,D.,etal.,2022. Emergentabilities of large language models. arXiv preprint arXiv:2206.07682 . Winner,E.,Brownell,H.,Happé,F.,Blum,A.,Pincus...
2022 arXiv
-
[2017]
arXiv preprint arXiv:1709.08568
The consciousness prior. arXiv preprint arXiv:1709.08568 . Blankertz, B., Lemm, S., Treder, M., Haufe, S., Müller, K.R.,
-
[2018]
PLoS computational biology 14, e1006343
Pyphi: A toolbox for integrated information theory. PLoS computational biology 14, e1006343. Milligan,K.,Astington,J.W.,Dack,L.A.,2007.Languageandtheoryofmind:Meta-analysisoftherelationbetweenlanguageabilityandfalse-belief understanding. Child development 78, 622–646. Minaee, ...
2007
-
[2019]
Consciousness and cognition 72, 49–59
The unfolding argument: Why iit and other causal structure theories cannot explain consciousness. Consciousness and cognition 72, 49–59. vanDuijn,M.,vanDijk,B.,Kouwenhoven,T.,deValk,W.,Spruit,M.,vanderPutten,P.,2023. Theoryofmindinlargelanguagemodels:Examining performance of 1...
2023
-
[2020]
arXiv preprint arXiv:2005.14165
Language models are few-shot learners. arXiv preprint arXiv:2005.14165 . Bubeck,S.,Chandrasekaran,V.,Eldan,R.,Gehrke,J.,Horvitz,E.,Kamar,E.,Lee,P.,Lee,Y.T.,Li,Y.,Lundberg,S.,etal.,2023. Sparksofartificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv...
2005 arXiv
-
[2021]
Frontiers in psychology 12, 749868
Global workspace theory (gwt) and prefrontal cortex: Recent developments. Frontiers in psychology 12, 749868. Bachmann,G.,Nagarajan,V.,2024. Thepitfallsofnext-tokenprediction,in:Proceedingsofthe41stInternationalConferenceonMachineLearning, pp. 2296–2318. Bahdanau, D., Cho, K.,...
2024
-
[2022]
3082–3101
Analyzing encoded concepts in transformer language models, in: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3082–3101. Saxe,R.,Carey,S.,Kanwisher,N.,2004. Understandingotherm...
2022
-
[2023]
arXiv preprint arXiv:2308.08708
Consciousness in artificial intelligence: insights from the science of consciousness. arXiv preprint arXiv:2308.08708 . Canolty, R.T., Edwards, E., Dalal, S.S., Soltani, M., Nagarajan, S.S., Kirsch, H.E., Berger, M.S., Barbaro, N.M., Knight, R.T.,
-
[2024]
arXiv preprint arXiv:2401.03568
Agent ai: Surveying the horizons of multimodal interaction. arXiv preprint arXiv:2401.03568 . Ethayarajh,K.,2019. Howcontextualarecontextualizedwordrepresentations?ComparingthegeometryofBERT,ELMo,andGPT-2embeddings, in:Inui,K.,Jiang,J.,Ng,V.,Wan,X.(Eds.),Proceedingsofthe2019Co...
2019 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.