Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Large Language Models as Computable Approximations to Solomonoff Induction

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training an LLM by loss minimization is a computable stand-in for Solomonoff induction, this paper argues.

desk verdict A definitional tautology presented as a theorem; the LLM's own distribution is re-derived by construction, and the practical heuristic is a known active-learning idea. read the letter →

arxiv 2505.15784 v1 pith:4B7YWZ4H submitted 2025-05-21 cs.LG cs.AIcs.CL

Jun Wan , Lingrui Mei This is my paper
classification cs.LGcs.AIcs.CL MSC 68Q3068T07
keywords Solomonoffinductionalgorithmicinformationtheorynext-tokenpredictionlanguagemodelingascompressionfew-shotexampleselectionin-contextlearningscalinglawsprefixKolmogorovcomplexity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a large language model trained by next-token loss minimization is a computable stand-in for Solomonoff induction, the idealized rule that predicts the next symbol by weighting all programs that produce the observed prefix. It claims two theorems: loss minimization is equivalent to searching for short programs that generate the training data, so the model induces an approximate Solomonoff prior $\tilde{M}(x)$; and the model's conditional next-token distribution $P_\theta(x_{t+1}|x_{1:t})$ asymptotically tracks the Solomonoff conditional $M(x_{t+1}|x_{1:t})$ up to a sequence-independent prefactor that cancels on normalization. If true, this makes in-context learning, few-shot adaptation, and scaling laws consequences of one convergence theorem rather than separate empirical regularities. The same argument motivates a concrete recipe: for few-shot classification, pick examples the model labels with lowest confidence, which the paper shows improves accuracy across Qwen and Llama models on three text benchmarks.

What carries the argument

The load-bearing construction is a prefix program $f(x,s)=(n(x)^{(2)}, s^{(2)}, e(x)^{(2)})$ whose components are Elias-gamma coded so the program set is prefix-free and can run on a universal prefix Turing machine. The approximate prior $\tilde{M}(x)=\sum_s 2^{-\ell(f(x,s))}$ is a subset of the true Solomonoff prior, and the argument works by tracking how loss minimization compresses $e(x)^{(2)}$, the arithmetic-coded encoding of $x$; shorter encodings translate into shorter programs and hence larger weight in $\tilde{M}(x)$. The paper's encoding-length relation $|e(x_{1:t})^{(2)}|\approx 2t-\sum_{i=1}^{t}\log_2 P(x_i|x_{1:i-1})$ is the step that produces the $t^2/(4(t+1)^2)$ prefactor in the inference theorem.

What would settle it

Take a frozen LLM and an arithmetic coder driven by its next-token probabilities, record $|e(x_{1:t})|$ for many prefixes, and compare the difference $|e(x_{1:t})|-|e(x_{1:t+1})|$ with $-\log_2 P(x_{t+1}|x_{1:t})$; a mismatch, or the absence of the $2t$ term in the measured coding length, would directly refute the identity behind Theorem 3. A second check is to compare normalized rankings of candidate next tokens under $M$ and under $P_\theta$: if the prefactor cancels, the rankings must coincide, so any divergence would show the prefactor or the coding-length model is doing real work.

Watch

Extended reading notes

Core claim

The paper's central claim is that the LLM training objective, read as a program-length optimization, constructs a computable semi-measure that sits inside the Solomonoff prior: every string $x$ is assigned probability $\tilde{M}(x)=\sum_{s}2^{-\ell(f(x,s))}$ over prefix-encoded programs built from model weights, decoding iterations, a seed, and the arithmetic-coded compression $e(x)^{(2)}$. Because the Solomonoff prior sums over all programs, $\tilde{M}(x)\le M(x)$, and the paper argues that lowering the training loss shortens $|e(x)^{(2)}|$, so the gap closes. The second theorem states that for a well-trained model, $M(x_{t+1}|x_{1:t})\approx \frac{t^2}{4(t+1)^2}P_\theta(x_{t+1}|x_{1:t})$ for large context length $t$, with the prefactor independent of the token and therefore removed by normalization. The paper presents this as the first formal bridge between LLM architectures and algorithmic information theory, and it uses the convergence bound for Solomonoff induction to explain why in-context examples, few-shot examples, more parameters, and longer inference horizons all help.

Load-bearing premise

The inference theorem assumes a specific arithmetic coding-length identity, $|e(x_{1:t})^{(2)}| \approx 2t - \sum_{i=1}^t \log_2 P(x_i|x_{1:i-1})$, in which the $2t$ term produces the $t^2/(4(t+1)^2)$ prefactor; if the identity fails, the prefactor is not as stated, and because the prefactor cancels on normalization the substantive claim rests entirely on the model's own next-token probabilities.

Editorial extensions

If this is right

  • In-context learning is explained as the universal prior adapting to a computable target distribution: a carefully designed context makes $M(0|x_{1:t})$ track an arbitrary computable $\mu(0|x_{1:t})$.
  • Few-shot examples accelerate convergence because they raise the weight of the target distribution in the error term of the Solomonoff convergence bound, so the model adapts with fewer tokens.
  • Scaling parameters amounts to approximating the Solomonoff prior more precisely, while inference-time scaling (longer context, more decoding steps) increases $t$ and speeds convergence to the true distribution.
  • Few-shot example selection should prefer low-confidence samples; in the paper's experiments this consistently beats high-confidence selection across SMS spam, emotion, and news classification, with the largest gains on smaller models.
  • Because the prefactor in the inference theorem is token-independent, the operational prediction of the theorem is that normalized LLM next-token probabilities match normalized Solomonoff conditionals, so comparisons between tokens do not depend on the prefactor.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the prefactor truly cancels under normalization, the theorem's empirical content is the ratio identity $|e(x_{1:t})|-|e(x_{1:t+1})|=-\log_2 P(x_{t+1}|x_{1:t})$; a direct measurement of this identity on real LLMs would separate the proportionality claim from the choice of coding scheme.
  • The low-confidence selection rule can be read as an active-learning acquisition function; one testable extension is whether iterating selection and fine-tuning on the chosen examples compounds the gains in a way ordinary random active learning does not.
  • The framework suggests that any compression-based training objective that shortens the description length of the training data should inherit a similar approximation guarantee, so comparing decoder-only transformers, mixture-of-experts, and retrieval-augmented models under the same loss could reveal which architectural features matter beyond loss minimization.
  • Because the convergence argument relies on the target distribution being computable, the paper leaves open whether naturally occurring language distributions are computable in the relevant sense; a testable probe is whether predictive error curves on real text follow the $K(\mu)$-dependent bound.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper claims to establish a formal connection between large language models and Algorithmic Information Theory. It introduces an 'approximate Solomonoff prior' M̄(x) defined via programs built from an LLM's own arithmetic code, proves (Theorem 2) that LLM training minimizes the length of such programs and thereby approximates the Solomonoff prior, and proves (Theorem 3) that next-token prediction Pθ approximates Solomonoff induction with an asymptotic prefactor t²/(4(t+1)²). It then uses these results to explain in-context learning, few-shot learning, and scaling laws, and proposes a low-confidence few-shot example selection strategy, which it evaluates on three text-classification datasets.

Significance. If the central theorems were correct, the paper would offer a unifying theoretical framework for LLM behavior, which would be significant. The paper also contains a practically testable proposal: selecting low-confidence examples for few-shot prompting. However, as detailed below, the main theoretical result (Theorem 3) is definitionally circular because M̄ is constructed from the LLM's own distribution, and the asserted upper bound in Theorem 2 is not established. The empirical study, while suggestive, lacks statistical rigor. The paper does not, in its current form, deliver the claimed formal connection.

major comments (4)
  1. [Section 4.2, Eq. (8) and Theorem 3] The definition of M̄(x) in Eq. (8) sums over programs f(x,s) whose third component is e(x), an arithmetic code for x computed using the LLM's own probabilities Pθ. Consequently, M̄(x1:t+1)/M̄(x1:t) is, up to a token-independent factor, equal to 2^{-|e(x1:t+1)|+|e(x1:t)|}, which by the arithmetic coding identity is Pθ(xt+1|x1:t). This holds without any reference to the Solomonoff prior M. The theorem therefore shows that M̄'s conditional recovers Pθ by construction, not that Pθ approximates M. The nonstandard code-length formula |e(x1:t)| ≈ 2t − Σ log₂ P(xi|x1:i−1) in Eq. (18) is not derived from any standard arithmetic-coding result; standard arithmetic coding gives −Σ log₂ P(xi|x1:i−1) plus at most O(1) bits. Inserting the 2t term merely manufactures the specific prefactor t²/(4(t+1)²), which the authors concede cancels under normalization. Hence the substantial claim of the paper—that next-token prediction approximates Solomonoff induction—is not established.
  2. [Section 4.1, Theorem 2 and Lemma 4] The upper bound M̄(x) ≤ M(x) is asserted after constructing a new universal prefix Turing machine U_F. The proof of Lemma 4 claims the existence of a string s that is not a prefix of any program in F, based only on F being prefix-free; this does not follow from the Kraft inequality for infinite prefix-free sets. More importantly, even if U_F satisfies U_F(f) = U(f) for f ∈ F, the Solomonoff prior M in Eq. (3) is defined with respect to a fixed UTM U, not U_F. The invariance theorem only gives M_{U_F}(x) ≤ c · M_U(x) for some constant c depending on the machines, not the pointwise inequality with constant 1. Finally, the assertion that decreasing the LLM loss makes M̄ approach M is unquantified: no convergence rate, no bound on the approximation error, and no argument that the omitted programs in M are negligible. Theorem 2 as stated is therefore not proven.
  3. [Section 4.3 and 4.4] The explanation of few-shot learning and the derivation of the low-confidence selection criterion are not rigorous. Equation (19) bounds a sum over all contexts of squared prediction errors; it does not directly imply that adding specific in-context examples 'increases μ(x1:t)' and 'accelerates convergence' in the sense used here. The sample-selection heuristic in Section 4.4 is motivated by an analogy (larger difference between M(0|x1:t) and μ(0|x1:t) is beneficial) that is never tied quantitatively to the convergence theorem. Without a formal statement of how the error term at a particular context depends on the selected examples, the proposed strategy remains an intuitive heuristic rather than a consequence of the theory.
  4. [Section 5, Table 1] The experimental results are reported as single accuracies with no error bars, no standard deviations, and no multiple seeds or independent runs. The selection procedure (Algorithm 1) is greedy and deterministic with temperature 0, so the reported numbers may be sensitive to the order of the candidate pool and to the initial prompt; this variability is not assessed. Furthermore, there is no random-selection baseline, so the observed gains cannot be attributed to the specific 'low-confidence' criterion versus merely selecting a non-random subset. The claim that low-confidence selection 'consistently' improves performance is not supported by statistical evidence.
minor comments (5)
  1. [Figure 1 and Notation] The figure contains a typo ('Kolmogrorov'), and the notation is inconsistent: M is used both for the Solomonoff prior and for the model in Algorithm 1, and the overline distinguishing M̄ from M is easy to miss in the text. A unified notation table (Appendix C) is present, but the main text should adhere to it more consistently.
  2. [Equation (10)] Equation (10) writes |e(x)(2)| ≈ |e(x)(2)| + 2 log |e(x)(2)|, which is confusing because the same symbol e(x)(2) appears on both sides. The authors presumably intend to distinguish the Elias-gamma-coded version from the raw arithmetic code; this should be made explicit.
  3. [Section 4.2, Eq. (18)] The formula |e(x1:t)| ≈ 2t − Σ log₂ P(xi|x1:i−1) is stated without proof or citation. Since this formula is load-bearing for the prefactor in Theorem 3, it needs a derivation or a reference to a standard result. As it stands, the 2t term appears ad hoc.
  4. [Section 4.1] The paper cites Delétang et al. [2023] for the claim that minimizing training loss aligns with minimizing the arithmetic-code length |e(x)|. That reference establishes that language models can be used as compressors, but the specific equivalence asserted here is not proven in the manuscript.
  5. [Appendix A] The limitations section appropriately acknowledges that the connection is an approximation, but it does not mention the more fundamental issue that Theorem 3's quantity M̄ is defined in terms of the LLM's own code, which makes the theorem definitional rather than substantive. The authors should address this circularity explicitly.

Circularity Check

2 steps flagged · score 9.0 of 10

Theorem 3 is definitionally circular: Mbar is built from the LLM's own arithmetic code, so its conditional distribution is Pθ by construction, and the 2t code-length term only manufactures a prefactor that cancels.

  1. self definitional [Section 4.1 Eq. 5-8; Section 4.2 Eq. 9 and Eq. 17-18]
    "the compression and encoding component uses the LLM in conjunction with arithmetic coding to losslessly compress the string x, resulting in a binary encoding e(x)(2) ... Next, we define the computable prior M (x) := ∞X s=1 2−ℓ(f (x,s))"

    The program f(x,s) whose lengths define the approximate prior contains e(x), an arithmetic code of x produced with the LLM's own probabilities Pθ. Arithmetic coding gives 2^{−|e(x)|} ≈ Pθ(x) up to an O(1) factor, so Mbar(x) is, by construction, a slowly varying token-independent prefactor times Pθ(x). The conditional Mbar(x_{t+1}|x_{1:t}) = Mbar(x_{1:t+1})/Mbar(x_{1:t}) therefore recovers Pθ(x_{t+1}|x_{1:t}) exactly after normalization. Eq. 9 then labels this ratio 'M(xt+1|x1:t)', so Theorem 3's claimed approximation to Solomonoff induction is an identity built into the definitions, not an independent derivation.

  2. other [Section 4.2, Eq. 18]
    "On the other hand, |e(x1:t)(2)| ≈ 2t − Pt i=1 log2 P (xi | x1:i−1), where P (xi | x1:i−1) is the LLM’s predicted probability for the next token."

    Standard arithmetic coding with model Pθ has length |e(x_{1:t})| ≈ −Σ log2 P(x_i|x_{<i}); there is no 2t term. The inserted 2t is what produces the t²/(4(t+1)²) prefactor in Eq. 11, and the paper concedes this prefactor 'is a value independent of the token, it will automatically cancel out during normalization.' With the standard length formula the same ratio still yields Pθ(x_{t+1}|x_{1:t}). Hence Eq. 18 is a dressed-up identity Pθ≈Pθ, not an approximation to the Solomonoff prior M.

full rationale

The paper's central AIT bridge is definitionally circular. The approximate prior Mbar of Eq. 8 is a sum over programs f(x,s) whose executable content is e(x), the LLM's own arithmetic code for x; therefore Mbar is essentially the LLM's distribution Pθ times a slowly varying factor. Eq. 9 replaces the true Solomonoff prior M by Mbar before proof, and Theorem 3 forms the ratio of Mbar, so the derived conditional is Pθ(x_{t+1}|x_{1:t}) by construction. The nonstandard code-length formula with the 2t term in Eq. 18 is not the arithmetic coding length; it only injects the t²/(4(t+1)²) prefactor, which the authors admit cancels under normalization. The 'Mbar ≤ M' upper bound is obtained by building a new universal machine UF around the paper's own program set F, and the convergence of Mbar to M as loss decreases is asserted rather than quantified. The empirical few-shot selection study is independent and not circular, but it does not support the claimed formal identification of next-token prediction with Solomonoff induction. Score 9 because the central result is forced by definition.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The paper's central claim rests on defining a prior from the LLM's own code, an unproven code-length offset, and qualitative use of Hutter's theorem. The few-shot selection method is a known active-learning heuristic.

free parameters (2)
  • 2t offset in arithmetic code length = not fitted, chosen by hand
    Section 4.2 assumes |e(x1:t)| ≈ 2t − Σ log2 P without derivation; standard arithmetic coding has no such offset. It is introduced to obtain the t^2/(4(t+1)^2) prefactor.
  • number of few-shot examples K = 10
    Chosen in experiments without sensitivity analysis; it affects the reported comparisons though not the theoretical claim.
assumptions (5)
  • standard math Solomonoff prior and Hutter's convergence theorem (Eq. 19) hold as stated
    Relied on in Section 4.3 to explain in-context learning, few-shot learning, and scaling laws.
  • domain assumption An LLM with fixed weights is a deterministic Turing machine and can be internalized into a universal Turing machine
    Used in Section 4.1 to simplify f(x,s) to (n,s,e) via Lemma 5, ignoring finite precision, sampling, and hardware details.
  • domain assumption Minimizing LLM loss is equivalent to minimizing arithmetic coding length
    Borrowed from Delétang et al. and Wan 2025 in Section 4.1 to identify loss minimization with compression.
  • ad hoc to paper The approximate prior Mbar is a sub-sum of the Solomonoff prior over the same universal Turing machine
    Section 4.1 claims 'Clearly Mbar forms a subset of the Solomonoff prior', but Mbar is defined with U_F rather than the U used in M; invariance is not proven for this sub-sum.
  • ad hoc to paper Low-confidence examples accelerate convergence to the target distribution
    Section 4.4 asserts this follows from the convergence theorem, but no bound or derivation is given.
invented entities (1)
  • Approximate Solomonoff prior Mbar(x)
    purpose: Bridge LLM training and inference to AIT; defined as a sum over seeds of 2^{-length(fbar(x,s))}.
    The construct is defined entirely in terms of the LLM's own compression code, so it has no independent falsifiable handle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Large Language Models as Computable Approximations to Solomonoff Induction." pith.science (2026). https://pith.science/paper/4B7YWZ4H

@misc{pith2026250515784,
  author       = {Pith},
  title        = {Pith review of: Large Language Models as Computable Approximations to Solomonoff Induction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4B7YWZ4H}},
  note         = {Machine review of arXiv:2505.15784}
}
read the original abstract

The rapid advancement of large language models (LLMs) calls for a rigorous theoretical framework to explain their empirical success. While significant progress has been made in understanding LLM behaviors, existing theoretical frameworks remain fragmented in explaining emergent phenomena through a unified mathematical lens. We establish the first formal connection between LLM architectures and Algorithmic Information Theory (AIT) by proving two fundamental results: (1) the training process computationally approximates Solomonoff prior through loss minimization interpreted as program length optimization, and (2) next-token prediction implements approximate Solomonoff induction. We leverage AIT to provide a unified theoretical explanation for in-context learning, few-shot learning, and scaling laws. Furthermore, our theoretical insights lead to a principled method for few-shot example selection that prioritizes samples where models exhibit lower predictive confidence. We demonstrate through experiments on diverse text classification benchmarks that this strategy yields significant performance improvements, particularly for smaller model architectures, when compared to selecting high-confidence examples. Our framework bridges the gap between theoretical foundations and practical LLM behaviors, providing both explanatory power and actionable insights for future model development.

Figures

Figures reproduced from arXiv: 2505.15784 by the authors.

Figure 1
Figure 1. Conceptual diagram of our theoretical framework linking LLM processes to AIT. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Hierarchical Solomonoff Induction: An Unbounded Machine Learning Model

    cs.LG 2026-08 conditional novelty 5.0 of 10

    HSI, a hyperprior over all Solomonoff priors, is shown equivalent to Solomonoff Induction while enabling dataset-conditioned prediction and a training-set error bound.

Reference graph

Works this paper leans on

77 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [2]

    SMS Spam Collection

    Tiago Almeida and Jos Hidalgo. SMS Spam Collection . UCI Machine Learning Repository, 2011. DOI : https://doi.org/10.24432/C5CC84

    doi:10.24432/c5cc84 2011

  2. [3]

    Rethinking Semantic Parsing for Large Language Models: Enhancing LLM Performance with Semantic Hints

    Kaikai An, Shuzheng Si, Helan Hu, Haozhe Zhao, Yuchi Wang, Qingyan Guo, and Baobao Chang. Rethinking semantic parsing for large language models: Enhancing llm performance with semantic hints. arXiv preprint arXiv:2409.14469, 2024 a

    work page Pith review arXiv 2024

  3. [4]

    Thread: A logic-based data organization paradigm for how-to question answering with retrieval augmented generation

    Kaikai An, Fangkai Yang, Liqun Li, Junting Lu, Sitao Cheng, Shuzheng Si, Lu Wang, Pu Zhao, Lele Cao, Qingwei Lin, et al. Thread: A logic-based data organization paradigm for how-to question answering with retrieval augmented generation. arXiv preprint arXiv:2406.13372, 2024 b

    arXiv 2024

  4. [5]

    Ultraif: Advancing instruction following from the wild

    Kaikai An, Li Sheng, Ganqu Cui, Shuzheng Si, Ning Ding, Yu Cheng, and Baobao Chang. Ultraif: Advancing instruction following from the wild. arXiv preprint arXiv:2502.04153, 2025

    arXiv 2025

  5. [6]

    Context-dpo: Aligning language models for context-faithfulness

    Baolong Bi, Shaohan Huang, Yiwei Wang, Tianchi Yang, Zihan Zhang, Haizhen Huang, Lingrui Mei, Junfeng Fang, Zehao Li, Furu Wei, et al. Context-dpo: Aligning language models for context-faithfulness. arXiv preprint arXiv:2412.15280, 2024 a

    arXiv 2024

  6. [7]

    Decoding by contrasting knowledge: Enhancing llms' confidence on edited facts

    Baolong Bi, Shenghua Liu, Lingrui Mei, Yiwei Wang, Pengliang Ji, and Xueqi Cheng. Decoding by contrasting knowledge: Enhancing llms' confidence on edited facts. arXiv preprint arXiv:2405.11613, 2024 b

    arXiv 2024

  7. [8]

    Is factuality enhancement a free lunch for llms? better factuality can lead to worse context-faithfulness

    Baolong Bi, Shenghua Liu, Yiwei Wang, Lingrui Mei, Junfeng Fang, Hongcheng Gao, Shiyu Ni, and Xueqi Cheng. Is factuality enhancement a free lunch for llms? better factuality can lead to worse context-faithfulness. arXiv preprint arXiv:2404.00216, 2024 c

    arXiv 2024

  8. [9]

    Parameters vs

    Baolong Bi, Shenghua Liu, Yiwei Wang, Yilong Xu, Junfeng Fang, Lingrui Mei, and Xueqi Cheng. Parameters vs. context: Fine-grained control of knowledge reliance in language models. arXiv preprint arXiv:2503.15888, 2025

    arXiv 2025

Show all 77 references
  1. [10]

    The description length of deep learning models

    L \'e onard Blier and Yann Ollivier. The description length of deep learning models. Advances in Neural Information Processing Systems, 31, 2018

    2018

  2. [11]

    A machine-independent theory of the complexity of recursive functions

    Manuel Blum. A machine-independent theory of the complexity of recursive functions. J. ACM, 14 0 (2): 0 322–336, April 1967 a . ISSN 0004-5411. doi:10.1145/321386.321395. URL https://doi.org/10.1145/321386.321395

    1967

  3. [12]

    On the size of machines

    Manuel Blum. On the size of machines. Information and Control, 11 0 (3): 0 257--265, 1967 b . ISSN 0019-9958. doi:https://doi.org/10.1016/S0019-9958(67)90546-3. URL https://www.sciencedirect.com/science/article/pii/S0019995867905463

    1967 doi

  4. [13]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffr...

    2020 arXiv

  5. [14]

    On the length of programs for computing finite binary sequences

    Gregory J Chaitin. On the length of programs for computing finite binary sequences. Journal of the ACM (JACM), 13 0 (4): 0 547--569, 1966

    1966

  6. [15]

    Algorithmic information theory

    Gregory J Chaitin. Algorithmic information theory. IBM journal of research and development, 21 0 (4): 0 350--359, 1977

    1977

  7. [16]

    Kolmogorov's contributions to information theory and algorithmic complexity

    Thomas M Cover, Peter Gacs, and Robert M Gray. Kolmogorov's contributions to information theory and algorithmic complexity. The annals of probability, 17 0 (3): 0 840--865, 1989

    1989

  8. [17]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025 arXiv

  9. [18]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, and et al

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei L...

    2025 arXiv

  10. [20]

    Language modeling is compression, 2024

    Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, Marcus Hutter, and Joel Veness. Language modeling is compression, 2024. URL https://arxiv.org/abs/2309.10668

    2024 arXiv

  11. [21]

    Longdocurl: a comprehensive multimodal long document benchmark integrating understanding, reasoning, and locating

    Chao Deng, Jiale Yuan, Pi Bu, Peijie Wang, Zhong-Zhi Li, Jian Xu, Xiao-Hui Li, Yuan Gao, Jun Song, Bo Zheng, et al. Longdocurl: a comprehensive multimodal long document benchmark integrating understanding, reasoning, and locating. arXiv preprint arXiv:2412.18424, 2024

    2024 arXiv

  12. [22]

    A survey on in-context learning, 2024

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. A survey on in-context learning, 2024. URL https://arxiv.org/abs/2301.00234

    2024 arXiv

  13. [23]

    Algorithmic randomness and complexity

    Rodney G Downey and Denis R Hirschfeldt. Algorithmic randomness and complexity. Springer Science & Business Media, 2010

    2010

  14. [24]

    Universal artificial intelligence: Practical agents and fundamental challenges

    Tom Everitt and Marcus Hutter. Universal artificial intelligence: Practical agents and fundamental challenges. Foundations of trusted autonomy, pages 15--46, 2018

    2018

  15. [25]

    Innate reasoning is not enough: In-context learning enhances reasoning large language models with less overthinking, 2025

    Yuyao Ge, Shenghua Liu, Yiwei Wang, Lingrui Mei, Lizhe Chen, Baolong Bi, and Xueqi Cheng. Innate reasoning is not enough: In-context learning enhances reasoning large language models with less overthinking, 2025. URL https://arxiv.org/abs/2503.19602

    2025 arXiv

  16. [26]

    Patchscopes: A unifying framework for inspecting hidden representations of language models, 2024

    Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscopes: A unifying framework for inspecting hidden representations of language models, 2024. URL https://arxiv.org/abs/2401.06102

    2024 arXiv

  17. [27]

    The llama 3 herd of models, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  18. [28]

    Learning universal predictors

    Jordi Grau-Moya, Tim Genewein, Marcus Hutter, Laurent Orseau, Gr \'e goire Del \'e tang, Elliot Catt, Anian Ruoss, Li Kevin Wenliang, Christopher Mattern, and Matthew Aitchison. Learning universal predictors. arXiv preprint arXiv:2401.14953, 2024

    2024 arXiv

  19. [29]

    Skywork open reasoner series

    Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork open reasoner series. https://capricious-hydrogen-41c.notion....

    2025

  20. [30]

    Execoder: Empowering large language models with executability representation for code translation

    Minghua He, Fangkai Yang, Pu Zhao, Wenjie Yin, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. Execoder: Empowering large language models with executability representation for code translation. arXiv preprint arXiv:2501.18460, 2025 b

    2025

  21. [31]

    Metagpt: Meta programming for multi-agent collaborative framework

    Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, et al. Metagpt: Meta programming for multi-agent collaborative framework. CoRR, abs/2308.00352, 2023

    2023 arXiv

  22. [32]

    Language models as zero-shot planners: Extracting actionable knowledge for embodied agents, 2022

    Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents, 2022. URL https://arxiv.org/abs/2201.07207

    2022 arXiv

  23. [33]

    Universal artificial intelligence: Sequential decisions based on algorithmic probability

    Marcus Hutter. Universal artificial intelligence: Sequential decisions based on algorithmic probability. Springer Science & Business Media, 2005

    2005

  24. [34]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems, 35: 0 22199--22213, 2022

    2022

  25. [35]

    Three approaches to the quantitative definition ofinformation’

    Andrei N Kolmogorov. Three approaches to the quantitative definition ofinformation’. Problems of information transmission, 1 0 (1): 0 1--7, 1965

    1965

  26. [36]

    An introduction to Kolmogorov complexity and its applications, volume 3

    Ming Li, Paul Vit \'a nyi, et al. An introduction to Kolmogorov complexity and its applications, volume 3. Springer, 2008

    2008

  27. [37]

    Lans: A layout-aware neural solver for plane geometry problem

    Zhong-Zhi Li, Ming-Liang Zhang, Fei Yin, and Cheng-Lin Liu. Lans: A layout-aware neural solver for plane geometry problem. arXiv preprint arXiv:2311.16476, 2023

    2023 arXiv

  28. [38]

    Cmmath: A chinese multi-modal math skill evaluation benchmark for foundation models

    Zhong-Zhi Li, Ming-Liang Zhang, Fei Yin, Zhi-Long Ji, Jin-Feng Bai, Zhen-Ru Pan, Fan-Hu Zeng, Jian Xu, Jia-Xin Zhang, and Cheng-Lin Liu. Cmmath: A chinese multi-modal math skill evaluation benchmark for foundation models. arXiv preprint arXiv:2407.12023, 2024

    2024 arXiv

  29. [40]

    From system 1 to system 2: A survey of reasoning large language models

    Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, et al. From system 1 to system 2: A survey of reasoning large language models. arXiv preprint arXiv:2502.17419, 2025 b

    2025 arXiv

  30. [41]

    Axis: Efficient human-agent-computer interaction with api-first llm-based agents, 2025

    Junting Lu, Zhiyang Zhang, Fangkai Yang, Jue Zhang, Lu Wang, Chao Du, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. Axis: Efficient human-agent-computer interaction with api-first llm-based agents, 2025. URL https://arxiv.org/abs/2409.17140

    2025 arXiv

  31. [42]

    Transformer-based image compression

    Ming Lu, Peiyao Guo, Huiqing Shi, Chuntong Cao, and Zhan Ma. Transformer-based image compression. arXiv preprint arXiv:2111.06707, 2021

    2021 arXiv

  32. [43]

    From understanding to utilization: A survey on explainability for large language models, 2024

    Haoyan Luo and Lucia Specia. From understanding to utilization: A survey on explainability for large language models, 2024. URL https://arxiv.org/abs/2401.12874

    2024 arXiv

  33. [44]

    Slang: New concept comprehension of large language models

    Lingrui Mei, Shenghua Liu, Yiwei Wang, Baolong Bi, and Xueqi Chen. Slang: New concept comprehension of large language models. arXiv preprint arXiv:2401.12585, 2024 a

    2024 arXiv

  34. [45]

    not aligned

    Lingrui Mei, Shenghua Liu, Yiwei Wang, Baolong Bi, Jiayi Mao, and Xueqi Cheng. " not aligned" is not" malicious": Being careful about hallucinations of large language models' jailbreak. arXiv preprint arXiv:2406.11668, 2024 b

    2024 arXiv

  35. [46]

    Hiddenguard: Fine-grained safe generation with specialized representation router

    Lingrui Mei, Shenghua Liu, Yiwei Wang, Baolong Bi, Ruibin Yuan, and Xueqi Cheng. Hiddenguard: Fine-grained safe generation with specialized representation router. arXiv preprint arXiv:2410.02684, 2024 c

    2024 arXiv

  36. [47]

    a1: Steep test-time scaling law via environment augmented generation, 2025

    Lingrui Mei, Shenghua Liu, Yiwei Wang, Baolong Bi, Yuyao Ge, Jun Wan, Yurong Wu, and Xueqi Cheng. a1: Steep test-time scaling law via environment augmented generation, 2025. URL https://arxiv.org/abs/2504.14597

    2025 arXiv

  37. [48]

    Locating and editing factual associations in gpt, 2023

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt, 2023. URL https://arxiv.org/abs/2202.05262

    2023 arXiv

  38. [49]

    Transformerlens

    Neel Nanda and Joseph Bloom. Transformerlens. https://github.com/TransformerLensOrg/TransformerLens, 2022

    2022

  39. [50]

    Gpt-4 technical report, 2024

    OpenAI . Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774

    2024 arXiv

  40. [51]

    Introducing openai o1-preview

    OpenAI. Introducing openai o1-preview. https://openai.com/index/introducing-openai-o1-preview/, 2024

    2024

  41. [52]

    Instruction tuning with gpt-4

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023

    2023 arXiv

  42. [54]

    A practical review of mechanistic interpretability for transformer-based language models, 2025

    Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. A practical review of mechanistic interpretability for transformer-based language models, 2025. URL https://arxiv.org/abs/2407.02646

    2025

  43. [55]

    CARER : Contextualized affect representations for emotion recognition

    Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. CARER : Contextualized affect representations for emotion recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3687--3697, Brussels, Belgium,...

    2018 doi

  44. [57]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024 b

    2024 arXiv

  45. [58]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024. URL https://arxiv.org/abs/2408.03314

    2024 arXiv

  46. [59]

    A preliminary report on a general theory of inductive inference

    Ray J Solomonoff. A preliminary report on a general theory of inductive inference. Citeseer, 1960

    1960

  47. [60]

    A formal theory of inductive inference

    Ray J Solomonoff. A formal theory of inductive inference. part i. Information and control, 7 0 (1): 0 1--22, 1964 a

    1964

  48. [61]

    A formal theory of inductive inference

    Ray J Solomonoff. A formal theory of inductive inference. part ii. Information and control, 7 0 (2): 0 224--254, 1964 b

    1964

  49. [62]

    Kimi k1.5: Scaling reinforcement learning with llms, 2025

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, and et al. Kimi k1.5: Scaling reinforcement learning with...

    2025 arXiv

  50. [63]

    Qwq: Reflect deeply on the boundaries of the unknown, November 2024

    Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown, November 2024. URL https://qwenlm.github.io/blog/qwq-32b-preview/

    2024

  51. [64]

    On computable numbers, with an application to the entscheidungsproblem

    Alan Mathison Turing et al. On computable numbers, with an application to the entscheidungsproblem. J. of Math, 58 0 (345-363): 0 5, 1936

    1936

  52. [65]

    Solomonoff induction: A solution to the problem of the priors? 2012

    Aron Vallinder. Solomonoff induction: A solution to the problem of the priors? 2012

    2012

  53. [66]

    Unifying two types of scaling laws from the perspective of conditional kolmogorov complexity

    Jun Wan. Unifying two types of scaling laws from the perspective of conditional kolmogorov complexity. arXiv preprint arXiv:2501.06802, 2025

    2025 arXiv

  54. [67]

    Label words are anchors: An information flow perspective for understanding in-context learning, 2023

    Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning, 2023. URL https://arxiv.org/abs/2305.14160

    2023 arXiv

  55. [68]

    Large action models: From inception to implementation, 2025 a

    Lu Wang, Fangkai Yang, Chaoyun Zhang, Junting Lu, Jiaxu Qian, Shilin He, Pu Zhao, Bo Qiao, Ray Huang, Si Qin, Qisheng Su, Jiayi Ye, Yudi Zhang, Jian-Guang Lou, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. Large action models: From inception to implementation, 20...

    2025 arXiv

  56. [69]

    Mv-math: Evaluating multimodal math reasoning in multi-visual contexts

    Peijie Wang, Zhong-Zhi Li, Fei Yin, Xin Yang, Dekang Ran, and Cheng-Lin Liu. Mv-math: Evaluating multimodal math reasoning in multi-visual contexts. arXiv preprint arXiv:2502.20808, 2025 b

    2025 arXiv

  57. [70]

    Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models, 2...

    2022 arXiv

  58. [71]

    Eva-kellm: A new benchmark for evaluating knowledge editing of llms, 2023

    Suhang Wu, Minlong Peng, Yue Chen, Jinsong Su, and Mingming Sun. Eva-kellm: A new benchmark for evaluating knowledge editing of llms, 2023. URL https://arxiv.org/abs/2308.09954

    2023 arXiv

  59. [72]

    Vulnerability of text-to-image models to prompt template stealing: A differential evolution approach

    Yurong Wu, Fangwen Mu, Qiuhong Zhang, Jinjing Zhao, Xinrun Xu, Lingrui Mei, Yang Wu, Lin Shi, Junjie Wang, Zhiming Ding, et al. Vulnerability of text-to-image models to prompt template stealing: A differential evolution approach. arXiv preprint arXiv:2502.14285, 2025

    2025 arXiv

  60. [73]

    Redstar: Does scaling long-cot data unlock better slow-reasoning systems? arXiv preprint arXiv:2501.11284, 2025

    Haotian Xu, Xing Wu, Weinong Wang, Zhongzhi Li, Da Zheng, Boyuan Chen, Yi Hu, Shijia Kang, Jiaming Ji, Yingying Zhang, et al. Redstar: Does scaling long-cot data unlock better slow-reasoning systems? arXiv preprint arXiv:2501.11284, 2025

    2025 arXiv

  61. [74]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  62. [75]

    Make pixels dance: High-dynamic video generation

    Yan Zeng, Guoqiang Wei, Jiani Zheng, Jiaxin Zou, Yang Wei, Yuchen Zhang, and Hang Li. Make pixels dance: High-dynamic video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8850--8860, 2024 a

    2024

  63. [76]

    Yan Zeng, Hanbo Zhang, Jiani Zheng, Jiangnan Xia, Guoqiang Wei, Yang Wei, Yuchen Zhang, Tao Kong, and Ruihua Song. What matters in training a gpt4-style language model with multimodal inputs? In Proceedings of the 2024 Conference of the North American Chapter of the Associatio...

    2024

  64. [77]

    Geoeval: benchmark for evaluating llms and multi-modal models on geometry problem-solving

    Jiaxin Zhang, Zhongzhi Li, Mingliang Zhang, Fei Yin, Chenglin Liu, and Yashar Moshfeghi. Geoeval: benchmark for evaluating llms and multi-modal models on geometry problem-solving. arXiv preprint arXiv:2402.10104, 2024 a

    2024 arXiv

  65. [78]

    Fuse, reason and verify: Geometry problem solving with parsed clauses from diagram

    Ming-Liang Zhang, Zhong-Zhi Li, Fei Yin, Liang Lin, and Cheng-Lin Liu. Fuse, reason and verify: Geometry problem solving with parsed clauses from diagram. arXiv preprint arXiv:2407.07327, 2024 b

    2024 arXiv

  66. [79]

    Character-level convolutional networks for text classification

    Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, 2015

    2015

  67. [80]

    Distributed rule vectors is a key mechanism in large language models' in-context learning, 2024

    Bowen Zheng, Ming Ma, Zhongqiao Lin, and Tianming Yang. Distributed rule vectors is a key mechanism in large language models' in-context learning, 2024. URL https://arxiv.org/abs/2406.16007

    2024

  68. [81]

    Vem: Environment-free exploration for training gui agent with value environment model

    Jiani Zheng, Lu Wang, Fangkai Yang, Chaoyun Zhang, Lingrui Mei, Wenjie Yin, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan, and Qi Zhang. Vem: Environment-free exploration for training gui agent with value environment model. arXiv preprint arXiv:2502.18906, 2025

    2025 arXiv

  69. [82]

    Trustrag: Enhancing robustness and trustworthiness in rag, 2025

    Huichi Zhou, Kin-Hei Lee, Zhonghao Zhan, Yue Chen, Zhenhao Li, Zhaoyang Wang, Hamed Haddadi, and Emine Yilmaz. Trustrag: Enhancing robustness and trustworthiness in rag, 2025. URL https://arxiv.org/abs/2501.00879

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.