Examining Two Hop Reasoning Through Information Content Scaling

David Johnston; Nora Belrose

REVIEW 3 major objections 5 minor 18 references

Examining Two Hop Reasoning Through Information Content Scaling

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that a transformer answering a two-hop question in a single forward pass has to store each underlying fact twice, once in each of two composed lookup functions, and that adding chain of thought removes that redundancy by…

desk verdict The generalization experiments are clean and worth taking seriously; the capacity-scaling evidence is undercut by a likely entropy-formula error and an unreproduced 2-bit reference line. read the letter →

arxiv 2502.03490 v2 pith:V4HN6MTT submitted 2025-02-05 cs.AI cs.LG

David Johnston , Nora Belrose This is my paper · ORCID

classification cs.AIcs.LG

keywords two-hopquestionansweringinformationcontentscalingknowledgecapacityfunctioncompositionchainofthoughttransformergeneralizationmemorizationvsreasoningmeasurement

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks why transformers are inconsistent at latent two-hop questions such as "Who is Bob's mother's boss?", where the answer requires composing two facts but no intermediate text is generated. It proposes that a single forward pass cannot apply a learned fact-lookup function twice, so the transformer must learn two separate copies of each fact, one for each hop. The evidence is information content scaling: the number of bits a trained model encodes grows with parameter count at roughly the rate predicted by a two-function-composition model, and generalization tests show the model can answer held-out complete questions but fails if any first-hop or second-hop component was absent. The same measurements on chain-of-thought models match a single-function model in which each fact is stored once. The paper presents this as support for the two-copies hypothesis rather than proof, and notes that supervised probing failed to reveal the intermediate entity.

What carries the argument

The central object is the paper's measured "information content" of a trained transformer, defined as dataset entropy minus the sum of cross-entropy losses over answer tokens, a quantity inherited from prior work on knowledge capacity. The comparison machinery is a set of entropy formulas that give different reference lines: one-hop entropy for recurrent composition, double one-hop entropy for two-function composition, and $|R|$ times one-hop entropy for independent memorization. Each candidate algorithm also has a distinct generalization signature, and the paper uses both measurements together. The load-bearing reference line is the empirical "2 bits per parameter" capacity, which the paper's own one-hop runs only approximate at about 1.6 bits per parameter.

What would settle it

Train a sweep of transformers on two-hop data in which every relation appears equally often as the first hop and as the second hop, and record the information content at zero train loss across a range of parameter counts; the two-function model predicts the curve must sit on the double-entropy line, while the recurrent model predicts it can sit on the one-hop entropy line, and which curve is actually reached would settle the claim.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that latent two-hop QA without chain of thought is implemented as $f_2(f_1(e_1,r),a)$ — two separate feed-forward fact-lookup functions — not as a single recurrent function $f(f(e_1,r),a)$ and not as a flat hash table $h(e_1,r,a)$. The three candidate algorithms predict different entropy budgets (one copy of each fact, two copies, or an independent entry per complete question) and different generalization patterns. Empirically, the information content of saturated 250K-to-15M parameter transformers scales along the two-function-composition curve, and generalization matches the paper's Table 1: models generalize to unseen complete questions but not to questions whose hop components were held out, even when all facts appeared in one-hop training. Chain-of-thought models instead scale like a single-function model. Supervised linear probes were unable to recover the intermediate entity, so in this setting capacity scaling and generalization agree with each other where probing does not.

Load-bearing premise

Everything rests on the claim that a saturated transformer has a fixed, architecture-independent capacity of roughly 2 bits per parameter; if that number shifts with training configuration, comparing measured information content against that line cannot identify which algorithm the model learned.

Editorial extensions

If this is right

Without chain of thought, a transformer that sees every fact only as one-hop questions will not compose those facts; training must expose each fact in the correct hop position of a two-hop question.
Allowing chain of thought should let a single stored fact serve both hops, so scaling curves for chain-of-thought models should be interpreted with a single-function entropy model.
Information content scaling can act as an interpretability check that agrees with generalization when activation probing fails, at least on tasks where candidate algorithms have distinguishable entropy formulas.
Small models can be trapped in a memorization local minimum, and capacity measurement identifies this regime because their information content falls below the composition curves.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the 2-bits-per-parameter reference line is configuration-dependent, the scaling argument becomes weaker; however, the paper's generalization predictions are independent of that constant and could be tested on natural-language data with controlled hop components.
The failed probing suggests the intermediate entity may be encoded nonlinearly or in a mixed representation; capacity scaling could be paired with nonlinear probes or causal interventions to locate where the first function's output is consumed by the second.
A practical extension is to apply the same entropy-comparison method to detect whether a model answers other multi-hop problems by composing general rules or by memorizing question-answer pairs, by checking which entropy formula its capacity curve approaches.
Chain-of-thought generalization on held-out first entities was inconsistent across runs, so whether chain-of-thought fully implements recurrent composition or only a partial version remains open and could be tested with more seeds.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. The generalization half is careful and largely convincing: without chain of thought, transformers generalize to held-out complete two-hop questions but not to held-out hop components, and that pattern matches two-function composition exactly. The capacity-scaling half, which is the advertised new method, rests on an entropy formula that appears to be wrong, and on a 2-bit-per-parameter reference line the authors themselves reproduce as about 1.6 bits per parameter. So the paper's main support for "learn each fact twice" is weaker than it looks.

What is genuinely new: the systematic holdout of individual hop components extends Wang et al.; the memorization-trap regime for small models is a nice, reproducible observation; and the probing failure is a useful negative result. The paper is also honest about its limitations, which counts for something.

Now the soft spots, in proportion. Equation (3) double-counts the four property attributes. With A defined as relations plus properties, f1 only needs the 17 relation tables and f2 needs all 21 attribute tables, so the two-function entropy should be |N| log N0 + |N| sum_{r in R} log|Vr| + |N| sum_{a in A} log|Va|, not 2N sum_{a in A} log|Va|. That is a concrete error in the central calculation, and it shifts the two-function entropy line by a non-trivial amount at their dataset sizes. On top of that, the capacity comparison leans on an external constant that their own one-hop runs do not reproduce, and the curves in Figures 1–3 are compared visually with no error bars or multiple seeds. The authors admit some of this, but the admission does not fix the double counting.

The generalization results alone are worth referee time. They are clean, match the table of predictions, and extend prior work in a useful way. The capacity-scaling claim needs a corrected entropy formula, released code, and a more careful treatment of the reference line before it can be taken as evidence.

Who is this for? People working on mechanistic interpretability and multi-hop reasoning. It deserves a serious referee, especially for the generalization half. My recommendation: send to peer review. The equation error is fixable, and the paper's honest treatment of its own uncertainty makes it a good candidate for revision rather than rejection.

Referee Report

3 major / 5 minor

Summary. The paper proposes information-capacity scaling as a complementary interpretability method for transformers and applies it to latent two-hop question answering. It trains numerous small transformers on synthetic one- and two-hop QA datasets and measures an 'information content' per parameter, comparing observed scaling curves against entropy models for three algorithmic hypotheses: independent memorization, two-function composition (f2(f1(e1,r),a)), and recurrent single-function composition. The authors report that, without chain of thought, capacity scaling best fits the two-function composition model, while with chain of thought the recurrent model fits better. They also test generalization under systematically held-out components and find that no-chain-of-thought generalization matches the two-function composition prediction: generalization to held-out complete questions but not to held-out hop components. A 'trapping' experiment shows that small models can be pushed into a non-generalizing independent-memorization regime. The paper is candid that the 2 bits-per-parameter capacity reference from Allen-Zhu & Li (2024) is not reproduced (their one-hop runs give ~1.6 bits/parameter) and that the scaling curves are more complex than the prior result.

Significance. If the capacity-scaling methodology is validated, it would provide a genuinely complementary tool to probing and behavioral tests for identifying which algorithm a transformer implements, and the paper's generalization results extend prior work by Wang et al. (2024) to a more systematic set of held-out components. The 'trapped memorization' phenomenon is an interesting empirical observation with clear potential for future mechanistic study. The paper also honestly reports a negative probing result, which strengthens the case that capacity scaling can sometimes be more informative than supervised probing. However, the central quantitative claim currently rests on an external capacity constant that the paper's own experiments do not reproduce, and on an entropy formula that appears to be miscalibrated for the two-function model. The qualitative generalization findings are clean and would remain valuable even if the capacity-scaling evidence were substantially revised.

major comments (3)

[Section 2.3, Eq. (3)] Equation (3) double-counts the four property attributes. In Section 2.1, A is defined as the set of relations and properties, and properties 'could be queried at the last hop, but could not constitute a first hop'. Under the two-function model, the first hop needs only relation values, while the second hop needs all attribute values. The dataset entropy should therefore be |N| log N0 + |N| ∑_{r∈R} log|Vr| + |N| ∑_{a∈A} log|Va|, not |N| log N0 + 2N ∑_{a∈A} log|Va|. As written, the formula either counts the four properties twice or omits the relation table entirely if A was intended to mean properties only. Since Eq. (3) defines the entropy baseline used in Figure 2 for the preferred model, this error shifts all information-content curves for the two-function comparison and can change the qualitative conclusion about which model 'kisses' the capacity line.
[Section 3.1, Figure 1] The capacity-scaling argument relies on the 2 bits-per-parameter line from Allen-Zhu & Li (2024) as the reference for whether a model has enough capacity to implement a given algorithm. Yet the paper's own one-hop runs, under the same hyperparameters used for the two-hop experiments, yield about 1.6 bits per parameter (Figure 1), and the authors explicitly note that the scaling behavior is 'more complex' than in the prior work. The observation that the two-hop curves 'almost kiss' the 2-bit line is therefore not an independent test of the two-function entropy model; it is a joint test of that model and of an unvalidated capacity constant. The paper should either recalibrate the reference line using its measured one-hop constant, provide a confidence band around the reference, or otherwise demonstrate that the qualitative comparison is robust to capacity constants in the range 1.6 to 2 bits per parameter. Without this, the abstract's claim that 'capacity scaling and generalization both support' the two-function hypothesis is stronger than the evidence supports.
[Appendix A.2] The effective-loss construction for two-function composition introduces a free parameter ϵ, which is chosen to maximize the right-hand side of the equation, subject to a 'fixed budget split between first and second hops' assumption. This is a modeling degree of freedom rather than a quantity derived from the transformer's loss. Because the resulting information-content curves are used to compare the two-function model against the alternatives in Figure 2, the sensitivity of the curves to alternative budget splits and to the assumed independence of per-hop probabilities should be quantified. The current presentation gives no indication of how much the inferred capacity scaling changes when ϵ is varied within plausible ranges, which weakens the discriminative power of the capacity measurement.

minor comments (5)

[Abstract and Introduction] There are several typos: 'concluding' should be 'conclusion' in the abstract, and 'intelligvence' should be 'intelligence' in Section 1.1.
[Table 1] The table is difficult to read because the algorithm labels are repeated in each row without clear column separators; the intended grouping of rows under each algorithm should be made visually explicit.
[Section 2.3, Eq. (1)] The notation A ∪ R is redundant because A already includes all relations and properties; this may confuse readers about whether the model uses a separate set for first-hop relations.
[Figure captions] Figure 8 caption uses 'µp' where 'µP' is meant, and Figure 9 caption contains the typo 'teh'.
[Section 3.2] The sentence 'the generalization performance ... precisely matches the prediction of Table 1' should be qualified, since the chain-of-thought results in Figure 7 show inconsistent generalization that the paper itself describes as not perfectly matching the recurrent-composition prediction; the no-chain-of-thought results do match precisely.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; capacity scaling compares predefined entropy models against an external 2-bit/param constant, with only a minor non-load-bearing self-citation.

full rationale

The paper's capacity-scaling argument is not circular. The two-function, recurrent, and independent computational models are defined by their entropies (Eqs. 2-4), computed from dataset parameters (N, N0, R, A, Va) before any model is trained, and the measured information content is dataset entropy minus cross-entropy loss (Eqs. 5-8), with effective losses derived in Appendices A.1 and A.2. No parameter is fitted to force the two-function curve to touch the capacity line: the 2-bit/param reference is an external empirical constant from Allen-Zhu & Li (2024), and the paper explicitly concedes it is testing two hypotheses at once ('we are testing two hypotheses at once (2 bits per parameter capacity as well as the computational model)'). The generalization signatures in Table 1 are logical consequences of the three algorithm definitions and are evaluated on held-out data; they do not reduce to the capacity fits. The only self-citation is Belrose et al. (2024) in the speculative explanation of the memorization trap (Section 3.3), and it is not load-bearing for the main claim. A modeling concern worth noting separately is Eq. (3): because A includes the four second-hop-only properties, 2N sum_{a in A} log|Va| counts those property entropies twice rather than once, which would bias the two-function entropy estimate; however, this is a calibration/accuracy issue, not a circularity, since the entropy is defined a priori and not tuned to the empirical curves. Overall, no derivation step equates its conclusion with its inputs by construction.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central inference rests on the external capacity constant, the feedforward architecture assumption, the saturation training assumption, and the effective-loss approximations. 'Native information content' is a measurement construct, not an independently evidenced entity, so it is not listed as an invented entity.

free parameters (1)

Knowledge capacity reference C = 2 bits per parameter (imported from Allen-Zhu & Li 2024, not fitted in this paper)
All capacity-scaling comparisons test whether observed information content approaches this line. The authors measure 1.6 bits per parameter for one-hop in their own setup (Section 3.1), so this reference value is a load-bearing input rather than an independently verified constant for this task.

assumptions (5)

domain assumption A transformer in a single feedforward pass cannot implement recurrent application of a learned function, so latent two-hop QA must use either independent memorization or two separate function copies f1 and f2.
Motivates the distinction between recurrent and two-function composition in Section 1.2. Reasonable for the Llama architecture used, but not proven; no recurrent or loop-based baselines are tested.
domain assumption Training to saturation, defined as loss decreasing by less than 1e-8 per step, lets models reach the information capacity limit assumed by the measurement.
The convergence criterion in Section 2.2 is used for all runs; the paper does not verify that longer training would change the capacity curves.
ad hoc to paper The effective-loss transformations in Appendices A.1 and A.2 faithfully lower-bound the information content of the composed functions (assuming independent per-item probabilities and a fixed budget split between hops).
These approximations are introduced specifically to make the capacity measurement tractable; the authors show exactness only at extremes of accuracy (A.1) and impose a fixed-budget assumption (A.2).
ad hoc to paper Algorithms with the same information-content scaling tend to have the same generalization behavior.
Stated in Section 1.1 as the bridge between capacity scaling and generalization; the authors note it is not clear how generally it applies.
domain assumption The synthetic profiles have randomly assigned, independent relations so that the task is purely combinatorial and the entropy formulas apply exactly.
Section 2.1 specifies that relations are independently uniformly random and that 'child' is not the inverse of 'parent'; this removes real-world relational structure that could otherwise change the information content estimates.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Examining Two Hop Reasoning Through Information Content Scaling." pith.science (2026). https://pith.science/paper/V4HN6MTT

@misc{pith2026250203490,
  author       = {Pith},
  title        = {Pith review of: Examining Two Hop Reasoning Through Information Content Scaling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V4HN6MTT}},
  note         = {Machine review of arXiv:2502.03490}
}

read the original abstract

Prior work has found that transformers have an inconsistent ability to learn to answer latent two-hop questions -- questions of the form "Who is Bob's mother's boss?" We study why this is the case by examining how transformers' capacity to learn datasets of two-hop questions and answers (two-hop QA) scales with their size, motivated by prior work on transformer knowledge capacity for simple factual memorization. We find that capacity scaling and generalization both support the hypothesis that latent two-hop QA requires transformers to learn each fact twice, while two-hop QA with chain of thought does not. We also show that with appropriate dataset parameters, it is possible to "trap" very small models in a regime where they memorize answers to two-hop questions independently, even though they would perform better if they could learn to answer them with function composition. Our findings show that measurement of capacity scaling can complement existing interpretability methods, though there are challenges in using it for this purpose.

Figures

Figures reproduced from arXiv: 2502.03490 by the authors.

**Figure 1.** Observed information content scaling on one-hop questions for 4 layer transformers. A loss of zero implies that the information content is equal to the dataset entropy, and a loss equal to predicting the uniform distribution over all answers in the dataset yields the information content level represented by “baseline” (this is nontrivial due to needing to learn the set of names in the dataset out of all possible n… view at source ↗

**Figure 3.** Observed information content scaling on two-hop questions with chain-of-thought generation for 4 layer transformers. The information content exceeds the capacity estimate for two function composition and approaches the capacity for recurrent composition, expected. performance on either individual hop (or both together) is too weak, it may be difficult to learn the composition algorithm. Second, models may choose to… view at source ↗

**Figure 5.** Anomalously low information content for models trained with only 4 relations. The estimated information content approximates the capacity curve if we assume independent memorization of all two hop questions. between generalizing and nongeneralizing models. See Tables 2 and 3. In particular, these results show that the intermediate entity is not more retrievable in models where we infer two-function composition than… view at source ↗

Figures from the paper (4 more)

**Figure 6.** Figure 6: Comparison of uniform distribution loss with generalization loss for models answering two hop questions questions without chain of thought with held out components. Values equal to or less than 0 indicate no generalization; in fact, every model in this plot does not ge…

**Figure 7.** Figure 7: Comparison of uniform distribution loss with generalization loss for models answering two hop questions questions with chain of thought with held out components. Values equal to or less than 0 indicate no generalization. For all components but Entity 1, at least one mo…

**Figure 8.** Figure 8: Additional measurements of 1 hop information content. These measurements were done using 4 relations instead of 17, and µp initialization was not used. The measured capacities were closer to 2 bits per parameter, but the trend less consistent than the main results. 14 …

**Figure 9.** Figure 9: Comparison of information content of 4 layer and 12 layer models. The less consistent results of teh 12 layer models may be attributable to the fact that the hyperparameters were tuned on the 4 layer model and not readjusted for the deeper architecture. 15 [PITH_FULL_…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 11 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

and Li, Y

Allen-Zhu , Z. and Li, Y. Physics of Language Models : Part 3.3, Knowledge Capacity Scaling Laws , April 2024

work page 2024
[3]

Probing classifiers: Promises, shortcomings, and advances

Belinkov, Y. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48 0 (1): 0 207--219, 2022

2022
[4]

T., and Fern, X

Belrose, N., Pope, Q., Quirke, L., Mallen, A. T., and Fern, X. Neural networks learn statistics of increasing complexity. In Forty-first International Conference on Machine Learning, 2024

work page 2024
[5]

DeepSeek-AI , Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F., Hao, G., Chen, G.,...

work page 2025
[6]

A., Mehta, H., Mishchenko, K., Khaled, A., and Cutkosky, A

Defazio, A., Yang, X. A., Mehta, H., Mishchenko, K., Khaled, A., and Cutkosky, A. The road less scheduled. arXiv preprint arXiv:2405.15682, 2024

arXiv 2024
[7]

K., Aitchison, M., Orseau, L., Hutter, M., and Veness, J

Delétang, G., Ruoss, A., Duquenne, P.-A., Catt, E., Genewein, T., Mattern, C., Grau-Moya, J., Wenliang, L. K., Aitchison, M., Orseau, L., Hutter, M., and Veness, J. Language modeling is compression. arXiv preprint arXiv:2309.10668, 2023. URL https://arxiv.org/abs/2309.10668

arXiv 2023
[8]

The llama 3 herd of models

Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

arXiv 2024

Show all 18 references

[9]

L., Jiang, L., Lin, B

Dziri, N., Lu, X., Sclar, M., Li, X. L., Jiang, L., Lin, B. Y., Welleck, S., West, P., Bhagavatula, C., Le Bras, R., Hwang, J., Sanyal, S., Ren, X., Ettinger, A., Harchaoui, Z., and Choi, Y. Faith and Fate : Limits of Transformers on Compositionality . Advances in Neural Infor...

2023
[10]

R., Ewart, A., and Sharkey, L

Huben, R., Cunningham, H., Smith, L. R., Ewart, A., and Sharkey, L. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2023

2023
[11]

T., Goel, S., Krishnamurthy, A., and Zhang, C

Liu, B., Ash, J. T., Goel, S., Krishnamurthy, A., and Zhang, C. Transformers Learn Shortcuts to Automata , May 2023

2023
[12]

Merrill, W., Sabharwal, A., and Smith, N. A. Saturated Transformers are Constant-Depth Threshold Circuits . Transactions of the Association for Computational Linguistics, 10: 0 843--856, August 2022. ISSN 2307-387X. doi:10.1162/tacl_a_00493

2022 doi
[13]

Attention is Turing-Complete

P \'e rez, J., Barcel \'o , P., and Marinkovic, J. Attention is Turing-Complete . Journal of Machine Learning Research, 22 0 (75): 0 1--35, 2021. ISSN 1533-7928

2021
[14]

Solomonoff, R. J. A formal theory of inductive inference. part i. Information and Control, 7 0 (1): 0 1--22, 1964. doi:10.1016/S0019-9958(64)90223-2

1964 doi
[15]

Grokked Transformers are Implicit Reasoners : A Mechanistic Journey to the Edge of Generalization , October 2024

Wang, B., Yue, X., Su, Y., and Sun, H. Grokked Transformers are Implicit Reasoners : A Mechanistic Journey to the Edge of Generalization , October 2024

2024
[16]

J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J

Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. Tensor Programs V : Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer , March 2022

2022
[17]

Do LLMs Really Think Step-by-step In Implicit Reasoning ?, January 2025

Yu, Y. Do LLMs Really Think Step-by-step In Implicit Reasoning ?, January 2025

2025
[18]

Zhang, Z., Lin, P., Wang, Z., Zhang, Y., and Xu, Z.-Q. J. Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizing , January 2025

2025

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

and Li, Y

Allen-Zhu , Z. and Li, Y. Physics of Language Models : Part 3.3, Knowledge Capacity Scaling Laws , April 2024

work page 2024

[3] [3]

Probing classifiers: Promises, shortcomings, and advances

Belinkov, Y. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48 0 (1): 0 207--219, 2022

2022

[4] [4]

T., and Fern, X

Belrose, N., Pope, Q., Quirke, L., Mallen, A. T., and Fern, X. Neural networks learn statistics of increasing complexity. In Forty-first International Conference on Machine Learning, 2024

work page 2024

[5] [5]

DeepSeek-AI , Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F., Hao, G., Chen, G.,...

work page 2025

[6] [6]

A., Mehta, H., Mishchenko, K., Khaled, A., and Cutkosky, A

Defazio, A., Yang, X. A., Mehta, H., Mishchenko, K., Khaled, A., and Cutkosky, A. The road less scheduled. arXiv preprint arXiv:2405.15682, 2024

arXiv 2024

[7] [7]

K., Aitchison, M., Orseau, L., Hutter, M., and Veness, J

Delétang, G., Ruoss, A., Duquenne, P.-A., Catt, E., Genewein, T., Mattern, C., Grau-Moya, J., Wenliang, L. K., Aitchison, M., Orseau, L., Hutter, M., and Veness, J. Language modeling is compression. arXiv preprint arXiv:2309.10668, 2023. URL https://arxiv.org/abs/2309.10668

arXiv 2023

[8] [8]

The llama 3 herd of models

Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

arXiv 2024