Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Discovering Hierarchical Latent Capabilities of Language Models via Causal Representation Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Language-model benchmark results trace a causal chain from general capability through instruction-following to math.

desk verdict A transparent, useful look at low-rank structure and heterogeneity in LLM benchmarks, but the claimed causal hierarchy is an artifact of the triangular parametrization, not a discovery. read the letter →

arxiv 2506.10378 v1 pith:2I3JERYW submitted 2025-06-12 cs.LG cs.AIcs.CLstat.ML

Jikai Jin , Vasilis Syrgkanis , Sham Kakade , Hanlin Zhang This is my paper
classification cs.LGcs.AIcs.CLstat.ML
keywords causalrepresentationlearninglanguagemodelevaluationlatentcapabilityfactorshierarchicalstructuralbenchmarkheterogeneityinstructionfollowingmathematicalreasoningidentifiability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the six benchmark scores of fine-tuned language models are generated by three latent capability factors, and that those factors are causally ordered rather than merely correlated. To show this, it develops Hierarchical Component Analysis (HCA), a causal representation learning method that treats the base model as a shared confounder and recovers latent factors that are linear in the benchmarks and structurally related across base-model domains. Applied to public leaderboard data on over 1,500 models built from four base models, the method finds a compact three-node chain, with general problem-solving feeding instruction-following and instruction-following feeding mathematical reasoning. A sympathetic reading is that evaluation results are therefore not a flat leaderboard: they encode an actionable hierarchy, and math-score gains should be interpreted with the mediating instruction-following capability in mind.

What carries the argument

Hierarchical Component Analysis (HCA) is the central object. It starts with independent component analysis (ICA) on each base-model domain to obtain an unmixing matrix $M_k$ that maps observed benchmarks to independent sources, and it exploits the decomposition $M_k = P_k B_k H$, where $B_k$ is the domain-specific lower-triangular structural weight matrix and $H$ is the shared unmixing inverse of the mixing matrix $G$. The algorithm's key step is row-residual extraction: after guessing an order, it projects each row of $M_k$ onto the span of earlier rows and requires the projected residuals across domains to be rank 1, recovering the next row of $H$ as the top singular vector; it then searches over row permutations and fits the best lower-triangular $B_k$'s, selecting the configuration with the smallest maximum inexactness coefficient. The paper proves that under an exact linear SCM with non-Gaussian sources and enough generic domains, $H$ is identifiable up to a left lower-triangular factor for $K \ge d$, and it relaxes the setting to $\alpha$-inexact SCMs measured by MIC, with the four-domain fit achieving MIC 0.04.

What would settle it

Take one of the four base models, fine-tune it only on instruction-following examples that contain no mathematical content, and compare its MATH Lvl 5 score against a control model fine-tuned on an equal number of non-instruction paragraphs; the claimed causal chain predicts a significantly larger math gain in the instruction group, so a null or reversed result would refute it.

Watch

Extended reading notes

Core claim

The paper's central claim is that, within the four base models that share an invariant subspace, observed benchmark vectors $x$ are approximately $G z$ for a shared mixing matrix $G$ and three latent capability factors $z_1, z_2, z_3$, and that within each base model those factors follow a linear structural causal model with one common directed graph: $z_1$ influences $z_2$ and $z_2$ influences $z_3$. The recovered factors align tightly with benchmarks, $z_1$ to BBH and MMLU-Pro (general problem solving), $z_2$ to IFEval (instruction following), and $z_3$ to MATH Lvl 5 (advanced mathematical reasoning), with the $z_3$ alignment reaching $R^2 = 1.00$. The paper supports the causal reading with an intervention-style comparison: fine-tuning on IFEval raises both IFEval and MATH scores while leaving other benchmarks roughly unchanged, and this transfer is stronger for Qwen models than for Llama models, matching the recovered edge weights. The identifiability guarantee holds only up to ancestor mixtures and recovers the graph as its transitive closure, so the direct claim is the ancestor ordering rather than the presence of every individual edge.

Load-bearing premise

The load-bearing premise is that the six benchmark scores of all models built from the four base models are, up to a small residual, a fixed linear mixture of the same three latent factors, and that inside each base model those factors follow one shared causal chain with nearly independent noise sources.

Editorial extensions

If this is right

  • The same post-training recipe can no longer be assumed to transfer across base models, because the recovered causal graphs have different edge weights for different base models.
  • Math-score gains should not be read purely as mathematical reasoning gains, since part of the effect is predicted to run through instruction-following; fine-grained and conditional scores are needed.
  • Scaling pretraining compute is the primary lever for the general capability node, whereas instruction-following and math are more responsive to fine-tuning, so budget allocation between pretraining and post-training can target whichever node is the bottleneck.
  • The HCA pipeline offers a no-retraining evaluation route: recover latent factors from existing leaderboard data, then test causal hypotheses with small targeted fine-tuning runs instead of full retraining sweeps.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: applying HCA to newer or larger base-model families would test whether the recovered chain is a stable property of post-training or specific to this leaderboard cohort.
  • Beyond the paper: a clean mediation test would fine-tune only on instruction-following data stripped of mathematical content and compare MATH gains against a token-matched control, separating genuine mediation from generic fine-tuning effects.
  • Beyond the paper: if the hierarchy is a structural law, math gains from instruction-following should eventually saturate as the general capability node becomes the bottleneck, a predicted nonlinearity the current data do not estimate.
  • Beyond the paper: the recovered ancestor ordering suggests a curriculum for post-training, general capability first, then instruction-following, then math specialization, which the paper motivates but does not test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a causal representation learning framework, Hierarchical Component Analysis (HCA), to recover latent capability factors from benchmark performance data of LLMs. The model assumes a shared linear mixing matrix across base models, with latent factors following a linear structural causal model per base model. Applied to Open LLM Leaderboard data for four base models, the authors report a three-node causal structure with direction from general problem-solving (z1), through instruction-following (z2), to mathematical reasoning (z3), supported by a minimal inexactness coefficient (MIC) of 0.04 and by SFT experiments. The paper also includes analyses of base-model heterogeneity, matrix completion, and MMLU subtasks.

Significance. The idea of using multi-domain heterogeneity to identify latent causal capabilities is timely and addresses a real gap in causal evaluation of LLMs. The HCA algorithm is novel and comes with theoretical identifiability results, and the code is publicly released. The additional experiments on base-model-controlled matrix completion and the stability analysis are useful contributions. However, the central claim of a sparse causal direction is not supported by the identification result (which recovers only a transitive closure) or by the empirical evidence; this substantially weakens the paper's contribution as a causal discovery result.

major comments (4)
  1. [§C.1, §C.2, Theorem 1, Figure 7] The identifiability guarantee of HCA only identifies the latent factors up to a lower-triangular transformation, i.e., up to the transitive closure of the true graph. In Section C.1 the authors explicitly state that HCA 'only recovers a transitive closure Ḡ of the true graph G' and that, 'for simplicity and due to the fundamental inexactness of our model', they do not test which edges exist, instead constraining each Bk to be upper-triangular. Consequently, the graph in Figure 7 includes all edges consistent with a total order, such as an edge z1→z3, and the sparse chain z1→z2→z3 claimed in the abstract is not identified. Theorem 1 itself is stated and proved for a complete DAG with all lower-triangular edges, so it certifies recovery only of the total order, not of sparsity. The 'clear causal direction' in the abstract is therefore a post hoc reading of an unverified modeling assumption.
  2. [§4.1, Figure 8, Table 3] The semantic labels of the latent factors are assigned by an OLS procedure that selects, for each factor, the benchmark maximizing R². The reported R²=1.00 for z3 versus MATH Lvl 5 is an in-sample fitting outcome, not a predictive validation. This circularity means the strong correlations cannot be used as evidence for the causal interpretation of the factors, and the apparent near-perfect alignment with MATH is partly a consequence of the label-selection procedure.
  3. [§4.2, Table 2] The SFT experiments do not provide direct interventional evidence for the specific z2→z3 edge. Fine-tuning on IFEval is an intervention on the whole model, not on the isolated latent factor z2, and the measured MATH improvements are small and inconsistent: Gemma-2-9B shows a decrease from 0.13 to 0.12 after IFEval SFT, and Llama-3-8B stays at 0.05. The 'Instruct' rows in Table 2 are observational leaderboard entries subject to selection bias, so they cannot serve as controlled causal evidence.
  4. [§4.1, Figure 18] The text states that 'running our algorithm on the subset of {1,2,4,5}, with Qwen2-7B excluded, achieves a minimal MIC of 0.04.' Figure 18, however, shows size-3 domain subsets with MIC values as low as 0.02. This inconsistency undermines the reported model selection: if other subsets yield smaller MIC, then the choice of {1,2,4,5} and the associated causal graph are not justified as the best-fitting model.
minor comments (5)
  1. [§3 (Definitions 1, 2) and §3.1] There are typos: 'wuth' should be 'with' in Definitions 1 and 2, and 'varisbles' should be 'variables' in Section 3.1. Also, 'benhmark' in Appendix G.3 should be 'benchmark'.
  2. [§2.1] The text defines Sinv = {1,2,4,5,6} as 'the index set of these seven models', but the set has only five elements; this should say 'five models'.
  3. [§4.1] The OLS formula is presented with 'P' instead of a summation symbol: 'zi ≈ P j<i ajzj +γ BxB +c'. The description of the adjustment could be more precise about what is being residualized and why.
  4. [Figure 2a caption] The caption labels 'Explained Variance Ratio' and 'Cumulative Explained Variance' in a way that appears to be swapped relative to the plot content; please check the labeling.
  5. [Appendix H] The sensitivity analysis shows that the z1 factor is not stable when the domain set is expanded, and the authors themselves note that 'it is hard to argue which one makes more sense.' This instability is not flagged in the main text, where z1 is described as 'foundational general capability.'

Circularity Check

2 steps flagged · score 6.0 of 10

Headline 'z2→z3' is not identified: HCA imposes a triangular order and the benchmark labels are fitted by in-sample R² maximization.

  1. fitted input called prediction [Section 4.1 (Figure 8a and Figure 8d)]
    "Moreover, in view of the ambiguity discussed in Section 3.1, we run an OLS zi ≈ P j<i ajzj + γB xB + c where xB represents the performance on benchmark B. For each i, we pick B that maximizes the R2 and replace zi with zi − P j<i ajzj to attain best-possible alignment between the recovered latent factors and their most indicative benchmarks."

    The capability labels are not predicted but selected: for each recovered factor z_i, the benchmark B with the largest in-sample R² is chosen, and the factor is then rotated toward that benchmark. Figure 8d reports R²=1.00 for z3 against MATH Lvl 5, so the claim that z3 'represents advanced mathematical reasoning' is a fitting outcome of this OLS selection step. The abstract's phrasing 'culminating in mathematical reasoning ability' thus restates the fitting procedure rather than an independent empirical finding.

  2. self definitional [Appendix C.1, Section 3.1, and Figure 7]
    "Compared with LiNGCReL, HCA only recovers a transitive closure ¯G of the true graph G. It is still possible to infer whether each edge in ¯G indeed exists in G (see appendix). For simplicity and due to the fundamental inexactness of our model, we do not perform this step here. Equivalently, we are only imposing the constraint that each Bk is upper-triangular, without assuming that any other entries are also zero."

    The paper's headline causal chain z1→z2→z3 is not a recovered structure: HCA deliberately does not test which edges of the transitive closure exist and instead constrains every Bk to be triangular (a total order). The nonzero weights in Figure 7 are therefore fitted coefficients of this imposed triangular model, not discovered edges. Theorem 1 guarantees recovery only up to left multiplication by a lower-triangular matrix, i.e., up to the transitive closure of a total order; the sparse direction 'instruction following → math' is an untested modeling assumption that the abstract then presents as a 'clear causal direction.'

full rationale

The central scientific claim—that latent capabilities form a chain z1→z2→z3 with instruction following causally improving math—is only partially supported by the paper's own derivation. The identifiability guarantee (Theorem 1) applies to an exact SCM and certifies recovery only up to a lower-triangular transformation, i.e., up to the transitive closure of a total order. Appendix C.1 states that the algorithm does not test which edges exist and instead imposes triangularity; hence the directed edges in Figure 7 are fitted parameters of an assumed order, not discoveries. Independently, the semantic labels (z1≈BBH, z2≈IFEval, z3≈MATH) are produced by an OLS step that selects, for each factor, the benchmark maximizing in-sample R²; the reported R²=1.00 between z3 and MATH is a fitting outcome. The abstract's 'clear causal direction... culminating in mathematical reasoning ability' therefore combines an imposed ordering with a post hoc labeling of the factors. The external SFT-on-IFEval experiment is a genuine intervention and provides some independent evidence that instruction tuning can raise MATH performance, which prevents a higher circularity score; however, it is not a quantitative test of the fitted model and is interpreted only as 'consistent with' the fitted weights. The subset choice (excluding Qwen2-7B after inspecting MIC) and the reliance on a shared mixing matrix across only four base models are post hoc selection steps that further limit the generality of the reported structure, though they are not definitional circularity. The self-citation to the same authors' JS24 for the 'intrinsic' nature of the ambiguity is context for why the paper treats the unidentified direction as benign, but the paper also contains its own proof, so the main circularity is the fitted-label and imposed-triangle reduction.

Assumptions & free parameters 4 free parameters · 6 assumptions · 1 invented entities

The central causal claim rests on a chain of modeling choices: a shared linear mixing matrix across base models, an inexact linear SCM with near-independent sources, a total-order parametrization of the graph, and post hoc domain selection. Each of these is plausible but untested, and the paper's own sensitivity analysis shows instability of the recovered factor z1 when the domain set changes.

free parameters (4)
  • latent dimension d0 = 3
    Chosen as the number of principal components explaining 95% variance in Figure 2a.
  • alignment benchmark per factor = z1: BBH, z2: IFEval, z3: MATH Lvl 5
    Each factor is aligned to the benchmark that maximizes OLS R2 (Section 4.1), making the semantic labels fit the data by construction.
  • domain subset = {1,2,4,5} (four base models)
    Selected as the subset minimizing the inexactness coefficient; including Qwen2-7B worsens MIC, and the choice is post hoc.
  • sigmoid scaling law parameters L,k,C0,b,tau = reported per benchmark in Figure 15
    Appendix E fits these parameters to benchmark scores; they are a side analysis, not central to the causal claim.
assumptions (6)
  • domain assumption Shared linear mixing: x = Gz with common G across base models (Hypothesis 1)
    Section 2.1, Hypothesis 1. The paper shows similarity of PC subspaces, not equality of the mixing matrix G.
  • domain assumption Linear SCM with independent or nearly independent sources within each domain and shared graph (Hypothesis 2)
    Section 3, Hypothesis 2 and Definition 2. The MIC of 0.04 is a residual, not a validation of the causal model.
  • domain assumption Node-level non-degeneracy across domains (Assumption 1)
    Appendix C.2, Assumption 1. Required for identifiability and hard to verify with only four domains.
  • standard math Sources are non-Gaussian with different distributions (Assumption 2)
    Appendix C.2, Assumption 2. Standard ICA condition.
  • ad hoc to paper Total-order parametrization: each Bk is upper-triangular (complete DAG)
    Section C.1, point 1. The algorithm imposes upper-triangularity, which presumes a total order and prevents recovery of sparse causal graphs.
  • domain assumption Base model is the only confounder and is correctly determined from model metadata
    Section 1.1 and Appendix B. The base model is inferred heuristically, so mislabeling could affect all downstream results.
invented entities (1)
  • Latent capability factors z1, z2, z3
    purpose: Represent unobserved capabilities explaining benchmark scores and causal structure
    The factors are latent and the semantic labels are assigned by maximizing R2 against benchmarks (Section 4.1), so their identity is fitted rather than independently verified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Discovering Hierarchical Latent Capabilities of Language Models via Causal Representation Learning." pith.science (2026). https://pith.science/paper/2I3JERYW

@misc{pith2026250610378,
  author       = {Pith},
  title        = {Pith review of: Discovering Hierarchical Latent Capabilities of Language Models via Causal Representation Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2I3JERYW}},
  note         = {Machine review of arXiv:2506.10378}
}
read the original abstract

Faithful evaluation of language model capabilities is crucial for deriving actionable insights that can inform model development. However, rigorous causal evaluations in this domain face significant methodological challenges, including complex confounding effects and prohibitive computational costs associated with extensive retraining. To tackle these challenges, we propose a causal representation learning framework wherein observed benchmark performance is modeled as a linear transformation of a few latent capability factors. Crucially, these latent factors are identified as causally interrelated after appropriately controlling for the base model as a common confounder. Applying this approach to a comprehensive dataset encompassing over 1500 models evaluated across six benchmarks from the Open LLM Leaderboard, we identify a concise three-node linear causal structure that reliably explains the observed performance variations. Further interpretation of this causal structure provides substantial scientific insights beyond simple numerical rankings: specifically, we reveal a clear causal direction starting from general problem-solving capabilities, advancing through instruction-following proficiency, and culminating in mathematical reasoning ability. Our results underscore the essential role of carefully controlling base model variations during evaluation, a step critical to accurately uncovering the underlying causal relationships among latent model capabilities.

Figures

Figures reproduced from arXiv: 2506.10378 by the authors.

Figure 1
Figure 1. Example of a Hierarchical model of capa [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. PCA analysis showing low-rank structure and domain hetero [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Principal component sub￾space similarity across domains. We next examine whether these patterns persist when we vary the model subset used for PCA. To this end, we isolate all Qwen2.5 variants (across scales) and, separately, the Qwen2.5- 14B model. As shown in Figure 2b, the distributions of their distances from the full-leaderboard rank-3 PC subspace diverge substantially. This divergence implies that a one-size-f… view at source ↗
Figures from the paper (30 more)
Figure 4
Figure 4. Figure 4: RMSE of global and local matrix completion approaches for two types of missing patterns. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Illustration of Definition 1. Intuitively, latent factors earlier in the topological ordering of the DAG are primitive, while later factors are progressively less primitive, as they inherits the variability in their ancestors. In practice, assuming exact SCMs is often …
Figure 6
Figure 6. Figure 6: Illustration of our setting and the key row-residual extraction step in our algorithm [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: The causal graphs that we recover for each domain. The numbers represent the weights of [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: The unmixing matrix and the alignment between benchmarks and capabilities via OLS. [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Sigmoid scaling law for BBH performance. Interpreting z1 (Foundational General Capability). As a root node in the causal graph, z1 a root node in our causal graph, likely represents a foundational, generalized capability. This interpretation is supported by its positiv…
Figure 10
Figure 10. Figure 10: Results for running PCA on individual domains. [PITH_FULL_IMAGE:figures/full_fig_p026_10.png]
Figure 11
Figure 11. Figure 11: Pairwise cosine distance matrix for 20 base models. between MoE models and non-MoE models. We also include two architectures upon which a vast majority of MoE models are built. We can see that there is little difference in the principal component subspaces. Different …
Figure 12
Figure 12. Figure 12: PCA Results comparing principal component subspaces for different criteria. [PITH_FULL_IMAGE:figures/full_fig_p028_12.png]
Figure 13
Figure 13. Figure 13: The average benchmark performance of fine-tuned models on the open LM leaderboard [PITH_FULL_IMAGE:figures/full_fig_p028_13.png]
Figure 14
Figure 14. Figure 14: Estimates of the average effect of fine-tuning. [PITH_FULL_IMAGE:figures/full_fig_p029_14.png]
Figure 15
Figure 15. Figure 15: Sigmoid scaling laws of benchmark accuracies for pretrained and fine-tuned models. Top [PITH_FULL_IMAGE:figures/full_fig_p030_15.png]
Figure 16
Figure 16. Figure 16: RMSEs of global v.s. local matrix completion for each possible set of fully observed [PITH_FULL_IMAGE:figures/full_fig_p031_16.png]
Figure 17
Figure 17. Figure 17: HCA’s recovery of the DGP, including the linear SCM (second column) and mixing [PITH_FULL_IMAGE:figures/full_fig_p032_17.png]
Figure 18
Figure 18. Figure 18: Overview of the MIC obtained by difference choices of domain indices. Here, as we [PITH_FULL_IMAGE:figures/full_fig_p033_18.png]
Figure 19
Figure 19. Figure 19: Approximation error of the low-rank latent factor space for the observed benchmark [PITH_FULL_IMAGE:figures/full_fig_p034_19.png]
Figure 20
Figure 20. Figure 20: HCA’s recovery of the DGP after removing badly fine-tuned models that have average [PITH_FULL_IMAGE:figures/full_fig_p035_20.png]
Figure 21
Figure 21. Figure 21: The causal graphs recovered for different models. The numbers represent the weights [PITH_FULL_IMAGE:figures/full_fig_p035_21.png]
Figure 22
Figure 22. Figure 22: The unmixing matrix and the alignment between benchmarks and capabilities via OLS. [PITH_FULL_IMAGE:figures/full_fig_p035_22.png]
Figure 23
Figure 23. Figure 23: figures for our analysis of Open LLM leaderboard v1. [PITH_FULL_IMAGE:figures/full_fig_p036_23.png]
Figure 24
Figure 24. Figure 24: Results for applying our method to open LM leaderboard v1. [PITH_FULL_IMAGE:figures/full_fig_p037_24.png]
Figure 25
Figure 25. Figure 25: The causal graphs recovered for different models. The numbers represent the weights of [PITH_FULL_IMAGE:figures/full_fig_p037_25.png]
Figure 26
Figure 26. Figure 26: The unmixing matrix and the alignment between benchmarks and capabilities via OLS. [PITH_FULL_IMAGE:figures/full_fig_p037_26.png]
Figure 27
Figure 27. Figure 27: Correlation matrix for the tasks in the MMLU benchmark. [PITH_FULL_IMAGE:figures/full_fig_p038_27.png]
Figure 28
Figure 28. Figure 28: HCA analysis of the MMLU by Task Leaderboard data of math-related subjects. [PITH_FULL_IMAGE:figures/full_fig_p039_28.png]
Figure 29
Figure 29. Figure 29: HCA analysis of the MMLU by Task Leaderboard data of physics-related subjects. [PITH_FULL_IMAGE:figures/full_fig_p039_29.png]
Figure 30
Figure 30. Figure 30: HCA analysis of the MMLU by Task Leaderboard data of three different subjects [PITH_FULL_IMAGE:figures/full_fig_p040_30.png]
Figure 31
Figure 31. Figure 31: The unmixing matrices of ICA for individual domains. [PITH_FULL_IMAGE:figures/full_fig_p041_31.png]
Figure 32
Figure 32. Figure 32: The unmixing matrix and the alignment between benchmarks and capabilities via OLS. [PITH_FULL_IMAGE:figures/full_fig_p041_32.png]
Figure 33
Figure 33. Figure 33: Causal graphs recovered from each domain. [PITH_FULL_IMAGE:figures/full_fig_p042_33.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Domain-Aware Scaling Laws Uncover Data Synergy

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Domain-aware scaling laws with fitted γ and σ synergy terms recover stable code-math interactions from observational LLM mixtures and correctly predict mixture rankings in controlled small-scale trainings.

  2. Prescriptive Scaling Reveals the Evolution of Language Model Capabilities

    cs.LG 2026-02 conditional novelty 6.0 of 10

    For most benchmarks, the best achievable post-training accuracy follows a stable sigmoid curve in pre-training compute; math reasoning is the exception, with a boundary that keeps rising over time.

Reference graph

Works this paper leans on

79 extracted references · 48 canonical work pages · cited by 2 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023

    arXiv 2023

  2. [2]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219 , 2024

    arXiv 2024

  3. [3]

    An integrated theory of the mind

    John R Anderson, Daniel Bothell, Michael D Byrne, Scott Douglass, Christian Lebiere, and Yulin Qin. An integrated theory of the mind. Psychological review , 111(4):1036, 2004

    work page 2004

  4. [4]

    Invariant risk minimization

    Martin Arjovsky, L \'e on Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893 , 2019

    arXiv 1907

  5. [5]

    A theory for emergence of complex skills in language models

    Sanjeev Arora and Anirudh Goyal. A theory for emergence of complex skills in language models. arXiv preprint arXiv:2307.15936 , 2023

    arXiv 2023

  6. [6]

    Act: A simple theory of complex cognition

    John R Anderson. Act: A simple theory of complex cognition. American psychologist , 51(4):355, 1996

    work page 1996

  7. [7]

    Claude 3.5 sonnet

    Anthropic. Claude 3.5 sonnet. https://www.anthropic.com/news/claude-3-5-sonnet, 2024

    2024

  8. [8]

    Invariant risk minimization games

    Kartik Ahuja, Shiori Sagawa, Harikrishnan Ramaswamy, Edward Kung, Yair Carmon, David Kr \"u ger, Amy Zhang, and Percy Liang. Invariant risk minimization games. In 38th International Conference on Machine Learning (ICML) , volume 139 of Proceedings of Machine Learning Research , pages 145--159, 2021

    work page 2021

Show all 79 references
  1. [9]

    Sample complexity of interventional causal representation learning

    Emre Acart \"u rk, Burak Var c , Karthikeyan Shanmugam, and Ali Tajer. Sample complexity of interventional causal representation learning. Advances in Neural Information Processing Systems , 37:39350--39385, 2024

    2024

  2. [10]

    An empirical study of scaling laws for transfer

    Matthew Barnett. An empirical study of scaling laws for transfer. arXiv preprint arXiv:2408.16947 , 2024

    2024 arXiv

  3. [11]

    Sparks of artificial general intelligence: Early experiments with gpt-4, 2023

    S \'e bastien Bubeck, Varun Chadrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023

    2023

  4. [12]

    Functional magnetic resonance imaging evidence for a hierarchical organization of the prefrontal cortex

    David Badre and Mark D'Esposito. Functional magnetic resonance imaging evidence for a hierarchical organization of the prefrontal cortex. Journal of cognitive neuroscience , 19(12):2082--2099, 2007

    2007

  5. [13]

    Johann Brehmer, Pim de Haan, Phillip Lippe, and Taco S. Cohen. Weakly supervised causal representation learning. In Advances in Neural Information Processing Systems 35 (NeurIPS) , pages 17066--17079, 2022

    2022

  6. [14]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022

    2022 arXiv

  7. [15]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877--1901, 2020

    1901

  8. [16]

    Doubly robust estimation in missing data and causal inference models

    Heejung Bang and James M Robins. Doubly robust estimation in missing data and causal inference models. Biometrics , 61(4):962--973, 2005

    2005

  9. [17]

    Learning linear causal representations from interventions under general nonlinear mixing

    Simon Buchholz, Goutham Rajendran, Elan Rosenfeld, Bryon Aragam, Bernhard Sch \"o lkopf, and Pradeep Ravikumar. Learning linear causal representations from interventions under general nonlinear mixing. Advances in Neural Information Processing Systems , 36:45419--45462, 2023

    2023

  10. [18]

    Rethink reporting of evaluation results in ai

    Ryan Burnell, Wout Schellaert, John Burden, Tomer D Ullman, Fernando Martinez-Plumed, Joshua B Tenenbaum, Danaja Rutar, Lucy G Cheke, Jascha Sohl-Dickstein, Melanie Mitchell, et al. Rethink reporting of evaluation results in ai. Science , 380(6641):136--138, 2023

    2023

  11. [19]

    Human cognitive abilities: A survey of factor-analytic studies

    John Bissell Carroll. Human cognitive abilities: A survey of factor-analytic studies . Cambridge university press, 1993

    1993

  12. [20]

    Structured matrix completion with applications to genomic data integration

    Tianxi Cai, T Tony Cai, and Anru Zhang. Structured matrix completion with applications to genomic data integration. Journal of the American Statistical Association , 111(514):621--633, 2016

    2016

  13. [21]

    Scaling instruction-finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research , 25(70):1--53, 2024

    2024

  14. [22]

    Skills-in-context prompting: Unlocking compositionality in large language models

    Jiaao Chen, Xiaoman Pan, Dian Yu, Kaiqiang Song, Xiaoyang Wang, Dong Yu, and Jianshu Chen. Skills-in-context prompting: Unlocking compositionality in large language models. arXiv preprint arXiv:2308.00304 , 2023

    2023 arXiv

  15. [23]

    The rising costs of training frontier ai models, 2024

    Ben Cottier, Robi Rahman, Loredana Fattorini, Nestor Maslej, and David Owen. The rising costs of training frontier ai models, 2024

    2024

  16. [24]

    Fundamental safety-capability trade-offs in fine-tuning large language models

    Pin-Yu Chen, Han Shen, Payel Das, and Tianyi Chen. Fundamental safety-capability trade-offs in fine-tuning large language models. arXiv preprint arXiv:2503.20807 , 2025

    2025 arXiv

  17. [25]

    Dorner, and Moritz Hardt

    Ricardo Dominguez-Olmedo, Florian E. Dorner, and Moritz Hardt. Training on the test task confounds evaluation and emergence. In The Thirteenth International Conference on Learning Representations , 2025

    2025

  18. [26]

    Identifiability, separability, and uniqueness of linear ica models

    Jan Eriksson and Visa Koivunen. Identifiability, separability, and uniqueness of linear ica models. IEEE signal processing letters , 11(7):601--604, 2004

    2004

  19. [27]

    Principal stratification in causal inference

    Constantine E Frangakis and Donald B Rubin. Principal stratification in causal inference. Biometrics , 58(1):21--29, 2002

    2002

  20. [28]

    Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars

    Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01307 , 2025

    2025 arXiv

  21. [29]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024

    2024 arXiv

  22. [30]

    A closer look at the limitations of instruction tuning

    Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Deepali Aneja, Zeyu Jin, Ramani Duraiswami, Dinesh Manocha, et al. A closer look at the limitations of instruction tuning. arXiv preprint arXiv:2402.05119 , 2024

    2024 arXiv

  23. [31]

    Time travel in llms: Tracing data contamination in large language models

    Shahriar Golchin and Mihai Surdeanu. Time travel in llms: Tracing data contamination in large language models. arXiv preprint arXiv:2308.08493 , 2023

    2023 arXiv

  24. [32]

    The false promise of imitating proprietary llms

    Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary llms. arXiv preprint arXiv:2305.15717 , 2023

    2023 arXiv

  25. [33]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 , 2025

    2025 arXiv

  26. [34]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 , 2020

    2009 arXiv

  27. [35]

    Training compute-optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022

    2022 arXiv

  28. [36]

    A sober look at progress in language model reasoning: Pitfalls and paths to reproducibility

    Andreas Hochlehnert, Hardik Bhatnagar, Vishaal Udandarao, Samuel Albanie, Ameya Prabhu, and Matthias Bethge. A sober look at progress in language model reasoning: Pitfalls and paths to reproducibility. arXiv preprint arXiv:2504.07086 , 2025

    2025

  29. [37]

    Does rlhf scale? exploring the impacts from data, model, and method

    Zhenyu Hou, Pengfan Du, Yilin Niu, Zhengxiao Du, Aohan Zeng, Xiao Liu, Minlie Huang, Hongning Wang, Jie Tang, and Yuxiao Dong. Does rlhf scale? exploring the impacts from data, model, and method. arXiv preprint arXiv:2412.06000 , 2024

    2024 arXiv

  30. [38]

    a rinen, Jarmo Hurri, Patrik O Hoyer, Aapo Hyv \

    Aapo Hyv \"a rinen, Jarmo Hurri, Patrik O Hoyer, Aapo Hyv \"a rinen, Jarmo Hurri, and Patrik O Hoyer. Independent component analysis . Springer, 2009

    2009

  31. [39]

    Causal discovery from heterogeneous/nonstationary data

    Biwei Huang, Kun Zhang, Jiji Zhang, Joseph Ramsey, Ruben Sanchez-Romero, Clark Glymour, and Bernhard Sch \"o lkopf. Causal discovery from heterogeneous/nonstationary data. Journal of Machine Learning Research , 21(89):1--53, 2020

    2020

  32. [40]

    Learning linear causal representations from general environments: Identifiability and intrinsic ambiguity

    Jikai Jin and Vasilis Syrgkanis. Learning linear causal representations from general environments: Identifiability and intrinsic ambiguity. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024

    2024

  33. [41]

    Mistral 7b

    Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L'elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, T...

    arXiv

  34. [42]

    The construct of creativity: Structural model for self-reported creativity ratings

    James C Kaufman, Jason C Cole, and John Baer. The construct of creativity: Structural model for self-reported creativity ratings. The Journal of Creative Behavior , 43(2):119--134, 2009

    2009

  35. [43]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 , 2020

    2001 arXiv

  36. [44]

    The architecture of cognitive control in the human prefrontal cortex

    Etienne Koechlin, Chrystele Ody, and Fr \'e d \'e rique Kouneiher. The architecture of cognitive control in the human prefrontal cortex. Science , 302(5648):1181--1185, 2003

    2003

  37. [45]

    Solving quantitative reasoning problems with language models

    Aitor Lewkowycz, Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with l...

    2022

  38. [46]

    Holistic evaluation of language models

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. Transactions on Machine Learning Research , 2023

    2023

  39. [47]

    Not-just-scaling laws: Towards a better understanding of the downstream impact of language model design decisions

    Emmy Liu, Amanda Bertsch, Lintang Sutawika, Lindia Tjuatja, Patrick Fernandes, Lara Marinov, Michael Chen, Shreya Singhal, Carolin Lawrence, Aditi Raghunathan, et al. Not-just-scaling laws: Towards a better understanding of the downstream impact of language model design decisi...

    2025

  40. [48]

    a tsch, Bernhard Sch \

    Francesco Locatello, Ben Poole, Gunnar R \"a tsch, Bernhard Sch \"o lkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In 37th International Conference on Machine Learning (ICML) , volume 119 of Proceedings of Machine Learning ...

    2020

  41. [49]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference...

    2022

  42. [50]

    Ian R. McKenzie, Alexander Lyzhov, Michael Martin Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Xudong Shen, Joe Cavanagh, Andrew George Gritsevskiy, Derik Kauffman, Aaron T. Kirtland, Zhengping Zhou, Yuhui Zhang, Sicong Huang, Daniel Wurgaft, Max Weiss, Al...

    2023

  43. [51]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:277...

    2022

  44. [52]

    Causal diagrams for empirical research

    Judea Pearl. Causal diagrams for empirical research. Biometrika , 82(4):669--688, 1995

    1995

  45. [53]

    On the identifiability of bayesian factor analytic models

    Panagiotis Papastamoulis and Ioannis Ntzoufras. On the identifiability of bayesian factor analytic models. Statistics and Computing , 32(2):23, 2022

    2022

  46. [54]

    Sloth: scaling laws for llm skills to predict multi-benchmark performance across families

    Felipe Maia Polo, Seamus Somerstep, Leshem Choshen, Yuekai Sun, and Mikhail Yurochkin. Sloth: scaling laws for llm skills to predict multi-benchmark performance across families. arXiv preprint arXiv:2412.06540 , 2024

    2024

  47. [55]

    Evolm: In search of lost language model training dynamics, 2025

    Zhenting Qi, Fan Nie, Alexandre Alahi, James Zou, Himabindu Lakkaraju, Yilun Du, Eric Xing, Sham Kakade, and Hanlin Zhang. Evolm: In search of lost language model training dynamics, 2025. Manuscript

    2025

  48. [56]

    Safetywashing: Do ai safety benchmarks actually measure safety progress? Advances in Neural Information Processing Systems , 37:68559--68594, 2025

    Richard Ren, Steven Basart, Adam Khoja, Alice Gatti, Long Phan, Xuwang Yin, Mantas Mazeika, Alexander Pan, Gabriel Mukobi, Ryan Kim, et al. Safetywashing: Do ai safety benchmarks actually measure safety progress? Advances in Neural Information Processing Systems , 37:68559--68...

    2025

  49. [57]

    A simpler approach to matrix completion

    Benjamin Recht. A simpler approach to matrix completion. Journal of Machine Learning Research , 12(12), 2011

    2011

  50. [58]

    Maddison, and Tatsunori Hashimoto

    Yangjun Ruan, Chris J. Maddison, and Tatsunori Hashimoto. Observational scaling laws and the predictability of langauge model performance. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024

    2024

  51. [59]

    The architecture of complexity

    Herbert A Simon. The architecture of complexity. In The Roots of Logistics , pages 335--361. Springer, 2012

    2012

  52. [60]

    Toward causal representation learning

    Bernhard Sch \"o lkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE , 109(5):612--634, 2021

    2021

  53. [61]

    Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems , 36:55565--55581, 2023

    Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems , 36:55565--55581, 2023

    2023

  54. [62]

    Linear causal disentanglement via interventions

    Chandler Squires, Anna Seigal, Salil S Bhate, and Caroline Uhler. Linear causal disentanglement via interventions. In International conference on machine learning , pages 32540--32560. PMLR, 2023

    2023

  55. [63]

    Challenging big-bench tasks and whether chain-of-thought can solve them

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261 , 2022

    arXiv

  56. [64]

    Matching methods for causal inference: A review and a look forward

    Elizabeth A Stuart. Matching methods for causal inference: A review and a look forward. Statistical science: a review journal of the Institute of Mathematical Statistics , 25(1):1, 2010

    2010

  57. [65]

    Multitask prompted training enables zero-shot task generalization

    Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyo...

    2022

  58. [66]

    How to grow a mind: Statistics, structure, and abstraction

    Joshua B Tenenbaum, Charles Kemp, Thomas L Griffiths, and Noah D Goodman. How to grow a mind: Statistics, structure, and abstraction. science , 331(6022):1279--1285, 2011

    2011

  59. [67]

    Gemma 2: Improving open language models at a practical size

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118 , 2024

    2024 arXiv

  60. [68]

    Reliable and efficient amortized model-based evaluation

    Sang Truong, Yuheng Tu, Percy Liang, Bo Li, and Sanmi Koyejo. Reliable and efficient amortized model-based evaluation. arXiv preprint arXiv:2503.13335 , 2025

    2025 arXiv

  61. [69]

    u gelgen, Michel Besserve, Liang Wendong, Luigi Gresele, Armin Keki \'c , Elias Bareinboim, David Blei, and Bernhard Sch \

    Julius von K \"u gelgen, Michel Besserve, Liang Wendong, Luigi Gresele, Armin Keki \'c , Elias Bareinboim, David Blei, and Bernhard Sch \"o lkopf. Nonparametric identifiability of causal representations from unknown interventions. Advances in Neural Information Processing Syst...

    2023

  62. [70]

    Yixin Wang and Michael I. Jordan. Desiderata for representation learning: A causal perspective. arXiv , 2022

    2022

  63. [71]

    Emergent abilities of large language models

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 , 2022

    2022 arXiv

  64. [72]

    Skill-mix: A flexible and expandable family of evaluations for ai models

    Dingli Yu, Simran Kaur, Arushi Gupta, Jonah Brown-Cohen, Anirudh Goyal, and Sanjeev Arora. Skill-mix: A flexible and expandable family of evaluations for ai models. arXiv preprint arXiv:2310.17567 , 2023

    2023 arXiv

  65. [73]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115 , 2024

    2024 arXiv

  66. [74]

    Unveiling the impact of coding data instruction fine-tuning on large language models reasoning

    Xinlu Zhang, Zhiyu Zoey Chen, Xi Ye, Xianjun Yang, Lichang Chen, William Yang Wang, and Linda Ruth Petzold. Unveiling the impact of coding data instruction fine-tuning on large language models reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume...

    2025

  67. [75]

    Identifiability guarantees for causal disentanglement from soft interventions

    Jiaqi Zhang, Kristjan Greenewald, Chandler Squires, Akash Srivastava, Karthikeyan Shanmugam, and Caroline Uhler. Identifiability guarantees for causal disentanglement from soft interventions. Advances in Neural Information Processing Systems , 36:50254--50292, 2023

    2023

  68. [76]

    When scaling meets llm finetuning: The effect of data, model and finetuning method

    Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets llm finetuning: The effect of data, model and finetuning method. In The Twelfth International Conference on Learning Representations , 2024

    2024

  69. [77]

    Echo chamber: Rl post-training amplifies behaviors learned in pretraining

    Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: Rl post-training amplifies behaviors learned in pretraining. arXiv preprint arXiv:2504.07912 , 2025

    2025 arXiv

  70. [78]

    Investigating the catastrophic forgetting in multimodal large language models

    Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma. Investigating the catastrophic forgetting in multimodal large language models. arXiv preprint arXiv:2309.10313 , 2023

    2023 arXiv

  71. [79]

    Causal representation learning from multiple distributions: A general setting

    Kun Zhang, Shaoan Xie, Ignavier Ng, and Yujia Zheng. Causal representation learning from multiple distributions: A general setting. In Forty-first International Conference on Machine Learning , 2024

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.