Pith. sign in

REVIEW 2 major objections 4 minor 34 references

The paper argues that Kolmogorov–Arnold Networks in small language models are best understood as audit instruments — every scalar edge function can be reconstructed and pruned — rather than as replacements for MLP feed-forward blocks, which

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 23:03 UTC pith:YQBOFF5M

load-bearing objection A careful, honest negative result on KAN FFN replacements at small scale, with a reusable edge-audit methodology; the 'no advantage' claim needs a matching-horizon MLP baseline to be fully tight. the 2 major comments →

arxiv 2607.15525 v1 pith:YQBOFF5M submitted 2026-07-17 cs.LG cs.AI

Kolmogorov--Arnold Networks for Small Language Models

Felippe Alves , Renato Vicente This is my paper
classification cs.LG cs.AI
keywords Kolmogorov–Arnold Networksfeed-forward replacementlanguage modelsinterpretabilityedge-function auditBabyLMpruningbenchmark equivalence
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests two promises of Kolmogorov–Arnold Networks (KANs) inside small language models: that their learned edge functions are interpretable, and that they can replace transformer feed-forward blocks. It finds the first promise, scoped to low-capacity bases, is real: every one of the 884,736 edge functions in a small B-spline KAN can be reconstructed, most are nonlinear, and activity-ranked pruning removes 20–25% with negligible loss. The second promise fails at this scale: on the standardized BabyLM benchmark with ten seeds and 59,875 BLiMP pairs, KAN-family and gated feed-forward variants are statistically equivalent to a vanilla MLP, and the one sub-point effect reverses on the supplement. Validation cross-entropy also fails to predict benchmark ranking. The paper's conclusion is that small-basis KANs are diagnostic instruments, not drop-in replacements, and that stronger evidence would be needed before claiming otherwise.

Core claim

The central claim, stated as the authors would state it: in a six-layer, roughly 10-million-parameter B-spline KAN language model, all 884,736 scalar feed-forward edge functions can be reconstructed and activity-ranked; 87.8% are meaningfully nonlinear, only 0.4% are inactive, and pruning the lowest-activity 20–25% costs less than 0.005 nats of validation loss. This audit is corpus-transferable (it replicates on BabyLM), but the impressive functional-PCA compression and closed-form fits are properties of the six-dimensional grid-2 basis, not of KANs in general. On standardized replacement tests, the KAN-family variants and SwiGLU improve validation loss over a GELU MLP yet are benchmark-equi

What carries the argument

The central object is the edge function of a KAN feed-forward layer: a learned univariate scalar function on each input-output edge, which for B-spline KANs is a weighted sum of a SiLU base and learned B-spline basis functions. The audit machinery reconstructs these 884,736 functions exactly, scores them by a nonlinearity score (NLS, the relative distance from the best affine fit), applies functional PCA and closed-form fitting, and uses reconstructed-curve activity to prune edges. The replacement comparison's key mechanism is a matched-horizon training recipe paired with a standardized zero-shot benchmark suite, which separates validation cross-entropy from benchmark accuracy and exposes th

Load-bearing premise

The matched 8,000-step training recipe — same batch size, same cosine schedule, no per-architecture tuning — is a fair test for every architecture; if KAN variants were systematically under-optimized by that recipe, the benchmark-equivalence conclusion could be an artifact.

What would settle it

Continue the SwiGLU MLP baseline at 286M parameters to the same 5,040 steps and reset schedule as the stabilized GR-KAN run: if GR-KAN no longer trails (or beats) the MLP at a matched horizon, the cautionary scaling conclusion changes. Similarly, a per-architecture hyperparameter sweep on BabyLM with 20 seeds that yields a consistent >1-point BLiMP advantage for any KAN variant would refute the no-consistent-advantage claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Small-basis KAN feed-forward layers can be used as interpretability instruments in small language models: every scalar edge function is reconstructable and rankable by activity.
  • The tested KAN-family and gated feed-forward networks are benchmark-equivalent to a vanilla MLP on BabyLM; validation-loss gains do not translate into standardized-accuracy gains.
  • Validation cross-entropy does not predict benchmark ranking at this scale, so small-scale validation-loss screens alone cannot support an architecture-replacement claim.
  • The strongest interpretability summaries (near-total fPCA compression, high closed-form fit rates) are properties of the low-capacity grid-2 basis, not of trained KAN layers in general; they weaken as the grid grows.
  • Larger stress tests remain cautionary: a parameter-matched MLPEdge transformer underperforms an MLP on Wikitext-103, and a corrected 286M-parameter rational GR-KAN trails a SwiGLU MLP baseline at matched horizons.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the validation-loss/benchmark dissociation holds generally, architecture research should adopt standardized zero-shot benchmarks as the primary outcome measure, treating validation loss as a weak proxy at small scale.
  • Because the interpretability results are scoped to low-capacity bases, the audit's usefulness at larger scale and higher grid sizes is open; future work may need new summary metrics for rougher, higher-dimensional edge functions.
  • The faithful, separable scalar edge decomposition could enable targeted interventions — edge-level pruning or editing — that MLP hidden units do not offer, even though no benchmark gain appears; this application is plausible but not demonstrated.
  • If the no-consistent-advantage result survives tuned recipes, the practical takeaway is to spend compute on MLP/SwiGLU baselines and reserve KANs for diagnosis, redirecting KAN research toward settings where their interpretable scalar decomposition is the goal rather than language-model throughput.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. This paper empirically separates two premises for KANs in small language models: (i) interpretability — that learned scalar edge functions in a B-spline KAN can be exhaustively reconstructed and summarized — and (ii) replacement — that KAN-family feed-forward networks improve over MLP baselines. At GuppyLM scale the authors reconstruct all 884,736 edges of a six-layer grid-2 B-spline KAN, reporting high nonlinearity, high fPCA compression, closed-form approximability, and graceful activity-based pruning; however, random-init controls and a grid-size sweep show that the fPCA/closed-form properties are substantially basis-imposed and scoped to small-basis KANs. On BabyLM Strict-Small with ten seeds and the official evaluation pipeline, KAN-family and gated variants improve validation cross-entropy over a GELU MLP but are benchmark-equivalent on BLiMP/EWoK, with the only robust BLiMP effect reversing on the supplement. Latency is worse for KAN-family FFNs, and larger stress tests (GPT-2-small MLPEdge, 286M GR-KAN on ClimbMix) remain cautionary. The paper concludes that small-basis KANs are useful audit instruments, not demonstrated drop-in FFN replacements.

Significance. The study's main value is a well-powered, reproducible evaluation of a currently popular architecture family, combined with a genuinely exhaustive interpretability audit. Strengths include: all 884,736 edge functions reconstructed rather than sampled; ten-seed BabyLM evaluation with a bit-exact HuggingFace wrapper and the public pipeline; a strong SwiGLU baseline; random-initialization controls; cross-corpus replication; separate reporting of quality and latency; explicit limitation statements; and a detailed provenance manifest. The grid-size sweep and the MLP pruning baseline usefully scope the interpretability claims. If the conclusions hold, the paper provides a standardized template for evaluating KAN-style FFN replacements and a concrete diagnostic instrument for small models. The main reservations — the fairness of a single shared training recipe for the negative conclusion and the 'comparable validation loss' wording in the grid sweep — are fixable with additional matched baselines or precise rewording.

major comments (2)
  1. [§4.1, grid-size sweep] The claim that 'all three grid sizes reach comparable validation loss' is contradicted by the numbers in the same paragraph: 0.288 (grid 2), 0.312 (grid 5), and 0.371 (grid 10 after 16,000 steps). A gap of 0.083 nats (≈29% relative) is not 'comparable' in the usual sense, and grid 20 reaches only 3.02. This matters because the next sentence — 'this degradation reflects basis capacity rather than undertraining' — is load-bearing for scoping the interpretability claim to small-basis KANs. If the larger-grid models are simply harder to optimize with the chosen recipe, the drop in fPCA and closed-form coverage could be an optimization artifact. Please either train grids 5 and 10 to a genuinely matched loss (or report the best achievable loss under a longer or better-tuned schedule), or explicitly replace 'comparable validation loss' with a statement that the larger grids did not reach the gr
  2. [§4.3 / Appendix C / §3.1] The central negative conclusion ('no consistent advantage') is obtained under a single shared 8,000-step training recipe for all architectures. Appendix C shows that a stabilized, longer 5,040-step GR-KAN continuation narrows the 286M-parameter validation-BPB gap from +0.0440 to +0.0152 relative to the MLP's 2,520-step endpoint, with no matched 5,040-step MLP continuation. The authors correctly label this as an optimization-stability observation rather than an architecture comparison, but it is direct evidence that the default recipe can systematically disadvantage KAN-family optimization. Because the BabyLM no-advantage result uses the same default recipe, the abstract's unqualified 'no consistent benchmark, quality, or latency advantage' is stronger than the matched-horizon evidence supports. I recommend adding a matched longer-horizon MLP baseline at least in the 286M stress test (and
minor comments (4)
  1. [Abstract and §5] The phrase 'no consistent benchmark, quality, or latency advantage' is in tension with Table 1, where all three non-MLP critical architectures have significantly lower BabyLM validation cross-entropy (3.770/3.781/3.800 vs 3.820) — a quality metric as defined in §3.2. Please clarify that 'quality' refers to standardized-benchmark quality, or explicitly acknowledge the validation-loss advantage and state why it does not constitute a replacement win.
  2. [§4.1] The sentence 'An earlier draft reported an 82.3% random-init figure...' is an important transparency note but breaks the flow of the results; consider moving it to an appendix or footnote and keeping the main text focused on the reproducible finding.
  3. [Table 1] 'Across ten seeds and 59,875 BLiMP minimal pairs' is ambiguous: 59,875 is the size of the BLiMP evaluation set per seed, not the total number of decisions across seeds. Please rephrase to avoid implying that only 59,875 decisions were made.
  4. [§4.2 / Appendix E] Because the same ten seeds are reused across architectures, paired-seed comparisons would be more natural for the BLiMP differences than Welch independent-sample tests. The paper flags this, but reporting paired differences as the primary analysis (or as a prominently placed robustness check) would strengthen the null conclusion.

Circularity Check

0 steps flagged

No significant circularity: the paper is an empirical benchmark study whose interpretability claims are checked against random-init controls and external standardized benchmarks.

full rationale

This is an empirical study, not a derivation, and none of its central claims reduce to its inputs by construction. The interpretability audit is self-contained: edge functions are reconstructed from the trained KAN, and the headline fPCA/closed-form findings are explicitly shown to be basis-capacity properties of the grid-2 architecture rather than learned effects, with random-initialization controls and a grid-size sweep (Section 4.1). The pruning result is compared against matched MLP baselines rather than being presented as a unique KAN property. The replacement claim rests on external standardized benchmarks (BabyLM, BLiMP, EWoK, Wikitext-103) with ten seeds, and the paper explicitly disclaims formal statistical equivalence (Appendix E). The stabilized 5,040-step GR-KAN continuation is labeled as an unmatched optimization-stability observation, not an architecture comparison, and the matched-horizon comparison is reported separately (Section 4.3, Appendix C). The only load-bearing assumption is the fairness of the shared 8,000-step training recipe; this is an external-validity limitation, not a circularity, because the results are not defined in terms of that assumption. There are no self-citations, no fitted parameter renamed as a prediction, and no imported uniqueness theorem. The paper even retracts an earlier unreproducible random-init figure, further indicating that the claims are not forced by construction.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 1 invented entities

The central claims are empirical and rest on analysis thresholds (NLS, activity, fPCA truncation, pruning tolerance) and domain assumptions (benchmark validity, training-recipe fairness). There are no fitted constants being 'predicted'; the most fragile premise is the grid-sweep comparability, which the evidence contradicts.

free parameters (6)
  • NLS nonlinearity threshold = 0.1
    Chosen threshold for 'meaningfully nonlinear' edges; 87.8% claim depends on it (sensitivity reported: 97.3% at 0.05, 66.8% at 0.20).
  • Inactive-edge activity threshold = 0.01
    Edges with activity ≤0.01 counted as inactive; affects 0.4% claim.
  • fPCA truncation = top-4 components
    Near-total variance claim uses 4 components; at grid-2 rank is capped at 6.
  • Closed-form library and R^2 threshold = 6 functions, R²≥0.99
    Top-50 active edges per layer; 93.3% coverage claim is library- and domain-dependent.
  • Pruning loss tolerance = ≤0.005 nats (20–25% prunable)
    The 'negligible loss increase' claim is defined by this tolerance; 30% prunable at ≤0.01.
  • Grid-size sweep 'comparable validation loss' = 0.288/0.312/0.371 for grids 2/5/10
    The claim that degradation reflects basis capacity rather than undertraining assumes comparability; the losses differ by ~29% relative CE.
axioms (5)
  • standard math B-spline grid-2 edge function space is 6-dimensional (5 splines + SiLU base), capping fPCA rank at 6.
    Used to reinterpret near-total top-4 fPCA variance as basis-imposed (Section 4.1).
  • domain assumption BabyLM Strict-Small zero-shot pipeline (BLiMP, supplement, EWoK) measures linguistic competence relevant for LLM architecture comparison.
    Central yardstick for the replacement claim; EWoK at chance for all models limits its discrimination.
  • domain assumption Validation cross-entropy on the held-out split is a meaningful proxy for LM quality.
    The paper's strongest negative result is that this proxy fails to predict benchmarks; the assumption is tested and rejected.
  • domain assumption The post-hoc MLP 'effective edge' probe is a fair comparator for NLS distributions.
    Authors flag it is not a faithful circuit representation; used only descriptively (Section 3.4).
  • ad hoc to paper All three grid-size sweep models reached comparable validation loss, so degradation is due to basis capacity.
    Grid losses are 0.288/0.312/0.371; not actually comparable, so the attribution is weaker than stated.
invented entities (1)
  • MLPEdge independent evidence
    purpose: Isolates edge-wise scalar functions without spline machinery to test whether KAN topology alone helps.
    Introduced and tested in the paper; falsifiable via its Wikitext-103 underperformance (PPL 20.96 vs 16.58), so it has an external handle despite being a new architecture.

pith-pipeline@v1.3.0-alltime-deepseek · 25417 in / 14205 out tokens · 146682 ms · 2026-08-01T23:03:44.654810+00:00 · methodology

0 comments
read the original abstract

Kolmogorov--Arnold Networks (KANs) replace fixed node activations with learned one-dimensional edge functions, offering an explicit interface for interpretation and a possible alternative to transformer feed-forward networks. We test these claims separately. In a six-layer, 10M-parameter B-spline KAN, we reconstruct all 884,736 feed-forward edges: 87.8\% exceed (NLS>0.1) and 0.4\% are inactive. Pruning the lowest-activity 20--25\% causes negligible loss increase, although structured MLP neuron pruning tolerates comparable sparsity. The audit replicates on BabyLM, but grid-size sweeps show that near-total fPCA compression and high closed-form-fit coverage are properties of the low-capacity grid-2 basis, not universal KAN behavior. For replacement, we evaluate MLP, SwiGLU, grouped Chebyshev, and rational GR-KAN networks on BabyLM. The KAN-family and gated variants improve validation loss over the GELU MLP, but this ordering does not transfer to standardized benchmarks: across ten seeds and 59,875 BLiMP pairs, accuracies span 62.4--63.1\%, EWoK remains at chance, and a (+0.7)-point GR-KAN effect on BLiMP reverses on the supplement. Larger tests are also cautionary: parameter-matched MLPEdge underperforms the MLP on Wikitext-103, and 286M-parameter GR-KAN remains below a SwiGLU ClimbMix baseline after stabilization. Thus, small-basis KANs provide a practical, corpus-transferable interface for auditing learned scalar transformations, but the tested replacements show no consistent benchmark, quality, or latency advantage over strong MLP baselines.

Figures

Figures reproduced from arXiv: 2607.15525 by Felippe Alves, Renato Vicente.

Figure 1
Figure 1. Figure 1: Graphical abstract. The study separates the interpretability claim from the replacement [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Edge-function evidence at GuppyLM scale. The audit reconstructs scalar functions rather [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Pruning robustness as a function of FFN-capacity sparsity, plotted as the increase in [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Standardized BabyLM evaluation for the four critical architectures, ordered by validation [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Small-scale validation loss with seed variation. Error bars show sample standard deviation [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Post-hoc MLP effective-edge NLS distributions for the first FFN layer of the GuppyLM [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Checkpoint-level validation BPB trajectories for the ClimbMix d12 stress test. The [PITH_FULL_IMAGE:figures/full_fig_p024_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Top four functional principal components across all six B-spline KAN FFN layers. [PITH_FULL_IMAGE:figures/full_fig_p025_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Representative high-activity KAN edge functions and their best fits from the six-function [PITH_FULL_IMAGE:figures/full_fig_p026_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 18 linked inside Pith

  1. [1]

    N., Kaiser, L., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. InAdvances in Neural Information Processing Systems, 2017

    2017

  2. [2]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. OpenAI technical report, 2019

    2019

  3. [3]

    Pointer sentinel mixture models

    Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017

    2017

  4. [4]

    Y., and Tegmark, M

    Liu, Z., Wang, Y., Vaidya, S., Ruehle, F., Halverson, J., Soljačić, M., Hou, T. Y., and Tegmark, M. KAN: Kolmogorov–Arnold Networks.International Conference on Learning Representations, 2025

    2025

  5. [5]

    KAN 2.0: Kolmogorov–Arnold Networks meet science

    Liu, Z., Ma, P., Wang, Y., Matusik, W., and Tegmark, M. KAN 2.0: Kolmogorov–Arnold Networks meet science. arXiv:2408.10205, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    and Wang, X

    Yang, X. and Wang, X. Kolmogorov–Arnold Transformer.International Conference on Learning Representations, 2025. arXiv:2409.10594

    Pith/arXiv arXiv 2025

  7. [7]

    A., Islam, M

    Somvanshi, S., Javed, S. A., Islam, M. M., Pandit, D., and Das, S. A survey on Kolmogorov– Arnold Network. arXiv:2411.06078, 2024. 28

    Pith/arXiv arXiv 2024

  8. [8]

    A practitioner’s guide to Kolmogorov–Arnold Networks

    Noorizadegan, A., Wang, S., and Ling, L. A practitioner’s guide to Kolmogorov–Arnold Networks. arXiv:2510.25781, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    and Wang, S

    Noorizadegan, A. and Wang, S. Scale-parameter selection in Gaussian Kolmogorov–Arnold Networks. arXiv:2604.21174, 2026

    Pith/arXiv arXiv 2026

  10. [10]

    EfficientKAN: Efficient implementation of KAN.https://github.com/Blealtan/ efficient-kan, 2024

    Blealtan. EfficientKAN: Efficient implementation of KAN.https://github.com/Blealtan/ efficient-kan, 2024

    2024

  11. [11]

    Kolmogorov–Arnold Networks are radial basis function networks

    Li, Z. Kolmogorov–Arnold Networks are radial basis function networks. arXiv:2405.06721, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    and Chen, H

    Bozorgasl, Z. and Chen, H. Wav-KAN: Wavelet Kolmogorov–Arnold Networks. arXiv:2405.12832, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Aghaei, A. A. rKAN: Rational Kolmogorov–Arnold Networks. arXiv:2406.14495, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    S., Keerthana, A

    Sidharth, S. S., Keerthana, A. R., Gokul, R., and Anas, K. P. Chebyshev polynomial-based Kolmogorov–Arnold Networks: An efficient architecture for nonlinear function approximation. arXiv:2405.07200, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    GLU variants improve transformer

    Shazeer, N. GLU variants improve transformer. arXiv:2002.05202, 2020

    Pith/arXiv arXiv 2002

  16. [16]

    Chowdhery, A. et al. PaLM: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240):1–113, 2022

    2022

  17. [17]

    Touvron, L. et al. LLaMA: Open and efficient foundation language models. arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  18. [18]

    Elhage, N. et al. Superposition, memorization, and double descent.Transformer Circuits Thread, 2022

    2022

  19. [19]

    and Tegmark, M

    Gurnee, W. and Tegmark, M. Finding neurons in a haystack: Case studies with sparse probing. Transactions on Machine Learning Research, 2023

    2023

  20. [20]

    Cunningham, H. et al. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv:2304.05923, 2023

    Pith/arXiv arXiv 2023

  21. [21]

    Bricken, T. et al. Towards monosemanticity: Decomposing language models with dictionary learning.Transformer Circuits Thread, 2023

    2023

  22. [22]

    Neural additive models: Interpretable machine learning with neural nets

    Agarwal, R., Melnick, L., Frosst, N., Zhang, X., Lengerich, B., Caruana, R., and Hinton, G. Neural additive models: Interpretable machine learning with neural nets. InAdvances in Neural Information Processing Systems, 2021

    2021

  23. [23]

    Interpretable machine learning for science with PySR and SymbolicRegression.jl

    Cranmer, M. Interpretable machine learning for science with PySR and SymbolicRegression.jl. arXiv:2305.01582, 2023

    Pith/arXiv arXiv 2023

  24. [24]

    Li, J. et al. DataComp-LM: In search of the next generation of training sets for language models. arXiv:2406.11794, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    nanochat: The best ChatGPT that $100 can buy

    Karpathy, A. nanochat: The best ChatGPT that $100 can buy. https://github.com/ karpathy/nanochat, 2025

    2025

  26. [26]

    and Li, Y

    Eldan, R. and Li, Y. TinyStories: How small can language models be and still speak coherent English? arXiv:2305.07759, 2023. 29

    Pith/arXiv arXiv 2023

  27. [27]

    Warstadt, A. et al. Findings of the BabyLM challenge.Proceedings of the 27th Conference on Computational Natural Language Learning, 2023

    2023

  28. [28]

    Evaluation pipeline for the 2025 BabyLM Challenge.https://github

    BabyLM Challenge. Evaluation pipeline for the 2025 BabyLM Challenge.https://github. com/babylm/evaluation-pipeline-2025, 2025

    2025

  29. [29]

    Warstadt, A., Parrish, A., Liu, H., Mohananey, A., Peng, W., Wang, S.-F., and Bowman, S. R. BLiMP: The benchmark of linguistic minimal pairs for English.Transactions of the Association for Computational Linguistics, 8:377–392, 2020

    2020

  30. [30]

    Ivanova, A. A. et al. Elements of World Knowledge (EWoK): A cognition-inspired framework for evaluating basic world knowledge in language models. arXiv:2405.09605, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    Zhang, Z. et al. TinyLlama: An open-source small language model. arXiv:2401.02385, 2024

    Pith/arXiv arXiv 2024

  32. [32]

    Hu, S. et al. MiniCPM: Unveiling the potential of small language models with scalable training strategies. arXiv:2404.06395, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    Groeneveld, D. et al. OLMo: Accelerating the science of language models.Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024

    2024

  34. [34]

    GuppyLM-60K-Generic: A fish-personality chatbot dataset.https://huggingface

    arman-bd. GuppyLM-60K-Generic: A fish-personality chatbot dataset.https://huggingface. co/datasets/arman-bd/guppylm-60k-generic, 2024. 30

    2024