Predictable Emergent Abilities of LLMs: Proxy Tasks Are All You Need

Bo-Wen Zhang; Boxiang Yang; Guang Liu; Yan Yan; Yifei Xue

REVIEW 4 major objections 4 minor 40 references

Predictable Emergent Abilities of LLMs: Proxy Tasks Are All You Need

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Emergent abilities like tool use can be predicted early in training from carefully chosen simple proxy tasks, and the paper shows the prediction holding across five differently trained models.

desk verdict A plausible two-stage proxy-task selection method, but the validation is too thin to back the strong-correlation claim; still worth a referee. read the letter →

arxiv 2412.07111 v1 pith:C25AIZD6 submitted 2024-12-10 cs.CL

Bo-Wen Zhang , Yan Yan , Boxiang Yang , Yifei Xue , Guang Liu This is my paper · ORCID

classification cs.CL

keywords emergentabilitiesproxytaskstoolutilizationearly-stageevaluationperformancepredictionscalinglawstaskrelevancerobustnessanalysis

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper takes aim at a known blind spot of scaling laws: they predict steady gains on tasks small models already perform, but not 'emergent' abilities that are absent in small or early-stage models. Its proposal is to predict such abilities anyway, by selecting simple, non-emergent tasks that are strongly and stably correlated with the target ability and blending their early-training scores into a single metric. Selection is data-driven: candidate tasks are ranked by how their scores across many existing models track the target benchmark's scores, then filtered by how robust each task is across small ensembles of models trained with different data versus different seeds. In a case study on tool use, the resulting early metric ranks five large models, trained with different annealing schedules and data mixes, almost exactly in the order of their final tool-use evaluations, while perplexity ranks them far less reliably. The paper's claim is that emergent abilities are predictable in advance if one measures the right simple tasks early.

What carries the argument

The load-bearing object is a two-stage selection pipeline built on a double-normalized performance matrix. Each candidate task is represented by a vector of scores across many models, normalized first across tasks and then across models, so that task difficulty and model scale do not dominate the comparison. Relevance to the target task is the Kendall rank correlation between the task vector and the target vector, chosen over Pearson and Spearman through a resampling check that keeps the variants whose top-ranked tasks survive random re-sampling of models. Robustness is the variance ratio $R_i = \sigma_{di}^2 / \sigma_{ri}^2$ between a task's scores on small models trained with varied data and on small models trained from varied seeds with identical data, the idea being that a stable proxy should be driven by data distribution, not random noise. The final proxy weight combines relevance with a sigmoid-transformed robustness score, and the early prediction is the normalized weighted average of the surviving tasks' scores.

What would settle it

Train a new family of models with a different architecture or substantially different data mix, and compare the paper's early proxy ranking against their final T-eval ranking; if the proxy ordering no longer matches, the relevance structure has shifted and the method fails. A cheaper check is to recompute the Kendall relevance rankings of the 42 candidate tasks on a fresh set of public models and see whether the top proxy tasks remain the same.

Watch

Extended reading notes

Core claim

The central claim is that the emergent ability of tool utilization, which scaling laws cannot forecast, is predictable from proxy tasks evaluated early in training. The authors build an early-stage metric from six selected multiple-choice tasks, weighting each by its Kendall correlation with the T-eval benchmark across 17 model pairs and by a robustness ratio that compares a task's variance across data-varied small models with its variance across seed-varied small models. They then show that this metric orders five large models, trained with or without learning-rate annealing and under three data-selection strategies, in nearly the same sequence as their final T-eval scores: only one pair of ten is reversed, versus four reversals for a perplexity-based ranking. This is presented as evidence that complex capabilities can be evaluated, and training configurations optimized, during the early phases of model development rather than after full training.

Load-bearing premise

The argument assumes that the task-relevance structure measured on 17 existing public model pairs carries over to newly trained models with different architectures, data mixtures, and training schedules, and the direct validation covers only five Qwen-based models under two kinds of training variation.

Editorial extensions

If this is right

Training configurations can be compared and adjusted at early checkpoints using the proxy metric, instead of waiting for full pretraining and fine-tuning.
Perplexity, the standard early-training metric, is a weaker ordering signal for tool-use capability than the selected proxy blend.
The same selection pipeline applies to other emergent abilities by substituting the target benchmark and the candidate task pool.
Because candidate-task scores are drawn from public leaderboards, choosing proxies requires no additional large-model evaluations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the relevance structure proves stable across training regimes, the proxy metric could serve as an optimization signal for data-mix and schedule choices, not just a diagnostic.
The variance-ratio robustness test is a general screening tool that could identify which benchmarks deserve trust at small scale, apart from any prediction goal.
The results support a picture in which tool use is not a discontinuous capacity but a composite of measurable precursors, suggesting that apparent 'emergence' may partly be an artifact of evaluation scale.
A stress test the paper does not run is applying the full pipeline to a second emergent ability and to a non-Qwen model family before the method is relied on.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this paper has a genuinely new two-stage method for picking proxy tasks, but the empirical evidence for the central claim is a lot weaker than the abstract suggests. If you're deciding whether to read it, treat it as an interesting case study, not a confirmed technique.

What's new: the combination of cross-model relevance (Kendall correlation on performance vectors) with a variance-ratio robustness score from small model ensembles is not in the cited literature. The authors also did real work: they assembled 17 chat/base pairs for relevance, trained eight 1.8B models from scratch for robustness, and then trained five 18B Qwen variants to test prediction. That's more effort than most 'emergent ability' papers put in, and the writing is generally clear about the two stages.

Soft spots: the prediction formula is under-specified. The relevance threshold, robustness threshold, and sigmoid slope k are never given, and the aggregated 'Proxy Task' score in Tables 6, 8, and 10 has no closed-form definition, so no one can reproduce the integration. The validation is five Qwen-18B runs, all from one architecture and one SFT recipe. That makes the transfer assumption real: relevance computed on public model pairs may not hold for a different architecture or training setup. With five items, one reversed pair out of ten is not strong evidence of a 'strong correlation,' and there are no error bars or magnitude predictions. The relevance weights are also fit to T-eval on the public model pairs, so the prediction is partly fitted to the target; the held-out models give some independence, but not enough to establish generalizability. There are also minor notation issues (m=42 vs 17 pairs; a mismatched Baichuan pair in Table 2), but those are fixable.

The reader's take is basically right that the central claim is not well-supported as stated. I'd add that the method is plausible and the experiments are real, so I'd rather see this go to a serious referee than get desk-rejected. The paper is for people who build early-stage evaluation pipelines or do model selection for tool-use; they may get useful ideas from the method, but they shouldn't rely on the current evidence as validation. If I were handling it, I'd send it to review with the expectation that the authors need to specify the full prediction rule and add cross-architecture validation before it's convincing.

Referee Report

4 major / 4 minor

Summary. The paper proposes a method for predicting emergent LLM abilities from proxy tasks before the ability has developed. It defines task relevance as the correlation of normalized performance vectors across 17 public chat/base model pairs, validates candidate tasks using a robustness score computed from small Qwen-1.8B models trained on different data mixtures or random seeds, and then claims to integrate relevance and robustness into a weighted proxy score. In a case study on T-eval tool utilization, the authors compare proxy-task rankings of five Qwen-18B pretraining checkpoints (two annealing variants and three data-mixture variants) with post-SFT T-eval rankings and report one reversed pair for their preferred Vbc strategy.

Significance. If the method were fully specified and validated, it would offer a practical way to monitor early-stage training checkpoints and to compare training configurations without running costly target-task evaluations. The idea of transferring correlation structures from public leaderboard models to new training runs is attractive, and the two-stage relevance-plus-robustness selection is clearly presented. The manuscript's strengths are its concrete case study, its use of publicly available benchmark results, and the reproducible small-model robustness experiments. However, the current evidence does not establish the central claim: the final prediction score is not given in closed form, the relevance and robustness thresholds are free parameters, the validation set consists of five same-architecture models, and the robustness metric's interpretation is questionable. As presented, the work is a proposal with an illustrative case study rather than a validated predictive method.

major comments (4)

[Section 3.3, Tables 6, 8, 10] The final prediction score is not defined in closed form. Section 3.3 defines Si = Ci * f(Ri) and normalized weights Wi, but the manuscript never reports the values of epsilon_C, epsilon_R, k, or the normalization used to produce the 'Proxy Task' and 'Vbc' numbers in Tables 6, 8, and 10. Since the Vbc column of Table 10 reproduces exactly the 'Proxy Task' rows of Tables 6 and 8, a reader cannot determine whether the published predictions actually implement the proposed relevance-robustness weighting, and the claimed integration cannot be independently reproduced. Please provide the exact formula and all parameter values, together with a sensitivity analysis of the results to those parameters.
[Section 4.3, Table 10] The validation is too narrow to support the claim of a 'strong correlation'. The five models are all 18B-parameter Qwen models with the same architecture and the same SFT recipe, and the only reported agreement statistic is the number of reversed pairs (1/10 for Vbc). With five items, a single inversion yields Kendall's tau = 0.8, and since three prediction strategies are compared, the evidence is marginal even before considering that the relevance weights were estimated from the same model family in Section 4.1. No confidence intervals, no per-model absolute errors, and no held-out architecture or training configuration are reported. The transfer of the correlation structure from 17 public model pairs to new training runs is therefore asserted rather than demonstrated.
[Section 3.2] The robustness score Ri = sigma^2_d / sigma^2_r is not a measure of robustness in the usual sense. A task whose scores vary strongly across data-mixture pretraining runs is highly sensitive to data distribution, which is a form of instability; interpreting a high variance ratio as 'stronger robustness' is a non sequitur. The small ensemble sizes (three random-noise models and five data-variability models) also make the variance ratio noisy, and no uncertainty is reported. The selection of C3 and CHID as the most robust tasks may therefore reflect task-specific sensitivity to data mixture rather than reliability.
[Sections 3.1 and 4.1, Table 2] The performance-vector notation is inconsistent and appears to be transposed. Section 3.1 defines m as the number of models and n as the number of tasks, but Section 4.1 says 'construct an m-dimensional performance vector, where m = 42', while 42 is the number of candidate tasks and the number of model pairs is 17. Please fix the notation and confirm that the correlation analysis uses the correct dimensions. Additionally, two rows in Table 2 pair models that do not match (Baichuan2-13B-Chat with Baichuan2-7B-Base, and Mistral-7B-Instruct-v0.2 with Mixtral-7B-v0.1); these mismatches should be corrected or justified, because they directly affect the estimated relevance values.

minor comments (4)

[Section 4.1, Figure 1] Figure 1 is referenced in the text but its content is not described; if it contains the top-task rankings, the reader cannot verify the claim that reasoning and problem-solving tasks dominate the top ten without the figure.
[Section 2] The citation for OpenCompass appears to be incorrect: the benchmark suite OpenCompass is not authored by Buitrago and Nystrom (2019), and the reference should be replaced with the appropriate OpenCompass documentation or paper.
[Section 4.3] The abstract and conclusion claim a 'strong correlation' between predicted and actual performance, but no correlation coefficient, confidence interval, or significance test is reported; the rank-inversion count alone is insufficient to support that wording.
[Section 3.3] The thresholds epsilon_C and epsilon_R are introduced but never specified or varied; if they are chosen by hand on the validation set, the method is not parameter-free, and the reported results may be optimistic.

Circularity Check

0 steps flagged · score 0.0 of 10

The proxy-task predictor is fit on 17 public model pairs and then applied to five newly trained Qwen checkpoints; the validation target (T-eval) is not used to construct the proxy scores, so the central claim is not circular, though its external-validity evidence is thin.

full rationale

The derivation chain is: estimate relevance C_i by correlating each candidate task's performance vector with T-eval across the 17 existing chat/base pairs (Sec. 4.1); estimate robustness R_i from variance ratios of small Qwen-1.8B ensembles (Sec. 4.2); form weighted proxy scores S_i = C_i f(R_i) and normalize (Sec. 3.3); then evaluate five newly pretrained Qwen-18B checkpoints on the chosen proxies early in training and separately, after SFT, on T-eval (Sec. 4.3). The T-eval scores of the five validation models enter only as the comparison target in Table 10, not as inputs to the proxy-score construction. The weights are fitted to a training set of 17 models that excludes the five validation runs, so the Table 10 rank comparison is an out-of-sample check. This is a standard supervised-transfer setup; it may be under-powered (n=5, one architecture, same SFT recipe) and the proxy aggregate is under-specified (epsilon_C, epsilon_R, k, and closed form of Vbc are missing), but those are reproducibility and external-validity concerns, not circularity. No self-citation is load-bearing and no result is equivalent to its inputs by construction. Therefore no circular step is identified.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method depends on several unspecified thresholds and an unstated prediction formula. The robustness metric's interpretation is an assumption, and the transferability of relevance scores from existing models to new training runs is assumed.

free parameters (4)

Relevance threshold epsilon_C
Mentioned in Section 3.3 as a filter for candidate tasks, but no value or selection procedure is given.
Robustness threshold epsilon_R
Same, no value given.
Sigmoid slope k in f(x)=1/(1+e^{-kx})
Introduced in Section 3.3 without specifying k or how it is chosen.
Number of selected proxy tasks = 6
The validation uses six tasks (C3, CMNLI, OCNLI, CHID, RTE, CMMLU), but the paper does not explain how thresholds yield exactly these six.

assumptions (4)

domain assumption Performance vectors across models can be normalized and correlated to reveal task relevance
Section 3.1 constructs feature and sample normalized performance vectors and assumes correlations among them reflect shared task requirements.
domain assumption A high variance ratio sigma_d^2/sigma_r^2 marks a robust proxy task
Section 3.2 defines robustness R_i = sigma_d^2/sigma_r^2 and interprets higher ratios as stronger robustness, though high sensitivity to data distribution could equally indicate instability.
domain assumption Proxy task performance at early training predicts target task performance after full training and SFT
Section 4.3 validates on 5 models, implicitly assuming the correlation structure observed on existing models transfers to new training runs.
domain assumption T-eval is a valid measure of emergent tool-use ability
Section 4 selects T-eval as the target benchmark without justifying it as the canonical tool-use metric.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Predictable Emergent Abilities of LLMs: Proxy Tasks Are All You Need." pith.science (2026). https://pith.science/paper/C25AIZD6

@misc{pith2026241207111,
  author       = {Pith},
  title        = {Pith review of: Predictable Emergent Abilities of LLMs: Proxy Tasks Are All You Need},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C25AIZD6}},
  note         = {Machine review of arXiv:2412.07111}
}

read the original abstract

While scaling laws optimize training configurations for large language models (LLMs) through experiments on smaller or early-stage models, they fail to predict emergent abilities due to the absence of such capabilities in these models. To address this, we propose a method that predicts emergent abilities by leveraging proxy tasks. We begin by establishing relevance metrics between the target task and candidate tasks based on performance differences across multiple models. These candidate tasks are then validated for robustness with small model ensembles, leading to the selection of the most appropriate proxy tasks. The predicted performance on the target task is then derived by integrating the evaluation results of these proxies. In a case study on tool utilization capabilities, our method demonstrated a strong correlation between predicted and actual performance, confirming its effectiveness.

Figures

Figures reproduced from arXiv: 2412.07111 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 6 canonical work pages

[1]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023
[2]

Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, M \'e rouane Debbah, \'E tienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. 2023. The falcon series of open language models. arXiv preprint arXiv:2311.16867

arXiv 2023
[3]

Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al. 2024. Foundational challenges in assuring alignment and safety of large language models. arXiv preprint arXiv:2404.09932

arXiv 2024
[4]

Sanjeev Arora and Anirudh Goyal. 2023. A theory for emergence of complex skills in language models. arXiv preprint arXiv:2307.15936

arXiv 2023
[5]

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

arXiv 2023
[6]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

2020
[7]

Paola A Buitrago and Nicholas A Nystrom. 2019. Open compass: accelerating the adoption of ai in open research. pages 1--9

work page 2019
[8]

Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, et al. 2023. T-eval: Evaluating the tool utilization capability step by step. arXiv preprint arXiv:2312.14033

arXiv 2023

Show all 40 references

[9]

Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. 2010. Recognizing textual entailment: Rational, evaluation and approaches--erratum. Natural Language Engineering, 16(1):105--105

2010
[10]

Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang. 2024. Understanding emergent abilities of language models from the loss perspective. arXiv preprint arXiv:2403.15796

2024 arXiv
[11]

Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, et al. 2024. Language models scale reliably with over-training and on downstream tasks. arXiv preprint arXiv:2403.08540

2024 arXiv
[12]

Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, et al. 2022. Predictability and surprise in large generative models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability,...

2022
[13]

Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. 2020. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701

2020 arXiv
[14]

Hai Hu, Kyle Richardson, Liang Xu, Lu Li, Sandra K \"u bler, and Lawrence S Moss. 2020. Ocnli: Original chinese natural language inference. arXiv preprint arXiv:2010.05444

2020 arXiv
[15]

Shengding Hu, Xin Liu, Xu Han, Xinrong Zhang, Chaoqun He, Weilin Zhao, Yankai Lin, Ning Ding, Zebin Ou, Guoyang Zeng, et al. 2023. Predicting emergent abilities with infinite resolution evaluation. In The Twelfth International Conference on Learning Representations

2023
[16]

Yuzhen Huang, Jinghan Zhang, Zifei Shan, and Junxian He. 2024. Compression represents intelligence linearly. arXiv preprint arXiv:2404.09937

2024 arXiv
[17]

Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212

2023 arXiv
[18]

Yuan Ling, Fanyou Wu, Shujing Dong, Yarong Feng, George Karypis, and Chandan K Reddy. 2023. International workshop on multimodal learning-2023 theme: Multimodal learning with foundation models. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mi...

2023
[19]

Sheng Lu, Irina Bigoulaeva, Rachneet Sachdeva, Harish Tayyar Madabushi, and Iryna Gurevych. 2023. Are emergent abilities in large language models just in-context learning? arXiv preprint arXiv:2309.01809

2023 arXiv
[20]

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843

2016 arXiv
[21]

Naoaki Okazaki, Kakeru Hattori, Hirai Shota, Hiroki Iida, Masanari Ohi, Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Rio Yokota, and Sakae Mizuki. 2024. Building a large japanese web corpus for large language models. arXiv preprint arXiv:2404.17733

2024 arXiv
[22]

Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data only. Advances in Neur...

2023
[23]

Claude Elwood Shannon. 1948. A mathematical theory of communication. The Bell system technical journal, 27(3):379--423

1948
[24]

Noam Shazeer. 2020. Glu variants improve transformer. arXiv preprint arXiv:2002.05202

2020 arXiv
[25]

Yusuxke Shibata, Takuya Kida, Shuichi Fukamachi, Masayuki Takeda, Ayumi Shinohara, Takeshi Shinohara, and Setsuo Arikawa. 1999. Byte pair encoding: A text compression scheme that accelerates pattern matching

1999
[26]

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri \`a Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv p...

2022 arXiv
[27]

Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063

2024
[28]

Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2020. Investigating prior knowledge for challenging chinese machine reading comprehension. Transactions of the Association for Computational Linguistics, 8:141--155

2020
[29]

Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261

2022 arXiv
[30]

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022 a . Emergent abilities of large language models. arXiv preprint arXiv:2206.07682

2022 arXiv
[31]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022 b . Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

2022
[32]

Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, and Ves Stoyanov. 2022. Training trajectories of language models across scales. arXiv preprint arXiv:2212.09803

2022 arXiv
[33]

Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. 2024. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36

2024
[34]

Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. 2022. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pages 1--10

2022
[35]

Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, et al. 2020. Clue: A chinese language understanding evaluation benchmark. arXiv preprint arXiv:2004.05986

2020 arXiv
[36]

Biao Zhang and Rico Sennrich. 2019. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32

2019
[37]

Chujie Zheng, Minlie Huang, and Aixin Sun. 2019. Chid: A large-scale chinese idiom dataset for cloze test. arXiv preprint arXiv:1906.01265

2019 arXiv
[38]

Yukun Zhu. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. arXiv preprint arXiv:1506.06724

2015 arXiv
[39]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[40]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023

[2] [2]

Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, M \'e rouane Debbah, \'E tienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. 2023. The falcon series of open language models. arXiv preprint arXiv:2311.16867

arXiv 2023

[3] [3]

Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al. 2024. Foundational challenges in assuring alignment and safety of large language models. arXiv preprint arXiv:2404.09932

arXiv 2024

[4] [4]

Sanjeev Arora and Anirudh Goyal. 2023. A theory for emergence of complex skills in language models. arXiv preprint arXiv:2307.15936

arXiv 2023

[5] [5]

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

arXiv 2023

[6] [6]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

2020

[7] [7]

Paola A Buitrago and Nicholas A Nystrom. 2019. Open compass: accelerating the adoption of ai in open research. pages 1--9

work page 2019

[8] [8]

Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, et al. 2023. T-eval: Evaluating the tool utilization capability step by step. arXiv preprint arXiv:2312.14033

arXiv 2023