Small Language Model as Data Prospector for Large Language Model

Di Yang; Haihong Wu; Hamid Alinejad-Rokny; Min Yang; Qiang Qu; Shiwen Ni

REVIEW 3 major objections 6 minor 19 references

Small Language Model as Data Prospector for Large Language Model

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A small language model can rank instruction data as well as a 7B model while using 58 times less compute.

desk verdict Plausible efficiency-oriented variant of NUGGETS with a useful task-set refinement ablation, but the headline 1-2% parity and 58x efficiency claims are unsupported by the experiments as presented. read the letter →

arxiv 2412.09990 v1 pith:XBUSGJVZ submitted 2024-12-13 cs.CL cs.AI

Shiwen Ni , Haihong Wu , Di Yang , Qiang Qu , Hamid Alinejad-Rokny , Min Yang This is my paper · ORCID

classification cs.CLcs.AI

keywords instructiondataselectionone-shotlearningsmalllanguagemodelprospectinggoldenscorefine-tuningperplexityefficiency

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Instruction-tuning quality depends less on data volume than on which examples you keep. The paper proposes SuperNUGGETS, a data-filtering method that scores each candidate instruction by how much it improves a small model's one-shot perplexity on a curated set of 100 test tasks. It claims this lets a 350M-parameter model (and even a 125M-parameter model) pick training data almost as well as the 7B-parameter scorer used by the original NUGGETS, losing only 1-2% in final win rate while cutting compute by a factor of 58. If true, high-quality data selection for LLM fine-tuning becomes a cheap preprocessing step rather than a large-model job. The paper also reports that fine-tuning on just the top 5% of Alpaca selected this way beats fine-tuning on all 52,002 examples.

What carries the argument

The Golden Score (Eq. 5) is the ranking object: for a candidate instruction $z_k$, $\mathrm{GS}(z_k)=\frac{1}{m}\sum_{i=1}^{m}\mathbb{I}[s_{\mathrm{one}}^i(z_k)>s_{\mathrm{zero}}^i]$, the fraction of predefined test tasks whose one-shot answer perplexity improves over zero-shot when $z_k$ is prepended. The paper's second load-bearing mechanism is the refined test set: 100 tasks assembled by taking the top 20 reward-model-scored instructions and 80 diverse examples from a greedy k-center clustering, which replaces 1,000 randomly sampled tasks and makes the scoring pass ten times cheaper.

What would settle it

Take the same candidate pool, score it with Opt-125m and with Llama2-7B, then fine-tune identical copies of a fixed base model on the 5% each scorer picks and on a random 5% baseline. If the Opt-125m subset does not beat the random subset on the held-out benchmark, or if the two scorers' rankings have no better-than-chance overlap, the claim that small-model Golden Scores transfer to large-model fine-tuning would be refuted.

Watch

Extended reading notes

Core claim

The central claim is that the information needed to prospect good instruction data lives in the one-shot behavior of a much smaller model, not in the size of the scorer. Given a pool of instruction examples and a small set of predefined tasks, SuperNUGGETS computes, for every candidate example, the fraction of tasks whose answer perplexity improves when that example is prepended as a one-shot prompt (the Golden Score). It then keeps the top n% of examples by that score. The paper shows this selection transfers: fine-tuning Llama2-7B on the top 5% scored by Opt-350m reaches a win rate of 23.98 versus 18.51 for the full dataset and 24.47 for the top 5% scored by Llama2-7B itself, and the overlap between the 7B and 125M rankings is 65% at the top 30% cutoff. The method's efficiency comes from replacing the 1,000 random test tasks with a 100-task set built by reward-model scoring and diversity clustering, which cuts the number of scoring passes by another factor of ten.

Load-bearing premise

The entire method assumes that a small model's one-shot perplexity improvement on 100 test tasks ranks instruction examples by their downstream fine-tuning value for a much larger model.

Editorial extensions

If this is right

Fine-tuning on SuperNUGGETS's top 5% of Alpaca (about 2,600 examples) outperforms fine-tuning on the full 52,002-example dataset across all three prospector sizes.
A 350M-parameter prospector (20 times smaller than Llama2-7B) and a 125M-parameter prospector (56 times smaller) both recover most of the data-selection signal, with final win rates within 1-2% of the 7B prospector.
Reducing the predefined task set from 1,000 random tasks to 100 refined tasks cuts scoring computations tenfold while matching the selection quality of the larger random set.
The rankings are stable across model sizes: the top 30% selected by Opt-125m and Opt-350m overlap with Llama2-7B's selection at 65% and 70%, respectively, so a small model can stand in for a large one.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the scoring pass is cheap, a natural extension the paper does not run is to scale the same filter to millions of candidate instructions, where an LLM scorer would be prohibitively expensive; nothing in the method prevents that scale.
The paper fine-tunes only Llama2-7B, so it does not establish that the same top-5% subset is optimal for larger or different base models; the transfer claim would be stronger if tested on a 13B or 70B fine-tune.
The success of a 100-task refined set suggests the predefined tasks can be optimized rather than sampled, and one could search over task sets to maximize agreement with an LLM's ranking, a direction the paper does not explore.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about this paper. The core idea is reasonable: swap the 7B prospector for a 125M/350M one and shrink the predefined task set from 1000 random examples to 100 refined ones. The second is that the headline numbers in the abstract—1-2% performance drop and 58x efficiency gain—are not established by the experiments that follow.

What is genuinely new is the two-step refinement of the task set: reward-model scoring to keep high-quality examples, then k-center greedy clustering for diversity. The ablation in Table 2 is the paper's real contribution. For every prospector, the refined-100 task set beats a random-100 set and is comparable to or better than the random-1000 set used in the original NUGGETS. That is a clean, useful result. The agreement between SLM and LLM selections (Table 3) also supports the intuition that small models can serve as proxies here. The evaluation is independent (Alpaca-Eval), so there is no circularity problem.

Where it falls short: the paper never runs the original NUGGETS under identical conditions. The closest row, Llama2-7B with 1000 random tasks, appears in Table 2 but is not labeled as NUGGETS and is not used to substantiate the 1-2% claim. Looking at the numbers yourself, the claim does not hold up. At top 5%, Opt-350m/refined-100 scores 23.98 vs Llama2-7B/random-1000's 21.49—SuperNUGGETS is better. At top 10% it is 21.99 vs 23.79, about a 1.8-point drop, and at top 1% it is 15.65 vs 18.63, a 3-point drop. That is not a uniform 1-2% decrease; it ranges from better to roughly 16% relative worse. The 58x efficiency figure is also unexplained. The paper's own arithmetic gives ~10x fewer inference calls (1000 vs 100 tasks) and 20-56x smaller models, which naively multiplies to 200-560x, not 58x. No wall-clock time, GPU hours, or FLOPs are reported. The number just appears in the abstract and conclusion.

There are also small errors: the text says 104 million inference runs when it is actually about 52 million, and the ablation section says "the validity letter of our refinement." No error bars or multiple seeds are reported anywhere, which matters for fine-tuning comparisons.

Who should read it: anyone working on instruction-data selection. The task-set refinement result is worth citing even if the efficiency claim is ignored. But the central comparison to NUGGETS needs to be redone with a proper baseline and a measured runtime or FLOP count before the 1-2% / 58x claims can be taken seriously. I would send it to peer review—the idea is solid enough and the ablation is informative—but with a strong request for major revision. The authors need to run NUGGETS as a real baseline, report actual efficiency metrics, and restate the performance comparison based on the table rather than the abstract.

Referee Report

3 major / 6 minor

Summary. The paper proposes SuperNUGGETS, a variant of the NUGGETS instruction-data prospecting method. SuperNUGGETS replaces the 7B-parameter data prospector with Opt-125m or Opt-350m small language models and replaces the original 1000-example random predefined task set with a refined 100-example set constructed via reward-model scoring and diversity-preserving clustering. The authors evaluate the filtered data by fine-tuning Llama2-7B on Alpaca subsets and measuring Alpaca-Eval win rates, reporting that top-5% selected data outperforms fine-tuning on the full 52k dataset, and claiming in the abstract and conclusion that SuperNUGGETS is only 1-2% less performant than NUGGETS while being 58 times more efficient.

Significance. If the central claims were fully supported, this would be a practically useful result: using a small model as a data prospector for a large model would substantially lower the cost of instruction-data selection while preserving quality. The observation that top-5% subsets selected by Opt-125m and Opt-350m outperform full-data fine-tuning is interesting and worth investigating. However, the headline parity and efficiency claims are currently not measured against the original NUGGETS pipeline, and the reported numbers do not consistently support the 1-2% figure. The paper does provide useful ablations showing that the refined 100-example task set outperforms a randomly sampled 100-example set and roughly matches a random 1000-example set.

major comments (3)

[Abstract and Section 5] The claim that SuperNUGGETS is only 1-2% less performant than NUGGETS is not supported by the reported numbers. Table 2 contains no row explicitly labeled "NUGGETS"; if the Llama2-7B / 1000-random row is intended as the original NUGGETS baseline, then the results vary widely by selection ratio. At top 5%, the Opt-350m and Opt-125m SuperNUGGETS rows are better than the baseline (23.98 and 22.11 vs. 21.49), while at top 10% the drops are about 7.6% and 10.2% (21.99 and 21.37 vs. 23.79), and at top 1% the drops are about 16% and 13% (15.65 and 16.15 vs. 18.63). A 1-2% decrease is therefore not a consistent reading of the table, and the paper should state explicitly which comparison supports the abstract's claim.
[Section 2 and Section 5] The 58x efficiency factor is asserted without measurement or a clear derivation. The paper's own numbers in Section 2 imply about 52,054,002 inference passes for the original method and about 5,252,202 for the 100-example refined set, a reduction of roughly 10x in inference count; combining this with the parameter ratios of 20x (Opt-350m vs. Llama2-7B) or 56x (Opt-125m vs. Llama2-7B) would suggest efficiency gains of roughly 200x or 560x, not 58x. No wall-clock, GPU-hour, or FLOP measurement appears anywhere in the paper. The efficiency claim needs either actual runtime data or a clearly stated calculation that explains the factor of 58.
[Section 2.2, Eq. (5)] The golden score in Eq. (5) is the load-bearing mechanism of the method: it assumes that the fraction of predefined tasks for which one-shot perplexity improves over zero-shot perplexity ranks instruction examples by downstream fine-tuning value, and that this ranking transfers from the 7B prospector to the 125m/350m prospectors. The paper currently validates this only indirectly: Table 3 shows overlap between the top-30% sets selected by different prospectors, and Table 1 shows that top-5% subsets beat full-data fine-tuning. A concrete test of the transfer assumption would be to report fine-tuning win rates for subsets selected by each prospector at the same ratios (which Table 2 partially does) and, more directly, to compute the rank correlation between golden scores and downstream fine-tuning gains for a sample of examples. Without such a test, the parity claim for the small-model prospector remains an assumption rather than a demonstrated result.

minor comments (6)

[Abstract] The phrase "high-quality quality data" in the abstract contains a duplicated word; it should read "high-quality data."
[Section 2, Motivation] The text states that filtering Alpaca requires "inference a total of 52,002 (zero-shot) + [52,002 × 1,000] (one-shot) = 52,054,002 times" and then says "104 million times." The sum is about 52 million, not 104 million; this arithmetic inconsistency should be corrected.
[Section 2.1] The description "encodes the first 20-10,000 data" and later "selects 80 examples from 20-1,000 data" is unclear: it should specify the exact index ranges (for example, ranks 20 through 10,000, and ranks 20 through 1,000) in a consistent notation.
[Section 3.1] The sentence "we use an Adam optimiser with a learning rate of 2 × 10−5, a learning rate of 2e-5" repeats the learning rate value; this should be presented once.
[Section 3.1] The text says "davincici -003 model," which contains a typo; it should be "text-davinci-003."
[Section 4] The phrase "The above experimental results illustrate the validity letter of our refinement" appears to contain an extra word; it should likely read "the validity of our refinement."

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: the golden-score filter is tested against external Alpaca-Eval, not fitted to it; minor self-citation overlap with NUGGETS is not load-bearing.

full rationale

SuperNUGGETS inherits the NUGGETS one-shot scoring mechanism (Eqs. 1-5), but the paper's own experiments are self-contained: the golden score ranks candidate instructions by perplexity improvement on a refined predefined task set, and the method's success is measured by Alpaca-Eval win_rate after fine-tuning Llama2-7B. The target metric is not used to fit any parameter in Eqs. 1-5, so the central selection result does not reduce to its inputs. The refined predefined set is constructed by reward-model scoring and kcenter-greedy diversity, not by optimizing final performance, so that step is also not circular. The abstract's '1-2% decrease' and '58x efficiency' claims are not backed by an actual NUGGETS baseline or runtime measurement in the paper (Tables 1-3 compare filtered vs full data and refined vs random predefined sets, with no NUGGETS row and no wall-clock/FLOP data); however, an unsupported comparative claim is a correctness/evidence problem, not a circularity. The only self-citation concern is that Min Yang is a coauthor of both NUGGETS (Li et al., 2023c) and this paper, and the method is explicitly a variant of that prior work; but the prior work is cited as a published method and the current evaluation is independent (Alpaca-Eval via gpt-4o-mini), so the self-citation is not load-bearing in a circular way. The limitations paragraph honestly notes that larger-scale fine-tuning was not tested. Overall circularity score is low.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new entities. It relies on several domain assumptions inherited from NUGGETS plus its own refinement choices. No parameters are fitted to the Alpaca-Eval target, but multiple hand-chosen cutoffs affect the selection pipeline and are not optimized or sensitivity-tested.

free parameters (5)

Refined test set size = 100
Hand-chosen in Section 2.1. The ablation shows 100 refined tasks roughly match 1000 random tasks, but the size is not optimized.
High-quality subset size = 20
Hand-chosen in Section 2.1 to seed the refined set from the top reward-model-scored examples.
Cluster-selected subset size = 80
Hand-chosen in Section 2.1 to complement the 20 seed examples after k-center greedy clustering.
Reward-model cutoff = top 10,000 of 52,002
Hand-chosen in Section 2.1 to reduce the pool before clustering. No sensitivity analysis is provided.
Selected data ratio highlighted = 5%
Top 5% is the best-performing ratio in Table 1 and is highlighted as the main result, but other ratios (1%, 10%, 30%, 50%) were also evaluated.

assumptions (5)

domain assumption One-shot in-context learning approximates implicit fine-tuning
Invoked in the Introduction via Dai et al. 2022. The entire golden-score computation is built on this proxy.
domain assumption Answer perplexity improvement on the predefined task set predicts which instruction examples improve fine-tuning
Central premise of Eqs. 1-5. No independent validation of the proxy itself is given.
domain assumption Reward-model score (deberta-v3-large-v2) is a valid quality signal for instruction data
Used in Section 2.1 to keep the top 10,000 examples. The paper does not analyze reward-model errors or calibration.
domain assumption K-center greedy clustering of semantic embeddings ensures diversity and coverage of the selected task set
Section 2.1 assumes dispersed centroids cover instruction types. No quantitative coverage check is provided.
domain assumption Opt-125m and Opt-350m are valid stand-ins for Llama2-7B for scoring one-shot perplexity
The core efficiency claim rests on golden-score rankings from small models matching those from Llama2-7B. Only overlap statistics in Table 3 support this.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Small Language Model as Data Prospector for Large Language Model." pith.science (2026). https://pith.science/paper/XBUSGJVZ

@misc{pith2026241209990,
  author       = {Pith},
  title        = {Pith review of: Small Language Model as Data Prospector for Large Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XBUSGJVZ}},
  note         = {Machine review of arXiv:2412.09990}
}

read the original abstract

The quality of instruction data directly affects the performance of fine-tuned Large Language Models (LLMs). Previously, \cite{li2023one} proposed \texttt{NUGGETS}, which identifies and selects high-quality quality data from a large dataset by identifying those individual instruction examples that can significantly improve the performance of different tasks after being learnt as one-shot instances. In this work, we propose \texttt{SuperNUGGETS}, an improved variant of \texttt{NUGGETS} optimised for efficiency and performance. Our \texttt{SuperNUGGETS} uses a small language model (SLM) instead of a large language model (LLM) to filter the data for outstanding one-shot instances and refines the predefined set of tests. The experimental results show that the performance of \texttt{SuperNUGGETS} only decreases by 1-2% compared to \texttt{NUGGETS}, but the efficiency can be increased by a factor of 58. Compared to the original \texttt{NUGGETS}, our \texttt{SuperNUGGETS} has a higher utility value due to the significantly lower resource consumption.

Figures

Figures reproduced from arXiv: 2412.09990 by the authors.

**Figure 1.** Comparison of Nuggets and SuperNuggets on the Alpaca-Eval benchmark. ing instruction fine-tuning by collecting larger, diverse, and complex datasets, often derived from open source data or expanded based on large language models. However, some recent studies (Bai et al., 2024; Zhou et al., 2023; Cao et al., 2023) have shown that smaller but carefully selected highquality datasets in the instruction fine-tuning ph… view at source ↗

**Figure 2.** Comparison of SuperNUGGETS and NUGGETS. sequentially selected as a one-shot example for contextual learning and scored by observing its impact on the perplexity of the predefined examples. This score reflects the correlation between the predefined examples and the candidate examples and serves as a criterion for data selection. Since the NUGGETS method needs to calculate the one-shot score and zero-shot score for … view at source ↗

**Figure 3.** Examples of instructions and their corresponding golden scores. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 2 canonical work pages

[1]

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

arXiv 2023
[2]

Yuelin Bai, Xinrun Du, Yiming Liang, Yonggang Jin, Ziqiang Liu, Junting Zhou, Tianyu Zheng, Xincheng Zhang, Nuo Ma, Zekun Wang, et al. 2024. Coig-cqia: Quality is all you need for chinese instruction fine-tuning. arXiv preprint arXiv:2403.18058

arXiv 2024
[3]

Yihan Cao, Yanbin Kang, and Lichao Sun. 2023. Instruction mining: High-quality instruction data selection for large language models. arXiv preprint arXiv:2307.06290

arXiv 2023
[4]

Xuxin Cheng, Zhihong Zhu, Hongxiang Li, Yaowei Li, Xianwei Zhuang, and Yuexian Zou. 2024. Towards multi-intent spoken language understanding via hierarchical attention and optimal transport. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17844--17852

work page 2024
[5]

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416

arXiv 2022
[6]

Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. 2022. Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers. arXiv preprint arXiv:2212.10559

arXiv 2022
[7]

Google. 2023. Palm 2 technical report. arXiv preprint arXiv:2305.10403

arXiv 2023
[8]

Yunjie Ji, Yong Deng, Yan Gong, Yiping Peng, Qiang Niu, Lei Zhang, Baochang Ma, and Xiangang Li. 2023. Exploring the impact of instruction data scaling on large language models: An empirical study on real-world use cases. arXiv preprint arXiv:2303.14742

arXiv 2023

Show all 19 references

[9]

Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, et al. 2023 a . M3it: A large-scale dataset towards multi-modal multilingual instruction tuning. arXiv preprint arXiv:2306.04387

2023 arXiv
[10]

Hashimoto

Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023 b . Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval

2023
[11]

Yunshui Li, Binyuan Hui, Xiaobo Xia, Jiaxi Yang, Min Yang, Lei Zhang, Shuzheng Si, Junhao Liu, Tongliang Liu, Fei Huang, et al. 2023 c . One shot learning as instruction data prospector for large language models. arXiv preprint arXiv:2312.10302

2023 arXiv
[12]

OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

2023 arXiv
[13]

Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. 2023. How far can camels go? exploring the state of instruction tuning on open resources. arXiv preprint arXiv:2306.04751

2023 arXiv
[14]

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022 a . Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560

2022 arXiv
[15]

Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, et al. 2022 b . Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. ...

2022
[16]

Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. 2023. Baize: An open-source chat model with parameter-efficient tuning on self-chat data. arXiv preprint arXiv:2304.01196

2023 arXiv
[17]

Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206

2023 arXiv
[18]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[19]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

arXiv 2023

[2] [2]

Yuelin Bai, Xinrun Du, Yiming Liang, Yonggang Jin, Ziqiang Liu, Junting Zhou, Tianyu Zheng, Xincheng Zhang, Nuo Ma, Zekun Wang, et al. 2024. Coig-cqia: Quality is all you need for chinese instruction fine-tuning. arXiv preprint arXiv:2403.18058

arXiv 2024

[3] [3]

Yihan Cao, Yanbin Kang, and Lichao Sun. 2023. Instruction mining: High-quality instruction data selection for large language models. arXiv preprint arXiv:2307.06290

arXiv 2023

[4] [4]

Xuxin Cheng, Zhihong Zhu, Hongxiang Li, Yaowei Li, Xianwei Zhuang, and Yuexian Zou. 2024. Towards multi-intent spoken language understanding via hierarchical attention and optimal transport. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17844--17852

work page 2024

[5] [5]

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416

arXiv 2022

[6] [6]

Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. 2022. Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers. arXiv preprint arXiv:2212.10559

arXiv 2022

[7] [7]

Google. 2023. Palm 2 technical report. arXiv preprint arXiv:2305.10403

arXiv 2023

[8] [8]

Yunjie Ji, Yong Deng, Yan Gong, Yiping Peng, Qiang Niu, Lei Zhang, Baochang Ma, and Xiangang Li. 2023. Exploring the impact of instruction data scaling on large language models: An empirical study on real-world use cases. arXiv preprint arXiv:2303.14742

arXiv 2023