REVIEW 7 cited by
Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Cases
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The success of ChatGPT has recently attracted numerous efforts to replicate it, with instruction-tuning strategies being a key factor in achieving remarkable results. Instruction-tuning not only significantly enhances the model's performance and generalization but also makes the model's generated results more consistent with human speech patterns. However current research rarely studies the impact of different amounts of instruction data on model performance, especially in the real-world use cases. In this paper we explore the performance of large language models based on instruction tuning across different scales of instruction data. An evaluation dataset consisting of 12 major online use cases is constructed in the experiment. With Bloomz-7B1-mt as the base model, the results show that 1) merely increasing the amount of instruction data leads to continuous improvement in tasks such as open-ended generation, 2) in tasks such as math and code, the model performance curve remains quite flat while increasing data size. We further analyze the possible causes of these phenomena and propose potential future research directions such as effectively selecting high-quality training data, scaling base models and training methods specialized for hard tasks. We will release our training and evaluation datasets, as well as model checkpoints.
Forward citations
Cited by 7 Pith papers
-
On the Effect of Instruction Tuning Loss on Generalization
Weighted Instruction Tuning, with low-to-moderate prompt weight and moderate-to-high response weight, beats the standard response-only instruction tuning loss in most of the 75 (model, dataset, benchmark) settings tested.
-
FinTeam: A Multi-Agent Collaborative Intelligence System for Comprehensive Financial Scenarios
A four-agent LLM pipeline trained with role-specific data improves human preference on comprehensive Chinese financial analysis tasks.
-
ClimateChat: Designing Data and Methods for Instruction Tuning LLMs to Answer Climate Change Queries
An automated pipeline creates climate instruction data, and fine-tuning a geoscience LLM on it improves climate question-answering accuracy over general instruction tuning.
-
Steel-LLM:From Scratch to Open Source -- A Personal Journey in Building a Chinese-Centric LLM
Steel-LLM is a fully open 1B-parameter Chinese-centric LLM trained from scratch on ~1T tokens with 8 GPUs, reaching 41.9 C-Eval and 36.1 CMMLU after SFT and DPO.
-
Fine-Tuning Causal LLMs for Text Classification: Embedding-Based vs. Instruction-Based Approaches
Embedding-based QLoRA fine-tuning of causal LLMs matches BERT on single-label patent classification with 10–30x fewer trainable parameters, while instruction-tuning wins on multi-label classification only with ≥100M t...
-
Optimizing Small Transformer-Based Language Models for Multi-Label Sentiment Analysis in Short Texts
On SemEval 2025 Task 11 short texts, generated data helped some BERT models, continued pretraining was mixed, and classification head changes barely mattered.
-
JoyTTS: LLM-based Spoken Chatbot With Voice Cloning
JoyTTS is an end-to-end spoken chatbot with voice cloning, built by feeding LLM hidden states into CosyVoice2 and releasing the training code.
Discussion (0). Continue with ORCID to comment.