pith. sign in

arxiv: 2111.10952 · v2 · pith:QHJ3P2XQnew · submitted 2021-11-22 · 💻 cs.CL · cs.LG

ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning

Vamsi Aribandi , Yi Tay , Tal Schuster , Jinfeng Rao , Huaixiu Steven Zheng , Sanket Vaibhav Mehta , Honglei Zhuang , Vinh Q. Tran
show 6 more authors
Dara Bahri Jianmo Ni Jai Gupta Kai Hui Sebastian Ruder Donald Metzler
This is my paper
classification 💻 cs.CL cs.LG
keywords multi-tasktasksexmixext5pre-traininglearningscalingtransfer
0
0 comments X
read the original abstract

Despite the recent success of multi-task learning and transfer learning for natural language processing (NLP), few works have systematically studied the effect of scaling up the number of tasks during pre-training. Towards this goal, this paper introduces ExMix (Extreme Mixture): a massive collection of 107 supervised NLP tasks across diverse domains and task-families. Using ExMix, we study the effect of multi-task pre-training at the largest scale to date, and analyze co-training transfer amongst common families of tasks. Through this analysis, we show that manually curating an ideal set of tasks for multi-task pre-training is not straightforward, and that multi-task scaling can vastly improve models on its own. Finally, we propose ExT5: a model pre-trained using a multi-task objective of self-supervised span denoising and supervised ExMix. Via extensive experiments, we show that ExT5 outperforms strong T5 baselines on SuperGLUE, GEM, Rainbow, Closed-Book QA tasks, and several tasks outside of ExMix. ExT5 also significantly improves sample efficiency while pre-training.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning

    cs.LG 2025-05 unverdicted novelty 6.0

    Entropy minimization on self-generated outputs elicits strong reasoning in pretrained LLMs, matching or exceeding supervised RL methods on benchmarks.

  2. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators

    cs.LG 2024-04 conditional novelty 6.0

    Length-controlled AlpacaEval applies regression adjustment to remove length bias from LLM auto-evaluations, raising Spearman correlation with Chatbot Arena from 0.94 to 0.98.

  3. The Flan Collection: Designing Data and Methods for Effective Instruction Tuning

    cs.AI 2023-01 conditional novelty 6.0

    The Flan Collection demonstrates that task balancing, data enrichment, and mixed prompt training are critical to effective instruction tuning, yielding stronger Flan-T5 models released publicly.

  4. A General Language Assistant as a Laboratory for Alignment

    cs.CL 2021-12 conditional novelty 6.0

    Ranked preference modeling outperforms imitation learning for language model alignment and scales more favorably with model size.

  5. PaLM 2 Technical Report

    cs.CL 2023-05 unverdicted novelty 5.0

    PaLM 2 reports state-of-the-art results on language, reasoning, and multilingual tasks with improved efficiency over PaLM.

  6. Galactica: A Large Language Model for Science

    cs.CL 2022-11 unverdicted novelty 5.0

    Galactica, a science-specialized LLM, reports higher scores than GPT-3, Chinchilla, and PaLM on LaTeX knowledge, mathematical reasoning, and medical QA benchmarks while outperforming general models on BIG-bench.