Pith. sign in

REVIEW 8 cited by

UL2: Unifying Language Learning Paradigms

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2205.05131 v3 pith:WHDFRWDA submitted 2022-05-10 cs.CL

Yi Tay , Mostafa Dehghani , Vinh Q. Tran , Xavier Garcia , Jason Wei , Xuezhi Wang , Hyung Won Chung , Siamak Shakeri
show 6 more authors
Dara Bahri Tal Schuster Huaixiu Steven Zheng Denny Zhou Neil Houlsby Donald Metzler
This is my paper
classification cs.CL
keywords pre-trainingmodelsobjectivesmodelachieveacrossdifferentdiverse
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Existing pre-trained models are generally geared towards a particular class of problems. To date, there seems to be still no consensus on what the right architecture and pre-training setup should be. This paper presents a unified framework for pre-training models that are universally effective across datasets and setups. We begin by disentangling architectural archetypes with pre-training objectives -- two concepts that are commonly conflated. Next, we present a generalized & unified perspective for self-supervision in NLP and show how different pre-training objectives can be cast as one another and how interpolating between different objectives can be effective. We then propose Mixture-of-Denoisers (MoD), a pre-training objective that combines diverse pre-training paradigms together. We furthermore introduce a notion of mode switching, wherein downstream fine-tuning is associated with specific pre-training schemes. We conduct extensive ablative experiments to compare multiple pre-training objectives and find that our method pushes the Pareto-frontier by outperforming T5 & GPT-like models across multiple diverse setups. By scaling our model up to 20B parameters, we achieve SOTA performance on 50 well-established supervised finetuning based NLP tasks. Our model also achieve strong results at in-context learning, outperforming 175B GPT-3 on zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot summarization. On 0-shot MMLU, UL2 20B outperforms T0 and T5 models. UL2 20B also works well with chain-of-thought prompting and reasoning, making it an appealing choice for research into reasoning at a small to medium scale of 20B parameters. Finally, we apply FLAN instruction tuning to the UL2 20B model, achieving MMLU and Big-Bench scores competitive to FLAN-PaLM 62B. We release Flax-based T5X checkpoints for the UL2 20B & Flan-UL2 20B.

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 98 citations worldwide. Full citation record

  1. S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Modelwith Spatio-Temporal Visual Representation

    cs.CV 2025-05 conditional novelty 7.0 of 10

    S4-Driver uses a multimodal LLM with a sparse 3D spatio-temporal volume representation to achieve self-supervised motion planning that rivals supervised methods on nuScenes and WOMD.

  2. Not All Tokens Matter: Data-Centric Optimization for Efficient Code Summarization

    cs.SE 2026-01 reject novelty 6.0 of 10

    The best token-pruning strategy for code summarization is language-dependent: AST inputs help Java but hurt Python, while function signatures are best for Python.

  3. Populate-A-Scene: Affordance-Aware Human Video Generation

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A fine-tuned text-to-video model inserts a person into a scene and generates an interaction video without bounding boxes or pose input, and its attention maps reveal a latent sense of affordance.

  4. Transition Matching: Scalable and Flexible Generative Modeling

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Transition Matching unifies flow matching and continuous autoregressive generation as discrete-time Markov processes, with three variants that improve text-to-image quality and speed.

  5. Overview of the ClinIQLink 2025 Shared Task on Medical Question-Answering

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A new 4,978-item medical QA benchmark shows modern LLMs handle true/false and multiple choice well but struggle with list questions and multi-hop reasoning.

  6. Sentence Splitter: Uncovering Latent Factual Structure for Self-Supervised Learning

    cs.CL 2026-07 conditional novelty 5.0 of 10

    A T5-based sentence splitter trained on verbalized knowledge-graph triples extracts prefix–tail pairs from raw text, and the resulting structure-aware supervision raises few-shot CommonsenseQA and ATOMIC completion scores.

  7. The Maskability Index: Predicting Task-Objective Alignment in Pretrained Language Models

    cs.CL 2026-07 conditional novelty 4.0 of 10

    A five-example DepthRank gap between masked and prefix templates sorts ATOMIC2020 relations into groups that match which prompt style generates better tails.

  8. A Survey on Generative Model Unlearning: Fundamentals, Taxonomy, Evaluation, and Future Direction

    cs.LG 2025-07 conditional novelty 4.0 of 10

    A survey and framework that categorizes generative model unlearning by point-wise versus concept-wise objectives, parameter-based versus non-parametric methods, and completeness/utility/efficiency evaluation.

Pith tools