REVIEW 5 cited by
AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
In this work, we demonstrate that multilingual large-scale sequence-to-sequence (seq2seq) models, pre-trained on a mixture of denoising and Causal Language Modeling (CLM) tasks, are more efficient few-shot learners than decoder-only models on various tasks. In particular, we train a 20 billion parameter multilingual seq2seq model called Alexa Teacher Model (AlexaTM 20B) and show that it achieves state-of-the-art (SOTA) performance on 1-shot summarization tasks, outperforming a much larger 540B PaLM decoder model. AlexaTM 20B also achieves SOTA in 1-shot machine translation, especially for low-resource languages, across almost all language pairs supported by the model (Arabic, English, French, German, Hindi, Italian, Japanese, Marathi, Portuguese, Spanish, Tamil, and Telugu) on Flores-101 dataset. We also show in zero-shot setting, AlexaTM 20B outperforms GPT3 (175B) on SuperGLUE and SQuADv2 datasets and provides SOTA performance on multilingual tasks such as XNLI, XCOPA, Paws-X, and XWinograd. Overall, our results present a compelling case for seq2seq models as a powerful alternative to decoder-only models for Large-scale Language Model (LLM) training.
Forward citations
Cited by 5 Pith papers
-
Not All Tokens Matter: Data-Centric Optimization for Efficient Code Summarization
The best token-pruning strategy for code summarization is language-dependent: AST inputs help Java but hurt Python, while function signatures are best for Python.
-
Optimizing Datasets for Code Summarization: Is Code-Comment Coherence Enough?
Filtering code summarization training data by code-comment coherence gives no better results than random selection, and halving the training set does not hurt performance.
-
Safe and Certifiable AI Systems: Concepts, Challenges, and Lessons Learned
The paper presents the TÜV AUSTRIA Trusted AI audit catalog, a statistical framework based on the Stochastic Application Domain Definition, minimum performance requirements, and independent-sample testing for certifyi...
-
Towards Reliable Generative AI-Driven Scaffolding: Reducing Hallucinations and Enhancing Quality in Self-Regulated Learning Support
A multi-agent reliability check and an LLM-as-a-judge quality check both reduce hallucinations in AI-generated study scaffolds, with the multi-agent check matching human expert judgments almost perfectly.
-
Survey of NLU Benchmarks Diagnosing Linguistic Phenomena: Why not Standardize Diagnostics Benchmarks?
A survey of NLU diagnostics benchmarks finds no shared naming convention or standard set of linguistic phenomena, and asks whether the field should build an ISO-like evaluation standard.
Discussion (0). Continue with ORCID to comment.