REVIEW 19 cited by
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
How well can NLP models generalize to a variety of unseen tasks when provided with task instructions? To address this question, we first introduce Super-NaturalInstructions, a benchmark of 1,616 diverse NLP tasks and their expert-written instructions. Our collection covers 76 distinct task types, including but not limited to classification, extraction, infilling, sequence tagging, text rewriting, and text composition. This large and diverse collection of tasks enables rigorous benchmarking of cross-task generalization under instructions -- training models to follow instructions on a subset of tasks and evaluating them on the remaining unseen ones. Furthermore, we build Tk-Instruct, a transformer model trained to follow a variety of in-context instructions (plain language task definitions or k-shot examples). Our experiments show that Tk-Instruct outperforms existing instruction-following models such as InstructGPT by over 9% on our benchmark despite being an order of magnitude smaller. We further analyze generalization as a function of various scaling parameters, such as the number of observed tasks, the number of instances per task, and model sizes. We hope our dataset and model facilitate future progress towards more general-purpose NLP models.
Forward citations
Cited by 19 Pith papers
-
Synthetic Visual Genome
A GPT-4V/GPT-4o pipeline for completing and refining scene graph annotations yields a dense synthetic dataset that, after instruction tuning, gives a 3B model strong relationship understanding and grounding results.
-
How Many Instructions Can LLMs Follow at Once?
IFScale measures instruction-following at densities from 10 to 500 constraints and finds that even top frontier models satisfy only about two-thirds of 500 simultaneous keyword instructions.
-
Interactive Reasoning: Visualizing and Controlling Chain-of-Thought Reasoning in Large Language Models
Interactive Reasoning, instantiated as Hippo, lets users view and edit an LLM's chain-of-thought as a tree, and a 16-person study reports improved perceived control, sense-making, and assumption awareness.
-
A Modular Taxonomy for Hate Speech Definitions and Its Impact on Zero-Shot LLM Classification Performance
A taxonomy of 14 hate speech definition components plus evidence that zero-shot LLM hate speech classification is sensitive to which definition is inserted into the prompt, with model-dependent effects.
-
Aligning Large Language Models with Implicit Preferences from User-Generated Content
PUGC turns unlabeled user-generated content into preference pairs by scoring policy responses against the source post as a reference, reaching 35.93% length-controlled win rate on AlpacaEval 2.0 with Mistral-7B-Instruct.
-
TailorSQL: An NL2SQL System Tailored to Your Query Workload
Adding workload-derived hints and tailored retrieval from past SQL queries improves NL2SQL execution accuracy and reduces prompt size on three benchmarks.
-
LAARA: Layer-Aware Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning
LAARA allocates LoRA ranks per layer from diagonal Fisher (gradient-based) estimates, reporting improved accuracy with fewer trainable parameters on GLUE and MathInstruct.
-
Agents of Diffusion: Enhancing Diffusion Language Models with Multi-Agent Reinforcement Learning for Structured Data Generation (Extended Version)
AoD pairs a frozen diffusion language model with two LLM agents that iteratively rewrite prompts from natural-language feedback, reporting better JSON diversity and validity, though the claimed RL mechanism and theore...
-
DICE: Dynamic In-Context Example Selection in LLM Agents via Efficient Knowledge Transfer
DICE dynamically retrieves the most relevant in-context demonstrations at each agent step, and in this preprint it raises exact-match and success-rate scores on HotpotQA, ALFWorld, and Webshop across ReAct, Reflexion,...
-
Instruction Learning Paradigms: A Dual Perspective on White-box and Black-box LLMs
A hybrid black-box and white-box instruction optimizer, built on InstructZero and INSTINCT, reports the highest mean score on 30 tasks but with small margins, missing error bars, and unreleased code.
-
Pruning General Large Language Models into Customized Expert Models
Cus-Prun identifies and removes neurons that are irrelevant to a user's target language, domain, and task, producing specialized expert models without post-training.
-
MINT: Multimodal Instruction Tuning with Multimodal Interaction Grouping
Grouping instruction-tuning datasets by redundancy, uniqueness, or synergy of text-image interaction improves vision-language model accuracy over single-task and unselective multi-task tuning.
-
RetrieveAll: A Multilingual Named Entity Recognition Framework with Large Language Models
RetrieveAll combines per-language LoRA adapters with retrieval of entity and context examples to improve multilingual NER, claiming an average 12.1% F1 gain on PAN-X.
-
Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct Technical Report
Foundation-Sec-8B-Instruct, an instruction-tuned 8B cybersecurity LLM, is released and claimed to beat Llama 3.1-8B-Instruct on CTIBench-RCM and CTIBench-MCQA while remaining competitive on general instruction-following.
-
Improving Task Diversity in Label Efficient Supervised Finetuning of LLMs
Weighted Task Diversity allocates the annotation budget across tasks in inverse proportion to the base model's average confidence, improving MMLU and AlpacaEval scores with up to 80% fewer labels.
-
Continual Learning for Generative AI: From LLMs to MLLMs and Beyond
A survey that categorizes continual learning methods for generative models into architecture-based, regularization-based, and replay-based paradigms across four model families.
-
Being Strong Progressively! Enhancing Knowledge Distillation of Large Language Models through a Curriculum Learning Framework
POCL wraps LLM knowledge distillation in a curriculum that increases data difficulty and temperature over stages, improving Rouge-L on small GPT-2 and OPT students, though ablations show temperature drives the gains.
-
Leveraging Large Language Models for Bengali Math Word Problem Solving with Chain of Thought Reasoning
A new Bengali math word problem dataset translated from GSM8K is benchmarked with chain-of-thought prompting, yielding 88% accuracy with LLaMA-3.3 70B on a 1,000-sample test subset.
-
PARAM-1 BharatGen 2.9B Model
A technical report on a 2.9B English-Hindi model whose headline evaluation numbers are internally inconsistent and whose promoted tokenizer was not used to train the final model.
Discussion (0). Sign in to comment.