{"work":{"id":"807e5cef-0bb3-4eae-a684-154fc2d6aa06","openalex_id":null,"doi":null,"arxiv_id":"2104.08773","raw_key":null,"title":"Cross-Task Generalization via Natural Language Crowdsourcing Instructions","authors":null,"authors_text":"Swaroop Mishra, Daniel Khashabi, Chitta Baral, Hannaneh Hajishirzi","year":2021,"venue":"cs.CL","abstract":"Humans (e.g., crowdworkers) have a remarkable ability in solving different tasks, by simply reading textual instructions that define them and looking at a few examples. Despite the success of the conventional supervised learning on individual datasets, such models often struggle with generalization across tasks (e.g., a question-answering system cannot solve classification tasks). A long-standing challenge in AI is to build a model that learns a new task by understanding the human-readable instructions that define it. To study this, we introduce NATURAL INSTRUCTIONS, a dataset of 61 distinct tasks, their human-authored instructions, and 193k task instances (input-output pairs). The instructions are obtained from crowdsourcing instructions used to create existing NLP datasets and mapped to a unified schema. Using this meta-dataset, we measure cross-task generalization by training models on seen tasks and measuring generalization to the remaining unseen ones. We adopt generative pre-trained language models to encode task-specific instructions along with input and generate task output. Our results indicate that models benefit from instructions when evaluated in terms of generalization to unseen tasks (19% better for models utilizing instructions). These models, however, are far behind an estimated performance upperbound indicating significant room for more progress in this direction.","external_url":"https://arxiv.org/abs/2104.08773","cited_by_count":null,"metadata_source":"pith","metadata_fetched_at":"2026-05-24T09:14:16.516419+00:00","pith_arxiv_id":"2104.08773","created_at":"2026-05-10T17:34:42.780495+00:00","updated_at":"2026-05-24T09:14:16.516419+00:00","title_quality_ok":true,"display_title":"Cross-Task Generalization via Natural Language Crowdsourcing Instructions","render_title":"Cross-Task Generalization via Natural Language Crowdsourcing Instructions"},"hub":{"state":{"work_id":"807e5cef-0bb3-4eae-a684-154fc2d6aa06","tier":"hub","tier_reason":"10+ Pith inbound or 1,000+ external citations","pith_inbound_count":19,"external_cited_by_count":null,"distinct_field_count":6,"first_pith_cited_at":"2021-10-15T17:08:57+00:00","last_pith_cited_at":"2025-12-11T16:55:30+00:00","author_build_status":"not_needed","summary_status":"needed","contexts_status":"needed","graph_status":"needed","ask_index_status":"not_needed","reader_status":"not_needed","recognition_status":"not_needed","updated_at":"2026-05-27T16:17:47.981453+00:00","tier_text":"hub"},"tier":"hub","role_counts":[{"context_role":"background","n":3}],"polarity_counts":[{"context_polarity":"background","n":3}],"runs":{},"summary":{},"graph":{},"authors":[]}}