A Unifying Scheme for Extractive Content Selection Tasks

Aviv Slobodkin; Ido Dagan; Ori Shapira; Shmuel Amar

REVIEW 1 major objections 4 minor 31 references

A Unifying Scheme for Extractive Content Selection Tasks

T0 review · 1 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a unified instruction-guided formulation of content selection lets a small fine-tuned model outperform much larger prompted models on six diverse tasks, with overall token-level F1 of 45.7 versus a 41.9 baseline in…

desk verdict A useful benchmark-plus-synthetic-data paper for content selection, with transfer gains that are consistent but need to be checked against possible document overlap between GENCS and the test sets. read the letter →

arxiv 2507.16922 v1 pith:INA3SWPN submitted 2025-07-22 cs.CL

Shmuel Amar , Ori Shapira , Aviv Slobodkin , Ido Dagan This is my paper · ORCID

classification cs.CL

keywords instruction-guidedcontentselectionextractivesynthetictrainingdatatransferlearningLLMfine-tuningtoken-levelF1IGCS-BENCHGENCS

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's aim is to show that many NLP tasks that select relevant text spans from source documents are one task, and can be modeled that way. It proposes instruction-guided content selection (IGCS), where the task definition and instance-specific query are given to a language model as an instruction, and introduces IGCS-BENCH, a unified benchmark of six existing human-annotated selection tasks. It also builds GENCS, a synthetic dataset of 12,490 LLM-generated instruction–selection pairs over seven corpora, and reports that fine-tuning small models on GENCS improves token-level F1 on the benchmark compared with zero- or few-shot prompting, with overall F1 of 45.7 versus a 41.9 baseline when no task-specific training data is used. The value, if the claim holds, is a cheap and reusable recipe for new content selection tasks, plus a generic evaluation metric.

What carries the argument

The carrying machinery is the IGCS instruction template, a natural-language instruction with optional query slots, the source document set, and a JSON array of verbatim spans as output, together with GENCS, the synthetic dataset of 12,490 (documents, instruction, selection) instances produced in three stages: GPT-4 writes five diverse instructions per document set; GPT-4, Claude3-Opus, and Gemini-1.5-Pro each propose selections; and union or majority merging of the three proposals yields two recall- and precision-oriented datasets. The mechanism that makes transfer work is the diversity of instructions and selection sizes across seven corpora, which forces the model to learn the generic operation of following an instruction and copying spans rather than memorizing one task distribution. Two inference-time mechanisms support this: document-level inference, which prompts per document and concatenates selections, and fuzzy Levenshtein grounding of generated output back to source spans.

What would settle it

Take the 500 sampled document sets per GENCS source corpus and the test documents of the six IGCS-BENCH tasks, compute exact and near-duplicate overlap, and if any overlap exists retrain the fine-tuned models with those GENCS instances removed, then re-measure whether the overall F1 gain of 45.7 versus 41.9 and the EVIDSENT gain of 72.1 versus 66.0 persist.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that extractive content selection, spanning evidence detection, salience detection, aspect-based sentence selection, extractive aspect summarization, and argument mining, is one task when formulated as instruction-guided content selection, and that a small model fine-tuned on diverse synthetic instruction–selection pairs transfers across all of these tasks. In the transfer-only setting, fine-tuning Llama-3-8B on GENCS achieves overall token-level F1 of 45.7, above the 41.9 best prompt baseline; in the supervision+transfer setting, adding GENCS to task-specific fine-tuning raises EVIDSENT F1 from 66.0 to 72.1. The paper further proposes token-level F1 as a generic metric, reporting near-perfect correlation (r > 0.99) with the average of original task-specific metrics.

Load-bearing premise

The observed transfer gains assume that no GENCS training document also appears among the IGCS-BENCH test documents, because the paper does not describe deduplication between the 500 sampled documents per GENCS corpus and the test sets.

Editorial extensions

If this is right

New content selection tasks can be tackled without any task-specific annotations by fine-tuning a small model on GENCS, with per-task results improving for four of the six benchmark tasks in the transfer-only setting.
When task-specific training data does exist, mixing in GENCS adds a further boost: EVIDSENT F1 rises from 66.0 with task-only fine-tuning to 72.1 when GENCS Union is added.
GENCS Union and GENCS Majority provide a precision–recall dial, with the union variant helping tasks with long selections such as ARGMINE and the majority variant helping short-selection tasks such as EVIDSENT.
Token-level F1 can serve as a unified evaluation metric across content selection tasks, since it correlates strongly with the four originally task-specific metrics.
Document-level inference, feeding one source document at a time and concatenating the selections, raises performance on multi-document tasks with large expected outputs such as ASPSEL and SALIENCE.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A likely reason GENCS works is that it teaches the selection routine rather than a domain; if so, the same dataset should transfer to selection tasks in languages or formats not present in the seven source corpora, an extension the paper does not test.
The reported gains could be partly inflated if any of the 500 sampled documents per GENCS corpus overlaps a benchmark test document, so a deduplicated rerun would settle whether the transfer signal is real.
The 0.61 human–LLM selection agreement suggests the synthetic references contain noise, and filtering out low-agreement instances before fine-tuning might sharpen the transfer signal.
Because the pipeline costs about $550 and scales by sampling more documents, the same three-step generation procedure could produce tailored synthetic selection data for specialized domains such as legal or biomedical documents on demand.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read the paper. The headline result is that fine-tuning a small model on GENCS, their three-LLM synthetic dataset, improves token-level F1 across six content selection tasks relative to zero/few-shot prompting, both alone and on top of task-specific training. The experiments are run across eight models with confidence intervals, permutation tests, ablations, and prompt variants. That is careful work. The benchmark unification is useful and the metric-correlation analysis is a nice addition.

The real soft spot is the unexamined overlap between GENCS training sources and IGCS-BENCH test documents. GENCS samples 500 document sets from PubMed, Wikipedia, Multi-News, Enron, books, hotel reviews, GitHub. The test tasks include EVIDSENT (medical abstracts from PubMed), ASPSUM/ASPSEL/EVIDPROP/SALIENCE (news, including Multi-News), and ARGMINE (debate articles). The paper describes no filtering or deduplication between GENCS source documents and the test splits. For a verbatim span-extraction task, having seen the source document during training can directly improve selection. The reported transfer-only gain of 45.7 vs 41.9 could be partly memorization. The authors are clearly contamination-aware—they filter ARGMINE splits on identical fields in Appendix A—but they do not apply the same reasoning to GENCS. The limitation section only mentions possible pretraining contamination, not this.

Other concerns are minor. ASPSEL has 51 test instances and SALIENCE 98; the ARGMINE split is new, so the comparison to the previously reported number is not direct. These do not change my reading.

The central claim, that a generic synthetic dataset helps small models transfer across content selection tasks, is plausible and consistently supported, but the leakage question is load-bearing and addressable. I would send this to reviewers and ask them to check whether any GENCS document or near-duplicate appears in the test sets, and ideally re-report the main numbers after filtering.

The paper deserves a serious referee. It provides a useful benchmark and a reusable synthetic data pipeline; the resources alone justify engagement. I would cite it once the contamination question is settled.

Referee Report

1 major / 4 minor

Summary. This paper proposes instruction-guided content selection (IGCS) as a unifying task formulation in which the task definition and the instance-specific query are supplied as a natural-language instruction. It introduces IGCS-BENCH, a benchmark built by converting six existing span/sentence-selection datasets (EVIDSENT, EVIDPROP, SALIENCE, ASPSEL, ASPSUM, ARGMINE) into a common prompt format, and GENCS, a synthetic training dataset generated with GPT-4, Claude-3-Opus, and Gemini-1.5-Pro over documents from seven corpora, with union and majority merging variants. The experiments fine-tune Llama-3-8B and other small models in transfer-only and supervision+transfer configurations and compare them with prompt-based baselines, reporting gains in token-level F1 (e.g., transfer-only overall F1 45.7 vs 41.9; EVIDSENT supervision+transfer F1 72.1 vs 66.0). The paper also proposes document-level inference and recommends token-level F1 as a generic evaluation metric, showing high correlation with task-specific metrics.

Significance. If the transfer gains are genuine, the paper makes a useful contribution: a reusable benchmark, a relatively inexpensive synthetic training resource, and a practical recipe for small-model content selection. The empirical apparatus is above the norm for this type of paper: bootstrap confidence intervals, permutation significance tests, ablations of the synthetic-data pipeline, prompt-variant robustness checks, human quality ratings of GENCS instructions and selections, and public release of models and datasets. The evaluation is not circular, since all six benchmark tasks use external human-annotated gold labels. The main empirical claim, however, depends on the training corpora used to create GENCS being disjoint from the benchmark test documents; that condition is currently unverified, and it is the decisive factor for my recommendation.

major comments (1)

[§4.1, Table 3, Appendix C.1; Appendix A] The most load-bearing issue is the absence of any reported check for overlap between the GENCS training document sets and the IGCS-BENCH test documents. The GENCS source corpora include PubMed and Multi-News, and the benchmark test sets include EVIDSENT/SCIFACT medical abstracts (Appendix A) and news-based tasks (EVIDPROP, SALIENCE, ASPSEL, ASPSUM). A document (or near-duplicate) from a test instance that also appears in a GENCS training instance can inflate transfer scores because the task is verbatim span selection: the fine-tuned model can memorize the source text and the kinds of spans requested for it, rather than learning a generalizable instruction-following behavior. The contamination filter described in Appendix A applies only to duplicates across ARGMINE splits, not to GENCS versus the benchmark, and Section 8's limitation paragraph addresses pretraining contamination only. I request: (i) report the number of exact and near-duplicate document overlaps between the GENCS training instances and each IGCS-BENCH test split; (ii) re-run the transfer-only and supervision+transfer comparisons after excluding all overlapping or near-duplicate training instances; or (iii) provide an ablation showing the main conclusions are unchanged when overlapping GENCS instances are removed. This is necessary because the headline claims (transfer-only overall F1 45.7 vs 41.9; EVIDSENT supervision+transfer 72.1 vs 66.0; Table 4) rest on the assumption that GENCS training does not contain benchmark test documents.

minor comments (4)

[§6.1.1 and Table 6] The sentence that Table 6 compares GENCS transfer results to the "analogous previously reported result" is imprecise, because the Best-reported column mixes settings, including a supervised result for ARGMINE as acknowledged in the table footnote; I recommend relabeling the column and moving the qualification into the main text.
[§3.3 and Appendix B] The bootstrap procedure and the permutation test are described, but the paper does not state whether the overall F1 differences (e.g., 45.7 vs 41.9 in Table 4) are tested with a paired permutation test across the six task means or only compared via the reported confidence intervals; please make this explicit.
[§5.3 and Appendix G] The fuzzy-match grounding threshold of 15% Levenshtein distance (max 10 tokens) is used both when constructing GENCS and when scoring all models; a one-paragraph sensitivity analysis over two nearby thresholds (e.g., 10% and 20%) would clarify whether the reported gains depend on this choice.
[Appendix A] For ARGMINE, the random subsample of 1,000 instances per split should be drawn with a fixed seed, and the seed should be reported, to make the benchmark exactly reproducible.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central transfer-learning claims are evaluated on external human-annotated benchmarks, not on quantities derived from the paper's own definitions.

full rationale

I find no circular step in the derivation chain. The paper's load-bearing result is empirical: fine-tuning on the synthetic GENCS dataset improves token-level F1 on the six IGCS-BENCH tasks. The target quantity, token-level F1, is computed against human-annotated gold spans from SCIFACT, SPARK, OPENASP, ASPECTNEWS, and DebateSum; none of these labels are constructed from GENCS or from the paper's equations. GENCS labels are produced by three LLMs and merged by union or majority vote, with human annotators independently rating instruction quality and selection agreement; its utility is then tested by transfer F1 on external gold data. The generic-metric correlation analysis in Section 6.3 compares two evaluation functions applied to the same model outputs; this is an empirical correlation claim, not a definitional identity, and the reported near-perfect correlation does not make the metric its own input. The paper cites prior work by overlapping authors for two benchmark datasets (OPENASP, Amar et al. 2023; SPARK, Ernst et al. 2024), but those datasets are externally published, human-annotated resources, so the overlap in authorship does not make the evaluation circular. The only substantive weakness I flag is the absence of any described deduplication between GENCS source corpora (PubMed, Wikipedia, Multi-News, etc., Section 4.1 and Appendix C.1) and IGCS-BENCH test documents; the paper's own limitation section notes only pretraining contamination, not this training-test overlap. This is a data-leakage and validity risk, not a circular-derivation risk, and it cannot be exhibited as an equation reducing a prediction to its inputs. Accordingly, the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper is empirical and resource-oriented rather than derivational. No free parameters are fitted to the benchmark; the hand-chosen constants are annotation pipeline settings (fuzzy match threshold, empty-selection proportion). The axioms are standard NLP assumptions about annotation quality and metric validity, plus the implicit reliability of LLM-generated training labels. No new theoretical entities are introduced.

free parameters (2)

Fuzzy match Levenshtein threshold = 15% of output token length, capped at 10 tokens
Chosen by hand after testing larger distances in Appendix G; affects how model outputs are grounded to source spans and thus all reported scores.
Empty selection instruction proportion = 5% of document sets per corpus
Design choice in the GENCS generation pipeline (Section 4.1) to include instructions with no relevant content; affects the distribution of training labels.

assumptions (3)

domain assumption Token-level F1 is a valid generic evaluation metric for content selection
The paper validates this empirically via system-level correlation with task-specific metrics (Section 6.3), but the correlation is computed on a limited number of model configurations and without confidence intervals.
domain assumption The human annotations in the six underlying datasets are reliable gold standards
The paper cites inter-annotator agreement figures from the original papers (Appendix A, Table 6) but does not reannotate or independently verify these gold labels.
domain assumption LLM-generated selections in GENCS are of sufficient quality to serve as training targets
Supported by human-LLM agreement of 0.61 F1 on a 105-instance sample (Section 4.2) and by downstream transfer gains, but the synthetic labels are not human-verified at scale.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Unifying Scheme for Extractive Content Selection Tasks." pith.science (2026). https://pith.science/paper/INA3SWPN

@misc{pith2026250716922,
  author       = {Pith},
  title        = {Pith review of: A Unifying Scheme for Extractive Content Selection Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/INA3SWPN}},
  note         = {Machine review of arXiv:2507.16922}
}

read the original abstract

A broad range of NLP tasks involve selecting relevant text spans from given source texts. Despite this shared objective, such \textit{content selection} tasks have traditionally been studied in isolation, each with its own modeling approaches, datasets, and evaluation metrics. In this work, we propose \textit{instruction-guided content selection (IGCS)} as a beneficial unified framework for such settings, where the task definition and any instance-specific request are encapsulated as instructions to a language model. To promote this framework, we introduce \igcsbench{}, the first unified benchmark covering diverse content selection tasks. Further, we create a large generic synthetic dataset that can be leveraged for diverse content selection tasks, and show that transfer learning with these datasets often boosts performance, whether dedicated training for the targeted task is available or not. Finally, we address generic inference time issues that arise in LLM-based modeling of content selection, assess a generic evaluation metric, and overall propose the utility of our resources and methods for future content selection models. Models and datasets available at https://github.com/shmuelamar/igcs.

Figures

Figures reproduced from arXiv: 2507.16922 by the authors.

**Figure 1.** Overall F1 scores on IGCS-BENCH for four methods across eight small language models of up-to 8B parameters from Qwen-2.5, SmolLM2 and Llama-3 families. All tested models benefit from fine-tuning with GENCS, especially smaller ones. The largest confidence interval across models is 2.0 (α = 0.05). paring the baseline rows of the two sections). Notably, results improve by enriching the finetuning data with transfer tr… view at source ↗

**Figure 2.** Difference in original metric scores between [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗

**Figure 3.** The prompt template for IGCS-BENCH tasks. The ⃝1 instruction for each task is detailed in [PITH_FULL_IMAGE:figures/full_fig_p021_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 5.** Figure 5: The prompt template for annotating selec [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]

**Figure 6.** Figure 6: Our annotation interface for manual con [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]

**Figure 7.** Figure 7: Manual annotation guidelines for naturalness and specificity for rating GENCS instructions. from 24 model configurations for ARGMINE and ASPSUM, and from 35 configurations for ASPSEL and EVIDSENT. The configurations vary along three dimensions: the choice of fine-tunin…

**Figure 8.** Figure 8: Overall F1 scores on IGCS-BENCH for eight models, evaluated across four methods and three prompt variants (see examples for the prompt variants in [PITH_FULL_IMAGE:figures/full_fig_p026_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 26 canonical work pages

[1]

You must keep the 5 instructions short and concise - as a single sentence

work page
[2]

Select {code|content}

Instructions must start with the words "Select {code|content}" as they are always for select- ing {code|content} from the {file|document}(s) and never for writing a new {code|text} nor paraphrasing the original content

work page
[3]

Instructions should not be too specific that hint on the an- swer and not too vague that cannot be fulfilled

work page
[4]

{Source File|Document} #0: Document 0 text

Write the instructions as a numbered list. {Source File|Document} #0: Document 0 text. . . {Source File|Document} #1: Document 1 text. . . . . . Figure 4: The prompt template for annotating IGCS in- structions is as follows: from each choice of { 1⃝| 2⃝}, we use 1⃝ for the GitHub code dataset and 2⃝ for all other datasets. For empty selection annotation, ...

work page
[5]

In Companion Pro- ceedings of the ACM Web Conference 2023 , WWW ’23 Companion, page 28–31, New York, NY , USA

Strum: Extractive aspect-based con- trastive summarization. In Companion Pro- ceedings of the ACM Web Conference 2023 , WWW ’23 Companion, page 28–31, New York, NY , USA. Association for Computing Machin- ery. Jia Guo, Liying Cheng, Wenxuan Zhang, Stan- ley Kok, Xin Li, and Lidong Bing. 2023. AQE: Argument quadruplet extraction via a quad-tagging augmente...

work page 2023
[7]

In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 15339–15353, Bangkok, Thailand

Same task, more tokens: the impact of input length on the reasoning performance of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 15339–15353, Bangkok, Thailand. Association for Computational Linguistics. Patrick Lewis, Barlas Oguz, Ruty Rinott, Se- bastian...

work page 2020
[8]

Transactions of the Associ- ation for Computational Linguistics , 12:933– 949

State of what art? a call for multi-prompt LLM evaluation. Transactions of the Associ- ation for Computational Linguistics , 12:933– 949. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou

work page
[10]

In Proceedings of the 61st Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers) , pages 9709– 9728, Toronto, Canada

Concise answers to complex ques- tions: Summarization of long-form answers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers) , pages 9709– 9728, Toronto, Canada. Association for Com- putational Linguistics. Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. ...

work page 2020

Show all 31 references

[11]

Gaurav Sahu, Olga Vechtomova, and Issam H

OpenReview.net. Gaurav Sahu, Olga Vechtomova, and Issam H. Laradji. 2025. A guide to effectively lever- aging LLMs for low-resource text summariza- tion: Data augmentation and semi-supervised approaches. In Findings of the Association for Computational Linguistics: NAACL 2025 ...

2025
[12]

In Proceedings of the 2024 Conference on Empir- ical Methods in Natural Language Processing , pages 11679–11695, Miami, Florida, USA

Major entity identification: A general- izable alternative to coreference resolution. In Proceedings of the 2024 Conference on Empir- ical Methods in Natural Language Processing , pages 11679–11695, Miami, Florida, USA. As- sociation for Computational Linguistics. Mirac Suzgun...

2024
[16]

Frontiers of Computer Science, 18(6):186357

Large language models for generative information extraction: a survey. Frontiers of Computer Science, 18(6):186357. Yumo Xu and Mirella Lapata. 2020. Coarse-to- fine query focused multi-document summariza- tion. In Proceedings of the 2020 Conference on Empirical Methods in Nat...

2020 arXiv
[17]

sen- tences

Extractive summarization via ChatGPT for faithful summary generation. In Findings of the Association for Computational Linguis- tics: EMNLP 2023 , pages 3270–3278, Singa- pore. Association for Computational Linguis- tics. Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, and Hoifu...

2024
[22]

Do not change spaces or punctuation, do not fix typos and avoid any other changes to the {code|content} you select

Output the exact verbatim {code|text phrases} from the {source file|document} (s). Do not change spaces or punctuation, do not fix typos and avoid any other changes to the {code|content} you select
[23]

Follow the instructions as closely as possible and pay care- ful attention to them
[24]

Instructions:

Output format must be a two level nested list, the first level is the instruction and the second level is the multi- ple {code|content} selections copied from the original {source file|document} (s). Instructions:
[25]

<instruction #1 text>
[26]

<instruction #2 text>
[27]

<instruction #3 text>
[28]

<instruction #4 text>
[29]

{Source File|Document} #0: Document 0 text

<instruction #5 text>. {Source File|Document} #0: Document 0 text. . . {Source File|Document} #1: Document 1 text. . . . . . Figure 5: The prompt template for annotating selec- tions. From every choice of { 1⃝| 2⃝}, we use 1⃝ for the GitHub code dataset and 2⃝ otherwise. inter...

2005
[30]

Select content that lists the factors increasing the likelihood of resumption of ovarian cyclicity (ROC) at 36 to 42 days in milk (DIM) from Document #1

Naturalness – The instruction is clear, fluent, plausible, and relevant to the context of the document set and its topic. • Clear – The instruction is unambiguous and understandable, given the topic. • Fluent – The instruction is written in natural, human-like language and adh...
[31]

Sentence 1

Specificity – The information sought by the instruction is central (1) or anecdotal (5) to the topic of the document set. • To assess specificity, imagine a conceptual tree of all information organized hierarchically within the document set. The measure reflects how deep into ...

1989
[2000]

In Fourth Conference on Com- putational Natural Language Learning and the Second Learning Language in Logic Workshop

Introduction to the CoNLL-2000 shared task chunking. In Fourth Conference on Com- putational Natural Language Learning and the Second Learning Language in Logic Workshop. David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajish...

2000
[2010]

In Proceedings of the 16th ACM SIGKDD Inter- national Conference on Knowledge Discovery and Data Mining , KDD ’10, page 783–792, New York, NY , USA

Latent aspect rating analysis on review text data: a rating regression approach. In Proceedings of the 16th ACM SIGKDD Inter- national Conference on Knowledge Discovery and Data Mining , KDD ’10, page 783–792, New York, NY , USA. Association for Comput- ing Machinery. Yiming W...

2024
[2017]

Proceedings of the AAAI Conference on Artificial Intelligence , 31(1)

SummaRuNNer: A Recurrent Neural Network Based Sequence Model for Extractive Summarization of Documents. Proceedings of the AAAI Conference on Artificial Intelligence , 31(1). E.W. Noreen. 1989. Computer-Intensive Methods for Testing Hypotheses: An Introduction . Wi- ley. OpenA...

1989 arXiv
[2019]

In Pro- ceedings of the 2nd Workshop on New Frontiers in Summarization , pages 64–69, Hong Kong, China

Towards annotating and creating sum- mary highlights at sub-sentence level. In Pro- ceedings of the 2nd Workshop on New Frontiers in Summarization , pages 64–69, Hong Kong, China. Association for Computational Linguis- tics. Regina Barzilay and Michael Elhadad. 1997. Us- ing l...

1997
[2021]

In Proceedings of the Neural Informa- tion Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Bench- marks 2021, December 2021, virtual

BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Proceedings of the Neural Informa- tion Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Bench- marks 2021, December 2021, virtual. James Thorne, Andreas Vla...

2021
[2022]

Danqi Chen, Adam Fisch, Jason Weston, and An- toine Bordes

How optimal is greedy decoding for ex- tractive question answering? In 4th Confer- ence on Automated Knowledge Base Construc- tion, AKBC 2022, London, UK, November 3-5, 2022. Danqi Chen, Adam Fisch, Jason Weston, and An- toine Bordes. 2017. Reading Wikipedia to an- swer open-d...

2022
[2023]

In Proceedings of the 61st Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers) , pages 2680–2697, Toronto, Canada

Generating EDU extracts for plan-guided summary re-ranking. In Proceedings of the 61st Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers) , pages 2680–2697, Toronto, Canada. Associa- tion for Computational Linguistics. Ojas Ahuja, Jiachen...

2022 arXiv
[2024]

CoRR, abs/2407.21783v1

The llama 3 herd of models. CoRR, abs/2407.21783v1. Bradley Efron and Robert J. Tibshirani. 1994. An Introduction to the Bootstrap , 1st edition. Chapman & Hall/CRC, New York, USA. Ori Ernst, Avi Caciularu, Ori Shapira, Ramakanth Pasunuru, Mohit Bansal, Jacob Goldberger, and I...

1994 arXiv
[2025]

In Proceedings of the 63nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vienna, Austria

Laquer: Localized attribution queries in content-grounded generation. In Proceedings of the 63nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vienna, Austria. Association for Com- putational Linguistics. Ella Hofmann-Coyle, Mayank Ku...

2022 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

You must keep the 5 instructions short and concise - as a single sentence

work page

[2] [2]

Select {code|content}

Instructions must start with the words "Select {code|content}" as they are always for select- ing {code|content} from the {file|document}(s) and never for writing a new {code|text} nor paraphrasing the original content

work page

[3] [3]

Instructions should not be too specific that hint on the an- swer and not too vague that cannot be fulfilled

work page

[4] [4]

{Source File|Document} #0: Document 0 text

Write the instructions as a numbered list. {Source File|Document} #0: Document 0 text. . . {Source File|Document} #1: Document 1 text. . . . . . Figure 4: The prompt template for annotating IGCS in- structions is as follows: from each choice of { 1⃝| 2⃝}, we use 1⃝ for the GitHub code dataset and 2⃝ for all other datasets. For empty selection annotation, ...

work page

[5] [5]

In Companion Pro- ceedings of the ACM Web Conference 2023 , WWW ’23 Companion, page 28–31, New York, NY , USA

Strum: Extractive aspect-based con- trastive summarization. In Companion Pro- ceedings of the ACM Web Conference 2023 , WWW ’23 Companion, page 28–31, New York, NY , USA. Association for Computing Machin- ery. Jia Guo, Liying Cheng, Wenxuan Zhang, Stan- ley Kok, Xin Li, and Lidong Bing. 2023. AQE: Argument quadruplet extraction via a quad-tagging augmente...

work page 2023

[6] [7]

In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 15339–15353, Bangkok, Thailand

Same task, more tokens: the impact of input length on the reasoning performance of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 15339–15353, Bangkok, Thailand. Association for Computational Linguistics. Patrick Lewis, Barlas Oguz, Ruty Rinott, Se- bastian...

work page 2020

[7] [8]

Transactions of the Associ- ation for Computational Linguistics , 12:933– 949

State of what art? a call for multi-prompt LLM evaluation. Transactions of the Associ- ation for Computational Linguistics , 12:933– 949. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou

work page

[8] [10]

In Proceedings of the 61st Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers) , pages 9709– 9728, Toronto, Canada

Concise answers to complex ques- tions: Summarization of long-form answers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers) , pages 9709– 9728, Toronto, Canada. Association for Com- putational Linguistics. Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. ...

work page 2020