Pith. sign in

REVIEW 18 cited by

A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2105.03011 v1 pith:IXQT57G3 submitted 2021-05-07 cs.CL

Pradeep Dasigi , Kyle Lo , Iz Beltagy , Arman Cohan , Noah A. Smith , Matt Gardner This is my paper
classification cs.CL
keywords questionsansweringquestiondatasetinformation-seekingresearchanswersexisting
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Readers of academic research papers often read with the goal of answering specific questions. Question Answering systems that can answer those questions can make consumption of the content much more efficient. However, building such tools requires data that reflect the difficulty of the task arising from complex reasoning about claims made in multiple parts of a paper. In contrast, existing information-seeking question answering datasets usually contain questions about generic factoid-type information. We therefore present QASPER, a dataset of 5,049 questions over 1,585 Natural Language Processing papers. Each question is written by an NLP practitioner who read only the title and abstract of the corresponding paper, and the question seeks information present in the full text. The questions are then answered by a separate set of NLP practitioners who also provide supporting evidence to answers. We find that existing models that do well on other QA tasks do not perform well on answering these questions, underperforming humans by at least 27 F1 points when answering them from entire papers, motivating further research in document-grounded, information-seeking QA, which our dataset is designed to facilitate.

Discussion (0). Sign in to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MesaNet: Sequence Modeling by Locally Optimal Test-Time Training

    cs.LG 2025-06 conditional novelty 7.0 of 10

    MesaNet uses conjugate-gradient-optimal test-time regression in a chunkwise-parallelizable recurrent layer, achieving strong language modeling and benchmark performance at up to 1B scale.

  2. TaskPress: Query-Agnostic KV Cache Compression via Task-Guided Pruning

    cs.AI 2026-08 conditional novelty 6.0 of 10

    A task guide acts as a meta-query to compress the KV cache once per task, so one compressed memory serves many future questions.

  3. Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Sleep-time Knowledge Seeding plus Dreaming lets LLMs expand capacity, distill fragile in-context memories into stable parameters, and self-improve without human labels.

  4. CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilation

    cs.LG 2026-02 unverdicted novelty 6.0 of 10

    Offline-learned head-reliability and risk-threshold tables make prefill-only KV compression recover about 97.7% of uncompressed LongBench accuracy at a 512-token-per-layer memory budget.

  5. Linguistic Nepotism: Trading-off Quality for Language Preference in Multilingual RAG

    cs.CL 2025-09 conditional novelty 6.0 of 10

    In multilingual retrieval-augmented generation, models cite English evidence more accurately than translated evidence, and this language preference can outweigh document relevance.

  6. EvolKV: Evolutionary KV Cache Compression for LLM Inference

    cs.LG 2025-09 conditional novelty 6.0 of 10

    CMA-ES search over per-layer KV cache budgets beats uniform and pyramidal compression heuristics on LongBench, NIAH, RULER, and GSM8K, and edges past the full cache on one code dataset at 1.5% of the budget.

  7. Cartridges: Lightweight and general-purpose long context representations via self-study

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A per-corpus trained KV cache, called a Cartridge, matches full-context in-context learning quality on long-document benchmarks while using up to 38.6x less serving memory.

  8. CLBench-V: Evaluating Multimodal Context Learning from Grounding to Knowledge Acquisition

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Multimodal context-learning benchmark CLBench-V separates grounding, information application, and knowledge acquisition; the best evaluated model scores 0.2847.

  9. Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse

    cs.CL 2026-02 unverdicted novelty 5.0 of 10

    Attention-sink weight is recast as an implicit MoE router per head, motivating a sink-aware head-balancing loss that yields small, consistent benchmark gains across three attention variants but rests on a definitional...

  10. LExI: Layer-Adaptive Active Experts for Efficient MoE Model Inference

    cs.LG 2025-09 conditional novelty 5.0 of 10

    LExI sets a different number of active experts per layer, found by weight-only sensitivity profiling and evolutionary search, improving MoE inference throughput with little accuracy loss.

  11. Learn from the Past: Fast Sparse Indexing for Large Language Model Decoding

    cs.LG 2025-05 reject novelty 5.0 of 10

    LFPS predicts which KV cache positions a decoder will attend to by reusing historical vertical and slash patterns plus local expansion, then runs exact Top-k only on the predicted candidate set.

  12. Ranking Free RAG: Replacing Re-ranking with Selection in RAG for Sensitive Domains

    cs.CL 2025-05 reject novelty 5.0 of 10

    METEORA uses DPO-tuned rationales to select and verify evidence chunks in RAG, and claims better recall, precision, evidence efficiency, and poisoning defense, though key evaluation details are missing.

  13. HoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Models

    cs.CL 2025-09 reject novelty 4.0 of 10

    HoPE replaces RoPE's sine/cosine rotations with hyperbolic functions plus an exponential damping term to enforce monotonic attention decay, but the claimed consistent superiority and the 'RoPE as special case' theorem...

  14. PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference

    cs.LG 2025-09 conditional novelty 4.0 of 10

    PagedEviction prunes the KV cache in whole blocks using a key-value norm ratio, speeding up long-context LLM inference in vLLM while staying close to full-cache accuracy.

  15. Deep Research Agents: A Systematic Examination And Roadmap

    cs.AI 2025-06 conditional novelty 4.0 of 10

    A survey that organizes LLM-powered deep research agents into static versus dynamic workflows and single versus multi agent architectures, and reviews their benchmarks and open challenges.

  16. A Comprehensive Graph Framework for Question Answering with Mode-Seeking Preference Alignment

    cs.CL 2025-06 conditional novelty 4.0 of 10

    GraphMPA combines an embedding-similarity hierarchical graph with mode-seeking preference optimization to improve RAG question answering on six datasets.

  17. Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey

    cs.CL 2025-06 conditional novelty 4.0 of 10

    A survey that classifies AI agent evaluation benchmarks along environment and capability axes, and proposes five traits that distinguish agents from chatbots.

  18. On the Comprehensibility of Multi-structured Financial Documents using LLMs and Pre-processing Tools

    cs.IR 2025-06 conditional novelty 4.0 of 10

    Preprocessing financial PDFs into text, tables, and chart data with existing tools improves LLM question-answering accuracy over direct GPT-4o image input in a small private evaluation.

Pith tools