MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documents

Ashish Sabharwal; Dan Roth; Harsh Trivedi; Mor Geva; Reut Tsarfaty; Tomer Wolfson; Tushar Khot; Yoav Goldberg

REVIEW 4 major objections 4 minor 1 cited by

MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documents

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MoNaCo: the first benchmark to pair natural, human-written questions with answers that span dozens to hundreds of evidence pages, and the paper reports that frontier LLMs still cap out at 61.2% F1.

desk verdict A benchmark that fills a real gap; the GPT-4O anchoring of factoid gold answers is a genuine caveat, but the central finding—frontier LLMs are far from solving these broad multi-document questions—holds up. read the letter →

arxiv 2508.11133 v2 pith:RVL3OSTA submitted 2025-08-15 cs.CL cs.AIcs.DB

Tomer Wolfson , Harsh Trivedi , Mor Geva , Yoav Goldberg , Dan Roth , Tushar Khot , Ashish Sabharwal , Reut Tsarfaty This is my paper · ORCID

classification cs.CLcs.AIcs.DB

keywords MoNaCoquestionansweringbenchmarkmulti-documentreasoninginformation-seekingLLMevaluationdecompositionretrieval-augmentedgenerationlistquestions

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MoNaCo is built around a gap in existing QA benchmarks: nobody had collected natural, human-written questions whose answers genuinely take hours of research across dozens or hundreds of pages. The paper claims it has closed that gap with 1,315 questions averaging 43.3 evidence pages each, annotated with full reasoning chains. It then reports that 15 frontier LLMs score at most 61.2% F1, with the best model perfect on only 38.7% of questions, and that errors are dominated by low recall and hallucination. A sympathetic reader should take away that broad, time-consuming information-seeking remains unsolved by current LLM agents, and that MoNaCo gives the field a way to measure progress.

What carries the argument

The load-bearing mechanism is a decomposed annotation pipeline built on QDMR question decompositions. Workers first write time-consuming questions from personas, then expert annotators decompose each question into intermediate steps, and a 31-operator execution engine automatically derives follow-up questions and runs aggregation operations (sort, top-n, group, set operations, arithmetic) while crowdworkers answer the remaining factoid and list subquestions. This decomposition is what makes it possible to annotate gold answers spanning hundreds of pages without any single annotator seeing the whole task, and the same decompositions let the authors compute complexity statistics and reasoning-template diversity.

What would settle it

Take a random sample of MoNaCo questions, have independent expert annotators (no GPT-4O, no authors) re-answer them from Wikipedia and compare to the gold answers; if the expert-gold agreement is substantially below ordinary human annotation agreement, the benchmark's scores are not measuring what they appear to measure.

Watch

Extended reading notes

Core claim

The paper's central claim is that MoNaCo is the first QA benchmark to combine genuinely natural, human-written questions with solutions that span dozens to hundreds of evidence documents. Each of the 1,315 questions comes with a gold QDMR-style decomposition into intermediate steps, 90,773 intermediate question-answer units, and evidence drawn from 36,194 distinct Wikipedia pages, with an average of 43.3 pages and 5.1 reasoning steps per question. Evaluated in a closed-book setting, the best frontier model reaches 61.2% F1; even with all gold evidence supplied as context, two strong models score only about 59% F1, and adding BM25 retrieval drops performance by over 12 points. The paper interprets these results as showing that aggregating and reasoning over dozens of facts is an open challenge for LLM-powered agents.

Load-bearing premise

The benchmark's gold answers are assumed to be correct, but factoid answers were accepted only when a crowdworker's answer agreed with GPT-4O's answer, so a systematic bias in GPT-4O's factual beliefs would be inherited by every gold answer and every reported F1 score.

Editorial extensions

If this is right

Frontier LLM agents are not close to solving broad, human-time-scale information-seeking tasks; the best model's 61.2% F1 and 38.7% perfect-answer rate leave large headroom.
Even when all gold evidence is placed in the prompt, strong models score only about 59% F1, so long-context reasoning over dozens of documents is itself unsolved.
Standard RAG can hurt: BM25 retrieval lowers performance by more than 12 F1 points, indicating current LLMs are not robust to even partially irrelevant retrieved context.
List questions are a major bottleneck: model recall drops sharply as the number of expected answers grows past roughly 20, so improving exhaustive set recall is a concrete target.
MoNaCo's per-question time-dependence labels (about half stable) and released answer timestamps make it usable as a longitudinal leaderboard for agentic search systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper's claims, MoNaCo's decomposition-and-executor machinery could be turned around to generate fresh benchmark questions automatically from Wikipedia snapshots, making contamination harder to manage.
The 16.1% of questions whose answers change yearly imply that a static leaderboard will decay in interpretability; using only the 49.8% time-independent subset for head-to-head model comparisons would give more stable rankings.
The hybrid factoid validation makes GPT-4O a de facto oracle for part of the gold standard; a manual re-annotation study on a sample would quantify how much of the reported gap is model failure versus benchmark noise.
Because MoNaCo contains 40,125 boolean and 8,549 list subquestions, it can double as a fine-grained diagnostic for the two subskills (exhaustive enumeration and yes/no verification) that broader agent benchmarks rarely isolate.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

MoNaCo is worth reading and worth taking seriously. The contribution is real: it is the first QA benchmark I know of that pairs human-written, natural information-seeking questions with evidence that typically spans dozens of Wikipedia pages (43.3 on average), and the pipeline that gets there—persona-based question elicitation plus QDMR decomposition plus distributed crowd answering—is a practical way to scale annotation of genuinely time-consuming questions. The complexity comparisons against eight benchmarks are persuasive, and the user study on naturalness, small as it is, supports the claim that these are not machine-generated trivia. The release of 1,315 questions with 90K intermediate answers, evidence, prompts, and predictions makes it immediately usable.

I have one real reservation and two smaller ones. The real one: factoid intermediate answers are accepted when a worker's answer agrees with GPT-4O's own answer (Section 3.3), so a systematic GPT-4O blind spot could be baked into the gold chains. The stress-test note is right that this is upstream of every reported F1. It is mitigated, not eliminated, by manual review of all 1,315 final aggregate answers and by human overlap for list questions. I do not think it invalidates the central claim—the model-vs-model gaps and the low-recall pattern are too large to be an artifact—but a benchmark whose gold answers are partly model-anchored should report a human-agreement calibration on a sample and say how many final-answer corrections the manual review produced. The smaller issues: GPT-4.1 is used as judge without any human-agreement study, none of the F1 scores come with confidence intervals, and there is no human ceiling. The last point is understandable given how time-consuming the questions are, but the absence makes "60% F1" harder to interpret as a capability gap; even a small expert sample would help.

Citation patterns look fine; the QDMR self-citations are appropriate and the related-work coverage is fair. One framing caveat: these questions are crowd-written with persona prompts, not harvested from real user logs; the paper is honest about this, and the user study at least measures perceived naturalness.

Who this is for: anyone evaluating long-context reasoning, agentic search, or retrieval-augmented QA. It deserves serious peer review. I would ask for the calibration, confidence intervals, and a human ceiling sample before accepting, but I would not desk-reject.

Referee Report

4 major / 4 minor

Summary. The paper introduces MoNaCo, a benchmark of 1,315 human-written, time-consuming questions whose gold solutions span dozens of Wikipedia pages (mean 43.3, median 12). The authors construct the benchmark with a decomposed annotation pipeline: crowd workers write persona-driven questions, trained workers annotate QDMR-style decompositions, and the execution engine derives intermediate questions that are answered by crowd workers with page/section/sentence evidence. Validation uses GPT-4O for factoid answer agreement, human overlap thresholds for list answers, and manual author review of all final aggregate answers. The paper evaluates 15 LLMs in closed-book, oracle-retrieval, and BM25-retrieval settings, reporting a top F1 of 61.2% (O3), with low recall and hallucinations as the main failure modes. A user study with 18 graduate students is used to argue that MoNaCo questions are more natural than machine-generated complex QA benchmarks while being far more complex than existing natural benchmarks. The benchmark, code, prompts, and predictions are publicly released.

Significance. If the gold answers and the LLM-judge evaluation are sound, MoNaCo fills a genuine gap: existing benchmarks either use natural but simple questions or complex but artificial ones, whereas MoNaCo combines human-written natural questions with evidence that spans dozens of pages. The resource itself is substantial: 90,773 intermediate questions, 36,194 evidence pages, list/boolean/factoid and table reasoning, and public releases of predictions and judge scores. The paper's complexity statistics (Table 3) and the multi-setting evaluation (closed-book, oracle, RAG) provide a useful testbed for agentic and deep-research systems. The explicit manual review of all 1,315 final answers and the human-overlap validation for list questions are notable strengths. The main risk is the dependence of the gold answers and the final scores on GPT-4O-family models, which this report treats as the central validity concern.

major comments (4)

[§3.3, Factoid QA Validation] The acceptance criterion for factoid intermediate answers requires a crowdworker's answer to be judged by GPT-4O as agreeing with GPT-4O's own answer; disagreements are re-annotated until agreement. With 42,099 non-list, non-boolean intermediate answers (§4.1), this protocol effectively conditions the gold reasoning chains on GPT-4O's factual beliefs and answer formatting. A systematic GPT-4O error on any factoid category would propagate into the aggregated final answers, and the manual review of the 1,315 final answers cannot reliably detect an incorrect intermediate value that still yields a plausible aggregate (e.g., a wrong denominator or a wrong date range). Since every reported score in Table 4, Figure 7, and Figure 9 is computed against these gold answers, this is a load-bearing validity risk. Please add an independent human review of a random sample of factoid intermediate answers and report the agreement rate, and report how often the worker's original answer was overridden by the GPT-4O agreement criterion.
[§5.1, LLM-as-judge evaluation] The evaluation uses GPT-4.1 as an LLM judge to decide answer equivalence and to extract precision/recall, but no human-agreement calibration is reported. The judge comes from the same model family as the GPT-4O used to construct the gold answers, so model predictions that resemble GPT-4O's answer style may be systematically favored. The limitation statement in §8 acknowledges possible judge mistakes but does not quantify them. Please provide a human-judge agreement study on a sample of at least 100-200 predictions and report judge error rates by answer type (factoid, list, boolean, numeric). Without such calibration, the headline '61.2% F1' is not known to be a faithful measurement of model performance on these gold answers.
[§4.2, Table 3] The complexity comparison relies on GPT-4O-generated decompositions for all non-MoNaCo benchmarks, while MoNaCo uses human-annotated decompositions. If GPT-4O under-decomposes the other benchmarks, the reported differences (5.1 vs. 3.2 reasoning steps; 38% vs. at most 12% aggregation) could overstate MoNaCo's relative complexity. This does not affect the #Pages evidence, but it does affect the 'more complex than all existing benchmarks' claim. Please validate the generated decompositions on a sample (e.g., human annotation of 100 questions per benchmark) or explicitly characterize how the comparison depends on this asymmetry.
[Table 4 and §5.2] No confidence intervals, variance estimates, or multiple runs are reported for any F1 score. With 1,315 questions, differences among top models (e.g., 61.18 vs. 60.11 for O3 vs. GPT-5) are within likely sampling noise, and the single-run nature also affects the 61.2% headline. Please report bootstrap confidence intervals over questions for the main table and state how many runs the numbers are based on. This is important for the benchmark's stated purpose of tracking progress over time.

minor comments (4)

[§5.1, Figure 12] Please clarify how F1 is computed for single numeric answers when the judge outputs a normalized similarity score rather than a binary precision label; the text says precision and recall are computed automatically, but the formula in Figure 12 only defines a precision-like value and the recall aggregation over factoid answers is not specified.
[§4.3, Figure 1] The 'naturalness' score on the x-axis of Figure 1 is described as a weighted average of two measures, but the weighting formula is not defined; please state the exact normalization and weights so the reader can reproduce the figure.
[Throughout] The benchmark name is typeset inconsistently as 'MoNaCo', 'MONACO', and 'M ONACO'; please choose one spelling and use it consistently in text, tables, and figure captions.
[Appendix A.1] The claim that automatic execution of operator steps 'leaves no room for potential calculation errors by humans' would be stronger if the executor's implementations of the 31 operators in Table 7 were unit-tested or validated against a small manually computed set; please state whether such tests were performed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark's empirical findings are not constructed from their inputs.

full rationale

The central claim—that frontier LLMs reach at most 61.2% F1 on MoNaCo—is an independent measurement against a human-annotated gold set, not a quantity fitted from the inputs. The pipeline has separate stages: crowd workers write questions, experts produce QDMR decompositions, workers answer intermediate questions with Wikipedia evidence, the executor computes aggregations, and final answers are manually reviewed by the authors. The factoid validation loop in §3.3 uses GPT-4O to accept answers only when a worker and GPT-4O agree, and the evaluation in §5.1 uses GPT-4.1 as a judge; this is a same-family-model validity concern rather than a derivational circularity. Gold answers are not defined as GPT-4O outputs: list answers require two-worker overlap above 0.77, final aggregate answers are author-reviewed, and all evaluated models are scored against that resulting gold. GPT-4O's own score of 48.98 F1 is not forced by construction. The reuse of QDMR from Wolfson et al. (2020) is a methodological citation, not a load-bearing self-citation that predetermines the benchmark's difficulty or the model results. No equation or fitted parameter is renamed as a prediction, and no claimed result reduces to its own input.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claims depend on three families of assumptions: gold-answer correctness (crowd workers, GPT-4O validation, and author review), the validity of QDMR decompositions and their 31 operator implementations, and the reliability of GPT-4.1 as an answer-equivalence judge. No new physical or formal entities are introduced. The free parameters are annotation-agreement thresholds, the numeric tolerance used by the judge, and the BM25 top-k setting.

free parameters (4)

list answer agreement threshold = 0.77
Two worker-annotated lists are considered in agreement only if the overlap score exceeds 0.77; this threshold determines which lists are accepted as gold.
list length difference threshold = 0.25
Accepted lists must have normalized length difference below 0.25; this filters list pairs that differ too much in size.
numerical tolerance margin in judge = 1 to 5.5 percentage points
The GPT-4.1 judge marks numeric answers as correct if within 1 to 5.5 percentage points; this directly affects precision, recall, and F1.
BM25 top-k = 20
The retrieval-augmented experiments feed the top-20 BM25 documents to the LLM; the RAG results depend on this choice.

assumptions (5)

domain assumption Persona-primed crowdworkers produce questions that reflect real-world information-seeking behavior
The paper relies on this to claim MoNaCo questions are natural; support comes only from a user study with 18 graduate students (§4.3).
domain assumption English Wikipedia is a complete and correct source for all answers
All gold answers are anchored to Wikipedia pages, sections, sentences, or table cells (§3.2); if Wikipedia is incomplete or wrong on a fact, the benchmark answer is wrong.
domain assumption The QDMR decomposition and the 31 operator implementations exactly capture the reasoning needed for the original question
Final answers are produced by executing these operators over leaf answers (§2.3); a wrong decomposition or operator semantics invalidates gold.
domain assumption GPT-4O answer agreement is a valid proxy for correctness of factoid gold answers
Factoid answers are accepted only when a worker agrees with GPT-4O under an LLM judge (§3.3); the paper does not validate this against human judges.
domain assumption GPT-4.1 judge correctly determines equivalence between model predictions and gold answers
All F1 scores rely on this judge, and the paper reports no human agreement study for the judge (§5.1, Limitations).

how reviews work

0 comments

Cite this review

Pith. "Pith review of MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documents." pith.science (2026). https://pith.science/paper/RVL3OSTA

@misc{pith2026250811133,
  author       = {Pith},
  title        = {Pith review of: MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RVL3OSTA}},
  note         = {Machine review of arXiv:2508.11133}
}

read the original abstract

Automated agents, powered by Large language models (LLMs), are emerging as the go-to tool for querying information. However, evaluation benchmarks for LLM agents rarely feature natural questions that are both information-seeking and genuinely time-consuming for humans. To address this gap we introduce MoNaCo, a benchmark of 1,315 natural and time-consuming questions that require dozens, and at times hundreds, of intermediate steps to solve -- far more than any existing QA benchmark. To build MoNaCo, we developed a decomposed annotation pipeline to elicit and manually answer real-world time-consuming questions at scale. Frontier LLMs evaluated on MoNaCo achieve at most 61.2% F1, hampered by low recall and hallucinations. Our results underscore the limitations of LLM-powered agents in handling the complexity and sheer breadth of real-world information-seeking tasks -- with MoNaCo providing an effective resource for tracking such progress. The MoNaCo benchmark, codebase, prompts and models predictions are all publicly available at: https://tomerwolgithub.github.io/monaco

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

From Interpretability to Control: Insights from Six Years of the TrustNLP Workshop
cs.CL 2026-08 conditional novelty 5.0 of 10

Truthfulness grew from zero papers in 2021-2022 to the largest topic by 2025-2026, while explainability declined and then resurged in 2026 through mechanistic interpretability.

Reference graph

Works this paper leans on

11 extracted references · 9 canonical work pages · cited by 1 Pith paper

[1]

Arthur Miller was born on October 17, 1915, so he turned 20 on October 17, 1935 and he turned 40 on October 17, 1955

work page 1915
[2]

Between October 17, 1935 and October 17, 1955 Arthur Miller wrote the following plays: No Villain (1936), They Too Arise (1937), Honors at Dawn (1938), The Grass Still Grows (1938), The Great Disobedience (1938), Listen My Children (1939), The Golden Years (1940), The Half-Bridge (1943), The Man Who Had All the Luck (1944), All My Sons (1947), Death of a ...

work page 1936
[3]

In ICML 2022 Workshop on Knowledge Retrieval and Language Models

Large language models are zero-shot rea- soners. In ICML 2022 Workshop on Knowledge Retrieval and Language Models. Satyapriya Krishna, Kalpesh Krishna, Anhad Mo- hananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, and Manaal Faruqui. 2024. Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation. Anastasia Krithara, Anastasios ...

work page 2022
[6]

In Proceedings of the 61st Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers) , pages 14032– 14047, Toronto, Canada

QUEST: A retrieval dataset of entity- seeking queries with implicit set operations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers) , pages 14032– 14047, Toronto, Canada. Association for Com- putational Linguistics. OpenAI. 2024. GPT-4 technical report. OpenAI. 2024. OpenAI o1 system car...

work page 2024
[8]

list questions

Measuring short-form factuality in large language models. Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. 2025. Browsecomp: A simple yet challenging benchmark for browsing agents. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi,...

work page 2020
[11]

Answers: 15 Figure 18: Excerpt of the few-shot chain-of-thought prompt which includes intermediate answers (CoT+Ans)

Therefore Miller had written 15 plays between October 17, 1935 and October 17, 1955. Answers: 15 Figure 18: Excerpt of the few-shot chain-of-thought prompt which includes intermediate answers (CoT+Ans)

work page 1935
[2016]

ArXiv, abs/1611.09268

MS MARCO: A human generated MA- chine Reading COmprehension dataset. ArXiv, abs/1611.09268. Bernd Bohnet, Vinh Q. Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, Tom Kwiatkowski, Ji Ma, Jianmo Ni, Tal Schuster, William W. Cohen, Michael Collins, Dipanjan Das, Donald Met- zler, ...

arXiv 2022
[2021]

In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)

BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multihop questions via single-hop question composition. Transaction...

work page 2022

Show all 11 references

[2022]

Transactions of the Association for Computational Linguistics, 10:111–126

Break, perturb, build: Automatic pertur- bation of reasoning paths through question de- composition. Transactions of the Association for Computational Linguistics, 10:111–126. Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, N. Chawla, Olaf Wiest, and Xianglian...

2024 arXiv
[2023]

Scientific Data, 10(1):170

Bioasq-qa: A manually curated corpus for biomedical question answering. Scientific Data, 10(1):170. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Ja- cob Devlin, Kenton Lee, Kristina Tou...

2019
[2024]

ExpertQA: Expert-curated questions and attributed answers. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies (Volume 1: Long Papers) , pages 3025–3045, Mexico City, Mexico. Associ...

2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Arthur Miller was born on October 17, 1915, so he turned 20 on October 17, 1935 and he turned 40 on October 17, 1955

work page 1915

[2] [2]

Between October 17, 1935 and October 17, 1955 Arthur Miller wrote the following plays: No Villain (1936), They Too Arise (1937), Honors at Dawn (1938), The Grass Still Grows (1938), The Great Disobedience (1938), Listen My Children (1939), The Golden Years (1940), The Half-Bridge (1943), The Man Who Had All the Luck (1944), All My Sons (1947), Death of a ...

work page 1936

[3] [3]

In ICML 2022 Workshop on Knowledge Retrieval and Language Models

Large language models are zero-shot rea- soners. In ICML 2022 Workshop on Knowledge Retrieval and Language Models. Satyapriya Krishna, Kalpesh Krishna, Anhad Mo- hananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, and Manaal Faruqui. 2024. Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation. Anastasia Krithara, Anastasios ...

work page 2022

[4] [6]

In Proceedings of the 61st Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers) , pages 14032– 14047, Toronto, Canada

QUEST: A retrieval dataset of entity- seeking queries with implicit set operations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers) , pages 14032– 14047, Toronto, Canada. Association for Com- putational Linguistics. OpenAI. 2024. GPT-4 technical report. OpenAI. 2024. OpenAI o1 system car...

work page 2024

[5] [8]

list questions

Measuring short-form factuality in large language models. Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. 2025. Browsecomp: A simple yet challenging benchmark for browsing agents. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi,...

work page 2020

[6] [11]

Answers: 15 Figure 18: Excerpt of the few-shot chain-of-thought prompt which includes intermediate answers (CoT+Ans)

Therefore Miller had written 15 plays between October 17, 1935 and October 17, 1955. Answers: 15 Figure 18: Excerpt of the few-shot chain-of-thought prompt which includes intermediate answers (CoT+Ans)

work page 1935

[7] [2016]

ArXiv, abs/1611.09268

MS MARCO: A human generated MA- chine Reading COmprehension dataset. ArXiv, abs/1611.09268. Bernd Bohnet, Vinh Q. Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, Tom Kwiatkowski, Ji Ma, Jianmo Ni, Tal Schuster, William W. Cohen, Michael Collins, Dipanjan Das, Donald Met- zler, ...

arXiv 2022

[8] [2021]

In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)

BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multihop questions via single-hop question composition. Transaction...

work page 2022