{"id":"449ed96d-b8f5-4503-bcdc-819ab1e224e4","arxiv_id":"2508.11133","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MoNaCo provides 1,315 human-written multi-step questions needing dozens to hundreds of Wikipedia pages, and shows that frontier LLMs answer at most 61.2% F1.","lead":"MoNaCo is a benchmark of 1,315 natural, time-consuming questions whose answers require combining evidence from dozens to hundreds of Wikipedia pages, far more than existing QA benchmarks. It is built with a new decomposed annotation pipeline, and frontier LLMs reach at most 61.2% F1, making it a resource for tracking agentic information-seeking systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Factoid gold answers are accepted only when a worker's answer matches GPT-4O's own output, so any systematic GPT-4O bias is baked into the benchmark and propagates to every reported F1 score; the final manual review of aggregate answers limits but does not remove this risk.","rationale":"The reader's weakest_assumption is exactly the concern I identify: the factoid gold answers are validated against GPT-4O, so a systematic GPT-4O bias would corrupt the benchmark. I agree this is the most load-bearing risk because it sits upstream of every reported number; if the gold answers are wrong or stylistically biased, the F1 scores cease to be meaningful measures of model capability. The final manual review of the 1,315 aggregate answers is a real mitigation, but it is not a complete one: the manual review is plausibility-oriented and cannot reliably detect a single wrong intermediate fact that shifts an aggregated result. List questions are human-validated by overlap and are less exposed to this bias, but factoid questions are numerous and feed into the same gold chains. I considered other potential concerns, such as the decomposition granularity inflating the reported page counts and the lack of human calibration for the GPT-4.1 judge, but the gold-answer anchor is more fundamental: a page-count inflation would weaken the novelty claim, while a gold-answer corruption would invalidate the empirical results. The proposed audit test, using independent human re-annotation on a sample, would settle whether the concern actually lands. Since the paper's verdict was already CONDITIONAL, and this concern is addressable by such an audit, I recommend no change to the verdict.","tokens_in":24679,"tokens_out":10763,"duration_ms":115189,"concrete_test":"Independently re-annotate a random sample of 200 factoid intermediate questions from the released benchmark using two fresh crowd workers with Wikipedia access, blinded to the existing gold answers and to GPT-4O outputs; require the two workers to agree, then measure agreement between this independent gold set and the released gold answers. Also have one expert annotator re-answer 50 complex questions end-to-end from Wikipedia without the provided decompositions and compare final answers. If independent agreement on factoid answers is below 95%, or expert final-answer agreement is below 90%, the gold standard is not independent of GPT-4O and the benchmark's scores are suspect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's validity depends on the gold answers being correct. Section 3.3 describes factoid QA validation: a worker answer is accepted only when GPT-4O judges it to agree with GPT-4O's own answer; disagreements trigger re-annotation until agreement. This design actively selects for GPT-4O's factual beliefs and even its answer formatting (e.g., dates as '23 years' vs. '1982–2005'). With 42,099 non-list, non-boolean intermediate answers, a systematic GPT-4O error on any factoid category would be propagated into the gold reasoning chains. The authors manually review all 1,315 final aggregate answers, which catches many errors, but an incorrect intermediate fact can change an aggregated output in a non-obvious way (a wrong denominator, a wrong set member, a wrong date range) and survive plausibility review. Since every reported result—Table 4's F1 scores, the 61.2% headline, the oracle and BM25 comparisons—is scored against these gold answers, the central empirical claim inherits GPT-4O's blind spots. This is the most load-bearing assumption because it is upstream of both the benchmark's correctness and all model evaluations.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MoNaCo, a benchmark of 1,315 human-written, time-consuming questions whose gold solutions span dozens of Wikipedia pages (mean 43.3, median 12). The authors construct the benchmark with a decomposed annotation pipeline: crowd workers write persona-driven questions, trained workers annotate QDMR-style decompositions, and the execution engine derives intermediate questions that are answered by crowd workers with page/section/sentence evidence. Validation uses GPT-4O for factoid answer agreement, human overlap thresholds for list answers, and manual author review of all final aggregate answers. The paper evaluates 15 LLMs in closed-book, oracle-retrieval, and BM25-retrieval settings, reporting a top F1 of 61.2% (O3), with low recall and hallucinations as the main failure modes. A user study with 18 graduate students is used to argue that MoNaCo questions are more natural than machine-generated complex QA benchmarks while being far more complex than existing natural benchmarks. The benchmark, code, prompts, and predictions are publicly released.","tokens_in":24926,"tokens_out":5964,"duration_ms":65491,"significance":"If the gold answers and the LLM-judge evaluation are sound, MoNaCo fills a genuine gap: existing benchmarks either use natural but simple questions or complex but artificial ones, whereas MoNaCo combines human-written natural questions with evidence that spans dozens of pages. The resource itself is substantial: 90,773 intermediate questions, 36,194 evidence pages, list/boolean/factoid and table reasoning, and public releases of predictions and judge scores. The paper's complexity statistics (Table 3) and the multi-setting evaluation (closed-book, oracle, RAG) provide a useful testbed for agentic and deep-research systems. The explicit manual review of all 1,315 final answers and the human-overlap validation for list questions are notable strengths. The main risk is the dependence of the gold answers and the final scores on GPT-4O-family models, which this report treats as the central validity concern.","major_comments":[{"comment":"The acceptance criterion for factoid intermediate answers requires a crowdworker's answer to be judged by GPT-4O as agreeing with GPT-4O's own answer; disagreements are re-annotated until agreement. With 42,099 non-list, non-boolean intermediate answers (§4.1), this protocol effectively conditions the gold reasoning chains on GPT-4O's factual beliefs and answer formatting. A systematic GPT-4O error on any factoid category would propagate into the aggregated final answers, and the manual review of the 1,315 final answers cannot reliably detect an incorrect intermediate value that still yields a plausible aggregate (e.g., a wrong denominator or a wrong date range). Since every reported score in Table 4, Figure 7, and Figure 9 is computed against these gold answers, this is a load-bearing validity risk. Please add an independent human review of a random sample of factoid intermediate answers and report the agreement rate, and report how often the worker's original answer was overridden by the GPT-4O agreement criterion.","section":"§3.3, Factoid QA Validation"},{"comment":"The evaluation uses GPT-4.1 as an LLM judge to decide answer equivalence and to extract precision/recall, but no human-agreement calibration is reported. The judge comes from the same model family as the GPT-4O used to construct the gold answers, so model predictions that resemble GPT-4O's answer style may be systematically favored. The limitation statement in §8 acknowledges possible judge mistakes but does not quantify them. Please provide a human-judge agreement study on a sample of at least 100-200 predictions and report judge error rates by answer type (factoid, list, boolean, numeric). Without such calibration, the headline '61.2% F1' is not known to be a faithful measurement of model performance on these gold answers.","section":"§5.1, LLM-as-judge evaluation"},{"comment":"The complexity comparison relies on GPT-4O-generated decompositions for all non-MoNaCo benchmarks, while MoNaCo uses human-annotated decompositions. If GPT-4O under-decomposes the other benchmarks, the reported differences (5.1 vs. 3.2 reasoning steps; 38% vs. at most 12% aggregation) could overstate MoNaCo's relative complexity. This does not affect the #Pages evidence, but it does affect the 'more complex than all existing benchmarks' claim. Please validate the generated decompositions on a sample (e.g., human annotation of 100 questions per benchmark) or explicitly characterize how the comparison depends on this asymmetry.","section":"§4.2, Table 3"},{"comment":"No confidence intervals, variance estimates, or multiple runs are reported for any F1 score. With 1,315 questions, differences among top models (e.g., 61.18 vs. 60.11 for O3 vs. GPT-5) are within likely sampling noise, and the single-run nature also affects the 61.2% headline. Please report bootstrap confidence intervals over questions for the main table and state how many runs the numbers are based on. This is important for the benchmark's stated purpose of tracking progress over time.","section":"Table 4 and §5.2"}],"minor_comments":[{"comment":"Please clarify how F1 is computed for single numeric answers when the judge outputs a normalized similarity score rather than a binary precision label; the text says precision and recall are computed automatically, but the formula in Figure 12 only defines a precision-like value and the recall aggregation over factoid answers is not specified.","section":"§5.1, Figure 12"},{"comment":"The 'naturalness' score on the x-axis of Figure 1 is described as a weighted average of two measures, but the weighting formula is not defined; please state the exact normalization and weights so the reader can reproduce the figure.","section":"§4.3, Figure 1"},{"comment":"The benchmark name is typeset inconsistently as 'MoNaCo', 'MONACO', and 'M ONACO'; please choose one spelling and use it consistently in text, tables, and figure captions.","section":"Throughout"},{"comment":"The claim that automatic execution of operator steps 'leaves no room for potential calculation errors by humans' would be stronger if the executor's implementations of the 31 operators in Table 7 were unit-tested or validated against a small manually computed set; please state whether such tests were performed.","section":"Appendix A.1"}],"recommendation":"major_revision","confidential_remarks":"This is a solid benchmark contribution with real practical value, and the central claims are likely defensible after additional validation. The main concern is the extent to which the gold answers and the judge are both anchored to GPT-4O-family models; I would advise the editor to require the authors to provide human-agreement evidence before acceptance. The comparison with other benchmarks via GPT-4O-generated decompositions is a secondary but real concern, as is the absence of confidence intervals. I do not see grounds for rejection; these are fixable within the paper's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"MoNaCo is worth reading and worth taking seriously. The contribution is real: it is the first QA benchmark I know of that pairs human-written, natural information-seeking questions with evidence that typically spans dozens of Wikipedia pages (43.3 on average), and the pipeline that gets there—persona-based question elicitation plus QDMR decomposition plus distributed crowd answering—is a practical way to scale annotation of genuinely time-consuming questions. The complexity comparisons against eight benchmarks are persuasive, and the user study on naturalness, small as it is, supports the claim that these are not machine-generated trivia. The release of 1,315 questions with 90K intermediate answers, evidence, prompts, and predictions makes it immediately usable.\n\nI have one real reservation and two smaller ones. The real one: factoid intermediate answers are accepted when a worker's answer agrees with GPT-4O's own answer (Section 3.3), so a systematic GPT-4O blind spot could be baked into the gold chains. The stress-test note is right that this is upstream of every reported F1. It is mitigated, not eliminated, by manual review of all 1,315 final aggregate answers and by human overlap for list questions. I do not think it invalidates the central claim—the model-vs-model gaps and the low-recall pattern are too large to be an artifact—but a benchmark whose gold answers are partly model-anchored should report a human-agreement calibration on a sample and say how many final-answer corrections the manual review produced. The smaller issues: GPT-4.1 is used as judge without any human-agreement study, none of the F1 scores come with confidence intervals, and there is no human ceiling. The last point is understandable given how time-consuming the questions are, but the absence makes \"60% F1\" harder to interpret as a capability gap; even a small expert sample would help.\n\nCitation patterns look fine; the QDMR self-citations are appropriate and the related-work coverage is fair. One framing caveat: these questions are crowd-written with persona prompts, not harvested from real user logs; the paper is honest about this, and the user study at least measures perceived naturalness.\n\nWho this is for: anyone evaluating long-context reasoning, agentic search, or retrieval-augmented QA. It deserves serious peer review. I would ask for the calibration, confidence intervals, and a human ceiling sample before accepting, but I would not desk-reject.","headline":"A benchmark that fills a real gap; the GPT-4O anchoring of factoid gold answers is a genuine caveat, but the central finding—frontier LLMs are far from solving these broad multi-document questions—holds up.","tokens_in":25503,"tokens_out":2203,"would_cite":true,"duration_ms":26517,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MoNaCo: the first benchmark to pair natural, human-written questions with answers that span dozens to hundreds of evidence pages, and the paper reports that frontier LLMs still cap out at 61.2% F1.","keywords":["MoNaCo","question answering benchmark","multi-document reasoning","information-seeking","LLM evaluation","question decomposition","retrieval-augmented generation","list questions"],"falsifier":"Take a random sample of MoNaCo questions, have independent expert annotators (no GPT-4O, no authors) re-answer them from Wikipedia and compare to the gold answers; if the expert-gold agreement is substantially below ordinary human annotation agreement, the benchmark's scores are not measuring what they appear to measure.","tokens_in":24476,"feed_emoji":"📚","tokens_out":5988,"duration_ms":54120,"temperature":0.7,"pith_summary":"MoNaCo is built around a gap in existing QA benchmarks: nobody had collected natural, human-written questions whose answers genuinely take hours of research across dozens or hundreds of pages. The paper claims it has closed that gap with 1,315 questions averaging 43.3 evidence pages each, annotated with full reasoning chains. It then reports that 15 frontier LLMs score at most 61.2% F1, with the best model perfect on only 38.7% of questions, and that errors are dominated by low recall and hallucination. A sympathetic reader should take away that broad, time-consuming information-seeking remains unsolved by current LLM agents, and that MoNaCo gives the field a way to measure progress.","feed_headline":"Top LLMs score just 61% on MoNaCo's 43-page questions","feed_subtitle":"Benchmark questions span 43 pages on average; even the best model leaves a wide gap.","key_machinery":"The load-bearing mechanism is a decomposed annotation pipeline built on QDMR question decompositions. Workers first write time-consuming questions from personas, then expert annotators decompose each question into intermediate steps, and a 31-operator execution engine automatically derives follow-up questions and runs aggregation operations (sort, top-n, group, set operations, arithmetic) while crowdworkers answer the remaining factoid and list subquestions. This decomposition is what makes it possible to annotate gold answers spanning hundreds of pages without any single annotator seeing the whole task, and the same decompositions let the authors compute complexity statistics and reasoning-template diversity.","core_discovery":"The paper's central claim is that MoNaCo is the first QA benchmark to combine genuinely natural, human-written questions with solutions that span dozens to hundreds of evidence documents. Each of the 1,315 questions comes with a gold QDMR-style decomposition into intermediate steps, 90,773 intermediate question-answer units, and evidence drawn from 36,194 distinct Wikipedia pages, with an average of 43.3 pages and 5.1 reasoning steps per question. Evaluated in a closed-book setting, the best frontier model reaches 61.2% F1; even with all gold evidence supplied as context, two strong models score only about 59% F1, and adding BM25 retrieval drops performance by over 12 points. The paper interprets these results as showing that aggregating and reasoning over dozens of facts is an open challenge for LLM-powered agents.","pith_inferences":["Beyond the paper's claims, MoNaCo's decomposition-and-executor machinery could be turned around to generate fresh benchmark questions automatically from Wikipedia snapshots, making contamination harder to manage.","The 16.1% of questions whose answers change yearly imply that a static leaderboard will decay in interpretability; using only the 49.8% time-independent subset for head-to-head model comparisons would give more stable rankings.","The hybrid factoid validation makes GPT-4O a de facto oracle for part of the gold standard; a manual re-annotation study on a sample would quantify how much of the reported gap is model failure versus benchmark noise.","Because MoNaCo contains 40,125 boolean and 8,549 list subquestions, it can double as a fine-grained diagnostic for the two subskills (exhaustive enumeration and yes/no verification) that broader agent benchmarks rarely isolate."],"forward_implications":["Frontier LLM agents are not close to solving broad, human-time-scale information-seeking tasks; the best model's 61.2% F1 and 38.7% perfect-answer rate leave large headroom.","Even when all gold evidence is placed in the prompt, strong models score only about 59% F1, so long-context reasoning over dozens of documents is itself unsolved.","Standard RAG can hurt: BM25 retrieval lowers performance by more than 12 F1 points, indicating current LLMs are not robust to even partially irrelevant retrieved context.","List questions are a major bottleneck: model recall drops sharply as the number of expected answers grows past roughly 20, so improving exhaustive set recall is a concrete target.","MoNaCo's per-question time-dependence labels (about half stable) and released answer timestamps make it usable as a longitudinal leaderboard for agentic search systems."],"supporting_citations":[{"why":"Supplies QDMR, the decomposition formalism the entire annotation pipeline relies on.","marker":"Wolfson et al. (2020)"},{"why":"Natural Questions, the natural-questions benchmark MoNaCo compares against for naturalness.","marker":"Kwiatkowski et al. (2019)"},{"why":"HotpotQA, the multi-hop benchmark used as a complexity comparison point.","marker":"Yang et al. (2018)"},{"why":"MuSiQue, the machine-generated complex benchmark contrast.","marker":"Trivedi et al. (2022)"},{"why":"QAMPARI, the list-QA benchmark that MoNaCo extends and compares with.","marker":"Amouyal et al. (2023)"},{"why":"FanoutQA, a multi-document benchmark in the naturalness comparison and related work.","marker":"Zhu et al. (2024)"},{"why":"Supplies the 'retrieval robustness' notion used to explain RAG's performance drop.","marker":"Yoran et al. (2024b)"},{"why":"The LLM-as-judge evaluation prompt MoNaCo adapts for scoring.","marker":"Phan et al. (2025)"}],"fun_headline_variants":["1,315 questions, avg 43 pages, top LLM only 61% F1","Best LLM scores 61% F1 on questions needing 43 docs on average","MoNaCo benchmark: 1,315 questions, 90k steps, best model 61% F1","LLM agents fail on real questions that need 43+ docs to answer","New QA benchmark: answers require 43 pages of evidence, LLMs at 61%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's gold answers are assumed to be correct, but factoid answers were accepted only when a crowdworker's answer agreed with GPT-4O's answer, so a systematic bias in GPT-4O's factual beliefs would be inherited by every gold answer and every reported F1 score.","fun_headline_variants_meta":{"raw":{"variants":["1,315 questions, avg 43 pages, top LLM only 61% F1","Best LLM scores 61% F1 on questions needing 43 docs on average","MoNaCo benchmark: 1,315 questions, 90k steps, best model 61% F1","LLM agents fail on real questions that need 43+ docs to answer","New QA benchmark: answers require 43 pages of evidence, LLMs at 61%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000785,"raw_usage":{"total_tokens":3447,"prompt_tokens":907,"completion_tokens":2540,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":2422}},"tokens_in":523,"tokens_out":2540,"duration_ms":16681,"temperature":1.0,"reasoning_tokens":2422,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:28:08.135531+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of MoNaCo questions, have independent expert annotators (no GPT-4O, no authors) re-answer them from Wikipedia and compare to the gold answers; if the expert-gold agreement is substantially below ordinary human annotation agreement, the benchmark's scores are not measuring what they appear to measure.","supporting_citations":[],"review_version":2}