{"id":"8f7b64b0-81ed-4e02-be4a-b97fb9196c87","arxiv_id":"2505.00263","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new 528,304-question benchmark over 103,638 Enron emails across 150 inboxes lets RAG pipelines be evaluated in a setting where LLMs cannot answer from memory.","lead":"The paper releases EnronQA, a question-answering benchmark built from more than 100,000 Enron emails with over 500,000 questions tied to 150 user inboxes. It is meant to test RAG systems on private-style documents, and it shows that on this benchmark, better retrieval directly improves answers, unlike Wikipedia-based benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The calibration result is partly enforced by the groundedness filter in §3.2, and the QA filters that certify gold answers are validated on only 41–200 human examples using the same model family that generated the questions.","rationale":"The reader's weakest assumption identifies the same load-bearing point: the four LLM-based checks are the only gatekeeping between raw emails and a claimed 528k high-quality QA pairs, and the human validation is very small relative to the dataset. I add the observation that the groundedness filter makes the headline calibration comparison partly by construction, but this is not itself a fatal flaw; for a RAG benchmark, excluding questions that models can answer without retrieval is a legitimate design choice. The benchmark is likely a useful resource, the filtering pipeline is described in sufficient detail to audit, and the authors explicitly disclose the sizes of their human evaluations. A conditional verdict remains appropriate: the paper should add independent human validation of final QA pairs, release the pipeline code, and clarify how much of the calibration result follows from the groundedness filter. I do not see grounds for rejection, and the concern does not move the verdict beyond conditional.","tokens_in":28885,"tokens_out":6729,"duration_ms":73349,"concrete_test":"Independently sample 300 Dev/Test QA pairs from the released dataset, and have two annotators not involved with the paper verify, for each item, that the gold answer is correct and fully supported by the labeled email and that the question is specific enough to identify that email among similar ones. Report the per-item gold-answer pass rate and Cohen's kappa. If the pass rate is below roughly 90% or kappa is below 0.6, the filtering pipeline is too lenient and the calibration curves in §4.2 should be recomputed after excluding the failed items.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central calibration claim in §4.2 ('EnronQA is the only benchmark where adding context is always better') depends on the groundedness check in §3.2: a question is admitted only if both Llama-3.1-70B and Mixtral-8x7B, given no email, fail to reproduce the gold answer. Low no-context accuracy is therefore partly a property of the filter, not an independent discovery about private documents; this is acceptable for a RAG benchmark, but it weakens the comparison to Wikipedia benchmarks that did not filter out memorized questions. More importantly, the same pipeline certifies correctness: gold answers are Llama-3.1-70B outputs accepted by the objective check, and the LLM judge's 0.98 F1 was measured on 200 match/no-match judgements, not on whether answers are factually correct. The quality ruleset was tuned on 41 human-labeled questions, with author inter-annotator Spearman correlation of 0.5. If the judge accepts same-family, self-consistent but wrong answers, or if some questions are unanswerable, the benchmark's quality and its low no-context baseline are misleading. The paper is not internally inconsistent, but the evidence for 'high-quality, correctly labeled' QA pairs is thin.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EnronQA, a synthetic question-answering benchmark built from the cleaned Enron email corpus, containing 103,638 emails and 528,304 QA pairs across 150 user inboxes. The dataset is constructed through a DSPy-based pipeline that generates questions with Llama-3.1-70B and filters them through four LLM-based checks: specificity, objectivity, groundedness, and a rule-based quality judge. The authors present dataset statistics, a calibration study comparing EnronQA to NaturalQuestions and TriviaQA, baseline RAG experiments with BM25 and ColBERTv2 over several LLMs, and a case study on LoRA-based memorization of facts. The paper's central claims are that EnronQA is the only evaluated benchmark where adding retrieved context always improves over a no-context baseline, and that it provides a better-calibrated testbed for retriever quality than Wikipedia-based benchmarks.","tokens_in":29170,"tokens_out":4238,"duration_ms":39866,"significance":"If the quality of the 528k synthetic QA pairs is adequately validated, EnronQA would be a valuable community resource: it is large, segmented by user, drawn from realistic private emails, and released with prompts and artifacts. The calibration result, if it holds under independent evaluation, would be useful for the RAG community because it addresses the memorization problem that afflicts Wikipedia-based benchmarks. The LoRA memorization study is also a relevant initial exploration. The paper is transparent about its pipeline and releases substantial supporting material, including prompts and chain-of-thought outputs, which strengthens reproducibility. The main weaknesses are that the evidence for question quality is thin (41 human-labeled-quality examples and 200 judge agreement examples), the evaluation stack is heavily concentrated in one model family, and the headline calibration claim is partly enforced by construction rather than observed independently.","major_comments":[{"comment":"Table 3 reports an average of 491.81 emails per user, but 103,638 emails divided by 150 users equals 690.92. This arithmetic inconsistency means the summary statistics as reported cannot be trusted until the table is corrected or the discrepancy is explained. Since the size and composition of the benchmark are central to the paper's claims, this needs to be fixed before publication.","section":"§4.1, Table 3"},{"comment":"The headline calibration claim that 'EnronQA is the only benchmark where adding context is always better than the no-context baseline' is partly enforced by the groundedness filter in §3.2: a question is admitted only when both Llama-3.1-70B and Mixtral-8x7B fail to reproduce the gold answer without the email. The no-context accuracy below 5% is therefore a property of the benchmark's construction, not an independent discovery about private documents. The comparison to NaturalQuestions and TriviaQA, which were not filtered this way, should be framed accordingly, and the authors should report calibration on an unfiltered or differently filtered subset to support the stronger interpretation.","section":"§4.2 vs. §3.2"},{"comment":"The validation of question quality is too thin for a resource of 528k pairs. The LLM judge's 0.98 F1 was measured on only 200 generated match/no-match judgements, and this measures answer-equivalence, not whether a gold answer is factually correct or whether a question is answerable. The quality ruleset was tuned on 21 development questions and tested on 20, with an inter-annotator Spearman correlation of only 0.5. Because the same Llama-3.1-70B family generates questions, filters them, and scores final accuracy, there is a real risk of self-consistency bias. The authors should provide independent human evaluation on a larger random sample, report agreement statistics, and check for unanswerable or ambiguous questions before presenting EnronQA as 'high quality.'","section":"§3.2, §4.2, §5.2"},{"comment":"The claim that LoRA memorization 'matches long-context performance at almost all scales' is not well supported by Table 5. At 5,000 facts the rank-1024 adapter collapses to 0.03 accuracy, and at 20,000 facts ranks 512, 1024, and 2048 yield 0.08, 0.00, and 0.03, respectively. Several cells show non-monotonic behavior across ranks, and no error bars, multiple seeds, or standard deviations are reported. This makes the conclusion about 'surprising capacity' fragile and should be either supported with variance estimates or substantially tempered.","section":"§6, Table 5"}],"minor_comments":[{"comment":"The text contains an unresolved placeholder '(??)' in the sentence about headroom for improving retriever quality; please replace it with the intended citation or number.","section":"§1"},{"comment":"The caption refers to 'Mixtral-7B-Instruct' while the text and reference correctly identify the model as Mixtral-8x7B-Instruct; please correct the model name.","section":"Fig. 2 caption"},{"comment":"There is a typo 'GPTo' that should read 'GPT-4o' or 'GPT4o' consistently.","section":"§5.2"},{"comment":"The column headers in Table 4 are difficult to parse because 'R@5' appears only under 'Query Rewrite' and the row labels mix retriever names with recall values. Please restructure the table so that each retriever's Recall@5 is clearly associated with the right column.","section":"Table 4"},{"comment":"The introductory sentence in B.7 is duplicated from B.5 and says 'These prompts are used to both answer the question given the context of an email or to produce an answer to the question with no grounding,' which does not describe the quality-evaluation prompt in B.7; please correct it.","section":"Appendix B.7"},{"comment":"The memorization experiment reports '10 epochs with rate 1e-4' but does not specify the optimizer, learning-rate schedule, or how accuracy is computed from the LLM judge's outputs; please add these details.","section":"§6.1"}],"recommendation":"major_revision","confidential_remarks":"The EnronQA resource is timely and likely to be useful, but the current evidence for the quality of the 528k synthetic QA pairs and the uniqueness of the calibration result is not yet convincing. The arithmetic inconsistency in Table 3 and the fragility of the memorization results in Table 5 are concrete issues that the authors can fix. The circularity concern raised by the second reader is legitimate and should be addressed with independent human evaluation and at least one evaluation using a different model family. I do not see a fatal flaw, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a genuinely useful resource and the authors did serious work, but the calibration finding is oversold and the human validation is thin. I'd send it to review with a request for fixes.\n\nWhat's new: EnronQA gives you 103k emails across 150 inboxes with 528k single-hop QA pairs, roughly 30x the QA count of ConcurrentQA. That scale, plus the private-document framing, makes it a real addition to RAG benchmarking. The filtering pipeline (minhash, subset dedup, quality filters) is sensible, and the DSPy-based generation with explicit specificity/objectivity/groundedness checks is a reasonable way to make synthetic QA at scale. The LoRA vs RAG vs long-context case study is a nice extra, though the results are unsurprising.\n\nThe soft spots are real but mostly addressable. First, the headline calibration result—'the only benchmark where adding context is always better'—is partly enforced by the groundedness filter: questions are kept only if two LLMs fail to reproduce the gold answer without the email. So low no-context accuracy is by construction, not a discovery about Enron emails. That's fine for a RAG benchmark, but the comparison to Wikipedia benchmarks is unfair, since those weren't filtered for memorization. The claim should be softened.\n\nSecond, Table 3 has an outright arithmetic error: 103,638 emails over 150 users is 690.9 per user, not 491.81. That suggests a different denominator; the authors need to fix or explain it.\n\nThird, the validation of the synthetic pipeline is thin: the quality ruleset was tuned on 41 human-labeled questions with inter-annotator Spearman 0.5, and the LLM judge's 0.98 F1 comes from 200 examples. The same model family (Llama-3.1-70B) generates, filters, and evaluates. That doesn't make the benchmark worthless, but it means the 'high quality' claim isn't independently confirmed. Authors should release the pipeline code and get a few hundred human validation labels on the final dataset.\n\nThe 'private' framing is a bit loose—the Enron corpus is public—but the authors are clear that they mean unmemorized, personal documents. I don't think that's a flaw worth holding against them.\n\nWho's this for? Anyone building retrievers or RAG pipelines for private/personalized settings, and people working on continued pretraining or memorization. Worth citing as a resource. Deserves a serious referee; with the statistical fix and softened claims it can be a solid benchmark paper.","headline":"Large, useful private-document RAG benchmark; the calibration claim is partly self-enforcing and human validation is thin, but the resource deserves review.","tokens_in":29674,"tokens_out":2494,"would_cite":true,"duration_ms":24105,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EnronQA, a benchmark of 103,638 private emails and 528,304 question-answer pairs across 150 inboxes, is calibrated so end-to-end RAG accuracy tracks retrieval quality rather than LLM memorization, with no-context accuracy below 5% and…","keywords":["EnronQA","retrieval-augmented generation","private-document benchmark","question-answer generation","retrieval calibration","personalized retrieval","memorization","LoRA"],"falsifier":"Have independent human annotators answer a random sample of roughly 500 EnronQA questions with no email in context; if more than a small percentage are answered correctly, the groundedness check is too lenient and the benchmark would not track retrieval quality. The same test can be rerun on each new generation of large language models, since the no-context baseline rising above 5% would signal that the corpus has leaked into training.","tokens_in":28683,"feed_emoji":"📧","tokens_out":8827,"duration_ms":84061,"temperature":0.7,"pith_summary":"EnronQA turns the cleaned Enron email corpus into a benchmark for retrieval-augmented generation over private documents: 103,638 emails, 528,304 synthetic question-answer pairs, and 150 separate user inboxes. The paper's central claim is that this benchmark is properly calibrated: without any retrieved context, a large language model scores below 5%, and every additional point of retrieval recall adds roughly 0.6 points of answer accuracy. That contrasts with Wikipedia-based benchmarks such as NaturalQuestions and TriviaQA, where a model can often answer from memorized knowledge alone and a retriever must be quite good before it helps at all. If the calibration claim holds, EnronQA lets practitioners read end-to-end RAG accuracy as a direct measure of retriever quality, and it provides a test bed for personalized retrieval over realistic per-user document stores.","feed_headline":"A 528k-question email benchmark makes RAG scores track retrieval","feed_subtitle":"Accuracy on this 103,638-email corpus reflects retrieval quality, not LLM memorization.","key_machinery":"The paper's working parts are the four-gate question admission pipeline and the per-inbox segmentation of the corpus. A question enters the benchmark only if it is specific, objective, grounded, and high quality by automated checks; this is what keeps no-context accuracy near zero and makes end-to-end accuracy move with retrieval recall. The 150 distinct inboxes are what let the same corpus support personalized retrieval experiments rather than only global retrieval.","core_discovery":"EnronQA is constructed by filtering the cleaned Enron email corpus down to 103,638 usable emails and generating 528,304 QA pairs with a multi-stage LLM pipeline. Each question must pass four automated checks before entering the benchmark: specificity, objectivity, groundedness, and a rules-based quality gate. The central discovery is about calibration: in simulated Recall@1 sweeps, EnronQA is the only benchmark tested where adding the correct context always improves accuracy over the no-context baseline, with no-context accuracy below 5% and nearly a 0.6% accuracy gain per 1% recall gain. The paper also reports baseline RAG results and a memorization case study in which LoRA adapters trained on up to 20,000 facts match long-context performance, while retrieval still outperforms both.","pith_inferences":["A consequence the paper leaves implicit: the specificity gate likely pushes questions toward distinctive named entities, which would explain the strong lexical-retrieval results; re-generating questions with paraphrased, entity-light wording would test whether dense retrieval closes the gap.","The calibration claim is time-sensitive; as future large language models train on broader data, the Enron corpus may leak into training, so re-running the no-context baseline on each new model generation is a simple contamination monitor.","The memorization results suggest an open capacity-versus-cost tradeoff; because LoRA adapters hold up to 20,000 facts, a natural next experiment is measuring at what fact count a large-rank adapter becomes cheaper or faster than maintaining a search index."],"forward_implications":["End-to-end accuracy on EnronQA can be read as a direct measure of retriever quality, since every point of recall adds roughly 0.6 points of accuracy.","Private-document RAG pipelines can be benchmarked without first controlling for whether the LLM already memorized the documents, which is not possible on Wikipedia-based benchmarks.","The benchmark's 150-inbox structure enables personalized retrieval evaluation, where systems must find and reason over the right user's documents.","The memorization case study shows LoRA adapters can recall up to 20,000 facts at a level comparable to putting the facts in context, while retrieval still outperforms both.","The benchmark is large enough for fine-tuning and continued-pretraining experiments, not just for evaluation."],"supporting_citations":[{"why":"Supplies the cleaned 2015 Enron email corpus that EnronQA filters into 103,638 documents.","marker":"[17]"},{"why":"Defines the related private-document RAG benchmark whose Enron emails are mapped back into EnronQA for compatibility.","marker":"[4]"},{"why":"One of the Wikipedia-based benchmarks used in the calibration comparison to show the memorization confound.","marker":"[41]"},{"why":"The trivia benchmark whose high no-context accuracy illustrates when retrieval gains are masked by memorized knowledge.","marker":"[32]"},{"why":"The LLM family used to generate, refine, judge, and answer questions in the pipeline and calibration experiments.","marker":"[15]"},{"why":"The second LLM family used in the objectivity and groundedness checks to require cross-model agreement.","marker":"[30]"},{"why":"The dense bi-encoder retriever used to mine the hard negative emails for the specificity check.","marker":"[50]"},{"why":"The dense retriever used in the RAG baselines and in the memorization case-study retrieval.","marker":"[64]"}],"fun_headline_variants":["Email benchmark ties RAG accuracy to retrieval, not memory","EnronQA: 528k private-email questions reveal retrieval's role","RAG scores on EnronQA climb with recall, not memorization","103k emails, 528k QA: benchmark isolates retrieval impact","Private-email QA set shows RAG needs context, not just tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the four automated checks performed by large language models correctly certify question quality, so the 528,304 question-answer pairs are genuinely answerable from their paired email and not from general knowledge.","fun_headline_variants_meta":{"raw":{"variants":["Email benchmark ties RAG accuracy to retrieval, not memory","EnronQA: 528k private-email questions reveal retrieval's role","RAG scores on EnronQA climb with recall, not memorization","103k emails, 528k QA: benchmark isolates retrieval impact","Private-email QA set shows RAG needs context, not just tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00022,"raw_usage":{"total_tokens":1427,"prompt_tokens":908,"completion_tokens":519,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":426}},"tokens_in":524,"tokens_out":519,"duration_ms":5282,"temperature":1.0,"reasoning_tokens":426,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:46:47.862847+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent human annotators answer a random sample of roughly 500 EnronQA questions with no email in context; if more than a small percentage are answered correctly, the groundedness check is too lenient and the benchmark would not track retrieval quality. The same test can be rerun on each new generation of large language models, since the no-context baseline rising above 5% would signal that the corpus has leaked into training.","supporting_citations":[],"review_version":1}