pith. machine review for the scientific record. sign in

arxiv: 2604.20763 · v1 · submitted 2026-04-22 · 💻 cs.IR · cs.AI· cs.LG

Recognition: unknown

Coverage, Not Averages: Semantic Stratification for Trustworthy Retrieval Evaluation

Andrew Klearman , Radu Revutchi , Rohin Garg , Rishav Chakravarti , Samuel Marc Denton , Yuan Xue
Authors on Pith no claims yet

Pith reviewed 2026-05-09 23:06 UTC · model grok-4.3

classification 💻 cs.IR cs.AIcs.LG
keywords retrieval evaluationsemantic stratificationquery set biascoverage guaranteesentity-based clusteringRAG evaluationinformation retrievalmetric reliability
0
0 comments X

The pith

Retrieval evaluation metrics are unreliable unless query sets cover the full semantic structure of the corpus instead of averaging over incomplete samples.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper treats retrieval evaluation as a statistical estimation problem whose reliability is limited by how the query set is assembled. Standard benchmarks use heuristic collections that leave large semantic regions of the corpus untested, producing unstable and hard-to-interpret scores. Semantic stratification counters this by first grouping documents into clusters based on the entities they contain, then generating new queries to cover the missing clusters. The resulting evaluation supplies formal coverage guarantees and shows exactly which semantic regions cause retrieval failures. Experiments confirm that this yields more stable assessments and clearer signals for choosing retrieval methods in systems like RAG.

Core claim

Retrieval evaluation is a statistical estimation task whose accuracy is bounded by the semantic coverage of the query set. Semantic stratification solves the coverage problem by partitioning the corpus into an interpretable global space of entity-based document clusters and systematically generating queries for every missing stratum, thereby delivering formal semantic coverage guarantees across retrieval regimes together with interpretable visibility into failure modes.

What carries the argument

Semantic stratification, which organizes documents into entity-based clusters to create an interpretable global semantic space and generates queries for the missing strata.

If this is right

  • Stratified evaluation supplies formal semantic coverage guarantees that hold across different retrieval methods and regimes.
  • Cluster-level analysis identifies structural signals that explain observed variance in retrieval performance.
  • The new metrics are more stable than aggregate averages computed on heuristically chosen queries.
  • Decision-making about which retrieval method to use in RAG pipelines becomes more trustworthy because failure modes are localized.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Evaluation benchmarks could be generated automatically from any target corpus rather than relying on fixed external query collections.
  • The same stratification approach might be extended to track how coverage changes when the corpus is updated over time.
  • Similar coverage-based diagnostics could be applied to other stages of retrieval-augmented pipelines such as passage ranking or answer generation.

Load-bearing premise

That grouping documents by entities produces a complete, unbiased semantic space that captures all relevant retrieval variations and allows query generation without introducing new selection biases.

What would settle it

A large independently drawn sample of queries from the full corpus on which the performance ranking or variance obtained from the stratified set does not match the ranking or variance from the independent sample.

Figures

Figures reproduced from arXiv: 2604.20763 by Andrew Klearman, Radu Revutchi, Rishav Chakravarti, Rohin Garg, Samuel Marc Denton, Yuan Xue.

Figure 1
Figure 1. Figure 1: shows that underrepresented clusters exhibit lower mean nDCG@10 when evaluated, but contribute little to aggregate scores due to their low query frequency. Together, these effects induce a structural bias in benchmark evaluation with aggregate metrics dominated by overrepre￾sented semantic regions and masking failures in prevalent but sparsely evaluated domains [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Semantic structure of a document corpus constructed via entity-based clustering. Query stratification converts query text into a set of entities. Cluster assignments, J, and ∆ form a single query coverage vector. Semantic entity extraction. We extract semantic entities, which are atomic and interpretable semantic units, from each document using a LLM. Specifically, we prompt the model to identify salient e… view at source ↗
Figure 3
Figure 3. Figure 3: Retrieval performance (nDCG@10) across semantic clus￾ters for different retrievers. Each box shows per-query score distri￾bution within a cluster; dashed line indicates overall nDCG@10. these patterns hold across additional datasets. 5.3. Coverage Analysis We assess semantic space coverage using three metrics: Minimum Semantic Coverage (MSC): the fraction of se￾mantic clusters “touched” by at least one eva… view at source ↗
Figure 5
Figure 5. Figure 5: demonstrates this sensitivity through 1,000 boot￾strapped evaluation sets. Using overall mean, text￾embedding-3-small wins only 32.2% of samples; using macro-average (equal weight per cluster), the same model wins 63.3%. If a practitioner cares equally about all seman￾tic regions regardless of query volume, macro-averaging better reflects their intent. Stratified evaluation makes these trade-offs explicit … view at source ↗
Figure 6
Figure 6. Figure 6: visualizes the semantic clusters across the NFCorpus dataset [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: FiQA: Distribution of nDCG@10 scores across semantic clusters, showing per-cluster performance variance. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: SciDocs: Distribution of nDCG@10 scores across semantic clusters, showing per-cluster performance variance. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: FiQA: Per-query performance comparison between retrieval systems, colored by semantic cluster. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: SciDocs: Per-query performance comparison between retrieval systems, colored by semantic cluster. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: FiQA: Win rate comparison between retrieval systems across bootstrap samples (n=300) [PITH_FULL_IMAGE:figures/full_fig_p025_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: SciDocs: Win rate comparison between retrieval systems across bootstrap samples (n=300). 25 [PITH_FULL_IMAGE:figures/full_fig_p025_12.png] view at source ↗
read the original abstract

Retrieval quality is the primary bottleneck for accuracy and robustness in retrieval-augmented generation (RAG). Current evaluation relies on heuristically constructed query sets, which introduce a hidden intrinsic bias. We formalize retrieval evaluation as a statistical estimation problem, showing that metric reliability is fundamentally limited by the evaluation-set construction. We further introduce \emph{semantic stratification}, which grounds evaluation in corpus structure by organizing documents into an interpretable global space of entity-based clusters and systematically generating queries for missing strata. This yields (1) formal semantic coverage guarantees across retrieval regimes and (2) interpretable visibility into retrieval failure modes. Experiments across multiple benchmarks and retrieval methods validate our framework. The results expose systematic coverage gaps, identify structural signals that explain variance in retrieval performance, and show that stratified evaluation yields more stable and transparent assessments while supporting more trustworthy decision-making than aggregate metrics.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper claims that heuristic query sets introduce intrinsic bias in retrieval evaluation, formalizes evaluation as a statistical estimation problem whose reliability is limited by evaluation-set construction, and introduces semantic stratification: documents are organized into an interpretable global space of entity-based clusters, queries are systematically generated for missing strata, yielding formal semantic coverage guarantees across retrieval regimes plus interpretable visibility into failure modes. Experiments across multiple benchmarks and retrieval methods are said to validate the framework by exposing coverage gaps, identifying structural signals for performance variance, and showing more stable assessments than aggregate metrics.

Significance. If the entity-based clustering construction indeed supplies an interpretable global semantic space that supports systematic query generation and formal coverage guarantees without introducing new selection biases, the work would strengthen evaluation methodology in information retrieval and RAG by shifting from opaque averages to coverage-aware, diagnostically transparent protocols. The statistical-estimation framing and the emphasis on corpus-grounded strata are positive contributions that could support more trustworthy decision-making.

major comments (2)
  1. [Abstract / semantic stratification construction] Abstract and the description of semantic stratification: the central claim that entity-based clusters produce an interpretable global semantic space sufficient for formal coverage guarantees rests on the unexamined assumption that entity co-occurrence or similarity captures all relevant retrieval dimensions; non-entity factors (temporal ordering, causal relations, sentiment polarity, discourse structure) are not shown to align with the clusters, so strata labeled 'complete' may still omit critical query variations and the claimed bias reduction does not follow.
  2. [Abstract] Abstract: the statement that 'experiments across multiple benchmarks and retrieval methods validate our framework' is unsupported by any description of experimental design, baselines, statistical tests, or the concrete procedure used to compute coverage guarantees, rendering the empirical support for the central claim unverifiable from the provided text.
minor comments (1)
  1. [Abstract] The abstract introduces 'semantic strata' and 'entity-based clusters' without a brief definitional sentence, which reduces immediate clarity for readers unfamiliar with the approach.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed comments, which help clarify the scope and presentation of our work. We respond to each major comment below and indicate the revisions we will make to the manuscript.

read point-by-point responses

  1. Referee: [Abstract / semantic stratification construction] Abstract and the description of semantic stratification: the central claim that entity-based clusters produce an interpretable global semantic space sufficient for formal coverage guarantees rests on the unexamined assumption that entity co-occurrence or similarity captures all relevant retrieval dimensions; non-entity factors (temporal ordering, causal relations, sentiment polarity, discourse structure) are not shown to align with the clusters, so strata labeled 'complete' may still omit critical query variations and the claimed bias reduction does not follow.

    Authors: We agree that entity-based clustering does not exhaustively capture every retrieval dimension. The framework defines semantic strata and coverage guarantees explicitly with respect to the entity space, which we selected because entities provide an interpretable, corpus-grounded partitioning that aligns with many knowledge-intensive retrieval tasks. Experiments demonstrate that this already exposes substantial gaps in standard query sets. We will add a new subsection in the discussion that explicitly states the scope of the entity-based approach, lists non-entity factors as orthogonal dimensions not covered by the current strata, and outlines how the stratification procedure could be extended (e.g., by adding temporal or sentiment-based partitions). This revision will make the assumptions and limitations transparent without altering the core technical contribution. revision: partial

  2. Referee: [Abstract] Abstract: the statement that 'experiments across multiple benchmarks and retrieval methods validate our framework' is unsupported by any description of experimental design, baselines, statistical tests, or the concrete procedure used to compute coverage guarantees, rendering the empirical support for the central claim unverifiable from the provided text.

    Authors: The abstract is a high-level summary; the full experimental design, benchmark details, retrieval methods, statistical tests, and exact procedure for computing coverage guarantees appear in Sections 4 and 5 of the manuscript. Nevertheless, the current abstract phrasing is too terse and could mislead readers who read only the abstract. We will revise the abstract to qualify the validation statement (e.g., “Experiments across multiple benchmarks and retrieval methods expose systematic coverage gaps and demonstrate more stable assessments than aggregate metrics”) and add a brief parenthetical reference to the methodology sections. revision: yes

Circularity Check

1 steps flagged

Semantic coverage guarantees reduce to definitional property of the stratification

specific steps
  1. self definitional [Abstract]
    "organizing documents into an interpretable global space of entity-based clusters and systematically generating queries for missing strata. This yields (1) formal semantic coverage guarantees across retrieval regimes and (2) interpretable visibility into retrieval failure modes."

    The coverage guarantees are yielded directly by the act of organizing into clusters and generating queries for missing strata, making the guarantee hold by the definition of the strata rather than through an independent mathematical derivation or empirical validation separate from the construction.

full rationale

The paper's formalization of retrieval evaluation as a statistical estimation problem and its experimental results across benchmarks stand as independent content. However, the central yield of 'formal semantic coverage guarantees' is presented as following directly from the definition and construction of entity-based strata plus query generation for missing ones, without an external derivation or benchmark that would make the guarantee non-tautological. This is a self-definitional element in the load-bearing claim but does not collapse the entire argument.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The framework rests on the domain assumption that entity-based clusters adequately represent the semantic structure of the corpus for query generation and coverage assessment.

axioms (1)
  • domain assumption Entity-based clusters form an interpretable global semantic space that captures the relevant variations for systematic query generation across retrieval regimes.
    Invoked directly in the description of semantic stratification as grounding evaluation in corpus structure.
invented entities (1)
  • semantic strata no independent evidence
    purpose: To define missing coverage areas in the evaluation set for generating targeted queries.
    New concept introduced to operationalize coverage guarantees; no independent evidence provided beyond the framework itself.

pith-pipeline@v0.9.0 · 5465 in / 1256 out tokens · 27832 ms · 2026-05-09T23:06:21.618406+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 7 canonical work pages

  1. [1]

    Cormack, Charles L

    Accessed: 2026. Cormack, G. V ., Clarke, C. L. A., and Buettcher, S. Re- ciprocal rank fusion outperforms condorcet and indi- vidual rank learning methods. InProceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’09, pp. 758–759, New York, NY , USA, 2009. Associa- tion for Computing Machiner...

    work page doi:10.1145/1571941.1572114 2026

  2. [2]

    ISBN 9 Coverage, Not Averages Semantic Stratification for Trustworthy Retrieval Evaluation 979-8-89176-288-6

    Association for Computational Linguistics. ISBN 9 Coverage, Not Averages Semantic Stratification for Trustworthy Retrieval Evaluation 979-8-89176-288-6. doi: 10.18653/v1/2025.acl-industry

    work page doi:10.18653/v1/2025.acl-industry 2025

  3. [3]

    2024 , pages =

    URL https://aclanthology.org/2025. acl-industry.33/. Guti´errez, B. J., Shu, Y ., Gu, Y ., Yasunaga, M., and Su, Y . Hipporag: Neurobiologically inspired long-term memory for large language models. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.),Advances in Neural Information Pro- cessing Systems, volume ...

    work page doi:10.52202/079017-1902 2025

  4. [4]

    findings-emnlp.279/

    URL https://aclanthology.org/2024. findings-emnlp.279/. Rahman, M. M., Kutlu, M., Elsayed, T., and Lease, M. Effi- cient test collection construction via active learning. In Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval, ICTIR ’20, pp. 177–184, New York, NY , USA, 2020. Associa- tion for Computing Machine...

    work page doi:10.1145/3409256.3409837 2024

  5. [5]

    , booktitle =

    URL https://openreview.net/forum? id=t4eB3zYWBK. Teixeira de Lima, R., Gupta, S., Berrospi Ramis, C., Mishra, L., Dolfi, M., Staar, P., and Vagenas, P. Know your RAG: Dataset taxonomy and generation strategies for evaluating RAG systems. In Rambow, O., Wanner, L., Apidianaki, M., Al-Khalifa, H., Eugenio, B. D., Schock- aert, S., Darwish, K., and Agarwal, ...

    work page doi:10.18653/v1/d18-1259 2025

  6. [6]

    neural network,

    doi: 10.18653/v1/2025.acl-long.418. URL https: //aclanthology.org/2025.acl-long.418/. 11 Coverage, Not Averages Semantic Stratification for Trustworthy Retrieval Evaluation A. Semantic Structure Construction A.1. Entity Extraction Model: GPT-4o-mini EXTRACTION_PROMPT = """Extract key entities from this document. For each entity provide: - name: the entity...

    work page doi:10.18653/v1/2025.acl-long.418 2025

  7. [7]

    AUTHENTICITY: Generate queries that real users would actually type, not artificial constructs

  8. [8]

    SPECIFICITY CONTROL: Match the requested query type precisely (short/long, ambiguous/specific)

  9. [9]

    ANSWERABILITY: The query MUST be answerable by the provided document

  10. [10]

    NO LEAKAGE: Do not copy exact phrases from the document title

  11. [11]

    BERT embeddings

    INFORMATION NEED: Express a genuine information need, not just keywords WHAT TO AVOID: - Queries that are too generic to meaningfully match the document - Queries that require information NOT in the document - Forced or unnatural combinations of keywords --- USER: Generate a search query that this document would answer. Query Type: {query_type_description...

  12. [12]

    DIRECT MENTION: The entity or a synonym appears in the query

  13. [13]

    heart disease prevention

    IMPLICIT REFERENCE: The query is about this entity even if not named explicitly Example: "heart disease prevention" -> relevant: "cardiovascular disease", "heart health", "coronary artery"

  14. [14]

    CRISPR applications

    REQUIRED KNOWLEDGE: Answering the query requires knowing about this entity Example: "CRISPR applications" -> relevant: "gene editing", "Cas9", "genetic engineering"

  15. [15]

    --- USER: Identify which entities from the candidate list are relevant to the search query

    SEMANTIC OVERLAP: The entity’s domain strongly overlaps with the query’s information need WHAT IS NOT RELEVANT: - Entities that are merely topically adjacent but not needed to answer the query - Entities from the same broad field but addressing different specific aspects - Entities that might appear in the same document but aren’t query-relevant PRECISION...

  16. [16]

    It appears in or is directly referenced by the query

  17. [17]

    It represents a key concept the query is asking about

  18. [18]

    Understanding this entity is necessary to answer the query

  19. [19]

    {query_text}

    It’s a person, organization, place, or technical term central to the query An entity is NOT RELEVANT if: - It’s from the same general field but addresses a different aspect - It might co-occur with query terms but isn’t about the query topic - It’s too broad or too specific relative to the query’s need Search Query: "{query_text}" Candidate Entities: {ent...

  20. [20]

    machine learning

    TOPICAL SIMILARITY != RELEVANCE Two documents about "machine learning" may address completely different aspects. The query might be about "ML for medical imaging" but doc is about "ML for NLP"

  21. [21]

    Context and the specific information need matter

    KEYWORD MATCHING != RELEVANCE Matching keywords doesn’t mean the document answers the query. Context and the specific information need matter

  22. [22]

    If I were the user, would this document satisfy my search?

    DOCUMENT QUALITY != RELEVANCE A poorly written document can still be relevant. A high-quality document on a different topic is not relevant. WHEN IN DOUBT: - Re-read the query and identify the core information need - Ask: "If I were the user, would this document satisfy my search?" - Lean toward precision: it’s better to miss a marginally relevant doc tha...

  23. [23]

    Does this document contain information the user is looking for?

  24. [24]

    Would returning this document be helpful for this specific query?

  25. [25]

    0, 2, 5"). If none are relevant, respond with

    Is the relevance substantive or merely superficial keyword overlap? Respond with ONLY the document numbers that are RELEVANT, 17 Coverage, Not Averages Semantic Stratification for Trustworthy Retrieval Evaluation comma-separated (e.g., "0, 2, 5"). If none are relevant, respond with "NONE". D. Examples and Coverage Analysis D.1. Example Semantic Cluster Ta...

    work page arXiv 2008