ScalableRAG: High-Quality RAG at Zero Ingestion Cost

Aditya Chakravarty; Hilaf Hasson; Jayant Thomas; Krishna Gogineni

REVIEW 5 major objections 6 minor 22 references

ScalableRAG: High-Quality RAG at Zero Ingestion Cost

T0 review · 5 major / 6 minor · reviewed 2026-07-31 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A RAG agent can match heavy knowledge-base methods without building any index at all, by keeping named document and value sets it can refine and aggregate on the fly.

desk verdict Real systems win on set-persistent agentic RAG, but the 7.36% headline is a macro-average artifact—not the core result. read the letter →

arxiv 2607.25135 v1 pith:KPBHYV7Y submitted 2026-07-27 cs.AI cs.LG

Hilaf Hasson , Aditya Chakravarty , Jayant Thomas , Krishna Gogineni This is my paper · ORCID

classification cs.AIcs.LG

keywords retrieval-augmentedgenerationagenticRAGzeroingestionset-algebraicworkspaceaggregativereasoningpatterndiscoverymulti-hopQA

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most high-performing retrieval-augmented systems pay a large upfront cost: an LLM reads every document to build a knowledge graph or a SQL-like table so the system can group and count later. This paper argues that the same grouping and aggregation can be done at question time with no ingestion at all—not even a vector database—when the thing you group by lines up with whole documents. The agent keeps a workspace of named document sets and extracted value sets, filters them with regex and set operations, and aggregates only after the right subset is in hand. On six corpora that method beats every baseline on average, including graph and schema systems, and wins outright on three of them. A second, limited-ingestion variant adds coarse embeddings and a one-time sample-based pattern catalog, still with LLM cost independent of corpus size, for a further lift on structured collections.

What carries the argument

The set-algebraic workspace: a registry of named document sets (always subsets of the corpus) and typed value sets (scalars, dates, or lists extracted per document), updated in place by tools such as apply_filter, extract_field/list, set_operation, filter_values, and aggregate, with every observation carrying relative diagnostics that guide the next refinement.

What would settle it

Run the same six-corpus, same-judge protocol on a question set deliberately built so gold answers require grouping by entities or relations that are not aligned one-to-one with documents; if zero-ingestion ScalableRAG then loses badly to knowledge-graph baselines while staying strong on document-aligned questions, the central claim’s scope is confirmed and its limit is real.

Watch

Extended reading notes

Core claim

Zero-Ingestion ScalableRAG shows that the aggregative operations people buy with knowledge graphs and extracted tables can be replicated by a stateful agent whose only input is a directory of plain-text files. By writing and reading a registry of document subsets and per-document value maps, and by enriching every tool result with diagnostics relative to the current working set, it reaches average LLM-as-judge accuracy about 7.36% above the next strongest baseline across six datasets, handily beating all systems on three corpora and trailing the best baseline only slightly on the other three.

Load-bearing premise

The method only works when the key you need to group by is in one-to-one correspondence with a subset of whole documents; questions that need entity-centric or cross-document keys that do not line up with documents fall outside its design.

Editorial extensions

If this is right

For corpora where documents already act as the natural primary key, teams can skip graph or schema ingestion and still answer count, average, min/max, and multi-hop subset questions.
Inference cost becomes the main budget: steps stay comparable to other agents while tokens stay far lower than agents that repeatedly dump large contexts without set persistence.
Limited-ingestion pattern discovery gives a constant-in-corpus-size path to better extraction on structured filings and transcripts without one LLM call per document.
Agentic RAG systems that do not persist intermediate sets will systematically underperform on aggregative questions even when they have a vector index.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same workspace idea could be layered under existing dense retrievers as a post-retrieval planner, turning top-k hits into named sets instead of replacing retrieval entirely.
If production logs show many questions that group by people, products, or events spanning multiple documents, hybrid routing—sets for document-aligned queries, graphs for the rest—becomes the practical deployment shape.
Diagnostic-rich tool observations may matter as much as the set algebra itself; other agent toolkits could adopt relative diagnostics without adopting the full ScalableRAG stack.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The useful thing here is not another ReAct wrapper. It’s a workspace of named document sets and value sets, with relative filters and diagnostics baked into every observation, so the agent can do count/avg/group-style work without building a KG or SQL table. That design choice is concrete, the code is shipped, and the Hotels result is the cleanest evidence: SRAG’s own aggregative corpus, zero ingestion, large margin. A-RAG loses there for exactly the reason they claim—no persistent sets.

What they do well: full prompts and traces, shared LLM-judge pipeline with answer normalization, honest Limitations §6 on the document-subset primary-key bet, and a clear split between zero-ingestion and limited-ingestion (constant-cost pattern discovery). Related work correctly flags that A-RAG/Interact-RAG don’t persist artifacts. Table 1 is readable; wins on MuSiQue/Transcripts/Hotels are real enough to take seriously as systems evidence.

Soft spots, in proportion. The stress-test on the 7.36% number is right. “All Datasets” is an unweighted mean of six accuracies with n from 100 to 1000. Micro-weighting by question count shrinks the gap over A-RAG to roughly a couple points. Biggest lifts sit on the small sets (Hotels 138, Transcripts 100); on the two large multi-hop benchmarks they’re at parity or slightly behind A-RAG. Transcripts is author-built from filename/label structure their tools exploit—fine as a stress case, not fine at equal weight with MuSiQue. No error bars, single model, some baseline NaNs. None of that kills the method; it kills treating 7.36% as settled.

The scope assumption (group-by key ↔ document subset) is load-bearing and they say so. That’s a feature boundary, not a hidden flaw, for the corpora they chose.

Who it’s for: people building enterprise/doc-QA agents who care about ingestion cost and aggregative questions. Not a theory paper. I’d bring it to a systems/RAG reading group with the weighting caveat on the slide. Deserves peer review; ask for micro-averages, uncertainty, and a clearer separation of author-built vs standard benchmarks. I’d engage and likely cite the workspace idea if I’m writing on agentic RAG.

Referee Report

5 major / 6 minor

Summary. The paper presents ScalableRAG, an agentic RAG framework whose distinguishing feature is a persistent workspace of named document sets and typed value sets that tools read from and write back to, with automatically computed diagnostics (selectivity gaps, anchor-token coverage, zero-result hints) attached to every observation. A Zero-Ingestion variant uses only regex/filename/set-algebra tools over a raw directory (no vector DB, no graph); a Limited-Ingestion variant adds coarse-chunk embeddings and a sample-based pattern-discovery/validation pipeline whose LLM cost is claimed constant in corpus size. On six corpora (MuSiQue, 2Wiki, an author-built Transcripts dataset, FinanceBench, ComplexTR, Hotels), the authors report LLM-judge accuracy of 75.45 macro-averaged for Zero-Ingestion vs. 68.09 for the strongest baseline (A-RAG), a 7.36-point headline margin, with wins concentrated on MuSiQue, Transcripts, and Hotels. Evaluation uses a shared GPT-4.1 judge pipeline with answer normalization, plus EM/F1; code, the verbatim system prompt, and full traces are released.

Significance. If the results hold, this is a useful contribution to agentic RAG: the set-registry workspace with diagnostic feedback is a clean, well-motivated design that demonstrably handles corpus-wide aggregation questions (e.g., the Hotels counting tasks and the Transcripts chamber comparisons in Appendix C) that stateless retrieval agents structurally miss, and it does so without per-document LLM ingestion. The manuscript is unusually transparent for a systems paper: the full system prompt is reproduced verbatim (Appendix A), context-management and tokenization rules are specified precisely (Appendix B), verbatim traces are provided (Appendix C), the Transcripts construction script and validator are described and released (Appendix D), the judge prompts are given in full (Appendix E), and code is public. The Limitations section honestly states the load-bearing scope assumption (grouping keys in 1-1 correspondence with document subsets). These are real strengths. The main reservations concern how the headline number is computed and the absence of any quantified cost comparison, which is the paper's central selling point.

major comments (5)

[Abstract / §4.2, Table 1 ('All Datasets' column)] The headline '7.36% above the next most competitive baseline' is an unweighted macro-average over six datasets of very unequal size (1,000 questions for MuSiQue/2Wiki vs. 100–200 for the other four). Recomputing from Table 1 weighted by question count gives ≈73.1 for Zero-Ingestion vs. ≈70.8 for A-RAG — a ≈2.3-point margin, not 7.36. The macro figure is inflated because the two largest wins are on the two smallest corpora (Hotels, 138 Q, +40.2 over A-RAG; Transcripts, 100 Q, +3.5), while on the two large standard benchmarks the method is at parity (MuSiQue 58.09 vs. 57.00) or behind (2Wiki 85.74 vs. 87.05). The per-dataset results are genuinely interesting and the Hotels margin is large; but the abstract's quantitative claim should be restated with both macro- and micro-averages, or the headline changed to the micro figure.
[§4.1 / Appendix D (Transcripts dataset)] Transcripts is author-constructed, and per Appendix D its gold answers are generated deterministically from exactly the structural features ScalableRAG's tools are designed to exploit: GPO filename conventions (hhrg/shrg/jhrg substrings, targeted by search_filenames) and line-anchored labeled fields (Present:, [Whereupon, ...], targeted by extract_field/apply_filter). The C.1 trace shows the agent solving a question purely via filename filters. This is not misconduct — construction is disclosed — but including this dataset at equal weight with corpora 10x its size in the headline average is not neutral, and it is one of the three datasets where the method 'handily out-performs all baselines.' The paper should (a) state in §4.1, not only in the appendix, that Transcripts was built by the authors from structure the method targets, and (b) report the headline average with and without it.
[§4.2, Tables 1–2] No error bars, seed variance, or significance tests are reported anywhere, and each configuration appears to be a single run. Several margins that the text draws conclusions from are within binomial noise at these sample sizes: FinanceBench (−1.1 vs. A-RAG, n=150, SE ≈ 3.7 points), ComplexTR (−2.5 vs. HippoRAG2, n=200, SE ≈ 3.4 points), and even the MuSiQue 'win' (+1.09, n=1000, SE ≈ 1.6 points). The claim 'only marginally missing maximum performance on the other three' is statistically unsupported as stated. At minimum, report Wilson/binomial confidence intervals per cell; ideally repeat runs at multiple seeds (or with judge reruns) for the small corpora.
[§1, §3.2, §4 (cost claims)] The paper's central framing is ingestion cost ('zero ingestion costs (not even a vector database)', 'constant number of LLM calls'), yet no ingestion-cost measurement is reported for any system: no LLM-call counts, wall-clock, or dollar cost for building HippoRAG2/GraphRAG/SRAG/AutoSchemaKG indices vs. ScalableRAG's pattern discovery (Algorithm 2 involves LLM-judged precision/recall sampling, up to 8 refinement rounds per pattern, and utility scoring — 'constant in corpus size' is plausible but never measured or even bounded analytically). A cost table (ingestion LLM calls/tokens per system per corpus, plus the inference tokens already in Table 2) is needed to substantiate the title claim; without it 'zero/low ingestion cost' is asserted rather than shown.
[Table 1 (AutoSchemaKG rows)] AutoSchemaKG shows NaN on 5 of 6 datasets (OOM or >24h on an n1-standard-8) and is nonetheless discussed as a baseline the method outperforms. A baseline that could not be run to completion on the shared hardware cannot support comparative claims; the text should either report it at a feasible scale/subsample or explicitly restrict claims to the systems that completed. Relatedly, SRAG's scores (3.33 on MuSiQue, 14.75 on 2Wiki) are far below what one would expect from its published results — please confirm the baseline configuration (model, extraction budget) matches the original paper's setup, since an under-resourced SRAG weakens the 'beats knowledge-structure methods' claim.

minor comments (6)

[§4.2 / Appendix E] The agent, the answer-normalization step, and the judge all use GPT-4.1. Same-model judging is known to introduce leniency bias; since all systems share the pipeline the relative comparison is partly protected, but a robustness check with a different judge model (or human audit on a sample) would strengthen the evaluation.
[§2, §4.2, Abstract, §6] Typos/formatting: 'survery' (§2); 'There havebeensomeimprovementsthatretaindense' (§2, missing spaces); 'available athttps' (Abstract, missing space); 'Zero-Ingesetion' (§4.2); 'is it-self' (§6); Listing 1 has two items labeled '(2)' (facilities_amenities_section and the pattern_search example).
[Table 1] The table is hard to parse: each cell packs judge accuracy and EM/F1 with inconsistent vertical ordering explained only in the caption. Consider splitting accuracy and EM/F1 into separate tables or panels, and bolding the best system per column.
[§3.1.1 / Appendix B] The distinction between 'keywords', 'anchor tokens', decomposition tokens, and gap-cluster tokens uses four similar tokenization rules; the Appendix B disambiguation list is helpful and a pointer to it should appear at first use in §3.1.1 rather than only in the appendix.
[§4.2] Table 2 shows A-RAG using dramatically more tokens on Transcripts/FinanceBench (318k vs. 26k per question); a sentence noting whether this is due to chunk_read of large documents, and whether token counts include the judge/normalization calls, would aid interpretation.
[§3.2.2] The pattern-validation thresholds (precision 0.90 / recall 0.85, utility ≥ 5, min validation 0.5, sample m=20, R=8) are stated as defaults with no sensitivity analysis; a brief ablation or justification (even one paragraph) would help readers assess fragility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical systems paper whose accuracy claims are measured against external corpora and independent baselines, not derived from quantities defined as the targets.

full rationale

ScalableRAG's central claims are empirical performance numbers (Table 1 LLM-as-judge accuracy, EM/F1) obtained by running an agent against six corpora and comparing to external baselines (HippoRAG2, SRAG, GraphRAG, A-RAG, Vanilla RAG, AutoSchemaKG). There is no first-principles derivation in which a fitted parameter or self-defined quantity is renamed a prediction. The workspace of document/value sets is an algorithmic design, not a definition that forces the reported accuracies. Pattern-discovery thresholds (precision/recall 0.90/0.85, utility ≥5, sample size 20) and the one-to-one document-subset premise (Limitations §6) are design choices and scope assumptions, not circular reductions of the accuracy claim. The author-built Transcripts gold (Appendix D) is deterministic dataset construction from public GPO structure; that can bias evaluation fairness but does not make the measured scores equal the inputs by construction. One related-work citation (HyQE, Zhou et al. 2024) shares an author but is not load-bearing for any result. No self-definitional loop, fitted-input-as-prediction, uniqueness import, or renaming of a known result appears in the derivation chain.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

The central accuracy claim rests on an empirical systems stack (ReAct-style tools, regex/set algebra, optional embeddings and sample-based patterns) plus the domain bet that QA groupings align with document subsets. Free parameters are engineering thresholds in pattern discovery and agent budgets, not physics-style fitted constants. No new physical entities; the ‘workspace’ is a software abstraction.

free parameters (4)

LLM pattern validation thresholds (precision/recall defaults 0.90/0.85, utility score ≥5, min validation 0.5) = 0.90 / 0.85 / ≥5 / 0.5
Chosen cutoffs decide which discovered patterns are exposed to the agent in Limited-Ingestion; they are not derived from first principles.
Pattern discovery sample size m and refinement rounds R = m=20, R=8
Default m=20 documents and up to R=8 refinement rounds cap ingestion LLM cost but affect pattern quality.
Agent max steps T and context budget B = Typical 5–15 steps; B≈0.6×Cmax (Appendix B)
Inference compute caps that can truncate hard trajectories; affect measured accuracy vs token use.
Embedding chunking policy (encoder-max large chunks, text-embedding-3-small) = coarse / model-max chunks
Limited-ingestion design choice; authors note finer chunks as in A-RAG might change results.

assumptions (4)

domain assumption Many corpus QA questions group by a primary key in one-to-one correspondence with a subset of documents, so document-set algebra suffices for aggregation.
Stated in abstract, intro (parallel to SRAG’s one-row-per-document bet), and Limitations §6; load-bearing for zero-ingestion success.
domain assumption Regex, filename patterns, and set operations over plain text can recover the grouping/aggregation operations that KG or SQL ingestion provide for the evaluated question mix.
Method §3.1; validated only empirically on the six corpora, not proved generally.
domain assumption ReAct-style tool calling with GPT-4.1 and the given prompts is a valid executor; LLM-as-judge scores after answer stripping track answer quality.
Experimental setup §4 and Appendix E; standard but imperfect evaluation axiom for modern RAG papers.
standard math Standard set algebra, regex matching, and safe arithmetic calculators behave as specified.
Underlying deterministic tools in §3.1; not controversial.

invented entities (2)

Set registry S(t) and value-set registry V(t) workspace independent evidence
purpose: Persist named document subsets and typed extractions so later tools can filter, intersect, and aggregate with diagnostics.
Core software abstraction of ScalableRAG; not a physical entity. Independent evidence is the public method/code and empirical tables, not an external natural phenomenon.
Gap clusters from pattern indexing
purpose: Surface keyword occurrences outside captured regex regions so the agent can avoid false counts.
Engineering construct in Limited-Ingestion pattern module (§3.2.2); usefulness shown via examples, not external theory.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ScalableRAG: High-Quality RAG at Zero Ingestion Cost." pith.science (2026). https://pith.science/paper/KPBHYV7Y

@misc{pith2026260725135,
  author       = {Pith},
  title        = {Pith review of: ScalableRAG: High-Quality RAG at Zero Ingestion Cost},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KPBHYV7Y}},
  note         = {Machine review of arXiv:2607.25135}
}

read the original abstract

Recent advances in RAG aim to optimize for performance by paying high ingestion costs for knowledge ingestion: building knowledge graphs or extracting SQL tables. In this work we show that the operations that such knowledge bases allow can be replicated with zero ingestion costs (not even a vector database); in fact our solution, Zero-Ingestion ScalableRAG, handily out-performs all baselines (including knowledge graph approaches) in three out of the six corpora considered here, and only marginally missing maximum performance on the other three, with average accuracy across all six datasets 7.36% above the next most competitive baseline. It achieves this by keeping a workspace of document sets and values sets that it can write into and read from, allowing for on-the-fly aggregative reasoning in all situations where grouping is required on a primary key that is in one to one correspondence with a subset of the total document set. Capping the number of LLM calls by a constant independent of the corpus size, we also introduce Limited-Ingestion ScalableRAG, which does use a minimal vector database as well as an automated pattern discovery from a sample of documents, to further improve accuracy at scale. Our code is available at https://github.com/cohesity/ScalableRAG .

Figures

Figures reproduced from arXiv: 2607.25135 by the authors.

**Figure 1.** Set-algebraic multi-hop reasoning. Each hop narrows the set, and the agent uses the intermediate set as both evidence and anchor for the next hop. documents, train a lightweight classifier over their embeddings (XGBoost by default; logistic regression and random forest are also supported), and apply it corpus-wide. In practice, the agent rarely invokes this pipeline, but it provides a principled fallback for corp… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 1 linked inside Pith

[1]

Your first apply_filter MUST use a SINGLE entity keyword, never a compound pattern with multiple terms

START BROAD –-one keyword at a time. Your first apply_filter MUST use a SINGLE entity keyword, never a compound pattern with multiple terms. For structural constraints (dates, categories, document types), prefer search_filenames –-filenames encode metadata reliably while text mentions can be incidental. Combine sets with set_operation(intersect ). False p...
[2]

read_docs from the beginning AND end of your set (use offset=-3)

EXPLORE from BOTH ENDS. read_docs from the beginning AND end of your set (use offset=-3). Documents are sorted by filename, so beginning/end often show different document types. Note ALL format variants you observe –-a general category may appear in several distinct formats. Build your refinement to cover ALL of them
[3]

Contains keyword

REFINE (mandatory for counting/aggregation). "Contains keyword"̸="Satisfies the condition." Write a more specific regex targeting the actual format you observed –-anchor to structural markers (section headings, field labels), not bare keywords. Apply WITHIN your working set. If the negative set has docs in a different format, build a separate filter and U...
[4]

extracted

NUMERICAL AGGREGATION. For questions asking for average, sum, min, max, or total of a value: first narrow to the exact relevant subset of docs (not "all" unless the question is about all docs). Then read a few docs to learn the exact format of the value. Then use extract_field with a regex capture group to scan EVERY doc in that subset –-never compute ext...
[5]

filter_type

VERIFY & ANSWER. For counts: read 2-3 docs from your NEGATIVE (excluded) set to verify your refinement regex did not miss valid variants. If you find a different format, build a second filter and UNION. Only then count_set on the refined set (not the broad set –-a broad count is an upper bound, never the answer). For facts: quote exact wording from the so...

2022
[8]

filter_type

was the mother of King George Tupou II. ===Automatically computed === Filename search counts: search_filenames("Back")→11 docs Set compositions –-composer_and_Back_pos (7)× has_BackInUSA_pos (1): intersect→0 union→8 composer_and_Back_pos \ has_BackInUSA_pos→7 has_BackInUSA_pos \ composer_and_Back_pos→1 Step 3: apply_filter({ "filter_type": "regex", "patte...

1959
[10]

Back in the U.S.A

was the mother of King George Tupou II. [Context audit –-'Back'in 2 of 3 positive docs:] Back in the U.S.A. "Back in the U.S.A." is... ...ormer, playing his past hits with local backup bands of variable quality. However, i... ===Automatically computed === Filename search counts: search_filenames("Chuck")→2 docs search_filenames("Berry")→3 docs Set composi...

2015
[11]

filter_type

was the mother of King George Tupou II. Step 1: apply_filter({ "filter_type": "regex", "pattern": "Meek Mill", "target_set": "all", "output_prefix": "has_meek_mill" }) Agent thought: The performer of'B Boy (song)'is Meek Mill, with guest appearances from Big Sean and ASAP Ferg. The question asks where the performer was detained. I will now search for info...

2015

Show all 22 references

[12]

Meek")→1 docs search_filenames(

was the mother of King George Tupou II. ===Automatically computed === Filename search counts: search_filenames("Meek")→1 docs search_filenames("Mill")→12 docs Set compositions –-has_b_boy_pos (1)×has_meek_mill_pos (2): intersect→1 union→2 has_b_boy_pos \ has_meek_mill_pos→0 ha...

2015
[13]

Cover-page metadata.The first ∼160 lines of each transcript contain the chamber identifier (e.g.,U.S. HOUSE OF REPRESENTATIVES, UNITED STATES SENATE, or aJOINT HEARING marker), the main committee heading (COMMITTEE ON ...), an optional subcommittee heading (SUBCOMMITTEE ON ......
[14]

Labeled-field markers in the body. CHRG transcripts use a small set of conventional line-anchored labels that recur across hearings: Present: and Members present: (formal attendance roll); Staff Present:; Also present: (visiting members); Available via the World Wide Web: (pub...
[15]

Single-document lookups(69/100): per-hearing questions about the commit- tee, the chamber, the hearing date, and the number of witnesses listed in the Con- tents block
[16]

Chamber-level aggregations(6/100): corpus-wide counts and comparisons across chambers, e.g., the total num- ber of House versus Senate hearings, the size of the House–Senate difference, and which chamber predominates corpus-wide or within a year
[17]

Chamber × year intersections and year totals(3/100): e.g., how many House hearings occurred in 2022; the total number of hearings in a given year

2022
[18]

Natural-language phrasing without lead- ing hints.Both the question text and the gold answers are phrased in natural language, without quoted regex markers or formatting hints

Labeled-field 3-way concept-majority questions(22/100): given three corpus- level features (e.g., an attendance roll, a published online transcript link, and an 11 AM adjournment notation), the ques- tion asks which of the three appears in the most hearings; chamber-restricted...
[19]

final answer

Answer-only extraction:transform a possibly verbose model output into a single “final answer” string (or INSUFFICIENT_EVIDENCE)
[20]

INSUFFICIENT_EVIDENCE

Judging:compare the extracted final an- swer against the gold answer and return a JSON verdict and score. Prompt for answer-only extraction (reason- ing removal): You are a QA assistant. You will be given: - the question - a model answer that may contain reasoning Your task: -...
[21]

incorrect

Count questions: - If the question is asking for a count/number, the answer is ONLY correct if the predicted number exactly matches the gold number. - If it is off by even 1, score MUST be 0 and verdict MUST be "incorrect"
[22]

correct",

List/set questions: - Treat the gold answer and model answer as sets of items. - Let L be the number of distinct gold items. - Start with score=1.0. - Apply a penalty of (1/L) for each missing gold item AND for each extra predicted item. - Treat possible aliases as correct. Do...
[1889]

composer

was the mother of King George Tupou II. ===Automatically computed === Filename search counts: search_filenames("composer")→4 docs Step 1: apply_filter({ "filter_type": "regex", "pattern": "Back", "target_set": "has_composer_pos", "output_prefix": "composer_and_Back" }) Agent t...

1989
[2017]

Father of Rock and Roll

was an American singer and songwriter, and one of the pioneers of rock and roll music. Nicknamed the "Father of Rock and Roll", Berry refined and developed rhythm and blues into the major elements that made rock and roll distinctive with songs such as "Maybellene" (1955), " Ro...

1955
[2025]

the X who

Autoschemakg: Autonomous knowledge graph construction through dynamic schema in- duction from web-scale corpora.arXiv preprint arXiv:2505.23628. Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini,...

2025 arXiv

Pith tools

Reviewed July 31, 2026 · model on record in the stance chip above.

[1] [1]

Your first apply_filter MUST use a SINGLE entity keyword, never a compound pattern with multiple terms

START BROAD –-one keyword at a time. Your first apply_filter MUST use a SINGLE entity keyword, never a compound pattern with multiple terms. For structural constraints (dates, categories, document types), prefer search_filenames –-filenames encode metadata reliably while text mentions can be incidental. Combine sets with set_operation(intersect ). False p...

[2] [2]

read_docs from the beginning AND end of your set (use offset=-3)

EXPLORE from BOTH ENDS. read_docs from the beginning AND end of your set (use offset=-3). Documents are sorted by filename, so beginning/end often show different document types. Note ALL format variants you observe –-a general category may appear in several distinct formats. Build your refinement to cover ALL of them

[3] [3]

Contains keyword

REFINE (mandatory for counting/aggregation). "Contains keyword"̸="Satisfies the condition." Write a more specific regex targeting the actual format you observed –-anchor to structural markers (section headings, field labels), not bare keywords. Apply WITHIN your working set. If the negative set has docs in a different format, build a separate filter and U...

[4] [4]

extracted

NUMERICAL AGGREGATION. For questions asking for average, sum, min, max, or total of a value: first narrow to the exact relevant subset of docs (not "all" unless the question is about all docs). Then read a few docs to learn the exact format of the value. Then use extract_field with a regex capture group to scan EVERY doc in that subset –-never compute ext...

[5] [5]

filter_type

VERIFY & ANSWER. For counts: read 2-3 docs from your NEGATIVE (excluded) set to verify your refinement regex did not miss valid variants. If you find a different format, build a second filter and UNION. Only then count_set on the refined set (not the broad set –-a broad count is an upper bound, never the answer). For facts: quote exact wording from the so...

2022

[6] [8]

filter_type

was the mother of King George Tupou II. ===Automatically computed === Filename search counts: search_filenames("Back")→11 docs Set compositions –-composer_and_Back_pos (7)× has_BackInUSA_pos (1): intersect→0 union→8 composer_and_Back_pos \ has_BackInUSA_pos→7 has_BackInUSA_pos \ composer_and_Back_pos→1 Step 3: apply_filter({ "filter_type": "regex", "patte...

1959

[7] [10]

Back in the U.S.A

was the mother of King George Tupou II. [Context audit –-'Back'in 2 of 3 positive docs:] Back in the U.S.A. "Back in the U.S.A." is... ...ormer, playing his past hits with local backup bands of variable quality. However, i... ===Automatically computed === Filename search counts: search_filenames("Chuck")→2 docs search_filenames("Berry")→3 docs Set composi...

2015

[8] [11]

filter_type

was the mother of King George Tupou II. Step 1: apply_filter({ "filter_type": "regex", "pattern": "Meek Mill", "target_set": "all", "output_prefix": "has_meek_mill" }) Agent thought: The performer of'B Boy (song)'is Meek Mill, with guest appearances from Big Sean and ASAP Ferg. The question asks where the performer was detained. I will now search for info...

2015