SimpleWikiSearch: A Clean Offline Wikipedia Environment for Agentic Search

Guanming Xiong; Penghui Zhang

REVIEW 3 major objections 5 minor 14 references

One fixed Wikipedia setup makes AI search-agent scores comparable

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 07:35 UTC pith:LT3SHFDN

load-bearing objection A genuinely useful environment contract for offline agentic search, but its reproducibility promise is currently promised, not delivered, and it needs an evidence-coverage audit before the scores can be trusted. the 3 major comments →

arxiv 2607.26070 v1 pith:LT3SHFDN submitted 2026-07-10 cs.IR cs.AI

SimpleWikiSearch: A Clean Offline Wikipedia Environment for Agentic Search

Guanming Xiong , Penghui Zhang This is my paper

classification cs.IR cs.AI

keywords agentic searchreproducibilityoffline Wikipediaretrieval-augmented generationhybrid retrievaltool-use agentsquestion answeringLLM evaluation

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the measured performance of agentic search over Wikipedia depends on every piece of the surrounding environment—the dump snapshot, page cleaning, chunking, retrieval backend, tool schema, observation format, and answer-submission rule—not just the language model. Because these details are usually under-specified, published scores are hard to compare or reproduce. The paper presents SimpleWikiSearch, an offline Wikipedia environment in which all of these choices are made explicit and released as runnable code, data, indexes, and interaction traces. Its stated contribution is the reference setup itself, not a new agent algorithm. If the paper is correct, the field gains a shared testbed where differences in reported accuracy can be attributed to the agent rather than the harness.

Core claim

SimpleWikiSearch specifies an end-to-end offline Wikipedia environment for agent question answering: a fixed English Wikipedia dump is cleaned into lightweight Markdown, sections are merged into 1536-token chunks with sentence-boundary preservation, a shared chunk-id space backs a keyword index and a dense embedding index with hybrid fusion by default, and agents interact through exactly three tools—search, open_url, and submit_answer—under a stated episode protocol (up to 20 model rounds, at most five tool calls per step). Evaluation on six QA datasets is reported with two metrics, lexical F1 and an LLM-judged accuracy, and the authors release all inference result files including message tr

What carries the argument

The central object is the environment contract: a deterministic pipeline from raw Wikipedia dump to final answer submission, including section-aware 1536-token chunking that preserves page titles and section structure, a hybrid retrieval stack (keyword plus dense embeddings, fused by reciprocal rank) over a single chunk-id space, a minimal three-tool interface with a fixed snippet and observation format, and a fixed evaluation protocol with both lexical and judged metrics. The load-bearing idea is that making every stage explicit and releasing it turns the environment into a first-class experimental object, so that agentic-search scores can be compared and reproduced.

Load-bearing premise

The paper assumes the cleaning and chunking pipeline preserves all answer-bearing sentences for its six benchmark questions, yet it reports no recall audit confirming that gold answers survive in the indexed chunks.

What would settle it

Run a recall audit on the released index: for each question in the six benchmarks, retrieve top-50 chunks under the default hybrid search and check whether any chunk contains the gold answer or an alias. If a substantial fraction (say, over 10%) of gold answers never appear in the top-50, the reported scores are properties of the pipeline, not the agent.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

Reported agentic-search scores can now be reproduced and compared directly, since the snapshot, chunking, indexes, and tool schema are fixed and released.
Baseline numbers become attributable to the agent's reasoning rather than to hidden harness choices, allowing cleaner system comparisons.
The random-300 subset offers a low-cost way to compare open-source and closed-source commercial models under identical conditions.
Release of interaction traces lets other researchers analyze tool-use patterns—how many searches, opens, and submissions occur—and separate reasoning failures from budget failures.
Future work on agent algorithms can swap in a new model while keeping the environment fixed, isolating agent-level progress.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

A direct recall audit—checking how often a gold-answer-bearing chunk is in the top-k results—would settle whether the chunking pipeline preserves the evidence needed for the six benchmarks; the paper does not report this, but the released indexes make it easy to compute.
The environment's hybrid retrieval stack invites ablations (keyword vs vector vs fused) that could quantify how much of an agent's accuracy comes from retrieval versus reasoning.
The section-aware 1536-token chunking design may generalize beyond Wikipedia to other long-document domains as an alternative to legacy 100-word passages.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A genuinely useful environment contract for offline agentic search, but its reproducibility promise is currently promised, not delivered, and it needs an evidence-coverage audit before the scores can be trusted.

read the letter

Here's my take on SimpleWikiSearch. The core claim is honest and useful: most agentic-search papers treat the search environment as a black box, and this paper specifies every component of an offline Wikipedia testbed — snapshot, cleaning, chunking, indexes, tool schema, observation format, agent-loop caps, evaluation protocol — and reports baselines. That is a real contribution to the benchmark/reproducibility conversation. The dual F1/LLM-judge reporting is sensible, and the tool-use statistics (rounds, search calls, submit rate) are a nice touch that most papers omit. The authors also position the work correctly: it's an environment, not a new agent.

Where it gets soft: the paper's defining promise is reproducibility, but at review time no code, indexes, or data are actually available — only a promise. That's the main reason I'd treat the numbers as plausible but not verified. Second, there's no evidence-coverage audit. The chunking pipeline merges sections into 1536-token chunks and strips link markup; nothing in the paper tests whether gold answers for the six datasets survive the transformation and are retrievable in the top-k. If some answers are dropped in cleaning or stranded in chunks the default RRF can't surface, every reported accuracy is capped by the environment. That's a load-bearing condition for a benchmark whose whole point is to isolate agent behavior. The reader flagged this as the weakest assumption, and I think that's right. It's not fatal — an audit could be added — but it should be required before people build on the scores. Third, the random-300 subset has no documented seed, which undermines cross-paper comparison for the closed-model results. And the LLM judge metric is reported without variance or any human-agreement check, so we don't know how stable a 1–2 point judge difference is.

The math and citation pattern look fine; this is measurement and specification, not derivation. The baseline numbers are plausible for the models used. The central argument — that the field needs an explicit environment contract — holds up.

Who should read it: anyone building agentic search or RAG benchmarks, or comparing agents over Wikipedia. It deserves a serious referee; the review should focus on the missing artifacts and the coverage audit, and on whether the seed and judge reliability get pinned down. I'd accept it as a workshop/benchmark paper after those revisions. I'll be honest: I'd bring it to reading group, and I'd probably cite it as a reference environment even now.

Referee Report

3 major / 5 minor

Summary. The paper introduces SimpleWikiSearch, an offline Wikipedia retrieval environment for agentic QA. It specifies a full pipeline: a fixed English Wikipedia dump (enwiki-20260601), a cleaning and 1536-token section-aware chunking procedure, Tantivy keyword and FAISS dense indexes with RRF fusion, a three-tool agent interface (search, open_url, submit_answer), a round/call-capped interaction protocol, and a dual evaluation metric (token-level F1 and LLM-judged accuracy). The authors report full-test and random-300 baseline results on 2Wiki, HotpotQA, MuSiQue, FRAMES, PopQA, and Bamboogle using Qwen3.5-4B and Qwen3.5-9B, plus commercial-model results on the random-300 subset. They also report per-question tool-use statistics. The stated contribution is not a new agent but the reference environment itself, intended to make agentic-search results reproducible and comparable.

Significance. If validated, this is a useful community resource: it makes explicit many environment details that are usually hidden in agentic-search papers, and the decision to release message trajectories and inference files is commendable. The two-metric reporting (lexical F1 plus judge accuracy) and the tool-use statistics are informative and go beyond single-score leaderboards. The paper is also honest that its contribution is the harness, not an algorithm. However, the significance depends entirely on two conditions: that the released artifact is actually available and runnable, and that the corpus construction does not silently destroy or hide the evidence needed for the benchmarks. Neither condition is currently demonstrated in the manuscript, so the contribution is well-motivated but not yet substantiated.

major comments (3)

[§2.1, §4, Tables 3–4] The core attribution claim — that reported scores reflect the agent rather than the harness — is not supported without an evidence-coverage audit. Section 2.1 removes explicit link markup and merges sections into 1536-token chunks with a 20% tolerance and sentence-boundary preservation; these operations can drop or split answer-bearing content. Tables 3 and 4 report only end-task F1 and judge accuracy. I ask for a per-dataset retrieval-recall experiment: for each gold answer, does a chunk containing (or entailing) the answer survive preprocessing, and does it appear in the top-k results of the default RRF stack (and of keyword/vector modes)? Without this, all final scores are upper bounds set by the corpus, and cross-model comparisons do not isolate agent behavior.
[§3, Table 4] The random-300 subset is not reproducible as specified. No seed or sampling procedure is given, yet Table 4 and all closed-source-model comparisons depend on this exact subset. Please specify the RNG and seed, or release the exact example IDs used. Without this, the random-300 numbers cannot be regenerated or audited, directly undermining the paper's reproducibility goal.
[§1, Conclusion, Abstract] The paper's central promise is an 'explicit and runnable' reference setup with released code and data, but the availability statement is only future-tense ('will be available') and no code, indexes, artifact versions, or build scripts are included in the submission. Since the contribution is the environment itself, this is load-bearing: a reviewer or reader cannot verify the claim as written. Please provide a versioned repository or a sufficiently detailed appendix for rebuilding the corpus and indexes, including exact versions of Tantivy, FAISS, SGLang, and the embedding model.

minor comments (5)

[Table 1] Typo: 'F AISS' should be 'FAISS'. Also 'chunk-1536 construction' is awkward; use '1536-token chunks'.
[§2.3 / Tables 5–6] The 'Submit %' metric is not defined. Clarify whether it is the fraction of episodes ending with a successful submit_answer before the round cap, as implied by §5.
[Table 2] The 'Alias %' column is not explained. State the source of aliases and how they are obtained for each dataset, since maximal-over-alias F1 is part of the metric.
[Table 4] Bamboogle has only n=125; small absolute differences between models should be interpreted cautiously. Consider adding confidence intervals or raw counts.
[§2.2] The open_url example in the text is truncated mid-sentence ('...'); include the full example or mark it as an excerpt for clarity.

Circularity Check

0 steps flagged

No circularity: the paper is an environment specification with externally measured baselines, not a derivation that reduces to its own inputs.

full rationale

SimpleWikiSearch makes no fitted-parameter or derived-prediction claim. Its central contribution is a specified reference setup: the paper states that 'Its contribution is this specified reference setup, rather than a new agent algorithm.' All reported numbers (Tables 3–6) are measurements of external QA datasets under a fixed retrieval stack and agent loop, evaluated by token-level F1 against gold answers and by an external LLM judge. The environment defaults (1536-token chunks, 20% tolerance, RRF fusion, 20-round cap, five tool calls per step) are explicitly hand-chosen design decisions, not parameters fit to the reported results. The only mild self-reference is that the Qwen tokenizer, embedding model, and evaluated open-source LLMs share the Qwen model family, but this does not make any score equivalent to an input by construction; the commercial-model comparisons provide independent evidence, and the environment itself is runnable and externally checkable. The skeptic's concern that the cleaning/chunking pipeline may drop gold answers is a real validity gap—no evidence-coverage or retrieval-recall audit is reported—but that is a missing validation for the benchmark, not circularity. No equation is shown to equal its input, and no load-bearing claim rests on a self-citation. Therefore the circularity score is 0.

Axiom & Free-Parameter Ledger

8 free parameters · 5 axioms · 0 invented entities

The paper introduces no theoretical invented entities; its 'free parameters' are hand-chosen environment and evaluation defaults that directly shape the reported numbers and are never ablated. The central validity assumptions are that the Wikipedia dump contains the evidence for all six datasets, the gold answers/aliases are correct, the judge model reliably maps paraphrases, the embedding model retrieves adequately, and the cleaning/chunking pipeline does not destroy answers.

free parameters (8)

Target chunk length = 1536 tokens
Section-aware chunks are merged toward 1536 tokens with 20% tolerance. This granularity is a hand-chosen default, not justified by ablations, and affects both retrieval and observation length.
Chunking tolerance band = 20%
Tolerance band for merging toward 1536 tokens; chosen by hand and never varied.
RRF fusion parameters = not stated
Default retrieval is RRF over keyword and vector candidates, but the fusion constant/rank cutoff is not specified, so the exact ranking is not fully reproducible.
Episode round cap = 20 rounds
Runner caps each episode at 20 model rounds; this hard budget directly affects submission-success statistics.
Tool-call cap per step = 5
Assistant step truncated to at most five tool calls; affects how many actions an agent can take.
Inference temperature / top-p = 0.7 / 0.95
Sampling defaults for all baselines; chosen without sensitivity analysis.
Random-300 subset seed = not stated
The 300-example subset's exact membership is not reproducible without a documented seed; central to the closed-model comparison track.
LLM judge model = gpt-5.4-mini-2026-03-17
Judge accuracy is determined by this single model and fixed prompt; no judge-stability analysis or human agreement is provided.

axioms (5)

domain assumption The enwiki-20260601 English Wikipedia dump is a complete, faithful knowledge source for all six QA datasets.
The environment's value depends on the dump covering the entities in 2Wiki, HotpotQA, MuSiQue, FRAMES, PopQA, and Bamboogle; no coverage audit is provided. Section 2.1.
domain assumption The gold answers and alias lists in the six datasets are correct and complete.
F1 and judge scores inherit any dataset noise; the alias column shows high alias rates (e.g., PopQA 100%) that directly inflate F1. Section 3.
domain assumption gpt-5.4-mini-2026-03-17 reliably judges factual equivalence.
The LLM-judged accuracy metric rests entirely on this single judge model; no human-agreement check or judge-stability analysis is given. Section 3.
domain assumption Qwen3-Embedding-0.6B produces query/chunk embeddings suitable for the six benchmarks.
Dense retrieval quality depends on this embedding model; no retrieval-quality metrics (recall@k) are reported. Section 2.1.
domain assumption The described cleaning pipeline and sentence-boundary preservation preserve answer-bearing content.
The pipeline removes explicit link markup and converts wikitext to lightweight Markdown; any content loss is unmeasured. Section 2.1.

pith-pipeline@v1.3.0-alltime-deepseek · 6947 in / 13253 out tokens · 131215 ms · 2026-08-02T07:35:36.503114+00:00 · methodology

0 comments

read the original abstract

Large language model (LLM)-based agentic search systems are often evaluated as if the underlying LLM were the only component that matters, yet their measured performance also depends on the surrounding search environment: the Wikipedia snapshot, preprocessing pipeline, chunking policy, retrieval backend, tool schema, observation format, and answer submission rule. These details are frequently under-specified, making it difficult to compare results or reproduce reported baselines. We present SimpleWikiSearch, whose corpus construction, retrieval stack, tool contract, and evaluation protocol are explicit and runnable. The environment starts from a full English Wikipedia dump, cleans and chunks the corpus, builds keyword and dense retrieval indexes, and exposes a minimal tool interface consisting of \texttt{search}, \texttt{open\_url}, and \texttt{submit\_answer}. We report baseline results on six QA datasets using open-source LLMs and provide a random-300 subset for comparisons with closed-source commercial models. SimpleWikiSearch provides a domain-specific agent harness and a controlled offline environment for reproducible agentic-search evaluation. Its contribution is this specified reference setup, rather than a new agent algorithm. Code and data will be available at: https://github.com/JimXiongGM/simple_wiki_search.

discussion (0)

Reference graph

Works this paper leans on

14 extracted references · 4 linked inside Pith

[1]

Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps

Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. InProceedings of the 28th International Conference on Computational Linguistics, COLING ’20, pages 6609–6625, Barcelona, Spain (Online), 2020. International Committee on Computational Linguistics. doi: 10. 1...

2020
[2]

LongRAG: Enhancing retrieval-augmented generation with long-context llms, 2024

Ziyan Jiang, Xueguang Ma, and Wenhu Chen. LongRAG: Enhancing retrieval-augmented generation with long-context llms, 2024. URLhttps://arxiv.org/abs/2406.15319

Pith/arXiv arXiv 2024
[3]

Ehsan Kamalloo, Charles L. A. Clarke, and Davood Rafiei. Limitations of open-domain question answering benchmarks for document-level reasoning. InProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’23, pages 2123–2128, New York, NY, USA, 2023. Association for Computing Machinery. doi: 10....

arXiv 2023
[4]

Dense passage retrieval for open-domain question answering

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP ’20, pages 6769–6781, Online, 2020. Association for Computational Linguistics. doi: 10....

2020
[5]

Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation, 2024

Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, and Manaal Faruqui. Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation, 2024. URLhttps://arxiv.org/abs/2409.12941

Pith/arXiv arXiv 2024
[6]

When not to trust language models: Investigating effectiveness of parametric and non-parametric memories

Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL ’23, pages 9802–9822, Toronto, Canada...

doi:10.18653/v1/2023.acl-long.546 2023
[7]

KILT: a benchmark for knowledge intensive language tasks

Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rockt¨ aschel, and Sebastian Riedel. KILT: a benchmark for knowledge intensive language tasks. InProceedings of the 2021 Conference of the North American Chapter of the Associa...

2021
[8]

Measuring and narrowing the compositionality gap in language models

Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. InFindings of the Association for Computational Linguistics: EMNLP 2023, Findings of EMNLP ’23, pages 5687–5711, Singapore,

2023
[9]

Qwen3.5: Towards native multimodal agents, February 2026

Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https: //qwen.ai/blog?id=qwen3.5

2026
[10]

MuSiQue: Multihop questions via single-hop question composition.Transactions of the Association for Computational Linguistics, 10:539–554, 2022

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions via single-hop question composition.Transactions of the Association for Computational Linguistics, 10:539–554, 2022. doi: 10.1162/tacl a 00475. URL https: //aclanthology.org/2022.tacl-1.31/

doi:10.1162/tacl 2022
[11]

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP ’18, pages 2369–2380, Brussels, Belgium, 2018. Association for Compu...

doi:10.18653/v1/d18-1259 2018
[12]

Qwen3 embedding: Advancing text embedding and reranking through foundation models, 2025

Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models, 2025. URL https: //arxiv.org/abs/2506.05176

Pith/arXiv arXiv 2025
[13]

Gonzalez, Clark Barrett, and Ying Sheng

Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang: Efficient execution of structured language model programs. InAdvances in Neural Information Processing Systems, volume 37, 2024. URLhttps://arxiv.org/abs/2312.07104. 9

Pith/arXiv arXiv 2024
[2023]

doi: 10.18653/v1/2023.findings-emnlp.378

Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.378. URLhttps://aclanthology.org/2023.findings-emnlp.378/

doi:10.18653/v1/2023.findings-emnlp.378 2023

[1] [1]

Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps

Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. InProceedings of the 28th International Conference on Computational Linguistics, COLING ’20, pages 6609–6625, Barcelona, Spain (Online), 2020. International Committee on Computational Linguistics. doi: 10. 1...

2020

[2] [2]

LongRAG: Enhancing retrieval-augmented generation with long-context llms, 2024

Ziyan Jiang, Xueguang Ma, and Wenhu Chen. LongRAG: Enhancing retrieval-augmented generation with long-context llms, 2024. URLhttps://arxiv.org/abs/2406.15319

Pith/arXiv arXiv 2024

[3] [3]

Ehsan Kamalloo, Charles L. A. Clarke, and Davood Rafiei. Limitations of open-domain question answering benchmarks for document-level reasoning. InProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’23, pages 2123–2128, New York, NY, USA, 2023. Association for Computing Machinery. doi: 10....

arXiv 2023

[4] [4]

Dense passage retrieval for open-domain question answering

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP ’20, pages 6769–6781, Online, 2020. Association for Computational Linguistics. doi: 10....

2020

[5] [5]

Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation, 2024

Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, and Manaal Faruqui. Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation, 2024. URLhttps://arxiv.org/abs/2409.12941

Pith/arXiv arXiv 2024

[6] [6]

When not to trust language models: Investigating effectiveness of parametric and non-parametric memories

Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL ’23, pages 9802–9822, Toronto, Canada...

doi:10.18653/v1/2023.acl-long.546 2023

[7] [7]

KILT: a benchmark for knowledge intensive language tasks

Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rockt¨ aschel, and Sebastian Riedel. KILT: a benchmark for knowledge intensive language tasks. InProceedings of the 2021 Conference of the North American Chapter of the Associa...

2021

[8] [8]

Measuring and narrowing the compositionality gap in language models

Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. InFindings of the Association for Computational Linguistics: EMNLP 2023, Findings of EMNLP ’23, pages 5687–5711, Singapore,

2023

[9] [9]

Qwen3.5: Towards native multimodal agents, February 2026

Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https: //qwen.ai/blog?id=qwen3.5

2026

[10] [10]

MuSiQue: Multihop questions via single-hop question composition.Transactions of the Association for Computational Linguistics, 10:539–554, 2022

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions via single-hop question composition.Transactions of the Association for Computational Linguistics, 10:539–554, 2022. doi: 10.1162/tacl a 00475. URL https: //aclanthology.org/2022.tacl-1.31/

doi:10.1162/tacl 2022

[11] [11]

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP ’18, pages 2369–2380, Brussels, Belgium, 2018. Association for Compu...

doi:10.18653/v1/d18-1259 2018

[12] [12]

Qwen3 embedding: Advancing text embedding and reranking through foundation models, 2025

Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models, 2025. URL https: //arxiv.org/abs/2506.05176

Pith/arXiv arXiv 2025

[13] [13]

Gonzalez, Clark Barrett, and Ying Sheng

Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang: Efficient execution of structured language model programs. InAdvances in Neural Information Processing Systems, volume 37, 2024. URLhttps://arxiv.org/abs/2312.07104. 9

Pith/arXiv arXiv 2024

[14] [2023]

doi: 10.18653/v1/2023.findings-emnlp.378

Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.378. URLhttps://aclanthology.org/2023.findings-emnlp.378/

doi:10.18653/v1/2023.findings-emnlp.378 2023