{"id":"c398003d-23a1-4c02-9639-f1b84dff10c4","arxiv_id":"2501.11301","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A question-to-question retrieval system that maps user queries to LLM-generated questions about Wikipedia passages and Wikidata triples, returning the linked source text without answer generation.","lead":"This paper proposes answering Wikipedia and Wikidata questions by first using an AI model to generate many questions for each text unit, then matching a user's query to that stored bank of questions instead of generating an answer. This design aims to return exact source passages quickly and without the fabricated answers that language models sometimes produce.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's high-precision and hallucination-free claims rest entirely on hand-picked cosine-similarity examples; no retrieval-accuracy evaluation is reported, leaving the load-bearing question-coverage assumption untested.","rationale":"The reader's weakest-assumption analysis pinpoints the untested comprehensiveness of generated questions, and I agree that this is the single load-bearing premise. The paper describes a coherent system, but the empirical evidence is limited to a few paraphrase-like examples with raw cosine scores; no evaluation against any benchmark, baseline, or even a held-out set is presented. The abstract and conclusion assert 'consistently above 0.9' and 'hallucination-free,' but these claims are not operationalized as retrieval accuracy. Because the method's entire value proposition depends on reliable retrieval of the correct passage, and retrieval accuracy is unmeasured, the central claim cannot be accepted as demonstrated. My concrete test—a standard NQ retrieval evaluation using the same components as the prototype—would settle whether the concern is real: if top-1 accuracy is high, the method has merit; if not, the central claim fails. Since this is the same concern the reader raised, I agree with the reader's verdict and see no need to adjust it.","tokens_in":8773,"tokens_out":3376,"duration_ms":38564,"concrete_test":"Evaluate the Q2Q retriever on a standard factoid QA benchmark, e.g., 500 questions from Natural Questions with known gold passages. For each question, generate questions with llama-3.1-8b-instruct for the passages in a BM25 top-100 candidate list, embed both queries and generated questions with bge-small-en-v1.5, retrieve the top-1 passage by cosine similarity, and measure top-1 passage accuracy (gold answer present in the retrieved paragraph). Also report the fraction of queries whose top match is a question generated from the gold passage (coverage). Compare against DPR and BM25 baselines. If top-1 accuracy is far below standard baselines or coverage is low (e.g., <80%), the high-precision and hallucination-free claims are unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"For the central claim to hold, question-to-question retrieval must guarantee that for every user query answerable by a passage, the top-1 nearest question embedding maps to a passage that actually answers the query. This requires both (a) comprehensive question generation coverage and (b) sufficient discriminative power in the embedding space. The paper provides no evidence for either: the only quantitative support is Table 2, containing 8 hand-picked queries, several of which are near-duplicates of the LLM-generated questions (e.g., 'France nuclear energy percentage?' vs. 'What percentage of France's electricity is nuclear?'). Even those scores are raw cosine similarities, not retrieval precision or recall; a high cosine to a generated question does not establish that the mapped passage contains the correct answer. The Limitations section concedes 'it might be possible to improve question generation coverage,' but coverage is exactly the load-bearing assumption: if a user query has no generated question in its vicinity, the argmax silently selects an unrelated question and returns the wrong passage with high confidence. Consequently, the 'hallucination-free' claim is also overreaching: generation hallucination is removed, but retrieval error remains a completely unmeasured source of false answers. The paper's own Section 2 defines the proper top-k retrieval accuracy metric but never reports it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a question-to-question (Q2Q) retrieval architecture for factoid question answering over Wikipedia and Wikidata. For each logical content unit (a paragraph or a textualized Wikidata triple), an instruction-tuned LLM generates a set of natural-language questions; these questions are embedded with a dense encoder and stored in a vector index. At inference time, a user query is embedded and matched against the question index via cosine similarity; the argmax match is used to retrieve the original content through a hash mapping. The paper claims that this approach achieves cosine similarity scores consistently above 0.9, high-precision retrieval, and hallucination-free responses, because no answer-generation step is used. The experiments reported are limited to hand-selected similarity examples in Tables 1-3 and a prototype indexing fewer than 1,000 Wikipedia articles; no retrieval precision, recall, accuracy, or comparison with existing systems is reported.","tokens_in":8992,"tokens_out":3829,"duration_ms":39604,"significance":"The core idea is interesting and potentially useful: converting declarative passages into a question space may reduce the well-known question-passage embedding mismatch in dense retrieval, and the method eliminates the cost and hallucination risk of answer generation at inference time. The paper also correctly identifies the relevant prior work, including document expansion by query prediction (Nogueira et al.) and unified structured/unstructured QA (UniK-QA). If the method were shown to work at scale on standard QA benchmarks, it could be a practical contribution to open-domain factoid QA. However, the paper as written provides only anecdotal evidence; the central claims are not backed by a systematic evaluation, and the load-bearing assumption that the generated question set covers all plausible user queries is merely asserted. The authors do clearly state the extractive scope and the limitation concerning question-generation coverage, which is helpful transparency but does not replace the missing measurements.","major_comments":[{"comment":"The central empirical claim, 'cosine similarity scores consistently above 0.9 for relevant question pairs,' is supported only by a handful of hand-picked examples. Several of these queries are near-verbatim paraphrases of the generated questions (e.g., 'France nuclear energy percentage?' vs. 'What percentage of France's electricity is nuclear?'), so the high scores are largely by construction. The paper itself defines the proper metric, top-k retrieval accuracy, in Section 2, but never reports it. A systematic evaluation is needed, e.g., using a standard QA benchmark such as Natural Questions or a random sample of natural user queries, with retrieval precision, recall, and end-to-end answer accuracy.","section":"Section 5 and Table 2"},{"comment":"The load-bearing assumption is that the LLM-generated question set is comprehensive enough that any user query answerable by a passage will be nearest to a generated question that maps to that passage. The Limitations section concedes 'it might be possible to improve question generation coverage,' but no coverage measurement is provided and no analysis of failure modes is given. If a user query has no generated question in its vicinity, the argmax selection silently chooses an unrelated question and returns the wrong passage with high confidence. The paper should estimate coverage, for example by measuring retrieval accuracy on held-out queries that were not used in generation or on natural user queries, and reporting recall at various k.","section":"Section 4.3 and Section 3"},{"comment":"The term 'hallucination-free' overstates the safety of the system. Removing the LLM generation step eliminates generation-based hallucination, but it does not eliminate retrieval error: a wrong argmax match will present incorrect content as the answer. Since retrieval accuracy is never measured, the claim that the system is 'hallucination-free' in any practical sense is not supported. The paper should revise the claim to 'no generated answer' or 'no generation hallucination,' and evaluate the system's accuracy on answer-bearing queries, rather than asserting that retrieval error cannot occur. This is not a matter of wording only; it changes the meaning of the central contribution.","section":"Section 4.2 and Abstract"}],"minor_comments":[{"comment":"Unresolved cross-references to 'Appendix ??' appear in Section 3; the prompts are actually in Appendices A.1 and A.2. Also, Listing 1 is not referenced in the text as Appendix B. All cross-references should be fixed before resubmission.","section":"Throughout"},{"comment":"There is an inconsistency in the embedding models: Table 1 reports 'text-embedding-004. Dimensions: 798,' while Section 4.1 states the prototype used baai/bge-small-en-v1.5 with 384-dimensional vectors. The authors should clarify which model was used for which experiment and whether the dimensionality value 798 is a typo (e.g., 768).","section":"Table 1 and Section 4.1"},{"comment":"The notation is inconsistent: the text first defines D documents and then uses M for both the number of passages and the number of content units. The formal definitions would be easier to follow if the symbols were unified and defined once.","section":"Section 2"},{"comment":"The claim that the vector store size is 'approximately ten-fold' larger than passage-based indexing is given without empirical justification. Since the number of generated questions per passage is not reported, the reader cannot verify this ratio. Reporting the average number of questions per passage and the actual index size would make this quantitative claim checkable.","section":"Section 4.1"},{"comment":"The screenshot in Figure 2 is described only in its caption; the text would benefit from a sentence explaining the user interaction (query input, article navigation, scrolling, and highlighting) so the reader knows how the prototype behaves.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads more like a technical report with illustrative examples than a full research paper. The central idea is plausible, but the absence of any systematic retrieval evaluation is a serious gap. I would be willing to consider a revised version that reports standard retrieval metrics on a benchmark dataset, measures question-generation coverage, and carefully narrows the 'hallucination-free' claim. If the authors cannot add such experiments, the paper would be better suited to a workshop or as a short systems paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a coherent write-up of an old idea with a new name. Question-to-question retrieval is document expansion by query prediction (Nogueira et al. 2019), which the paper honestly cites. The Wikidata triple-to-text part follows UniK-QA. What's left is an engineering proposal: index LLM-generated questions for each passage, match user queries to question embeddings, return the mapped passage. That could be useful for low-latency extractive QA, but nothing in the paper demonstrates that it works.\n\nWhat's good: the paper is honest about its building blocks, the system description is clear, and the limitations section concedes that question coverage may be incomplete. The prototype is real, though small (under 1,000 articles). The idea of avoiding generation entirely is worth a footnote as one way to control hallucination cost.\n\nThe soft spots are large. The only quantitative evidence is Tables 1-3: a handful of hand-picked queries, most of which are near-verbatim paraphrases of the generated questions. The abstract and conclusion claim cosine similarity above 0.9 and 'highly precise retrieval,' but no retrieval precision, recall, or top-k accuracy is reported anywhere. The paper itself defines top-k retrieval accuracy in Section 2 and never measures it. The load-bearing assumption is that the generated question set covers all plausible user queries; if a query has no close generated question, the argmax silently returns the wrong passage with high confidence. The paper concedes this in Limitations but does not test it. 'Hallucination-free' is also overstated: generation hallucination is removed, but retrieval error remains unmeasured.\n\nThe novelty is modest. Applying query-prediction expansion to Wikipedia/Wikidata with dense retrieval is natural, and the paper does not add a new algorithm or a new result. The examples are selected to look good, not to quantify behavior.\n\nWho is this for? Someone thinking about lightweight, generation-free factoid retrieval might skim it for the architecture. But as a research paper claiming empirical effectiveness, it falls short of the evidentiary bar. I would not send it to a serious referee as is. If the author adds real retrieval evaluation on a standard benchmark (e.g., Natural Questions, TriviaQA) and reports coverage statistics, it could become a modest workshop contribution. My recommendation: desk reject or return for major revision with mandatory evaluation.","headline":"Repackaged document expansion with no evaluation; a coherent proposal, not a demonstrated result.","tokens_in":9526,"tokens_out":2469,"would_cite":false,"duration_ms":24976,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that indexing each Wikipedia paragraph and Wikidata triple with LLM-generated questions, then matching user queries to those questions by cosine similarity, retrieves the correct source with scores above 0.9 and makes…","keywords":["question-to-question retrieval","hallucination-free QA","dense vector retrieval","Wikipedia question answering","Wikidata question answering","question generation","cosine similarity matching"],"falsifier":"Run a held-out evaluation where real user queries (for example from Wikipedia search logs) are matched against the question index, with annotators judging whether the retrieved paragraph answers the query; if a large share of answerable queries fail to retrieve the correct content, the coverage assumption—and with it the claim of consistently high precision—breaks.","tokens_in":8556,"feed_emoji":"🔍","tokens_out":9466,"duration_ms":74931,"temperature":0.7,"pith_summary":"This paper proposes a retrieval-only design for open-domain question answering. Instead of embedding passages and comparing a user's question to them, an instruction-tuned LLM first generates a set of plausible questions for every Wikipedia paragraph and every Wikidata statement; those questions are embedded into a vector store, mapped back to the original content by hash. A user query is embedded in the same space, and the highest cosine similarity selects the generated question whose corresponding paragraph or triple is returned. Because both sides are questions, similarity is high—above 0.9 for relevant pairs—so the top match points reliably to the content that answers the query. Presenting the retrieved original text directly eliminates the answer-generation step and, with it, the risk of hallucinated or fabricated responses.","feed_headline":"Question-to-question retrieval scores above 0.9 for Wikipedia QA","feed_subtitle":"Indexing LLM-generated questions instead of passages removes the hallucination-prone generation step.","key_machinery":"The mechanism that carries the argument is the generated-question index. For each content unit—a paragraph with its article and section context, or a Wikidata triple rendered as text—an instruction-tuned LLM produces a set of natural-language questions the unit can answer; these are embedded with a text-embedding model, stored in a vector store, and linked to the source through a SHA-256 hash of the unit. At query time, the user query is embedded and the argmax over cosine similarity selects the closest generated question, whose hash retrieves the original unit. The design's force comes from making the indexed objects questions rather than declarative passages, so query and index entry share the same interrogative form.","core_discovery":"The central claim is that question-to-question retrieval—indexing the knowledge base with generated questions rather than the content itself—achieves both high retrieval precision and hallucination-free responses. The paper reports cosine similarity consistently above 0.9 for relevant question pairs, against the 0.4–0.7 range typical of question-to-passage matching, and argues that directly returning the retrieved source text makes answer generation unnecessary. It further claims that the same approach works for Wikidata by converting each triple to a text form and generating questions from it, and that through triple metadata it makes images, audio, and 3D models addressable as answers.","pith_inferences":["The paper does not measure generation coverage; the decisive extension would be a held-out test using real search queries to quantify how many user phrasings fall outside the generated question sets.","A second inference: because the method retrieves the source unit itself, a minimum-similarity threshold could act as a built-in refusal mechanism, letting the system say 'I don't know' instead of guessing—a property current RAG systems typically lack.","Third, the same index design would likely transfer to non-encyclopedic corpora such as legal or technical documentation, where the mismatch between interrogative queries and declarative text is equally acute.","Fourth, multi-hop questions, which the paper explicitly leaves out, could be addressed by generating questions over chained content units (adjacent paragraphs or linked Wikidata triples), trading index size for reasoning scope."],"forward_implications":["Because inference-time generation is replaced by a vector lookup and a hash dereference, query latency and cost fall well below generation-based RAG systems.","The answer shown to the user is always the original Wikipedia text or Wikidata statement, so the system cannot fabricate content that did not come from the knowledge base.","Selective re-indexing via content hashes means Wikipedia's constant editing can be accommodated by regenerating questions only for changed paragraphs.","The approach treats textual facts and metadata for images, audio, and 3D models uniformly, so multimodal questions like 'show me the flag' or 'how does a lion roar' resolve to the same retrieval pipeline.","The roughly tenfold expansion of the index from question generation still fits within modern vector database capacities, keeping the approach scalable to full Wikipedia."],"supporting_citations":[{"why":"Surveys the hallucination problem in retrieval-augmented generation, motivating the paper's removal of the generation step.","marker":"Gao et al., 2024"},{"why":"Defines retrieval-augmented generation (RAG), the baseline architecture the proposed method replaces.","marker":"Lewis et al., 2020"},{"why":"Provides the dense passage retrieval formulation and top-k retrieval metric that the paper's question-indexing approach reconfigures.","marker":"Karpukhin et al., 2020"},{"why":"Introduces document expansion by query prediction, the conceptual precedent for generating queries from content units.","marker":"Nogueira et al., 2019"},{"why":"Establishes the text rendering of Wikidata triples that the paper uses before question generation.","marker":"Oguz et al., 2022"},{"why":"Describes Wikidata, the structured knowledge base the paper extends its method to.","marker":"Vrandečić and Krötzsch, 2014"},{"why":"Defines the open-domain Wikipedia QA task and the need for a retriever, framing the problem the paper addresses.","marker":"Chen et al., 2017"}],"fun_headline_variants":["Question-to-question retrieval beats passage matching for QA","Index questions, not passages: hallucination-free retrieval","Q2Q retrieval: >0.9 cosine, no answer generation needed","Question indexing for precise Wikipedia and Wikidata QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the generated question set for each paragraph or triple is complete enough that every user question answerable by that content will match one of its generated questions more closely than any content generated from elsewhere.","fun_headline_variants_meta":{"raw":{"variants":["Question-to-question retrieval beats passage matching for QA","Index questions, not passages: hallucination-free retrieval","Q2Q retrieval: >0.9 cosine, no answer generation needed","Question indexing for precise Wikipedia and Wikidata QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000145,"raw_usage":{"total_tokens":1119,"prompt_tokens":826,"completion_tokens":293,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":442,"completion_tokens_details":{"reasoning_tokens":227}},"tokens_in":442,"tokens_out":293,"duration_ms":3483,"temperature":1.0,"reasoning_tokens":227,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:25:29.758447+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a held-out evaluation where real user queries (for example from Wikipedia search logs) are matched against the question index, with annotators judging whether the retrieved paragraph answers the query; if a large share of answerable queries fail to retrieve the correct content, the coverage assumption—and with it the claim of consistently high precision—breaks.","supporting_citations":[],"review_version":1}