{"id":"04e3f166-36fa-4e03-b8c0-2a7e75474aa8","arxiv_id":"2505.10089","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"XRAG is a new cross-lingual RAG benchmark showing that large language models frequently ignore the question language and fail at combining evidence from documents in different languages.","lead":"This paper introduces XRAG, a benchmark for evaluating AI systems that answer questions using documents written in a different language than the question. It finds that current models often answer in the wrong language and struggle to combine information across languages, which has direct consequences for multilingual AI products.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that every XRAG question requires cross-document reasoning is not directly tested: the pipeline selects sub-QA pairs from both articles but never checks whether a single article suffices, and the human QA statistics for article-level sufficiency are not reported.","rationale":"The paper is careful and honest overall: human QA, Cohen's kappa 0.71, no-retrieval checks, and the controlled analysis in Section 5.4 are all good practice. However, the central claim that XRAG questions require cross-document reasoning hinges on the unverified necessity of each supporting article. The pipeline ensures that the generated question was constructed from sub-QA pairs drawn from both summaries, but this is a generation-side heuristic; it does not establish that the resulting question is unanswerable from one article. The human QA guideline even collects exactly the information needed to test this (Sufficient/Some/None labels), but the paper does not report the distribution or explain whether 'Sufficient' cases were rejected. The no-retrieval experiment in Section 5.2 checks external knowledge, not cross-document composition. If a single-article oracle shows high accuracy, the benchmark's 'cross-document' label and the Section 5.4 interpretation of cross-lingual reasoning would both need to be softened, though the RLC finding and the raw performance numbers would remain useful. The reader's CONDITIONAL verdict already points in this direction; I see no reason to move it, only to add this verification to the release requirements.","tokens_in":22674,"tokens_out":7323,"duration_ms":73641,"concrete_test":"On a random sample of at least 200 instances per language setting from the released XRAG data, run a single-article oracle: for each instance, present the same question to a strong LLM twice, once with only supporting article 1 and once with only supporting article 2 (no distractors), and score with the paper's LLM-judge protocol. If the fraction of questions answered correctly from at least one single article is material (e.g., >15%), the 'both articles necessary' assumption fails and the cross-document reasoning claim needs qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Load-bearing assumption: each instance's two supporting articles are individually necessary and jointly sufficient for the gold answer (Section 3; pipeline Step 3 in Section 4.2; QA guideline Figure 17). The pipeline's 'select at least one sub-QA pair from each set' rule makes cross-document generation likely, but it does not verify that the final question cannot be answered from a single article alone; one article can incidentally contain all needed facts, or the answer can be inferred from one article plus the question's phrasing. The human QA step (Figure 17, item 4) asks annotators to label each article as Sufficient/Some/None, but the paper neither reports this distribution nor states that 'Sufficient' examples were discarded. Section 5.2's no-retrieval experiment (Table 3) shows only that English questions are not answerable from parametric knowledge; it does not test the necessity of each supporting article, and it is not run on the non-English question versions. If a nontrivial fraction of instances are answerable from one supporting article, the benchmark's 'cross-document reasoning' claim is overstated, and the Section 5.4 controlled analysis becomes ambiguous: the +ES jump could reflect better single-document reading of one article rather than integration of evidence across two documents. This is not fatal by itself, but it is the key empirical gap that should be closed before the benchmark's central claim is accepted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces XRAG, a benchmark for evaluating LLMs in cross-lingual retrieval-augmented generation (RAG). XRAG consists of two settings: monolingual retrieval, where four non-English query languages are answered using English supporting and distracting documents, and multilingual retrieval, where documents in both the query language and English are provided. Questions are generated from recent News Crawl articles via a GPT-4o workflow that first creates simple QA pairs from each of two related articles and then combines them into cross-document questions (aggregation, comparison, multi-hop, set), followed by human quality control, human translation into German, Spanish, Chinese, and Arabic, and automatic selection of topically related distractors. The authors evaluate five LLMs and report two findings: (1) in monolingual retrieval, models frequently answer in English instead of the query language, and (2) in multilingual retrieval, translating supporting articles into English yields larger accuracy gains than translating the question, suggesting the main challenge is cross-lingual reasoning rather than non-English generation. The paper also includes no-retrieval and oracle-retrieval bounds, a human upper bound of 85%, and an LLM-judge panel validated against human annotations with Cohen's kappa of 0.71.","tokens_in":23021,"tokens_out":3483,"duration_ms":34114,"significance":"If the benchmark's design assumptions hold, XRAG fills a real gap: existing cross-lingual QA benchmarks (XQA, XOR QA) are largely English-document and span-extraction oriented, whereas XRAG provides controlled monolingual and multilingual retrieval settings with distractors, recency-based external-knowledge requirements, and human quality control. The reported human-LLM gap (85% vs. 55.5% for GPT-4o) and the low no-retrieval scores (below 16%) support the benchmark's difficulty and its potential as a reasoning testbed. The two empirical findings—response language correctness failures and the dominant effect of translating supporting articles—are plausible and would be useful for system developers. The paper also deserves credit for shipping a comparatively complete evaluation stack: a three-judge panel with majority vote, a language detector, a human-judge agreement check, and a controlled decomposition of the multilingual retrieval setting across five models (Tables 6, 9–11).","major_comments":[{"comment":"The central claim that every XRAG question requires cross-document reasoning rests on the assumption that the two supporting articles are individually necessary and jointly sufficient. The construction pipeline (Section 4.2, Step 3) only enforces that at least one sub-QA pair is selected from each source; it does not check whether the final question can be answered from a single article alone (e.g., because one article incidentally contains all needed facts or the question phrasing gives away the answer). The human QA guideline (Figure 17, item 4) asks annotators to label each article as Sufficient/Some/None, but the paper neither reports this distribution nor states that examples labeled 'Sufficient' were discarded. Without this information, a nontrivial fraction of instances could be answerable from one supporting article, which would overstate the benchmark's cross-document reasoning claim and make the +ES improvement in Section 5.4 ambiguous (it could reflect better single-document comprehension rather than integration of evidence across two documents). Please report the annotation distribution and either discard or re-verify all instances where either article is labeled 'Sufficient', or provide direct evidence that such instances are negligible.","section":"Section 3 and Section 4.2 Step 3"},{"comment":"The controlled analysis that supports the paper's second main conclusion is not statistically grounded. The differences between conditions, e.g., GPT-4o's average improvement from +EQ to +EQ,+ES is 2.91 points (58.25 to 61.16, Table 6), and per-language improvements vary (En+Es improves by only 1.33 points; En+Ar by 3.67), are computed over 300-item subsets per language without confidence intervals, significance tests, or effect sizes. Across Tables 6 and 9–11, some 'improvements' are within 1–2 points and could be noise. Please report per-condition accuracy with confidence intervals (e.g., bootstrap) and apply a paired significance test (e.g., McNemar's test) across the full 1,200-item multilingual set and per language. Without this, the claim that 'the main challenge lies in reasoning over retrieved information across languages' is not yet empirically supported.","section":"Section 5.4, Table 6 and Appendix D.1"},{"comment":"The no-retrieval check is run only on English questions from the monolingual retrieval setting (as stated in the text and Figure 21). The translated German, Spanish, Chinese, and Arabic versions of these questions are not directly tested for answerability from parametric knowledge. Since the benchmark's stated property is that questions require external knowledge to answer, and since translation can alter lexical cues or expose answers differently across languages, the cross-lingual validity of this property is not established. Please run the no-retrieval experiment on the translated non-English questions (or, if a retriever is needed, at least on a representative sample per language) and report the resulting accuracies. If the no-retrieval check cannot be run for all languages, explicitly acknowledge this limitation and discuss the risk it poses to the cross-lingual conclusions.","section":"Section 5.2, Table 3"},{"comment":"The Response Language Correctness (RLC) finding is presented only for German and Chinese in Figure 5, even though the monolingual retrieval setting includes Spanish and Arabic. The paper states that 'all evaluated models struggle with response language correctness' but provides no RLC numbers for Spanish and Arabic. Without these, it is unclear whether the phenomenon is universal or language-dependent. Please provide RLC rates for all four languages (e.g., a table with percentages per model and language), or explicitly state that only German and Chinese were analyzed and why.","section":"Figure 5"}],"minor_comments":[{"comment":"The sentence 'Vu et al. (2024) construct a a dynamic QA benchmark' contains a duplicated article 'a a'; please fix.","section":"Section 2"},{"comment":"The hyphen in 'design-ed-' in the abstract appears to be a linebreak artifact; please ensure the final PDF renders cleanly.","section":"Abstract and Section 1"},{"comment":"The controlled analysis uses a single order of interventions (EQ then ES then ED). A less assumption-laden analysis would also test the reverse order or a factorial design, since the marginal effect of ES may depend on whether EQ is already applied.","section":"Section 5.4"},{"comment":"The selection of distracting articles uses a multilingual dense retriever and a two-week publication gap, but the paper does not describe how the number of distractors (six) was chosen or whether the retrieved distractors were filtered for redundancy or near-duplicate content. A brief clarification would improve reproducibility.","section":"Section 4.4"},{"comment":"The Cohen's kappa of 0.71 is reported for the LLM judge panel versus human evaluators, but the paper does not report the raw agreement rate or per-judge agreement. Adding these would help readers assess the reliability of the evaluation.","section":"Appendix C.2"},{"comment":"The retrieval quality investigation in Appendix A relies on proprietary data (SimilarWeb traffic) and a proprietary evaluation by Claude 3.5 Sonnet; the results are not reproducible from the paper. Please clearly label this as preliminary evidence and, if possible, provide an anonymized description of the query set or a reproducible proxy.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid benchmark submission but is not yet ready for acceptance because the core 'cross-document' property is not verified and the headline controlled analysis lacks statistical rigor. Both issues are fixable within the scope of a revision: report the Sufficient/Some/None annotation distribution and filter appropriately, and add CIs/significance tests. I also note that GPT-4o is used for question generation, is one of the evaluated models, and sits on the judge panel; the paper acknowledges this in footnote 5, but the interaction between generation, evaluation, and conclusion should be explicitly discussed in the final version. The use of a proprietary dataset in Appendix A and a closed model (Nova Pro) is not fatal, but it limits external reproducibility; the authors may want to state this clearly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nXRAG is a solid, useful benchmark paper, not a breakthrough, but it earns a serious referee. The two empirical findings – that models answer in English too often when documents are English and questions are German or Chinese, and that in multilingual retrieval the bottleneck is reasoning over mixed-language evidence rather than generating in the target language – are genuinely new and worth testing. The construction pipeline is careful: recent news, human verification, distractor selection, a human upper bound of 85% versus GPT-4o's 62.4% with oracle retrieval, and low no-retrieval scores. That is a real resource.\n\nMy hesitation is on the load-bearing assumption that every question truly requires two supporting articles. The pipeline selects sub-QA pairs from both sets, but nothing verifies that one article alone is insufficient. The human QA guideline collects Sufficient/Some/None labels for each article, but the paper does not report that distribution or state that 'Sufficient' examples were discarded. If a nontrivial fraction of instances can be answered from a single article, the 'cross-document reasoning' claim is overstated, and the controlled analysis in Table 6 becomes ambiguous – the +ES jump might just reflect better single-document reading rather than evidence integration.\n\nThe other soft spots are more mundane. The headline controlled-analysis numbers are averages over 300-item subsets with differences of 1 to 5 points and no confidence intervals or significance tests. The response-language finding in Figure 5 covers only German and Chinese, not Spanish or Arabic. And no data or code is released, so independent verification is currently impossible. The GPT-4o-as-generator-and-judge circularity is real but mild: the judges are a panel of three with majority vote, and the findings hold across models.\n\nI would not desk-reject this. It deserves referee time, but the dataset should be released and the sufficiency of individual supporting articles should be directly checked. If the cross-document requirement holds up, this is a valuable resource for RAG evaluation.\n\nBest,\n[Your name]","headline":"A useful new cross-lingual RAG benchmark with two plausible findings; needs verification that questions truly require both documents before the central claim is accepted.","tokens_in":23525,"tokens_out":2401,"would_cite":false,"duration_ms":23886,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"XRAG, a new benchmark for cross-lingual retrieval-augmented generation, shows that LLMs' main difficulty is reasoning across languages rather than generating non-English text.","keywords":["cross-lingual RAG","retrieval-augmented generation","benchmark","response language correctness","multilingual retrieval","cross-document reasoning","question generation","LLM-as-a-judge"],"falsifier":"Run the no-retrieval and single-supporting-article conditions on the German, Spanish, Chinese, and Arabic versions of XRAG: if any evaluated model scores near its two-article retrieval accuracy without outside documents, or if one supporting article alone suffices for a large share of questions, the claim that XRAG requires external knowledge and cross-document reasoning would be falsified.","tokens_in":22513,"feed_emoji":"🌐","tokens_out":4702,"duration_ms":40766,"temperature":0.7,"pith_summary":"The paper introduces XRAG, a benchmark for cross-lingual retrieval-augmented generation where users ask in German, Spanish, Chinese, or Arabic while retrieved evidence is in English, or in a mix of English and the question language. Each question is built from two recent news articles so that external knowledge and cross-document reasoning are required, and six topically related distractors simulate imperfect retrieval. Benchmarking five LLMs, the paper reports two previously unreported challenges: in the monolingual retrieval setting, models often respond in English instead of the question language, and in the multilingual retrieval setting, the bottleneck is reasoning over information scattered across languages rather than producing non-English text. Because even the English-only version leaves models well below human accuracy, XRAG also serves as a reasoning benchmark independent of cross-lingual difficulty.","feed_headline":"Cross-lingual RAG fails first on reasoning, not on language output","feed_subtitle":"A new benchmark shows LLMs answer in the wrong language and struggle to combine mixed-language documents.","key_machinery":"The construction pipeline is the central mechanism: from News Crawl articles dated after the models' knowledge cutoffs, the authors (1) pair related articles by shared entities or by multilingual retrieval over international events, (2) prompt an LLM to summarize each article and generate simple single-step Q&A pairs, (3) prompt it to connect those pairs into cross-document questions of four types—aggregation, comparison, multi-hop, and set—such that at least one piece of evidence comes from each article, and (4) have professional annotators verify facts and translate the questions. The controlled analysis in the multilingual setting then isolates the source of difficulty by successively replacing the question, supporting articles, and distractors with English equivalents, attributing the largest accuracy gain to translating the supporting articles.","core_discovery":"XRAG claims that cross-lingual RAG should be evaluated as two distinct scenarios—monolingual retrieval, where a non-English query is answered from English documents, and multilingual retrieval, where evidence comes in both the query language and English—and that existing cross-lingual QA datasets are too simple and too English-centric to expose the real failure modes. Using questions that require combining information from two supporting articles while ignoring distractors, the paper finds that all five evaluated LLMs suffer from response language correctness problems in the monolingual setting, and that in the multilingual setting replacing non-English supporting articles with English translations improves accuracy more than replacing the question language. That pattern indicates that cross-lingual reasoning over retrieved evidence, not non-English generation, is the core difficulty.","pith_inferences":["A testable extension is to feed the model machine-translated versions of the non-English supporting articles: if accuracy then approaches the English-monolingual baseline, the deficit is evidence access, not inference per se.","The response-language-correctness finding suggests that constrained decoding with a language identifier, or an explicit language tag in the prompt, may close much of the monolingual-retrieval gap without retraining.","The two-scenario split implies that retrieval quality should be evaluated separately in each language: a retriever serving one shared English index and one native index may need different relevance criteria.","If XRAG-style questions continue to be generated from recent news, the benchmark can be refreshed over time, keeping it a moving target against parametric-knowledge memorization."],"forward_implications":["Deploying a RAG system with an English-only knowledge base for non-English users requires explicit response-language enforcement; current models silently switch to English.","Adding native-language retrieval alongside English retrieval changes the failure profile: the bottleneck becomes fusing evidence across languages, not generating the answer language.","Benchmarks for multilingual QA that do not force external-knowledge use may overstate cross-lingual RAG ability; XRAG-style questions separate retrieval need from reasoning ability.","The four-question taxonomy (aggregation, comparison, multi-hop, set) gives evaluation designers a reusable template for cross-document reasoning questions.","Cross-document reasoning over mixed-language evidence is a distinct capability worth measuring separately from language generation."],"supporting_citations":[{"why":"Establishes the retrieval-augmented generation paradigm that XRAG evaluates.","marker":"(Lewis et al., 2020)"},{"why":"XOR QA, an existing cross-lingual open-retrieval dataset that the paper positions XRAG against.","marker":"(Asai et al., 2021)"},{"why":"XQA, another existing cross-lingual QA dataset criticized as too simple and English-only.","marker":"(Liu et al., 2019)"},{"why":"Prior analysis of cross-lingual RAG on existing datasets; the baseline XRAG is designed to go beyond.","marker":"(Chirkova et al., 2024)"},{"why":"Source of the four cross-document question types and the LLM-as-a-judge evaluation style.","marker":"(Yang et al., 2024)"},{"why":"Provides the LLM-as-a-Judge protocol used to score answers.","marker":"(Zheng et al., 2023)"},{"why":"News Crawl is the data source for article pairs and distractors.","marker":"(Nagel, 2016)"}],"fun_headline_variants":["Cross-lingual RAG: reasoning over mixed-language docs is the wall","New XRAG benchmark pinpoints cross-lingual RAG's real failure","LLMs stumble on cross-lingual RAG reasoning, not language output","XRAG: response-language errors, then cross-language reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark rests on the assumption that each question's two supporting articles are individually necessary and jointly sufficient, so that a correct answer really requires reading both; if many questions can be answered from one article or from the model's own knowledge, the cross-document reasoning measurement fails.","fun_headline_variants_meta":{"raw":{"variants":["Cross-lingual RAG: reasoning over mixed-language docs is the wall","New XRAG benchmark pinpoints cross-lingual RAG's real failure","LLMs stumble on cross-lingual RAG reasoning, not language output","XRAG: response-language errors, then cross-language reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00045,"raw_usage":{"total_tokens":2235,"prompt_tokens":876,"completion_tokens":1359,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":1279}},"tokens_in":492,"tokens_out":1359,"duration_ms":8736,"temperature":1.0,"reasoning_tokens":1279,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:17:11.381787+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the no-retrieval and single-supporting-article conditions on the German, Spanish, Chinese, and Arabic versions of XRAG: if any evaluated model scores near its two-article retrieval accuracy without outside documents, or if one supporting article alone suffices for a large share of questions, the claim that XRAG requires external knowledge and cross-document reasoning would be falsified.","supporting_citations":[],"review_version":1}