{"id":"9ef4df4d-93bd-4100-bf38-65ec16307233","arxiv_id":"2504.13425","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"SecMulti-RAG, which adds pre-written expert reports and filtered external LLM knowledge to standard RAG, wins 56-92 percent of pairwise evaluations against single-source RAG on automotive report generation.","lead":"This paper presents SecMulti-RAG, an enterprise retrieval system that combines internal documents, pre-written expert reports, and on-demand external LLM knowledge, gated by a safety filter. It reports that this multi-source system beats a traditional single-source RAG in 56 to 92 percent of pairwise comparisons for report generation in the Korean automotive domain.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported win rates rely on a test set drawn from the same keyword-report taxonomy that defines SecMulti-RAG's extra knowledge source, with Claude-generated gold answers; no evidence that this generalizes to organic user queries.","rationale":"The reader's weakest assumption and my own analysis point to the same load-bearing concern: the end-to-end evaluation is designed around the same keyword-report taxonomy that defines SecMulti-RAG's novel knowledge source, with gold answers and judge both coming from Claude. This makes the reported win rates a measure of performance on the system's intended query distribution, but not of generalization to organic user queries. The paper itself acknowledges the limited, self-constructed data in its Limitations, and the retriever evaluation's inclusion of test splits in the chunk pool (§5.2) is a clear methodological flaw that inflates MAP. I do not see an internal inconsistency in the framework's design; the multi-source retrieval and filtering are reasonable and the security-filter results, while imperfect (74% recall on hard queries), are honestly reported. The central weakness is strictly the validity of the performance claim's evaluation. Since the reader already conditioned acceptance on an independent test set and ablations, and my concern is the same, the verdict should remain CONDITIONAL rather than being changed to REJECT. The proposed concrete test—an organic-query set with engineer-authored gold—would settle whether the win rates are an artifact or a genuine advantage.","tokens_in":16451,"tokens_out":5370,"duration_ms":46725,"concrete_test":"Collect a held-out set of at least 50 organic user queries written by engineers who did not participate in keyword curation, with gold answers authored by engineers (not generated by Claude), then run the same pairwise evaluation of Traditional RAG vs. SecMulti-RAG with human judges blind to system identity. If SecMulti-RAG's correctness win rate falls below 50% on this organic set, the reported win rates are artifacts of the test-query distribution and the Claude-generated gold loop; if the win rate remains above 60%, the central claim is supported for real deployment queries.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline win rates (79.3–91.9% LLM-judged, 56.3–70.4% human-judged) are computed on the 135 held-out keyword–report pairs from the pre-written expert-knowledge pipeline that also supplies SecMulti-RAG's supplementary retrieval source (§5.1.2, §7.1). The keywords in Appendix D.1 are exactly the problem topics for which pre-written expert knowledge is generated, so every end-to-end test query is one for which the system's new knowledge source was deliberately designed, while Traditional RAG is restricted to internal documents that the same curation process found insufficient. Gold answers are the Claude-generated pre-written reports (§7.1, D.2), and the LLM judge is the same model family (Claude 3.7 Sonnet, §G.1), making 'correctness' a match to the augmentation source's content and style rather than an independent ground truth. The retriever evaluation is additionally inflated by including test splits in the chunk pool (§5.2). Thus the win rates are an upper-bound estimate on the system's own design distribution, not evidence that SecMulti-RAG outperforms Traditional RAG on organic enterprise queries. A secondary but independent concern is the confidentiality filter's 74.35% recall on security-sensitive queries without project names (§4.3), which weakens the 'prevents data leakage' claim; however, that issue is reported transparently, whereas the evaluation circularity is not acknowledged and directly undermines the central performance claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SecMulti-RAG, a retrieval-augmented generation framework for enterprise settings that combines three knowledge sources: an internal document base, pre-written expert reports for anticipated queries, and on-demand external LLM knowledge that is invoked only when a confidentiality filter deems the user query safe. The generator is a locally hosted Qwen-2.5-14B model, fine-tuned on Korean automotive QA data. The authors report that SecMulti-RAG outperforms traditional RAG (which uses only the internal knowledge base) on a report generation task, with LLM-judged win rates of 79.3–91.9% across correctness, richness, and helpfulness, and human-judged win rates of 56.3–70.4%. The paper also evaluates the confidentiality filter and the fine-tuned BGE-M3 retriever, and includes deployment-oriented design choices such as progressive filter training and a cap on external documents per query.","tokens_in":16715,"tokens_out":2880,"duration_ms":27285,"significance":"If the reported results are valid, the paper would make a useful applied contribution: it demonstrates a workable architecture for enterprise RAG that combines internal, pre-curated, and on-demand external knowledge while routing queries through a confidentiality filter. The system description is concrete and includes training details, GPU configurations, prompt templates, and a link to open-source retriever code. The human evaluation, though small, is a welcome complement to the LLM judge. However, the manuscript's central empirical claim is weakened by a substantially self-referential evaluation design: the test queries and gold answers are generated from the same expert-knowledge taxonomy that defines the supplementary retrieval source, and the judge is the same model family that produced the gold answers. As a result, the headline win rates are better interpreted as an upper-bound demonstration on the system's own design distribution than as evidence of superiority on organic enterprise queries.","major_comments":[{"comment":"The end-to-end evaluation is circular. The 135 test queries are the held-out keyword–report pairs from the pre-written expert knowledge pipeline that also supplies one of SecMulti-RAG's retrieval sources, so every test query is one for which the system's new knowledge source was intentionally designed. The gold answers are the Claude-generated pre-written reports (Appendix D.2), and the LLM judge is Claude 3.7 Sonnet (Appendix G.1), the same model family that generated those reports. The reported win rates (79.3–91.9% LLM-judged, 56.3–70.4% human-judged) are therefore not evidence that SecMulti-RAG outperforms Traditional RAG on organic user queries. To support the central claim, the authors should either evaluate on independently collected user queries that are not derived from the same keyword–report taxonomy, or explicitly reframe the contribution as a design-space demonstration and discuss the distribution shift. This issue is load-bearing for the paper's main conclusion.","section":"§5.1.2, §7.1, Appendix D.2, Appendix G.1"},{"comment":"The retriever evaluation includes all splits (training, validation, and test) in the chunk pool, as stated in §5.2, which means the test query's gold chunk is always present in the pool. On a collection of only 6,165 chunks, this inflates MAP and makes the comparison between vanilla and fine-tuned BGE-M3 less informative. The authors should report MAP with the test-split chunks excluded from the pool, and ideally with queries that are not tied to the pre-written expert knowledge taxonomy, so that the reported MAP values (0.5965, 0.7027, 0.7099) reflect retrieval quality rather than the design of the test split.","section":"§5.2, §5.3"},{"comment":"The confidentiality filter achieves only 74.35% recall on security-sensitive queries without project names, meaning roughly a quarter of such queries would be flagged safe and sent to external LLMs. Since 'prevents data leakage' is a core claimed contribution, the authors should either quantify the actual leakage risk in the end-to-end run (e.g., how many of the 135 test queries were misclassified and whether the retrieved external documents contained sensitive information) or explicitly restrict the security claim to easy queries and treat the hard-case recall as an open limitation. The current presentation is transparent but does not connect this 74.35% figure to the paper's security guarantee.","section":"§4.3"}],"minor_comments":[{"comment":"There are several typos and stylistic slips: 'for sercurity' in the Introduction, 'approch' in the Conclusion, and 'publicily available code' in §5.2. These should be corrected.","section":"§1, §8"},{"comment":"The reference list cites 'Anthropic. 2024. The claude 3 model family' but the paper uses Claude 3.7 Sonnet; a specific citation for the 3.7 model would help reproducibility.","section":"References"},{"comment":"The description of human evaluation is under-specified: it is unclear how many human annotators performed the pairwise judgments, whether they were blind to system identity, and how disagreements were resolved. Adding this information would increase confidence in the human results.","section":"§7.1"},{"comment":"The heuristic of 'at most one external knowledge document per query' is described but not systematically justified; a sensitivity analysis (e.g., varying the cap between 0 and 3) would clarify its effect on the reported win rates.","section":"§5.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a well-written systems paper with a concrete deployment scenario, but the central evaluation is self-referential in a way that the authors do not acknowledge. The reported win rates are likely to be attractive to readers, yet they cannot be taken at face value as a comparison against traditional RAG on realistic queries. I recommend major revision: the authors should either add an independent evaluation (e.g., organic user queries, independent gold answers, or an external judge) or substantially narrow the claims and explicitly discuss the distribution mismatch. The retriever MAP inflation from including test splits in the pool should also be addressed. The paper fits the venue's applied scope, but the current evaluation design does not support the headline conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid engineering paper, not a research breakthrough. The new thing is the specific combination—internal chunks plus pre-written expert reports plus gated on-demand external LLM knowledge, with a local generator—applied to Korean automotive report generation. The framework is sensible, and the deployment discussion (progressive filter training, query rewriting, at-most-one-external-doc heuristic) is more grounded than most industry RAG papers. The filter results are reported honestly: 99% recall on easy sensitive queries but 74.35% on hard ones, and human annotators only reach 76.44%. So the authors don't oversell the security filter.\n\nThe soft spot is exactly where your stress-test note points. The 135 end-to-end test queries are held-out keyword–report pairs from the same pre-written expert-knowledge pipeline that populates SecMulti-RAG's supplementary retrieval source. Every test query is therefore one for which the system's new knowledge source was designed, while Traditional RAG is limited to internal docs the same curation process deemed insufficient. The gold answers are the Claude-generated reports, and the LLM judge is Claude. That doesn't make the win rates meaningless—human evaluators also preferred SecMulti-RAG, especially on richness—but it does mean the 79–91% LLM win rates are an upper bound on the system's own design distribution. The retriever evaluation including test splits in the chunk pool is acknowledged in Section 5.2, but it still inflates MAP. The 18.5% internal-document retrieval rate among test queries is another sign the test set favors the pre-written source.\n\nI would not call this fatal. For an industry-track paper the framework stands on its own, and the authors are transparent about many limitations. But the central comparative claim needs an independent set of organic user queries, or at least a test set not sampled from the same taxonomy that generates the extra knowledge. Without that, the paper is an existence proof of a reasonable architecture, not a demonstrated superiority over traditional RAG. I'd want code/data released, or at least an independent eval, before trusting the numbers.\n\nWho is this for? Practitioners building secure enterprise RAG in low-resource or proprietary settings. I'd bring it to a reading group as an example of a well-motivated system with a cautionary evaluation design. A serious editor should send it to peer review—the engineering is thoughtful and the flaws are identifiable and fixable—but reviewers should push for a cleaner comparison.","headline":"A plausible enterprise RAG engineering pattern whose headline win rates are inflated by a self-referential evaluation; worth refereeing, but the central comparison needs an independent test set.","tokens_in":17265,"tokens_out":1829,"would_cite":false,"duration_ms":17638,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SecMulti-RAG, which adds pre-written expert knowledge and filtered external LLM knowledge to internal retrieval, beats traditional RAG on enterprise reports, winning 79.3-91.9% of LLM-judged and 56.3-70.4% of human-judged comparisons.","keywords":["retrieval-augmented generation","multi-source retrieval","confidentiality filter","data leakage prevention","enterprise knowledge base","open-source LLM","Korean automotive domain","report generation"],"falsifier":"Use a fresh set of user queries that were not used to build the pre-written expert knowledge, with gold answers verified by domain experts independent of the generation pipeline; if SecMulti-RAG's correctness win rate against traditional RAG falls to 50% (within confidence intervals), the claimed advantage does not generalize beyond the curated test set.","tokens_in":16220,"feed_emoji":"🔐","tokens_out":10103,"duration_ms":73958,"temperature":0.7,"pith_summary":"The paper proposes SecMulti-RAG, a framework for enterprise retrieval-augmented generation that draws on three knowledge sources instead of one: internal corporate documents, pre-written expert knowledge for anticipated queries, and on-demand external knowledge from a closed-source LLM, which is only consulted when a confidentiality filter deems the user query safe. The central claim is that this hybrid retrieval strategy, paired with a locally hosted open-source generator, makes enterprise RAG both more complete and more secure than traditional single-source RAG. On a report-generation task in the Korean automotive industry, SecMulti-RAG is preferred over traditional RAG 79.3-91.9% of the time by an LLM judge and 56.3-70.4% by human evaluators across correctness, richness, and helpfulness. The practical significance is that enterprises can get better, richer answers without sending proprietary data to external LLM providers.","feed_headline":"Multi-source RAG wins 79-92% of report comparisons","feed_subtitle":"Pre-written expert knowledge plus filtered external LLM knowledge makes enterprise reports richer and more complete.","key_machinery":"The load-bearing mechanism is the confidentiality-preserving filter combined with the three-source retrieval pool. A fine-tuned lightweight classifier (Qwen-2.5-3B-Instruct) labels each user query as safe or security-sensitive; only safe queries trigger on-demand generation from an external LLM (GPT-4o), whose output is then indexed back into the retrieval pool for future use. Retrieval is performed by a fine-tuned BGE-M3 embedding model over a vector-indexed pool of internal chunks and pre-written expert knowledge, with a heuristic that limits each query to at most one external-knowledge document. The generator is a fine-tuned Qwen-2.5-14B-Instruct model, so proprietary data stays on local infrastructure unless the filter explicitly allows external augmentation.","core_discovery":"The central discovery is that a RAG system does not have to choose between retrieval depth and data security. By maintaining three complementary retrieval sources—a dynamically updated internal knowledge base, pre-generated expert reports for anticipated queries, and external LLM-generated technical background that is only invoked after a query passes a fine-tuned confidentiality filter—the system can produce substantially more complete and detailed engineering reports than a baseline that retrieves only from internal documents. The authors further show that a moderate-size open-source generator (Qwen-2.5-14B-Instruct) fine-tuned on domain QA pairs can deliver these gains, suggesting that high-quality multi-source retrieval can compensate for a smaller local generator. The evaluation uses pairwise win rates against traditional RAG, with Claude 3.7 Sonnet as the LLM judge and with expert human evaluators, on 135 held-out keyword–report pairs in the automotive crash-safety domain.","pith_inferences":["If the system is tested on independent user queries that were not used to build the pre-written expert knowledge, the reported win rates could shrink, since the current test queries and gold answers come from the same curation pipeline.","The retriever evaluation's practice of including test splits in the chunk pool likely inflates the reported MAP improvements over the vanilla embedding model, so a strictly held-out retrieval evaluation could narrow the apparent gap.","Because the filter's recall on hard queries without project names (74.35%) is below human performance, a query-rewriting fallback—flag the query and transform it into a safer version—could let sensitive queries benefit from external knowledge without exposure; the paper mentions this option but does not evaluate it.","The 'at most one external document' heuristic is a free parameter; a learned document-selection strategy that weighs provenance and query intent together might extract more value from the external source without degrading factual grounding."],"forward_implications":["Enterprises can deploy RAG without sending proprietary query data to closed-source LLM providers: queries flagged sensitive bypass external retrieval entirely, while safe queries still benefit from external knowledge.","Indexing on-demand external knowledge back into the pool means the system improves over time: even queries later classified as sensitive can retrieve previously generated external background documents without making new external calls.","A modest open-source generator can be competitive with larger closed-source models when the retrieval layer is rich enough, lowering inference cost and removing per-token external API dependence.","The filter's high precision (97.93%) suggests that when it flags a query as sensitive, that decision is reliable; the lower recall on hard queries without project names (74.35%) points to the need for human-in-the-loop or progressive deployment.","The approach transfers to question answering beyond report generation, as the authors' preliminary experiments suggest."],"supporting_citations":[{"why":"Defines the original RAG formulation that serves as the baseline system.","marker":"Lewis et al., 2020"},{"why":"Provides the enterprise knowledge base corpus and QA pairs used to train the retriever and generator.","marker":"Choi et al., 2025"},{"why":"Supplies the BGE-M3 multilingual embedding model that is fine-tuned into the retriever.","marker":"Chen et al., 2024"},{"why":"Supplies the Qwen2.5 model family used for the filter (3B) and generator (14B).","marker":"Yang et al., 2024"},{"why":"Provides the external LLM (GPT-4o) that generates on-demand background knowledge for safe queries.","marker":"OpenAI et al., 2024"},{"why":"Claude 3.7 Sonnet generates the query dataset, pre-written expert reports, and serves as LLM judge.","marker":"Anthropic, 2024"},{"why":"Establishes the LLM-as-a-judge pairwise evaluation protocol used to compute win rates.","marker":"Zheng et al., 2023"},{"why":"Presents the multi-source RAG approach that SecMulti-RAG extends with internal retrieval and security filtering.","marker":"Wu et al., 2024"}],"fun_headline_variants":["Secure multi-source RAG beats old RAG by 79-92%","RAG with security filter wins 79-92% of showdowns","Three-source RAG: safer, richer, and 79-92% better","Enterprise RAG: add filtered external knowledge, win 9/10","Local LLM + filtered external help: RAG wins 79-92%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation's test queries and gold answers are generated from the same pre-written expert knowledge that populates the retrieval pool, so the reported win rates over traditional RAG may shrink with independent real user queries and independently verified answers.","fun_headline_variants_meta":{"raw":{"variants":["Secure multi-source RAG beats old RAG by 79-92%","RAG with security filter wins 79-92% of showdowns","Three-source RAG: safer, richer, and 79-92% better","Enterprise RAG: add filtered external knowledge, win 9/10","Local LLM + filtered external help: RAG wins 79-92%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000427,"raw_usage":{"total_tokens":2183,"prompt_tokens":938,"completion_tokens":1245,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":1144}},"tokens_in":554,"tokens_out":1245,"duration_ms":8824,"temperature":1.0,"reasoning_tokens":1144,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:08:29.541862+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use a fresh set of user queries that were not used to build the pre-written expert knowledge, with gold answers verified by domain experts independent of the generation pipeline; if SecMulti-RAG's correctness win rate against traditional RAG falls to 50% (within confidence intervals), the claimed advantage does not generalize beyond the curated test set.","supporting_citations":[],"review_version":1}