{"id":"8f884227-cfed-447a-96e5-9c3d3bc9ba12","arxiv_id":"2502.09596","paper_version":1,"verdict":"UNVERDICTED","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":3,"one_line_summary":"A configurable multi-agent RAG system that rewrites queries from conversation context, routes to knowledge sources by embedding centroids, and generates answers with separately produced citations, shown in three industrial use cases without measured results.","lead":"KIMAs is a configurable multi-agent system for knowledge-intensive question answering that combines conversation-aware query rewriting, embedding-based source routing, reranking, and a two-stage answer-and-citation pipeline. The paper describes three deployed use cases but provides no quantitative evaluation and no released code.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Routing runs on the raw user query in parallel with context rewriting (§3.5), so multi-turn follow-ups with pronouns can be routed to the wrong knowledge agents; no routing accuracy data is reported to rule this out.","rationale":"The reader's weakest_assumption already identifies routing reliability as the central risk; I agree and sharpen it by locating the exact ordering problem in Section 3.5. The paper's own motivating example is a multi-turn query whose referent exists only in prior conversation, and such queries are precisely where raw-query centroid routing can fail. Because no routing or end-to-end measurements are reported, the paper cannot currently support its performance claims, which supports the reader's UNVERDICTED verdict. I am not claiming the system is wrong; the proposed comparison would either vindicate the parallel ordering or show it needs revision. The critique is architectural and evidentiary, not a judgment about the authors. Credit is due for a clearly described, configurable architecture, but that alone does not establish the claimed improvements in coherency, routing, or latency.","tokens_in":13235,"tokens_out":4041,"duration_ms":44016,"concrete_test":"Release the code and run a controlled experiment on 100 multi-turn follow-up queries that contain pronouns or omitted referents (e.g., after a previous answer mentions 'the Werewolf game', ask 'Where can I find the code for it?'). Compare the Section 3.5 parallel pipeline, which routes on the raw query, against a serial variant that routes on the context-manager-rewritten query. Measure routing recall, defined as whether the top-K routed agents include the agent whose source contains the referent, and final answer accuracy via human or LLM judgment. If raw-query routing recall is materially lower, the parallelization order should be changed or routing should wait for the rewrite.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.5 parallelizes Query ingest: the context manager's conversation-context rewrite runs concurrently with query routing, so routing uses the raw user query, not the enriched query. In the paper's motivating multi-turn example, a follow-up like 'Where can I find the code for it?' has an unresolved referent, and the embedding used for centroid routing is computed on that fragment. If routing selects the wrong retrieval agents, the later context-aware rewrite is applied only to the wrong sources, and retrieval cannot recover. Manual mix-in and score scaling adjust static similarity ranges; they do not supply conversational context at query time. No routing accuracy, retrieval recall, or end-to-end comparison is reported anywhere in the paper, so there is no evidence that this ordering is safe in practice. The architecture is coherent and the three deployments are plausible, but this design tension is the point where the central claim of multi-turn conversational coherency combined with efficient routing is least secure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents KIMAs, a configurable multi-agent framework for knowledge-intensive question answering. The system is organized around three agent types: a context manager that rewrites conversational queries and digests conversation history, retrieval agents that access heterogeneous knowledge sources (local vector databases, online search engines, domain-specific HTTP APIs), and a summarizer that generates final answers with references. Routing is performed by embedding-based clustering over each agent's knowledge chunks, with optional developer-supplied mix-in text and score scaling. The pipeline is parallelized at the query-ingest and retrieval stages. The paper reports three deployment use cases: an AgentScope Q&A chatbot, a ModelScope Q&A chatbot, and an Olympic-commentary bot on Weibo. No quantitative evaluation is provided; the use cases are described through screenshots and qualitative configuration summaries.","tokens_in":13519,"tokens_out":3089,"duration_ms":30678,"significance":"If validated, KIMAs would be a practically useful open-source framework for building RAG-based conversational applications over heterogeneous knowledge sources. The paper's design choices are reasonable and clearly motivated: the separation of conversation-context and knowledge-context rewriting, the centroid-based routing with manual intervention, and the two-stage look-back citation generation are all sensible engineering contributions. The paper also gives concrete configuration recipes for three deployed systems, which is valuable for practitioners. However, the central claims about improved retrieval accuracy, multi-turn coherency, efficient routing, and low latency are currently unsupported by any quantitative measurement. There are no baselines, ablations, benchmarks, or error analyses, and the primary deployments are the authors' own products, so the effectiveness claims rest on self-referential demonstration. The architectural ideas are promising, but the evidence needed to support the paper's stated contributions is not yet present.","major_comments":[{"comment":"The paper makes quantitative claims in the abstract (improved retrieval accuracy, multi-turn conversational coherency, efficient routing, low latency), but Section 4 provides only screenshots and qualitative descriptions, and Section 5 concludes on the basis of these demonstrations. There are no retrieval accuracy numbers, routing precision/recall figures, end-to-end answer quality metrics, latency distributions, or comparisons against a single-agent RAG baseline or an existing framework such as LlamaIndex or LangChain. The only quantitative statement is the claim in Section 4.3 that end-to-end latency is reduced to less than 10 seconds per post, but no measurement methodology or data is given. Without this evidence, the core claims of the paper are not testable. I would request a systematic evaluation with at least retrieval recall, routing accuracy, answer faithfulness or correctness, and latency, on both the deployed use cases and a public benchmark.","section":"Section 4 and Section 5"},{"comment":"The optimized pipeline runs query routing on the raw user query in parallel with the context manager's conversation-context rewrite. As the motivating example in Section 3.1 shows, a follow-up such as \"Where can I find the code for it?\" contains an unresolved pronoun that only the conversation history can resolve. Under the parallel execution described in Section 3.5, the embedding used for centroid routing is computed from the unresolved fragment, so routing may activate the wrong retrieval agents. The later context-aware rewrite is then applied only to the wrong sources, and retrieval cannot recover the omitted information. The paper does not report any routing accuracy measurements for multi-turn queries, so there is no evidence that this ordering is safe. This design tension directly affects the central claim of combining multi-turn coherency with efficient routing; I would expect either a sequential routing-after-rewrite option, or an evaluation showing that routing on raw queries is sufficient in conversational settings.","section":"Section 3.5, Parallelization 1"},{"comment":"The routing mechanism has several free parameters that are not analyzed: the weight of the manual mix-in relative to local knowledge embeddings, the score scaling factor, and the number of activated retrieval agents (top-K). These parameters are described as developer preferences, but no guidance or sensitivity analysis is given, and no measurement shows how routing quality or end-to-end answer quality changes with them. Since the paper claims \"efficient knowledge routing and retrieval\" as a key property, the evaluation should include routing accuracy against a labeled set of queries, the overhead of the centroid-based routing compared with LLM-based routing, and the effect of the mix-in and scaling parameters on routing decisions.","section":"Section 3.3.2"},{"comment":"The summarization section claims that the reranking model is \"more reliable and general\" than reciprocal rank fusion, and that the look-back citation strategy is robust and low-latency, but neither claim is measured. In particular, there is no evaluation of citation accuracy, no comparison of the look-back approach against the one-step approach, and no measurement of the latency added by the second-stage citation generation. The paper also notes that \"we tested several approaches\" but reports only the one-step approach and the final look-back design, without data from those tests. These mechanisms are load-bearing for the claims of \"simple but effective filter and reference generation,\" so they need direct experimental support.","section":"Section 3.4"}],"minor_comments":[{"comment":"Reference [13] is titled \"Kimi.ai\" but the URL points to perplexity.ai; the reference should be corrected to the actual source.","section":"References"},{"comment":"References [29] and [30] are duplicate entries for the same Chain-of-Thought paper; one should be removed and the citations merged.","section":"References"},{"comment":"In the \"Knowledge sources configuration\" paragraph, the sentence \"these knowledge sources can be hosted and retrieved locally because the retrieval standard is more\" is incomplete and should be finished.","section":"Section 4.2"},{"comment":"In the \"Pipeline configuration\" paragraph, \"routine mechanism\" should be \"routing mechanism.\"","section":"Section 4.2"},{"comment":"The caption says \"Agent A is roused to conduct knowledge retrieval\"; \"roused\" should be \"routed\" or \"activated.\"","section":"Figure 3 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper is closer to a system demonstration or technical report than a conventional research paper. Its main contributions are architectural and configurational, and the three deployed use cases are plausible evidence of practical usefulness. However, the manuscript currently lacks the experimental grounding that would justify its strong abstract claims. The self-referential nature of the evaluation—use cases on the authors' own AgentScope and ModelScope products, and a routing backbone from a paper with overlapping authors—makes independent validation particularly important. I would encourage the editor to treat the requested experiments as a necessary condition for acceptance rather than a cosmetic addition."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a system paper, not an experimental one. The architecture is clearly described and the integration of known pieces—context rewriting, centroid-based routing, reranking, two-stage citation generation—is sensible. The look-back citation stage is the most interesting bit: generate the answer in stream mode, then do a second pass to assign references. That is a pragmatic workaround for the JSON-output failure mode, and it deserves credit. The writing is also honest about being a technical report, and the three deployments are plausible.\n\nThe soft spots are real and they are in the measurement, not the design. There are no baselines, no ablations, no routing accuracy numbers, no end-to-end comparison. The routing backbone is from the authors' own prior work, and the flagship use cases are their own products, so the evidence is self-referential. The stress-test concern is legitimate: in §3.5, routing runs in parallel with the conversation-context rewrite, meaning the raw user query is used for centroid matching. In a multi-turn follow-up like \"Where can I find the code for it?\", the unresolved pronoun goes straight to routing. If the wrong agents are activated, the later rewrite cannot fix the sources that were never retrieved. The manual mix-ins and score scaling adjust static similarity ranges; they do not inject conversational context. This is not a fatal flaw—maybe in practice the embeddings handle it, or the top-K agents are broad enough—but the paper gives no evidence that it is safe, and the central claim of multi-turn coherency depends on it.\n\nI agree with the reader's UNVERDICTED verdict. The architecture is worth reading, but the paper's claims about accuracy, coherency, and latency are all unsupported by measurement. The lack of released code and data makes verification harder.\n\nWho is this for? Engineers building knowledge-intensive QA chatbots who want design ideas. It is not ready as a citable result. A serious reviewer should engage with it, because the design choices are worth discussing and the look-back citation stage is a practical contribution, but acceptance would require a real evaluation section.\n\nRecommendation: send to peer review, but be clear that it needs quantitative evaluation and an analysis of the routing-rewrite parallelization before it can be accepted.","headline":"A coherent configurable RAG multi-agent system design with a genuinely useful look-back citation trick, but no quantitative evaluation and a real routing/context parallelization tension that the paper leaves unaddressed.","tokens_in":13945,"tokens_out":1337,"would_cite":false,"duration_ms":15872,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"KIMAs is a configurable multi-agent pipeline that enriches queries with conversation and knowledge context, routes each query to the best-suited knowledge sources using embedding-cluster centroids, and generates cited answers with…","keywords":["retrieval-augmented generation","multi-agent systems","query rewrite","knowledge routing","citation generation","configurable pipeline","knowledge-intensive QA","low-latency RAG"],"falsifier":"Take a set of knowledge sources with known topics, plant queries whose answers lie in exactly one source, and check whether the embedding-centroid routing places that source in the top-K activated agents. A second decisive test: use a follow-up question with pronouns whose antecedent is in the conversation history and run the optimized parallel pipeline; if routing activates the wrong source because it used the un-enriched query, the claim that context management improves retrieval in multi-turn conversations fails.","tokens_in":13035,"feed_emoji":"🤖","tokens_out":5399,"duration_ms":50812,"temperature":0.7,"pith_summary":"The paper describes KIMAs, a configurable multi-agent system for building knowledge-intensive question-answering applications on top of large language models (LLMs). Its central claim is that the combination of conversation-aware query rewriting, per-agent knowledge routing by embedding-cluster centroids, reranking-based filtering, and a two-stage look-back citation mechanism lets developers handle heterogeneous knowledge sources without building heavy data pipelines. The system is designed so that each piece can be switched on or off by configuration, and the optimized pipeline runs query routing and context analysis in parallel to keep latency close to simple retrieval-augmented generation (RAG). The authors support the claim by showing how the same framework is configured for three operational applications, from a small repository Q&A bot to a high-throughput social-media bot with sub-ten-second latency.","feed_headline":"KIMAs routes each query to the right knowledge source","feed_subtitle":"Three live deployments show a configurable pipeline handling mixed sources, multi-turn conversation, and citation under ten seconds.","key_machinery":"The load-bearing mechanism is embedding-cluster routing. For each retrieval agent, the embeddings of its knowledge chunks are clustered and the centroids serve as compact synopses of what that agent knows; at query time, the query embedding is compared with all centroids and only the top-K agents are activated for retrieval. The other pillars are the context manager's two rewrites (one for retrieval, one for the summarizer), the retrieval agents' pluggable rewrite strategies (prompt, retrieval, keyword, HyDE, and translation), and the rerank-then-look-back citation pipeline. The parallel execution is what reconciles these sequential-looking stages with low latency.","core_discovery":"On its own terms, the paper's discovery is a system design: KIMAs decomposes a RAG application into three agent types — a context manager that rewrites a user query with conversation context and distills history for the summarizer, retrieval agents that rewrite queries to fit their own knowledge sources, and a summarizer that reranks and filters retrieved chunks and generates the final answer. Knowledge sources are represented by embeddings of their chunks, and routing compares the query embedding to each agent's cluster centroids, with optional developer-supplied mix-in text and score scaling to impose preference. Citation generation uses a look-back strategy: the answer is streamed first, then the model is asked which retrieved chunks support it. The authors report that three production configurations run with reliable performance, including an endpoint that responds in under ten seconds by disabling the context manager and using keyword rewrite.","pith_inferences":["The routing mechanism is not limited to knowledge-source selection: the same centroid-comparison idea could route queries to tools, functions, or specialized agents whose descriptions are embedded, making it a general component-routing primitive.","The paper does not report routing accuracy; an implicit testable claim is that centroid similarity survives the heterogeneity of chunks (code versus prose) after score scaling. A direct measurement of recall@K of correct sources would settle this.","The score-scaling knob is hand-tuned per source; a natural extension is to learn the scaling factors from implicit user feedback (for example, which cited references users click) rather than setting them manually.","The parallelization structure implies that end-to-end latency is governed by the slowest parallel branch, typically retrieval or reranking, so further latency gains would come from accelerating reranking rather than from the LLM calls."],"forward_implications":["A developer can build an application over local vector databases, search-engine APIs, and domain HTTP APIs by editing a configuration file rather than writing a new retrieval stack.","Because routing activates only the top-K agents, cost and latency scale with the number of relevant sources rather than the total number of configured sources.","Disabling the context manager is a usable latency lever; the Olympic use case reports under ten seconds end-to-end with only keyword rewrite and direct summarization.","The look-back citation generation lets the answer stream to the user immediately, with references appearing after a short pause, avoiding the latency and formatting risks of one-shot structured-output citation.","The same agent types and pipeline can be reconfigured for small, large, and turbo-scale applications, which is the paper's evidence of generality."],"supporting_citations":[{"why":"Supplies the embedding-centroid routing algorithm that KIMAs adapts for selecting which retrieval agents to activate.","marker":"[32]"},{"why":"Provides the multi-agent platform infrastructure on which KIMAs is implemented.","marker":"[5]"},{"why":"Provides the vector-database functions KIMAs inherits for local knowledge storage and retrieval.","marker":"[18]"},{"why":"Defines the HyDE query-rewrite strategy that KIMAs includes as one built-in retrieval-agent rewrite option.","marker":"[6]"},{"why":"Serves as one built-in vector database backend for local knowledge retrieval.","marker":"[4]"},{"why":"Presents reciprocal rank fusion, which KIMAs considers and rejects as a reranking method, motivating the reranking-model filter.","marker":"[3]"},{"why":"Documents the difficulty of citation generation that KIMAs's look-back strategy is designed to overcome.","marker":"[7]"}],"fun_headline_variants":["KIMAs: A RAG system that routes, rewrites, and cites","Multi-agent RAG with context-aware query rewriting","KIMAs: Three agents to handle mixed knowledge sources","KIMAs cuts response time with agent pipeline"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The system works only if the routing step reliably picks the knowledge sources that contain the answer, and in the optimized pipeline it makes that choice from the raw user query before conversation context has been added.","fun_headline_variants_meta":{"raw":{"variants":["KIMAs: A RAG system that routes, rewrites, and cites","Multi-agent RAG with context-aware query rewriting","KIMAs: Three agents to handle mixed knowledge sources","KIMAs cuts response time with agent pipeline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000484,"raw_usage":{"total_tokens":2380,"prompt_tokens":929,"completion_tokens":1451,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":1383}},"tokens_in":545,"tokens_out":1451,"duration_ms":11600,"temperature":1.0,"reasoning_tokens":1383,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T20:54:01.951751+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of knowledge sources with known topics, plant queries whose answers lie in exactly one source, and check whether the embedding-centroid routing places that source in the top-K activated agents. A second decisive test: use a follow-up question with pronouns whose antecedent is in the conversation history and run the optimized parallel pipeline; if routing activates the wrong source because it used the un-enriched query, the claim that context management improves retrieval in multi-turn conversations fails.","supporting_citations":[{"cited_title":"Talk to right specialists: Routing and planning in multi-agent system for question answering, 2025","cited_arxiv_id":null,"evidence_quote":"Supplies the embedding-centroid routing algorithm that KIMAs adapts for selecting which retrieval agents to activate."},{"cited_title":"LlamaIndex: Build ai knowledge assistants over your enterprise data","cited_arxiv_id":null,"evidence_quote":"Provides the vector-database functions KIMAs inherits for local knowledge storage and retrieval."},{"cited_title":"Precise zero-shot dense retrieval without relevance labels","cited_arxiv_id":null,"evidence_quote":"Defines the HyDE query-rewrite strategy that KIMAs includes as one built-in retrieval-agent rewrite option."},{"cited_title":"Elasticsearch.software], version, 6(1), 2018","cited_arxiv_id":null,"evidence_quote":"Serves as one built-in vector database backend for local knowledge retrieval."}],"review_version":1}