Scaling External Knowledge Input Beyond Context Windows of LLMs via Multi-Agent Collaboration

Fei Huang; Ming Yan; Peng Li; Yang Liu; Zhennan Wan; Zijun Liu

arxiv: 2505.21471 · v2 · submitted 2025-05-27 · 💻 cs.CL

Scaling External Knowledge Input Beyond Context Windows of LLMs via Multi-Agent Collaboration

Zijun Liu , Zhennan Wan , Peng Li , Ming Yan , Fei Huang , Yang Liu This is my paper

Pith reviewed 2026-05-19 12:40 UTC · model grok-4.3

classification 💻 cs.CL

keywords multi-agent collaborationLLM context windowsexternal knowledge integrationinference-time scalingmulti-hop question answeringagent orchestrationparallel processing

0 comments

The pith

Multi-agent coordination lets LLMs integrate external knowledge beyond their context windows without training.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to demonstrate that current limits on how much retrieved knowledge an LLM can use stem from fixed context windows and from specific problems in how multiple agents are currently orchestrated. By identifying two core bottlenecks in existing multi-agent designs, the authors build ExtAgents, a framework that coordinates agents to distribute and process large knowledge inputs in parallel at inference time. This approach avoids the information loss that comes from extending a single model's context and delivers stronger results than other non-training methods on the same volume of knowledge. Benchmarks including an enhanced multi-hop question answering test and long survey generation show the gains hold whether the total input fits inside one context window or greatly exceeds it, while parallelism keeps runtime efficient.

Core claim

ExtAgents is a multi-agent framework that overcomes two identified bottlenecks in prior agent orchestration designs, enabling scalable integration of external knowledge at inference time without longer-context training and producing higher performance than existing non-training methods on the same knowledge volume, whether that volume lies inside or outside the model's context window.

What carries the argument

ExtAgents, the multi-agent framework whose coordination mechanisms distribute external knowledge across agents for parallel processing.

Load-bearing premise

The two core bottlenecks in existing agent orchestration are the main obstacles to scaling knowledge input, and the new coordination mechanisms fix them without adding offsetting errors or latency.

What would settle it

A direct test on the enhanced multi-hop QA benchmark where knowledge input exceeds the context window and the coordination mechanisms are removed or replaced shows no remaining performance gain over baseline non-training methods.

read the original abstract

With the rapid advancement of post-training techniques for reasoning and information seeking, large language models (LLMs) can incorporate a large quantity of retrieved knowledge to solve complex tasks. However, the limited context window of LLMs obstructs scaling the amount of external knowledge input, prohibiting further improvement. Existing context window extension methods inevitably cause information loss. LLM-based multi-agent methods emerge as a new paradigm to handle massive input in a distributional manner, where we identify two core bottlenecks in existing agent orchestration designs. In this work, we develop a multi-agent framework, \textbf{\ExtAgents}, to overcome the bottlenecks and enable better scalability in inference-time knowledge integration without longer-context training. Benchmarked with our enhanced multi-hop question answering test, \textbf{$\boldsymbol{\infty}$Bench+}, and other public test sets including long survey generation, \ExtAgents significantly enhances the performance over existing non-training methods with the same amount of external knowledge input, regardless of whether it falls \emph{within or exceeds the context window}. Moreover, the method maintains efficiency due to high parallelism. We believe further study in the coordination of LLM agents on increasing external knowledge input could benefit real-world applications.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

ExtAgents gives a training-free way to feed LLMs more external knowledge than the context window holds by fixing two orchestration bottlenecks in multi-agent setups, with reported gains on multi-hop QA.

read the letter

The core takeaway is that this paper shows how to scale external knowledge input past the context limit using a multi-agent system called ExtAgents, without any extra training. It splits the knowledge across agents and adds coordination steps to handle the distribution, then tests it on an enhanced multi-hop QA benchmark plus long survey generation. The claim is that it beats other non-training baselines whether the total knowledge fits in the window or not, while keeping things efficient through parallelism.

Referee Report

2 major / 2 minor

Summary. The manuscript proposes ExtAgents, a multi-agent collaboration framework to scale external knowledge input for LLMs beyond context window limits without longer-context training. It identifies two core bottlenecks in existing agent orchestration designs and develops coordination mechanisms to enable distributed knowledge integration. The work introduces an enhanced multi-hop QA benchmark ∞Bench+ and evaluates on this plus public datasets for tasks including long survey generation, claiming significant performance gains over existing non-training methods with equivalent external knowledge input, whether inside or outside the context window, while preserving efficiency via high parallelism.

Significance. If the results hold, the contribution could be meaningful for inference-time scaling of knowledge integration in LLMs via multi-agent systems, offering an alternative to context extension techniques that incur information loss. The emphasis on coordination to handle distributed facts in multi-hop settings addresses a practical bottleneck, and the new ∞Bench+ benchmark may support further work. Credit is due for focusing on non-training methods and parallelism for efficiency. However, the moderate soundness rating and absence of detailed ablations limit the assessed impact pending stronger verification of the coordination robustness.

major comments (2)

[Abstract; Method section describing coordination protocol] The central claim that ExtAgents' coordination mechanisms address the two bottlenecks without introducing offsetting integration errors or incomplete reasoning paths is load-bearing for the scalability assertion (abstract and method description). The skeptic concern that inter-agent communication may fail to synthesize cross-chunk facts in multi-hop QA is not yet dispelled by the reported evidence; without explicit analysis of relevance signal propagation or error rates in message passing, gains over chunked single-agent baselines remain unverified for out-of-window inputs.
[Experiments and results section] Evaluation on ∞Bench+ and public sets reports performance enhancements, but the review notes moderate support due to missing full experimental details, ablations, and error analysis. This weakens the claim of consistent superiority 'regardless of whether it falls within or exceeds the context window' until such controls are provided to rule out confounding factors like prompt engineering or agent count.

minor comments (2)

[Introduction] Clarify the exact definitions of the two core bottlenecks early in the introduction with concrete examples from prior agent orchestration work to improve readability.
[Throughout manuscript] Ensure all benchmark names (e.g., ∞Bench+) and method names (ExtAgents) are formatted consistently in bold or italics across sections and figures.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our work. We provide detailed responses to the major comments and indicate revisions to address the raised concerns.

read point-by-point responses

Referee: The central claim that ExtAgents' coordination mechanisms address the two bottlenecks without introducing offsetting integration errors or incomplete reasoning paths is load-bearing for the scalability assertion (abstract and method description). The skeptic concern that inter-agent communication may fail to synthesize cross-chunk facts in multi-hop QA is not yet dispelled by the reported evidence; without explicit analysis of relevance signal propagation or error rates in message passing, gains over chunked single-agent baselines remain unverified for out-of-window inputs.

Authors: We recognize the need for stronger verification of the coordination mechanisms' ability to synthesize cross-chunk facts. Our results on ∞Bench+ show that ExtAgents outperforms chunked single-agent baselines on multi-hop QA tasks, which inherently require effective propagation of relevance signals across agents. This performance differential supports that the mechanisms mitigate integration errors. To further dispel concerns, we will add an explicit analysis of message passing, including relevance signal tracking and error rate estimation, in the revised manuscript. revision: partial
Referee: Evaluation on ∞Bench+ and public sets reports performance enhancements, but the review notes moderate support due to missing full experimental details, ablations, and error analysis. This weakens the claim of consistent superiority 'regardless of whether it falls within or exceeds the context window' until such controls are provided to rule out confounding factors like prompt engineering or agent count.

Authors: We agree that additional details and controls would bolster the claims. In the revised manuscript, we will provide fuller experimental details, include ablations varying agent counts and prompt engineering approaches, and incorporate error analysis. These additions will help confirm that the observed superiority holds consistently for both in-context and out-of-context window scenarios, independent of the mentioned confounding factors. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical framework with independent benchmark validation

full rationale

The paper introduces ExtAgents as an engineering solution to two identified bottlenecks in multi-agent orchestration for scaling external knowledge beyond LLM context windows. Claims rest on direct performance comparisons against existing non-training methods using the enhanced ∞Bench+ multi-hop QA benchmark and other public datasets, with results reported for both in-window and out-of-window inputs. No equations, fitted parameters, or predictions are defined in terms of the target outcomes; the coordination mechanisms are presented as novel design choices whose efficacy is measured externally rather than assumed by construction. No self-citation chains or uniqueness theorems are invoked as load-bearing premises in the provided description. The derivation chain is therefore self-contained and non-circular.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The contribution is an empirical framework rather than a mathematical derivation; no explicit free parameters, axioms, or invented entities are introduced beyond standard LLM and agent-system assumptions.

pith-pipeline@v0.9.0 · 5743 in / 1032 out tokens · 34435 ms · 2026-05-19T12:40:19.098430+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Foundation/RealityFromDistinction.lean reality_from_one_distinction unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

We develop a multi-agent framework, ExtAgents, to overcome the bottlenecks and enable better scalability in inference-time knowledge integration... featuring two key components: global knowledge synchronization... and knowledge-accumulating reasoning, which gradually integrates and increases the updated knowledge from Seeking Agents to Reasoning Agent throughout multiple rounds of reasoning.
IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

the bandwidth of Chain of Agents and LongAgent is 2, and the bandwidth of LLM×MapReduce is O(L/|m|)... ExtAgents implements global knowledge synchronization... Topk(Mt) = arg max ...

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
cs.SE 2026-04 accept novelty 5.0

LLM agent progress depends on externalizing cognitive functions into memory, skills, protocols, and harness engineering that coordinates them reliably.
A Multi-Agent Framework for Automated Exploit Generation with Constraint-Guided Comprehension and Reflection
cs.SE 2026-04 unverdicted novelty 5.0

Vulnsage, a multi-agent framework, generates 34.64% more exploits than prior tools and verified 146 zero-day vulnerabilities in real-world open-source libraries.