REVIEW 4 major objections 4 minor 31 references
C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation
T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read Training a 1.5B proxy between retriever and LLM lifts RAG accuracy to 63.5% without touching either component.
desk verdict A novel multi-agent proxy for RAG with tree-rollout credit assignment, but an unnamed LLM judge supplies both the training reward and every reported accuracy, so the effect sizes are unquantified. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the tree-structured rollout with Monte Carlo credit assignment. During training, for each question the Reasoning Router is forced to expand all top-level strategies—[No Retrieval], single-pass [Retrieval], and [Planning]—and then later actions are sampled stochastically with branching factor $K(t)=2$ for depths up to 4 and 1 beyond, building a tree of full trajectories whose leaves carry the system reward ($R=1$ for a correct final answer, $0$ otherwise). Each agent's action node receives the credit reward $r_{credit}(s^i_t, a^i_t) = (1/|L|)\sum_{\ell \in L} R_\ell$, the average of the leaf rewards reachable from that node, and these expectation-based credits are fed into PPO. This converts a single sparse question-level reward into dense per-action supervision and redistributes sampling effort from the question level to the action level.
What would settle it
Run the exact same trained proxies and baselines on the six in-domain datasets, but score answers with exact match after light normalization or with human raters blind to which system produced each answer; if the C-3PO advantage shrinks to near zero or reverses on single-hop sets, then the reported gains come from judge acceptance rather than from the credit-assignment training.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that misalignment between an independently built retriever and LLM is addressable as a learned communication problem: a single small proxy model, prompted with different role instructions, can act as all three go-betweens and be optimized end-to-end against the final answer quality. The proxy chooses among direct answering, single-pass retrieval, and a planning-driven iterative loop, so the LLM is called only once for the plan and once for the final answer. Training uses multi-agent PPO where the retriever and LLM are treated as a fixed environment, and the paper's proposed tree-structured rollout with Monte Carlo credit assignment is what makes joint optimization stable, converting one sparse system-level 0/1 reward into dense per-action expected rewards. With that procedure the trained 0.5B and 1.5B proxies outperform all compared baselines on average, and hold up when the retriever, datasets, and LLM server are swapped for unseen ones.
Load-bearing premise
The reported accuracies all rest on an LLM judge that decides whether the predicted answer is semantically equivalent to the gold answer, and the paper does not name that judge or report how often it agrees with human raters; if that judge only prefers the proxy's longer, structured outputs, the measured gains overstate real correctness.
Editorial extensions
If this is right
- A trained proxy can act as a drop-in adapter for closed or commercial RAG stacks: it requires no retriever or LLM fine-tuning and keeps working when the retriever is replaced by Google Search and the LLM by models such as Llama3.3-70B or GPT-4o-mini.
- Multi-hop reasoning is where the method pays off most, with reported gains of +15.5 points on 2WikiMultiHopQA, +13.2 on HotpotQA, and +12.9 on MuSiQue over the strongest baseline.
- The tree-structured rollout makes RL training stable enough that, unlike single-trajectory reward assignment, it reliably improves on the supervised warm-up model.
- The whole adapter can be a 0.5B or 1.5B model adding about 4.8 seconds per question, which the paper argues is practical for edge deployment.
Reading between the lines
- Editorial inference: the same tree-rollout credit assignment should transfer to other multi-stage language-agent pipelines—tool selection, web browsing, or code repair—where the only reward signal arrives at the end.
- Editorial inference: because all rewards come from an LLM judge, a controlled study that varies only output formatting (forcing baselines to emit the same Thought/Action traces) would separate content gains from presentation gains.
- Editorial inference: the training dynamics show the [Planning] strategy rising to roughly 60-70% of decisions, so an ablation that removes the LLM-generated roadmap would test how much of the multi-hop gain depends on the plan rather than on the proxy's retrieval decisions.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes C-3PO, a proxy-centric alignment framework for retrieval-augmented generation. A small proxy model implements three agents (Reasoning Router, Information Filter, and Decision Maker) that decide whether retrieval is needed, generate and rewrite queries, filter retrieved documents, and plan multi-step retrieval, while the retriever and the LLM remain frozen. The proxy is trained with a tree-structured rollout and Monte Carlo credit assignment under PPO, with the system-level reward determined by an LLM judge. Experiments on six in-domain datasets and two out-of-distribution datasets with unseen retrievers and LLMs report large gains over baselines (average 62.08% with a 0.5B proxy and 63.53% with a 1.5B proxy, versus 52.71% for the strongest full-coverage baseline), plus an HLE score of 5.79%.
Significance. If the reported results hold under a transparent evaluation, C-3PO is a valuable plug-and-play method: it avoids modifying the retriever or the LLM, uses only a 0.5B/1.5B proxy, and shows consistent gains across datasets and OOD settings, with code released. The Monte Carlo credit assignment in Eq. (5) is a sensible mechanism for distributing sparse system-level rewards, and the ablations support the value of the tree-structured rollout. The main significance risk is that every reported number, including the RL reward itself, depends on an unidentified LLM judge, so the measured gains could partly reflect judge acceptance rather than answer correctness. The HLE claim also lacks a same-server Standard RAG baseline, making the comparison to base and proprietary models difficult to interpret.
major comments (4)
- [Appendix D.2; Sections 5.2 and 6.2] The evaluation metric is an LLM judge that compares predicted answers with gold answers for semantic equivalence, but the judge model, its decoding parameters, and any human-agreement study are not reported. The same binary judgment appears to be the system-level reward used in Eq. (5), so the proxy is optimized to maximize this judge's acceptance. If the judge is biased toward verbose or structured outputs, or is from the same model family as the generator, the gains in Tables 1 and 2 could be inflated. Please identify the judge, report agreement against human labels on a sample of each benchmark, and re-score at least a subset with a different judge or with human annotators.
- [Section 6.6, Table 4] The HLE evaluation compares C-3PO with Qwen2.5-72B against the base model without retrieval and against proprietary models, but it does not include a Standard RAG baseline using the same LLM server and the same Google Search retriever. Without such a baseline, the improvement from 4.27% to 5.79% cannot be attributed to C-3PO's proxy; part of the gain could come from retrieval itself. Please add Standard RAG and, if feasible, an intermediate-module baseline under the same server/retriever configuration.
- [Section 5.1, Eqs. (1), (5), (9)] The deterministic rollout forces the Reasoning Router to take [No Retrieval], [Retrieval], and [Planning] actions that are not sampled from the policy pi_theta, and Section 5.2 states that all nodes are added to a replay buffer for PPO. The PPO loss in Eq. (9) uses importance ratios with respect to pi_theta_old as if the data were on-policy. Please clarify whether forced branches are included in the policy-gradient updates and how the behavior distribution for those branches is accounted for; if they are included without an off-policy correction, the 'on-policy' description in Section 5 is inaccurate.
- [Section 6.2, Tables 1-2] No variance or significance testing is reported. Given that the differences on single-hop datasets are relatively small (about +1.5 to +3.5 percentage points) and the evaluation is an automated judge, it would strengthen the paper to report standard errors over at least three independent runs or bootstrap resamples, especially for the headline averages.
minor comments (4)
- [Table 1] The 'Average' column is computed over different dataset subsets for different baselines (e.g., Self-RAG 72B averages only 3 datasets, with value 58.33). This makes the claimed comparison against SKR-KNN's full-coverage average of 52.71 misleading. Please report averages over a common subset or clearly mark the number of datasets included in each average.
- [Appendix A.3 and Section 6.5] The text alternates between 'out-of-distribution' and 'out-of-generalization'; please unify the terminology throughout the paper.
- [Appendix B.1 and Figure 6] There are several typos: 'we details' should be 'we detail', 'fo each agent' should be 'for each agent', and 'displaies' in the Figure 6 caption should be 'displays'.
- [Figure 6] The density plots for NaturalQuestions and PopQA have y-axis values above 1, which is possible for a density but can confuse readers; consider using a probability mass function over depths or adding a note explaining that these are density values.
Circularity Check
No circular derivation: C-3PO's credit assignment is a standard Monte Carlo estimator, and its self-citations are not load-bearing.
full rationale
The paper's derivation chain is self-contained with respect to circularity. The central technical component, Monte Carlo credit assignment in Eq. (5), estimates each agent-node reward as the average system-level reward over reachable leaf nodes; this is a standard expectation estimator, not a fitted quantity that is later renamed as a prediction. The PPO objective in Eqs. (6)-(11) is a conventional RL formulation, and the reported in-domain and out-of-distribution accuracies come from held-out test questions and unseen retrievers/LLMs, so the main generalization claim is not forced by the training fit. The self-citations (Chen et al. 2024a, 2024b, 2025) appear only in general background citation lists for RAG and multi-agent systems and are not used as load-bearing evidence for any uniqueness claim or core derivation. One evaluation-validity concern exists: Appendix D.2 describes an LLM-based semantic-equivalence judge, and the same kind of judge supplies the binary system-level reward used in RL training, while its identity and human-agreement rate are not reported. However, this is a potential measurement-bias issue rather than a circularity: optimizing toward a metric and then reporting that metric is standard practice, and the paper does not define its conclusion in terms of the training reward or fit the judge to the test outputs. Therefore, no circular step can be exhibited from the paper's own equations, and the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (6)
- Branching factor K(t) =
2 for depth <=4, 1 for depth >4
- Maximum tree depth =
13
- Warm-up epochs =
3
- PPO learning rates =
policy 5e-7, value 5e-6
- KL coefficient =
0.005
- Number of training questions per dataset =
6000
assumptions (4)
- standard math PPO and GAE provide unbiased policy updates
- domain assumption Monte Carlo estimate in Eq. (5) is an unbiased estimator of conditional expected reward
- domain assumption The three-agent decomposition (router, filter, decision maker) is sufficient to capture the coordination needed for RAG
- domain assumption The LLM judge in Appendix D.2 measures answer correctness faithfully
Cite this review
Pith. "Pith review of C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation." pith.science (2026). https://pith.science/paper/YDE5R2BO
@misc{pith2026250206205,
author = {Pith},
title = {Pith review of: C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/YDE5R2BO}},
note = {Machine review of arXiv:2502.06205}
}
read the original abstract
Retrieval-augmented generation (RAG) systems face a fundamental challenge in aligning independently developed retrievers and large language models (LLMs). Existing approaches typically involve modifying either component or introducing simple intermediate modules, resulting in practical limitations and sub-optimal performance. Inspired by human search behavior -- typically involving a back-and-forth process of proposing search queries and reviewing documents, we propose C-3PO, a proxy-centric framework that facilitates communication between retrievers and LLMs through a lightweight multi-agent system. Our framework implements three specialized agents that collaboratively optimize the entire RAG pipeline without altering the retriever and LLMs. These agents work together to assess the need for retrieval, generate effective queries, and select information suitable for the LLMs. To enable effective multi-agent coordination, we develop a tree-structured rollout approach for reward credit assignment in reinforcement learning. Extensive experiments in both in-domain and out-of-distribution scenarios demonstrate that C-3PO significantly enhances RAG performance while maintaining plug-and-play flexibility and superior generalization capabilities.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Consider the specificity, complexity, and clarity of the question
**Evaluate the Question**: Assess whether a precise answer can be provided based on the existing knowledge of LLM. Consider the specificity, complexity, and clarity of the question
-
[2]
Guo, T., Chen, X., Wang, Y ., Chang, R., Pei, S., Chawla, N
URL https://openreview.net/forum?id= 4jqOV6NlUz. Guo, T., Chen, X., Wang, Y ., Chang, R., Pei, S., Chawla, N. V ., Wiest, O., and Zhang, X. Large language model based multi-agents: A survey of progress and challenges. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI 2024, Jeju, South Korea, August 3-9, 20...
work page 2024
-
[3]
Ho, X., Duong Nguyen, A.-K., Sugawara, S., and Aizawa, A
URL https://www.ijcai.org/proceedings/ 2024/890. Ho, X., Duong Nguyen, A.-K., Sugawara, S., and Aizawa, A. Constructing a multi-hop QA dataset for compre- hensive evaluation of reasoning steps. In Scott, D., Bel, N., and Zong, C. (eds.), Proceedings of the 28th Inter- national Conference on Computational Linguistics, pp. 6609–6625, Barcelona, Spain (Onlin...
arXiv 2024
-
[4]
URL https: //doi.org/10.48550/arXiv.2410.21276
doi: 10.48550/ARXIV .2410.21276. URL https: //doi.org/10.48550/arXiv.2410.21276. Izacard, G., Caron, M., Hosseini, L., Riedel, S., Bojanowski, P., Joulin, A., and Grave, E. Unsupervised dense infor- mation retrieval with contrastive learning. Trans. Mach. Learn. Res., 2022, 2022. URL https://openreview. net/forum?id=jKN1pXi7b0. Joshi, M., Choi, E., Weld, ...
-
[5]
The predicted answer is considered correct if it matches ANY ONE of the provided correct answers
-
[6]
The matching can be exact or semantically equivalent to any of the correct answers
-
[7]
URL https: //doi.org/10.48550/arXiv.2403.08295
doi: 10.48550/ARXIV .2403.08295. URL https: //doi.org/10.48550/arXiv.2403.08295. Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V ., Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., Button, K., Knight, M., Chess, B., and Schulman, J. Webgpt: Browser-assisted question-answering with hu- man fee...
-
[13]
**Decision Categories:** - If the question is complex and requires a planning phase before retrieval, your response should be: [Planning] - If the question requests specific information that you believe the LLM does not possess or pertains to recent events or niche topics outside LLM’s knowledge scope, format your response as follows: [Retrieval] ‘YOUR QU...
Show all 31 references
-
[14]
Concentrate solely on determining the necessity for retrieval or planning
**Focus on Assessment**: Avoid providing direct answers to the questions. Concentrate solely on determining the necessity for retrieval or planning. State of Reasoning Router Now, process the following question: Question: {question} Output (All possible Actions) of Reasoning R...
-
[15]
**Analyze Relevance**: Evaluate each document whether it aligns with the objectives of the current retrieval step and contains a direct answer to the question
-
[16]
**Thought Process**: Provide a brief analysis for each document, considering both the answer content and the retrieval objectives
-
[17]
**Filter Documents**: After your thought process, generate a list of document indices indicating which documents to retain. State of Information Filter Now, process the following question: Current step’s objectives: {objective} (only for [Planning] mode) Question: {question} D...
-
[18]
**Evaluate Existing Documents**: Assess the existing documents to determine if it is sufficient to answer the question
-
[19]
**Follow the Plan**: Understand the next steps outlined in the plan
-
[20]
**Decision Categories:** - If the existing documents is insufficient and requires additional retrieval, respond with: [Retrieval] ‘YOUR QUERY HERE‘ - If the existing documents is adequate to answer the question, respond with: [LLM]
-
[21]
**Focus on Action**: Do not answer the question directly; concentrate on identifying the next appropriate action based on the existing documents, plan, and question. State of Decision Maker Now, process the following question: Existing Documents: {accumulated documents} Roadma...
-
[22]
Determine what key information we currently know and what additional information is needed through retrieval
**Question Analysis**: Identifying the core components of the question. Determine what key information we currently know and what additional information is needed through retrieval
-
[23]
Focus on the planning process rather than providing direct answers
**Step By Step Planning**: Develop a detailed plan step by step. Focus on the planning process rather than providing direct answers
-
[24]
**Focus on Planning**: Keep your response clear and structured, concentrating solely on the analysis and planning aspects. Now, process the following question: Question: {question} Example of generated roadmap (Take What nationality is the director of film The Caper Of The Gol...
2023
-
[25]
Consider semantic equivalence, not just exact string matching
-
[26]
Ignore minor differences in formatting, spacing, or capitalization
-
[27]
For numerical answers, consider acceptable margin of error if applicable
-
[28]
For text answers, focus on the core meaning rather than exact wording
-
[31]
True” if the predicted answer is correct, or “False
Return only “True” if the predicted answer is correct, or “False” if it is incorrect. Now, process the following question: Question: {question} Correct Answer: {true answer} Predicted Answer: {long answer} 22
-
[463]
naacl-long.463
URL https://doi.org/10.18653/v1/2024. naacl-long.463. Sun, H.-L., Sun, Z., Peng, H., and Ye, H.-J. Mitigating visual forgetting via take-along visual conditioning for multi-modal long cot reasoning. In ACL, 2025a. Sun, H.-L., Zhou, D.-W., Li, Y ., Lu, S., Yi, C., Chen, Q.-G., ...
2024 doi
-
[691]
findings-emnlp.691
URL https://doi.org/10.18653/v1/2023. findings-emnlp.691. Wei, Z., Chen, W.-L., and Meng, Y . InstructRAG: Instruct- ing retrieval-augmented generation via self-synthesized rationales. In The Thirteenth International Conference on Learning Representations , 2025. URL https:// ...
-
[2017]
doi: 10.18653/v1/P17-1147
Association for Computational Linguistics. doi: 10.18653/v1/P17-1147. URL https://aclanthology. org/P17-1147/. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A. P., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelc...
-
[2018]
doi: 10.18653/v1/D18-1259
Association for Computational Linguistics. doi: 10.18653/v1/D18-1259. URL https://aclanthology. org/D18-1259/. Yu, T., Zhang, S., and Feng, Y . Auto-rag: Au- tonomous retrieval-augmented generation for large lan- guage models. CoRR, abs/2411.19443, 2024a. doi: 10. 48550/ARXIV ...
-
[2023]
URL https: //doi.org/10.48550/arXiv.2308.03281
doi: 10.48550/ARXIV .2308.03281. URL https: //doi.org/10.48550/arXiv.2308.03281. Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net,...
-
[2024]
URL https: //doi.org/10.48550/arXiv.2407.21783
doi: 10.48550/ARXIV .2407.21783. URL https: //doi.org/10.48550/arXiv.2407.21783. Fan, W., Ding, Y ., Ning, L., Wang, S., Li, H., Yin, D., Chua, T., and Li, Q. A survey on RAG meeting llms: Towards retrieval-augmented large language models. In Baeza- Yates, R. and Bonchi, F. (e...
-
[4436]
doi: 10.18653/V1/2024.ACL-LONG.242
Association for Computational Linguistics, 2024. doi: 10.18653/V1/2024.ACL-LONG.242. URL https: //doi.org/10.18653/v1/2024.acl-long.242. 12 C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation Tang, Y . and Yang, Y . Multihop-rag...
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.