REVIEW 4 major objections 7 minor 1 cited by
PaperPilot treats multi-turn scientific literature search as building and editing executable DAGs of paper-search operators, and a 9B model trained this way beats its base toolset agent on retrieval while cutting workflow errors to zero.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-12 09:23 UTC pith:BNE6FY6L
load-bearing objection Solid systems paper: editable paper-search DAGs plus corruption-based training stabilize multi-turn tool use for a 9B model; main caveat is gold-aware simulation, not a broken claim. the 4 major comments →
Multi-Turn Agentic Scientific Literature Search via Workflow Induction
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Framing multi-turn scientific literature search as workflow induction—constructing and refining an executable DAG of typed paper-search operators from an anchor paper, query, and interaction history—gives a controllable interface that improves retrieval quality and eliminates workflow execution errors for a 9B model trained with supervised workflow imitation and preference optimization over controlled corruptions, relative to the same model using the toolset without that training.
What carries the argument
Workflow induction: an executable directed acyclic graph whose nodes are typed paper-search operators (keyword search, citation expansion, filtering, scoring, reranking, evidence extraction, etc.) and whose edges carry intermediate paper sets, scores, keywords, or evidence; user feedback is applied as structural and parametric edits to this DAG rather than as extra query text.
Load-bearing premise
The controlled multi-turn protocol—hidden gold papers built from citation graphs and related-work cohorts, plus a fixed large-model user simulator under leakage control—is a faithful enough stand-in for real researchers’ evolving intent that the measured multi-turn gains will transfer outside the benchmark.
What would settle it
Run the same multi-turn protocol with real researchers (not a simulator) on the same anchor–query cases and check whether PaperPilot-9B still improves Hit@5, MRR, and nDCG@10 over the untrained 9B toolset agent while keeping workflow execution errors near zero; a clear drop or reversal of those gains would falsify the central claim about controllable alignment with complex scientific intent.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes PaperPilot, a multi-turn scientific literature search agent that frames retrieval as induction and refinement of an executable DAG over a typed operator library (keyword search, citation expansion, filtering, scoring, reranking, evidence extraction). Given an anchor paper and query, the agent induces a workflow, executes it, and edits structure/parameters from user feedback. PaperPilot-9B is trained with supervised imitation of teacher workflows (filtered by gold-in-top-5 success) plus IPO-style preference optimization on controlled workflow corruptions. On a hold-out benchmark spanning five search directions, multi-turn PaperPilot-9B improves over the base Qwen3.5-9B toolset agent (Hit@5 58.0→77.0, MRR 47.5→59.4, nDCG@10 26.8→32.5) and reduces workflow execution errors from 9.5% to 0%, with additional workflow-generation/editing ablations, candidate-pool sensitivity, cost comparisons, and a small human study.
Significance. If the multi-turn gains transfer beyond the controlled protocol, the work offers a practical control interface for agentic literature search: inspectable, editable DAGs rather than opaque free-form tool traces or fixed pipelines. The combination of a typed paper-search toolset, explicit Induce/Refine/Execute loop, and training via imitation plus corruption-based preference pairs is a concrete systems contribution. Strengths include a reproducible hold-out protocol, workflow-level metrics (Table 3; Figs. 6–7), cost-effectiveness reporting, honest limitations, and a human study that at least partially grounds the claim outside pure simulation. The paper is timely for agentic IR and scientific assistants, even if absolute SOTA retrieval is not claimed for the 9B model alone.
major comments (4)
- §3.1–3.2 and App. C.2: The central multi-turn claim (Abstract; Fig. 5; Table 2) rests on a fixed Qwen3.5-397B-A17B user simulator conditioned on hidden gold-paper metadata, with leakage controls that block titles/authors but still allow preference-level answers. Because gold sets are built from citation-graph signals, human filtering, LLM-assisted synthesis, and related-work cohorts (App. C.1), simulator feedback may systematically favor operator edits (year filters, citation direction, NLI axes) that PaperPilot is designed to apply. The human study (Table 4; App. C.8) is encouraging but small (six volunteers, five sessions) and insufficient alone to establish transfer. Please either (i) report a larger real-user evaluation with the same metrics, (ii) ablate gold-access vs gold-blind simulation, or (iii) narrow the claim to “under controlled multi-turn protocols with leakage-controlled s
- Table 2 / Fig. 4–5: For Qwen3.5-9B Toolset, multi-turn interaction degrades Hit@5 (69.0→58.0) while workflow execution errors rise (2.0%→9.5%); PaperPilot-9B restores Hit@5 to 77.0 with 0% errors. The headline improvement therefore confounds two effects: (a) learning to emit valid executable DAGs under refinement, and (b) learning better search strategies. Without a breakdown of retrieval quality restricted to non-errored base-model runs, or an oracle that repairs only syntax/type errors of the base agent, it is unclear how much of the claimed “workflow induction” benefit is error-rate reduction versus intent-aligned operator composition. Please add this disentangling analysis; it is load-bearing for attributing gains to editable workflows rather than to execution robustness alone.
- Table 2: PaperPilot-9B is reported only in the multi-turn “Workflow + Toolset” row; there is no single-turn PaperPilot-9B result and no multi-turn fixed-workflow PaperPilot-9B result. Given that adaptive toolset use already helps untrained models in single-turn (e.g., Qwen3.5-9B Workflow 48.5 vs Toolset 69.0 Hit@5), the contribution of training versus multi-turn refinement versus toolset access cannot be fully separated. Please add the missing single-turn (and, if feasible, multi-turn fixed-workflow) PaperPilot-9B numbers under the same protocol so the training effect is isolated.
- §2.3 Training data construction: Supervision retains teacher turns where the gold paper appears in the top-5 under direction-specific success conditions (5,540 examples), then builds preference pairs by corrupting those successful workflows (1,733 hard pairs). This is reasonable for imitation learning, but it couples supervision quality to the same gold construction used at evaluation and may bias the model toward teacher strategies that already exploit citation-graph structure. Please report (i) teacher success rates by direction, (ii) how often rejected corruptions would still retrieve gold under execution, and (iii) whether preference optimization improves retrieval metrics beyond SFT alone (SFT-only checkpoint on the hold-out set).
minor comments (7)
- Table 1 capability matrix uses strict ✓/△/× definitions; a short footnote defining each column’s inclusion criteria (already partly in Related Work) would reduce ambiguity for systems marked △.
- Fig. 3 action distribution: n decreases sharply after turn 7; clarify whether percentages are conditional on remaining active cases and whether early finalize is treated as success or truncation.
- §4.2 / Fig. 8 / Table 6: Sensitivity over K is useful; state whether K is total first-stage candidates or per-source top-k, and whether rerank budget is held fixed when K grows.
- App. C.6: Report base model identity for PaperPilot-9B (presumably Qwen3.5-9B), LoRA rank/alpha, and total training tokens or wall-clock for reproducibility.
- Notation: Gt, Pt, Ht appear in §2.2 before the full formalization in App. A.2; a brief forward pointer would help.
- Typos/consistency: “PAPERPILOT” vs “PaperPilot” casing varies; “Qwen3.5-397B-A17B” vs “Qwen3.5-397B” should be standardized; arXiv id 2607.00597v2 date line is fine but ensure camera-ready author list matches contributions footnote.
- Human study App. C.8: Specify interface (Figs. 9–10), blinding, and whether participants saw the same anchors/queries across systems; report variance or per-participant ranges for Table 4.
Circularity Check
No derivation-style circularity: empirical systems results on hold-out retrieval, not predictions forced by fitted inputs or self-definition.
full rationale
PaperPilot is an empirical multi-turn IR/agent systems paper. Its load-bearing claims are measured retrieval metrics (Hit@5, MRR, nDCG, workflow error rate) on a hold-out benchmark under a fixed multi-turn protocol, comparing a trained 9B toolset agent to the untrained base and other baselines (Table 2; Fig. 5). Training is standard SFT on teacher workflows filtered for gold-in-top-5 success plus IPO-style preference pairs from controlled structural/semantic corruptions (§2.3)—that is imitation and preference learning, not a claim that a fitted constant predicts a quantity defined by the same fit. There is no self-definitional equation chain, no uniqueness theorem imported from the authors to forbid alternatives, no ansatz smuggled in as a first-principles result, and no renaming of a known closed-form law as a new derivation. Self-citations in Related Work are ordinary positioning, not load-bearing premises that force the headline gains. Concerns about gold construction (citation graph + LLM synthesis) and a gold-conditioned user simulator (App. C.1–C.2) are evaluation-validity / transfer issues, not circular reductions of the form Eq. X ≡ Eq. Y by construction. Per the analyzer rules, that is not circularity; score 0 with empty steps is the correct finding.
Axiom & Free-Parameter Ledger
free parameters (4)
- SFT learning rate / epochs / sequence length =
2e-4, 3 epochs, 14336
- IPO/DPO beta and preference-stage LR =
β=0.2, LR 3e-5
- First-stage candidate pool scale K =
base ~8–10
- Max clarification turns / top-k cutoffs =
4–5 clarifications; top-50 lists
axioms (5)
- domain assumption Scientific multi-turn intent can be adequately expressed as edits over a fixed library of typed paper-search operators composed as a DAG.
- domain assumption Five discrete search directions (predecessor, successor, sibling, benchmark, survey) cover the evaluation intents of interest.
- ad hoc to paper Teacher trajectories that place gold papers in top-5 under direction-specific success conditions are high-quality workflow supervision.
- ad hoc to paper An LLM user simulator with gold access but leakage controls provides valid multi-turn feedback for ranking systems.
- domain assumption Standard next-token SFT plus IPO-style preference optimization on corrupted workflows improves executable workflow editing.
invented entities (3)
-
PAPERPILOT-TOOLSET (typed paper-search operator library)
independent evidence
-
Workflow-induction framing for multi-turn literature search (Induce/Refine/Execute loop)
no independent evidence
-
Controlled workflow-corruption preference pairs
no independent evidence
read the original abstract
Scientific literature search often requires more than retrieving papers from a single query: users' intents are underspecified, preference-dependent, and evolve through interaction. Existing search agents typically rely on fixed pipelines or implicit language-only reasoning, making their search strategies difficult to control, inspect, and refine. We introduce PaperPilot, a multi-turn literature search agent that frames scientific search as workflow induction. Given an anchor paper and a user query, PaperPilot constructs an executable DAG of paper-search operators, including keyword search, citation expansion, filtering, scoring, reranking, and evidence extraction. User feedback is then used to refine both the query and the workflow itself. We train PaperPilot with supervised workflow imitation and preference optimization over controlled workflow corruptions. Experiments show that PaperPilot-9B improves over the base Qwen3.5-9B toolset agent under multi-turn interaction, increasing Hit@5 from 58.0 to 77.0, MRR from 47.5 to 59.4, and nDCG@10 from 26.8 to 32.5, while reducing workflow execution errors from 9.5% to 0%. These results show that explicit, editable search workflows provide an effective and controllable interface for aligning literature search agents with complex scientific intent.
Figures
Forward citations
Cited by 1 Pith paper
-
Fetch-then-Explore: Decoupling Selection from Extraction over a Persistent Workspace for Search Agents
Fetch-then-Explore, which stores fetched pages in a per-question workspace and extracts evidence on demand with grep/read, beats visit-and-read and browsing baselines on BrowseComp across three LLM backbones.
Reference graph
Works this paper leans on
-
[1]
Aho and Jeffrey D
Alfred V. Aho and Jeffrey D. Ullman , title =. 1972
1972
-
[2]
arXiv preprint arXiv:2606.05622 , year=
AdaPlanBench: Evaluating Adaptive Planning in Large Language Model Agents under World and User Constraints , author=. arXiv preprint arXiv:2606.05622 , year=
-
[3]
arXiv preprint arXiv:2511.02734 , year=
CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents , author=. arXiv preprint arXiv:2511.02734 , year=
-
[4]
arXiv preprint arXiv:2601.11004 , year=
NAACL: Noise-AwAre Verbal Confidence Calibration for LLMs in RAG Systems , author=. arXiv preprint arXiv:2601.11004 , year=
-
[5]
Publications Manual , year = "1983", publisher =
1983
-
[6]
Jiayu Liu and Rui Wang and Qing Zong and Qingcheng Zeng and Tianshi Zheng and Haochen Shi and Dadi Guo and Baixuan Xu and Chunyang Li and Yangqiu Song , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2601.11004 , eprinttype =. 2601.11004 , timestamp =
-
[7]
Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243
-
[8]
Scalable training of
Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of
-
[9]
Dan Gusfield , title =. 1997
1997
-
[10]
Tetreault , title =
Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =
2015
-
[11]
A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =
Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =
-
[12]
arXiv preprint arXiv:2606.20997 , year=
BioInsight: Multi-Agent Orchestration for Interactive Biomedical Knowledge Discovery , author=. arXiv preprint arXiv:2606.20997 , year=
-
[13]
Control Large Language Models via Divide and Conquer
Li, Bingxuan and Wang, Yiwei and Meng, Tao and Chang, Kai-Wei and Peng, Nanyun. Control Large Language Models via Divide and Conquer. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.18653/v1/2024.emnlp-main.850
-
[14]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =
Li, Bingxuan and Cui, Yiming and He, Yicheng and Wang, Yiwei and Zhang, Shu and Wen, Longyin and Niu, Yulei , title =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =. 2026 , pages =
2026
-
[15]
arXiv preprint arXiv:2603.07978 , year=
Osexpert: Computer-use agents learning professional skills via exploration , author=. arXiv preprint arXiv:2603.07978 , year=
-
[16]
METAL : A Multi-Agent Framework for Chart Generation with Test-Time Scaling
Li, Bingxuan and Wang, Yiwei and Gu, Jiuxiang and Chang, Kai-Wei and Peng, Nanyun. METAL : A Multi-Agent Framework for Chart Generation with Test-Time Scaling. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025. doi:10.18653/v1/2025.acl-long.1452
-
[17]
Yao, Shunyu and Zhao, Jeffrey and Yu, Dian and Du, Nan and Shafran, Izhak and Narasimhan, Karthik and Cao, Yuan , booktitle =
-
[18]
Advances in Neural Information Processing Systems (NeurIPS) , year =
Reflexion: Language Agents with Verbal Reinforcement Learning , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =
-
[19]
Advances in Neural Information Processing Systems (NeurIPS) , year =
Toolformer: Language Models Can Teach Themselves to Use Tools , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =
-
[20]
and Moazam, Hanna and Miller, Heather and Zaharia, Matei and Potts, Christopher , booktitle =
Khattab, Omar and Singhvi, Arnav and Maheshwari, Paridhi and Zhang, Zhiyuan and Santhanam, Keshav and Vardhamanan, Sri and Haq, Saiful and Sharma, Ashutosh and Joshi, Thomas T. and Moazam, Hanna and Miller, Heather and Zaharia, Matei and Potts, Christopher , booktitle =
-
[21]
Zhang, Jiayi and Xiang, Jinyu and Yu, Zhaoyang and Teng, Fengwei and Chen, Xionghui and Chen, Jiaqi and Zhuge, Mingchen and Cheng, Xin and Hong, Sirui and Wang, Jinlin and Zheng, Bingnan and Liu, Bang and Luo, Yuyu and Wu, Chenglin , booktitle =
-
[22]
arXiv preprint arXiv:2601.11957 , year=
PEARL: Self-Evolving Assistant for Time Management with Reinforcement Learning , author=. arXiv preprint arXiv:2601.11957 , year=
-
[23]
arXiv preprint arXiv:2501.05366 , year =
Search-o1: Agentic Search-Enhanced Large Reasoning Models , author =. arXiv preprint arXiv:2501.05366 , year =
-
[24]
2025 , howpublished =
2025
-
[25]
Li, Yutong and Yuan, Lu and Chen, Yuming and Wang, Pengfei and Zhang, Ningyu , journal =
-
[26]
Lu, Chris and Lu, Cong and Lange, Robert Tjarko and Foerster, Jakob and Clune, Jeff and Ha, David , journal =. The
-
[27]
Li, Kaichen and Zhao, Tian and Yang, Quanyu and Wu, Yong and Cao, Yu , journal =
-
[28]
Whitfield, Sonia and Hofmann, Matthew Axel , journal =. Elicit:
-
[29]
Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
Symbolic representation for any-to-any generative tasks , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
-
[30]
arXiv preprint arXiv:2402.01788 , year=
Litllm: A toolkit for scientific literature review , author=. arXiv preprint arXiv:2402.01788 , year=
-
[31]
arXiv preprint arXiv:2409.13740 , year=
Language agents achieve superhuman synthesis of scientific knowledge , author=. arXiv preprint arXiv:2409.13740 , year=
-
[32]
arXiv preprint arXiv:2411.14199 , year=
Openscholar: Synthesizing scientific literature with retrieval-augmented lms , author=. arXiv preprint arXiv:2411.14199 , year=
-
[33]
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
Pasa: An llm agent for comprehensive academic paper search , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
-
[34]
Researchagent: Iterative research idea generation over scientific literature with large language models , author=. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=
2025
-
[35]
arXiv preprint arXiv:2405.15784 , year=
Clarinet: Augmenting language models to ask clarification questions for retrieval , author=. arXiv preprint arXiv:2405.15784 , year=
-
[36]
Proceedings of the web conference 2020 , pages=
Generating clarifying questions for information retrieval , author=. Proceedings of the web conference 2020 , pages=
2020
-
[37]
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=
Assisting in writing wikipedia-like articles from scratch with large language models , author=. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=
2024
-
[38]
Advances in neural information processing systems , volume=
Autosurvey: Large language models can automatically write surveys , author=. Advances in neural information processing systems , volume=
-
[39]
2026 , howpublished =
2026
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.