Pith. sign in

REVIEW 4 major objections 3 minor 3 cited by

SciSage: A Multi-Agent Framework for High-Quality Scientific Survey Generation

T0 review · 4 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A six-agent pipeline that reflects while drafting beats current survey generators

desk verdict Useful system and benchmark, but the headline coherence claim likely measures outlines and the human-eval numbers contradict each other; worth refereeing after major revision. read the letter →

arxiv 2506.12689 v2 pith:OXSGETAX submitted 2025-06-15 cs.AI cs.IR

Xiaofeng Shi , Qian Kou , Yuduo Li , Ning Tang , Jinxin Xie , Longbin Yu , Songjing Wang , Hua Zhou This is my paper · ORCID
classification cs.AIcs.IR
keywords multi-agentsystemsscientificsurveygenerationself-reflectionretrieval-augmentedbenchmarkLLMevaluationstructuralcoherencecitationaccuracy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SciSage tries to show that the biggest gap between machine-written and human-written surveys is not retrieval or outline quality but the absence of critical revision during writing. Its six-agent pipeline inserts a hierarchical Reflector that critiques the outline, each section, and the assembled manuscript before the next draft is written, looping until feedback stops. Against LLM×MapReduce-V2 and AutoSurvey on the new SurveyScope benchmark, the paper reports document coherence of 80.37 (vs. 78.64) and citation $F_1$ of 0.46 (vs. 0.14), with ablation showing the reflection component itself accounts for much of the gap. The authors also present human evaluations in which experts preferred human-written surveys on 7 of 10 topics, so the claim is that reflection closes much of the structural and citation gap while leaving analytical depth and mathematical precision as open challenges.

What carries the argument

The central mechanism is the hierarchical Reflector agent, a generate–reflect–regenerate loop that runs at three scopes: outline, section, and full document. At full-document scope it convenes a panel of LLM personas (editor, senior professor, peer reviewer) with majority-vote identification of weak sections and a structured revision plan. The Reflector does the work of converting an unreflective single-pass writer into a self-correcting one; ablation shows document coherence rising from 71.25 to 81.48 and critical-thinking scores from 69.70 to 77.93 when reflection is enabled.

What would settle it

Swap the judge: have a different model family (or blinded expert raters) score the same generated surveys on the 10-point rubric. If SciSage no longer separates from LLM×MapReduce-V2 on document coherence, or if the cognitive scores invert on topics where humans preferred the human-written survey, the claimed advantage is an artifact of self-judging.

Watch

Extended reading notes

Core claim

The paper claims that embedding a critic into the generation loop—rather than relying on a single-pass draft plus post-editing—is what makes automated surveys structurally coherent and citation-faithful. The Reflector audits the outline for completeness and logic, audits each section for accuracy and evidential support and can trigger new retrieval, and audits the full text through a panel of simulated expert personas whose majority vote feeds a revision plan back into retrieval and composition. On SurveyScope, a curated set of 46 high-impact computer-science surveys (2020–2025, average 322 citations), this yields document-level coherence 80.37, critical-thinking 77.58, and citation $F_1$ 0.46 with 1,510 of 3,844 human-cited references correctly matched—while the nearest baseline on citation $F_1$ matches 392. Human experts found the generated surveys broader and faster to produce, but preferred human-written surveys on 7 of 10 topics, mainly for depth, precise formulas, and style.

Load-bearing premise

The headline numbers assume that Qwen3-32B—the same model family that wrote the surveys—is an unbiased judge of coherence, critical thinking, and citation quality, and that the rescaled 10-point rubric preserves the ranking a 100-point scale would give.

Editorial extensions

If this is right

  • If the reflect-when-you-write result holds, automated survey systems should embed critics at every drafting stage rather than bolting on a final edit.
  • Citation quality becomes a measurable property of the pipeline: $F_1$ 0.46 with 1,510 exact overlaps shows retrieval-driven composition can align with human reference choices at an actionable scale.
  • The 10-point scoring rubric rescaled to 0–100 is a methodological claim: the standard 100-point rubric saturates and cannot separate modern systems, so finer-grained evaluation is needed.
  • SurveyScope's recency and citation controls give future survey-generation work a common, higher-impact testbed than the older 20-paper SURVEY EVAL_TEST and AutoSurvey sets.
  • The mixed human results imply the practical payoff is breadth and drafting efficiency, not replacing expert authorship of analytic content.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the benchmark's retrieval sources overlap the manually curated SurveyScope papers, the citation-$F_1$ gains could partly measure retrieval access rather than reflection; a clean test is to hold out the benchmark papers from the retrieval corpus and re-measure.
  • The same hierarchical-reflection pattern could transfer to other long-form, citation-heavy writing—technical reports, regulatory documents, textbooks—where structural coherence and source fidelity matter more than one-shot fluency.
  • A stricter extension of the paper's own metric-saturation warning: future evaluation should shift from topical-relevance and section-coherence scores toward document-level argumentation and claim-support alignment, where the paper shows the largest discriminative gaps.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes SciSage, a multi-agent LLM framework for automated scientific survey generation built around a 'reflect-when-you-write' paradigm, in which a hierarchical Reflector agent critiques and refines generated content at outline, section, and document levels. The authors also introduce SurveyScope, a benchmark of 46 high-impact computer science survey papers (2020–2025) with recency and citation-based quality controls. Experiments compare SciSage against OpenScholar (w/ SciSage), AutoSurvey, and LLM×MapReduce-V2, reporting gains in document coherence (80.37 vs. 78.64) and citation F1 (0.46 vs. 0.14 for the strongest baseline), together with a human evaluation by domain experts. The paper claims that SciSage significantly outperforms state-of-the-art baselines, especially in structural coherence and reference accuracy.

Significance. If the reported results were fully supported, SciSage would be a useful contribution to automated survey generation, and the SurveyScope benchmark would provide a valuable community resource. The framework is described in sufficient detail to be reproducible, and the release of code and benchmark is a concrete strength. The reflect-when-you-write mechanism is a plausible and interesting design choice. However, the evaluation as currently presented contains load-bearing problems: the 'document coherence' metric appears to score outlines rather than full generated documents, the human evaluation numbers are internally contradictory, and the LLM judge is the same model family as the generator. These issues mean that the central claims of superiority over baselines are not currently established by the manuscript's evidence.

major comments (4)
  1. [§5.1.2, Appendix A.6] The 'Document-Level Structure' metric—which is the basis for the headline claim of +1.73/+2.8 points in document coherence (Table 4, §6.2.1) and for the ablation gains in Table 6—does not actually evaluate the full generated survey document. The prompt in Appendix A.6 is titled 'Prompt for Evaluation Document Outline', takes [Topic], [Skeleton], and [OUTLINE] as inputs, and defines its scoring criteria (structural coherence, conceptual depth, critical thinking) in terms of an outline's section organization and coverage. There is no input slot for the finished survey text. Consequently, the reported 'Document' scores measure outline quality, not the coherence of the final manuscript. This is a construct-validity failure that directly undermines the primary quantitative comparison. The authors must either re-run the document-level evaluation on full generated documents or substantially re-frame the claims.
  2. [Abstract, §1, §5.2, Table 7, §9] The human evaluation results are reported inconsistently in ways that cannot all be true. The Abstract states '3 wins vs. 7 losses against human-written surveys'; the Introduction states 'Human experts prefer SciSage's drafts on 7 out of 10 topics'; Table 7 lists only one topic where 'SciSage is better', two where 'Same', and seven where 'Human is better'; and the Conclusion states a '30% win rate'. These statements are mutually contradictory. The authors need to reconcile the counts and present a single, accurate summary of the human study. As it stands, the reader cannot tell whether the system won 1, 3, or 7 of the comparisons, and the Introduction's claim is contradicted by the paper's own Table 7.
  3. [§5.1, §6.2.1] The automatic evaluation framework uses Qwen3-32B as the judge for systems that are also built on Qwen3-32B, including SciSage itself. The paper further replaced the 100-point rubric from LLM×MapReduce-V2 with a 10-point rubric to increase score variance (Section 5.1.1), which suggests that the original rubric was not discriminative. This setup creates a risk of evaluator bias in favor of the proposed system, since the judge may prefer outputs that match its own stylistic tendencies. The authors should provide evidence that the Qwen3-32B judge is not biased, for example by reporting agreement with human judgments or by using an independent judge (e.g., a different model family or human preference judgments) for at least a subset of the evaluation.
  4. [Abstract, §1, Table 4] The reported effect sizes are inconsistent across the paper. The Abstract claims '+32% in citation F1 scores', which appears to be an absolute percentage-point difference (0.46 vs. 0.14 for AutoSurvey), not a relative improvement; relative to AutoSurvey the improvement is about 228%. The Introduction claims '+2.8 points in document coherence', but Table 4 shows a difference of 1.73 points (80.37 vs. 78.64) against LLM×MapReduce-V2, and larger differences against other baselines. The authors should state the comparison basis explicit for every reported gain and ensure the abstract, introduction, and results tables are consistent.
minor comments (3)
  1. [Table 5, §7.1] Table 5 is captioned 'Comparison of SCISAGE with and without Reflection', but the surrounding text in §7.1 describes an ablation of query understanding (w/ Q.U. vs. w/o Q.U.). This caption appears to be a copy-paste error and should be corrected.
  2. [Throughout] There are numerous spelling and formatting errors, including 'adajecent' in Table 3, 'retrievd' in §5.1.3, 'Pormpt' in Appendix A.4 and A.5, 'Comparsion' in Figure 12, 'scitific' in §2, 'understading' in §3.1, and 'trails' instead of 'trials' in Algorithm 1. The model name is inconsistently written as 'Qwen3-32B' and 'QWEN 3-32B', and the system name as 'SciSage' and 'SCISAGE'. These should be cleaned up.
  3. [§6.2.1] The statement that SciSage achieves 'perfect topical relevance (100)' is not a meaningful differentiator because LLM×MapReduce-V2 also achieves 100 and the metric is acknowledged as saturated in §8. The discussion should avoid highlighting a saturated metric as evidence of superiority.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central claim is an empirical system comparison, not a derivation that reduces to its inputs; only minor evaluator overlap and an outline-vs-document construct concern appear.

full rationale

The paper's central claim—that SciSage's hierarchical reflect-when-you-write agents improve document coherence and citation F1—is an empirical result obtained by running SciSage and three external baselines on a new benchmark, SurveyScope. No fitted parameter is renamed as a prediction, no equation is defined in terms of the quantity it is said to predict, and no load-bearing claim rests on a self-citation or on a uniqueness theorem from the authors' prior work. The main result is supported by ablations (Table 6) that toggle the reflection mechanism and measure the effect, which is a causal comparison rather than a construction-level identity. The automatic evaluation does use Qwen3-32B both as generator and as judge, and the rubrics are authored by the same team, but this is an evaluator-overlap/validity risk, not a circular derivation: the scores are not obtained by plugging the system's own outputs back into the system's own parameters. There is also a construct-validity concern that the "Document" coherence score may actually be an outline score, since Appendix A.6 is titled "Prompt for Evaluation Document Outline" and its placeholders are [TOPIC], [Skeleton], and [OUTLINE]; however, this is a measurement-mismatch issue, not a case of the claimed result being equivalent to the input by construction. The paper's own Section 8 acknowledges metric saturation for Topical Relevance and Section Coherence, further indicating that the headline discriminative claims are limited to specific metrics rather than being forced by the framework's definition. Overall, the derivation chain is self-contained and empirically grounded; the identified concerns belong under correctness/validity risk, not circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central comparison rests on a chain of domain assumptions about LLM judging, benchmark quality proxies, and baseline fairness. None of these are formal axioms; they are evaluation choices that, if wrong, would change the conclusions. No new physical or mathematical entities are introduced.

free parameters (3)
  • outline_max_reflections = 2
    Hand-chosen cap on outline reflection rounds; ablations do not vary it, so reported quality gains may depend on this budget.
  • section_reflection_max_turns = 2
    Hand-chosen cap for section reflection; the reflection ablation is tested only at this budget.
  • global_reflection_max_turns = 2
    Hand-chosen cap for full-document reflection; the contribution of global reflection is evaluated only with this fixed setting.
assumptions (4)
  • domain assumption LLM-based evaluation with Qwen3-32B is a valid proxy for expert human assessment of survey quality.
    All headline scores in Table 4 come from this evaluator; the authors note metric saturation and do not report correlation with human judgments.
  • domain assumption Citation count and publication recency are appropriate proxies for paper quality in benchmark construction.
    SurveyScope's quality controls are defined solely by these two criteria, which may favor popular rather than technically important work.
  • domain assumption Human-written reference lists are the complete ground truth for citation F1.
    Citation F1 in Section 5.1.3 penalizes any model reference not appearing in the human survey's bibliography, assuming human lists are exhaustive.
  • domain assumption Baselines run under official default configurations provide fair comparisons.
    OpenScholar is modified with SciSage outlines and queries, so this assumption is questionable for that baseline.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SciSage: A Multi-Agent Framework for High-Quality Scientific Survey Generation." pith.science (2026). https://pith.science/paper/OXSGETAX

@misc{pith2026250612689,
  author       = {Pith},
  title        = {Pith review of: SciSage: A Multi-Agent Framework for High-Quality Scientific Survey Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OXSGETAX}},
  note         = {Machine review of arXiv:2506.12689}
}
read the original abstract

The rapid growth of scientific literature demands robust tools for automated survey-generation. However, current large language model (LLM)-based methods often lack in-depth analysis, structural coherence, and reliable citations. To address these limitations, we introduce SciSage, a multi-agent framework employing a reflect-when-you-write paradigm. SciSage features a hierarchical Reflector agent that critically evaluates drafts at outline, section, and document levels, collaborating with specialized agents for query interpretation, content retrieval, and refinement. We also release SurveyScope, a rigorously curated benchmark of 46 high-impact papers (2020-2025) across 11 computer science domains, with strict recency and citation-based quality controls. Evaluations demonstrate that SciSage outperforms state-of-the-art baselines (LLM x MapReduce-V2, AutoSurvey), achieving +1.73 points in document coherence and +32% in citation F1 scores. Human evaluations reveal mixed outcomes (3 wins vs. 7 losses against human-written surveys), but highlight SciSage's strengths in topical breadth and retrieval efficiency. Overall, SciSage offers a promising foundation for research-assistive writing tools.

Figures

Figures reproduced from arXiv: 2506.12689 by the authors.

Figure 1
Figure 1. Overview of the SCISAGE framework. 3 Method In this section, we introduce SciSage, a LLM-based Multi-Agent framework designed for automated scientific survey generation. Inspired by the cognitive and iterative behaviors of expert authors, SciSage leverages a coordinated architecture of specialized agents that unfolds through three interconnected components—Query Understanding and Rewrite, Retrieval and Content Gener… view at source ↗
Figure 2
Figure 2. Overview of the SurveyScope construction pipeline. 1. Domain Extraction from Existing Benchmarks. We began by collecting open-source benchmark datasets and identifying their covered academic research domains. This extraction was performed using the Qwen3-32B model [9], guided by a structured prompt (see Appendix A.2). 2. Topic Completion with Expert and LLM Assistance. To ensure comprehensive coverage, we augmented … view at source ↗
Figure 3
Figure 3. Distribution of topics in SurveyScope Recency of Publications SurveyScope includes papers published between 2020 and 2025, capturing recent advances in computer science. The release of ChatGPT [33] in late 2022 led to a sharp increase in large language model (LLM) research, resulting in a notable concentration of publications in 2023 and 2024 [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Publication year distribution in SurveyScope. (a) Histogram showing the number of papers published each year. (b) Kernel density estimate illustrating publication trends over time. High Citation Count Papers in SurveyScope demonstrate strong citation impact. The most c…
Figure 5
Figure 5. Figure 5: Citation statistics for SurveyScope. (a) Distribution of citation counts across the dataset. (b) Cumulative plot showing that over 52% of the papers have received more than 100 citations. Summary SurveyScope stands out from existing benchmarks through its broad topical…
Figure 6
Figure 6. Figure 6: Comparison of different benchmarks on Category Count, Topic Diversity, Data Volume, Year Span, Max Citations, Avg Citations 5 Evaluation To comprehensively evaluate the quality of generated content compared to human-written counterparts, we adopt a two-fold evaluation …
Figure 7
Figure 7. Figure 7: Score comparison: direct 100-point (blue) vs. scaled from 10-point rubric (purple). [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Human evaluation results comparing SCISAGE with HUMAN WRITTEN papers Strengths: Broad Coverage and Summarization SCISAGE excels at generating content that is broad in scope and performs as well as or better than human authors on summarization tasks. For example, in are…
Figure 9
Figure 9. Figure 9: Radar chart illustrating topic distribution across [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: Boxplot showing publication year distributions across benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Boxplot of citation counts across benchmarks. Papers in [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: Comparsion of differents benchmarks 22 [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: Example of generated outline 25 [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Externalizing Research Synthesis and Validation in AI Scientists through a Research Harness

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    Xcientist is a research harness that externalizes an AI scientist's literature grounding, idea evolution, experiments, and repairs into auditable artifacts, demonstrated on memory, traffic forecasting, and PDE-solving tasks.

  2. SurveyLens: A Discipline-Aware Benchmark for Automatic Survey Generation

    cs.CL 2026-02 conditional novelty 6.0 of 10

    SurveyLens benchmarks AI survey generators on 1,000 human-written surveys from 10 disciplines, finding Deep Research agents most robust and reference quality weakest everywhere.

  3. DeepSurvey-Bench: Evaluating Academic Value of Automatically Generated Scientific Surveys

    cs.AI 2026-01 conditional novelty 5.0 of 10

    A benchmark with human-annotated academic-value labels and seven LLM-judged metrics for evaluating whether automatically generated scientific surveys have real academic value.

Reference graph

Works this paper leans on

56 extracted references · 29 canonical work pages · cited by 3 Pith papers

  1. [1]

    Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases

    Lutz Bornmann, Robin Haunschild, and Rüdiger Mutz. Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases. Humanities and Social Sciences Communications, 8(1):1–15, 2021

    work page 2021

  2. [2]

    Autosurvey: Large language models can automatically write surveys

    Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Qingsong Wen, Wei Ye, et al. Autosurvey: Large language models can automatically write surveys. Advances in Neural Information Processing Systems, 37:115119–115145, 2024

    2024

  3. [3]

    Deep reinforcement learning: A survey

    Xu Wang, Sen Wang, Xingxing Liang, Dawei Zhao, Jincai Huang, Xin Xu, Bin Dai, and Qiguang Miao. Deep reinforcement learning: A survey. IEEE Transactions on Neural Networks and Learning Systems, 35(4):5064– 5078, 2022

    work page 2022

  4. [4]

    A survey on large language models: Applications, challenges, limitations, and practical usage

    Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints, 2023

    2023

  5. [5]

    A survey on large language model based autonomous agents

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345, 2024

    2024

  6. [6]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    arXiv 2024

  7. [7]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    arXiv 2023

  8. [8]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

Show all 56 references
  1. [9]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  2. [10]

    Assisting in writing wikipedia-like articles from scratch with large language models

    Yijia Shao, Yucheng Jiang, Theodore A Kanell, Peter Xu, Omar Khattab, and Monica S Lam. Assisting in writing wikipedia-like articles from scratch with large language models. arXiv preprint arXiv:2402.14207, 2024

    2024 arXiv

  3. [11]

    Into the unknown unknowns: Engaged human learning through participation in language model agent conversations

    Yucheng Jiang, Yijia Shao, Dekun Ma, Sina J Semnani, and Monica S Lam. Into the unknown unknowns: Engaged human learning through participation in language model agent conversations. arXiv preprint arXiv:2408.15232, 2024

    2024 arXiv

  4. [12]

    Llm × mapreduce-v2: Entropy-driven convolutional test-time scaling for generating long-form articles from extremely long resources

    Haoyu Wang, Yujia Fu, Zhu Zhang, Shuo Wang, Zirui Ren, Xiaorong Wang, Zhili Li, Chaoqun He, Bo An, Zhiyuan Liu, et al. Llm × mapreduce-v2: Entropy-driven convolutional test-time scaling for generating long-form articles from extremely long resources. arXiv preprint arXiv:2504....

    2025 arXiv

  5. [13]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634–8652, 2023

    2023

  6. [14]

    Self-rag: Learning to retrieve, gener- ate, and critique through self-reflection

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, gener- ate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, 2023

    2023

  7. [15]

    Self-reflection in llm agents: Effects on problem-solving performance

    Matthew Renze and Erhan Guven. Self-reflection in llm agents: Effects on problem-solving performance. arXiv preprint arXiv:2405.06682, 2024

    2024 arXiv

  8. [16]

    Openscholar: Synthesizing scientific literature with retrieval-augmented lms

    Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D’arcy, et al. Openscholar: Synthesizing scientific literature with retrieval-augmented lms. arXiv preprint arXiv:2411.14199, 2024

    2024 arXiv

  9. [17]

    Surveyforge: On the outline heuristics, memory-driven generation, and multi-dimensional evaluation for automated survey writing

    Xiangchao Yan, Shiyang Feng, Jiakang Yuan, Renqiu Xia, Bin Wang, Bo Zhang, and Lei Bai. Surveyforge: On the outline heuristics, memory-driven generation, and multi-dimensional evaluation for automated survey writing. arXiv preprint arXiv:2503.04629, 2025

    2025 arXiv

  10. [18]

    Interactivesurvey: An llm-based personalized and interactive survey paper generation system

    Zhiyuan Wen, Jiannong Cao, Zian Wang, Beichen Guo, Ruosong Yang, and Shuaiqi Liu. Interactivesurvey: An llm-based personalized and interactive survey paper generation system. arXiv preprint arXiv:2504.08762, 2025

    2025 arXiv

  11. [19]

    Introducing deep research, 2024

    OpenAI. Introducing deep research, 2024. Accessed: 2025-05-19

    2024

  12. [20]

    Gemini deep research overview, 2024

    Google DeepMind. Gemini deep research overview, 2024. Accessed: 2025-05-19

    2024

  13. [21]

    A survey on llm-based multi-agent systems: workflow, infrastructure, and challenges

    Xinyi Li, Sai Wang, Siqi Zeng, Yu Wu, and Yi Yang. A survey on llm-based multi-agent systems: workflow, infrastructure, and challenges. Vicinagearth, 1(1):9, 2024

    2024

  14. [22]

    Large language model based multi-agents: A survey of progress and challenges

    Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01680, 2024

    2024 arXiv

  15. [23]

    Metagpt: Meta programming for multi-agent collaborative framework

    Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352, 3(4):6, 2023

    2023 arXiv

  16. [24]

    Autogen: Enabling next-gen llm applications via multi-agent conversation

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen llm applications via multi-agent conversation. arXiv preprint arXiv:2308.08155, 2023

    2023 arXiv

  17. [25]

    Unleashing the emergent cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration

    Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing the emergent cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration. arXiv preprint arXiv:2307.05300, 2023

    2023 arXiv

  18. [26]

    Generative agents: Interactive simulacra of human behavior

    Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22, 2023

    2023

  19. [27]

    Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents

    Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, et al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents. arXiv preprint arXiv:2308.10848, 2(4):6, 2023

    2023 arXiv

  20. [28]

    Chatdev: Communicative agents for software development

    Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. Chatdev: Communicative agents for software development. arXiv preprint arXiv:2307.07924, 2023

    2023 arXiv

  21. [29]

    Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization

    Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization. arXiv preprint arXiv:2310.02170, 2023. 15 SciSage Technical Report

    2023 arXiv

  22. [30]

    Encouraging divergent thinking in large language models through multi-agent debate

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305.19118, 2023

    2023 arXiv

  23. [31]

    Improving factuality and reasoning in language models through multiagent debate

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning, 2023

    2023

  24. [32]

    Agentnet: Decentralized evolutionary coordination for llm-based multi-agent systems

    Yingxuan Yang, Huacan Chai, Shuai Shao, Yuanyi Song, Siyuan Qi, Renting Rui, and Weinan Zhang. Agentnet: Decentralized evolutionary coordination for llm-based multi-agent systems. arXiv preprint arXiv:2504.00587, 2025

    2025 arXiv

  25. [33]

    Chatgpt and open-ai models: A preliminary review

    Konstantinos I Roumeliotis and Nikolaos D Tselikas. Chatgpt and open-ai models: A preliminary review. Future Internet, 15(6):192, 2023

    2023

  26. [34]

    A probabilistic interpretation of precision, recall and f-score, with implication for evaluation

    Cyril Goutte and Eric Gaussier. A probabilistic interpretation of precision, recall and f-score, with implication for evaluation. In European conference on information retrieval, pages 345–359. Springer, 2005. 16 SciSage Technical Report A Prompt Template A.1 Prompt for Query ...

    2005

  27. [35]

    Machine Learning

    Research Domain: Identify the broad academic field the query falls into. Examples: Computer Science, Medicine, Physics, Sociology, History, Linguistics. Be as specific as reasonably possible (e.g., "Machine Learning" if clearly indicated within Computer Science, otherwise "Com...

  28. [36]

    You MUST choose one of the following predefined types: survey, method, application, analysis, position, theory, benchmark, dataset, OTHER

    Query Type: Determine the type of information or paper the user is likely seeking. You MUST choose one of the following predefined types: survey, method, application, analysis, position, theory, benchmark, dataset, OTHER . If none of the specific types fit well, use OTHER

  29. [37]

    latest advancements in using LLMs for code generation

    Research Topic: Pinpoint the specific subject, concept, or entities at the core of the query. This should be a concise phrase representing the main focus. For example, if the query is "latest advancements in using LLMs for code generation", the topic could be "LLMs for code ge...

  30. [38]

    Semantic clarity issues

  31. [39]

    Contextual fit for search/research scenarios

  32. [40]

    Based on the following paper title, please complete the two tasks below:

    Overly complex or verbose phrasing If rewriting is needed, create a revised version that: • Maintains the original semantic meaning • Is more precise and concise • Is better suited for search/research purposes A.2 Prompt for Benchmark Topic Classification You are an expert in ...

  33. [41]

    Extract the main research topic of the paper (expressed as a concise phrase, such as: Robustness in NLP Models, Multimodal Learning, LLM Safety, etc.)

  34. [42]

    Assign the extracted topic to one of the following high-level categories: Category List:

  35. [43]

    Medical / Biomedical 17 SciSage Technical Report

  36. [44]

    Finance / Domain-specific

  37. [45]

    Benchmarking / Evaluation

  38. [46]

    Now I want to gain a comprehensive overview of the current research hotspots across the field

    Other Paper Title: {title} Please return the result in the following format: Research Topic: [your topic] Category: [your chosen category] A.3 Prompt for Benchmark Topic Completion You are an expert in computer science research. Now I want to gain a comprehensive overview of t...

  39. [47]

    Uses precise terminology consistently, avoids colloquial language entirely, and maintains a scholarly tone throughout

    Academic Formality (10 points) Demonstrates flawless academic rigor. Uses precise terminology consistently, avoids colloquial language entirely, and maintains a scholarly tone throughout. Sentence structures are sophisticated and intentionally crafted to support analytical dep...

  40. [48]

    Sentences are logically structured with seamless transitions

    Clarity & Readability (10 points) Writing is exceptionally clear, concise, and unambiguous. Sentences are logically structured with seamless transitions. The argument progresses smoothly with no unnecessary complexity. Any ambiguity or minor inefficiency reduces the score

  41. [49]

    Repetition is only acceptable for structural clarity, such as reinforcing terminology or aiding transitions

    Redundancy (10 points) Uniqueness: Every sentence should contribute new value. Repetition is only acceptable for structural clarity, such as reinforcing terminology or aiding transitions. Efficiency: Arguments must be logically coherent and free from unnecessary repetition. Re...

  42. [50]

    Clearly identifies significant gaps, weaknesses, and areas for improvement

    Critical Analysis (10 points) Offers a deep and incisive critique of methodologies, results, and underlying assumptions. Clearly identifies significant gaps, weaknesses, and areas for improvement. Challenges assumptions with well-supported arguments and proposes concrete alternatives

  43. [51]

    Demon- strates strong subject-matter understanding and contributes genuinely original perspectives

    Original Insights (10 points) Proposes novel, well-supported interpretations or frameworks based on the reviewed literature. Demon- strates strong subject-matter understanding and contributes genuinely original perspectives. Insights are well-integrated with existing research,...

  44. [52]

    Suggestions are concrete, actionable, and closely tied to gaps identified in the literature

    Future Directions (10 points) Clearly articulates promising research directions with strong justification. Suggestions are concrete, actionable, and closely tied to gaps identified in the literature. Demonstrates foresight by proposing innovative approaches or methodologies. [...

  45. [53]

    Sections and subsections are clearly organized, transitions are smooth, and the narrative progression is coherent

    Structural Coherence & Narrative Logic (10 points) Ideal Standard: The outline presents a well-structured, logically flowing framework. Sections and subsections are clearly organized, transitions are smooth, and the narrative progression is coherent. Scoring Guidance: Deduct p...

  46. [54]

    There is a balance of breadth and depth, with core debates and historical development of the field clearly reflected

    Conceptual Depth & Thematic Coverage (10 points) Ideal Standard: The outline captures key themes, concepts, and subfields comprehensively and insightfully. There is a balance of breadth and depth, with core debates and historical development of the field clearly reflected. Sco...

  47. [55]

    It synthesizes viewpoints into a coherent scholarly vision

    Critical Thinking & Scholarly Synthesis (10 points) Ideal Standard: The outline integrates perspectives critically, addressing contradictions, methodologi- cal tensions, and open research questions. It synthesizes viewpoints into a coherent scholarly vision. Scoring Guidance: ...

  48. [56]

    SurveyScope exhibits broader and more balanced domain coverage

    Medical / Biomedical Benchmarking / Evaluation Dialogue Systems Finance / Domain-specific LLMs (General) LLMs Efficiency LLMs Safety Medical / Biomedical Multimodal NLP Other Robotics 2 4 6 8 SurveyScope SurveyEval_T est AutoSurvey Figure 9: Radar chart illustrating topic dist...

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.