Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TreeReview claims that structuring LLM peer review as a tree of fine-grained questions, each answered from a few highly relevant paper chunks and then aggregated bottom-up, beats strong baselines on review depth and specificity while…

desk verdict A useful, honestly-reported framework for structured LLM review generation whose quality claims outrun the evidence on retrieval recall and judge self-reference, but the efficiency win is real and the work deserves referee time. read the letter →

arxiv 2506.07642 v3 pith:PWA53VYX submitted 2025-06-09 cs.CL

Yuan Chang , Ziyue Li , Hengyuan Zhang , Yuanbo Kong , Yanru Wu , Hayden Kwok-Hay So , Zhijiang Guo , Liya Zhu
show 1 more author
Ngai Wong
This is my paper
classification cs.CL
keywords TreeReviewautomatedpeerquestiondecompositionretrieval-augmentedgenerationdynamicexpansionlong-documentreasoningLLM-as-judgeevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the best way to get an LLM to review a long scientific paper is not to feed it the whole paper at once, nor to orchestrate a team of collaborating agents, but to decompose the review into a tree of increasingly specific questions, answer each leaf question from a small set of relevant paper chunks, and then aggregate the answers from leaf to root. The authors build TreeReview around this two-stage design, adding a dynamic expansion step in which the model generates follow-up questions when the answers it has collected do not yet resolve an intermediate question. They evaluate on a benchmark of 80 ICLR and NeurIPS papers with human-written reviews, using both LLM judges and five expert human evaluators. A sympathetic reader would care because the claimed result is that reviews become more comprehensive, specific, and technically deep while consuming roughly one fifth of the tokens a multi-agent alternative uses, which is exactly the trade-off that would make automated review assistance practical.

What carries the argument

The carrying object is the review question tree. Top-down, a question-generator agent decomposes each non-leaf question into at most $W_{\max}$ sub-questions following the MECE principle, with the branching budget shrinking by one at each depth level (5, then 4, then 3) up to a maximum depth of 4. Bottom-up, a synthesizer agent answers each leaf question using only the top-3 chunks of 1024 tokens selected by LongLLMLingua's question-aware perplexity reranking, so each fine-grained question is answered from roughly 3,000 tokens of focused context rather than the whole ~20,000-token paper, and then aggregates sub-answer pairs into answers for intermediate nodes. The dynamic expansion mechanism is the third piece: at 38.54% of intermediate nodes on average the synthesizer declares the collected evidence insufficient and generates up to two follow-up questions, which are grafted into the tree, decomposed if needed, and answered before synthesis proceeds. The final root step then reads the full paper with the aggregated question-answer pairs as explicit reasoning traces.

What would settle it

An audit where human annotators mark the passages that support each leaf question and check whether the top-3 selected chunks contain them: if the decisive passage frequently falls outside the selected chunks while judged review quality stays constant, the retrieval mechanism is not what carries the result, whereas if withholding it drops quality sharply, the chunk-selection assumption is confirmed as load-bearing.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that hierarchical question decomposition plus bottom-up synthesis is a better organizing principle for LLM review generation than either monolithic long-context prompting or multi-agent collaboration. The framework first has a question-generator agent recursively split broad review questions into fine-grained sub-questions, guided by metadata only so the decomposition stays exploratory, and then a synthesizer agent answers each leaf question from the top-3 paper chunks selected by a perplexity-based reranker, aggregates answers for intermediate nodes, and finally reads the full paper at the root to produce the review. When the aggregate evidence for an intermediate question is judged insufficient, the synthesizer generates follow-up questions that grow the tree dynamically. In LLM-as-Judge evaluation the method reports the highest scores on specificity, comprehensiveness, and technical depth, and in human pairwise comparison it wins between 66% and 90% of matchups against baselines, while using 80.2% fewer tokens per paper than the MARG multi-agent framework on the feedback-comments task.

Load-bearing premise

The whole synthesis rests on the assumption that the three 1,024-token chunks ranked highest by the perplexity filter contain everything needed to answer each leaf question, since only about 3,000 of roughly 20,000 tokens are available per question and the final full-paper pass comes too late to repair details missed at the leaves.

Editorial extensions

If this is right

  • If the central claim holds, LLM-assisted review can be made both deeper and cheaper: the reported 80.2% token reduction versus MARG comes with higher, not lower, judged quality, which changes the cost calculus for practical reviewer-assistance tools.
  • Dynamic expansion is doing real work, not just decoration: removing it drops the actionable-feedback Jaccard alignment from 15.33% to 12.98% and ITF-IDF specificity from 4.62 to 4.01, and the mechanism also compensates for a smaller initial tree.
  • The approach transfers across fields: on a separate 40-paper set from biology, climate science, and quantum science, the reported gains in technical depth and evidence support persist, suggesting the benefit is not a machine-learning-domain artifact.
  • Rating prediction matches a specialized fine-tuned model: TreeReview's mean squared error on overall rating (2.12) is on par with or better than the SEA-E fine-tuned model (2.30), without any task-specific training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My reading, beyond the paper: the retrieval-then-aggregate design implies the ceiling on review specificity is set by chunk selection, not by the synthesizer, so a testable extension would vary k (currently 3) and measure whether marginal review quality rises with k, which the paper does not report.
  • A natural extension the authors leave implicit is to apply the same question-tree machinery to other long-document comprehension tasks, such as grant proposal review, patent examination, or clinical protocol critique, where the same problem of details scattered through a long document dominates.
  • Because the leaf answers must cite evidence from the selected chunks, the framework produces a built-in audit trail linking each review claim to a paper location; a practical deployment could surface these citations to human reviewers, a workflow consequence the paper notes but does not develop.
  • The evaluation uses LLM judges and a small expert panel; a stronger test would pit TreeReview reviews against human reviews in a blind author-facing study measuring how often authors act on the feedback, which is ultimately the outcome that matters.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces TreeReview, a framework that structures LLM-based scientific peer review as a hierarchical question-answering process. In a top-down stage, a question generator recursively decomposes the review task into a tree of fine-grained questions; in a bottom-up stage, an answer synthesizer answers leaf questions using question-aware retrieval over paper chunks, aggregates answers from leaves to the root, and dynamically generates follow-up questions when evidence is deemed insufficient. The final root step combines the full paper with the collected question-answer pairs to produce a review or actionable feedback comments. The authors construct a benchmark of 80 ICLR-2024 and NeurIPS-2023 papers with human reviews, evaluate full-review generation and feedback-comment generation against several baselines, and report LLM-as-judge scores, semantic alignment metrics, human pairwise win-rates, and token usage. The main claims are that TreeReview outperforms strong baselines in comprehensiveness, specificity, and technical depth while reducing token usage by up to 80.2% relative to MARG.

Significance. If the central claims hold, TreeReview is a practically useful contribution: it combines decomposition, retrieval, and dynamic expansion in a way that appears to produce more specific and evidence-grounded reviews at substantially lower token cost than multi-agent baselines. The paper also contributes an open benchmark and code, which are valuable for future work on automated review generation. The framing of review generation as a hierarchical question-answering process, rather than as direct long-context generation or multi-agent communication, is a reasonable and clearly specified design. The human evaluation, despite its modest size, is a genuine strength because it partially validates the automated metrics. The main caveat is that the quality advantage is contingent on the retrieval component selecting all content required by leaf questions, and the paper does not yet provide retrieval-recall or sensitivity evidence for this load-bearing step.

major comments (3)
  1. [Section 3.3, Eq. (2); Appendix A.1] The central quality claim rests on the assumption that the top-3 chunks selected by LongLLMLingua (L=1024 tokens, k=3) contain all content needed to answer each leaf question. A paper in this benchmark averages roughly 19,000 tokens, so each leaf answer sees only about 3,000 tokens. The leaf prompt in Figure 14 explicitly forbids inference beyond the provided context and asks the model to acknowledge absence; any such gap propagates through the aggregation in Eq. (3) to the final review, and the root prompt in Figure 16 instructs the model to use the Q&A pairs as a guiding framework, so the root step is anchored on what the leaves supplied. The paper reports no retrieval-recall statistics, no rate of acknowledged missing content, and no sensitivity analysis for k or L. Appendix E.2 even concedes that the method struggles with fine-grained inconsistencies spanning different sections, which is exactly the failure mode that truncated retrieval would create. Because the 80% token reduction is achieved precisely by this k=3 choice, both the efficiency and quality claims are jointly contingent on an unvalidated hyperparameter. I recommend adding retrieval-recall measurements (e.g., how often leaf questions are answered with full confidence versus acknowledged absence) and a k/L sensitivity study before the main claims can be accepted.
  2. [Section 4.2, Fig. 3; Section 4.3, Table 2] The primary evidence for full-review quality is LLM-as-judge evaluation using Gemini-2.5-Pro, while the reviews themselves are generated by Gemini-2.0-Flash. The paper reports only mean scores and an ICC across three scoring runs; it does not report per-dimension standard deviations, confidence intervals, or significance tests against the baselines. The headline improvements (e.g., specificity +12.27%, comprehensiveness +11.22%, technical depth +6.45%) are therefore presented without statistical support. For the feedback-comments task, the reference comments are extracted and merged from human reviews by Gemini-2.5-Pro (Appendix B.1) and then aligned with generated comments by the same model family, which introduces a partial self-reference in the evaluation. The human evaluation in Section 4.4 is helpful but covers only 20 papers and reports pseudo win-rates without confidence intervals. I recommend reporting variance and significance tests for the LLM-judge scores, and ideally adding a second judge model or a human-rated subset to break the generator-judge model-family overlap.
  3. [Section 4.5, Tables 3 and 4] The domain-generalization claim is based on 40 Nature Communications papers evaluated against only two or three baselines (DGE, SORT, DPW, and TreeReview) and with no significance testing. The text says the method 'consistently outperforms baselines across almost all evaluation metrics,' but the tables report single-point estimates, and the gap between TreeReview and SORT on several dimensions is modest. Given that the earlier benchmark already lacks significance tests, the multi-domain results do not yet establish that the advantage is robust across fields. At minimum, the authors should report per-dimension variance and a significance test (e.g., paired bootstrap or Wilcoxon) for the main comparisons.
minor comments (5)
  1. [Section 3.2, Eq. (1)] There is an inconsistent use of agent symbols: Section 3.2 text says 'a specialized Question Generator agent M_a' for Eq. (1), but Eq. (1) uses M_q. This should be harmonized.
  2. [Appendix A.1, Eq. (5)] Equation (5) defines perplexity using p(q_i, x_restrict | chunk_j), while the main text Eq. (2) and the preceding description say the top-k chunks are selected based on p(q_i | chunk). The relationship between the restrictive statement x_restrict and the conditional probability should be clarified.
  3. [Section 4.7, Table 5] The cost analysis covers only the feedback-comments task, not the full-review task. The abstract and conclusion emphasize the 80% token reduction; it would be useful to report token usage for full-review generation as well, since the root step reads the full paper and the dynamic expansion may behave differently in that setting.
  4. [Appendix E.2] The error-mode analysis identifies that the method misses fine-grained inconsistencies spanning different sections. This is a direct consequence of the leaf retrieval strategy and should be discussed in the main text rather than only in an appendix, since it is relevant to the method's claimed depth of analysis.
  5. [Section 4.4] The human evaluation aggregates judgments from two evaluators per paper-pair into a pseudo win-rate, but the denominator for each comparison is not stated. Reporting the number of comparisons per cell would improve transparency.

Circularity Check

1 steps flagged · score 2.0 of 10

No derivation-level circularity: the TreeReview pipeline is constructive and does not fit parameters to its evaluation targets.

  1. other [Appendices B.1 and C.5 (also Section 4.3)]
    "Both the extraction and merging processes are implemented using Gemini-2.5-Pro. ... We utilize Gemini-2.5-Pro for this evaluation."

    The quantitative expert-alignment results in Table 2 are produced through a self-referential loop: Gemini-2.5-Pro extracts and consolidates human review comments into the reference set, and the same model then performs the many-to-many matching and relatedness/specificity judgments that define Precision, Recall, and Jaccard. The metric therefore partly measures Gemini-2.5-Pro's consistency with its own paraphrase and merging decisions rather than independent agreement with the original human comments. This is not a mathematical reduction of the method's output to its input, and the paper's human pairwise evaluation and embedding-based SN metrics provide independent evidence, so the circularity is partial and evaluation-level rather than derivation-level.

full rationale

TreeReview is an empirical pipeline rather than a derivation: the question tree is generated by an LLM from metadata, leaf questions are answered from retrieved chunks, answers are aggregated bottom-up, and the root review is generated from the full paper plus the Q&A trace. No free parameter is fitted to the evaluation targets, and none of the claimed improvements are forced by construction. The top-3 LongLLMLingua retrieval step restricts available context, and the paper's own error analysis (Appendix E.2) concedes that fine-grained cross-section inconsistencies are hard to capture, but this is a recall/coverage limitation, not circularity. The main caveat is the feedback-comment alignment evaluation: reference comments are built and judged by Gemini-2.5-Pro, so parts of the reported 'expert-aligned' advantage may reflect evaluator self-consistency. However, human evaluators independently preferred TreeReview at high win-rates, and semantic-similarity SN metrics show a consistent direction, so the central claim retains independent support. No load-bearing self-citation, imported uniqueness theorem, or renamed-known-result circularity was found.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The framework is an empirical system, so the main ledger entries are design hyperparameters and domain assumptions rather than mathematical axioms. The central claimed advantage depends on question-decomposition coverage, retrieval quality, and the validity of LLM-based evaluation; these are assumptions, with partial independent support from human evaluation.

free parameters (6)
  • Dmax (maximum tree depth) = 4
    Hand-chosen depth cap controls how fine-grained the decomposition becomes; no sensitivity sweep reported.
  • W^max_1 (max root sub-questions) = 5
    Hand-chosen width; decreasing width per depth is justified by intuition, not by tuning experiments.
  • W^exp_max (max dynamic follow-ups per node) = 2
    Limits how many new questions can be added during bottom-up aggregation.
  • k (retrieved chunks per leaf question) = 3
    Determines how much paper content reaches leaf answers; critical for specificity and coverage.
  • L (chunk size) = 1024 tokens
    Chunk size for segmentation before reranking.
  • t (ITF-IDF similarity threshold) = 0.5
    Threshold for counting comment matches in the specificity metric.
assumptions (4)
  • domain assumption A review question tree that satisfies MECE and is generated from title, abstract, and table of contents can cover all important aspects of a paper.
    The coverage claim depends on decomposition quality; invoked in Section 3.2 and the prompt in Fig. 13.
  • domain assumption Perplexity-based top-3 chunk selection from LongLLMLingua retrieves all content needed to answer each leaf question.
    Section 3.3 and Appendix A.1; if a critical detail falls outside the top-3 chunks, leaf answers miss it.
  • domain assumption LLM-as-judge scores from Gemini-2.5-Pro on eight dimensions are a valid measure of review quality, and human evaluator preferences correlate with them.
    Main results in Section 4.2 and Appendix C.2; supported by a reported correlation of about 0.90, but not externally calibrated.
  • domain assumption Reference feedback comments produced by Gemini-2.5-Pro extraction and merging from human reviews are a faithful ground truth for alignment evaluation.
    Appendix B.1 and C.5; only 15 manual checks are reported, and the same model family performs matching, so bias is possible.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review." pith.science (2026). https://pith.science/paper/PWA53VYX

@misc{pith2026250607642,
  author       = {Pith},
  title        = {Pith review of: TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PWA53VYX}},
  note         = {Machine review of arXiv:2506.07642}
}
read the original abstract

While Large Language Models (LLMs) have shown significant potential in assisting peer review, current methods often struggle to generate thorough and insightful reviews while maintaining efficiency. In this paper, we propose TreeReview, a novel framework that models paper review as a hierarchical and bidirectional question-answering process. TreeReview first constructs a tree of review questions by recursively decomposing high-level questions into fine-grained sub-questions and then resolves the question tree by iteratively aggregating answers from leaf to root to get the final review. Crucially, we incorporate a dynamic question expansion mechanism to enable deeper probing by generating follow-up questions when needed. We construct a benchmark derived from ICLR and NeurIPS venues to evaluate our method on full review generation and actionable feedback comments generation tasks. Experimental results of both LLM-based and human evaluation show that TreeReview outperforms strong baselines in providing comprehensive, in-depth, and expert-aligned review feedback, while reducing LLM token usage by up to 80% compared to computationally intensive approaches. Our code and benchmark dataset are available at https://github.com/YuanChang98/tree-review.

Figures

Figures reproduced from arXiv: 2506.07642 by the authors.

Figure 1
Figure 1. Current LLM-driven review methods face key [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of TreeReview framework. (A) Top-down Question Generation Stage: The overall process of hierarchical question decomposition. (B) Bottom-up Answer Aggregation Stage: The overall process of aggregating answers from leaf to root, producing the final review. (a) Decomposing a non-leaf question. (b) Answering a leaf question. (c) Dynamically raising follow-up sub-questions and synthesizing the answer for an inte… view at source ↗
Figure 3
Figure 3. LLM evaluation scores across quality dimen [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Proportion of generated comments judged as “highly related” and “more specific” in LLM-based alignment evaluation across different methods. In [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Human evaluation pair-wise win-rates for full [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: t-SNE visualization of sampled papers show [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Instructions for extracting feedback comments from human reviews. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Instructions for merging multiple sets of feedback comments from different human reviewers. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Instructions for the LLM-as-Judge evaluation. [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]
Figure 10
Figure 10. Figure 10: The interface used for human evaluation. It includes guidelines for evaluators, the paper PDF, and pairs [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: Example of review question tree constructed by [PITH_FULL_IMAGE:figures/full_fig_p026_11.png]
Figure 12
Figure 12. Figure 12: Example of full review produced by TreeReview [PITH_FULL_IMAGE:figures/full_fig_p027_12.png]
Figure 13
Figure 13. Figure 13: Prompt for the Question Generator in decomposing review questions. [PITH_FULL_IMAGE:figures/full_fig_p028_13.png]
Figure 14
Figure 14. Figure 14: Prompt for the Answer Synthesizer in answering leaf questions. [PITH_FULL_IMAGE:figures/full_fig_p029_14.png]
Figure 15
Figure 15. Figure 15: Prompt for the Answer Synthesizer in aggregating answers and generating follow-up questions for [PITH_FULL_IMAGE:figures/full_fig_p030_15.png]
Figure 16
Figure 16. Figure 16: Prompt for the Answer Synthesizer in generating the full review at the root level. [PITH_FULL_IMAGE:figures/full_fig_p031_16.png]
Figure 17
Figure 17. Figure 17: Prompt for the Answer Synthesizer in generating actionable feedback comments at the root level. [PITH_FULL_IMAGE:figures/full_fig_p032_17.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AI4Research: A Survey of Artificial Intelligence for Scientific Research

    cs.CL 2025-07 conditional novelty 4.0 of 10

    A survey that organizes AI-for-research work into five tasks, comprehension, survey, discovery, writing, and peer review, and compiles associated tools and benchmarks.

Reference graph

Works this paper leans on

46 extracted references · 41 canonical work pages · cited by 1 Pith paper

  1. [1]

    Experiments are not convincing

    If multiple reviewers mention the same issue, combine them into a single comment that preserves all details, ensuring no duplicate comments. If one reviewer provides a more detailed explanation than another on the same point, include the more comprehensive version with all specifics. For example, a reviewer might give a high-level comment like "Experiment...

    work page

  2. [2]

    If conflicting comments exist between reviewers, preserve all conflicting viewpoints in the final list, do not attempt to resolve contradictions

    work page

  3. [3]

    Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei

    Multilingual large language models and curse of multilinguality.arXiv preprint arXiv:2406.10602. Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei

    arXiv

  4. [4]

    Ensure each comment in your final list is fully contextualized and can stand alone as a complete comment

    work page

  5. [5]

    InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1658–1677, Bangkok, Thailand

    LongLLMLingua: Accelerating and enhanc- ing LLMs in long context scenarios via prompt com- pression. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1658–1677, Bangkok, Thailand. Association for Computational Linguistics. Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zu...

    work page 2018

  6. [6]

    **Evidence Support**: Does the review reference specific examples, sections, or data from the paper to substantiate its observations and feedback? Is the referenced content faithful to the original paper?

    work page

  7. [7]

    **Consistency**: Is the review internally consistent? Does it contain contradictory viewpoints?

    work page

  8. [8]

    Section Title > Sub-Section Title >

    EPO: Hierarchical LLM agents with environ- ment preference optimization. InProceedings of the 2024 Conference on Empirical Methods in Natu- ral Language Processing, pages 6401–6415, Miami, Florida, USA. Association for Computational Lin- guistics. Wenqing Zheng, SP Sharan, Ajay Kumar Jaiswal, Kevin Wang, Yihan Xi, Dejia Xu, and Zhangyang Wang. 2023a. Outl...

    arXiv 2024

Show all 46 references
  1. [10]

    • Technical Depth: Evaluates if the review demon- strates a strong understanding of the paper’s tech- nical content and the relevant research area

    We design the following eight distinct quality dimensions: • Comprehensiveness: Assesses whether the re- view covers all crucial aspects of the paper, such as the significance of the research problem, inno- vation, methodological soundness, etc. • Technical Depth: Evaluates if...

    2024

  2. [11]

    Do not reframe comments as reported speech or add attributions

    Try to preserve original wording, voice, and phrasing whenever possible, with minimal rewording only when necessary for clarity or to properly merge similar comments. Do not reframe comments as reported speech or add attributions

  3. [13]

    First consolidated comment

    Do not add new critiques or suggestions that weren’t present in the original review comments. Your output should be a single JSON array of strings, where each string is a complete, consolidated comment. Do not include any numbering, bullet points, or other special markers in t...

    2024

  4. [15]

    none”, “weak

    repetitive segments within one review and 2) generic segments that appear across reviews for multiple papers. A higher ITF-IDF score indicates that the generated comments are more specific to the content of the individual paper and less generic across different papers. The ITF...

    2024

  5. [16]

    **Comprehensiveness**: Does the review assess all important dimensions of the paper, including the significance of the research question, innovation and originality, methodological rigor, experimental design and analysis, potential impact on the field, and other key aspects?

  6. [17]

    **Technical Depth**: Does the review demonstrate a thorough understanding of the paper’s content and the related research domain? Does it identify subtle yet significant technical issues?

  7. [18]

    **Clarity**: Does the review accurately and clearly identify specific strengths, weaknesses, and unclear aspects of the paper?

  8. [19]

    **Constructiveness**: Is the review constructive and helpful in nature? Can the provided suggestions or insights really help improve the paper?

  9. [20]

    **Specificity**: Is the review focused on particular issues within the given paper, rather than being overly generic or applicable to other papers?

  10. [23]

    C om pr eh en siv en es s

    **Overall Quality**: Considering all aspects, how would you score the overall quality of the review? Before assigning any scores, carefully analyze the review against each evaluation criterion, thinking step-by-step. For each criterion, first provide a concise reason, then ass...

  11. [24]

    The impact of the α parameter in the distance calculation is not discussed

    The domain transformation method seems arbitrary and lacks a clear explanation of why the specific distance metrics and spatial arrangement are chosen. The impact of the α parameter in the distance calculation is not discussed. Ours · · · The relevance of mini-ImageNet, a data...

  12. [25]

    superior performance by outstripping the TablLMO in a 4-shot task with a maximum 0.11 AUC and a STUNT in a 1-shot setting, where it led on average by3.17%accuracy

    The experimental results are only briefly mentioned ("superior performance by outstripping the TablLMO in a 4-shot task with a maximum 0.11 AUC and a STUNT in a 1-shot setting, where it led on average by3.17%accuracy"). There is no detailed experimental setup, datasets used, e...

  13. [26]

    Only use information explicitly stated in the provided Relevant Context

  14. [27]

    If the text is ambiguous or incomplete, acknowledge the limitation and refrain from filling gaps with assumptions

    Avoid making inferences, predictions, or hypotheses that are not directly supported by the text. If the text is ambiguous or incomplete, acknowledge the limitation and refrain from filling gaps with assumptions

  15. [28]

    Avoid casual phrasing, exaggeration, or emotional language

    Use formal, precise, and objective language. Avoid casual phrasing, exaggeration, or emotional language

  16. [29]

    Sufficient Evidence

    Provide Detailed Evidence: For each comment, include specific evidence from the given context (e.g., quotes, section references, or data points) to justify your point. INPUT: - Review Question: {QUESTION} - Relevant Context: {CONTEXT} OUTPUT FORMAT: A single string containing ...

  17. [30]

    Focus on evaluating the paper’s claims, methodology, and conclusions critically

    Critical Reviewer Perspective: From the perspective of a peer reviewer, not the author. Focus on evaluating the paper’s claims, methodology, and conclusions critically. Avoid defending the paper or emphasizing its contributions without sufficient evidence

  18. [31]

    Never reference external knowledge or invent claims

    Input-Bound Synthesis: Use only the provided sub-Q&A pairs. Never reference external knowledge or invent claims

  19. [32]

    Synthesize insights that go beyond surface-level observations, critically evaluating the strength of evidence and exploring the broader implications of the findings

    Analytical Depth: Dive deeply into the sub-answers to uncover patterns, contradictions, and gaps. Synthesize insights that go beyond surface-level observations, critically evaluating the strength of evidence and exploring the broader implications of the findings

  20. [33]

    Highlight any significant findings or unresolved issues

    Critical Thinking: Consider the implications of the sub-answers and how they collectively address the main question. Highlight any significant findings or unresolved issues

  21. [34]

    Provide Detailed Evidence: For each insight in your synthesized answer, include specific evidence from the sub-Q&A pairs (e.g., quotes, section references, or data points) to justify your point

  22. [35]

    chain_of_thought

    Chain of Thought: Clearly articulate your reasoning process, showing how you derived your conclusions from the sub-answers. This should include a step-by-step explanation of your thought process. OUTPUT FORMAT FOR SUFFICIENT EVIDENCE: A JSON object containing the chain of thou...

  23. [36]

    Evidence Assessment: If the provided sub-Q&A pairs are insufficient to answer the main question, propose up to MAX QUESTION NUM follow-up questions that need to be answered to address the main question adequately

  24. [37]

    Determine what additional information is required to address the main question adequately

    Analytical Depth: Analyze the sub-answers to identify specific areas where the evidence is lacking or contradictory. Determine what additional information is required to address the main question adequately

  25. [38]

    chain_of_thought

    Chain of Thought: Clearly articulate your reasoning process, showing how you identified the gaps in the evidence and why the proposed follow-up questions are necessary. This should include a step-by-step explanation of your thought process. OUTPUT FORMAT FOR INSUFFICIENT EVIDE...

  26. [39]

    Firstly, you should carefully read through the entire paper

  27. [40]

    Secondly, it’s important to use the questions and their corresponding answers as a guiding framework to help you deeply understand the paper and ensure a comprehensive review

  28. [41]

    REQUIREMENTS

    Based on the analysis from the first two steps, compose a thorough and comprehensive review. REQUIREMENTS

  29. [42]

    Instead, use the insights from them to inform your review process

    While the question-answer pairs are important inputs for your analysis, your review should focus on the paper itself and avoid directly mentioning the Q&A pairs. Instead, use the insights from them to inform your review process

  30. [43]

    In your review, you must cover the following aspects: [ICLR and NIPS Reviewer Guideline] INPUT - Paper Content: PAPER CONTENT - Questions and answers: QUESTIONS AND ANSWERS OUTPUT FORMAT Here is the template for a review format. You must follow this format to output the integr...

  31. [44]

    You should first carefully read through the entire paper

  32. [45]

    It’s important to use the review questions and their corresponding answers as reference to guide and enhance your review thinking process. However, if after reading the entire paper you think some viewpoints or insights in the review Q&A pairs to be incorrect or insufficient, ...

  33. [46]

    feedback comment1

    Identify weak points of the paper, and write them as feedback comments. For each of your comments, it should: - Focus on the paper’s weaknesses, limitations, potential flaws, and areas for improvement, or raise questions that highlight the need for clarification and further an...

  34. [2019]

    Surv., 52(6)

    Academic plagiarism detection: A systematic literature review.ACM Comput. Surv., 52(6). Zhaolin Gao, Kianté Brantley, and Thorsten Joachims

  35. [2022]

    Ari Kobren, Barna Saha, and Andrew McCallum

    Decomposed prompting: A modular approach for solving complex tasks.CoRR, abs/2210.02406. Ari Kobren, Barna Saha, and Andrew McCallum. 2019. Paper matching with local fairness constraints. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & D...

    2019 arXiv

  36. [2023]

    InFindings of the Association for Computational Linguistics: EMNLP 2023

    Not all languages are created equal in llms: Improving multilingual capability by cross-lingual- thought prompting. InFindings of the Association for Computational Linguistics: EMNLP 2023. Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dong- sheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu

    2023

  37. [2024]

    Robert E Gropp, Scott Glisson, Stephen Gallo, and Lisa Thompson

    Reviewer2: Optimizing review genera- tion through prompt generation.arXiv preprint arXiv:2402.10886. Robert E Gropp, Scott Glisson, Stephen Gallo, and Lisa Thompson. 2017. Peer review: A system under stress. BioScience, 67(5):407–410. Xiao-Yu Guo, Yuan-Fang Li, and Gholamreza ...

    2017 arXiv

  38. [2025]

    InThe Thirteenth Inter- national Conference on Learning Representations

    Cycleresearcher: Improving automated re- search via automated review. InThe Thirteenth Inter- national Conference on Learning Representations. Hang Yang, Hao Chen, Hui Guo, Yineng Chen, Ching- Sheng Lin, Shu Hu, Jinrong Hu, Xi Wu, and Xin Wang. 2024a. Llm-medqa: Enhancing medi...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.