Large language models for automated scholarly paper review: A survey

Hongfeng Xu; Jialiang Lin; Jiandong Chen; Yuwen Jiang; Zhenzhen Zhuang

REVIEW 5 major objections 5 minor 2 cited by

Large language models for automated scholarly paper review: A survey

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This survey tries to establish a comprehensive map of automated scholarly paper review as it has developed in the era of large language models, covering models, methods, datasets, source code, publisher policies, and open problems.

desk verdict A genuinely useful survey of LLM-based automated paper review, with a few fixable citation and sourcing errors; worth peer review. read the letter →

arxiv 2501.10326 v2 pith:QUK66XFY submitted 2025-01-17 cs.AI cs.CLcs.DL

Zhenzhen Zhuang , Jiandong Chen , Hongfeng Xu , Yuwen Jiang , Jialiang Lin This is my paper · ORCID

classification cs.AIcs.CLcs.DL

keywords automatedscholarlypaperreviewlargelanguagemodelspeerLLM-basedgenerationmulti-agentsystemsdatasetspublisherpoliciesacademicpublishing

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This survey sets out to establish a comprehensive map of automated scholarly paper review (ASPR) as it exists in the era of large language models. It argues that LLMs have turned ASPR from a theoretical idea into a rapidly growing research area, one in which automated systems now coexist with and assist traditional peer review. The authors collect work from 2023 to 2024 using snowballing from five seed papers, then organize it by which models are used, which technical bottlenecks have been solved, which methods generate review reports, and which datasets and source code are available. They also catalog performance problems and publisher policies. If the map is accurate, it gives researchers a reliable reference for the tools, gaps, and open challenges in building automated paper review.

What carries the argument

The survey's organizing machinery is the ASPR pipeline—screening, main review, comment generation, review-quality assessment, and author response—used as a grid to classify papers, datasets, and code. Onto this grid the authors map a set of LLM capabilities that resolve earlier bottlenecks (long-text modeling, multimodal input, multi-turn conversation, instant knowledge retrieval) and a set of generation methods (prompting, fine-tuning, multi-agent frameworks) that produce review reports. This structure is what lets the survey turn a collection of papers into a holistic picture of the field.

What would settle it

Search a bibliographic database for papers combining 'peer review' and 'large language model' published before January 2023, or in journals and conferences not connected to the five seed papers; finding a substantial number of such works would show the survey's corpus is incomplete and falsify the comprehensiveness claim.

Watch

Extended reading notes

Core claim

The central claim is that LLM-driven ASPR has reached a stage the authors call the coexistence phase, in which automated review serves as a human assistant rather than a full replacement, and that this phase is now substantial enough to survey. The paper catalogs the field across five axes: the LLMs used, with closed-source models like GPT-4 dominating usage and open-source models catching up; the technical bottlenecks solved, including long-text processing, multimodal input, multi-turn conversation, and real-time knowledge acquisition; the methods for generating review reports, namely prompt engineering, supervised fine-tuning, and multi-agent frameworks; the available datasets and source code; and the current performance issues, including insufficient comprehension, bias, hallucination, confidentiality risks, and limited customization. The paper also reports that most major publishers prohibit reviewers from using AI-generated content tools, while a few are building in-house AI review assistants, and it extracts from academia a set of recommendations including private deployment, reviewer training, transparency, and alignment with scholarly values.

Load-bearing premise

The survey's claim to provide a holistic view depends on the assumption, stated in Section 1, that snowballing from five seed papers restricted to 2023–2024 captures all the field's significant work; if important LLM-based review systems exist outside that window or citation chain, the picture is incomplete.

Editorial extensions

If this is right

A researcher new to the field can use the survey's tables to select an LLM, a review-generation method, and a dataset for a specific subtask without redoing the literature search.
The performance findings imply that current LLM-generated reviews are not yet dependable enough to replace human reviewers: they are prone to false positives in error detection, inflated scores, and hallucinated references.
Because most publishers currently prohibit AI-generated review content, scaling ASPR depends on policy change or on the in-house AI tools a few publishers are already building.
The scarcity of multimodal review datasets is the main constraint on multimodal LLM-based review, so dataset construction may be a higher-leverage activity than new model development.
The survey's catalog of open challenges—hallucination correction, multimodal generation, reasoning models, generative attacks, low-resource private deployment, and personalized review—defines a concrete research agenda.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A likely next step, if the documented trend continues, is that full-scale ASPR will first deploy in low-stakes tasks such as desk screening and meta-review, where a wrong decision is less costly than in final accept/reject judgments.
The survey's taxonomy implies a testable hypothesis: fine-tuned open-source reviewers will close much of the quality gap with closed-source general-purpose models once large, high-quality review datasets become available.
The survey reports author helpfulness ratings but not whether authors would consent to fully automated review; that consent question may be the true adoption bottleneck.
One missing element in the surveyed corpus is longitudinal evidence on whether LLM-assisted review actually changes editorial outcomes; a causal study comparing acceptance rates with and without LLM assistance would test the coexistence-phase premise.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a genuinely useful survey of LLMs for automated scholarly paper review (ASPR) in 2023-2024, and the summary tables—models, datasets, code, publisher policies—are the real contribution. It is curation, not new research, but the curation is mostly careful and the organization is clear. I disagree with any take that calls this a weak paper; as a reference map it earns its keep. The central “holisic view” claim is broadly supported by the cited literature.

The soft spots are real but individually minor. Table 3 misattributes ARIES to Couto et al. (2024) when the ACL paper is D’Arcy et al. (2024); that’s the kind of error a careful reader will catch and it undermines trust in the table. The Llama 3 “23.6% logical reasoning gain” figure is cited to Touvron et al. (2023a) and Liang et al. (2023), but neither source actually says that, so it’s unsourced. Claude 3 is listed as MoE-Decoder without official confirmation—speculative. And the snowballing protocol is described in one sentence: five seeds, forward/backward search, 2023-2024 window, but no iteration count, screening log, or saturation criterion. The stress-test note makes a fair point: as reported, the protocol isn’t auditable, so the “holisic view” claim is weaker than it could be. But I don’t think that’s lethal—the paper covers a wide range of work beyond the seeds, including independent datasets and systems, so the map is likely reasonably complete in spirit. It’s a transparency problem, not a fatal sampling flaw.

The paper also leans on the authors’ earlier ASPR work (Lin et al. 2023a) for definitions and ethics, which is acceptable but creates a slight self-citation skew. The publisher policy table is useful and appropriately labeled where unknown.

Bottom line: the paper is for researchers entering this area, or for anyone wanting a quick overview of models, datasets, and publisher stances. I’d cite it, and I’d send it to peer review—it deserves referee time, but only with the expectation that the authors fix the citation errors, source or remove the Llama 3 claim, and document the search protocol more fully.

Referee Report

5 major / 5 minor

Summary. This survey reviews the state of automated scholarly paper review (ASPR) in the era of large language models, organizing the literature into the LLMs used, the technological capabilities they bring (long-text modeling, multimodality, multi-turn dialogue, knowledge acquisition), methods for generating reviews (prompting, fine-tuning, multi-agent frameworks), new datasets, source code and online systems, observed performance and limitations, publisher policies, academic suggestions, and future directions. The literature is collected by snowballing from five seed papers, restricted to works published between 2023 and 2024 that use LLMs for ASPR.

Significance. If the collected corpus is accepted as representative, the survey provides a useful and well-structured map of a fast-moving field, and its consolidated tables of models, datasets, source code, and publisher policies are practical resources for researchers. The authors also make a commendable effort to cover the full pipeline from screening to author response, and they identify several concrete open problems such as multimodal review input, reasoning-model integration, and privately hosted deployment. However, the central claim of a 'comprehensive' or 'holistic' view depends on the completeness and replicability of the snowballing procedure, and that procedure is currently not auditable; several specific factual errors in the survey tables and text further reduce confidence. The contribution is therefore valuable but not yet fully reliable.

major comments (5)

[Section 1 (research method)] The snowballing protocol is under-specified and cannot support the 'comprehensive review'/'holistic view' claim. The text names five seed papers and says the collection was expanded by forward and backward citation searches following Wohlin (2014), but it gives no inclusion/exclusion rationale for the seeds, no number of iterations, no list of screened and excluded papers, and no saturation criterion. Without such documentation, the corpus is not auditable or replicable, and any LLM-based ASPR work that neither cites nor is cited by the five seeds is systematically invisible. Please provide a protocol document (e.g., a PRISMA-style flow diagram), report the iterations and decisions, and ideally complement snowballing with an independent database search to verify recall.
[Table 3 and references] Table 3 misattributes ARIES to Couto et al. (2024) in both the 'Comment generation' and 'Author response' rows. The linked repository (github.com/allenai/aries) and the reference list identify ARIES as D'Arcy et al. (2024), 'ARIES: A corpus of scientific paper edits made in response to peer reviews.' This is a factual error in a table whose purpose is to provide reproducible resources, and it is inconsistent with the same work's correct listing under author response in Table 2.
[Section 2.1, page 3] The statement that 'Llama 3, by incorporating 17% structured code data into its pre-training corpus, has improved its zero-shot logical reasoning capability by 23.6% compared to its predecessor' is not supported by the cited references, Touvron et al. (2023a) and Liang et al. (2023), both of which predate Llama 3 and contain no such statistics. Provide a verifiable primary source (e.g., the Llama 3 model card) or remove the quantitative claim.
[Section 2.2 and Table 1] The architecture classification of Claude 3 as 'MoE-Dec' in Table 1 is not justified by the cited Anthropic source, which does not describe Claude 3 as a mixture-of-experts model. Additionally, the text in Section 2.2 attributes Gemini 1.5's MoE architecture to 'Xue et al. (2024)' but that reference is a paper on wireless distributed MoE, not the Gemini 1.5 technical report. Please correct the citation to Gemini Team (2024) and either provide a source for Claude 3's architecture or mark it as undisclosed.
[Section 1, last paragraph] The survey explicitly defers all discussion of biases, fairness, ethics, accountability, and misuse to the authors' earlier work (Lin et al., 2023a). While the earlier work is relevant, this deferral is a substantive gap for a survey claiming a holistic view of LLM-driven ASPR: the new context of LLM-generated reviews introduces or sharpens several of these issues (e.g., training-data contamination, reviewer-accountability, disclosure policies), and simply referring readers elsewhere is not a substitute for an integrated treatment. Add at least a summary subsection on these issues and explain how the 2023-2024 literature addresses them.

minor comments (5)

[Section 4.1, page 11] The claim that Robertson (2023) 'appears to be the first research paper to employ LLMs for large-scale generation of review comments' is a strong attribution that the survey itself hedges with 'appears to be'; please either substantiate the priority claim with a systematic check or rephrase it as an early example.
[Section 8.5, page 21] The statement that 'a growing phenomenon has emerged where different reviewers utilize LLMs to generate review comments that are strikingly homogeneous' lacks a supporting citation; please cite evidence or qualify it as an observation from the reviewed literature.
[References] There are two separate entries for 'D'Arcy et al. (2024)' (MARG and ARIES), which creates ambiguity in in-text citations; please disambiguate with suffixes (e.g., D'Arcy et al., 2024a, 2024b).
[Section 3.1, page 9] Please use consistent capitalization for model names, e.g., 'gpt-3.5-turbo-16k' should be 'GPT-3.5-turbo-16k' or 'gpt-3.5-turbo-16k' as an API identifier, and clarify which is intended.
[Table 1] The 'ChatReviewer 2023' row in Table 3 does not list an author in the approach column; for consistency, include the citation (Ni, 2023) or a footnote.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; self-citations frame the survey but none of its findings reduce to the authors' own prior work.

full rationale

This is a literature survey, not a derivation chain: there are no fitted parameters, no equations, and no quantities predicted from inputs. The authors' self-citations (notably Lin et al. 2023a) supply the ASPR concept, review-phase taxonomy, a definition of ASPR as one-turn, and a pointer to earlier ethics discussion, but the survey's substantive claims—which LLMs are used, which methods and datasets exist, performance findings, publisher policies, and academic suggestions—are summaries of independent cited works with their own empirical content. No passage exhibits a reduction of a conclusion to an input: no equation is shown to equal another equation by construction, no fitted parameter is renamed as a prediction, and no external result is replaced by a self-citation chain. The snowballing protocol is under-specified and not auditable, but that is a coverage and rigor risk, not circularity; an incomplete search does not make the survey's contents equivalent to its selection criteria. Accordingly, no circular step is flagged, and the score reflects only the presence of minor, non-load-bearing self-citations.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No empirical model or fitting is performed, so there are no free parameters or invented entities. The survey relies on background assumptions about literature coverage and on the accuracy of secondhand reporting.

assumptions (4)

domain assumption Snowballing from five seed papers is sufficient to identify the relevant ASPR literature.
Section 1 describes snowballing with seeds Hosseini and Horbach 2023, Robertson 2023, Biswas et al. 2023, Liu and Shah 2024, and Liang et al. 2024b; the coverage of the whole survey depends on this.
domain assumption The 2023-2024 publication window captures the LLM era for automated scholarly paper review.
Section 1 sets the inclusion window to 2023-2024 because the LLM era matured with ChatGPT; earlier or later relevant work could be excluded by this boundary.
domain assumption The authors' previous ASPR concept and terminology (Lin et al. 2023a) provides the correct frame for the field.
Sections 1 and 2 build on Lin et al. 2023a for definitions, pipeline, and as a baseline for what has changed; this frames which papers count as ASPR.
domain assumption Quantitative findings quoted from the surveyed papers are transcribed accurately.
Section 7 reports accuracy figures such as 86.6% for checklist verification and 7 of 13 error-identification successes without independent re-validation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Large language models for automated scholarly paper review: A survey." pith.science (2026). https://pith.science/paper/QUK66XFY

@misc{pith2026250110326,
  author       = {Pith},
  title        = {Pith review of: Large language models for automated scholarly paper review: A survey},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QUK66XFY}},
  note         = {Machine review of arXiv:2501.10326}
}

read the original abstract

Large language models (LLMs) have significantly impacted human society, influencing various domains. Among them, academia is not simply a domain affected by LLMs, but it is also the pivotal force in the development of LLMs. In academic publication, this phenomenon is represented during the incorporation of LLMs into the peer review mechanism for reviewing manuscripts. LLMs hold transformative potential for the full-scale implementation of automated scholarly paper review (ASPR), but they also pose new issues and challenges that need to be addressed. In this survey paper, we aim to provide a holistic view of ASPR in the era of LLMs. We begin with a survey to find out which LLMs are used to conduct ASPR. Then, we review what ASPR-related technological bottlenecks have been solved with the incorporation of LLM technology. After that, we move on to explore new methods, new datasets, new source code, and new online systems that come with LLMs for ASPR. Furthermore, we summarize the performance and issues of LLMs in ASPR, and investigate the attitudes and reactions of publishers and academia to ASPR. Lastly, we discuss the challenges and future directions associated with the development of LLMs for ASPR. This survey serves as an inspirational reference for the researchers and can promote the progress of ASPR for its actual implementation.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

How Far Are AI Scientists from Changing the World?
cs.AI 2025-07 conditional novelty 4.0 of 10

This survey proposes a four-level capability framework for AI Scientist systems and, using an AI reviewer, finds that current systems produce papers rated well below normal scientific standards.
Research quality evaluation by AI in the era of Large Language Models: Advantages, disadvantages, and systemic effects
cs.DL 2025-06 conditional novelty 4.0 of 10

A review arguing LLM-based quality scores could surpass bibliometrics in accuracy and coverage, but with unknown biases and gaming risks that currently block real-world use.

Reference graph

Works this paper leans on

107 extracted references · 34 canonical work pages · cited by 2 Pith papers

[1]

title Peer Review at Science Journals

author AAAS ( year 2024 ). title Peer Review at Science Journals . https://www.science.org/content/page/peer-review-science-publications

2024
[2]

title FAQ: Can I use AI writing assistants to write my review? https://2023.aclweb.org/blog/review-acl23/#faq-can-i-use-ai-writing-assistants-to-write-my-review

author ACL ( year 2023 ). title FAQ: Can I use AI writing assistants to write my review? https://2023.aclweb.org/blog/review-acl23/#faq-can-i-use-ai-writing-assistants-to-write-my-review

2023
[3]

title ACM Peer Review Policy

author ACM ( year 2023 ). title ACM Peer Review Policy . https://www.acm.org/publications/policies/peer-review

2023
[4]

title ACM Peer Review Policy Frequently Asked Questions ,

author ACM ( year 2024 ). title ACM Peer Review Policy Frequently Asked Questions ,

2024
[5]

( year 2023 )

author AI, S. ( year 2023 ). title Stability ai launches the first of its stablelm suite of language models . journal Blog post, Apr \/ ,

2023
[6]

title Ethics for Editors and Reviewers

author AIP ( year 2024 ). title Ethics for Editors and Reviewers . https://publishing.aip.org/resources/researchers/policies-and-ethics/editors-and-reviewers/

2024
[7]

, author Elsayed, M

author Alshami, A. , author Elsayed, M. , author Ali, E. , author Eltoukhy, A. E. , & author Zayed, T. ( year 2023 ). title Harnessing the power of ChatGPT for automating systematic review process: Methodology, case study, limitations, and future directions . journal Systems \/ , volume 11 \/ , pages 351

2023
[8]

( year 2024 )

author Anthropic, A. ( year 2024 ). title The claude 3 model family: Opus, sonnet, haiku . journal Claude-3 Model Card \/ , volume 1 \/

2024

Show all 107 references

[9]

author Bharti, P. K. , author Navlakha, M. , author Agarwal, M. , & author Ekbal, A. ( year 2023 ). title PolitePEER: does peer review hurt? A dataset to gauge politeness intensity in the peer reviews . journal Language Resources and Evaluation \/ , . :10.1007/s10579-023-09662-3

2023 doi
[10]

, author Dobaria, D

author Biswas, S. , author Dobaria, D. , & author Cohen, H. L. ( year 2023 ). title ChatGPT and the future of journal reviews: A feasibility study . journal The Yale Journal of Biology and Medicine \/ , volume 96 \/ , pages 415--420 . :10.59249/SKDH9286

2023 doi
[11]

author Biswas, S. S. ( year 2024 ). title AI-assisted academia: Navigating the nuances of peer review with ChatGPT 4 . journal The Journal of Pediatric Pharmacology and Therapeutics \/ , volume 29 \/ , pages 441--445 . :10.5863/1551-6776-29.4.441

2024 doi
[12]

, & author Watanabe, N

author Bougie, N. , & author Watanabe, N. ( year 2024 ). title Generative Adversarial Reviews: When LLMs Become the Critic . journal arXiv preprint arXiv:2412.10415 \/ ,

2024 arXiv
[13]

, author Mann, B

author Brown, T. , author Mann, B. , author Ryder, N. , author Subbiah, M. , author Kaplan, J. D. , author Dhariwal, P. , author Neelakantan, A. , author Shyam, P. , author Sastry, G. , author Askell, A. , author Agarwal, S. , author Herbert-Voss, A. , author Krueger, G. , aut...

2020
[14]

, author Schlichtkrull, M

author Chamoun, E. , author Schlichtkrull, M. , & author Vlachos, A. ( year 2024 ). title Automated focused feedback generation for scientific writing assistance

2024
[15]

author Chawla, D. S. ( year 2024 ). title Is ChatGPT corrupting peer review? Telltale words hint at AI use . journal Nature \/ , volume 628 \/ , pages 483--484 . :10.1038/d41586-024-01051-2

2024 doi
[16]

, author Li, Z

author Chiang, W.-L. , author Li, Z. , author Lin, Z. , author Sheng, Y. , author Wu, Z. , author Zhang, H. , author Zheng, L. , author Zhuang, S. , author Zhuang, Y. , author Gonzalez, J. E. , author Stoica, I. , & author Xing, E. P. ( year 2023 ). title Vicuna: An open-sourc...

2023
[17]

, author Ai, Q

author Chu, Z. , author Ai, Q. , author Tu, Y. , author Li, H. , & author Liu, Y. ( year 2024 ). title Automatic Large Language Model Evaluation via Peer Review . In booktitle Proceedings of the 33rd ACM International Conference on Information and Knowledge Management \/

2024
[18]

( year 2023 )

author Claude, I. ( year 2023 ). title Introducing Claude . https://www.anthropic.com/news/introducing-claude note claude1

2023
[19]

, author Hayes, M

author Conover, M. , author Hayes, M. , author Mathur, A. , author Meng, X. , author Xie, J. , author Wan, J. , author Ghodsi, A. , author Wendell, P. , & author Zaharia, M. ( year 2023 ). title Hello Dolly: Democratizing the magic of ChatGPT with open models . journal Databri...

2023
[20]

( year 2023 )

author Conroy, G. ( year 2023 ). title How ChatGPT and other AI tools could disrupt scientific publishing . journal Nature \/ , volume 622 \/ , pages 234--236 . :10.1038/d41586-023-03144-w

2023 doi
[21]

author Couto, P. H. , author Ho, Q. P. , author Kumari, N. , author Rachmat, B. K. , author Khuong, T. G. H. , author Ullah, I. , & author Sun-Hosoya, L. ( year 2024 ). title RelevAI-Reviewer: A Benchmark on AI Reviewers for Survey Paper Relevance . In booktitle CAp 2024 - Con...

2024
[22]

, author Hope, T

author D'Arcy, M. , author Hope, T. , author Birnbaum, L. , & author Downey, D. ( year 2024 ). title MARG: Multi-agent review generation for scientific papers . journal arXiv preprint arXiv:2401.04259 \/ ,

2024 arXiv
[23]

, & author Te'eni, D

author Drori, I. , & author Te'eni, D. ( year 2024 ). title Human-in-the-Loop AI Reviewing: Feasibility, Opportunities, and Risks . journal Journal of the Association for Information Systems \/ , volume 25 \/ , pages 98--109 . :10.17705/1jais.00867

2024 doi
[24]

, author Wang, Y

author Du, J. , author Wang, Y. , author Zhao, W. , author Deng, Z. , author Liu, S. , author Lou, R. , author Zou, H. P. , author Venkit, P. N. , author Zhang, N. , author Srinath, M. , author Zhang, H. R. , author Gupta, V. , author Li, Y. , author Li, T. , author Wang, F. ,...

2024
[25]

, author Jauhri, A

author Dubey, A. , author Jauhri, A. , author Pandey, A. , author Kadian, A. , author Al-Dahle, A. , author Letman, A. , author Mathur, A. , author Schelten, A. , author Yang, A. , & author Fan, A. ( year 2024 ). title The Llama 3 herd of models . journal arXiv preprint arXiv:...

2024 arXiv
[26]

, author Kuznetsov, I

author Dycke, N. , author Kuznetsov, I. , & author Gurevych, I. ( year 2023 ). title NLPeer: A Unified Resource for the Computational Study of Peer Review . :10.18653/v1/2023.acl-long.277

2023 doi
[27]

, author Garmendia, X

author Díaz, O. , author Garmendia, X. , & author Pereira, J. ( year 2024 ). title Streamlining the review process: AI-generated annotations in research manuscripts . journal arXiv preprint arXiv:2412.00281 \/ ,

2024 arXiv
[28]

, author Ross, A

author D’Arcy, M. , author Ross, A. , author Bransom, E. , author Kuehl, B. , author Bragg, J. , author Hope, T. , & author Downey, D. ( year 2024 ). title ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviews

2024
[29]

title The use of generative AI and AI-assisted technologies in the journal peer review process

author Elsevier ( year 2024 ). title The use of generative AI and AI-assisted technologies in the journal peer review process . https://www.elsevier.com/about/policies-and-standards/generative-ai-policies-for-journals#2-for-reviewers

2024
[30]

title How to...Review a journal, case study or book proposal ,

author Emerald ( year 2024 ). title How to...Review a journal, case study or book proposal ,

2024
[31]

author Faizullah, A. R. B. M. , author Urlana, A. , & author Mishra, R. ( year 2024 ). title LimGen: Probing the LLMs for Generating Suggestive Limitations of Research Papers . :10.1007/978-3-031-70344-7\_7

2024 doi
[32]

, author Kendall-Taylor, J

author Flanagin, A. , author Kendall-Taylor, J. , & author Bibbins-Domingo, K. ( year 2023 ). title Guidance for Authors, Peer Reviewers, and Editors on Use of AI, Language Models, and Chatbots . journal JAMA \/ , volume 330 \/ , pages 702--703 . :10.1001/jama.2023.12500

2023
[33]

( year 2024 )

author Francis, T. . ( year 2024 ). title Guidelines for peer reviewers . https://editorresources.taylorandfrancis.com/reviewer-guidelines/

2024
[34]

title Artificial Intelligence to help meet global demand for high-quality, objective peer-review in publishing

author Frontiers ( year 2020 ). title Artificial Intelligence to help meet global demand for high-quality, objective peer-review in publishing . https://www.frontiersin.org/news/2020/07/01/artificial-intelligence-peer-review-assistant-aira/

2020
[35]

title Editorial policies and publication ethics

author Frontiers ( year 2024 ). title Editorial policies and publication ethics . https://www.frontiersin.org/guidelines/policies-and-publication-ethics

2024
[36]

, author Brantley, K

author Gao, Z. , author Brantley, K. , & author Joachims, T. ( year 2024 ). title Reviewer2: Optimizing review generation through prompt generation . journal arXiv preprint arXiv:2402.10886 \/ ,

2024 arXiv
[37]

, author Gudibande, A

author Geng, X. , author Gudibande, A. , author Liu, H. , author Wallace, E. , author Abbeel, P. , author Levine, S. , & author Song, D. ( year 2023 ). title Koala: A dialogue model for academic research . journal Blog post, April \/ , volume 1 \/ , pages 6

2023
[38]

( year 2024 )

author Gomez, A. ( year 2024 ). title Introducing command r+: A scalable llm built for business . journal Online Blog \/ ,

2024
[39]

, author Gupta, M

author Guo, E. , author Gupta, M. , author Deng, J. , author Park, Y.-J. , author Paget, M. , & author Naugler, C. ( year 2024 ). title Automated paper screening for clinical reviews using large language models: Data analysis study . journal Journal of Medical Internet Researc...

2024
[40]

, author Shang, G

author Guo, Y. , author Shang, G. , author Rennard, V. , author Vazirgiannis, M. , & author Clavel, C. ( year 2023 ). title Automatic analysis of substantiation in scientific peer reviews . :10.18653/v1/2023.findings-emnlp.684

2023 doi
[41]

, author Gao, C

author Han, Z. , author Gao, C. , author Liu, J. , author Zhang, J. , & author Zhang, S. Q. ( year 2024 ). title Parameter-efficient fine-tuning for large models: A comprehensive survey . journal arXiv preprint arXiv:2403.14608 \/ ,

2024 arXiv
[42]

, & author Horbach, S

author Hosseini, M. , & author Horbach, S. P. J. M. ( year 2023 ). title Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review . journal Research Integrity and Peer Review \/...

2023 doi
[43]

, author Yu, W

author Huang, L. , author Yu, W. , author Ma, W. , author Zhong, W. , author Feng, Z. , author Wang, H. , author Chen, Q. , author Peng, W. , author Feng, X. , author Qin, B. , & author Liu, T. ( year 2024 ). title A Survey on Hallucination in Large Language Models: Principles...

2024 doi
[44]

, & author Ahmadi, Z

author Idahl, M. , & author Ahmadi, Z. ( year 2024 ). title OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews . journal arXiv preprint arXiv:2412.11948 \/ ,

2024 arXiv
[45]

title Become an IEEE reviewer

author IEEE ( year 2024 ). title Become an IEEE reviewer . https://journals.ieeeauthorcenter.ieee.org/submit-your-article-for-peer-review/become-an-ieee-reviewer/

2024
[46]

author Jiang, A. Q. , author Sablayrolles, A. , author Mensch, A. , author Bamford, C. , author Chaplot, D. S. , author Casas, D. d. l. , author Bressand, F. , author Lengyel, G. , author Lample, G. , & author Saulnier, L. ( year 2023 ). title Mistral 7B . journal arXiv prepri...

2023 arXiv
[47]

, author Ghosal, T

author Kumar, S. , author Ghosal, T. , & author Ekbal, A. ( year 2023 ). title When reviewers lock horns: Finding disagreements in scientific peer reviews . :10.18653/v1/2023.emnlp-main.1038

2023 doi
[48]

, author Afzal, O

author Kuznetsov, I. , author Afzal, O. M. , author Dercksen, K. , author Dycke, N. , author Goldberg, A. , author Hope, T. , author Hovy, D. , author Kummerfeld, J. K. , author Lauscher, A. , & author Leyton-Brown, K. ( year 2024 ). title What can natural language processing ...

2024 arXiv
[49]

, author Kilcher, Y

author Köpf, A. , author Kilcher, Y. , author von Rütte, D. , author Anagnostidis, S. , author Tam, Z. R. , author Stevens, K. , author Barhoum, A. , author Nguyen, D. , author Stanley, O. , author Nagyfi, R. , author Es, S. , author Suri, S. , author Glushkov, D. , author Dan...

2023
[50]

( year 2017 )

author Laine, C. ( year 2017 ). title Scientific misconduct hurts . journal Annals of Internal Medicine \/ , volume 166 \/ , pages 148--149 . :10.7326/M16-2550

2017 doi
[51]

author Latona, G. R. , author Ribeiro, M. H. , author Davidson, T. R. , author Veselovsky, V. , & author West, R. ( year 2024 ). title The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Rates . journal arXiv preprint arXiv:2405.02150 \/ ,

2024 arXiv
[52]

author Leung, T. I. , author de Azevedo Cardoso, T. , author Mavragani, A. , & author Eysenbach, G. ( year 2023 ). title Best practices for using AI tools as an author, peer reviewer, or editor . journal Journal of Medical Internet Research \/ , volume 25 \/ , pages e51584 . :...

2023 doi
[53]

, author Hovy, E

author Li, M. , author Hovy, E. , & author Lau, J. ( year 2023 ). title Summarizing multiple documents with conversational structure for meta-review generation . :10.18653/v1/2023.findings-emnlp.472

2023 doi
[54]

, author Sun, J

author Li, M. , author Sun, J. , & author Tan, X. ( year 2024 a ). title Evaluating the effectiveness of large language models in abstract screening: a comparative analysis . journal Systematic Reviews \/ , volume 13 \/ , pages 219 . :10.1186/s13643-024-02609-x

2024 doi
[55]

, author Huang, J

author Li, S. , author Huang, J. , author Zhuang, J. , author Shi, Y. , author Cai, X. , author Xu, M. , author Wang, X. , author Zhang, L. , author Ke, G. , & author Cai, H. ( year 2024 b ). title SciLitLLM: How to Adapt LLMs for Scientific Literature Understanding . journal ...

2024 arXiv
[56]

, author Izzo, Z

author Liang, W. , author Izzo, Z. , author Zhang, Y. , author Lepp, H. , author Cao, H. , author Zhao, X. , author Chen, L. , author Ye, H. , author Liu, S. , author Huang, Z. , author McFarland, D. , & author Zou, J. Y. ( year 2024 ). title Monitoring AI-modified content at ...

2024
[57]

, author Peng, Z

author Lin, E. , author Peng, Z. , & author Fang, Y. ( year 2024 ). title Evaluating and Enhancing Large Language Models for Novelty Assessment in Scholarly Publications . journal arXiv preprint arXiv:2409.16605 \/ ,

2024 arXiv
[58]

, author Song, J

author Lin, J. , author Song, J. , author Zhou, Z. , author Chen, Y. , & author Shi, X. ( year 2023 a ). title Automated scholarly paper review: Concepts, technologies, and challenges . journal Information Fusion \/ , volume 98 \/ , pages 101830 . :10.1016/j.inffus.2023.101830

2023
[59]

, author Song, J

author Lin, J. , author Song, J. , author Zhou, Z. , author Chen, Y. , & author Shi, X. ( year 2023 b ). title MOPRD: A multidisciplinary open peer review dataset . journal Neural Computing and Applications \/ , volume 35 \/ , pages 24191--24206 . :10.1007/s00521-023-08891-5

2023 doi
[60]

, author Wang, Y

author Lin, J. , author Wang, Y. , author Yu, Y. , author Zhou, Y. , author Chen, Y. , & author Shi, X. ( year 2022 ). title Automatic analysis of available source code of top artificial intelligence conference papers . journal International Journal of Software Engineering and...

2022 doi
[61]

, & author Shah, N

author Liu, R. , & author Shah, N. B. ( year 2024 ). title ReviewerGPT? An exploratory study on using large language models for paper reviewing . In booktitle AAAI SDU Workshop \/

2024
[62]

, & author Hsiao, S

author Manyika, J. , & author Hsiao, S. ( year 2023 ). title An overview of Bard: an early experiment with generative AI . journal AI. Google Static Documents \/ , volume 2 \/

2023
[63]

, author Hellen, N

author Marvin, G. , author Hellen, N. , author Jjingo, D. , & author Nakatumba-Nabende, J. ( year 2024 ). title Prompt engineering in large language models . :10.1007/978-981-99-7962-2\_30

2024 doi
[64]

title Guidelines for Reviewers

author MDPI ( year 2024 ). title Guidelines for Reviewers . https://www.mdpi.com/reviewers

2024
[65]

, author Mathur, A

author Mehta, V. , author Mathur, A. , author Anjali, A. K. , & author Fiorillo, L. ( year 2024 ). title The application of ChatGPT in the peer-reviewing process . journal Oral Oncology Reports \/ , volume 9 \/ , pages 100227 . :10.1016/j.oor.2024.100227

2024
[66]

author Nath, K. A. , author Conway, M. , & author Fonseca, R. ( year 2024 ). title AI in peer review: Publishing’s panacea or a pandora’s box of problems? journal Mayo Clinic Proceedings \/ , volume 99 \/ , pages 10--12 . :10.1016/j.mayocp.2023.11.013

2024 doi
[67]

( year 2024 )

author Nature, S. ( year 2024 ). title Editorial policies - Artificial Intelligence . https://www.springernature.com/gp/policies/editorial-policies

2024
[68]

, author Blackbourn, H

author Ong, C. , author Blackbourn, H. , & author Migliori, G. ( year 2023 ). title GPT-4, artificial intelligence and implications for publishing . journal The International Journal of Tuberculosis and Lung Disease \/ , volume 27 \/ , pages 425--426 . :10.5588/ijtld.23.0143

2023 doi
[69]

title Introducing ChatGPT

author OpenAI ( year 2022 ). title Introducing ChatGPT . https://openai.com/blog/chatgpt

2022
[70]

title GPT-4 technical report

author OpenAI ( year 2023 ). title GPT-4 technical report . journal arXiv preprint arXiv:2303.08774 \/ ,

2023 arXiv
[71]

, author Alcaide, E

author Peng, B. , author Alcaide, E. , author Anthony, Q. , author Albalak, A. , author Arcadinho, S. , author Biderman, S. , author Cao, H. , author Cheng, X. , author Chung, M. , & author Grella, M. ( year 2023 ). title Rwkv: Reinventing rnns for the transformer era . journa...

2023 arXiv
[72]

author Press, C. U. ( year 2024 a ). title Core editorial policies for journals . https://www.cambridge.org/core/services/publishing-ethics/core-editorial-policies-journals

2024
[73]

( year 2024 b )

author Press, M. ( year 2024 b ). title Policy on the use of AI tools by MIT Press authors . https://mitpress.mit.edu/for-authors/

2024
[74]

, author Lauscher, A

author Purkayastha, S. , author Lauscher, A. , & author Gurevych, I. ( year 2023 ). title Exploring Jiu-Jitsu argumentation for writing peer review rebuttals . :10.18653/v1/2023.emnlp-main.894

2023 doi
[75]

( year 2023 )

author Robertson, Z. ( year 2023 ). title GPT4 is slightly helpful for peer-review assistance: A pilot study . journal arXiv preprint arXiv:2307.05492 \/ ,

2023 arXiv
[76]

, author Mesgar, M

author Ruggeri, F. , author Mesgar, M. , & author Gurevych, I. ( year 2023 ). title A dataset of argumentative dialogues on scientific papers . :10.18653/v1/2023.acl-long.425

2023 doi
[77]

, author Jenko, N

author Saad, A. , author Jenko, N. , author Ariyaratne, S. , author Birch, N. , author Iyengar, K. P. , author Davies, A. M. , author Vaishya, R. , & author Botchu, R. ( year 2024 ). title Exploring the potential of ChatGPT in the peer review process: An observational study . ...

2024
[78]

title Using AI in peer review and publishing

author Sage ( year 2024 ). title Using AI in peer review and publishing . https://us.sagepub.com/en-us/nam/using-ai-in-peer-review-and-publishing

2024
[79]

, author Singh, A

author Sahoo, P. , author Singh, A. K. , author Saha, S. , author Jain, V. , author Mondal, S. , & author Chadha, A. ( year 2024 ). title A systematic survey of prompt engineering in large language models: Techniques and applications . journal arXiv preprint arXiv:2402.07927 \/ ,

2024 arXiv
[80]

, author Hubig, N

author Scherbakov, D. , author Hubig, N. , author Jansari, V. , author Bakumenko, A. , & author Lenert, L. A. ( year 2024 ). title The emergence of Large Language Models (LLM) as a tool in literature reviews: an LLM automated systematic review . journal arXiv preprint arXiv:24...

2024 arXiv
[81]

, author Strinzel, M

author Severin, A. , author Strinzel, M. , author Egger, M. , author Barros, T. , author Sokolov, A. , author Mouatt, J. V. , & author Müller, S. ( year 2023 ). title Relationship between journal impact factor and the thoroughness and helpfulness of peer reviews . journal PLOS...

2023 doi
[82]

, author Habibnia, H

author Shcherbiak, A. , author Habibnia, H. , author Böhm, R. , & author Fiedler, S. ( year 2024 ). title Evaluating science: A comparison of human and AI reviewers . journal Judgment and Decision Making \/ , volume 19 \/ , pages e21 . :10.1017/jdm.2024.24

2024 doi
[83]

, author Biller-Andorno, N

author Spitale, G. , author Biller-Andorno, N. , & author Germani, F. ( year 2023 ). title AI model GPT-3 (dis)informs us better than humans . journal Science Advances \/ , volume 9 \/ , pages eadh1850 . :10.1126/sciadv.adh1850

2023 doi
[84]

, author Wang, G

author Taechoyotin, P. , author Wang, G. , author Zeng, T. , author Sides, B. , & author Acuna, D. ( year 2024 ). title MAMORX: Multi-agent multi-modal scientific review generation with external knowledge . In booktitle Neurips 2024 Workshop Foundation Models for Science: Prog...

2024
[85]

( year 2023 )

author Takagi, S. ( year 2023 ). title Speculative exploration on the concept of artificial agents conducting autonomous research . journal arXiv preprint arXiv:2312.03497 \/ ,

2023 arXiv
[86]

, author Lyu, D

author Tan, C. , author Lyu, D. , author Li, S. , author Gao, Z. , author Wei, J. , author Ma, S. , author Liu, Z. , & author Li, S. Z. ( year 2024 ). title Peer review as a multi-turn and long-context dialogue with role-based interactions . journal arXiv preprint arXiv:2406.0...

2024 arXiv
[87]

, author Gulrajani, I

author Taori, R. , author Gulrajani, I. , author Zhang, T. , author Dubois, Y. , author Li, X. , author Guestrin, C. , author Liang, P. , & author Hashimoto, T. B. ( year 2023 ). title Alpaca: A strong, replicable instruction-following model . journal Stanford Center for Resea...

2023
[88]

, author Georgiev, P

author Team, G. , author Georgiev, P. , author Lei, V. I. , author Burnell, R. , author Bai, L. , author Gulati, A. , author Tanzer, G. , author Vincent, D. , author Pan, Z. , author Wang, S. , author Mariooryad, S. , author Ding, Y. , author Geng, X. , author Alcober, F. , au...

2024 doi arXiv
[89]

( year 2024 )

author Thelwall, M. ( year 2024 ). title Can ChatGPT evaluate research quality? journal Journal of Data and Information Science \/ , volume 9 \/ , pages 1--21 . :10.2478/jdis-2024-0013

2024 doi
[90]

, & author Cui, A

author Tian, E. , & author Cui, A. ( year 2023 ). title GPTZero: Towards detection of AI-generated text using zero-shot and supervised methods . journal GPTZero \/ ,

2023
[91]

, author Lavril, T

author Touvron, H. , author Lavril, T. , author Izacard, G. , author Martinet, X. , author Lachaux, M.-A. , author Lacroix, T. , author Rozière, B. , author Goyal, N. , author Hambro, E. , & author Azhar, F. ( year 2023 a ). title Llama: Open and efficient foundation language ...

2023 arXiv
[92]

, author Martin, L

author Touvron, H. , author Martin, L. , author Stone, K. , author Albert, P. , author Almahairi, A. , author Babaei, Y. , author Bashlykov, N. , author Batra, S. , author Bhargava, P. , & author Bhosale, S. ( year 2023 b ). title Llama 2: Open foundation and fine-tuned chat m...

2023 arXiv
[93]

, author Segev, B

author Tyser, K. , author Segev, B. , author Longhitano, G. , author Zhang, X.-Y. , author Meeks, Z. , author Lee, J. , author Garg, U. , author Belsten, N. , author Shporer, A. , & author Udell, M. ( year 2024 ). title AI-driven review systems: Evaluating LLMs in scalable and...

2024 arXiv
[94]

, author Chen, S

author Wang, L. , author Chen, S. , author Jiang, L. , author Pan, S. , author Cai, R. , author Yang, S. , & author Yang, F. ( year 2024 ). title Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies . journal arXiv preprint arXiv:2410.19878 \/ ,

2024 arXiv
[95]

( year 2024 )

author Weber, R. ( year 2024 ). title The Other Reviewer: RoboReviewer . journal Journal of the Association for Information Systems \/ , volume 25 \/ , pages 85--97 . :10.17705/1jais.00866

2024 doi
[96]

title The role of Artificial Intelligence in Research Exchange

author Wiley ( year 2024 ). title The role of Artificial Intelligence in Research Exchange . https://www.wiley.com/en-us/network/publishing/research-publishing/peer-review/the-role-of-artificial-intelligence-in-research-exchange

2024
[97]

, author Xiao, B

author Yang, A. , author Xiao, B. , author Wang, B. , author Zhang, B. , author Bian, C. , author Yin, C. , author Lv, C. , author Pan, D. , author Wang, D. , author Yan, D. , author Yang, F. , author Deng, F. , author Wang, F. , author Liu, F. , author Ai, G. , author Dong, G...

2023 arXiv
[98]

, author Yang, B

author Yang, A. , author Yang, B. , author Hui, B. , author Zheng, B. , author Yu, B. , author Zhou, C. , author Li, C. , author Li, C. , author Liu, D. , author Huang, F. , author Dong, G. , author Wei, H. , author Lin, H. , author Tang, J. , author Wang, J. , author Yang, J....

2024 arXiv
[99]

, author Fu, C

author Yin, S. , author Fu, C. , author Zhao, S. , author Li, K. , author Sun, X. , author Xu, T. , & author Chen, E. ( year 2023 ). title A survey on multimodal large language models . journal arXiv preprint arXiv:2306.13549 \/ ,

2023 arXiv
[100]

, author Ding, Z

author Yu, J. , author Ding, Z. , author Tan, J. , author Luo, K. , author Weng, Z. , author Gong, C. , author Zeng, L. , author Cui, R. , author Han, C. , author Sun, Q. , author Wu, Z. , author Lan, Y. , & author Li, X. ( year 2024 ). title Automated peer reviewing in paper ...

2024
[101]

, author Liu, P

author Yuan, W. , author Liu, P. , & author Neubig, G. ( year 2022 ). title Can we automate scientific reviewing? journal Journal of Artificial Intelligence Research \/ , volume 75 \/ , pages 171--212 . :10.1613/jair.1.12862

2022 doi
[102]

, author Liu, X

author Zeng, A. , author Liu, X. , author Du, Z. , author Wang, Z. , author Lai, H. , author Ding, M. , author Yang, Z. , author Xu, Y. , author Zheng, W. , & author Xia, X. ( year 2022 ). title Glm-130b: An open bilingual pre-trained model . journal arXiv preprint arXiv:2210....

2022 doi arXiv
[103]

author Zhao, W. X. , author Zhou, K. , author Li, J. , author Tang, T. , author Wang, X. , author Hou, Y. , author Min, Y. , author Zhang, B. , author Zhang, J. , & author Dong, Z. ( year 2023 ). title A survey of large language models . journal arXiv preprint arXiv:2303.18223 \/ ,

2023 arXiv
[104]

, author Sheng, Y

author Zheng, L. , author Sheng, Y. , author Chiang, W. , author Zhang, H. , author Gonzalez, J. , & author Stoica, I. ( year 2023 ). title Fastchat . journal URl: https://github. com/lm-sys/FastChat \/ ,

2023
[105]

, author Huang, X

author Zhou, H. , author Huang, X. , author Pu, H. , & author Qi, Z. ( year 2024 a ). title May generative AI be a reviewer on an academic paper? In booktitle EEKE-AII \/

2024
[106]

, author Chen, L

author Zhou, R. , author Chen, L. , & author Yu, K. ( year 2024 b ). title Is LLM a reliable reviewer? A comprehensive evaluation of LLM on automatic paper reviewing tasks . In booktitle LREC-COLING \/

2024
[107]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

title Peer Review at Science Journals

author AAAS ( year 2024 ). title Peer Review at Science Journals . https://www.science.org/content/page/peer-review-science-publications

2024

[2] [2]

title FAQ: Can I use AI writing assistants to write my review? https://2023.aclweb.org/blog/review-acl23/#faq-can-i-use-ai-writing-assistants-to-write-my-review

author ACL ( year 2023 ). title FAQ: Can I use AI writing assistants to write my review? https://2023.aclweb.org/blog/review-acl23/#faq-can-i-use-ai-writing-assistants-to-write-my-review

2023

[3] [3]

title ACM Peer Review Policy

author ACM ( year 2023 ). title ACM Peer Review Policy . https://www.acm.org/publications/policies/peer-review

2023

[4] [4]

title ACM Peer Review Policy Frequently Asked Questions ,

author ACM ( year 2024 ). title ACM Peer Review Policy Frequently Asked Questions ,

2024

[5] [5]

( year 2023 )

author AI, S. ( year 2023 ). title Stability ai launches the first of its stablelm suite of language models . journal Blog post, Apr \/ ,

2023

[6] [6]

title Ethics for Editors and Reviewers

author AIP ( year 2024 ). title Ethics for Editors and Reviewers . https://publishing.aip.org/resources/researchers/policies-and-ethics/editors-and-reviewers/

2024

[7] [7]

, author Elsayed, M

author Alshami, A. , author Elsayed, M. , author Ali, E. , author Eltoukhy, A. E. , & author Zayed, T. ( year 2023 ). title Harnessing the power of ChatGPT for automating systematic review process: Methodology, case study, limitations, and future directions . journal Systems \/ , volume 11 \/ , pages 351

2023

[8] [8]

( year 2024 )

author Anthropic, A. ( year 2024 ). title The claude 3 model family: Opus, sonnet, haiku . journal Claude-3 Model Card \/ , volume 1 \/

2024