REVIEW 4 cited by
The emergence of Large Language Models (LLM) as a tool in literature reviews: an LLM automated systematic review
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Objective: This study aims to summarize the usage of Large Language Models (LLMs) in the process of creating a scientific review. We look at the range of stages in a review that can be automated and assess the current state-of-the-art research projects in the field. Materials and Methods: The search was conducted in June 2024 in PubMed, Scopus, Dimensions, and Google Scholar databases by human reviewers. Screening and extraction process took place in Covidence with the help of LLM add-on which uses OpenAI gpt-4o model. ChatGPT was used to clean extracted data and generate code for figures in this manuscript, ChatGPT and Scite.ai were used in drafting all components of the manuscript, except the methods and discussion sections. Results: 3,788 articles were retrieved, and 172 studies were deemed eligible for the final review. ChatGPT and GPT-based LLM emerged as the most dominant architecture for review automation (n=126, 73.2%). A significant number of review automation projects were found, but only a limited number of papers (n=26, 15.1%) were actual reviews that used LLM during their creation. Most citations focused on automation of a particular stage of review, such as Searching for publications (n=60, 34.9%), and Data extraction (n=54, 31.4%). When comparing pooled performance of GPT-based and BERT-based models, the former were better in data extraction with mean precision 83.0% (SD=10.4), and recall 86.0% (SD=9.8), while being slightly less accurate in title and abstract screening stage (Maccuracy=77.3%, SD=13.0). Discussion/Conclusion: Our LLM-assisted systematic review revealed a significant number of research projects related to review automation using LLMs. The results looked promising, and we anticipate that LLMs will change in the near future the way the scientific reviews are conducted.
Forward citations
Cited by 4 Pith papers
-
Can Large Language Models Match the Conclusions of Systematic Reviews?
On 284 medical questions derived from Cochrane systematic reviews, the best of 24 LLMs, DeepSeek V3, matches expert conclusions 62.40% of the time, and all tested models struggle with uncertain or low-quality evidence.
-
Leveraging Large Language Models for Generating Research Topic Ontologies: A Multi-Disciplinary Study
Fine-tuned open-weight LLMs classify research-topic relationships with up to 93.5% F1 on a new multi-disciplinary benchmark, and cross-domain transfer loses only about 5 points.
-
AmbiGraph-Eval: Can LLMs Effectively Handle Ambiguous Graph Queries?
A benchmark organized by a six-type taxonomy of ambiguous graph queries reportedly shows that nine LLMs, including top models, frequently produce wrong query translations.
-
Not All Jokes Land: Evaluating Large Language Models Understanding of Workplace Humor
Five LLMs frequently misclassify the appropriateness of workplace humor, especially offensive and neutral jokes, on a new 304-item industrial humor dataset.
Discussion (0). Continue with ORCID to comment.