Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Automatic Evaluation Metrics for Artificially Generated Scientific Research

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A simple title-and-abstract score predictor agrees with human reviewers more often than LLM-based reviewers do, the paper claims.

desk verdict A well-built empirical study of score prediction on human papers whose headline promise about AI-generated research outruns the evidence. read the letter →

arxiv 2503.05712 v1 pith:LMVHOUR4 submitted 2025-02-14 cs.CY cs.AIcs.LG

Niklas Höpner , Leon Eshuijs , Dimitrios Alivanistos , Giacomo Zamprogno , Ilaria Tiddi This is my paper
classification cs.CYcs.AIcs.LG
keywords automaticevaluationmetricscitationcountpredictionreviewscoreAI-generatedscientificresearchhypothesisannotationLLMreviewercomparisonquality
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Automatic evaluation of AI-generated science is stuck between costly expert reviews and unreliable LLM reviewers. This paper tries to make the case that predicting a paper's citation count or its human review score, directly from its text, can serve as cheap automatic quality metrics. To do this, it assembles a large unified dataset of scientific submissions with citation counts and machine-annotated research hypotheses. Its experiments show that citation count prediction works better than review score prediction, that a simple model using only title and abstract outperforms LLM-based reviewers in matching human scores, and that human reviewers remain the most consistent benchmark.

What carries the argument

The load-bearing object is a learned score function $f_\theta$ that maps fixed text embeddings to a scalar quality score, trained with a pairwise ranking objective and an $\ell^1$ regression objective. Different parts of a paper are embedded separately, such as title and abstract, research hypothesis, or individual sections, and when context is used the paper embedding is combined with embeddings of its references through a one-layer Transformer. The data pipeline matters as much as the model: PDFs are parsed and classified into sections, each submission is annotated with a research hypothesis by a prompted language model, and citation counts are gathered from a scholarly database. The hypothesis is that these quality scores are predictable enough from content alone to serve as evaluation metrics.

What would settle it

Run the same title-and-abstract score predictor on a set of machine-generated papers whose later citation counts or human review scores are known; if its pairwise accuracy drops near chance while human-review consistency stays stable, the transfer claim is falsified.

Watch

Extended reading notes

Core claim

The central discovery, on the paper's own terms, is that a simple score-prediction model trained on text embeddings can rank papers by later citation impact above random and can agree with human review scores more reliably than LLM judges, while still falling short of a second human reviewer. Review-score prediction is harder than citation prediction, and predicting scores from a research hypothesis alone is harder than predicting them from the full paper. The paper also reports that citation counts and review scores correlate only weakly, meaning the two metrics capture different aspects of what makes research influential or well received.

Load-bearing premise

The evaluation metrics are meant for AI-generated papers, but every experiment measures human-written submissions, so the paper assumes that ranking accuracy transfers across that gap.

Editorial extensions

If this is right

  • Citation-count prediction from title and abstract is viable: pairwise comparison accuracy reaches about 66.5% on a computational-linguistics corpus and about 65% on ICLR submissions, so content alone carries a usable ranking signal.
  • Research-hypothesis-only prediction beats random but is markedly worse than title-and-abstract prediction, so evaluating idea-generation systems on hypotheses alone will lose signal.
  • Review score prediction across venues is near random; only venue-specific models reach about 60% pairwise accuracy, implying review scores need standardization before they can serve as a general metric.
  • A simple title-and-abstract model (correlation about 0.33 with human review scores) outperforms an LLM-based reviewer (about 0.16) but not a second human reviewer (about 0.41), so content-based scoring is the cheaper of the available automated options.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The transfer from human to AI-generated papers is the paper's load-bearing bet; since all experiments use human submissions, the reported accuracies are an upper-bound estimate until the predictor is tested on generated text.
  • The near-random cross-venue review-score predictions suggest that review-score normalization rather than model capacity is the binding constraint; normalizing per venue before training could be a direct extension.
  • A combined target that mixes citation and review signals might be more stable than either alone, since the two correlate only weakly.
  • The research-hypothesis results imply that judging idea-generation systems purely on their hypotheses forfeits most of the signal available in the full paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes automatic evaluation metrics for AI-generated scientific research, specifically citation count prediction and review score prediction, trained on content-only representations (titles, abstracts, hypotheses) of papers. The authors build a large unified OpenReview dataset enriched with citations, references, and GPT-annotated research hypotheses, and report three main findings: citation count prediction is more viable than review score prediction; predicting scores from the research hypothesis is more difficult than from the full paper; and a simple title-and-abstract prediction model outperforms LLM-based reviewers, though it falls short of human-level consistency. The paper also analyzes correlations between citations and review scores and provides qualitative Shapley-value analyses.

Significance. If the central claim were established, the paper would offer a cheap, content-based evaluation method for AI-generated research and a substantial new resource (the unified OpenReview dataset with hypothesis annotations). The release of code and dataset, the use of multiple random seeds, and the documented training details are concrete strengths. However, the headline application to AI-generated scientific content is never tested: every experiment trains and evaluates on human-written OpenReview papers. The comparison with LLM reviewers also lacks variance or significance testing. These gaps are load-bearing for the stated purpose, so the contribution is currently more a reproducible study of citation/review-score predictability on human papers than a validated evaluation metric for AI-generated research.

major comments (4)
  1. [§1, §5.3, Abstract] The central claim that citation count prediction and review score prediction serve as automatic evaluation metrics for AI-generated scientific research is untested. All experiments (Section 5.2, Table 2, Table 3, Figure 3; Section 5.3, Table 5) train, validate, and test exclusively on human-written OpenReview submissions (ACL-OCL, ICLR, NeurIPS). The paper itself cites Si et al. (2024) and Lu et al. (2024) for the observation that LLM-generated scientific text differs lexically and structurally, but no experiment or analysis addresses the distribution shift from human to AI-generated papers or hypotheses. The Limitations section concedes that citation and review scores are imperfect proxies but does not mention this transfer problem. For the headline conclusion to hold, the authors need to test the trained predictors on AI-generated papers or hypotheses (for example, papers produced by the AI Scientist) or else substantially scale back the claimed scope.
  2. [§5.3, Table 5] The claim that the simple prediction model "outperforms LLM-based reviewers" is not supported with statistical evidence. The LLM reviewer results (accuracy 0.548, 0.552; Spearman 0.145, 0.064) are point estimates from a single run, while the RSP results are reported as mean±std over five seeds. On a subsample of 200 papers, the observed differences may be within noise. Please provide multiple runs of the LLM reviewer (varying prompts or sampling), confidence intervals, and a paired significance test (e.g., bootstrap or McNemar) before drawing this conclusion.
  3. [§3, Eq. (1)] The pairwise comparison loss is written as L(d1,d2;θ)=x log y − (1−x) log(1−y). This is not the standard cross-entropy loss: it lacks the overall minus sign, so minimizing it does not correspond to maximizing the Bernoulli likelihood and the objective can be driven to negative infinity. The correct loss should be −[x log y + (1−x) log(1−y)] (or equivalently x log(1/y)+(1−x) log(1/(1−y))). Please correct the equation and ensure the implementation matches the corrected form.
  4. [Abstract, §5.2, Table 2, Figure 3] The finding "predicting scores is more difficult purely from the research hypothesis than from the full paper" is not supported by the reported experiments. Table 2 compares Hypothesis against Title+Abstract, Introduction, Related Work, Methodology, Experiments & Results, and Conclusion, but never against the full paper as a single input representation. Figure 3 shows a "Full Paper" context, but only with Title+Abstract as the paper representation, and Hypothesis is not included in that comparison. The claim should be reworded to "than from the title and abstract" or the missing full-paper condition with hypothesis input should be added.
minor comments (5)
  1. [Appendix D] The section classifier accuracy on OpenReview is reported as "0.93 ± 0.1"; given the mean of 0.93, a standard deviation of 0.1 seems too large and is likely a typo for 0.01. Please check and correct.
  2. [Appendix F] The heading "Additional resutls" contains a typo; it should be "Additional results".
  3. [Figure 7] The schematic of the review object lists "Repoducibility", which should be "Reproducibility".
  4. [Table 3] For the "All" dataset, the review score prediction results are statistically indistinguishable from random (accuracy ≈ 0.50, ρs ≈ 0.00–0.01), yet the table bolds a "best" model based on Spearman correlation. Consider adding a note or significance indicator to prevent overinterpretation of near-zero correlations.
  5. [General] The model name "SPECTER2" is spelled inconsistently as "SPECTER2", "Specter2", and "Specter 2" in different places; please unify the notation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the predictions are supervised against external human labels and the reported comparisons are independent of the paper's own construction choices.

full rationale

The paper trains citation-count and review-score predictors on human-written OpenReview and ACL-OCL papers with labels obtained from external sources: citation counts from Semantic Scholar and review scores from OpenReview reviews. The target labels are therefore not defined in terms of the model inputs (title/abstract, sections, hypothesis, references), and no fitted parameter is renamed as a prediction. The GPT-3.5-generated research hypotheses serve only as alternative input representations for the predictors; they are not used as ground-truth quality scores, so the central comparisons (citation vs. review prediction, title/abstract vs. hypothesis, RSP vs. LLM reviewers, RSP vs. human consistency) are evaluated against held-out external human labels. No load-bearing self-citation was found: the SPECTER2 embeddings and the LLM-reviewer baselines come from external prior work, and the paper's own dataset releases are resources rather than arguments that determine the conclusions. The main weakness, namely that all experiments use human-written papers while the stated application is AI-generated research, is a distribution-shift validity concern and not a circularity, because the reported numbers measure prediction of external labels and would remain meaningful if the transfer assumption held. No circular step can be exhibited by quoting a reduction of a prediction to its inputs, so the correct finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on standard supervised learning assumptions plus two domain leaps: that the proxies measure quality and that human-trained predictors work on AI-generated papers. The second leap is the most fragile because it is never tested. The free parameters are model tuning choices and dataset construction decisions, not scientific constants.

free parameters (4)
  • Model hyperparameters (learning rate, dropout, hidden size, epochs) = lr=5e-5, dropout=0.3, hidden=256 (no-context); context model hidden=1024; grid-searched on validation
    These are fitted on validation data and affect the reported accuracies, but they are standard tuning choices rather than scientific parameters.
  • LDA topic count = 13
    Chosen to match the 13 ICLR-2023 fields of study; topic clustering is used to test whether topic-specific models improve review score prediction (Table 4).
  • LLM comparison subsample size = 200
    Subsample of 200 papers from each 2024 test set for the LLM versus RSP comparison; cost-driven and not justified as statistically powered.
  • Review score normalization mapping = manual venue field mapping to [0,1]
    Authors manually mapped each venue's review fields to the unified review schema (Table 9); this choice affects all review score results.
assumptions (4)
  • domain assumption Review scores and citation counts are valid proxies for scientific quality.
    Invoked in Sections 1 and 3 as the basis for the two prediction tasks; the paper itself notes in the Limitations that both are imperfect proxies.
  • domain assumption Predictors trained on human-written OpenReview papers transfer to AI-generated scientific content.
    The abstract and conclusion motivate the metrics for AI-generated research, but no AI-generated paper or hypothesis is evaluated, so transfer is assumed rather than demonstrated.
  • domain assumption GPT-3.5 research hypothesis annotations accurately capture each paper's hypothesis.
    Appendix E uses a one-shot prompt to gpt-3.5-turbo and validates on only 13 first authors rating 32 hypotheses; completeness scores are moderate.
  • domain assumption GROBID parsing and the trained section classifier correctly extract and label paper sections.
    Appendix C and D rely on these tools to build section-level inputs; the text acknowledges that GROBID parsing errors are a potential source of error.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automatic Evaluation Metrics for Artificially Generated Scientific Research." pith.science (2026). https://pith.science/paper/LMVHOUR4

@misc{pith2026250305712,
  author       = {Pith},
  title        = {Pith review of: Automatic Evaluation Metrics for Artificially Generated Scientific Research},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LMVHOUR4}},
  note         = {Machine review of arXiv:2503.05712}
}
read the original abstract

Foundation models are increasingly used in scientific research, but evaluating AI-generated scientific work remains challenging. While expert reviews are costly, large language models (LLMs) as proxy reviewers have proven to be unreliable. To address this, we investigate two automatic evaluation metrics, specifically citation count prediction and review score prediction. We parse all papers of OpenReview and augment each submission with its citation count, reference, and research hypothesis. Our findings reveal that citation count prediction is more viable than review score prediction, and predicting scores is more difficult purely from the research hypothesis than from the full paper. Furthermore, we show that a simple prediction model based solely on title and abstract outperforms LLM-based reviewers, though it still falls short of human-level consistency.

Figures

Figures reproduced from arXiv: 2503.05712 by the authors.

Figure 1
Figure 1. Architecture of the context model in case of [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Pearson correlation heat map for the different [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Spearman correlations for context-based mod [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Scatter plots of predicted review scores and groundtruth review scores on a subset of the test set of [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 6
Figure 6. Figure 6: Schematic overview of the Reference object. [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Schematic overview of the review object. The [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Number of submissions per conference Parameter Value Learning Rate 0.0001 Batch Size 128 # Epochs 20 Hardware NVIDIA GeForce RTX 3080 (10GB) Training Time 25min [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Example of the Google Form used to collect the survey data. Parts of the form is blacked out to guarantee [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Frequency of the 20 most frequent field of [PITH_FULL_IMAGE:figures/full_fig_p030_10.png]
Figure 11
Figure 11. Figure 11: Illustrative Shapley values for titles and abstracts in the review score prediction model trained on the [PITH_FULL_IMAGE:figures/full_fig_p032_11.png]
Figure 12
Figure 12. Figure 12: Illustrative Shapley values for research hypotheses in the review score prediction model trained on the [PITH_FULL_IMAGE:figures/full_fig_p033_12.png]
Figure 13
Figure 13. Figure 13: Illustrative Shapley values for titles and abstracts in the citation score prediction model trained on the [PITH_FULL_IMAGE:figures/full_fig_p034_13.png]
Figure 14
Figure 14. Figure 14: Illustrative Shapley values for research hypotheses in the citation score prediction model trained on the [PITH_FULL_IMAGE:figures/full_fig_p035_14.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. How Far Are AI Scientists from Changing the World?

    cs.AI 2025-07 conditional novelty 4.0 of 10

    This survey proposes a four-level capability framework for AI Scientist systems and, using an AI reviewer, finds that current systems produce papers rated well below normal scientific standards.

  2. AI4Research: A Survey of Artificial Intelligence for Scientific Research

    cs.CL 2025-07 conditional novelty 4.0 of 10

    A survey that organizes AI-for-research work into five tasks, comprehension, survey, discovery, writing, and peer review, and compiles associated tools and benchmarks.

Reference graph

Works this paper leans on

53 extracted references · 29 canonical work pages · cited by 2 Pith papers

  1. [1]

    Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. 2024. https://doi.org/10.48550/ARXIV.2404.07738 Researchagent: Iterative research idea generation over scientific literature with large language models . CoRR, abs/2404.07738

    arXiv doi:10.48550/arxiv.2404.07738 2024

  2. [2]

    Xiaomei Bai, Fuli Zhang, and Ivan Lee. 2019. https://doi.org/10.1016/J.JOI.2019.01.010 Predicting the citations of scholarly paper . J. Informetrics, 13(1):407--418

    work page doi:10.1016/j.joi.2019.01.010 2019

  3. [3]

    Prabhat Kumar Bharti, Shashi Ranjan, Tirthankar Ghosal, Mayank Agrawal, and Asif Ekbal. 2021. https://doi.org/10.1007/978-3-030-91669-5\_33 Peerassist: Leveraging on paper-review interactions to predict peer review decisions . In Towards Open and Trustworthy Digital Societies - 23rd International Conference on Asia-Pacific Digital Libraries, ICADL 2021, V...

    work page doi:10.1007/978-3-030-91669-5 2021

  4. [4]

    Steven Bird. 2006. https://doi.org/10.3115/1225403.1225421 NLTK: the natural language toolkit . In ACL 2006, 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, Sydney, Australia, 17-21 July 2006 . The Association for Computer Linguistics

    work page arXiv 2006

  5. [5]

    Blei, Andrew Y

    David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. J. Mach. Learn. Res., 3(null):993–1022

    work page 2003

  6. [6]

    Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes

    Daniil A. Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. 2023. https://doi.org/10.1038/S41586-023-06792-0 Autonomous chemical research with large language models . Nat., 624(7992):570--578

    doi:10.1038/s41586-023-06792-0 2023

  7. [7]

    Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D

    Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D. White, and Philippe Schwaller. 2024. https://doi.org/10.1038/S42256-024-00832-8 Augmenting large language models with chemistry tools . Nat. Mac. Intell., 6(5):525--535

    doi:10.1038/s42256-024-00832-8 2024

  8. [8]

    Mike D'Arcy, Tom Hope, Larry Birnbaum, and Doug Downey. 2024. https://doi.org/10.48550/ARXIV.2401.04259 MARG: multi-agent review generation for scientific papers . CoRR, abs/2401.04259

    arXiv doi:10.48550/arxiv.2401.04259 2024

Show all 53 references
  1. [9]

    Gideon Maillette de Buy Wenniger, Thomas van Dongen, and Lambert Schomaker. 2023. https://doi.org/10.48550/ARXIV.2308.07971 Multischubert: Effective multimodal fusion for scholarly document quality prediction . CoRR, abs/2308.07971

    2023 doi arXiv

  2. [10]

    Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint...

    2023

  3. [11]

    Nils Dycke, Ilia Kuznetsov, and Iryna Gurevych. 2023. https://doi.org/10.18653/V1/2023.ACL-LONG.277 Nlpeer: A unified resource for the computational study of peer review . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon...

    2023 doi

  4. [12]

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M \" u ller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. 2024. https://openreview.net/forum?id=FPnUhsQJ5B Scaling rectifi...

    2024

  5. [13]

    Gustavo L \' u cius Fernandes and Pedro O. S. Vaz de Melo. 2024. https://doi.org/10.1007/S00799-023-00382-1 Enhancing the examination of obstacles in an automated peer review system . Int. J. Digit. Libr., 25(2):341--364

    2024 doi

  6. [14]

    Fu and Constantin F

    Lawrence D. Fu and Constantin F. Aliferis. 2008. https://knowledge.amia.org/amia-55142-a2008a-1.625176/t-001-1.626020/f-001-1.626021/a-045-1.626429/a-046-1.626426 Models for predicting and explaining citation count of biomedical articles . In AMIA 2008, American Medical Inform...

    2008

  7. [15]

    Martin Funkquist, Ilia Kuznetsov, Yufang Hou, and Iryna Gurevych. 2023. https://doi.org/10.18653/V1/2023.EMNLP-MAIN.455 Citebench: A benchmark for scientific citation text generation . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, E...

    2023 doi

  8. [16]

    Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. https://doi.org/10.18653/V1/2023.EMNLP-MAIN.398 Enabling large language models to generate text with citations . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Sin...

    2023 doi

  9. [17]

    Jun Hirako, Ryohei Sasano, and Koichi Takeda. 2023. https://doi.org/10.18653/V1/2023.FINDINGS-EACL.84 Realistic citation count prediction task for newly published papers . In Findings of the Association for Computational Linguistics: EACL 2023, Dubrovnik, Croatia, May 2-6, 202...

    2023 doi

  10. [18]

    Jun Hirako, Ryohei Sasano, and Koichi Takeda. 2024. https://doi.org/10.48550/ARXIV.2410.04404 Cimate: Citation count prediction effectively leveraging the main text . CoRR, abs/2410.04404

    2024 doi arXiv

  11. [19]

    Alfonso Ib \' a \ n ez, Pedro Larra \ n aga, and Concha Bielza. 2009. https://doi.org/10.1093/BIOINFORMATICS/BTP585 Predicting citation count of Bioinformatics papers within four years of publication . Bioinform., 25(24):3303--3309

    2009 doi

  12. [20]

    Shah, and Fei Fang

    Steven Jecmen, Minji Yoon, Vincent Conitzer, Nihar B. Shah, and Fei Fang. 2023. https://doi.org/10.1145/3543507.3583424 A dataset on malicious paper bidding in peer review . In Proceedings of the ACM Web Conference 2023, WWW 2023, Austin, TX, USA, 30 April 2023 - 4 May 2023 , ...

    2023

  13. [21]

    Silva, Osvaldo N

    Adilson Vital Jr., Filipi N. Silva, Osvaldo N. Oliveira Jr., and Diego R. Amancio. 2024. https://doi.org/10.48550/ARXIV.2407.19942 Predicting citation impact of research papers using GPT and other text embeddings . CoRR, abs/2407.19942

    2024 doi arXiv

  14. [22]

    Hovy, and Roy Schwartz

    Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard H. Hovy, and Roy Schwartz. 2018. https://doi.org/10.18653/V1/N18-1149 A dataset of peer reviews (peerread): Collection, insights and NLP applications . In Proceedings of the 2018 Conf...

    2018 doi

  15. [23]

    Mina Lee, Percy Liang, and Qian Yang. 2022. https://doi.org/10.1145/3491102.3502030 Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities . In CHI '22: CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA, ...

    2022

  16. [24]

    Ruochen Li, Teerth Patel, Qingyun Wang, and Xinya Du. 2024. https://doi.org/10.48550/ARXIV.2408.14033 Mlr-copilot: Autonomous machine learning research based on large language models agents . CoRR, abs/2408.14033

    2024 doi

  17. [25]

    Siqing Li, Wayne Xin Zhao, Eddy Jing Yin, and Ji - Rong Wen. 2019. https://doi.org/10.18653/V1/D19-1497 A neural citation count prediction model based on peer review text . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th In...

    2019 doi

  18. [26]

    Ryan Liu and Nihar B. Shah. 2023. https://doi.org/10.48550/ARXIV.2306.00622 Reviewergpt? an exploratory study on using large language models for paper reviewing . CoRR, abs/2306.00622

    2023 doi arXiv

  19. [27]

    Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024. https://doi.org/10.48550/ARXIV.2408.06292 The AI scientist: Towards fully automated open-ended scientific discovery . CoRR, abs/2408.06292

    2024 doi arXiv

  20. [28]

    Lundberg and Su - In Lee

    Scott M. Lundberg and Su - In Lee. 2017. https://proceedings.neurips.cc/paper/2017/hash/8a20a8621978632d76c43dfd28b67767-Abstract.html A unified approach to interpreting model predictions . In Advances in Neural Information Processing Systems 30: Annual Conference on Neural In...

    2017

  21. [29]

    Mendoza, Wojciech Kusa, Alaa El - Ebshihy, Ronin Wu, David Pride, Petr Knoth, Drahomira Herrmannova, Florina Piroi, Gabriella Pasi, and Allan Hanbury

    \' O scar E. Mendoza, Wojciech Kusa, Alaa El - Ebshihy, Ronin Wu, David Pride, Petr Knoth, Drahomira Herrmannova, Florina Piroi, Gabriella Pasi, and Allan Hanbury. 2022. https://aclanthology.org/2022.sdp-1.31 Benchmark for research theme classification of scholarly documents ....

    2022

  22. [30]

    Panitan Muangkammuen, Fumiyo Fukumoto, Jiyi Li, and Yoshimi Suzuki. 2022. https://doi.org/10.18653/V1/2022.FINDINGS-EMNLP.164 Exploiting labeled and unlabeled data via transformer fine-tuning for peer-review score prediction . In Findings of the Association for Computational L...

    2022 doi

  23. [31]

    Panitan Muangkammuen, Fumiyo Fukumoto, Jiyi Li, and Yoshimi Suzuki. 2023. Intermediate-task transfer learning for peer review score prediction. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific C...

    2023

  24. [32]

    Vinod Nair and Geoffrey E. Hinton. 2010. https://icml.cc/Conferences/2010/papers/432.pdf Rectified linear units improve restricted boltzmann machines . In Proceedings of the 27th International Conference on Machine Learning (ICML-10), June 21-24, 2010, Haifa, Israel, pages 807...

    2010

  25. [33]

    OpenAI. 2023. https://doi.org/10.48550/ARXIV.2303.08774 GPT-4 technical report . CoRR, abs/2303.08774

    2023 doi arXiv

  26. [34]

    Guilliermo Owen. 1977. Values of games with a priori unions. In Mathematical economics and game theory: Essays in honor of Oskar Morgenstern, pages 76--88. Springer

    1977

  27. [35]

    Barbara Plank and Reinard van Dalen. 2019. https://ceur-ws.org/Vol-2414/paper12.pdf Citetracked: A longitudinal dataset of peer reviews and citations . In Proceedings of the 4th Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for D...

    2019

  28. [36]

    Biqing Qi, Kaiyan Zhang, Haoxiang Li, Kai Tian, Sihang Zeng, Zhang - Ren Chen, and Bowen Zhou. 2023. https://doi.org/10.48550/ARXIV.2311.05965 Large language models are zero shot hypothesis proposers . CoRR, abs/2311.05965

    2023 doi arXiv

  29. [37]

    Radim R eh u r ek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora . In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks , pages 45--50, Valletta, Malta. ELRA. http://is.muni.cz/publication/884893/en

    2010

  30. [38]

    Shaurya Rohatgi, Yanxia Qin, Benjamin Aw, Niranjana Unnithan, and Min - Yen Kan. 2023. https://doi.org/10.18653/V1/2023.EMNLP-MAIN.640 The ACL OCL corpus: Advancing open science in computational linguistics . In Proceedings of the 2023 Conference on Empirical Methods in Natura...

    2023 doi

  31. [39]

    Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. 2024. https://doi.org/10.48550/ARXIV.2409.04109 Can llms generate novel research ideas? A large-scale human study with 100+ NLP researchers . CoRR, abs/2409.04109

    2024 doi arXiv

  32. [40]

    Amanpreet Singh, Mike D'Arcy, Arman Cohan, Doug Downey, and Sergey Feldman. 2023. https://doi.org/10.18653/V1/2023.EMNLP-MAIN.338 Scirepeval: A multi-format benchmark for scientific document representations . In Proceedings of the 2023 Conference on Empirical Methods in Natura...

    2023 doi

  33. [41]

    Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov

    Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. https://doi.org/10.5555/2627435.2670313 Dropout: a simple way to prevent neural networks from overfitting . J. Mach. Learn. Res., 15(1):1929--1958

    2014

  34. [42]

    Moritz Staudinger, Wojciech Kusa, Florina Piroi, and Allan Hanbury. 2024. An analysis of tasks and datasets in peer reviewing. In Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024), pages 257--268

    2024

  35. [43]

    Thomas van Dongen, Gideon Maillette de Buy Wenniger, and Lambert Schomaker. 2020. https://doi.org/10.18653/V1/2020.SDP-1.17 Schubert: Scholarly document chunks with bert-encoding boost citation count prediction . In Proceedings of the First Workshop on Scholarly Document Proce...

    2020 doi

  36. [44]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html Attention is all you need . In Advances in Neural I...

    2017

  37. [45]

    Gang Wang, Qi Peng, Yanfeng Zhang, and Mingyang Zhang. 2021 a . https://doi.org/10.1007/978-3-030-85896-4\_6 What have we learned from openreview? In Web and Big Data - 5th International Joint Conference, APWeb-WAIM 2021, Guangzhou, China, August 23-25, 2021, Proceedings, Part...

    2021 doi

  38. [46]

    Kehan Wang, Wenxuan Shi, Junsong Bai, Xiaoping Zhao, and Liying Zhang. 2021 b . https://doi.org/10.1007/S11192-021-04026-6 Prediction and application of article potential citations based on nonlinear citation-forecasting combined model . Scientometrics, 126(8):6533--6550

    2021 doi

  39. [47]

    Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. 2024. https://doi.org/10.18653/V1/2024.ACL-LONG.18 Scimon: Scientific inspiration machines optimized for novelty . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

    2024 doi

  40. [48]

    Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, and Erik Cambria. 2024. https://doi.org/10.18653/V1/2024.FINDINGS-ACL.804 Large language models for automated open-domain scientific hypotheses discovery . In Findings of the Association for Computational Linguisti...

    2024 doi

  41. [49]

    Weizhe Yuan, Pengfei Liu, and Graham Neubig. 2022. https://doi.org/10.1613/JAIR.1.12862 Can we automate scientific reviewing? J. Artif. Intell. Res., 75:171--212

    2022 doi

  42. [50]

    Fang Zhang and Shengli Wu. 2024. https://doi.org/10.1007/S11192-024-05086-0 Predicting citation impact of academic papers across research areas using multiple models and early citations . Scientometrics, 129(7):4137--4166

    2024 doi

  43. [51]

    Ruiyang Zhou, Lu Chen, and Kai Yu. 2024. https://aclanthology.org/2024.lrec-main.816 Is LLM a reliable reviewer? A comprehensive evaluation of LLM on automatic paper reviewing tasks . In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Langu...

    2024

  44. [52]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  45. [53]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.