Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

PreScience: A Dataset and Benchmark for Scientific Forecasting

T0 review · 4 major / 6 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that scientific forecasting can be decomposed into four generative tasks and benchmarked at scale, and shows that current AI systems, when chained into an end-to-end simulation, produce synthetic research that is measurably

desk verdict PreScience is a solid, carefully built forecasting benchmark with real temporal discipline; the diversity/novelty headline is plausible but depends on a metric used outside its validation envelope. read the letter →

arxiv 2602.20459 v2 pith:U2HK5TZ3 submitted 2026-02-24 cs.AI cs.CL

Anirudh Ajith , Amanpreet Singh , Jay DeYoung , Nadav Kunievsky , Austin C. Kozlowski , Oyvind Tafjord , James Evans , Daniel S. Weld
show 2 more authors
Tom Hope Doug Downey
This is my paper · ORCID
classification cs.AIcs.CL
keywords scientificforecastingbenchmarkdatasetcontributiongenerationLACERScoresimulationdiversityandnoveltycollaboratorpredictioncitationimpact
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PreScience tries to establish whether AI systems trained only on the scientific record up to a fixed date can forecast what comes next. It decomposes a future paper into four linked predictions—who collaborates, which prior work they build on, what contribution they produce, and what impact it has—and packages 98,000 recent AI papers plus 500,000 related papers, all with temporally aligned metadata that blocks leakage of future information. The paper introduces LACERScore, an LLM-based judge of conceptual similarity between generated and real contributions, and shows that even frontier models reach only about 5.6 on a 1–10 scale when asked to reproduce a paper's title and abstract from its key references. When the task models are composed into a 12-month simulation of daily scientific production, the synthetic corpus is systematically less diverse and trends toward being less novel than human-authored research from the same period. A careful reader should care because this reframes 'automated discovery' as a measurable forecasting problem with concrete headroom.

What carries the argument

Two mechanisms carry the argument. First, the generative factorization of a scientific advance into collaborators, prior work, contribution, and impact—conditioned on the temporal history H<t—provides both a modeling scaffold and a way to compose task models into a closed-loop simulation of scientific production. Second, LACERScore (Lattice of Automatically Constructed Exemplars for Reference Score) is an LLM-as-judge metric that calibrates a 1–10 conceptual-similarity scale without expensive human annotation: a key reference anchors score 1, a paraphrase of the target anchors score 10, and an LLM generates intermediate title-abstract pairs that interpolate between the two, which are then us

What would settle it

Collect 500 or more pairwise human similarity judgments from annotators outside the author team on held-out target abstracts versus generated candidates, and compute LACERScore's rank correlation with those judgments; if it drops well below the 0.53 human-human baseline or below a simple embedding metric, then the diversity/novelty collapse may be an artifact of the judge rather than the simulated corpus. A second check: measure the same synthetic and natural corpora with a non-embedding novelty signal (for example, lexical or topical surprise) and see whether the 'less diverse and less novel'

Watch

Extended reading notes

Core claim

The central claim is that scientific forecasting can be operationalized as a generative decomposition P(C, R, A, I | H<t): given the history of past papers, predict the team C, the influential prior work R, the advance A (title and abstract), and the impact I. On this decomposition, the paper builds a benchmark from 98,000 recent AI papers, with author identities disambiguated, citation and collaboration graphs, and metadata snapshotted to each paper's publication date so no future information leaks into model inputs. Across four tasks, current methods leave large gaps: the best collaborator prediction reaches nDCG 0.41, prior-work selection nDCG 0.13, contribution generation scores only 5.6

Load-bearing premise

The entire headline conclusion—that synthetic research is less diverse and less novel than real research—depends on LACERScore faithfully measuring what humans mean by conceptual similarity, yet the metric's human validation rests on only 250 rankings from five annotators who are themselves among the paper's authors, with human-human agreement at just 0.53 Kendall tau-b.

Editorial extensions

If this is right

  • If the benchmark is sound, the community gains a reusable, leakage-controlled resource for training and evaluating models that anticipate the content of future papers, not just their citation counts.
  • LACERScore, if it transfers, gives automated-discovery systems a practical proxy for conceptual similarity of research contributions, replacing surface-level text metrics that compress the scale to nearly no dynamic range.
  • The headroom numbers (best contribution generation at 5.6/10, prior-work selection nDCG 0.13) imply that current models under-exploit the rich author-history and citation-graph signals packaged in the dataset.
  • The simulated-corpus result, if correct, is a measurable limitation for automated-discovery pipelines: chaining today's models over many steps collapses the diversity and novelty of the output, so open-ended generation needs explicit diversity-maintenance mechanisms.
  • Because test papers postdate the knowledge cutoffs of the evaluated models, the benchmark provides a window for contamination-controlled evaluation of future frontier models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors leave implicit: the same temporal-snapshot design could be applied to fields beyond AI—wherever a large preprint or publication corpus with author histories and citations exists—turning scientific forecasting from a bespoke exercise into a template for domain-specific benchmarks.
  • A testable extension: use LACERScore as a training reward or filtering signal in a generation loop to see whether explicitly penalizing closeness to existing synthetic outputs preserves diversity over longer rollouts, directly addressing the mode-collapse the paper observes.
  • The paper's strongest negative result may be sensitive to the choice of diversity measure; an independent check with non-embedding novelty signals (e.g., lexical or structural novelty, or out-of-corpus topic classifiers) would confirm whether the collapse is a property of LLM generation or of the embedding neighborhood geometry used for scoring.
  • The human-human agreement of 0.53 reported in validation suggests that 'conceptual similarity' is itself subjective; a production version of this benchmark will likely need to track multiple judge distributions, not a single 1–10 scale.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces PreScience, a large-scale benchmark for scientific forecasting centered on 98K recent AI papers (with companion metadata yielding 502K papers in total), and decomposes a scientific advance into four generative components—collaborators (C), prior work (R), advance (A), and impact (I). Seven exemplar tasks are instantiated: contribution generation, collaborator prediction, prior work selection, citation count prediction, future combination prediction, and two topic trend forecasting variants. The authors develop baselines from heuristics to frontier LLMs, propose a new LLM-as-judge metric LACERScore for evaluating generated contribution descriptions, and compose task models into a 12-month end-to-end simulation. They report that frontier LLMs achieve only moderate LACERScore agreement with ground-truth contributions (GPT-5, 5.6 on a 1–10 scale) and that the synthetic corpus is systematically less diverse and less novel than human-authored research from the same period. Dataset and code are released.

Significance. If the results hold, PreScience would be a valuable community resource: it is large-scale, temporally controlled, and intentionally designed for forecasting rather than retrospective analysis. The strongest aspects are the snapshot-timed metadata, the reported contamination checks around model knowledge cutoffs (Table 8), the honest limitations section, and the public release of data and code. The paper is also useful in framing scientific forecasting as a composition of interdependent generative tasks. The headline corpus-level claim—that LLM-based simulation collapses diversity and novelty—is important for automated-discovery pipelines. However, that claim is measured entirely by LACERScore, whose validation is narrow (250 rankings from 5 author-annotators on 10 targets, Appendix G.3), so the significance depends on additional validation of the metric in the corpus-level regime. The benchmark and the per-task evaluations are, in themselves, a solid contribution even if the diversity claim requires further support.

major comments (4)
  1. [§2.3, §4.5, Figure 4, Appendix G.3] The corpus-level diversity/novelty result in §4.5 is measured exclusively with LACERScore, but LACERScore is validated only as a ranking metric for comparing a generated contribution to a ground-truth abstract. Appendix G.3 describes 250 human rankings from five annotators (drawn from the paper's own authors) on 10 targets, with human–human τb = 0.53; LACERScore only approaches that agreement. Using the same metric to score pairwise similarity between arbitrary title–abstract pairs—within-month nearest neighbors for diversity, and against H<t for novelty—is an extrapolation with no human validation. In addition, the 1–10 scale is calibrated by GPT-5-generated interpolations anchored at a key reference of the target, which is also an input to generation. The headline 'less diverse / less novel' could therefore be an artifact of the judge. Please add a targeted validation of LACERScore on
  2. [§4.5, Table 9] The generator and judge overlap compounds the previous concern: GPT-5 is used to generate the synthetic corpus, and GPT-5 is also the LACERScore judge for the reported results. Table 9 shows only that two judges agree on relative rankings for the contribution-generation task (Pearson 0.97, Spearman 0.81); it does not test agreement on the corpus-level pairwise scores used in Figure 4. If GPT-5 systematically prefers outputs that resemble its own style, synthetic papers will appear more self-similar (less diverse) and more similar to their conditioning references (less novel). Please report the Figure 4 protocol with an independent judge (e.g., Claude Opus 4.5) on the same pairs, or provide evidence that corpus-level LACERScore scores are stable across judge choices.
  3. [§4.5 Evaluation protocol] The diversity and novelty estimates depend on several free choices: n = 100 papers/month, k = 10 neighbors, the GRIT embedding space, and retrieval-pool subsampling. No sensitivity analysis is reported. Since LACERScore is applied to k-nearest-neighbor pairs, changing k, n, or the embedding model changes the pair distribution, and the mean scores could shift. To establish that the Figure 4 gap is a property of the synthetic corpus rather than the specific retrieval configuration, please report robustness across n, k, and embedding models (GTR, Specter2, GRIT), and state the number of rollouts and the variance of the gap across runs.
  4. [§2.4, §3, Table 1] The impact prediction target is defined as the cumulative citation count at time t + 12 months. The test period spans October 2024 to October 2025, and the manuscript is dated February 2026. For test papers published after February 2025, t + 12 months is after the stated data/preprint date, so complete 12-month citation counts cannot be observed. The paper does not specify the exact observation cutoff for citation data, nor does it state that the test set is restricted to papers with fully observed 12-month windows. As written, the impact prediction results and the test-set 'Avg. Citations @ 12m' in Table 1 are not reproducible. Please clarify the cutoff and either restrict the test set accordingly or define the target differently.
minor comments (6)
  1. [Abstract and §4.3] The abstract says frontier LLMs fall 'short of human performance', but the comparison is to ground-truth papers, not to human forecasters. Suggest rephrasing to 'short of the ground-truth contributions'.
  2. [§2.3] The definition of 'key reference with median n-gram overlap' should specify the tokenization (word vs. character n-grams) and whether the overlap is computed over the title, abstract, or both.
  3. [Algorithm 1, §4.5] SAMPLERESEARCHTEAM takes a parameter p_new that is estimated in line 1 but never defined in the main text. Please clarify how the probability of a new collaborator is estimated and how it is used.
  4. [Appendix D.1, Table 8] The contamination check covers only three models and a one-month pre/post window. A more systematic check—e.g., comparing scores on papers published well before and well after each cutoff over a longer interval—would strengthen the claim that cutoff effects are negligible.
  5. [Appendix G.2.1] The LACERScore judge prompt is extremely long and contains five fully worked interpolation examples. It would help readers to state the average output token length and the cost per scored pair, since the metric's practical usability depends on this.
  6. [§4.1, Table 2] The Frequency baseline outperforms all embedding methods on collaborator prediction (nDCG 0.41 vs. 0.28). It may be worth a sentence in the main text noting that this is consistent with prior link-prediction findings, to preempt the impression that the embedding baselines are unintentionally weak.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark targets are real future papers, and no prediction is equivalent to its inputs by construction.

full rationale

PreScience is a new forecasting benchmark whose targets are externally defined real arXiv papers, not outputs of the models being evaluated. The train/test split is temporally future (train Oct 2023–Oct 2024; test Oct 2024–Oct 2025), so the benchmark results do not reduce to the model inputs. LACERScore is an LLM-as-judge metric calibrated with a key reference as the score-1 anchor, but it is not a fitted parameter; contribution-generation scores are measured against the actual ground-truth abstract, and the paper explicitly checks robustness across two different LLM judges (Table 9, Pearson 0.97), so the reported relative rankings are not forced by the metric's construction. The corpus-level diversity/novelty finding depends on LACERScore's validity as a pairwise similarity measure, and its human validation is limited (250 rankings from five author-annotators, human-human τb=0.53; Appendix G.3), but that is a measurement-validity concern rather than a definitional circularity: nothing in the paper's equations makes a synthetic paper's diversity score equal to its generation input. No load-bearing self-citation chain or imported uniqueness theorem is used. The paper is self-contained against an external target distribution, and the headline results are empirical claims that could in principle be falsified by other metrics or future data.

Assumptions & free parameters 5 free parameters · 7 assumptions · 2 invented entities

The benchmark's results rest on external data sources and modeling choices the paper inherits rather than proves: Semantic Scholar's highly-influential-citations classifier defines the key-reference ground truth and LACERScore's score-1 anchor; S2AND disambiguation defines author identities; 12-month citation counts stand in for impact; the C-R-A-I factorization orders the tasks. LACERScore itself is an invented measurement construct whose only external validation is 250 self-annotated human rankings, and the diversity/novelty findings inherit its validity.

free parameters (5)
  • LACERScore scale anchors = 1 = key reference at median n-gram overlap; 10 = LLM paraphrase
    Hand-chosen calibration defining the absolute 1-10 scale of the headline metric (§2.3); changing anchors shifts absolute scores, though relative rankings were checked to be stable.
  • Corpus-evaluation sampling (n=100 papers/month, k=10 neighbors, GRIT embedding space) = n=100, k=10, GRIT
    Hand-chosen evaluation protocol for diversity/novelty in §4.5; results could shift with different pool sizes, embeddings, or retrieval pools.
  • Target-paper filter: 1-10 key references, non-empty author histories = 1-10 key refs; authors with histories
    Dataset inclusion rule (§3) that changes benchmark composition; excludes zero-key-reference and many-key-reference papers, and first-time authors.
  • Hierarchical-clustering centroid count m = ⌊√n⌋ = m = floor(sqrt(n))
    Hyperparameter for author multi-interest representation in collaborator/prior-work baselines (§4.1-4.2).
  • Impact horizon and transform = 12-month citations; log transform
    Task design: citation count at t+12 months as regression target with log transform (§2.4, §4.4).
assumptions (7)
  • domain assumption Semantic Scholar's 'highly influential citations' production classifier identifies a paper's influential prior work
    Defines ground-truth key references for prior-work selection (§2.2) and anchors LACERScore's score-1 (§2.3). No accuracy figures for the classifier are reported.
  • domain assumption S2AND author-name disambiguation yields correct author identities
    Author histories and collaborator-prediction ground truth depend on disambiguated identities (§3); errors propagate into histories and citation attribution.
  • domain assumption 12-month citation counts are a valid proxy for scientific impact
    Impact prediction target and framing (§2.4); the paper itself notes slower-recognized contributions are missed (Limitations, §6).
  • domain assumption The generative factorization P(C|H)P(R|C,H)P(A|C,R,H)P(I|C,R,A,H) is a usable model of a scientific advance
    Eq. 1 in §2; the paper explicitly states this is an operational scaffold, not a causal theory, and that components co-evolve in reality.
  • domain assumption LACERScore judge outputs track human conceptual-similarity judgments outside the 250-item validation set
    Underlies contribution-generation numbers and corpus diversity/novelty findings (§4.3, §4.5); validated only on 250 self-annotated rankings (Appendix G.3).
  • domain assumption Temporal metadata snapshots (citation counts, h-indices at publication time) are leakage-free and accurate
    The central leakage defense in §3; if snapshot timestamps are wrong, future information contaminates features.
  • domain assumption The empirical daily paper-count distribution P_daily estimated on the train period applies to the test period
    Used in Algorithm 1 corpus simulation; the paper itself notes resulting pools slightly underestimate ground-truth counts (§4.5).
invented entities (2)
  • LACERScore independent evidence
    purpose: LLM-as-judge metric scoring conceptual similarity of generated vs. ground-truth title-abstract pairs on a calibrated 1-10 scale
    Validated against 250 human rankings (Kendall τb approaching human IAA of 0.53) and robust to judge choice (Pearson 0.97 across GPT-5/Claude Opus judges) — a falsifiable handle, though the human validation is self-annotated.
  • C-R-A-I generative decomposition
    purpose: Four-component model (Collaborators, References, Advance, Impact) factoring the joint distribution of a scientific contribution (Eq. 1)
    The paper itself disclaims it as a causal theory (§2); it is a scaffold whose validity is only tested indirectly through the four task metrics, with no independent external validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PreScience: A Dataset and Benchmark for Scientific Forecasting." pith.science (2026). https://pith.science/paper/U2HK5TZ3

@misc{pith2026260220459,
  author       = {Pith},
  title        = {Pith review of: PreScience: A Dataset and Benchmark for Scientific Forecasting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U2HK5TZ3}},
  note         = {Machine review of arXiv:2602.20459}
}
read the original abstract

Can AI systems trained on the existing scientific record forecast the advances that will follow? We introduce PreScience, a dataset and benchmark for scientific forecasting built around 98K recent AI research papers, together with companion papers covering author publication histories and citation links, yielding 502K papers in total. The resulting paper records include titles, abstracts, disambiguated author identities, influential references, topic labels, citation trajectories, and metadata snapshotted to respect temporal cutoffs. We instantiate seven exemplar tasks: five paper-anchored tasks -- contribution generation, collaborator prediction, prior work selection, citation count prediction, and future combination prediction -- and two aggregate topic trend forecasting variants. We develop baselines ranging from simple heuristics and embedding methods to frontier language models and agentic systems, and introduce LACER, an LLM-based metric for evaluating similarity of generated contribution descriptions that agrees better with human judgments than existing metrics. Finally, we compose task models to generate a 12-month synthetic corpus and find that the resulting papers are systematically less diverse and less novel than human-authored research from the same period. We release the PreScience dataset (https://huggingface.co/datasets/allenai/prescience) and code (https://github.com/allenai/prescience).

Figures

Figures reproduced from arXiv: 2602.20459 by the authors.

Figure 1
Figure 1. Our generative decomposition of a scientific advance. A team of collaborators (C) identifies a set of foundational prior work (R) that they build upon to produce a scientific advance (A) which goes on to achieve impact (I). All of these steps are conditioned on historical scientific advances H <t, and the resulting advance is incorporated back into the history to inform future advances. tion generation – generating … view at source ↗
Figure 2
Figure 2. LACERScore approaches human-level agreement with human similarity judgments, outperforming other metrics. of its evaluations in Appendix G.5. Validating LACERScore. We validate LACERScore us￾ing 250 human similarity rankings from 5 expert annotators across 10 targets and 10 candidate generations (sourced from four strong LLMs) per target. Annotators ranked can￾didates by conceptual similarity to the ground-truth abs… view at source ↗
Figure 3
Figure 3. Prediction performance as familiarity increases. (a) Collaborator prediction. (b) Prior-work prediction. serve as compute-efficient7 baselines for scientific text gen￾eration. As points of reference, we also evaluate a gold paraphrase of the target abstract, a random key reference, and a random paper from the same primary arXiv category. We report results with GPT-5 (gpt-5-2025-08-07) as the LACERScore judge [PITH_… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Simulated (synthetic) papers (a) are less diverse and (b) trend towards being less novel compared to ground truth (natural) papers that correspond to the same time period. When novelty is measured relative to the fixed pre-simulation corpus (c) this trend disappears. b…
Figure 5
Figure 5. Figure 5: visualizes key properties of the PreScience dataset over target papers, including distributions of author counts per paper, author publication history lengths, key reference counts, and citation trajectories. These statistics highlight the heavy-tailed and heterogeneou…
Figure 6
Figure 6. Figure 6: analyzes collaborator prediction performance across two sources of variation. Panel (a) shows that nDCG typically decreases as the first author’s publication history length grows, indicating that larger and more crowded collaboration neighborhoods dilute the signal ava…
Figure 7
Figure 7. Figure 7: presents analyses of prior work selection performance across author experience, number of references, and team size. Across all three views, we observe limited and non-monotonic variation in nDCG and R-Precision across baselines, suggesting that no single factor strong…
Figure 8
Figure 8. Figure 8: Contribution Generation D.3. Contribution Generation LLM Prompt We provide below, the prompt we use with the baselines we list in [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: a plots the predictions of the XGBoost regressor trained using the full set of features described in Section 4.4. We find that the model exhibits clear heteroscedasticity: variance in prediction error increases with citation magnitude, indicating that highly cited pape…
Figure 10
Figure 10. Figure 10: Diversity of Authors and Prior work surfaced during corpus generation. F.2. Discussion Realistically simulating corpus rollouts can be difficult. Even assuming access to models that can perform individual tasks well, it can be challenging to use them to generate reali…
Figure 11
Figure 11. Figure 11: Primary arXiv topics of ground truth (natural) and simulated (synthetic) papers. Natural papers show significant seasonal variation while synthetic papers do not [PITH_FULL_IMAGE:figures/full_fig_p030_11.png]
Figure 12
Figure 12. Figure 12: Local clustering coefficient (i.e. the fraction of pairs of key references of the target paper that cite another of its key references) G. Selecting a Corpus Generation Metric G.1. FacetScore Unsatisfied with existing measures of textual similarity (ROUGE-L, BERTScore…
Figure 13
Figure 13. Figure 13: Agreement between humans, models, and aggregates. ‘Human’ refers to an average agreement among all five annotators – agreement computed against all the human annotators (excluding self-agreement), and then averaged. LACERScore is the only automated metric that reaches…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Scientific reasoning does not reliably translate into scientific forecasting in frontier AI

    cs.AI 2026-05 unverdicted novelty 7.0 of 10

    Introduces the CUSP benchmark across 4760 events and finds frontier AI models can pick plausible directions but fail to predict whether or when scientific advances will occur, with performance varying by domain and in...

Reference graph

Works this paper leans on

33 extracted references · 15 linked inside Pith · cited by 1 Pith paper

  1. [2]

    org/CorpusID:276647576

    URL https://api.semanticscholar. org/CorpusID:276647576. 10 PreScience: A Benchmark for Forecasting Scientific Contributions Chen, J., Zhang, K., Li, D., Feng, Y ., Zhang, Y ., and Deng, B. Structuring scientific innovation: A framework for modeling and discovering impactful knowledge combinations.ArXiv, abs/2503.18865, 2025a. URL https://api.semanticscho...

    arXiv 2023

  2. [5]

    org/CorpusID:280011519

    URL https://api.semanticscholar. org/CorpusID:280011519. Frohnert, F., Gu, X., Krenn, M., and van Nieuwen- burg, E. P. L. Discovering emergent connections in quantum physics research via dynamic word embed- dings.Machine Learning: Science and Technology, 6, 2024. URL https://api.semanticscholar. org/CorpusID:273963065. Fu, J., Zhang, X., Pashami, S., Rahi...

    arXiv 2024

  3. [6]

    org/CorpusID:273821733

    URL https://api.semanticscholar. org/CorpusID:273821733. Gy˝orffy, B., Herman, P., and Szab ´o, I. Research funding: past performance is a stronger predictor of future scientific output than reviewer scores. J. Informetrics, 14:101050, 2020. URL https: //api.semanticscholar.org/CorpusID: 219933512. Hadˇzi´c, A., Papez, M., and Pevn ´y, T. Distillation of ...

    arXiv 2020

  4. [8]

    org/CorpusID:277451644

    URL https://api.semanticscholar. org/CorpusID:277451644. Jansen, P. A., C ˆot´e, M.-A., Khot, T., Bransom, E., Dalvi, B., Majumder, B. P., Tafjord, O., and Clark, P. Discoveryworld: A virtual environ- ment for developing and evaluating automated sci- entific discovery agents.ArXiv, abs/2406.06769,

    arXiv

  5. [11]

    org/CorpusID:86850665

    URL https://api.semanticscholar. org/CorpusID:86850665. Koopmann, T., Kobs, K., Herud, K., and Hotho, A. Cobert: Scientific collaboration prediction via sequen- tial recommendation.2021 International Conference on Data Mining Workshops (ICDMW), pp. 45–54,

    2021

  6. [12]

    org/CorpusID:246081502

    URL https://api.semanticscholar. org/CorpusID:246081502. Lee, A. X. W., Yeung, P.-H., and Rajapakse, J. C.Subcorti- cal Masks Generation in CT Images via Ensemble-Based Cross-Domain Label Transfer, pp. 160–174. Springer Na- ture Switzerland, July 2025. ISBN 9783031986949. doi: 10.1007/978-3-031-98694-9 12. URL http://dx. doi.org/10.1007/978-3-031-98694-9_...

    arXiv 2025

  7. [13]

    org/CorpusID:281682043

    URL https://api.semanticscholar. org/CorpusID:281682043. Li, X., Wang, M., Wang, C., Fu, Y ., and Wang, X. Novsrc: A novelty-oriented scientific collabo- rators recommendation model.International Jour- nal of Advanced Computer Science and Applications,

  8. [14]

    org/CorpusID:268818672

    URL https://api.semanticscholar. org/CorpusID:268818672. Liben-Nowell, D. and Kleinberg, J. The link prediction prob- lem for social networks. InProceedings of the Twelfth International Conference on Information and Knowl- edge Management, CIKM ’03, pp. 556–559, New York, NY , USA, 2003a. Association for Computing Machinery. ISBN 1581137230. doi: 10.1145/...

    arXiv

Show all 33 references
  1. [15]

    org/CorpusID:271854887

    URL https://api.semanticscholar. org/CorpusID:271854887. Lundberg, S. M. and Lee, S.-I. A unified approach to inter- preting model predictions. InProceedings of the 31st In- ternational Conference on Neural Information Processing Systems, NIPS’17, pp. 4768–4777, Red Hook, NY , USA,

  2. [18]

    Margatina, K., Wang, S., Vyas, Y ., Anna John, N., Be- najiba, Y ., and Ballesteros, M

    URL https://openreview.net/forum? id=vyflgpwfJW. Margatina, K., Wang, S., Vyas, Y ., Anna John, N., Be- najiba, Y ., and Ballesteros, M. Dynamic benchmark- ing of masked language models on temporal concept drift with multiple views. In Vlachos, A. and Augen- stein, I. (eds.),P...

    2023 arXiv

  3. [19]

    org/CorpusID:209370497

    URL https://api.semanticscholar. org/CorpusID:209370497. Muennighoff, N., SU, H., Wang, L., Yang, N., Wei, F., Yu, T., Singh, A., and Kiela, D. Generative repre- sentational instruction tuning. InThe Thirteenth In- ternational Conference on Learning Representations,

  4. [20]

    Munasinghe, L

    URL https://openreview.net/forum? id=BC4lIvfSzv. Munasinghe, L. and Ichise, R. Time score: A new fea- ture for link prediction in social networks.IEICE Trans. Inf. Syst., 95-D:821–828, 2012. URL https://api. semanticscholar.org/CorpusID:30012200. Mysore, S., Cohan, A., and Hop...

    2012

  5. [23]

    org/CorpusID:272986926

    URL https://api.semanticscholar. org/CorpusID:272986926. Radensky, M., Shahid, S., Fok, R., Siangliulue, P., Hope, T., and Weld, D. S. Scideator: Human- llm scientific idea generation grounded in research- paper facet recombination.ArXiv, abs/2409.14634,

    arXiv

  6. [24]

    org/CorpusID:272827497

    URL https://api.semanticscholar. org/CorpusID:272827497. Redman, B. Science evaluation: Peer review, bibliomet- rics, and research impact assessment. InReconstructing Research Integrity, pp. 127–148. Springer, 2023. Riechers, P. M., Elliott, T. J., and Shai, A. S. Neural net- ...

    2023 arXiv

  7. [25]

    org/CorpusID:273346445

    URL https://api.semanticscholar. org/CorpusID:273346445. Su, H., Chen, R., Tang, S., Yin, Z., Zheng, X., Li, J., Qi, B., Wu, Q., Li, H., Ouyang, W., Torr, P., Zhou, B., and Dong, N. Many heads are better than one: Improved scientific idea generation by a LLM-based multi-agent ...

  8. [26]

    ISBN 979-8-89176-251-0

    Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long

    2025 doi

  9. [28]

    org/CorpusID:274060096

    URL https://api.semanticscholar. org/CorpusID:274060096. Thelwall, M. et al. Predicting article quality scores with ma- chine learning: The u.k. research excellence framework. Quantitative Science Studies, 4(2):547–573, 2023. 15 PreScience: A Benchmark for Forecasting Scientif...

    2023

  10. [30]

    org/CorpusID:273695165

    URL https://api.semanticscholar. org/CorpusID:273695165. Weis, J. W. and Jacobson, J. Delphi: A machine learning framework for early alert of high-impact research.Nature Biotechnology, 2021. Xi, X., Guo, Y ., and Duan, W. Recommendation of aca- demic collaborators: A methodolo...

    2021

  11. [31]

    org/CorpusID:235259334

    URL https://api.semanticscholar. org/CorpusID:235259334. Yang, Y ., Dan, S., Roth, D., and Lee, I. Benchmarking llm guardrails in handling multilingual toxicity, 2024. URL https://arxiv.org/abs/2410.22153. Yu, H., Hong, Z., Cheng, Z., Zhu, K., Xuan, K., Yao, J., Feng, T., and ...

    2024 arXiv

  12. [32]

    org/CorpusID:274992362

    URL https://api.semanticscholar. org/CorpusID:274992362. Zhang, T., Kishore, V ., Wu, F., Weinberger, K. Q., and Artzi, Y . Bertscore: Evaluating text generation with bert.ArXiv, abs/1904.09675, 2019. URL https: //api.semanticscholar.org/CorpusID: 127986044. Zhang, Y ., Tang, ...

    1904 arXiv

  13. [33]

    an image is worth a thousand words

    URL https://api.semanticscholar. org/CorpusID:259081885. Zhao, C., Pisu, P., Comert, G., Begashaw, N., Vaidyan, V ., and Hubig, N. C. Causal interpretability for ad- versarial robustness: A hybrid generative classification approach, 2025. URL https://arxiv.org/abs/ 2412.20025....

    2025

  14. [331]

    naacl-main.331/

    URL https://aclanthology.org/2022. naacl-main.331/. Ni, J., Qu, C., Lu, J., Dai, Z., Hernandez Abrego, G., Ma, J., Zhao, V ., Luan, Y ., Hall, K., Chang, M.-W., and Yang, Y . Large dual encoders are generalizable retrievers. In Gold- berg, Y ., Kozareva, Z., and Zhang, Y . (ed...

    2022 doi

  15. [669]

    emnlp-main.669/

    URL https://aclanthology.org/2022. emnlp-main.669/. Ni, Z., Wang, Y ., Zhou, R., Han, Y ., Guo, J., Liu, Z., Yao, Y ., and Huang, G. Enat: Rethinking spatial-temporal interactions in token-based image synthesis, 2024. URL https://arxiv.org/abs/2411.06959. NIST TREC. Common eva...

    2022 arXiv

  16. [1368]

    acl-long.1368/

    URL https://aclanthology.org/2025. acl-long.1368/. Subramanian, S., King, D., Downey, D., and Feldman, S. S2AND: A Benchmark and Evaluation System for Author Name Disambiguation. InJCDL ’21: Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2021, JCDL ’21, N...

    2025

  17. [2002]

    org/CorpusID:1981391

    URL https://api.semanticscholar. org/CorpusID:1981391. Jin, L., Ruan, Z., Mai, H., and Shang, J. Verilocc: End-to- end cross-architecture register allocation via llm, 2025. URLhttps://arxiv.org/abs/2506.17506. Kanakaris, N., Giarelis, N., Siachos, I., and Karacapi- lidis, N. S...

    2025 arXiv

  18. [2017]

    ISBN 9781510860964

    Curran Associates Inc. ISBN 9781510860964. 13 PreScience: A Benchmark for Forecasting Scientific Contributions Luu, K., Wu, X., Koncel-Kedziorski, R., Lo, K., Ca- chola, I., and Smith, N. A. Explaining relation- ships between scientific documents. InAnnual Meet- ing of the Ass...

  19. [2019]

    org/CorpusID:209450869

    URL https://api.semanticscholar. org/CorpusID:209450869. Hu, E. J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. InInternational Conference 12 PreScience: A Benchmark for Forecasting Scienti...

    2022 arXiv

  20. [2020]

    org/CorpusID:236459799

    URL https://api.semanticscholar. org/CorpusID:236459799. Majumder, B. P., Surana, H., Agarwal, D., Mishra, B. D., Meena, A., Prakhar, A., V ora, T., Khot, T., Sabharwal, A., and Clark, P. Discoverybench: Towards data-driven discovery with large language models. InThe Thirteent...

  21. [2021]

    org/CorpusID:232133644

    URL https://api.semanticscholar. org/CorpusID:232133644. Feng, N., Sui, Y ., Hou, S., Cresswell, J. C., and Wu, G. Re- sponse quality assessment for retrieval-augmented gen- eration via conditional conformal factuality.Proceed- ings of the 48th International ACM SIGIR Conferen...

  22. [2022]

    org/CorpusID:251597885

    URL https://api.semanticscholar. org/CorpusID:251597885. Wang, D., Song, C., and Barab ´asi, A.-L. Quantifying long-term scientific impact.Science, 342(6154):127–132, 2013a. Wang, D., Song, C., and ´Laszl´o Barab´asi, A. Quantifying long-term scientific impact.Science, 342:127...

    2023 arXiv

  23. [2023]

    org/CorpusID:265054862

    URL https://api.semanticscholar. org/CorpusID:265054862. Chuan, P. M., Son, L. H., Ali, M., Khang, T. D., Huong, L. T., and Dey, N. Link prediction in co-authorship networks based on hybrid content similarity metric. Appl. Intell., 48(8):2470–2486, 2018. doi: 10.1007/ S10489-0...

    2018

  24. [2024]

    org/CorpusID:270380311

    URL https://api.semanticscholar. org/CorpusID:270380311. J¨arvelin, K. and Kek¨al¨ainen, J. Cumulated gain-based evalu- ation of ir techniques.ACM Trans. Inf. Syst., 20:422–446,

  25. [2025]

    ISBN 979-8-89176-189-6

    Association for Computational Linguistics. ISBN 979-8-89176-189-6. URLhttps://aclanthology. org/2025.naacl-long.342/. Boyack, K. W., Smith, C., and Klavans, R. To- ward predicting research proposal success.Sciento- metrics, 114:449–461, 2018. URL https://api. semanticscholar.o...

    2025 arXiv

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.