{"id":"eb168a91-3c2a-4f49-91cc-b6ab8072ec14","arxiv_id":"2505.08941","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"ForeCite fine-tunes causal language models with a linear head to predict average monthly citation rates of biomedical papers, reporting a Spearman rho of 0.826 when trained on 16 percent of the corpus and tested on a random holdout.","lead":"Researchers at Memorial University trained language models to predict how often academic papers get cited, reaching a 0.826 rank correlation on a test set of biomedical papers. The paper is a candidate benchmark for automated research evaluation, but the headline result comes from a random split and the state-of-the-art comparison is across different datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline rho=0.826 is measured on a 90/10 random split, but the paper's own temporal holdout (§4.3) shows Pearson r decaying from 0.713 to 0.511; until the largest model is evaluated on a temporal split, the 27-point improvement over prior work is not established.","rationale":"The reader's weakest assumption is the same one I would flag: a random split is not a valid proxy for forecasting future papers, and the paper's own temporal decay experiment provides internal evidence that the random-split result is optimistic. I considered two other concerns: (a) the 27-point improvement is a cross-dataset comparison (ForeCite on biomedical text vs CiMaTe on computational linguistics/biology and Li et al. with peer-review text), which could inflate the gap; and (b) Qwen2.5-14B's pretraining data may overlap the random-split test period, since test papers published before the model's release can appear in pretraining. Both are real but secondary: the temporal-split issue alone is sufficient to undermine the 'forecasting long-term influence' framing, and it is directly testable with the paper's own protocol. The paper deserves credit for reporting the temporal holdout and ablations honestly, but the interpretation in §5.1 that 'temporal holdout experiments demonstrate the robustness of our approach' is not supported by the reported decay. A temporal evaluation of the headline configuration would settle the question; the current CONDITIONAL verdict stands.","tokens_in":11299,"tokens_out":5953,"duration_ms":57202,"concrete_test":"Run a strict temporal holdout with the exact headline configuration (Qwen2.5-14B, 16% of the corpus): train on a random subset of papers published before 2023 and compute Spearman correlation on monthly cohorts from January 2023 through December 2024, using the same preprocessing and target transform as §3. If the 12-month Spearman is materially below the random-split 0.826, or below the prior baselines cited in §5.1 (0.556, 0.436), the headline 27-point improvement does not transfer to future-paper forecasting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that ForeCite forecasts future citation rates, quantified by rho=0.826 for Qwen2.5-14B on 16% data, a 27-point gain over previous SOTA (§5.1, §6). But §3.3 defines the evaluation as a 90/10 random split over papers from 2000–2024. A random split lets training and test papers from the same publication year and topic cohort appear on both sides, so the model can exploit year-specific citation patterns and field trends rather than predict unseen future work. The paper's own temporal holdout (§4.3) is the relevant test: Bloom-560m trained only on pre-2023 papers gives r=0.713 in January 2023, drops to 0.631 one month later, and to 0.511 by mid-2025. The temporal experiment was run only on the smallest model, so there is no evidence that the 14B model's random-split rho=0.826 survives temporal separation. The abstract and conclusion describe temporal holdout as confirming practical robustness, but the reported decay points the other way. What would restore the claim is a temporal evaluation of the model configuration used for the headline result. Until then, the 27-point improvement over prior work is not a claim about future papers, only about a random subset of the same corpus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"ForeCite appends a linear regression head to pre-trained causal language models and fine-tunes via QLoRA to predict log-transformed average monthly citation rates from full-text Markdown of biomedical papers. The authors build a 900K+ corpus from Elsevier API, train five LMs (0.5B–14B) on 1%–16% of the data under a 90/10 random split, and report the headline result rho=0.826 (Pearson r=0.844) for Qwen2.5-14B, which they describe as a 27-point improvement over prior SOTA. They also present scaling-law fits, a temporal holdout study on Bloom-560m, gradient saliency, ablations, and pre-training comparisons. The paper claims state-of-the-art forecasting of future citation rates with 'practical robustness' confirmed by temporal holdout.","tokens_in":11665,"tokens_out":5486,"duration_ms":49931,"significance":"If the results held under temporal evaluation, ForeCite would be a substantial advance: it is a simple, end-to-end regression adapter for LLMs, applied to a large curated corpus, with a systematic scaling-law grid, honest reporting of temporal decay, and useful ablations and saliency analysis. The authors are transparent about the limitations of single-run experiments and compute constraints. However, the headline result is currently based on a random split and on cross-dataset comparisons, so the significance hinges on the requested revisions. The temporal holdout with Bloom-560m is a good-faith experiment and suggests the framework works out-of-distribution for a few months, but it does not cover the model used for the headline claim.","major_comments":[{"comment":"The abstract and §6 claim that ForeCite predicts 'future citation rates' and that 'temporal holdout experiments confirm practical robustness', but the headline rho=0.826 is measured on a 90/10 random split defined in §3.3 over papers published 2000–2024. A random split interleaves training and test papers from the same years and topic cohorts, so the model can exploit year-specific citation patterns rather than forecast unseen future work. The paper's own temporal holdout (§4.3, Figure 5) shows Bloom-560m's Pearson r decaying from 0.713 at January 2023 to 0.631 one month later and 0.511 by mid-2025, while the random-split test r for the same model is 0.721 (Table 8). Since the temporal experiment was run only on the smallest model, there is currently no evidence that the 14B configuration's rho=0.826 survives temporal separation. A temporal evaluation of the headline configuration (or a clear reframing of the claims to 'within-corpus ranking' rather than 'forecasting') is necessary to support the central claim.","section":"§3.3, §4.3, §6"},{"comment":"The claimed 27-point improvement over prior state-of-the-art is computed by comparing rho=0.826 on ForeCite's biomedical corpus with rho=0.556 from Li et al. (which uses peer-review text) and rho=0.436 from CiMaTe (computational linguistics and biological domains), while R²=0.706 is compared with R²=0.4/0.454 from SChuBERT/MultiSChuBERT. These are cross-dataset comparisons: none of the baselines was re-implemented or evaluated on the ForeCite corpus. A state-of-the-art claim requires either re-running the baselines under the same train/test protocol or restricting the comparison to numbers obtained on the same benchmark. The raw improvement in Table 8 is a within-ForeCite result; the '27-point' wording is only valid if the baseline numbers are directly comparable.","section":"§5.1, Table 8"},{"comment":"The scaling-law extrapolation to Llama 4 Behemoth (r(288, 100)=0.9413, rho=0.9325) is used in §4.1 and §6 to argue that larger models would push performance 'significantly farther'. However, the tanh model of Eq. (1) is fitted to 25 points with p in [0.5, 14.8]B and d in [1, 16]%, then evaluated at p=288B and d=100%, far outside the fitted range. No confidence intervals, residual diagnostics, or held-out validation of the functional form are provided, and each grid point is a single run. The extrapolation should be explicitly labeled as speculative, with uncertainty quantification or a validation experiment on at least one larger model or data volume, before it is cited as evidence in the conclusion.","section":"§4.1, Eq. (1), Appendix E"}],"minor_comments":[{"comment":"'medical articles publish between 2000 and 2024' should be 'medical articles published between 2000 and 2024'.","section":"§3.1"},{"comment":"'This is support by our findings' should be 'This is supported by our findings'.","section":"§5.1"},{"comment":"'Figure I demonstrates' should refer to Figure 13, not the paper's section letter.","section":"Appendix I"},{"comment":"The abstract states 'comprehensive scaling-law analysis ... data volumes' but data volumes never exceed 16% of the corpus; suggest saying 'fractions of the corpus up to 16%'.","section":"Abstract"},{"comment":"The 'Open science commitment' says code will be released upon publication, but no repository or data-access link is provided; please add the URL or state availability more concretely.","section":"§1"},{"comment":"The extrapolated MAE of approximately 0.03 is reported without explanation of how it is derived from the tanh fit; clarify the calculation.","section":"§4.1"},{"comment":"The '100% data∗' point is a theoretical estimate; the asterisk is explained only in the caption, and the fitted point could be more clearly distinguished from measured points.","section":"Figure 4b"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a preprint under review. The authors are honest about their limitations, but the abstract and conclusion overstate temporal robustness given the paper's own temporal decay experiment. The editor may wish to require either a temporal evaluation of the largest model or a reframing of the central claim. The cross-dataset SOTA comparison is a common weakness but should be addressed for a journal publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nWorth a look. This is a clean, large-scale empirical study that does something prior work hasn't: end-to-end fine-tuning of causal LMs with a linear head to regress log average monthly citation counts. The scaling-law grid across five model sizes and five data fractions is thorough, and the temporal holdout, saliency, and ablations sections are honest.\n\nThe best part is the architecture works. Qwen2.5-14B on 16% of 900K biomedical papers gets rho=0.826 on a random holdout, and even the small Bloom-560m gets r=0.713 initially on a temporal holdout. That is a real achievement.\n\nBut the headline overreaches. rho=0.826 comes from a random 90/10 split over 2000-2024 papers. Test papers are interleaved with training papers, so the model can exploit year-specific citation inflation and field trends. The paper's own temporal holdout, run only on Bloom-560m, shows r=0.713 for Jan 2023, dropping to 0.631 a month later and 0.511 by mid-2025. That is a fast decay. The conclusion calls this 'practical robustness,' but the data point the other way. Until the largest model is evaluated on a temporal split, the 27-point improvement over prior work is not a claim about future papers.\n\nThe baseline comparison is also weaker than stated. The 27-point gain is against numbers from other datasets and domains (CiMaTe on computational linguistics, Li et al. with peer-review text). No re-implementation, so some of the gap may be dataset and label differences. The gap is large enough that a real improvement likely exists, but '27-point' is not quantified fairly.\n\nOther soft spots are minor: code and data pending, single-run experiments, manual hyperparameter tuning, and the Llama 4 extrapolation is clearly labeled as theoretical. The authors acknowledge these in the limitations section, which is to their credit.\n\nBottom line: the framework and the empirical characterization are new and useful, but the main claim needs a temporal evaluation of the headline configuration. For people in scientometrics and scholarly document analysis, this is a useful reference point. This paper deserves a serious referee, not a desk reject. A conditional verdict asking for temporal holdout results on the 14B model and a fairer baseline comparison would be appropriate.","headline":"Solid empirical study with a straightforward recipe and honest reporting, but the headline forecast claim rests on a random split while the paper's own temporal holdout shows decay.","tokens_in":12154,"tokens_out":2578,"would_cite":false,"duration_ms":23724,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ForeCite claims that appending a linear regression head to a pre-trained causal language model and fine-tuning with QLoRA predicts average monthly citation rates from manuscript text alone, reaching Spearman $\\rho = 0.826$ on a 900K-paper…","keywords":["citation prediction","causal language models","regression head","QLoRA fine-tuning","scaling laws","temporal holdout","scientific impact forecasting","text-based prediction"],"falsifier":"Train ForeCite on papers published up to 2020 and evaluate on papers published in 2024--2025, with no overlap in publication month; if the Spearman correlation on that chronological holdout falls below the previous best $\\rho = 0.556$, then the reported $\\rho = 0.826$ overstates the model's ability to predict future citation rates. The paper already contains a partial version of this test: the Bloom-560m temporal holdout decays from $r = 0.713$ to $r = 0.511$ by mid-2025.","tokens_in":11076,"feed_emoji":"📈","tokens_out":10164,"duration_ms":83643,"temperature":0.7,"pith_summary":"This paper argues that a pre-trained causal language model, fitted with a single linear regression head and fine-tuned end-to-end, can rank academic papers by future citation rate from the manuscript text alone. On a curated corpus of more than 900,000 biomedical papers published between 2000 and 2024, the largest configuration reaches a Spearman rank correlation of $\\rho = 0.826$ on a randomly held-out test split, a 27-point improvement over the previous best reported predictor. The authors also find that test performance grows with model size and training-data volume, that the model attributes most of its signal to titles and abstracts, and that correlations decay quickly when the model is evaluated on papers published after its training window. If these results are correct, citation forecasting no longer requires citation graphs, peer-review text, or hand-built features: a language model reading the paper can do the ranking.","feed_headline":"Text-only LLM predicts citation impact at rho = 0.826","feed_subtitle":"A 14B model reading only manuscript text beats prior citation predictors by 27 points.","key_machinery":"The machinery is a linear regression head attached to the final hidden state of a pre-trained causal language model, trained in two phases: first with the base model frozen to train only the head, then jointly with the whole model via QLoRA (quantized low-rank adaptation of a 4-bit base model). The target is the average monthly citation rate, log-transformed and standardized with training-split statistics, which maps the skewed count distribution to a near-Gaussian regression target. A bounded least-squares scaling law, $f(p,d) = \\tanh(\\beta_0 + \\beta_1 \\log_2 p + \\beta_2 \\log_2 d)$, with $p$ the parameter count and $d$ the percentage of the corpus, is used to extrapolate performance across model sizes and data volumes.","core_discovery":"ForeCite's central claim is that end-to-end regression with a causal transformer—not classification, not embeddings feeding a separate regressor—is sufficient to forecast citation impact from text. The claim is quantified: with the Qwen2.5-14B base model fine-tuned on 16% of the corpus via QLoRA, the model attains $\\rho = 0.826$ and $R^2 = 0.706$ on a random test split, beating the cited previous state of the art ($\\rho = 0.556$ with peer-review text, $R^2 = 0.454$ with figures) by a wide margin. The paper's scaling-law analysis indicates that the result is not a plateau: extrapolating a bounded least-squares fit to a 288B-parameter model trained on the full corpus yields $r \\approx 0.94$ and $\\rho \\approx 0.93$. The temporal holdout, in contrast, shows a model released in 2022 decaying from $r = 0.713$ to $r = 0.511$ within roughly two years of out-of-distribution papers.","pith_inferences":["The headline $\\rho = 0.826$ rests on a random 90/10 split of papers published 2000--2024; because training and test papers can share publication windows, part of the measured signal may be era-specific style and field trends rather than intrinsic citable quality, so a strict chronological split is the deployment test.","The observed decay from $r = 0.713$ to $r = 0.511$ within about two years suggests the useful forecasting horizon of such models is short; practical systems would need confidence intervals that widen with time since training.","If titles and abstracts dominate the attribution, authors could plausibly game the predictor by rewriting abstracts in the rewarded style; a testable defense is section-level dropout during training, then measuring whether attribution spreads to the body text.","The same linear-head-on-causal-LM recipe should transfer to other text-to-outcome regression problems, such as predicting replication, clinical uptake, or peer-review scores, whenever a large labeled text corpus exists."],"forward_implications":["Citation ranking becomes a text-only operation: no citation graph, author profile, or peer-review text is needed to exceed previously reported predictive correlations.","Performance appears to follow a predictable scaling law, so the authors extrapolate that a 288B-parameter model trained on the full corpus would reach $r \\approx 0.94$ and $\\rho \\approx 0.93$.","Titles and abstracts carry most of the attribution, yet removing them costs less than $0.02$ in Pearson $r$, suggesting that abstract-only inputs could rank papers cheaply without catastrophic loss.","The temporal holdout profile—strong for about one month, then steadily decaying—implies that practical deployments would need periodic retraining or online adaptation to stay accurate.","Reasoning-specialized fine-tuning does not help and slightly hurts citation prediction, suggesting that general semantic knowledge, not logical specialization, drives the signal."],"supporting_citations":[{"why":"Supplies the previous state-of-the-art Spearman correlation ($\\rho = 0.436$) that ForeCite's $\\rho = 0.826$ is compared against.","marker":"[Hirako et al., 2024]"},{"why":"Gives the strongest prior number ($\\rho = 0.556$) obtained with peer-review text, the baseline ForeCite beats using only manuscript text.","marker":"[Li et al., 2019]"},{"why":"Provides the SChuBERT text-chunk baseline with $R^2 = 0.4$ that ForeCite's $R^2 = 0.706$ exceeds.","marker":"[van Dongen et al., 2020]"},{"why":"Provides the multimodal MultiSChuBERT baseline with $R^2 = 0.454$ that ForeCite beats using text alone.","marker":"[Wenniger et al., 2023]"},{"why":"Supplies the QLoRA method used to fine-tune the base language model jointly with the regression head.","marker":"[Dettmers et al., 2023]"},{"why":"Provides the empirical scaling-law methodology on which the paper's tanh extrapolation to larger models and data is based.","marker":"[Kaplan et al., 2020]"},{"why":"Establishes the few-shot and semantic capabilities of large pre-trained language models that justify adapting them for citation prediction.","marker":"[Brown et al., 2020]"}],"fun_headline_variants":["LLM reads abstracts, predicts citations at 0.826","ForeCite: text-only model beats prior predictors by 27","Text-trained LLM forecasts citations, rho 0.826","From text to citations: 14B model scores 0.826"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported $\\rho = 0.826$ depends on treating a random 90/10 split of the 2000--2024 corpus as a stand-in for future papers; if test papers share publication windows with training papers, the model can learn era-specific style and field trends, and the paper's own temporal holdout (decay from $r = 0.713$ to $r = 0.511$) shows that this premise only partially holds.","fun_headline_variants_meta":{"raw":{"variants":["LLM reads abstracts, predicts citations at 0.826","ForeCite: text-only model beats prior predictors by 27","Text-trained LLM forecasts citations, rho 0.826","From text to citations: 14B model scores 0.826"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"total_tokens":1395,"prompt_tokens":942,"completion_tokens":453,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":379}},"tokens_in":558,"tokens_out":453,"duration_ms":5136,"temperature":1.0,"reasoning_tokens":379,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:44:02.054188+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train ForeCite on papers published up to 2020 and evaluate on papers published in 2024--2025, with no overlap in publication month; if the Spearman correlation on that chronological holdout falls below the previous best $\\rho = 0.556$, then the reported $\\rho = 0.826$ overstates the model's ability to predict future citation rates. The paper already contains a partial version of this test: the Bloom-560m temporal holdout decays from $r = 0.713$ to $r = 0.511$ by mid-2025.","supporting_citations":[],"review_version":1}