Pith. sign in

REVIEW 4 major objections 7 minor 49 references

Benchmark for Evaluation and Analysis of Citation Recommendation Models

T0 review · 4 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper establishes a standardized benchmark for local citation recommendation and reports that BM25, the simplest method, achieves the best Recall@10 and MRR@10 on most of its diagnostic datasets.

desk verdict A careful proposal for a citation-recommendation benchmark with a useful taxonomy, but the evaluation is confounded and the paper is not a finished benchmark. read the letter →

arxiv 2412.07713 v1 pith:MKN2UQPS submitted 2024-12-10 cs.IR cs.AIcs.DL

Puja Maharjan This is my paper · ORCID
classification cs.IRcs.AIcs.DL
keywords citationrecommendationbenchmarkdiagnosticdatasetsS2ORCcontextBM25baselineRecall@10MRR@10
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Citation recommendation systems are built on many different datasets, context definitions, and metrics, so results across papers are hard to compare. This paper argues that the field needs a standardized benchmark and delivers one: a set of diagnostic datasets, derived from the S2ORC and S2AG corpora, each isolating a single feature of the citing context such as sentence length, citation position, citation intent, part-of-speech surroundings, field, year, popularity, or low-resource status. On these datasets the paper evaluates four models—two neural local recommendation systems, the Galactica language models, and a BM25 baseline—and reports Recall@10 and MRR@10 in a uniform way. The main empirical result is that BM25, despite being the simplest method, achieves the best recall and MRR on most datasets, with neural models winning on specific classes such as sentence length and intent. If the benchmark is used by the community, model comparisons become meaningful and the conditions under which complex models actually help become visible.

What carries the argument

The central object is the set of eight diagnostic datasets built by stratified sampling from the S2ORC full-text corpus and S2AG metadata. For each dataset, a single feature of the citation context is isolated—field, publication year group, citation-count decile, context length class, normalized citation position, citation intent, part-of-speech of the preceding or following word, or low-resource field—while other features are held fixed; for example, the length dataset keeps the citation at the end of the sentence. The extraction pipeline samples papers hierarchically by field, year, and citation count, parses sentences with a scientific text-processing pipeline, keeps only sentences containing exactly one citation, verifies the citation with a regular-expression citation parser, replaces the citation with the <REF> placeholder, and assigns the sentence to a class. These datasets are what make the evaluation diagnostic: a model's score on a given dataset shows how well it handles that specific contextual feature.

What would settle it

Re-run the dataset construction after manually verifying a random sample of S2ORC citation links, or after replacing the citation parser with a stricter one, and compare the results to Table 1; if BM25's lead over LCR and Galactica shrinks or reverses, the headline result is an artifact of noisy citation mapping. More directly, ask annotators whether the masked sentence alone identifies the cited paper; if it often does not, the diagnostic task is not measuring what the benchmark claims.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims to establish the first comprehensive benchmark for evaluating citation recommendation models, focused on local (context-based) recommendation. The load-bearing empirical claim is that BM25 ranking, a classical lexical retrieval baseline, outperforms the neural and large-language-model alternatives on most diagnostic datasets in both Recall@10 and MRR@10, and that the remaining wins are concentrated in specific classes: LCR beats BM25 on recall for the context-length and citation-intent datasets, while BM25 dominates on fields, years, citation-count groups, and the low-resource set. The paper also reports feature-level patterns, such as neural models doing better on short contexts and BM25 doing better when the citation appears at the start of the sentence. The contribution is therefore twofold: a reusable evaluation resource with controlled context features, and an initial comparative result showing where simple retrieval still wins.

Load-bearing premise

The benchmark's scores inherit whatever errors are in S2ORC's in-text citation links: if a citation mention is mapped to the wrong paper, or the one-citation sentence is too ambiguous to identify the intended reference, every diagnostic dataset and score is affected.

Editorial extensions

If this is right

  • Researchers can evaluate any new citation recommendation model on the same diagnostic datasets and the same Recall@10 and MRR@10 metrics, making published results directly comparable.
  • BM25 becomes a default baseline that more complex models must beat; its strong performance means simple lexical retrieval is the bar, not the floor.
  • Models can be profiled by feature, e.g., LCR's recall advantage on length and intent datasets and BM25's advantage on low-resource fields point to where architectural investment pays off.
  • Because each dataset controls for other context features, differences in scores can be attributed to the feature under study rather than to accidental dataset variation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The author's explanation that BM25 queries the full S2AG metadata suggests a testable separation: ablating BM25's access to metadata would reveal how much of its lead comes from the citation context itself versus from matching titles and other bibliographic fields.
  • The same diagnostic construction could be extended to global citation recommendation, where the input is the whole paper rather than a single sentence; the paper notes such extensions as future work.
  • The benchmark's usefulness depends on the accuracy of S2ORC's citation annotations; an independent re-annotation study of a random sentence sample would show whether the reported rankings are stable under annotation noise.
  • Because the paper reports only Recall@10 and MRR@10, adding nDCG and F1 would test whether the ranking conclusions survive a change of metric—something the author lists among the limitations.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. This paper proposes a benchmark for local citation recommendation. Using S2ORC and S2AG, the authors construct eight diagnostic datasets designed to isolate different features of the citation context (field, publication year, citation count, context length, citation location, citation intent, POS of surrounding words, and low-resource fields) and evaluate four model families (NCN, LCR, Galactica at three sizes, and a BM25 baseline) with Recall@10 and MRR@10 across all classes. Section 4 reports that BM25 achieves the best Recall and MRR for most datasets, and the Conclusion asserts that a comprehensive benchmark for evaluating and comparing citation recommendation models is established.

Significance. Standardized, diagnostic benchmarks are genuinely needed in citation recommendation, and the paper identifies a sensible, literature-motivated decomposition of citation-context features; the eight dataset dimensions, the inclusion of multiple Galactica sizes, and the release of code and data (GitHub and Google Drive links in Section 1) are useful steps. If the datasets, protocol, and comparison were fully specified, this could become a reusable evaluation resource. As it stands, however, the central claims cannot be assessed: dataset sizes are not reported anywhere, the evaluation protocol is not reproducible from the text, the headline model comparison is admitted in Section 4 to be confounded by unequal retrieval pools, and no error analysis is provided despite the paper's stated analytic goal. The paper is therefore better read as a description of a benchmark pipeline than as an established benchmark with verified results.

major comments (4)
  1. [§4 and Table 1] The headline comparison is confounded by unequal candidate pools. The paper's own explanation — "BM25 demonstrates superior overall performance compared to other models, primarily because it has the advantage of querying the entire S2AG dataset, which comprises millions of records" — concedes that BM25 ranks over the full S2AG index while no candidate pool, negative-sample set, or index is reported for NCN, LCR, or the Galactica models. Because Recall@10 and MRR@10 are computed over whatever candidate set each system returns, the Table 1 finding that "BM25 shows the best performance in terms of Recall and MRR for most datasets" does not support any conclusion about model quality and contradicts the benchmark's stated purpose of standardized comparison. The evaluation must be re-run with identical candidate pools for all models (or with BM25 restricted to each model's pool).
  2. [§3.2.1 and §5] No dataset sizes are reported. The sampling procedure states an upper limit of 10 papers per class group (step 4), but the final number of cited papers, citing sentences, and sentences per class per diagnostic dataset are never given, and the Limitations section concedes that "The size of the diagnostic dataset could be more significant." Without per-class sample counts, the Table 1 metrics are uninterpretable (e.g., the Fields/Biology gap of 0.195 vs 0.1094 between BM25 and the best neural model could be sampling noise on a small test set), and the released resource cannot be evaluated by other researchers. The paper must report total and per-class sizes, the number of samples removed at each filtering stage, and ideally confidence intervals for the reported metrics.
  3. [§4 and Table 1] The evaluation protocol is underspecified to the point of irreproducibility. The text does not say how the extracted sentence with <REF> is encoded for each model, how candidate papers are generated for NCN, LCR, and Galactica, what negative samples or context-window sizes are used (prior work in §2 uses 100, 200, 600, or 2048 tokens), or what inference hyperparameters are applied. In addition, Table 1 contains many zero entries — Galactica 125m is 0.0 for nearly every class — but the paper neither discusses these nor provides the error analysis that its diagnostic-dataset framing (Section 3) promises. The authors should document the full protocol and add an error analysis that accounts for the zeros.
  4. [§3.2.2] The validity of all downstream scores rests on S2ORC's in-text citation annotations linking each citation mention to the cited paper's corpus ID, but no quality check is reported. The paper itself documents malformed citations such as "(Lee et al., 1997a(Lee et al., 1997b)", and the only manual validation mentioned is for the POS dataset (§3.1.5) plus a general reference to manual checking in Section 5, with no agreement or error statistics. The authors should report the fraction of candidate sentences rejected by the citation parser, the rate at which S2ORC annotations were found to be incorrect, and any manual validation results, since a noisy mapping would propagate into every dataset and every score in Table 1.
minor comments (7)
  1. [§3] Throughout Section 3 the text mixes future and past tense, e.g., "The position will then be normalized" in Section 3.1.2 and "a dataset will be created" in Section 3.1.5, which is confusing given that Section 4 reports completed experiments; the manuscript should be rewritten in a consistent tense that reflects what was actually done.
  2. [Table 1] In the Citation Count 8 row for Galactica 6.7b, the entry "0.0363 00170" appears to have a missing decimal point and should presumably read "0.0363 0.0170".
  3. [References] Reference [3] is misattributed; the author name "Bridger Waleed Ammar" should be "Waleed Ammar".
  4. [§3.1.1 and §3.1.2] The class boundaries for context length (short/medium/long) and citation location (first/middle/last) are never stated; the paper should report the actual token ranges and normalized-position cutoffs derived from the mean and standard deviation.
  5. [§4] The sentence "Additionally, BM25 achieves higher MRR across all classes and for BM25, except for the preceding Preposition and Others categories" is unclear about which model's POS classes are meant.
  6. [§4] The explanation that Galactica's weaker performance on citations at the start or middle of a sentence arises from "the Transformer model's decoder, which retains information about the preceding context before the masked token" is not an accurate description of a decoder-only causal language model and should be revised.
  7. [§6] The Conclusion's claim that a comprehensive benchmark is "established" is undercut by the Limitations section's own admissions of small dataset sizes, a limited metric set, incomplete model coverage, and the absence of a standard evaluation framework; the conclusion should be qualified accordingly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark is assembled from external corpora and the model evaluations are independent of the dataset construction.

full rationale

The paper's central claim is that a benchmark with diagnostic datasets has been established. The datasets are built from S2ORC and S2AG by stratified sampling of cited papers and extracting citing sentences that contain exactly one citation (Sections 3.2.1 and 3.2.2). No model parameter is fitted to the benchmark, no prediction is derived from the benchmark's own outputs, and no load-bearing premise is justified by a self-citation: the author cites no prior work of her own. The evaluation in Section 4 compares externally trained models (NCN, LCR, Galactica) and a BM25 baseline on the constructed datasets; the reported scores are empirical measurements over held-out style tasks, not quantities that are equal to the dataset definitions by construction. The only notable concern is that BM25 'has the advantage of querying the entire S2AG dataset' while the candidate pools for the neural models are not reported, which makes Recall@10 and MRR@10 comparisons inequitable. That is a correctness or validity limitation, not circular reasoning: the benchmark's construction does not assume the conclusion that BM25 is best, and the asymmetry does not make the comparison true by definition. The limitation section itself acknowledges coverage and evaluation-metric gaps, further indicating that the contribution is an open empirical resource rather than a closed self-referential derivation.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper introduces diagnostic datasets as artifacts, but no theoretical entities such as particles, mediators, or forces. The central assumptions are about data annotation quality and the adequacy of single-sentence contexts, which are listed in the axioms.

free parameters (6)
  • z-score outlier threshold = 3
    Used in Section 3.1.1 to remove sentence-length outliers before defining context length classes; chosen by hand.
  • low-resource field threshold = 3%
    Defined in Section 3.1.4 and used in Section 3.2.1 step 1 to decide which fields are low-resource.
  • citation count lower bound for sampled cited papers = 5
    Section 3.2.1 step 3 starts sampling at citation count 5 to ensure cited papers have enough citing contexts.
  • sampling upper limit per class group = 10
    Section 3.2.1 step 4 sets an upper limit of 10 sampled papers per group though only 3 records may be required.
  • year bins = 1996-2000, 2001-2005, 2006-2010, 2011-2015, 2016-2020
    Five-year groups chosen in Section 3.2.1 step 2 based on record availability.
  • context length class boundaries (short, medium, long)
    Section 3.1.1 says boundaries come from mean and standard deviation, but the actual token thresholds are not reported, which blocks exact replication.
assumptions (4)
  • domain assumption S2ORC and S2AG annotations are accurate enough to serve as ground truth for citations and metadata.
    Section 3 relies on S2ORC bibliography references and S2AG metadata without reporting validation, yet all diagnostic datasets inherit this dependency.
  • domain assumption A single sentence with one citation is a valid unit for citation recommendation evaluation.
    Section 3.2.2 filters sentences to one citation and replaces it with <REF>, but no evidence is given that such sentences contain enough information to identify the correct cited paper.
  • domain assumption Stratified sampling by field, year, and citation count yields diagnostic datasets that are representative and balanced.
    Section 3.2.1 assumes the strata capture the variability needed for fair comparison, yet no dataset statistics are reported to verify balance.
  • domain assumption Excluding fields with less than 3% of papers, except for the low-resource set, avoids skew while preserving enough data.
    This design choice in Section 3.2.1 step 1 is not analyzed for its effect on generalizability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmark for Evaluation and Analysis of Citation Recommendation Models." pith.science (2026). https://pith.science/paper/MKN2UQPS

@misc{pith2026241207713,
  author       = {Pith},
  title        = {Pith review of: Benchmark for Evaluation and Analysis of Citation Recommendation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MKN2UQPS}},
  note         = {Machine review of arXiv:2412.07713}
}
read the original abstract

Citation recommendation systems have attracted much academic interest, resulting in many studies and implementations. These systems help authors automatically generate proper citations by suggesting relevant references based on the text they have written. However, the methods used in citation recommendation differ across various studies and implementations. Some approaches focus on the overall content of papers, while others consider the context of the citation text. Additionally, the datasets used in these studies include different aspects of papers, such as metadata, citation context, or even the full text of the paper in various formats and structures. The diversity in models, datasets, and evaluation metrics makes it challenging to assess and compare citation recommendation methods effectively. To address this issue, a standardized dataset and evaluation metrics are needed to evaluate these models consistently. Therefore, we propose developing a benchmark specifically designed to analyze and compare citation recommendation models. This benchmark will evaluate the performance of models on different features of the citation context and provide a comprehensive evaluation of the models across all these tasks, presenting the results in a standardized way. By creating a benchmark with standardized evaluation metrics, researchers and practitioners in the field of citation recommendation will have a common platform to assess and compare different models. This will enable meaningful comparisons and help identify promising approaches for further research and development in the field.

Figures

Figures reproduced from arXiv: 2412.07713 by the authors.

Figure 1
Figure 1. Citation position in the context sentences. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Combined preceding POS of the citation. The combined POS tags in the dataset are utilized. The exact process is repeated for the following POS as well. The distribution is examined to identify common POS patterns. Subsequently, these POS categories are utilized to differentiate the POS tags of the preceding and following words. Classes are created based on the preceding and following POS tags, with sub-classes for t… view at source ↗
Figure 4
Figure 4. Data distribution of papers, from years 2000 to 2023 [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Citation count distribution based on fields, where [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Process of data selection Our sampling strategy encompasses a sequence of steps and implements a hierarchical approach of stratified sampling similar to the one described in the work by Trost (1986) [38]. This hierarchical strategy comprises multiple tiers: initially, …
Figure 7
Figure 7. Figure 7: Process of paper selection papers is retrieved from the S2ORC dataset. The data is then saved in different collections in the database. 3.2.2 Sentence Extraction. We will follow the steps defined below for the extraction of the citing sentences, as shown in [PITH_FULL…
Figure 8
Figure 8. Figure 8: Process of sentence extraction (c) Context Type: The length of the citing sentence belongs to the class "medium" and the reference is at the "end" of the sentence (d) POS of Surrounding Words: The length of the citing sen￾tence belongs to the class "medium" and the ref…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 36 canonical work pages

  1. [1]

    Zafar Ali, Guilin Qi, Pavlos Kefalas, Shah Khusro, Inayat Khan, and Khan Muham- mad. 2022. SPR-SMN: Scientific paper recommendation employing SPECTER with memory network. Scientometrics 127, 11 (2022), 6763–6785

    work page 2022

  2. [2]

    Zafar Ali, Irfan Ullah, Amin Khan, Asim Ullah Jan, and Khan Muhammad. 2021. An overview and evaluation of citation recommendation models. Scientometrics 126, 5 (2021), 4083–4119

    work page 2021

  3. [3]

    Ha, Rodney Michael Kinney, Sebastian Kohlmeier, Kyle Lo, Tyler C

    Bridger Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu A. Ha, Rodney Michael Kinney, Sebastian Kohlmeier, Kyle Lo, Tyler C. Murray, Hsu-Han Ooi, Matthew E. Peters, Joanna L. Power, Sam Skjonsberg, Lucy Lu Wang, Christopher Wilhelm, Zheng Yuan, Madeleine va...

    work page 2018

  4. [4]

    MA Angrosh, Stephen Cranefield, and Nigel Stanger. 2010. Context identification of sentences in related work sections using a conditional random field: towards intelligent digital libraries. In Proceedings of the 10th annual joint conference on Digital libraries. 293–302

    work page 2010

  5. [5]

    Steven Bethard and Dan Jurafsky. 2010. Who should I cite: learning literature search models from citation behavior. InProceedings of the 19th ACM international conference on Information and knowledge management . 609–618

    work page 2010

  6. [6]

    Chandra Bhagavatula, Sergey Feldman, Russell Power, and Bridger Waleed Am- mar. 2018. Content-Based Citation Recommendation. ArXiv abs/1802.08301 (2018). https://api.semanticscholar.org/CorpusID:3536005

    arXiv 2018

  7. [7]

    Steven Bird, Robert Dale, Bonnie J Dorr, Bryan R Gibson, Mark Thomas Joseph, Min-Yen Kan, Dongwon Lee, Brett Powley, Dragomir R Radev, Yee Fan Tan, et al. 2008. The ACL Anthology Reference Corpus: A Reference Dataset for Bibliographic Research in Computational Linguistics.. In LREC

    work page 2008

  8. [8]

    Kevin W Boyack, Nees Jan van Eck, Giovanni Colavizza, and Ludo Waltman

    work page

Show all 49 references
  1. [9]

    Xiaoyan Cai, Junwei Han, Wenjie Li, Renxian Zhang, Shirui Pan, and Libin Yang. 2018. A three-layered mutually reinforced model for personalized citation recommendation. IEEE transactions on neural networks and learning systems 29, 12 (2018), 6026–6037

    2018

  2. [10]

    Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld

  3. [11]

    Gritsenko, Zhe Zhao, Neil Houlsby, Fer- nando Diaz, Donald Metzler, and Oriol Vinyals

    Mostafa Dehghani, Yi Tay, Alexey A. Gritsenko, Zhe Zhao, Neil Houlsby, Fer- nando Diaz, Donald Metzler, and Oriol Vinyals. 2021. The Benchmark Lot- tery. ArXiv abs/2107.07002 (2021). https://api.semanticscholar.org/CorpusID: 235810239

    2021 arXiv

  4. [12]

    Travis Ebesu and Yi Fang. 2017. Neural citation network for context-aware citation recommendation. In Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval . 1093–1096

    2017

  5. [13]

    Michael Färber and Adam Jatowt. 2020. Citation recommendation: approaches and datasets. International Journal on Digital Libraries 21, 4 (2020), 375–405

    2020

  6. [14]

    Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna Clinciu, Dipanjan Das, Kaustubh D

    Sebastian Gehrmann, Tosin P. Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna Clinciu, Dipanjan Das, Kaustubh D. Dhole, Wanyu Du, Esin Durmus, Ondrej Dusek, Chris C. Emezue, Varun Gangal, Cristina Garbac...

    2021 arXiv

  7. [15]

    C Lee Giles, Kurt D Bollacker, and Steve Lawrence. 1998. CiteSeer: An automatic citation indexing system. In Proceedings of the third ACM conference on Digital libraries. 89–98

    1998

  8. [16]

    Nianlong Gu, Yingqiang Gao, and Richard HR Hahnloser. 2022. Local citation recommendation with hierarchical-attention text encoder and SciBERT-based reranking. In Advances in Information Retrieval: 44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10...

    2022

  9. [17]

    Qi He, Jian Pei, Daniel Kifer, Prasenjit Mitra, and Lee Giles. 2010. Context-aware citation recommendation. In Proceedings of the 19th international conference on World wide web. 421–430

    2010

  10. [18]

    Wenyi Huang, Zhaohui Wu, Chen Liang, Prasenjit Mitra, and C Giles. 2015. A neural probabilistic model for context based citation recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 29

    2015

  11. [19]

    Wenyi Huang, Zhaohui Wu, Prasenjit Mitra, and C Lee Giles. 2014. Refseer: A citation recommendation system. In IEEE/ACM joint conference on digital libraries. IEEE, 371–374

    2014

  12. [20]

    Chanwoo Jeong, Sion Jang, Eunjeong Park, and Sungchul Choi. 2020. A context- aware citation recommendation model with BERT and graph convolutional networks. Scientometrics 124 (2020), 1907–1922

    2020

  13. [21]

    Graham, F.Q

    Rodney Michael Kinney, Chloe Anastasiades, Russell Authur, Iz Beltagy, Jonathan Bragg, Alexandra Buraczynski, Isabel Cachola, Stefan Candra, Yoganand Chan- drasekhar, Arman Cohan, Miles Crawford, Doug Downey, Jason Dunkelberger, Oren Etzioni, Rob Evans, Sergey Feldman, Joseph ...

    2023 arXiv

  14. [22]

    Petr Knoth and Zdenek Zdrahal. 2012. CORE: three access levels to underpin open access. D-Lib Magazine 18, 11/12 (2012), 1–13

    2012

  15. [23]

    Hyeyoung Ko, Suyeon Lee, Yoonseo Park, and Anna Choi. 2022. A survey of recommendation systems: recommendation models, techniques, and application fields. Electronics 11, 1 (2022), 141

    2022

  16. [24]

    Suchetha N Kunnath, David Pride, Bikash Gyawali, and Petr Knoth. 2020. Overview of the 2020 WOSP 3C citation context classification task. InProceedings of the 8th International Workshop on Mining Scientific Publications . Association for Computational Linguistics, 75–83

    2020

  17. [25]

    Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Michael Kinney, and Daniel S. Weld. 2020. S2ORC: The Semantic Scholar Open Research Corpus. InAnnual Meet- ing of the Association for Computational Linguistics . https://api.semanticscholar. org/CorpusID:215416146

    2020

  18. [26]

    Yonghe Lu, Meilu Yuan, Jiaxin Liu, and Minghong Chen. 2023. Research on semantic representation and citation recommendation of scientific papers with multiple semantics fusion. Scientometrics 128, 2 (2023), 1367–1393

    2023

  19. [27]

    Zoran Medić and Jan Šnajder. 2020. Improved local citation recommendation based on context enhanced with global information. In Proceedings of the First Workshop on Scholarly Document Processing . 97–103. 9

    2020

  20. [28]

    Rodrigo Nogueira, Zhiying Jiang, Kyunghyun Cho, and Jimmy Lin. 2020. Navigation-based candidate expansion and pretrained language models for cita- tion recommendation. Scientometrics 125, 3 (2020), 3001–3016

    2020

  21. [29]

    Xiang Ren, Jialu Liu, Xiao Yu, Urvashi Khandelwal, Quanquan Gu, Lidan Wang, and Jiawei Han. 2014. Cluscite: Effective citation recommendation by information network-based clustering. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data...

    2014

  22. [30]

    Anna Ritchie, Stephen Robertson, and Simone Teufel. 2008. Comparing citation contexts for information retrieval. In Proceedings of the 17th ACM conference on Information and knowledge management . 213–222

    2008

  23. [31]

    Tarek Saier and Michael Färber. 2019. Bibliometric-Enhanced arXiv: A Data Set for Paper-Based and Citation-Based Tasks.. In BIR@ ECIR. 14–26

    2019

  24. [32]

    Tarek Saier and Michael Färber. 2020. unarXive: a large scholarly data set with publications’ full-text, annotated in-text citations, and links to metadata. Sciento- metrics 125, 3 (2020), 3085–3108

    2020

  25. [33]

    Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. 2008. Ar- netminer: extraction and mining of academic social networks. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining. 990–998

    2008

  26. [34]

    Zehra Taşkın and Umut Al. 2018. A content-based citation analysis study based on text categorization. Scientometrics 114, 1 (2018), 335–357

    2018

  27. [35]

    Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic

    Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony S. Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic

  28. [36]

    Sonita Te, Amira Barhoumi, Martin Lentschat, Frédérique Bordignon, Cyril Labbé, and François Portet. 2022. Citation Context Classification: Critical vs Non-critical. In Proceedings of the Third Workshop on Scholarly Document Processing . 49–53

    2022

  29. [37]

    Nimbeshaho Thierry, Bing-Kun Bao, and Zafar Ali. 2023. RAR-SB: research article recommendation using SciBERT with BiGRU. Scientometrics 128, 12 (2023), 6427–6448

    2023

  30. [38]

    Jan E Trost. 1986. Statistically nonrepresentative stratified sampling: A sampling technique for qualitative studies. Qualitative sociology 9, 1 (1986), 54–57

    1986

  31. [39]

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier bench- mark for general-purpose language understanding systems. Advances in neural information processing systems 32 (2019)

    2019

  32. [40]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. GLUE: A Multi-Task Benchmark and Analysis Plat- form for Natural Language Understanding. In BlackboxNLP@EMNLP. https: //api.semanticscholar.org/CorpusID:5034059

    2018

  33. [41]

    Leipeng Wang, Yuan Rao, Qinyu Bian, and Shuo Wang. 2020. Content-based hybrid deep neural network citation recommendation method. In International conference of pioneering computer scientists, engineers and educators . Springer, 3–20

    2020

  34. [42]

    Mingyang Wang, Jiaqi Zhang, Shijia Jiao, Xiangrong Zhang, Na Zhu, and Guang- sheng Chen. 2020. Important citation identification by exploiting the syntactic and contextual information of citations. Scientometrics 125 (2020), 2109–2129

    2020

  35. [43]

    Dustin Wright and Isabelle Augenstein. 2021. CiteWorth: Cite-Worthiness De- tection for Improved Scientific Document Understanding. In Findings. https: //api.semanticscholar.org/CorpusID:235166702

    2021

  36. [44]

    Libin Yang, Yu Zheng, Xiaoyan Cai, Hang Dai, Dejun Mu, Lantian Guo, and Tao Dai. 2018. A LSTM based model for personalized context-aware citation recommendation. IEEE access 6 (2018), 59618–59627

    2018

  37. [45]

    Jun Yin and Xiaoming Li. 2017. Personalized citation recommendation via convo- lutional neural networks. InWeb and Big Data: First International Joint Conference, APWeb-W AIM 2017, Beijing, China, July 7–9, 2017, Proceedings, Part II 1 . Springer, 285–293

    2017

  38. [46]

    Xiaodan Zhu, Peter Turney, Daniel Lemire, and André Vellino. 2015. Measuring academic influence: Not all citations are equal. Journal of the Association for Information Science and Technology 66, 2 (2015), 408–427. 10

    2015

  39. [2018]

    Journal of Informetrics 12, 1 (2018), 59–73

    Characterizing in-text citations in scientific articles: A large-scale analysis. Journal of Informetrics 12, 1 (2018), 59–73

    2018

  40. [2020]

    ArXiv abs/2004.07180 (2020)

    SPECTER: Document-level Representation Learning using Citation- informed Transformers. ArXiv abs/2004.07180 (2020). https://api.semanticscholar. org/CorpusID:215768677

    2020 arXiv

  41. [2022]

    ArXiv abs/2211.09085 (2022)

    Galactica: A Large Language Model for Science. ArXiv abs/2211.09085 (2022). https://api.semanticscholar.org/CorpusID:253553203

    2022 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.