Pith. sign in

REVIEW 2 major objections 5 minor 3 references

The changing role of cited papers over time: An analysis of highly cited papers based on a large full-text dataset

T0 review · 2 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a highly cited paper's role shifts systematically as it ages: it is cited earlier in the citing text, mentioned fewer times, more often bundled with other references, and becomes less topically and bibliographically r

desk verdict A careful descriptive study whose central claim—that HCPs shift to symbolic referencing as they age—is not actually identified, because aging is perfectly collinear with global citation-practice changes over the same period. read the letter →

arxiv 2509.04190 v1 pith:S5JBEYDI submitted 2025-09-04 cs.DL

Gege Lin , Nees Jan van Eck , Haiyan Hou , Zhigang Hu This is my paper · ORCID
classification cs.DL
keywords citationanalysiscontextfull-texthighlycitedpaperssentimentfunctionpublicationrelatednessresearchevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks what happens to a famous paper's role as it grows older. Following 883 papers published in 2000 that each accumulated at least 1,000 citations, the authors examine the full text of more than 220,000 later papers that cite them. They find a consistent drift: citations to these aging papers migrate toward the opening sections of the citing text, each paper is mentioned fewer times, and citations increasingly appear in clusters with other references. Textual and bibliographic overlap between the cited paper and its citing papers also declines. Together these shifts support the paper's central claim — that highly cited works move from direct topical and methodological engagement toward general, background, and symbolic referencing — and they matter because conventional citation counts treat every citation as equal.

What carries the argument

The mechanism is large-scale full-text citation parsing. Each citing paper is decomposed into references, reference mentions, in-text citations, and citation sentences, and every citation to a target paper is assigned a location on a normalized text-progression scale from 0 to 1. Five characteristics are then measured per citing year: citation location (begin/middle/end of text), whether the reference is mentioned once or multiple times (single- vs. multiple-mentioned references), whether the in-text citation stands alone or bundles several references (single- vs. multi-reference citations), the sentiment of the citation sentence measured by a lexicon-based sentiment tool, and relatedness me

What would settle it

Measure the same five characteristics for a matched control group of papers published in 2000 that received few citations rather than thousands. If this control group shows the same drift — citations moving to the opening sections, fewer mentions per paper, more multi-reference bundles, declining textual and bibliographic relatedness — then the pattern tracks changes in citing conventions over time, not the aging of celebrated papers.

Watch

Extended reading notes

Core claim

The central claim is that the function of a highly cited paper changes systematically as the paper ages, and that the change is visible in the full text of the papers citing it. For a cohort of 883 papers published in 2000 with at least 1,000 citations each, the authors track five citation characteristics across citing years 2000–2016. The share of citations in the opening third of citing papers rises from about 36% to 56%, while the share in the closing third falls from about 37% to 19%; the average number of mentions per citing paper drops from 1.8 to 1.4; the share of multi-reference citations grows from 32.7% to 45.7%; and both textual similarity (title/abstract embeddings) and shared-re

Load-bearing premise

The paper assumes the temporal trends it observes are caused by the aging of the cited papers, yet it studies a single cohort without a comparison group; if citation practices across all of science changed over the same 2000–2016 window, those global changes alone could produce the same patterns.

Editorial extensions

If this is right

  • Citation counts are a flattening measure: for old classics they increasingly count acts of recognition rather than acts of use, so equal-weight counting systematically overstates the current role of aged highly cited papers.
  • Impact metrics become more truthful if differentiated by citation context — a repeated mention in the methods section should outweigh a single bundled mention in the introduction.
  • Science mapping and scholarly search can be improved by weighting links by textual or bibliographic relatedness, since the paper shows relatedness declines with age and thereby separates intellectual debt from mere acknowledgment.
  • Full-text sources become a practical necessity for credible citation analysis; reference lists alone cannot reveal the shift the paper documents.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The design compares one cohort across time rather than treated versus untreated papers, so an untested alternative reading is that global changes in citing practice between 2000 and 2016 — more references per paper, more bundled citations — produced the observed drift independently of any paper's age. A control group of ordinary papers published in 2000 would settle which reading is right.
  • If symbolic citation is real, then the same measurements could detect 'awakenings' — old papers that regain substantive engagement — as rebounds in mention frequency or relatedness that raw citation counts hide.
  • The slight rise in citation-sentence positivity (compound score from about 0.04 to 0.11) invites a sharper question: whether prestige citations carry more praise in passing than work cited for its actual content, which would let sentiment separate ceremonial from substantive citation.
  • A testable prediction follows: extending the same cohort beyond 2016 should show the begin-part share, multi-reference share, and relatedness declines continuing but decelerating, as the cohort settles into a stable symbolic phase.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper analyzes how the role of highly cited papers (HCPs) changes as they age, using a cohort of 883 HCPs all published in 2000 and the full text of 220,335 citing papers from the Elsevier ScienceDirect corpus (2000-2016). It examines five citation characteristics: location in the citing text, number of mentions (SMR/MMR), in-text citation type (SRC/MRC), citation sentiment, and textual/bibliographic relatedness. The authors report that as HCPs age they are cited earlier in papers, mentioned fewer times, more frequently cited together with other references, and become less textually and bibliographically related to citing papers. They interpret these patterns as a shift from direct topical and methodological engagement toward more general, background, and symbolic referencing, invoking Small's (1978) concept of 'concept symbols.' The paper shares data and code via Zenodo and GitHub. The central descriptive findings are clearly computed, but the core interpretative claim is threatened by a time-trend confound: because all HCPs are from 2000, citation age is perfectly collinear with citing year, and no control group of non-HCP references is used.

Significance. If the 'symbolic referencing' interpretation were supportable, the paper would be a valuable contribution to full-text citation analysis and to the literature on the changing function of highly cited works. The study is built on an unusually large full-text corpus (220k+ papers) and covers multiple citation dimensions simultaneously, which is a strength. The authors also provide open data and code, enhancing reproducibility. However, the central claim about aging-driven shifts is underdetermined by the presented evidence, since global temporal changes in citation practices—which the authors themselves cite via Boyack et al. (2018)—could produce the same trends. The paper is therefore best viewed as a descriptive account of temporal trends in how HCPs are cited, rather than a demonstrated aging effect. With additional controls or a re-framed interpretation, the study could still make a meaningful contribution.

major comments (2)
  1. [Section 4, Figures 2-11] The central interpretation that HCPs become 'background/symbolic' references as they age is confounded by global citation-practice changes. Because all HCPs were published in 2000, citation age equals citing year minus 2000, making age and temporal trends in citing practices perfectly collinear. The paper cites Boyack et al. (2018) showing that reference counts and in-text citation counts increased over 2000-2016, but it never uses a control group of non-HCP references. This matters mechanically: longer reference lists make multi-reference citations (MRC share, Fig. 6: 32.7% to 45.7%) more likely; overall trends in referencing density could also affect average mention counts (Fig. 5: 1.8 to 1.4) and relatedness measures (Figs. 10-11). Without a non-HCP control or an alternative identification strategy, the observed trends cannot be attributed to aging. The limitations paragraph (Section
  2. [Abstract and Section 5] The claim 'These patterns indicate a shift from direct topical and methodological engagement toward more general, background, and symbolic referencing' is the paper's central conclusion, but it is a causal interpretation of what are purely descriptive correlations with citing year. Even if the trends are robust, they could reflect changes in how all papers are cited over time (e.g., increasing reference list length, IMRaD conventions) rather than a special aging trajectory of HCPs. The conclusion should either be substantially softened to 'temporal changes in citation patterns' or the authors must provide a control comparison that isolates the age effect.
minor comments (5)
  1. [Abstract] The abstract states 'nearly 900 highly cited papers (HCPs) published between 2000 and 2016', but Section 3 clarifies that HCPs were all published in 2000; the 2000-2016 range applies to the citing papers. Please rephrase to avoid ambiguity.
  2. [Section 3 vs Section 5] The full-text coverage is reported as 20.9% in Section 3 ('accounting for 20.9% of the total number of citing papers') but as 14.3% in the limitations paragraph of Section 5. Reconcile these numbers.
  3. [Section 4.2] Typo: 'has decreasing from 1.8 in 2000 to 1.4 in 2016' should be 'has decreased'. Also, minor grammar issues appear elsewhere (e.g., 'Ding et. al' in Section 2.1).
  4. [Section 4.4] VADER is designed for social-media-style text. Its suitability for scientific citation sentences is not justified. A brief validation or acknowledgment of this limitation would strengthen the sentiment analysis, although the relative trends are probably robust.
  5. [Section 4.5] The description of textual relatedness is underspecified: it is not clear whether SciBERT was applied to title+abstract concatenated or separately, and whether the [CLS] token or mean-pooled embeddings were used. Please clarify.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: all reported trends are directly measured from full-text data; conclusions are interpretive, not built into the measures.

full rationale

The paper's central findings—earlier citation location, fewer mentions, more multi-reference citations, stable neutral sentiment, and declining relatedness—are computed directly from parsed full-text data, not derived from fitted parameters, self-defined equivalences, or citations to the authors' prior work. Each measure is operationally defined (text progression i/n, SMR/MMR, MRC/SRC, VADER sentiment scores, SciBERT cosine similarity, Ochiai bibliographic coupling) and then plotted over citing years; the 'shift toward background/symbolic referencing' is an interpretive summary of those observed trends, not an equation that reduces to the definitions. The paper does cite prior work by the authors (e.g., Hu et al. 2017 for MMR, Lin et al. 2019 for MRC, Boyack et al. 2018 for the Elsevier corpus), but these citations supply terminology, data provenance, or background facts about global citation inflation; they do not by themselves generate the HCP-specific temporal patterns. The aging-vs-citing-year confound noted by a skeptical reader is a real threat to causal interpretation, but it is an identification/validity issue, not circularity: the descriptive trends are still independent observations. The limitations section acknowledges data coverage issues but omits this confound, which is a completeness or correctness concern, not a circular-derivation concern. The analysis is self-contained against the data, and no prediction is produced by fitting a parameter to a subset of the data then reporting it as a finding.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities or fitted parameters. Its conclusions rest on domain assumptions about the validity of established measurement tools (VADER, SciBERT) and on the representativeness of the Elsevier corpus, plus the interpretation that MMRs are more essential, all drawn from prior literature.

assumptions (4)
  • domain assumption Multiple mentions (MMR) indicate more essential references than single mentions (SMR)
    Paper cites Ding et al. (2013) and Hu et al. (2017) for this interpretation; it is used to conclude that declining MMR implies less essential engagement.
  • domain assumption VADER sentiment scores, developed for social media text, transfer to scientific citation sentences
    The paper uses VADER without validating it on scientific text; this enters in Section 4.4.
  • domain assumption SciBERT embeddings of titles/abstracts capture topical relatedness
    Used in Section 4.5 to measure textual relatedness; no validation of this assumption for this task is reported.
  • domain assumption The Elsevier full-text subset is representative of all citing papers
    The paper argues consistency of ~20% across years, but does not check disciplinary or journal-level representativeness; this affects all temporal trends.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The changing role of cited papers over time: An analysis of highly cited papers based on a large full-text dataset." pith.science (2026). https://pith.science/paper/S5JBEYDI

@misc{pith2026250904190,
  author       = {Pith},
  title        = {Pith review of: The changing role of cited papers over time: An analysis of highly cited papers based on a large full-text dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S5JBEYDI}},
  note         = {Machine review of arXiv:2509.04190}
}
read the original abstract

This paper examines how the role of cited papers evolves over time by analyzing nearly 900 highly cited papers (HCPs) published between 2000 and 2016 and the full text of over 220,000 papers citing them. We investigate multiple citation characteristics, including citation location within the full text, reference and in-text citation types, citation sentiment, and textual and bibliographic relatedness between citing and cited papers. Our findings reveal that as HCPs age, they tend to be cited earlier in papers citing them, mentioned fewer times in the full text, and more often cited alongside other references. Citation sentiment remains predominantly neutral, while both textual and bibliographic similarity between HCPs and their citing papers decline over time. These patterns indicate a shift from direct topical and methodological engagement toward more general, background, and symbolic referencing. The findings highlight the importance to consider citation context rather than relying solely on simple citation counts. Large-scale full-text analyses such as ours can help refine measures of scientific impact and advance scholarly search and science mapping by uncovering more nuanced connections between papers.

Figures

Figures reproduced from arXiv: 2509.04190 by the authors.

Figure 1
Figure 1. Number of citing papers and percentage of citing papers with a full text in different [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Average location of citations to HCPs within the full text of citing papers across [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Percentage of citations to HCPs located in the begin part, middle part, and end part of [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Percentage of references to HCPs that are mentioned only once (SMR) or multiple [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Average number of mentions of references to HCPs within the full text of citing papers [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Percentage of in-text citations to HCPs that include only one reference (SRC) or multiple references (MRC) across different citing years [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Average number of references in in-text citations to HCPs across different citing years [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Proportion of the text of sentences containing citations to HCPs associated with a [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Average sentiment compound score of the text of sentences containing citations to [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Average textual relatedness between HCPs and their citing papers. [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Average reference relatedness between HCPs and their citing papers. [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

3 extracted references · 3 canonical work pages

  1. [340]

    https://doi.org/10.1177/030631277800800305 Tahamtan, I., & Bornmann, L. (2019). What do citation counts measure? An updated review of studies on citations in scientific documents published between 2006 and 2018. Scientometrics, 121(3), 1635-1684. https://doi.org/10.1007/s11192-019-03243-4 Thelwall, M. (2019a). Should citations be counted separately from e...

    work page Pith review arXiv doi:10.48550/arxiv.2005.04534 2019

  2. [427]

    https://doi.org/10.1002/asi.23179

    work page doi:10.1002/asi.23179

  3. [5052]

    Highly cited references in PLOS ONE and their in-text usage over time

    https://doi.org/10.1007/s11192-022-04437-z Gross, P . L., & Gross, E. M. (1927). College libraries and chemical education. Science, 66(1713), 385–389. https://doi.org/10.1126/science.66.1713.385 Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National academy of Sciences , 102(46), 16569 –16572. ht...

    work page Pith review arXiv doi:10.48550/arxiv.1903.11693 1927

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.