{"id":"7a500aea-0d7f-4a71-87da-37c38591a60b","arxiv_id":"2509.04190","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"As highly cited papers age, they are cited earlier and more briefly and become less related to the citing papers, consistent with a shift toward symbolic referencing.","lead":"This paper tracks how nearly 900 highly cited papers are cited in 220,000 full-text papers over 17 years, finding that older papers are cited earlier, more briefly, and as background rather than deep engagement. It matters because it quantifies on a large scale how citation counts can overstate the current relevance of famous older papers, though the trend may partly reflect general changes in writing style.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Aging effect is unidentified: all trends are collinear with citing-year global citation-practice changes; no non-HCP control is used.","rationale":"I read the paper as a careful descriptive analysis with valuable full-text data and openly shared code and statistics. The strongest claim, however, is causal-interpretive: it asserts that as HCPs age their citation role shifts toward symbolic referencing. The data alone, plotting metrics against citing year for a single 2000 cohort, cannot separate age from period effects. The reader's weakest assumption identifies exactly this confound, and I find it load-bearing. The paper acknowledges global citation-practice changes only in the literature review, not as a control or robustness check. A non-HCP reference control or multiple publication cohorts would settle the issue. I also note that the abstract says HCPs were published between 2000 and 2016 while the methods section says all 883 were published in 2000; this inconsistency should be corrected but is secondary. Since the reader already issued a conditional verdict based on the same concern, I recommend no change to that verdict.","tokens_in":13498,"tokens_out":3814,"duration_ms":41377,"concrete_test":"Build a matched control set from the same 220,335 citing papers: for each HCP citation, sample one non-HCP reference also published in 2000 from the same citing paper (or match on field). Recompute Figures 2-11 for this control set across citing years. If the control shows the same upward trend in begin-part share and MRC share and the same declines in mentions and relatedness, the HCP-aging interpretation collapses; if control trends are flat while HCP trends move, the confound is refuted. Equivalently, rerun the analysis on a second HCP cohort from 2005 and compare metrics at equal citation ages; discrepancies at equal ages would expose citing-year contamination.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central interpretative claim—that older HCPs shift from direct engagement to background/symbolic citation—rests entirely on trends in Figures 2-11 plotted against citing year. Because every HCP was published in 2000, citation age = citing year − 2000, so 'aging' and 'global changes in citing practices between 2000 and 2016' are perfectly collinear. The paper itself cites Boyack et al. (2018) showing that reference counts and in-text citation counts increased over this period, but it never uses non-HCP references as a control. This matters mechanically: the MRC share increase (32.7%→45.7%) could largely be a consequence of longer reference lists making co-citation in a single parenthesis more likely. Similarly, earlier average citation location, fewer mentions, and lower title/abstract similarity could reflect changing IMRaD conventions, reference inflation, and shifting citation norms rather than the aging of these particular papers. The paper's own limitations section does not acknowledge this confound. Thus the descriptive patterns are informative, but the 'shift toward symbolic referencing' conclusion is underdetermined.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper analyzes how the role of highly cited papers (HCPs) changes as they age, using a cohort of 883 HCPs all published in 2000 and the full text of 220,335 citing papers from the Elsevier ScienceDirect corpus (2000-2016). It examines five citation characteristics: location in the citing text, number of mentions (SMR/MMR), in-text citation type (SRC/MRC), citation sentiment, and textual/bibliographic relatedness. The authors report that as HCPs age they are cited earlier in papers, mentioned fewer times, more frequently cited together with other references, and become less textually and bibliographically related to citing papers. They interpret these patterns as a shift from direct topical and methodological engagement toward more general, background, and symbolic referencing, invoking Small's (1978) concept of 'concept symbols.' The paper shares data and code via Zenodo and GitHub. The central descriptive findings are clearly computed, but the core interpretative claim is threatened by a time-trend confound: because all HCPs are from 2000, citation age is perfectly collinear with citing year, and no control group of non-HCP references is used.","tokens_in":13779,"tokens_out":3215,"duration_ms":30887,"significance":"If the 'symbolic referencing' interpretation were supportable, the paper would be a valuable contribution to full-text citation analysis and to the literature on the changing function of highly cited works. The study is built on an unusually large full-text corpus (220k+ papers) and covers multiple citation dimensions simultaneously, which is a strength. The authors also provide open data and code, enhancing reproducibility. However, the central claim about aging-driven shifts is underdetermined by the presented evidence, since global temporal changes in citation practices—which the authors themselves cite via Boyack et al. (2018)—could produce the same trends. The paper is therefore best viewed as a descriptive account of temporal trends in how HCPs are cited, rather than a demonstrated aging effect. With additional controls or a re-framed interpretation, the study could still make a meaningful contribution.","major_comments":[{"comment":"The central interpretation that HCPs become 'background/symbolic' references as they age is confounded by global citation-practice changes. Because all HCPs were published in 2000, citation age equals citing year minus 2000, making age and temporal trends in citing practices perfectly collinear. The paper cites Boyack et al. (2018) showing that reference counts and in-text citation counts increased over 2000-2016, but it never uses a control group of non-HCP references. This matters mechanically: longer reference lists make multi-reference citations (MRC share, Fig. 6: 32.7% to 45.7%) more likely; overall trends in referencing density could also affect average mention counts (Fig. 5: 1.8 to 1.4) and relatedness measures (Figs. 10-11). Without a non-HCP control or an alternative identification strategy, the observed trends cannot be attributed to aging. The limitations paragraph (Section","section":"Section 4, Figures 2-11"},{"comment":"The claim 'These patterns indicate a shift from direct topical and methodological engagement toward more general, background, and symbolic referencing' is the paper's central conclusion, but it is a causal interpretation of what are purely descriptive correlations with citing year. Even if the trends are robust, they could reflect changes in how all papers are cited over time (e.g., increasing reference list length, IMRaD conventions) rather than a special aging trajectory of HCPs. The conclusion should either be substantially softened to 'temporal changes in citation patterns' or the authors must provide a control comparison that isolates the age effect.","section":"Abstract and Section 5"}],"minor_comments":[{"comment":"The abstract states 'nearly 900 highly cited papers (HCPs) published between 2000 and 2016', but Section 3 clarifies that HCPs were all published in 2000; the 2000-2016 range applies to the citing papers. Please rephrase to avoid ambiguity.","section":"Abstract"},{"comment":"The full-text coverage is reported as 20.9% in Section 3 ('accounting for 20.9% of the total number of citing papers') but as 14.3% in the limitations paragraph of Section 5. Reconcile these numbers.","section":"Section 3 vs Section 5"},{"comment":"Typo: 'has decreasing from 1.8 in 2000 to 1.4 in 2016' should be 'has decreased'. Also, minor grammar issues appear elsewhere (e.g., 'Ding et. al' in Section 2.1).","section":"Section 4.2"},{"comment":"VADER is designed for social-media-style text. Its suitability for scientific citation sentences is not justified. A brief validation or acknowledgment of this limitation would strengthen the sentiment analysis, although the relative trends are probably robust.","section":"Section 4.4"},{"comment":"The description of textual relatedness is underspecified: it is not clear whether SciBERT was applied to title+abstract concatenated or separately, and whether the [CLS] token or mean-pooled embeddings were used. Please clarify.","section":"Section 4.5"}],"recommendation":"major_revision","confidential_remarks":"The paper's descriptive statistics and data sharing are strengths, but the central inference from 'temporal trends' to 'aging effects on citation function' is not identified. The authors should be pushed to either add a non-HCP control group (e.g., randomly sampled references from the same citing papers, matched on publication year and field) or to reframe the conclusions as secular changes in citation practices affecting HCPs. This is a fixable issue within the manuscript's scope, hence my major-revision recommendation rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look, but the headline conclusion is overreach. The paper tracks ~900 highly cited papers (all from 2000) and 220k citing full texts across 17 years, measuring citation location, mention counts, in-text citation type, sentiment, and relatedness. That is a lot of careful descriptive work, and they share code and data. If you need a baseline for full-text citation features, this is a useful resource.\n\nThe problem is that every trend is plotted against citing year. Since all HCPs were published in 2000, citation age = citing year − 2000. So aging and global changes in citation practices over 2000–2016 are perfectly collinear. The paper even cites Boyack et al. (2018) showing reference counts and in-text citation counts increased over this period, but it never uses non-HCP references as a control. That matters mechanically: the MRC share rising from 32.7% to 45.7% could largely be a consequence of longer reference lists making co-citation in a single parenthesis more likely. Earlier citation location, fewer mentions, and lower title/abstract similarity could also reflect shifting IMRaD conventions, reference inflation, or other field-wide changes rather than the aging of these specific papers. The limitations section doesn't acknowledge this identification problem.\n\nSo the descriptive patterns are real and worth having, but the interpretation—\"a shift from direct topical and methodological engagement toward general, background, and symbolic referencing\"—is underdetermined. That is not a minor caveat; it is the paper's whole point.\n\nStill, the paper deserves a serious referee. The methods are transparent, the data are shared, and the descriptive results could seed more careful work. A referee should ask for a control group of non-HCP references matched by citing year, or a cohort design with HCPs from different publication years, or at minimum an explicit acknowledgment that age and citing-year effects cannot be separated here. I would not cite the current version for the symbolic-referencing claim, but I might cite it as a descriptive baseline. Send it to review, but expect that the central conclusion will need to be substantially qualified.","headline":"A careful descriptive study whose central claim—that HCPs shift to symbolic referencing as they age—is not actually identified, because aging is perfectly collinear with global citation-practice changes over the same period.","tokens_in":14198,"tokens_out":1655,"would_cite":false,"duration_ms":17699,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a highly cited paper's role shifts systematically as it ages: it is cited earlier in the citing text, mentioned fewer times, more often bundled with other references, and becomes less topically and bibliographically r","keywords":["citation analysis","citation context","full-text analysis","highly cited papers","citation sentiment","citation function","publication relatedness","research evaluation"],"falsifier":"Measure the same five characteristics for a matched control group of papers published in 2000 that received few citations rather than thousands. If this control group shows the same drift — citations moving to the opening sections, fewer mentions per paper, more multi-reference bundles, declining textual and bibliographic relatedness — then the pattern tracks changes in citing conventions over time, not the aging of celebrated papers.","tokens_in":13458,"feed_emoji":"📉","tokens_out":9888,"duration_ms":81990,"temperature":0.7,"pith_summary":"This paper asks what happens to a famous paper's role as it grows older. Following 883 papers published in 2000 that each accumulated at least 1,000 citations, the authors examine the full text of more than 220,000 later papers that cite them. They find a consistent drift: citations to these aging papers migrate toward the opening sections of the citing text, each paper is mentioned fewer times, and citations increasingly appear in clusters with other references. Textual and bibliographic overlap between the cited paper and its citing papers also declines. Together these shifts support the paper's central claim — that highly cited works move from direct topical and methodological engagement toward general, background, and symbolic referencing — and they matter because conventional citation counts treat every citation as equal.","feed_headline":"As classics age, citations turn symbolic, not substantive","feed_subtitle":"Full-text evidence from 220,000 citing papers: aging classics drift from method to mere mention.","key_machinery":"The mechanism is large-scale full-text citation parsing. Each citing paper is decomposed into references, reference mentions, in-text citations, and citation sentences, and every citation to a target paper is assigned a location on a normalized text-progression scale from 0 to 1. Five characteristics are then measured per citing year: citation location (begin/middle/end of text), whether the reference is mentioned once or multiple times (single- vs. multiple-mentioned references), whether the in-text citation stands alone or bundles several references (single- vs. multi-reference citations), the sentiment of the citation sentence measured by a lexicon-based sentiment tool, and relatedness me","core_discovery":"The central claim is that the function of a highly cited paper changes systematically as the paper ages, and that the change is visible in the full text of the papers citing it. For a cohort of 883 papers published in 2000 with at least 1,000 citations each, the authors track five citation characteristics across citing years 2000–2016. The share of citations in the opening third of citing papers rises from about 36% to 56%, while the share in the closing third falls from about 37% to 19%; the average number of mentions per citing paper drops from 1.8 to 1.4; the share of multi-reference citations grows from 32.7% to 45.7%; and both textual similarity (title/abstract embeddings) and shared-re","pith_inferences":["The design compares one cohort across time rather than treated versus untreated papers, so an untested alternative reading is that global changes in citing practice between 2000 and 2016 — more references per paper, more bundled citations — produced the observed drift independently of any paper's age. A control group of ordinary papers published in 2000 would settle which reading is right.","If symbolic citation is real, then the same measurements could detect 'awakenings' — old papers that regain substantive engagement — as rebounds in mention frequency or relatedness that raw citation counts hide.","The slight rise in citation-sentence positivity (compound score from about 0.04 to 0.11) invites a sharper question: whether prestige citations carry more praise in passing than work cited for its actual content, which would let sentiment separate ceremonial from substantive citation.","A testable prediction follows: extending the same cohort beyond 2016 should show the begin-part share, multi-reference share, and relatedness declines continuing but decelerating, as the cohort settles into a stable symbolic phase."],"forward_implications":["Citation counts are a flattening measure: for old classics they increasingly count acts of recognition rather than acts of use, so equal-weight counting systematically overstates the current role of aged highly cited papers.","Impact metrics become more truthful if differentiated by citation context — a repeated mention in the methods section should outweigh a single bundled mention in the introduction.","Science mapping and scholarly search can be improved by weighting links by textual or bibliographic relatedness, since the paper shows relatedness declines with age and thereby separates intellectual debt from mere acknowledgment.","Full-text sources become a practical necessity for credible citation analysis; reference lists alone cannot reveal the shift the paper documents."],"supporting_citations":[{"why":"Supplies the 'concept symbols' idea — the theoretical frame for reading aging citations as symbolic rather than substantive.","marker":"(Small, 1978)"},{"why":"Provides the full-text corpus this study reuses and the baseline showing average references and in-text citations grew over time, against which the cohort's drift must be interpreted.","marker":"(Boyack et al., 2018)"},{"why":"Defines multiple-mentioned references (MMR), the classification behind the finding that aging HCPs are mentioned fewer times.","marker":"(Hu et al., 2017)"},{"why":"Introduces CountX/CountOne mention counting that the single- versus multiple-mention analysis builds on.","marker":"(Ding et al., 2013)"},{"why":"Earlier evidence that highly cited references migrate toward methodology sections with age, which this study extends to full-text location.","marker":"(Otto et al., 2019)"},{"why":"Location-filtered citation counting, the precedent showing citation position matters for identifying essential references.","marker":"(Zhao & Strotmann, 2020)"},{"why":"Original bibliographic coupling measure used to compute reference relatedness between HCPs and citing papers.","marker":"(Kessler, 1963)"},{"why":"Grounds the textual relatedness measurement via title and abstract similarity of publication pairs.","marker":"(Colavizza et al., 2018)"},{"why":"The lexicon-based sentiment tool used to score citation sentences.","marker":"(Hutto & Gilbert, 2014)"},{"why":"The scientific-text language model used to embed titles and abstracts for the textual relatedness measure.","marker":"(Beltagy et al., 2019)"}],"fun_headline_variants":["Aging classics: from method to mere mention","Old papers cited as symbols, not substance","Classic papers drift from substance to symbolism over time","As papers age, citations shift from method to background","Citation context reveals: aging classics become symbolic"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The paper assumes the temporal trends it observes are caused by the aging of the cited papers, yet it studies a single cohort without a comparison group; if citation practices across all of science changed over the same 2000–2016 window, those global changes alone could produce the same patterns.","fun_headline_variants_meta":{"raw":{"variants":["Aging classics: from method to mere mention","Old papers cited as symbols, not substance","Classic papers drift from substance to symbolism over time","As papers age, citations shift from method to background","Citation context reveals: aging classics become symbolic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000345,"raw_usage":{"total_tokens":1730,"prompt_tokens":742,"completion_tokens":988,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":917}},"tokens_in":486,"tokens_out":988,"duration_ms":9511,"temperature":1.0,"reasoning_tokens":917,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:18:05.180666+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the same five characteristics for a matched control group of papers published in 2000 that received few citations rather than thousands. If this control group shows the same drift — citations moving to the opening sections, fewer mentions per paper, more multi-reference bundles, declining textual and bibliographic relatedness — then the pattern tracks changes in citing conventions over time, not the aging of celebrated papers.","supporting_citations":[],"review_version":1}