Pith. sign in

REVIEW 3 major objections 13 references

Jointly ranking every reference inside a citing paper lets LLMs distinguish high-impact citations more reliably than scoring each one alone.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 18:48 UTC pith:TANZ5ZWV

load-bearing objection Clean joint-ranking method for relative citation impact that beats the prior LLM baseline with solid ablations and a useful release; the High-only mapping is a real but secondary caveat, not a collapse of the claim. the 3 major comments →

arxiv 2603.26791 v3 pith:TANZ5ZWV submitted 2026-03-25 cs.DL cs.AIcs.CLcs.CY

Crystal: Characterizing Relative Impact of Scholarly Publications

Hannah Collison , Benjamin Van Durme , Daniel Khashabi This is my paper
classification cs.DL cs.AIcs.CLcs.CY
keywords citation impactlistwise rankinglarge language modelspositional biasscientific evaluationACL Test-of-TimeCrystal-Bank
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Most citation-impact tools judge a single citation by looking only at the sentences that mention it. Crystal instead asks a language model to rank every reference that appears in the same paper at once, so each citation is judged relative to its neighbors. To blunt the models’ well-known preference for items that appear early or late in a list, the authors shuffle the order three times, run the ranking, and take a majority vote. On a public set of human-labeled citations the method lifts accuracy by roughly nine and a half points and F1 by eight points over the previous best system, while making far fewer model calls. When the same pipeline is run over tens of thousands of papers that cite classic ACL work, the papers that later won Test-of-Time awards sit near the top of Crystal’s high-impact tallies. The authors release both the code and a 46 800-paper bank of rankings so others can reuse the signal.

Core claim

Evaluating all of a paper’s citations jointly, rather than in isolation, supplies comparative context that lets an LLM separate truly indispensable references from background ones more accurately; three randomized ranking passes with majority vote further stabilize the labels against positional bias.

What carries the argument

Crystal’s listwise ranking prompt that simultaneously assigns every reference in a citing paper to High / Medium / Low impact, executed three times under random order and aggregated by majority vote (or by reciprocal-rank fusion plus ordinal regression).

Load-bearing premise

That the three-level impact definitions written into the prompt, plus a three-shuffle majority vote, produce labels that line up with what humans mean by “impact,” even though the gold labels are only binary and the pilot check used only six co-authors from the same lab.

What would settle it

Run the identical three-shuffle majority-vote pipeline on a larger, multi-lab set of author-ranked reference lists and measure whether Spearman rank correlation with the human rankings stays above 0.7 and whether the accuracy gap over independent scoring remains positive.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Impact tallies can be computed for entire fields at O(number of papers) rather than O(number of citation edges), making large-scale analysis practical.
  • Open-weight models already match closed models on this task, so the method can be run without proprietary APIs.
  • High-impact citation counts become a usable, time-resolved signal that can be compared against awards, novelty scores, or retrieval benchmarks.
  • The released 46.8 k-paper bank supplies ready-made training and evaluation data for future impact models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same joint-ranking idea could be applied inside survey papers or grant proposals to surface the few references that actually drive the argument.
  • Temporal traces of high-impact citations may help predict which recent papers will still be cited a decade later.
  • Because the prompt forces an explicit High/Medium/Low partition, the method can serve as a cheap filter before expensive human review of citation importance.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. Crystal proposes to characterize citation impact by jointly ranking all references inside a citing paper with an LLM judge, rather than scoring each citation context in isolation. To reduce positional bias the reference list is ranked three times under random permutations and impact labels (High/Medium/Low) are aggregated by majority vote; an optional RRF+ordinal-regression path is also offered. On the human-annotated binary impact dataset of Arnaout et al. (2025), after mapping only High to “impact-revealing,” Crystal improves accuracy by +9.5 % and F1 by +8.3 % on average across GPT-5.1, o4-mini and Qwen3-30B relative to the prior UKP LLM baseline, while using asymptotically fewer LLM calls (O(n) papers vs O(m) edges). Ablations isolate the joint formulation, three-class granularity and permutation voting as the sources of gain. A case study of ACL 1996/2000 papers shows that high-impact citation counts under Crystal place Test-of-Time winners among the top four papers of their years. The authors release Crystal-Bank (46.8 k papers with rankings and labels) and code.

Significance. If the reported gains hold under a more carefully validated label mapping, Crystal supplies a practical, open-weight-compatible alternative to edge-level impact classifiers and a large public resource (Crystal-Bank) for downstream bibliometric work. The efficiency argument (O(n) vs O(m) LLM calls) and the demonstration that an open 30 B model already outperforms prior closed-model baselines are concrete engineering contributions. The ACL ToT alignment, while only correlational, offers an external, long-horizon sanity check that raw citation counts lack. These elements together make the work useful for the digital-libraries and science-of-science communities even if the absolute magnitude of the accuracy lift is later revised.

major comments (3)
  1. §A / Table 3 and Ablation 1 (Table 4): evaluation maps only High → impact-revealing while Medium and Low both become “other.” Ablation 1 shows that the three-class formulation mainly raises precision by parking borderline cases in Medium. Because the gold labels are binary, the headline +9.5 % / +8.3 % deltas partly reflect this post-hoc mapping choice. A sensitivity analysis that also treats Medium as positive (or a small human re-annotation of the Medium band) is needed to confirm that the claimed superiority of joint ranking is not inflated by a stricter decision threshold.
  2. §F (pilot study): the prompt is validated only by six co-author annotators from the same lab ranking their own papers (Spearman ρ > 0.7). That experiment checks ranking order under the authors’ High/Medium/Low definitions; it does not establish that the High class coincides with the binary “impact-revealing” notion used by Arnaout et al. A modest external validation (even 50–100 double-annotated contexts drawn from the evaluation set) would substantially strengthen the load-bearing assumption that the prompt produces labels faithful to the gold standard.
  3. §5 / §C and Figure 11: when reference lists exceed ~40 items, all three models omit substantial numbers of papers; omitted items are scored as errors. Because UKP scores every edge independently, the comparison is not fully apples-to-apples for long-reference papers. Reporting accuracy stratified by reference-list length (or restricting the main table to papers with complete rankings) would clarify how much of the gain survives under a fairer protocol.

Circularity Check

0 steps flagged

No significant circularity: Crystal is an empirical LLM ranking method evaluated on an external human-annotated binary impact dataset; labels and rankings are not defined in terms of fitted parameters or self-referential theorems.

full rationale

The paper's central claim is that jointly ranking all references of a citing paper (via three randomized LLM passes + majority vote) outperforms the prior UKP impact classifier by +9.5% accuracy / +8.3% F1 on the held-out Arnaout et al. (2025) human labels. Impact categories (High/Medium/Low) are supplied as fixed prompt definitions drawn from classic citation-function literature (Moravcsik & Murugesan 1975); they are not fitted to the evaluation data. Evaluation-time mapping (High → impact-revealing; Medium/Low → other) is an explicit post-hoc choice documented in Table 3 and §A, not a circular redefinition of the gold standard. Ablations isolate joint ranking, label granularity, and permutation voting as independent design choices. The pilot (six co-author rankings, Spearman ρ > 0.7) validates prompt alignment with author judgments under the same definitions, not the binary gold labels used for headline metrics. The ACL ToT case study and Crystal-Bank release are external applications. No self-definitional equations, fitted-input-as-prediction, load-bearing self-citation uniqueness claims, or renaming of known results appear. The alternative ordinal-regression post-processor (§E) is trained only on non-test majority-vote labels and is not required for the main results. The derivation chain is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 2 invented entities

The central performance claim rests on the reliability of LLM listwise ranking under the supplied impact definitions, the adequacy of three random permutations plus majority vote, the quality of Semantic Scholar citation contexts, and the external human gold labels; no free parameters are fitted to the evaluation metrics themselves.

free parameters (3)
  • number of ranking permutations = 3
    Fixed at three independent randomized orderings whose majority vote produces the final label; chosen by the authors rather than derived.
  • RRF constant k = 60
    Set to 60 following Cormack et al. for the optional ordinal-regression aggregation path.
  • ordinal-regression L2 alpha = 1.0
    Regularization strength set to 1.0 for the alternative label-assignment model.
axioms (3)
  • domain assumption An LLM given the full set of citation contexts of a paper can produce a ranking of relative impact that is more accurate than independent per-citation judgments.
    Core modeling premise stated in §4 and validated only by the pilot study and the main evaluation.
  • ad hoc to paper Three random permutations plus majority vote sufficiently mitigate positional bias for this task.
    Adopted from Tang et al. (2024) PSC; ablation 3 shows it helps weaker models but is not proven optimal.
  • domain assumption The binary human labels of Arnaout et al. (2025) constitute a reliable gold standard for impact-revealing versus other citations.
    All accuracy/F1 numbers are computed against this external set after label mapping.
invented entities (2)
  • Crystal joint-ranking procedure no independent evidence
    purpose: Produce calibrated high/medium/low impact labels for every reference inside a citing paper via multi-pass LLM ranking.
    The method itself is the paper’s primary technical contribution.
  • Crystal-Bank no independent evidence
    purpose: 46.8k-paper resource of rankings and impact labels for downstream use.
    Released artifact constructed by running Crystal on the evaluation and ACL corpora.

pith-pipeline@v1.1.0-grok45 · 29233 in / 2629 out tokens · 38255 ms · 2026-07-13T18:48:29.412175+00:00 · methodology

0 comments
read the original abstract

Assessing a cited paper's impact is typically done by analyzing its citation context in isolation within the citing paper. While this focuses on the most directly relevant text, it prevents relative comparisons across all the works a paper cites. We propose Crystal, which instead jointly ranks all cited papers within a citing paper using large language models (LLMs). To mitigate LLMs' positional bias, we rank each list three times in a randomized order and aggregate the impact labels through majority voting. This joint approach leverages the full citation context, rather than evaluating citations independently, to more reliably distinguish impactful references. Crystal outperforms a prior state-of-the-art impact classifier by +9.5% accuracy and +8.3% F1 on a dataset of human-annotated citations. Crystal further gains efficiency through fewer LLM calls and outperforms prior baselines using an open-weight model, enabling scalable, cost-effective citation impact analysis. In a case study of ACL Test-of-Time award-winning papers, we find that Crystal's impact characterizations align closely with long-term scientific recognition. We release Crystal-Bank, a 46.8k-paper dataset with rankings and impact labels, along with code.

Figures

Figures reproduced from arXiv: 2603.26791 by Benjamin Van Durme, Daniel Khashabi, Hannah Collison.

Figure 1
Figure 1. Figure 1: Overall performance of MVO-based methods on CEC benchmark functions: baseline algorithms (left) and FA-based variants (right). The x-axis shows the number of results statistically similar to the best, and the y-axis shows the number of best results. Algorithms in the top-right achieve the best performance. See Tables5 and 7 for details. 5.1 Results on Benchmark Functions This section analyzes the results o… view at source ↗
Figure 2
Figure 2. Figure 2: Performance in terms of the absolute error (the lower the better) scored by the seven mixed-variable opti￾mization methods on the CEC benchmark for the unimodal functions. more detailed analysis of algorithm behavior across different landscape characteristics, and the respective convergence plots are presented in the additional material (Appendix B). As shown in [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance in terms of the absolute error (the lower the better) scored by the seven mixed-variable opti￾mization methods on the CEC benchmark for the multimodal functions. 5.2 Results on Engineering Design Problems To further investigate the performance of the proposed algorithm on real-world mixed-variable optimization problems, three engineering design problems of varying complexity are considered. The… view at source ↗
Figure 4
Figure 4. Figure 4: Performance in terms of the absolute error (the lower the better) scored by the seven MVO methods on composite functions in the CEC benchmark [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Performance of MVO methods on the three engineering design problems (BEAM, CSD, and Vessel), measured by absolute error (top) and convergence curves (bottom). Lower values indicate better performance. 5.3 Ablation Study of the Proposed FAmv Components This section presents an ablation study to analyze the contribution of the main components of FAmv, namely the mixed￾variable movement mechanism and the para… view at source ↗
Figure 6
Figure 6. Figure 6: The description of this parameter is presented in Section 3.2 [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Convergence curves of the MVO methods on unimodal functions in the CEC benchmark [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Convergence curves of the MVO methods on multimodal functions in the CEC benchmark. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Convergence curves of the MVO methods on composite functions in the CEC benchmark. C Full results on the impact of the mixed-variable movement mechanism in FA variants This section presents the results obtained by the FA-based variants that investigate the proposed mixed-variable move￾ment mechanism and the parameter adaptation strategy, as detailed in Sections 3.2 and 3.3, respectively. In total, eight FA… view at source ↗
Figure 10
Figure 10. Figure 10: Performance scored by the FA-based variants on the CEC benchmark for the unimodal functions [PITH_FULL_IMAGE:figures/full_fig_p023_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Performance scored by the FA-based variants on the CEC benchmark for the multimodal functions. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Performance scored by the FA-based variants on the CEC benchmark for the composition functions [PITH_FULL_IMAGE:figures/full_fig_p024_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Convergence curves of the FA-based variants on unimodal functions in the CEC benchmark. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Convergence curves of the FA-based variants on multimodal functions in the CEC benchmark [PITH_FULL_IMAGE:figures/full_fig_p025_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Convergence curves of the FA-based variants on composition functions in the CEC benchmark. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_15.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

13 extracted references · 3 linked inside Pith

  1. [1]

    Tomoki Ikoma and Shigeki Matsubara

    Identifying important citations using contex- tual information from full text.2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pages 1–8. Tomoki Ikoma and Shigeki Matsubara. 2023. On the use of language models for function identification of citations in scholarly papers. InProceedings of the Second Workshop on Information Extraction from Scient...

    Pith/arXiv arXiv 2017

  2. [2]

    InProceedings of the 2022 conference of the North American chapter of the association for compu- tational linguistics: Human language technologies, pages 1875–1889

    Multicite: Modeling realistic citations requires moving beyond the single-sentence single-label set- ting. InProceedings of the 2022 conference of the North American chapter of the association for compu- tational linguistics: Human language technologies, pages 1875–1889. Willem JM Levelt, Ardi Roelofs, and Antje S Meyer

    2022

  3. [3]

    Behavioral and brain sciences, 22(1):1–38

    A theory of lexical access in speech production. Behavioral and brain sciences, 22(1):1–38. Saurav Manchanda and George Karypis. 2021. Evalu- ating scholarly impact: Towards content-aware bib- liometrics. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 6041–6053, Online and Punta Cana, Do- minican Republic...

    2021

  4. [4]

    Fabian Pedregosa-Izquierdo

    Some results on the function and quality of citations.Social studies of science, 5(1):86–92. Fabian Pedregosa-Izquierdo. 2015.Feature extraction and supervised learning on fMRI: from practice to theory. Ph.D. thesis, Université Pierre et Marie Curie- Paris VI. Jason Priem, Heather Piwowar, and Richard Orr. 2022. Openalex: A fully-open index of scholarly w...

    Pith/arXiv arXiv 2015

  5. [5]

    Xiaodan Zhu, Peter Turney, Daniel Lemire, and André Vellino

    Sciimpact: A multi-dimensional, multi-field benchmark for scientific impact prediction.arXiv preprint arXiv:2604.17141. Xiaodan Zhu, Peter Turney, Daniel Lemire, and André Vellino. 2015. Measuring academic influence: Not all citations are equal.Journal of the Association for Information Science and Technology, 66(2):408–427. A Citation Intent vs. Citation...

    Pith/arXiv arXiv 2015

  6. [8]

    rank": integer •

    Low-impact citations:These citations provide background, context, or perfunctory acknowledgement, but the core contribution of your paper is not dependent on them in any strong way. • Background or definitional citations:References used to define a task (e.g., Question Answering), introduce a general problem area, or acknowledge standard terminology. The ...

    2025

  7. [9]

    impact-revealing

    “impact-revealing” citations:These are the papers without which your own work would not have been possible. They supply essential conceptual, methodological, or operational ingredients. • Conceptual or operational indispensability:The reference provides a unique conceptual insight, methodological innovation, dataset, or technique that is directly instrume...

  8. [10]

    rank": integer •

    “other” citations:These are papers that helped you write your paper, but were not fundamentally irreplaceable. You could have used an alternative prior work or formulation, but you chose this one because it was particularly useful, clear, or canonical. These citations could also provide background, context, or perfunctory acknowledgement, but the core con...

    2025

  9. [11]

    They supply essential conceptual, methodological, or operational ingredients

    High-impact citations:These are the papers without which your own work would not have been possible. They supply essential conceptual, methodological, or operational ingredients. • Conceptual or operational indispensability:The reference provides a unique conceptual insight, methodological innovation, dataset, or technique that is directly instrumental to...

  10. [12]

    You could have used an alternative prior work or formulation, but you chose this one because it was particularly useful, clear, or canonical

    Medium-impact citations:These are papers that helped you write your paper, but were not fundamentally irreplaceable. You could have used an alternative prior work or formulation, but you chose this one because it was particularly useful, clear, or canonical. • Conceptual or operational contribution (non-unique):The reference conveys an idea, dataset, or m...

  11. [13]

    paperId": string •

    Low-impact citations:These citations provide background, context, or perfunctory acknowledgement, but the core contribution of your paper is not dependent on them in any strong way. • Background or definitional citations:References used to define a task (e.g., Question Answering), introduce a general problem area, or acknowledge standard terminology. The ...

  12. [14]

    o4-mini exhibits the steepest degradation, while GPT-5.1 and Qwen3-30B are more robust

    However, they omit more references as the list grows longer. o4-mini exhibits the steepest degradation, while GPT-5.1 and Qwen3-30B are more robust. Figure 12: Distribution of citation context length (in characters) for cases unanimously classified by QWEN3-30B, GPT-5.1, andO4-MINIunder Crystal.Left:unanimous false positives (FP, n=45) versus unanimous tr...

    2009

  13. [15]

    rank": 1,

    leveraged the reinforcement learning to automatically prune the convolution channels. Our method also connects to PinSage [21] and GAT [15]. We run the 4 models on the datasets and respective splits from [Yang et al., 2016]. Discriminative models are known to suffer from catastrophic forgetting when learning sequentially from examples from a single class ...

    2016