Pith. sign in

REVIEW 3 major objections 7 minor 52 references

Fine-tuned DeBERTa-large finds opioid impacts on Reddit with F1 0.61, beating GPT-4o prompts at 0.44.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Fine-tuned DeBERTa-large outperforms LLMs on extracting clinical and social impacts from opioid-use Reddit posts (relaxed token F1 0.61 vs 0.44), yet remains below human agreement (kappa 0.81).

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A useful new dataset and a fair empirical comparison, but the 'significantly outperforming' claim outruns the statistics. the 3 major comments →

arxiv 2508.19467 v1 pith:YVDAVVHM submitted 2025-08-26 cs.CL cs.AIcs.IR

Inference Gap in Domain Expertise and Machine Intelligence in Named Entity Recognition: Creation of and Insights from a Substance Use-related Dataset

Sumon Kanti Dey , Jeanne M. Powell , Azra Ismail , Jeanmarie Perrone , Abeed Sarker This is my paper
classification cs.CL cs.AIcs.IR
keywords named entity recognitionopioid usesocial mediaRedditin-context learningfine-tuningDeBERTaclinical NLP
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that for extracting self-reported clinical and social consequences of opioid use from Reddit posts, a domain-specific fine-tuned encoder model performs substantially better than prompting large language models. It introduces RedditImpacts 2.0, a first-person-focused annotated dataset with refined guidelines, and evaluates BERT, RoBERTa, DeBERTa, BioBERT and GPT-4o, Llama, and Gemma under zero- and few-shot in-context learning. The best model, DeBERTa-large, reaches relaxed token-level F1 0.61 (95% CI 0.43–0.62) versus 0.44 for the best LLM, GPT-4o with 3-shot prompting. The paper also reports that about half the training data is enough to reach plateau performance, and that even the best model remains below human inter-annotator agreement (Cohen's kappa 0.81). These findings matter because social media is an underused signal for addiction surveillance, and the comparison clarifies when fine-tuning is still needed rather than prompting alone.

Core claim

The central claim is that a fine-tuned DeBERTa-large model trained as a BIO token tagger on RedditImpacts 2.0 extracts ClinicalImpacts and SocialImpacts from first-person opioid narratives more accurately than state-of-the-art LLMs used with zero-, three-, or five-shot in-context learning. On the held-out test set, DeBERTa-large achieves relaxed token-level F1 0.61 (precision 0.75, recall 0.52), with stronger performance on ClinicalImpacts (F1 0.66) than SocialImpacts (F1 0.50); GPT-4o with three-shot prompting reaches F1 0.44 overall, with SocialImpacts much weaker at 0.26. The paper further claims the gap is not simply about LLM capacity, since error analysis shows GPT-4o mislabels social-

What carries the argument

The task is framed as sequence labeling with BIO tags, where each token is labeled O, B-ClinicalImpacts, I-ClinicalImpacts, B-SocialImpacts, or I-SocialImpacts. The top-performing system is a DeBERTa-large encoder with a linear token-level classification head, fine-tuned with cross-entropy, optionally augmented with a conditional random field layer that enforces valid tag transitions. Evaluation uses a relaxed token-level F1 that credits partial span overlap, appropriate for noisy social-media boundaries; LLM prompts are built by retrieving semantically similar training examples with sentence embeddings and feeding them as few-shot demonstrations.

Load-bearing premise

The test-set labels are treated as ground truth even though most test posts were annotated by a single human expert after the annotators agreed on a shared subset; if the two annotators systematically differ on the remaining posts, the measured F1 scores and the DeBERTa-versus-GPT-4o gap could shift.

What would settle it

Re-annotate the held-out test posts with both annotators plus adjudication, then recompute relaxed token-level F1 for DeBERTa-large and GPT-4o 3-shot against the consensus labels; if DeBERTa's margin over GPT-4o disappears or becomes statistically insignificant, the central comparison collapses.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • On this task, domain-specific fine-tuning of an encoder beats LLM prompting by about 17 F1 points, so fine-tuning remains the stronger default for token-level clinical NER.
  • Training on roughly 50% of the labeled data reaches performance comparable to the full dataset, so smaller annotation budgets can support deployment in low-resource settings.
  • Few-shot in-context learning gives only marginal, not statistically significant, gains over zero-shot; adding more examples can slightly hurt performance.
  • A substantial gap remains to human expertise, indicating headroom for models that better capture implicit impacts and first-person-only disclosures.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • An untested extension suggested by the error analysis is a two-stage pipeline that first classifies whether a post is first-person and then runs NER; such a filter could reduce many of GPT-4o's guideline violations, but the paper did not evaluate it.
  • Because the test set has only 278 posts and SocialImpacts entities are less frequent than ClinicalImpacts, the reported 17-point gap may be sensitive to how social-impact spans are drawn; re-running on a larger or re-annotated test set would sharpen the comparison.
  • The data-efficiency plateau implies that further gains may come more from better annotation guidelines or model architecture than from labeling more examples; a direct test would compare a second-round guideline revision against doubling the training set.
  • The relaxed token-level F1 metric credits partial span overlap, so the model-vs-human comparison is not directly on exact spans; a strict-boundary evaluation could change the practical assessment of deployability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces RedditImpacts 2.0, a refined named-entity-recognition dataset for identifying self-reported ClinicalImpacts and SocialImpacts in first-person Reddit narratives about nonmedical opioid use. The authors describe revised annotation guidelines, report an inter-annotator agreement of Cohen's kappa 0.81, and evaluate a range of fine-tuned encoder-based models (with and without CRF) and LLMs under zero-, 3-, and 5-shot in-context learning. On a 278-post test set, the best fine-tuned DeBERTa-large model achieves a relaxed token-level F1 of 0.61 [0.43, 0.62], while the best LLM, GPT-4o with 3-shot prompting, achieves 0.44 [0.39, 0.51]. The paper concludes that fine-tuned encoders significantly outperform prompting-based LLMs for this domain, presents a qualitative error analysis, and reports a data-efficiency experiment suggesting that 50% of the training data suffices. The dataset, guidelines, and training scripts are publicly released.

Significance. If the empirical claims hold, this is a useful contribution to clinical NLP and social-media public-health surveillance. The dataset fills a gap by focusing on first-person, self-reported consequences of opioid use, and the refined guidelines address known annotation-noise problems in the earlier Reddit-Impacts resource. The systematic comparison of PLMs, CRF-augmented PLMs, and modern LLMs is relevant, and the public release of data and code supports reproducibility. The qualitative error analysis is informative. However, the central quantitative claim—that DeBERTa-large 'consistently outperforms' or 'significantly outperforms' LLMs—is not currently supported by the reported statistics, and the relaxed metric has a technical ambiguity that can affect the reported F1 values. These issues are fixable but require additional analysis.

major comments (3)
  1. [§4, Table 3; §6 Conclusion] The headline claim that DeBERTa-large 'significantly outperforms' GPT-4o is not supported by the reported confidence intervals. DeBERTa-large has F1 0.61 [0.43, 0.62] and GPT-4o has F1 0.44 [0.39, 0.51] in Table 3; these intervals overlap substantially. The entity-specific intervals in Table 4 also overlap (e.g., SocialImpacts: 0.50 [0.25, 0.62] vs. 0.26 [0.18, 0.34]; ClinicalImpacts: 0.66 [0.45, 0.67] vs. 0.51 [0.44, 0.60]). No significance test is reported, and the test set is small (278 posts, 364 entities). The conclusion in Section 6 uses the word 'significantly,' but the evidence presented does not establish statistical significance. Please add a paired test (e.g., paired bootstrap over posts, or McNemar-type test on token/span decisions) or soften the claim to 'outperformed in this evaluation.' Reporting a strict span-level F1 would also strengthen the claim.
  2. [§3.5, Eq. for relaxed F1] The relaxed F1 definition is underspecified because no one-to-one matching between predicted and gold spans is defined. The text says M_T is the set of span pairs with non-zero token-level overlap and then computes TPT as the sum of Overlap(g,p) over all such pairs. As written, a single predicted span overlapping multiple gold spans (or vice versa) contributes multiple times, inflating true positives relative to a standard span-to-span match. This can distort the absolute F1 values and, depending on how models produce overlapping/duplicate spans, may also affect the relative comparison. Please specify the matching procedure (e.g., greedy matching, maximum-weight matching, or an explicit token-level TP definition that counts each gold and predicted token at most once).
  3. [§3.2, Data Annotation; Table 2] The reliability of the test-set labels is not fully established. Cohen's kappa of 0.81 was computed only on the co-annotated subset used for guideline alignment; after agreement, the remaining data were divided between the two annotators, meaning each test post received a single annotation. Systematic annotator bias in the test split could distort the reported F1 and the model comparison. Please report the distribution of the two annotators' posts in the test split, provide a per-split agreement estimate, or carry out adjudication on the test set. This is load-bearing because the main comparison is an empirical measurement against this ground truth.
minor comments (7)
  1. [Abstract; Table 3(b)] The GPT-4o 3-shot 95% CI is given as [0.37, 0.51] in the abstract but as [0.39, 0.51] in Table 3(b). Please make these consistent.
  2. [Table 4; §4] Table 4 reports DeBERTa-large ClinicalImpacts F1 as 0.66, while the text in Section 4 says the F1 is 0.60. One of these is wrong; please correct and ensure the corresponding CI is aligned.
  3. [Appendix A] The sentence 'A notable effort in this area is the work by Ge et al. [16]...' is duplicated verbatim. Please remove the duplicate.
  4. [Appendix C] There is a typo: 'SociallImpacts' should be 'SocialImpacts' in the example under 'Ambiguous context (assumed impact).'
  5. [Table D2] The prompt text contains 'ONL Y' instead of 'ONLY' in rule 1. Please fix.
  6. [§4.1 / Abstract] The abstract claims that the fine-tuned model outperforms LLMs 'in ... adherence to task-specific guidelines,' but this is supported only by qualitative examples in Section 4.1, not by a quantitative metric. If this is a headline claim, define and report a guideline-violation rate; otherwise soften the wording.
  7. [§5.4, Figure 2] The data-efficiency experiment does not describe how the subsampled training sets were selected, how many random seeds were used, or how the CIs in Figure 2 were computed. Please add these details so the plateau claim is reproducible.

Circularity Check

0 steps flagged

No circular derivation: the central model comparison is an empirical measurement on a held-out test set; self-citations are background context, not load-bearing.

full rationale

The paper's central claim is that a fine-tuned DeBERTa-large model outperforms LLM prompting on a newly annotated NER dataset. This is an empirical result: the model is trained on a train/validation split and evaluated on a disjoint test set (Section 3.3, Table 3). No target quantity is defined in terms of the conclusion, and no fitted parameter is renamed as a prediction. The relaxed token-level F1 metric (Section 3.5) is a custom evaluation choice, but it is applied symmetrically to all systems; it does not encode the DeBERTa-vs-GPT-4o ordering. The data-efficiency and error analyses are likewise measured outcomes, not derived from the claim they support. The paper cites prior work by the same group ([16], [10], [32]) to motivate the dataset lineage and task framing, but these citations are contextual: the improvements in RedditImpacts 2.0 are documented through new annotation guidelines and new experiments, and the encoder-vs-LLM comparison rests on the reported test-set F1 scores and CIs, not on the cited papers. The annotation-reliability caveat (Cohen's kappa measured only on the co-annotated subset, with remaining test data singly annotated) is a legitimate data-quality or statistical concern, but it is not a circularity. There is no exhibited reduction of a predicted result to its own input by construction, and no load-bearing self-citation chain. The score reflects only minor self-citation that is not load-bearing.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

The paper's conclusions rest on the authors' annotation guidelines, their custom relaxed metric, and their assumption about attributing impacts to opioids; these are reasonable domain choices but are not externally established.

axioms (3)
  • domain assumption ClinicalImpacts and SocialImpacts are useful, mutually exclusive categories that can be reliably labeled in first-person Reddit narratives.
    Defined in Appendix C and used for all annotations; the evaluation depends on this taxonomy.
  • ad hoc to paper The relaxed token-level overlap metric is a valid measure of entity recognition quality.
    Introduced in Section 3.5 as a custom metric; it credits partial span matches and may inflate scores relative to strict matching.
  • domain assumption Opioid involvement is assumed for any impact unless a non-opioid cause is explicitly stated.
    Stated as a rule in the prompt (Appendix D.2, rule 4) and in the annotation guidelines; this shapes the gold labels and model behavior.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Inference Gap in Domain Expertise and Machine Intelligence in Named Entity Recognition: Creation of and Insights from a Substance Use-related Dataset." pith.science (2026). https://pith.science/paper/YVDAVVHM

@misc{pith2026250819467,
  author       = {Pith},
  title        = {Pith review of: Inference Gap in Domain Expertise and Machine Intelligence in Named Entity Recognition: Creation of and Insights from a Substance Use-related Dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YVDAVVHM}},
  note         = {Machine review of arXiv:2508.19467}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Nonmedical opioid use is an urgent public health challenge, with far-reaching clinical and social consequences that are often underreported in traditional healthcare settings. Social media platforms, where individuals candidly share first-person experiences, offer a valuable yet underutilized source of insight into these impacts. In this study, we present a named entity recognition (NER) framework to extract two categories of self-reported consequences from social media narratives related to opioid use: ClinicalImpacts (e.g., withdrawal, depression) and SocialImpacts (e.g., job loss). To support this task, we introduce RedditImpacts 2.0, a high-quality dataset with refined annotation guidelines and a focus on first-person disclosures, addressing key limitations of prior work. We evaluate both fine-tuned encoder-based models and state-of-the-art large language models (LLMs) under zero- and few-shot in-context learning settings. Our fine-tuned DeBERTa-large model achieves a relaxed token-level F1 of 0.61 [95% CI: 0.43-0.62], consistently outperforming LLMs in precision, span accuracy, and adherence to task-specific guidelines. Furthermore, we show that strong NER performance can be achieved with substantially less labeled data, emphasizing the feasibility of deploying robust models in resource-limited settings. Our findings underscore the value of domain-specific fine-tuning for clinical NLP tasks and contribute to the responsible development of AI tools that may enhance addiction surveillance, improve interpretability, and support real-world healthcare decision-making. The best performing model, however, still significantly underperforms compared to inter-expert agreement (Cohen's kappa: 0.81), demonstrating that a gap persists between expert intelligence and current state-of-the-art NER/AI capabilities for tasks requiring deep domain knowledge.

Figures

Figures reproduced from arXiv: 2508.19467 by Abeed Sarker, Azra Ismail, Jeanmarie Perrone, Jeanne M. Powell, Sumon Kanti Dey.

Figure 1
Figure 1. Figure 1: NER framework for detecting Social and Clinical Impacts in user-generated text. In the output, Clinical corresponds to ClinicalImpacts and Social corresponds to SocialImpacts . 3.1. Task Overview We formulate the NER task as a sequence-labeling problem using the BIO (Begin, Inside, Outside) tagging scheme. This scheme assigns one of three tags to each token: B to indicate the beginning of an entity, I for … view at source ↗
Figure 2
Figure 2. Figure 2: F1 score with 95% CIs across training data sizes. x-axis shows the number of training samples with the corresponding percentage of the full dataset; shaded area represents the 95% CI for each point. Training a model for specialized tasks can be expensive due to the need for domain expert annotation—leading to substantial time and financial costs [49, 50]. Research suggests that expert annotators take 10- 3… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 43 canonical work pages · 5 internal anchors

  1. [1]

    American Psychiatric Association, Opioid use disorder (2023), Accessed: 2025-06-06

    work page 2023

  2. [2]

    Mayo Clinic, The role of healthcare professionals in opioid addiction prevention (n.d.), Accessed: 2025-06-06

    work page 2025

  3. [3]

    Cerd´ a, N

    M. Cerd´ a, N. Krawczyk, L. Hamilton, K. E. Rudolph, S. R. Friedman and K. M. Keyes, A critical review of the social and behavioral contributions to the overdose epidemic, Annual review of public health 42, 95 (2021)

    work page 2021

  4. [4]

    C. Lin, S. J. Cousins, Y. Zhu, S. E. Clingan, L. J. Mooney, E. Kan, F. Wu and Y.-I. Hser, A scoping review of social determinants of health’s impact on substance use disorders over the life course, Journal of substance use and addiction treatment , p. 209484 (2024)

    work page 2024

  5. [5]

    Darolia and C

    R. Darolia and C. Heflin, The social and community consequences of the opioid epidemic, The ANNALS of the American Academy of Political and Social Science 703, 7 (2022)

    work page 2022

  6. [6]

    Strike, S

    C. Strike, S. Robinson, A. Guta, D. H. Tan, B. O’Leary, C. Cooper, R. Upshur and S. Chan Caru- sone, Illicit drug use while admitted to hospital: Patient and health care provider perspectives, Plos one 15, p. e0229713 (2020)

    work page 2020

  7. [7]

    Cooper and S

    S. Cooper and S. Nielsen, Stigma and social support in pharmaceutical opioid treatment popu- lations: A scoping review, International Journal of Mental Health and Addiction 15, 452 (2017)

    work page 2017

  8. [8]

    Cheetham, L

    A. Cheetham, L. Picco, A. Barnett, D. I. Lubman and S. Nielsen, The impact of stigma on people with opioid use disorder, opioid treatment, and policy, Substance abuse and rehabilitation , 1 (2022)

    work page 2022

  9. [9]

    Lokala, O

    U. Lokala, O. C. Phukan, T. G. Dastidar, F. Lamy, R. Daniulaityte and A. Sheth, Detecting substance use disorder using social media data and the dark web: time-and knowledge-aware study, JMIRx Med 5, p. e48519 (2024)

    work page 2024

  10. [10]

    Sarker, A

    A. Sarker, A. DeRoos and J. Perrone, Mining social media for prescription medication abuse monitoring: a review and proposal for a data-centric framework,Journal of the American Medical Informatics Association 27, 315 (2020)

    work page 2020

  11. [11]

    Giorgi, D

    S. Giorgi, D. B. Yaden, J. C. Eichstaedt, L. H. Ungar, H. A. Schwartz, A. Kwarteng and B. Curtis, Predicting us county opioid poisoning mortality from multi-modal social media and psychological self-report data, Scientific reports 13, p. 9027 (2023)

    work page 2023

  12. [12]

    K. A. Carpenter, A. T. Nguyen, D. A. Smith, I. A. Samori, K. Humphreys, A. Lembke, M. V. Kiang, J. C. Eichstaedt and R. B. Altman, Which social media platforms facilitate monitoring the opioid crisis?, PLOS Digital Health 4, p. e0000842 (2025)

    work page 2025

  13. [13]

    Humphreys, R

    K. Humphreys, R. Saitz, N. D. Volkow, C. L. Shover, T. F. Babor, B. G. Carr, W. N. Evans, D. A. Fiellin, S. A. Glantz, W. Hall, D. Heller, D. H. Jernigan, R. J. MacCoun, B. K. Madras, S. Satel, B. Vicknasingam, S. E. Wakeman, R. West, D. P. Wilson, D. Ziedonis, L. R. Zindel, P. Das, O. A. Olukoya, S. L. Proctor, S. J. Tye, E. Wakeman and C. H. Wilkins, Re...

    work page 2022

  14. [14]

    Coppersmith, R

    G. Coppersmith, R. Leary, P. Crutchley and A. Fine, Natural language processing of social media as screening for suicide risk, Biomedical informatics insights 10, p. 1178222618792860 (2018)

    work page 2018

  15. [15]

    Beyer, Joint economic committee releases new report on the economic toll of the opioid crisis (2023), Accessed: 2025-06-15

    D. Beyer, Joint economic committee releases new report on the economic toll of the opioid crisis (2023), Accessed: 2025-06-15

    work page 2023

  16. [16]

    Y. Ge, S. Das, K. O’Connor, M. A. Al-Garadi, G. Gonzalez-Hernandez and A. Sarker, Reddit- impacts: A named entity recognition dataset for analyzing clinical and social effects of substance use derived from social media, arXiv preprint arXiv:2405.06145 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  17. [17]

    Cho and H

    H. Cho and H. Lee, Biomedical named entity recognition using deep neural networks with contextual information, BMC bioinformatics 20, 1 (2019)

    work page 2019

  18. [18]

    Recognising Biomedical Names: Challenges and Solutions

    X. Dai, Recognising biomedical names: Challenges and solutions, arXiv preprint arXiv:2106.12230 (2021)

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  19. [19]

    Luo, C.-H

    L. Luo, C.-H. Wei, P.-T. Lai, R. Leaman, Q. Chen and Z. Lu, Aioner: all-in-one scheme-based biomedical named entity recognition using deep learning, Bioinformatics 39, p. btad310 (2023)

    work page 2023

  20. [20]

    S. Liu, A. Wang, X. Xiu, M. Zhong, S. Wu et al., Evaluating medical entity recognition in health care: Entity model quantitative study, JMIR Medical Informatics 12, p. e59782 (2024)

    work page 2024

  21. [21]

    Karisani and E

    P. Karisani and E. Agichtein, Did you really just have a heart attack? towards robust detection of personal health mentions in social media, in Proceedings of the 2018 World Wide Web Confer- ence (WWW ’18) , (International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland, 2018)

    work page 2018

  22. [22]

    Magge, A

    A. Magge, A. Klein, A. Miranda-Escalada, M. A. Al-Garadi, I. Alimova, Z. Miftahutdinov, E. Farr´ e, S. Lima-L´ opez, I. Flores, K. O’Connor and G. Gonzalez-Hernandez, Overview of the sixth social media mining for health applications (#smm4h) shared tasks at naacl 2021, in Proceedings of the Sixth Social Media Mining for Health Workshop and Shared Task (SM...

    work page 2021

  23. [23]

    Weissenbacher, A

    D. Weissenbacher, A. Sarker, A. Magge, A. Daughton, K. O’Connor, M. Paul and G. Gonzalez, Overview of the fourth social media mining for health (smm4h) shared tasks at acl 2019, in Proceedings of the Fourth Social Media Mining for Health Applications (#SMM4H) Workshop & Shared Task , (Association for Computational Linguistics, Florence, Italy, 2019)

    work page 2019

  24. [24]

    Scepanovic, E

    S. Scepanovic, E. Martin-Lopez, D. Quercia and K. Baykaner, Extracting medical entities from social media, in Proceedings of the ACM Conference on Health, Inference, and Learning (ACM CHIL), (Association for Computing Machinery, New York, NY, USA, 2020)

    work page 2020

  25. [25]

    Opioid Named Entity Recognition (ONER-2025) from Reddit

    G. Sidorov, M. Ahmad, I. Ameer, M. Usman and I. Batyrshin, Opioid named entity recognition (oner-2025) from reddit, arXiv preprint arXiv:2504.00027 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  26. [26]

    Grattafiori, A

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan et al. , The llama 3 herd of models, arXiv preprint arXiv:2407.21783 (2024)

    Pith/arXiv arXiv 2024

  27. [27]

    G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ram´ e, M. Rivi` ereet al., Gemma 3 technical report, arXiv preprint arXiv:2503.19786 (2025)

    Pith/arXiv arXiv 2025

  28. [28]

    Hurst, A

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford et al. , Gpt-4o system card, arXiv preprint arXiv:2410.21276 (2024)

    Pith/arXiv arXiv 2024

  29. [29]

    Z. Zhan, S. Zhou, M. Li and R. Zhang, Ramie: Retrieval-augmented multi-task information extraction with large language models on dietary supplements, Journal of the American Medical Informatics Association 32, 545 (March 2025)

    work page 2025

  30. [30]

    W. Zhou, S. Zhang, Y. Gu, M. Chen and H. Poon, Universalner: Targeted distillation from large language models for open named entity recognition, arXiv preprint arXiv:2308.03279 (2023)

    Pith/arXiv arXiv 2023

  31. [31]

    C. Shyr, Y. Hu, L. Bastarache, A. Cheng, R. Hamid, P. Harris and H. Xu, Identifying and extracting rare diseases and their phenotypes with large language models, Journal of Healthcare Informatics Research 8, 438 (2024)

    work page 2024

  32. [32]

    Y. Ge, Y. Guo, S. Das, M. A. Al-Garadi and A. Sarker, Few-shot learning for medical text: A review of advances, trends, and opportunities, Journal of Biomedical Informatics 144, p. 104458 (August 2023), Epub 2023 Jul 23

    work page 2023

  33. [33]

    N. Ding, G. Xu, Y. Chen, X. Wang, X. Han, P. Xie, H.-T. Zheng and Z. Liu, Few-nerd: A few-shot named entity recognition dataset, arXiv preprint arXiv:2105.07464 (2021)

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  34. [34]

    Cohen, A coefficient of agreement for nominal scales, Educational and psychological measure- ment 20, 37 (1960)

    J. Cohen, A coefficient of agreement for nominal scales, Educational and psychological measure- ment 20, 37 (1960)

    work page 1960

  35. [35]

    A. J. Viera and J. M. Garrett, Understanding interobserver agreement: the kappa statistic, Family Medicine 37, 360 (May 2005)

    work page 2005

  36. [36]

    Q. Lu, R. Li, A. Wen, J. Wang, L. Wang and H. Liu, Large language models struggle in token- level clinical named entity recognition, in AMIA Annual Symposium Proceedings , (American Medical Informatics Association, 2025). PMID: 40417588; PMCID: PMC12099373

    work page 2025

  37. [37]

    Keraghel, S

    I. Keraghel, S. Morbieu and M. Nadif, A survey on recent advances in named entity recognition, arXiv preprint arXiv:2401.10825 (2024)

    Pith/arXiv arXiv 2024

  38. [38]

    S. Wang, X. Sun, X. Li, R. Ouyang, F. Wu, T. Zhang, J. Li and G. Wang, Gpt-ner: Named entity recognition via large language models, arXiv preprint arXiv:2304.10428 (2023)

    Pith/arXiv arXiv 2023

  39. [39]

    M. S. Obeidat, M. S. A. Nahian and R. Kavuluru, Do llms surpass encoders for biomedical ner?, arXiv preprint arXiv:2504.00664 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  40. [40]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee and K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers) , (Association for Computational Linguistics...

    work page 2019

  41. [41]

    Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer and V. Stoyanov, Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv:1907.11692 (2019)

    Pith/arXiv arXiv 1907

  42. [42]

    P. He, X. Liu, J. Gao and W. Chen, Deberta: Decoding-enhanced bert with disentangled atten- tion, arXiv preprint arXiv:2006.03654 (2020)

    Pith/arXiv arXiv 2006

  43. [43]

    J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So and J. Kang, Biobert: a pre-trained biomedical language representation model for biomedical text mining, Bioinformatics 36, 1234 (2020)

    work page 2020

  44. [44]

    T. H. Dang, H.-Q. Le, T. M. Nguyen and S. T. Vu, D3ner: biomedical named entity recog- nition using crf-bilstm improved with fine-tuned embeddings of various linguistic information, Bioinformatics 34, 3539 (2018)

    work page 2018

  45. [45]

    Y. Qiu, L. Dong, W. Zhang, H. Xing and J. Huang, A diffusion enhanced crf and bilstm frame- work for accurate entity recognition, Scientific Reports 15, p. 19670 (2025)

    work page 2025

  46. [46]

    Reimers and I

    N. Reimers and I. Gurevych, Sentence-bert: Sentence embeddings using siamese bert-networks, arXiv preprint arXiv:1908.10084 (2019)

    Pith/arXiv arXiv 1908

  47. [47]

    G. H. B. Andrade, S. Yada and E. Aramaki, Comparative evaluation of boundary-relaxed anno- tation for entity linking performance, in Proceedings of the 61st Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Papers) , (Association for Computational Linguistics, Toronto, Canada, 2023)

    work page 2023

  48. [48]

    Segura-Bedmar, P

    I. Segura-Bedmar, P. Mart ´ ınez and M. Herrero-Zazo, Semeval-2013 task 9: Extraction of drug- drug interactions from biomedical texts (ddiextraction 2013), in Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh Inter- national Workshop on Semantic Evaluation (SemEval 2013) , (Association for Computa...

    work page 2013

  49. [49]

    Z. P. Majdik, S. S. Graham, J. C. Shiva Edward, S. N. Rodriguez, M. S. Karnes, J. T. Jensen, J. B. Barbour and J. F. Rousseau, Sample size considerations for fine-tuning large language models for named entity recognition tasks: methodological study, Jmir ai 3, p. e52095 (2024)

    work page 2024

  50. [50]

    Lopez, A

    I. Lopez, A. Swaminathan, K. Vedula, S. Narayanan, F. Nateghi Haredasht, S. P. Ma, A. S. Liang, S. Tate, M. Maddali, R. J. Gallo et al. , Clinical entity augmented retrieval for clinical information extraction, npj Digital Medicine 8, p. 45 (2025)

    work page 2025

  51. [51]

    Y. Chen, T. A. Lask, Q. Mei, Q. Chen, S. Moon, J. Wang, K. Nguyen, T. Dawodu, T. Cohen, J. C. Denny et al. , An active learning-enabled annotation system for clinical named entity recognition, BMC medical informatics and decision making 17, 35 (2017)

    work page 2017

  52. [52]

    Lafferty, A

    J. Lafferty, A. McCallum and F. C. N. Pereira, Conditional random fields: Probabilistic models for segmenting and labeling sequence data, in Proceedings of the 18th International Conference on Machine Learning (ICML) , (Morgan Kaufmann, Williamstown, MA, USA, 2001)

    work page 2001

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.