Pith. sign in

REVIEW 3 major objections 5 minor 25 references

Few-Shot Biomedical Relation Extraction with Large Language Models: A Viable Alternative to Supervised Learning?

T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Prompt-based few-shot LLMs can beat supervised models on biomedical relation extraction when performance is measured per relation type.

desk verdict The formulation comparison and cost analysis are worth reading, but the 'viable alternative' claim outruns the evidence: macro-F1 is undefined, and the Association-ambiguity explanation is an untested premise. read the letter →

arxiv 2606.15412 v2 pith:XX263G64 submitted 2026-06-13 cs.CL cs.AI

Jakob Mraz , Tomaž Curk , Blaž Zupan This is my paper · ORCID
classification cs.CLcs.AI
keywords biomedicalrelationextractionfew-shotpromptinglargelanguagemodelspairwiseclassificationjointgenerationmacro-F1annotationambiguityBioREDirect
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that few-shot prompting of large language models is a viable alternative to supervised training for biomedical relation extraction, especially in low-resource settings. The authors compare two prompt designs—pairwise classification and joint generation—on the BioREDirect dataset and find a clear precision-recall trade-off. Their strongest evidence is a macro-F1 score of 0.45 for the best prompt-based model versus 0.38 for a supervised BERT baseline, with the gap driven by rare relation types. The paper argues that the apparent micro-F1 deficit (0.44 vs. 0.56) stems mostly from one ambiguously defined relation type, not from a general inability to extract relations.

What carries the argument

Prompt-based learning with two task formulations: pairwise classification, which makes one model call per entity pair, and joint generation, which extracts all relations from a chunk of entities in one call. A tunable parameter k — the number of entities per chunk in joint generation — lets the model balance precision, recall, and computational cost, since smaller chunks recover recall lost in full-document generation while retaining higher precision than pairwise classification. The macro-F1 metric is the load-bearing evaluation choice, because it weights each relation type equally rather than letting the abundant 'Association' class dominate the score.

What would settle it

Re-annotate the BioREDirect test set's Association-labeled pairs under a forced-choice protocol where annotators must pick the most specific relation among Bind, Positive Correlation, Negative Correlation, and others; if a majority of Association labels are confirmed rather than matched to the LLM's more specific predictions, then the macro-F1 argument would be weakened and the supervised baseline's micro-F1 lead would stand.

Watch

Extended reading notes

Core claim

On the BioREDirect document-level relation extraction dataset, a prompt-based few-shot LLM reaches a macro-F1 of 0.45, surpassing the supervised PubMedBERT baseline at 0.38, while its micro-F1 of 0.44 trails the baseline's 0.56. The authors attribute most of the micro-F1 gap to the 'Association' relation type, which they characterize as a fallback label used when annotators cannot assign a more specific relation. They provide a worked example where text explicitly states 'bind' and 'negative correlation' but the gold annotation says 'Association', suggesting that LLM predictions can be more specific than the schema allows. Under macro-F1, which they argue better reflects performance across i

Load-bearing premise

The claim that prompt-based models are truly competitive rests on the premise that the 'Association' relation type is an inherently ambiguous fallback label, so that macro-F1 is the more meaningful comparison and the micro-F1 gap is largely an artifact of the annotation scheme.

Editorial extensions

If this is right

  • Biomedical relation extraction can be performed without task-specific training in low-resource settings, with performance on rare relation types exceeding that of supervised models.
  • Practitioners can choose between pairwise classification (higher recall, more calls) and joint generation (higher precision, up to 25x cheaper) depending on downstream priorities.
  • The k-constrained generation setting offers a practical dial for trading off extraction coverage against computational cost.
  • Well-defined relation schemas—with distinct, unambiguous type definitions—are essential for fairly comparing LLM-based and supervised approaches.
  • If the Association-class ambiguity explanation is correct, the true capability gap between prompt-based and supervised systems is much smaller than raw micro-F1 suggests.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same macro-F1 advantage might appear in other relation extraction benchmarks with a dominant, vaguely defined 'other' class, making macro-F1 a more honest default for imbalanced extraction tasks.
  • A direct test of the paper's central explanation would be to re-annotate a sample of Association-labeled pairs with a forced-choice among the more specific relation types; if expert re-annotators often agree with the LLM's specific predictions, the micro-F1 gap would shrink.
  • The k-constrained generation idea could be generalized to other structured output tasks—such as event extraction or coreference—where a controllable chunk size trades completeness for precision at inference time.
  • Since the paper notes that reasoning helps MoE models more than dense ones, future work could investigate whether lightweight verification prompts can close the remaining micro-F1 gap without the cost of full chain-of-thought.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper evaluates few-shot prompt-based biomedical relation extraction with medium-sized open-weight LLMs on the BioREDirect dataset, comparing two task formulations: pairwise classification and joint generation (with a tunable context size k). It reports micro-F1 and macro-F1 for EP+RT extraction, per-relation results, and a token-cost analysis. The central claims are: (i) the best few-shot LLM reaches micro-F1 0.44, above previous few-shot results (0.34) but below the supervised PubMedBERT baseline (0.56); (ii) the micro-F1 gap is largely due to the ambiguous Association relation type; and (iii) under macro-F1, prompt-based LLMs outperform the supervised baseline (0.45 vs. 0.38), especially on rare relation types. The paper concludes that few-shot LLMs are a viable alternative to supervised BioRE in low-resource settings.

Significance. If the main claims hold, the paper would make a useful contribution to low-resource biomedical relation extraction by showing that prompt-based LLMs can be competitive with supervised models on a public document-level benchmark, while also providing a systematic comparison of pairwise classification and joint generation together with cost measurements. The strengths of the paper are its use of a public dataset, per-relation reporting, the k-constrained generation idea, a transparency-oriented cost model, and the open code repository. However, the central macro-F1 claim is currently undermined by an internal inconsistency between Table 2 and Table 3, and the explanation of the Association-related gap is asserted rather than quantified. The paper also explicitly acknowledges run-to-run variation but provides no confidence intervals or significance tests, which is particularly important because the macro-F1 advantage is concentrated in relation types with very few test instances (Comparison n=13, Conversion n=13). These issues are fixable within the manuscript's scope, but they are load-bearing for the 'viable alternative' conclusion.

major comments (3)
  1. [§3.4, Tables 2 and 3] The macro-F1 numbers in Table 2 are not reproducible from the per-relation F1 values in Table 3. For PubMedBERT, macro-averaging the seven relation types with test instances in Table 3 gives (0.54+0.56+0.66+0.74+0.47+0.25+0.00)/7 = 0.46, and including Drug Interaction as 0.00 gives 0.40; neither equals the reported 0.38. Similarly, for gemma-4-31B-it with k=N and reasoning enabled, Table 3 gives a macro average of about 0.53 over seven types (or 0.46 over eight), whereas Table 2 reports 0.45. The averaging rule (which classes are included, whether Drug Interaction with zero instances is counted, and whether per-class F1 is weighted by anything) must be stated explicitly and the tables reconciled. Because the headline '0.45 vs. 0.38' macro-F1 advantage is the basis for the paper's main conclusion, this inconsistency is load-bearing.
  2. [§4, Fig. 3] The claim that the micro-F1 gap is 'largely attributable to performance on the Association class' and that 'the true performance gap may be smaller' is not quantified. Association is the largest relation class (2,759 of 6,036 test instances in Table 1), so if most Association labels are correct, the 0.44 vs. 0.56 micro-F1 gap is real and the 'viable alternative' conclusion is weakened. The only evidence offered is one qualitative example (Fig. 3) and an assertion about annotation ambiguity. The paper should quantify the Association contribution directly, e.g., by reporting micro-F1 excluding Association, by measuring agreement on a re-annotated sample of Association instances, or by a counterfactual analysis that relabels borderline cases. Without such evidence, the central reinterpretation of the micro-F1 gap is unsupported.
  3. [§3.4 and Limitations] The paper acknowledges in the Limitations section that 'small variations in model outputs were observed across runs' and that 'repeated evaluations and confidence intervals should therefore be used.' However, all headline results in Tables 2 and 3 are point estimates with no confidence intervals, error bars, or significance tests. This is especially important for the macro-F1 advantage on rare relation types (Comparison n=13, Conversion n=13), where the difference between 0.45 and 0.38 may be driven by a handful of instances. Without variance estimates, the claim that prompt-based approaches 'outperform' the supervised baseline under macro-F1 is not statistically grounded.
minor comments (5)
  1. [Table 2 caption] The slash-separated values (e.g., 0.35 / 0.44) are not explained in the table caption. Stating explicitly that the two values correspond to reasoning disabled/enabled would greatly improve readability.
  2. [§4, Table 4] The text claims joint generation reduces computational costs by 'up to 25×' compared to pairwise classification. The largest ratio in Table 4 appears to be about 19.7 (Qwen3.5-27B: 33.83 / 1.72). Please specify which two configurations produce the 25× figure, or correct the claim.
  3. [§3.3] The term 'reasoning' is used throughout but not defined at first use. Clarify whether this refers to chain-of-thought prompting, an internal reasoning mode of the model, or something else.
  4. [§3.3] For k-constrained joint generation, the aggregation procedure is described as majority vote with 'ties discarded as ambiguous.' The behavior when a pair appears in multiple overlapping subsets and receives different relations is clear, but the treatment of ties between a relation and None should be stated precisely.
  5. [General] Table 1 lists Drug Interaction with a test count of 0, yet this class may be included in macro-F1 computations. State explicitly how zero-instance classes are handled in the macro average, since this affects comparability across tables.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's comparisons rest on external baselines, a public dataset, and measured LLM outputs; the macro-F1 argument is post hoc but not definitionally circular.

full rationale

The paper's central comparisons are empirical and externally grounded. The few-shot LLM results are measured on the public BioREDirect test split, and the supervised baselines (PubMedBERT, GPT-3.5 fine-tune) come from prior external work by Lai et al. [11]; no fitted parameter is renamed as a prediction, and no equation in the paper defines its outputs in terms of its inputs. The claim that the micro-F1 gap is largely attributable to the Association class is an interpretive premise supported by one qualitative example and a per-class table, not by a derivation that assumes the conclusion; it is a post hoc explanation, and a possibly weak empirical claim, but not circular. The shift from micro-F1 to macro-F1 is a metric-selection decision, not a circular reduction: macro-F1 is defined independently of the models' outputs and is not constructed from the LLM predictions. The paper also explicitly acknowledges the low counts of rare relation types and reports a reproducibility limitation, which counts against overconfidence but not against circularity. No load-bearing self-citation appears: the authors' own prior work is not invoked to justify the method or forbid alternatives. The dataset and baselines are external and independently checkable, so the derivation chain does not reduce to its own inputs.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No fitted parameters or new entities; the load-bearing choices are interpretive assumptions about the dataset and metric rather than mathematical derivations.

assumptions (3)
  • ad hoc to paper The Association relation type in BioREDirect is inherently ambiguous and masks more specific relations, so macro-F1 should be preferred for judging capability.
    Section 4 introduces this to reinterpret the micro-F1 gap; it is not tested by re-annotation or ablation.
  • domain assumption Evaluation with gold entity annotations isolates relation extraction from named entity recognition.
    Section 3.1 and 3.3 use pre-annotated entities; real pipelines would require NER, so end-to-end viability is not demonstrated.
  • standard math Exact matching against BioREDirect gold annotations is a valid evaluation of extraction quality.
    Section 3.4 defines metrics this way; this is conventional but treats the annotation scheme as ground truth, which the Association discussion partly challenges.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Few-Shot Biomedical Relation Extraction with Large Language Models: A Viable Alternative to Supervised Learning?." pith.science (2026). https://pith.science/paper/XX263G64

@misc{pith2026260615412,
  author       = {Pith},
  title        = {Pith review of: Few-Shot Biomedical Relation Extraction with Large Language Models: A Viable Alternative to Supervised Learning?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XX263G64}},
  note         = {Machine review of arXiv:2606.15412}
}
read the original abstract

Biomedical relation extraction (BioRE) is a key step in transforming biomedical literature into structured knowledge. Most existing approaches rely on supervised models trained on costly annotated datasets, limiting their scalability and adaptability across relation types and domains. We investigate few-shot BioRE using prompt-based learning with large language models (LLMs) and compare two task formulations: pairwise classification, which predicts relations for individual entity pairs, and joint generation, which extracts multiple relations in a single model call. Experiments on the BioREDirect dataset reveal a clear precision-recall trade-off. Pairwise classification achieves higher recall, whereas joint generation is more precise and computationally efficient. The best-performing model achieves a micro-F1 score of 0.44, substantially outperforming previous few-shot results (0.34) while remaining below the supervised baseline (0.56). Much of this gap is attributable to a single ambiguously defined relation type. When evaluated using macro-F1, which better captures performance across relation types in an imbalanced setting, prompt-based approaches outperform the supervised baseline (0.45 vs. 0.38), particularly on rare relation types. These findings highlight the potential of LLMs for BioRE in low-resource settings and underscore the importance of well-defined relation schemas.

Figures

Figures reproduced from arXiv: 2606.15412 by the authors.

Figure 1
Figure 1. Conversion of unstructured text into structured biomedical knowledge. A text passage is first annotated with named entities using NER (A). Next, RE is applied to identify semantic relationships between entities (B). The output is organized into a KG, where nodes represent entities and edges represent their relationships (C). The example is adapted from the BioREDirect dataset [11], which we use in our analyses. wide… view at source ↗
Figure 2
Figure 2. Comparison of BioRE task formulations for an input text with N = 5 entities. The color highlights represent the annotated entities. In the pairwise classification setting, each entity pair is evaluated independently, resulting in 10 model calls. In the joint generation setting, each call contains a subset of k entities, constructed such that every entity pair co-occurs in at least one call. With k = 3, all pairs are… view at source ↗
Figure 3
Figure 3. Example of BioREDirect annotation ambiguity: the abstract contains three entities – SNHG15 (lncRNA), miR-18a (miRNA), and CXCL13 (protein). All pairwise relations are labeled as Association. However, the text provides evidence for more specific relations. The relation between SNHG15 and miR-18a is explicitly described as Bind (“SNHG15 was found to bind to miR-18a”), while the statement “Silencing of SNHG15 led to CX… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 1 linked inside Pith

  1. [1]

    Agrawal, S

    M. Agrawal, S. Hegselmann, H. Lang, Y. Kim, and D. Sontag. Large language models are few-shot clinical information extractors. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors,Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1998–2022, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational ...

    2022

  2. [2]

    Ben Abacha and P

    A. Ben Abacha and P. Zweigenbaum. Automatic extraction of semantic relations between medical entities: a rule based approach.Journal of Biomedical Semantics, 2(5):S4, Oct. 2011

    2011

  3. [3]

    Q. Chen, Y. Hu, X. Peng, Q. Xie, Q. Jin, A. Gilson, M. B. Singer, X. Ai, P.-T. Lai, Z. Wang, V. K. Keloth, K. Raja, J. Huang, H. He, F. Lin, J. Du, R. Zhang, W. J. Zheng, R. A. Adelman, Z. Lu, and H. Xu. Benchmarking large language models for biomedical natural language processing applications and recommendations.Nature Communications, 16(1):3280, Apr. 2025

    2025

  4. [4]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In J. Burstein, C. Doran, and T. Solorio, editors,Proceedings of the 2019 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Pape...

    2019

  5. [5]

    J. A. Diaz-Garcia and J. A. D. Lopez. A survey on cutting-edge relation extraction techniquesbasedonlanguagemodels.Artificial Intelligence Review,58(9):287,July 2025

    2025

  6. [6]

    Goyal and N

    N. Goyal and N. Singh. Named entity recognition and relationship extraction for biomedical text: A comprehensive survey, recent advancements, and future research directions.Neurocomputing, 618:129171, Feb. 2025

    2025

  7. [7]

    Huang, X

    M. Huang, X. Zhu, Y. Hao, D. G. Payan, K. Qu, and M. Li. Discovering patterns to extract protein–protein interactions from full texts.Bioinformatics, 20(18):3604– 3612, Dec. 2004

    2004

  8. [8]

    Acomprehensiveevaluationof large Language models on benchmark biomedical text processing tasks.Computers in Biology and Medicine, 171:108189, Mar

    I.Jahan,M.T.R.Laskar,C.Peng,andJ.X.Huang. Acomprehensiveevaluationof large Language models on benchmark biomedical text processing tasks.Computers in Biology and Medicine, 171:108189, Mar. 2024

    2024

Show all 25 references
  1. [9]

    S. Ji, S. Pan, E. Cambria, P. Marttinen, and P. S. Yu. A Survey on Knowledge Graphs: Representation, Acquisition, and Applications.IEEE Transactions on Neural Networks and Learning Systems, 33(2):494–514, Feb. 2022. Few-Shot BioRE with LLMs: A Viable Alternative to Supervised ...

    2022

  2. [10]

    Jimenez Gutierrez, N

    B. Jimenez Gutierrez, N. McNeal, C. Washington, Y. Chen, L. Li, H. Sun, and Y. Su. Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors,Findings of the Associ- ation for Computational Linguistics: EMNLP 202...

    2022

  3. [11]

    Lai, C.-H

    P.-T. Lai, C.-H. Wei, S. Tian, R. Leaman, and Z. Lu. Enhancing biomedical relation extraction with directionality.Bioinformatics, 41(Supplement_1):i68–i76, July 2025

    2025

  4. [12]

    Liu and S

    Y. Liu and S. Zhu. A comprehensive evaluation of document level biomedical rela- tionextractionusinglargelanguagemodels. In2025 IEEE International conference on Medical Artificial Intelligence (MedAI), pages 119–123, Nov. 2025

    2025

  5. [13]

    Novoa, M

    J. Novoa, M. Chagoyen, C. Benito, F. J. Moreno, and F. Pazos. PMIDigest: Interactive Review of Large Collections of PubMed Entries to Distill Relevant Information.Genes, 14(4):942, Apr. 2023

    2023

  6. [14]

    C. Peng, X. Yang, K. E. Smith, Z. Yu, A. Chen, J. Bian, and Y. Wu. Model tuning or prompt Tuning? a study of large language models for clinical concept and relation extraction.Journal of Biomedical Informatics, 153:104630, May 2024

    2024

  7. [15]

    Y. Peng, A. Rios, R. Kavuluru, and Z. Lu. Extracting chemical–protein relations with ensembles of SVM and deep learning models.Database, 2018:bay073, Jan. 2018

    2018

  8. [16]

    Shang, Y

    Y. Shang, Y. Guo, S. Hao, and R. Hong. Biomedical Relation Extraction via Adaptive Document-Relation Cross-Mapping and Concept Unique Identifier, Jan

  9. [17]

    Sänger, S

    M. Sänger, S. Garda, X. D. Wang, L. Weber-Genzel, P. Droop, B. Fuchs, A. Akbik, and U. Leser. HunFlair2 in a cross-corpus evaluation of biomedical named entity recognition and normalization tools.Bioinformatics, 40(10):btae564, Oct. 2024

    2024

  10. [18]

    Q. Wei, Z. Ji, Y. Si, J. Du, J. Wang, F. Tiryaki, S. Wu, C. Tao, K. Roberts, and H. Xu. Relation Extraction from Clinical Narratives Using Pre-trained Language Models.AMIA Annual Symposium Proceedings, 2019:1236–1245, Mar. 2020

    2019

  11. [19]

    D. Xu, W. Chen, W. Peng, C. Zhang, T. Xu, X. Zhao, X. Wu, Y. Zheng, Y. Wang, and E. Chen. Large language models for generative information extraction: a survey.Frontiers of Computer Science, 18(6):186357, Nov. 2024

    2024

  12. [20]

    Y. Yao, D. Ye, P. Li, X. Han, Y. Lin, Z. Liu, Z. Liu, L. Huang, J. Zhou, and M. Sun. DocRED: A Large-Scale Document-Level Relation Extraction Dataset. InA.Korhonen,D.Traum,andL.Màrquez,editors,Proceedings of the 57th Annual Meeting of the Association for Computational Linguist...

    2019

  13. [21]

    Zhang, M

    J. Zhang, M. Wibert, H. Zhou, X. Peng, Q. Chen, V. K. Keloth, Y. Hu, R. Zhang, H. Xu, and K. Raja. A Study of Biomedical Relation Extraction Using GPT Models.AMIA Summits on Translational Science Proceedings, 2024:391–400, May 2024

    2024

  14. [22]

    D. Zhao, J. Wang, H. Lin, X. Wang, Z. Yang, and Y. Zhang. Biomedical cross- sentence relation extraction via multihead attention and graph convolutional net- works.Applied Soft Computing, 104:107230, June 2021

    2021

  15. [23]

    L. Zhao, L. Kang, and Q. Guo. Zero-shot document-level biomedical relation ex- traction via scenario-based prompt design in two-stage with LLM.Computational Biology and Chemistry, 123:108978, Aug. 2026

    2026

  16. [24]

    X. Zhao, Y. Deng, M. Yang, L. Wang, R. Zhang, H. Cheng, W. Lam, Y. Shen, and R. Xu. A Comprehensive Survey on Relation Extraction: Recent Advances and New Frontiers.ACM Comput. Surv., 56(11):293:1–293:39, July 2024

    2024

  17. [2025]

    arXiv:2501.05155 [cs]

    arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.