Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Improving Automatic Evaluation of Large Language Models (LLMs) in Biomedical Relation Extraction via LLMs-as-the-Judge

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper shows that LLM-as-judge evaluation of biomedical relation extraction fails below 50% accuracy on unstructured generator outputs and that JSON-structured output plus domain-adaptation fine-tuning recovers about 15 points of…

desk verdict Solid first benchmark of LLM-as-judge for biomedical relation extraction; the structured-output finding is likely real, but the headline 15% figure is overclaimed and the human labels need validation. read the letter →

arxiv 2506.00777 v1 pith:264EE335 submitted 2025-06-01 cs.CL

Md Tahmid Rahman Laskar , Israt Jahan , Elham Dolatabadi , Chun Peng , Enamul Hoque , Jimmy Huang This is my paper
classification cs.CL
keywords biomedicalrelationextractionLLM-as-judgestructuredoutputformattingdomainadaptationtransferlearningevaluationbenchmarklargelanguagemodelsJSON
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether LLM-as-judge evaluation, which works reasonably in general text generation, can be trusted for biomedical relation extraction. The authors benchmark eight judges against responses from five generators on three datasets and find that most judges fall below 50% accuracy, far short of human evaluation. They argue the main cause is that generator outputs do not follow a standard format, and they show that forcing JSON-structured output raises judge accuracy by about 15% on average across datasets. A domain-adaptation fine-tuning step then lets small open-weight judges approach or exceed closed-source zero-shot judges, making automated evaluation practical for a task where human annotation is slow and costly.

What carries the argument

The mechanism is a pair of formatting and training interventions. Structured output formatting requires the LLM-generator to emit predicted relations as JSON, so each drug-target or drug-disease pair is machine-readable and the judge sees clearly separated items. The judge prompt asks the LLM to output JSON with two keys, correctly_predicted_relations and total_predicted_relations, comparing AI predictions against gold relations; exact-match accuracy and RMSE between judge output and human labels measure the judge. Domain adaptation fine-tunes an open-source judge on a small amount of human-annotated judgment data from a different dataset, transferring the skill of counting correct and total relations from one relation type to another under low-resource conditions.

What would settle it

An independent blinded study in which annotators label the same generator responses without knowing which condition is structured, with inter-annotator agreement reported, would settle the claim: if the structured-vs-unstructured accuracy gap disappears or shrinks below statistical significance, the paper's central conclusion fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLM-based judges are not inherently unsuitable for biomedical relation extraction; they are defeated by the unstructured, paraphrase-heavy responses that relation-extraction LLMs produce, and this failure is largely fixable. In the paper's terms, structured output formatting for LLM-generated responses consistently improves judge performance by about 15% on average, with statistically significant gains (p < 0.05) across BC5CDR, DDI, and KD-DTI. The authors further claim that fine-tuning an open-source judge on human-annotated judgment data from one relation-extraction dataset transfers well to another, beating zero-shot closed-source judges in their experiments. On the paper's account, the evaluation bottleneck is not domain knowledge in the judge but format alignment between what generators emit and what judges are asked to count.

Load-bearing premise

The human-annotated gold labels against which every judge accuracy is measured were created by two of the paper's authors without reported inter-annotator agreement, so any bias in those labels, especially in favor of structured outputs, directly inflates the claimed improvement.

Editorial extensions

If this is right

  • Biomedical relation extraction pipelines can replace human evaluation with LLM judges once generator output is structured, cutting cost and turnaround time.
  • Relation extraction benchmarks should standardize JSON output, because format variation alone is shown to cost judges roughly 15 points in accuracy.
  • Open-source judges fine-tuned on a few hundred annotated judgment samples can match or beat proprietary zero-shot judges, lowering deployment cost.
  • The released 36k-sample judgment dataset gives the community a training and calibration resource for future biomedical evaluation models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the 15% gain may not come from JSON semantics but from parseable, enumerated output; a direct test using a different systematic delimiter format would clarify the mechanism.
  • The transfer-learning result suggests counting relations is a portable skill across relation types; a natural extension would test whether the same judge adapts to other structured biomedical tasks such as event extraction or phenotype annotation.
  • The error analysis indicates even the best judge conflates distinct drugs and merges multiple side effects; future judging protocols should therefore report entity-level agreement in addition to aggregate counts, which the paper does not implement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies LLMs-as-the-judge for biomedical relation extraction. It benchmarks eight LLM judges on responses generated by five LLM generators across three datasets (BC5CDR, DDI, KD-DTI), using exact-match accuracy and RMSE against human gold labels for the two judgment counts (correctly predicted relations and total predicted relations). The paper reports that zero-shot LLM judges perform poorly on the unstructured generator outputs from Jahan et al. (2024), proposes structured JSON formatting of generator responses as a remedy, claims an average improvement of about 15%, and introduces a domain-adaptation fine-tuning approach that transfers judgment knowledge between datasets. The authors release 36k human- and LLM-annotated judgment samples.

Significance. If the findings hold, the paper makes a useful empirical contribution: it is the first systematic benchmark of LLM judges for biomedical relation extraction, it identifies a simple and cheap intervention (structured JSON output) with a consistent directional effect across all 24 judge/dataset cells, and it shows that fine-tuned open-source judges can approach or exceed closed-source zero-shot judges. The public release of the judgment data is a valuable resource for follow-up work. However, the central quantitative claims are conditioned on two load-bearing weaknesses: the gold labels for the structured-versus-unstructured comparison were produced by two authors without reported inter-annotator agreement or blinding, and the comparison uses a single generator (GPT-4-Turbo). These issues make the exact magnitude of the reported gains uncertain, even though the direction of the effect is credible.

major comments (3)
  1. [Abstract; §5.2, Table 2] The abstract's claim of 'about 15% (on average)' improvement is not supported by Table 2. Averaging the 24 structured/unstructured exact-match accuracy pairs gives a mean gain of about 10.6 percentage points (from roughly 40.7% to 51.3%); per-dataset averages are about 23.4 points for BC5CDR, 4.4 for DDI, and 3.9 for KD-DTI. If a relative gain were intended, 10.6/40.7 is about 26%, not 15%. The direction of the effect is consistent—all 24 cells show a positive EM gain—but the headline magnitude is overstated and should be corrected or replaced with a range based on Table 2.
  2. [§5.2; Ethical Considerations] The gold labels used for the central structured-versus-unstructured comparison were created by two of the authors, who are not reported to be blind to the output-format condition, with disagreements resolved through discussion and no inter-annotator agreement statistic reported. The Ethical Considerations section confirms that no additional compensation was needed because two authors conducted the annotation. Since every judge-accuracy number in Tables 1–3 and the claimed average gain is measured against these labels, a systematic annotation asymmetry between the structured and unstructured conditions could inflate the reported improvement. I am not questioning the authors' intent; the method section needs inter-annotator agreement (e.g., Cohen's or Fleiss' kappa), a blinded or independently generated annotation protocol, or an external validation set before the headline numbers can be taken at face value.
  3. [§4.2, §5.2] The structured-versus-unstructured comparison is conducted with GPT-4-Turbo as the only generator. The paper justifies this by noting that earlier LLMs fail to follow structured-format instructions, but the abstract and contributions state the benefit of structured formatting as a general phenomenon. With a single generator, the gains could be idiosyncratic to GPT-4-Turbo's output quality or format-following behavior rather than to structured formatting per se. The paired t-test in §5.2 is over judges, not generators, so it does not address generator variability. The claim should be restricted to GPT-4-Turbo-generated responses, or the experiments extended to at least one additional modern generator on one dataset, before a general conclusion is drawn.
minor comments (5)
  1. [§3.2] The text 'Y AML' should read 'YAML'.
  2. [§5.2, Table 2] The sentence 'structured formatting consistently improves the performance for all models' is too strong: for Claude-3-Haiku on DDI, the structured RMSE (3.01) is worse than the unstructured RMSE (2.80). The claim should be limited to exact-match accuracy, or qualified as 'for EM accuracy and for RMSE in most cases.'
  3. [§5.2] The paired t-test is reported only as p < 0.05; the test statistic, the exact p-value, and the pairing unit (judges) should be reported, ideally with a multiple-comparison note given that two metrics and three datasets are tested.
  4. [§4.4] The inference temperature is set to 1.0 for all models, which introduces sampling variability in the judge outputs; the paper does not report how many runs or seeds were used to stabilize the exact-match and RMSE estimates. Reporting variance or running each evaluation multiple times would strengthen the comparison.
  5. [Table 3] The 'Data' column in Table 3 is ambiguous: it should clearly label which dataset is used for fine-tuning and which for evaluation (e.g., 'Fine-tune: BC5CDR → Evaluate: KD-DTI'), and the zero-shot baselines quoted in parentheses should be explicitly linked to the structured columns of Table 2.

Circularity Check

0 steps flagged · score 1.0 of 10

No derivation reduces to its own inputs; reported accuracies are empirical measurements against human gold labels, so no circular step is identified.

full rationale

The paper's central claims—that LLM-based judges are unreliable in biomedical relation extraction and that structured generator output improves judge accuracy—are empirical measurements rather than derivations from assumed conclusions. Table 1 evaluates judge annotations against human labels reused from Jahan et al. (2024), while Table 2 compares structured and unstructured GPT-4-Turbo responses using freshly produced human annotations described in Section 5.2. The structured-output improvement could have failed to materialize; it is not forced by the definition of structured output or by the exact-match metric. Similarly, the domain-adaptation result in Table 3 fine-tunes on one dataset and evaluates on another, which is a held-out evaluation rather than a fitted prediction. The only notable concerns are validity concerns rather than circularity: the Section 5.2 gold labels were created by two of the authors, with disagreements resolved by discussion and no inter-annotator agreement statistic reported, as disclosed in the Ethical Considerations. That could bias the measured improvement if annotation was not condition-blind, but it does not make the result equivalent to the paper's own inputs by construction. Likewise, the zero-shot benchmark reuses responses and labels from Jahan et al. (2024), a heavily overlapping prior paper, but this is data reuse with externally checkable artifacts rather than a load-bearing self-citation used to forbid alternatives. Because no equation is reduced to another equation and no fitted parameter is renamed as a prediction, no circular step is identified. The score of 1 reflects minor self-reliance on the authors' own prior dataset and labels, not a logical circle.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

This is an empirical benchmark, so there are no fitted model parameters in a derivation. The listed free parameters are hand-chosen experimental settings. The claims rely on the human annotations and benchmark gold labels as ground truth. No new entities are introduced.

free parameters (4)
  • LLM inference temperature = 1.0
    Set to 1.0 for generator and judge decoding to allow diverse outputs; chosen by hand with no sensitivity analysis.
  • Fine-tuning learning rate = 2e-5
    Used for Qwen and Phi domain-adaptation fine-tuning; no sensitivity analysis reported (Section 4.4).
  • Fine-tuning epochs = 3
    Fixed at 3 epochs for domain adaptation; no early stopping or schedule described.
  • Maximum output tokens = 300 (1000 for reasoning models)
    Cap on judge output length; reasoning models get 1000 tokens for thinking.
assumptions (3)
  • domain assumption The human annotations by the two author-annotators are treated as ground truth for judge accuracy.
    The entire evaluation uses these labels as gold; no inter-annotator agreement or external validation is reported (Section 5.2).
  • domain assumption The three benchmark datasets and their gold relation annotations are reliable and correctly interpreted.
    The paper uses BC5CDR, DDI, and KD-DTI without independent verification of their gold labels.
  • domain assumption All reported LLM-judge outputs were correctly parsed into the required JSON counts.
    BioMistral was excluded for format failures (Appendix A.4.1), but the paper does not report parse-failure rates for the eight judges used.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Automatic Evaluation of Large Language Models (LLMs) in Biomedical Relation Extraction via LLMs-as-the-Judge." pith.science (2026). https://pith.science/paper/264EE335

@misc{pith2026250600777,
  author       = {Pith},
  title        = {Pith review of: Improving Automatic Evaluation of Large Language Models (LLMs) in Biomedical Relation Extraction via LLMs-as-the-Judge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/264EE335}},
  note         = {Machine review of arXiv:2506.00777}
}
read the original abstract

Large Language Models (LLMs) have demonstrated impressive performance in biomedical relation extraction, even in zero-shot scenarios. However, evaluating LLMs in this task remains challenging due to their ability to generate human-like text, often producing synonyms or abbreviations of gold-standard answers, making traditional automatic evaluation metrics unreliable. On the other hand, while human evaluation is more reliable, it is costly and time-consuming, making it impractical for real-world applications. This paper investigates the use of LLMs-as-the-Judge as an alternative evaluation method for biomedical relation extraction. We benchmark 8 LLMs as judges to evaluate the responses generated by 5 other LLMs across 3 biomedical relation extraction datasets. Unlike other text-generation tasks, we observe that LLM-based judges perform quite poorly (usually below 50% accuracy) in the biomedical relation extraction task. Our findings reveal that it happens mainly because relations extracted by LLMs do not adhere to any standard format. To address this, we propose structured output formatting for LLM-generated responses that helps LLM-Judges to improve their performance by about 15% (on average). We also introduce a domain adaptation technique to further enhance LLM-Judge performance by effectively transferring knowledge between datasets. We release both our human-annotated and LLM-annotated judgment data (36k samples in total) for public use here: https://github.com/tahmedge/llm_judge_biomedical_re.

Figures

Figures reproduced from arXiv: 2506.00777 by the authors.

Figure 1
Figure 1. An example of human evaluation of LLM￾generated outputs in comparison to the gold labels. The drug “dex” in the gold relation is predicted as “dexam￾ethasone” in LLM response. While exact match will fail in this case, human evaluators can recognize. the volume of unstructured biomedical text grows exponentially, efficient and accurate relation ex￾traction is no longer a convenience but a necessity for advancing medi… view at source ↗
Figure 2
Figure 2. Average judgment accuracy based on our eval [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. An Overview of LLM-based Relation Extraction System. After constructing the input prompt for the [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: An example of the LLM-generated outputs for [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Performance comparison between Qwen mod [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Comparing fine-tuned smaller models, Qwen [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Our Evaluation Process where both the (c) [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A three-party-decoupled multi-turn chat benchmark finds closed and open models nearly tied on subjective empathy/persona scores but separated by up to 9× on objective intent tracking, with reasoning and persona format...

Reference graph

Works this paper leans on

55 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [1]

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    arXiv 2024

  2. [2]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  3. [3]

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. arXiv preprint arXiv:2305.10403

    arXiv 2023

  4. [4]

    Anthropic. 2023. https://www-cdn.anthropic.com/files/4zrzovbb/website/bd2a28d2535bfb0494cc8e2a3bf135d2e7523226.pdf Model card and evaluations for claude models

    work page 2023

  5. [5]

    Anthropic. 2024. https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf The claude 3 model family: Opus, sonnet, haiku

    2024

  6. [6]

    Do, Yan Xu, and Pascale Fung

    Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. https://doi.org/10.18653/v1/2023.ijcnlp-main.45 A multitask, multilingual, multimodal evaluation of C hat GPT on reasoning, hallucination, and interactivity . In Proceedings of the 13...

    doi:10.18653/v1/2023.ijcnlp-main.45 2023

  7. [7]

    Elisa Bassignana and Barbara Plank. 2022. https://doi.org/10.18653/v1/2022.acl-srw.7 What do you mean by relation extraction? a survey on datasets and study on scientific relation classification . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 67--83, Dublin, Ireland. Associatio...

    doi:10.18653/v1/2022.acl-srw.7 2022

  8. [8]

    Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, et al. 2024. Lessons from the trenches on reproducible evaluation of language models. arXiv preprint arXiv:2405.14782

    arXiv 2024

Show all 55 references
  1. [9]

    Qijie Chen, Haotong Sun, Haoyang Liu, Yinghui Jiang, Ting Ran, Xurui Jin, Xianglu Xiao, Zhimin Lin, Hongming Chen, and Zhangmin Niu. 2023. An extensive benchmark study on biomedical text generation and mining with chatgpt. Bioinformatics, 39(9):btad557

    2023

  2. [10]

    Xue-Yong Fu, Md Tahmid Rahman Laskar, Elena Khasanova, Cheng Chen, and Shashi Bhushan TN. 2024. Tiny titans: Can smaller large language models punch above their weight in the real world for meeting summarization? arXiv preprint arXiv:2402.00841

    2024 arXiv

  3. [11]

    Siddhant Garg, Thuy Vu, and Alessandro Moschitti. 2020. Tanda: Transfer and adapt pre-trained transformer models for answer sentence selection. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7780--7788

    2020

  4. [12]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  5. [13]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo. 2024. https://arxiv.org/abs/2411.15594 A survey on llm-as-a-judge . Preprint, arXiv:2411.15594

    2024 arXiv

  6. [14]

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C \'e sar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. 2023. Textbooks are all you need. arXiv preprint arXiv:2306.11644

    2023 arXiv

  7. [15]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    2025

  8. [16]

    Mar \' a Herrero-Zazo, Isabel Segura-Bedmar, Paloma Mart \' nez, and Thierry Declerck. 2013. The ddi corpus: An annotated corpus with pharmacological substances and drug--drug interactions. Journal of biomedical informatics, 46(5):914--920

    2013

  9. [17]

    Yutai Hou, Yingce Xia, Lijun Wu, Shufang Xie, Yang Fan, Jinhua Zhu, Tao Qin, and Tie-Yan Liu. 2022. Discovering drug--target interaction knowledge from biomedical literature. Bioinformatics, 38(22):5100--5107

    2022

  10. [18]

    Israt Jahan, Md Tahmid Rahman Laskar, Chun Peng, and Jimmy Huang. 2023. Evaluation of C hat GPT on biomedical tasks: A zero-shot comparison with fine-tuned generative transformers. In The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 32...

    2023

  11. [19]

    Israt Jahan, Md Tahmid Rahman Laskar, Chun Peng, and Jimmy Xiangji Huang. 2024. A comprehensive evaluation of large language models on benchmark biomedical text processing tasks. Computers in Biology and Medicine, page 108189

    2024

  12. [20]

    AQ Jiang, A Sablayrolles, A Mensch, C Bamford, DS Chaplot, D de las Casas, F Bressand, G Lengyel, G Lample, L Saulnier, et al. 2023. Mistral 7b (2023). arXiv preprint arXiv:2310.06825

    2023 arXiv

  13. [21]

    Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. 2023. Prometheus: Inducing fine-grained evaluation capability in language models. In The Twelfth International Conference on Learning Repr...

    2023

  14. [22]

    Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. Prometheus 2: An open source language model specialized in evaluating other language models. In Proceedings of the 2024 Confer...

    2024

  15. [23]

    Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Micka \"e l Rouvier, and Richard Dufour. 2024. Biomistral: A collection of open-source pretrained large language models for medical domains. In Findings of the Association for Computational Linguistics ACL 2...

    2024

  16. [24]

    Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, and Jimmy Huang. 2024 a . https://aclanthology.org/2024.emnlp-main.764 A ...

    2024

  17. [25]

    Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Huang. 2023 a . A systematic study and comprehensive evaluation of C hat GPT on benchmark datasets. In Findings of the Association for Computational Linguistics: ACL 2023, p...

    2023

  18. [26]

    Md Tahmid Rahman Laskar, Xue-Yong Fu, Cheng Chen, and Shashi Bhushan TN. 2023 b . https://doi.org/10.18653/v1/2023.emnlp-industry.33 Building real-world meeting summarization systems using large language models: A practical perspective . In Proceedings of the 2023 Conference o...

    2023 doi

  19. [27]

    Md Tahmid Rahman Laskar, Enamul Hoque, and Jimmy Xiangji Huang. 2022. Domain adaptation with pre-trained transformers for query-focused abstractive text summarization. Computational Linguistics, 48(2):279--320

    2022

  20. [28]

    Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Ridwan Mahbub, Ahmed Masry, Mizanur Rahman, Amran Bhuiyan, Mir Tafseer Nayeem, Shafiq Joty, Enamul Hoque, and Jimmy Huang. 2025. Judging the judges: Can large vision-language models fairly evaluate chart comprehension and reasoni...

    2025 arXiv

  21. [29]

    Md Tahmid Rahman Laskar, Elena Khasanova, Xue-Yong Fu, Cheng Chen, and Shashi Bhushan Tn. 2024 b . https://aclanthology.org/2024.emnlp-industry.86 Query- OPT : Optimizing inference of large language models via multi-query instructions in meeting summarization . In Proceedings ...

    2024

  22. [30]

    Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024 a . Llms-as-judges: a comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579

    2024 arXiv

  23. [31]

    Jiao Li, Yueping Sun, Robin J Johnson, et al. 2016. Biocreative v cdr task corpus: a resource for chemical disease relation extraction. Database, 2016

    2016

  24. [32]

    Yinghao Li, Rampi Ramprasad, and Chao Zhang. 2024 b . https://doi.org/10.18653/v1/2024.findings-emnlp.295 A simple but effective approach to improve structured language model output for information extraction . In Findings of the Association for Computational Linguistics: EMNL...

    2024 doi

  25. [33]

    Zhiyong Lu, Yifan Peng, Trevor Cohen, Marzyeh Ghassemi, Chunhua Weng, and Shubo Tian. 2024. Large language models in biomedicine and health: current research landscape and future directions. Journal of the American Medical Informatics Association, 31(9):1801--1811

    2024

  26. [34]

    Renqian Luo, Liai Sun, Yingce Xia, et al. 2022. BioGPT: generative pre-trained transformer for biomedical text generation and mining . Briefings in Bioinformatics, 23(6)

    2022

  27. [35]

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. Large language models: A survey. arXiv preprint arXiv:2402.06196

    2024 arXiv

  28. [36]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3...

    2022

  29. [37]

    Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general-purpose natural language processing task solver? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1339--1384

    2023

  30. [38]

    Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulczyn, Fan Zhang, Tim Strother, Chunjong Park, Elahe Vedadi, et al. 2024. Capabilities of gemini models in medicine. arXiv preprint arXiv:2404.18416

    2024 arXiv

  31. [39]

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2023. Large language models encode clinical knowledge. Nature, pages 1--9

    2023

  32. [40]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  33. [41]

    Shubo Tian, Qiao Jin, Lana Yeganova, Po-Ting Lai, Qingqing Zhu, Xiuying Chen, Yifan Yang, Qingyu Chen, Won Kim, Donald C Comeau, et al. 2024. Opportunities and challenges for chatgpt and large language models in biomedicine and health. Briefings in Bioinformatics, 25(1):bbad493

    2024

  34. [42]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 a . Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  35. [43]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  36. [44]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  37. [45]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical met...

    2020

  38. [46]

    Congying Xia, Chen Xing, Jiangshu Du, Xinyi Yang, Yihao Feng, Ran Xu, Wenpeng Yin, and Caiming Xiong. 2024. https://doi.org/10.18653/v1/2024.acl-long.40 FOFO : A benchmark to evaluate LLM s' format-following capability . In Proceedings of the 62nd Annual Meeting of the Associa...

    2024 doi

  39. [47]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024 a . Qwen2 technical report. arXiv preprint arXiv:2407.10671

    2024 arXiv

  40. [48]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024 b . Qwen2.5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  41. [49]

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. 2024 c . Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122

    2024 arXiv

  42. [50]

    Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. 2023. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792

    2023

  43. [51]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223

    2023 arXiv

  44. [52]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

  45. [53]

    Zexuan Zhong and Danqi Chen. 2021. A frustratingly easy approach for entity and relation extraction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 50--61

    2021

  46. [54]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  47. [55]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.