Zero-Shot End-to-End Relation Extraction in Chinese: A Comparative Study of Gemini, LLaMA and ChatGPT

Chuanqi Shi; Hang Zhang; Shaoshuai Du; Xinyu Qiu; Yanxin Shen; Yixian Shen; Yiyi Tao

REVIEW 5 major objections 5 minor 17 references

Zero-Shot End-to-End Relation Extraction in Chinese: A Comparative Study of Gemini, LLaMA and ChatGPT

T0 review · 5 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper reports that gpt-4-turbo achieves the top joint F1 score of 0.367 for zero-shot end-to-end Chinese relation extraction, while Gemini flash-8b offers the lowest latency at 0.57 seconds and LLaMA trails on both.

desk verdict The accuracy ranking is plausible but the paper gives you no way to verify it: the scorer is undefined, the sample is unspecified, and one F1 row is arithmetically wrong. read the letter →

arxiv 2502.05694 v1 pith:ROBHX4VI submitted 2025-02-08 cs.CL cs.AIcs.LG

Shaoshuai Du , Yiyi Tao , Yixian Shen , Hang Zhang , Yanxin Shen , Xinyu Qiu , Chuanqi Shi This is my paper · ORCID

classification cs.CLcs.AIcs.LG

keywords zero-shotrelationextractionend-to-endChineseNLPlargelanguagemodelsChatGPTGeminiLLaMAsemanticmatching

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish a three-way comparison of large language model families on zero-shot end-to-end relation extraction in Chinese, a task where the model must find entities and their relations from raw sentences without any annotated training examples or pre-identified entities. It reports that OpenAI's gpt-4-turbo gives the best accuracy with a joint F1 of 0.367, that Gemini's gemini-1.5-flash-8b is fastest at 0.57 seconds per example, and that LLaMA models trail on both axes. A practitioner would care because the results amount to a model-selection guide: choose gpt-4-turbo for accuracy-sensitive extraction and Gemini flash models for latency-sensitive applications. The paper also claims that LLaMA's underperformance indicates a need for further adaptation before open-weight models are ready for this task.

What carries the argument

The central machinery is an API-based zero-shot prompting workflow combined with a semantic matching evaluator. A fixed Chinese instruction asks the model to output subject-predicate-object triples for raw sentences from DuIE 2.0, and semantic matching, rather than exact string matching, decides whether an extracted triple counts as correct. This design carries the entire comparison: the same prompt-and-scoring pipeline applied to 12 models is what makes the accuracy ranking in Table II and the latency ranking in Figure 2 interpretable as a model-to-model comparison.

What would settle it

Publish the exact prompt template, decoding parameters, test sample, and scoring code used for Table II; if an independent run with those settings does not reproduce gpt-4-turbo's F1 of 0.367 as the top score and gemini-1.5-flash-8b's 0.57 seconds as the lowest latency, the claimed ranking is wrong. A quicker check is to replace the semantic-matching scorer with exact string matching: if the model ordering changes, then the reported ranking reflects the scorer rather than extraction ability.

Watch

Extended reading notes

Core claim

The paper's central empirical finding is a consistent accuracy-latency trade-off across 12 models from three families on the DuIE 2.0 benchmark under a zero-shot, end-to-end protocol. Across all models, gpt-4-turbo reaches the highest joint F1 of 0.367, gpt-4 reaches the highest precision of 0.363, and gpt-4o-mini reaches the highest recall of 0.432. Gemini models are the fastest, with gemini-1.5-flash-8b at 0.57 seconds and gemini-1.5-flash at 0.91 seconds, while their best F1 is 0.309 for gemini-1.5-pro. LLaMA models, including the 405b variant, achieve at most 0.178 F1. Even the best model stays below 0.37 F1, so zero-shot Chinese end-to-end relation extraction is shown to be workable but far from solved.

Load-bearing premise

The ranking rests on the assumption that every model was measured under the same prompt, decoding settings, test sentences, and semantic-matching rule, but the paper does not report those details.

Editorial extensions

If this is right

If the ranking holds, practitioners should choose gpt-4-turbo when extraction quality dominates and gemini-1.5-flash-8b when response time dominates.
A best F1 of 0.367 implies that zero-shot Chinese end-to-end relation extraction is not yet reliable enough for high-precision production use without further tuning or few-shot examples.
Gemini flash models show that low-latency Chinese relation extraction is achievable, but only by accepting a drop in accuracy relative to OpenAI's top models.
Even the largest LLaMA models are not competitive on this task without adaptation, so open-weight alternatives need targeted Chinese relation-extraction training.
The accuracy-latency trade-off made explicit here can guide real-time system design, where sub-second inference may outweigh a moderate F1 loss.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A prompt-variation sweep would test the paper's implicit assumption that one Chinese prompt template works fairly across all three API families, since zero-shot Chinese extraction is known to be prompt-sensitive.
The semantic matching scorer is described only by reference, so quantifying its tolerance would make the absolute F1 values reproducible and comparable with string-based baselines.
Because all calls go through provider-serving APIs, the reported latencies mix model computation with scheduling and network overhead; a controlled local deployment study would isolate model speed.
A direct next experiment the paper does not run is to split DuIE 2.0 by relation type and check whether the model ranking changes on long-tail relations, a direction its conclusion already gestures toward.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a useful comparison to have, but the current write-up doesn't back it up. The authors benchmark 12 model configurations on zero-shot end-to-end Chinese RE using DuIE 2.0 and report F1 plus latency. That's new — nothing in their cited prior work compares these three families on this task and dataset. The latency numbers are a nice addition for practitioners.

The good parts: the paper is honest about scope, it cites the relevant prior work on zero-shot RE and the Wu et al. medical KG paper, and the main ranking (ChatGPT best accuracy, Gemini fastest, LLaMA weak) is plausible and internally consistent with the larger pattern of reported numbers.

The soft spots are real and load-bearing. Section III-B says semantic matching was used but doesn't say what it actually was: no embedding model, no threshold, no component-level vs sentence-level logic. Section IV-A says 'latest APIs' with no prompt template, no temperature, no max tokens, no sample size, no repeated runs. The differences between the top models are small (0.367 vs 0.343 vs 0.309), so any of these could flip the ordering. The stress-test note about the scoring rule is exactly right.

There's also a concrete red flag in Table II: gemini-1.5-flash is listed with P=0.231, R=0.424, F1=0.293, but the harmonic mean is 0.299. That's not a rounding artifact at three decimals. It means the reported F1 was not computed from the reported precision and recall, which casts doubt on the rest of the table.

Bottom line: this paper is not ready as is. It could be made reproducible in a revision — release the prompt, the sample, the parsing and scoring code, and confidence intervals. As published, it's a reference point you can't verify. I'd send it to peer review only with major revision, because the comparison is worth having if it's done properly. I wouldn't cite it in its current form.

Recommendation: desk reject or major revision? I'd lean toward sending it back for major revision with a request for full artifacts and a fixed table. It's a legitimate empirical question and the authors are close to having a useful benchmark.

Referee Report

5 major / 5 minor

Summary. This paper compares 12 large language models (four OpenAI, four Google Gemini, four Meta LLaMA) on zero-shot end-to-end relation extraction from Chinese text using the DuIE 2.0 benchmark. It reports joint precision, recall, and F1 for each model, along with API latency measurements. The central claims are that gpt-4-turbo achieves the highest F1 (0.367), Gemini models provide the lowest latency (gemini-1.5-flash-8b at 0.57 s), and LLaMA models underperform on both accuracy and speed. The evaluation uses a prompt shown in Figure 1 and a 'semantic matching' method described only at a high level in Section III-B. The paper concludes with a discussion of accuracy-latency trade-offs and argues for further adaptation of smaller or alternative models for Chinese RE.

Significance. The paper addresses a genuinely underexplored task—zero-shot end-to-end relation extraction in Chinese—and provides a broad comparison of 12 models, including latency, which is rare in the RE literature. If the reported numbers are reliable, the ranking gives practitioners a model-selection guide and motivates further work on adapting smaller or alternative models. The authors also explicitly propose a semantic-matching evaluation approach to accommodate paraphrase variation, which is a useful contribution. However, the significance is conditional on the reproducibility of Table II and Figure 2, because the evaluation protocol is not fully documented; the disclosed details are insufficient to rule out that the differences between the top models (F1 0.367, 0.343, 0.309) arise from scoring choices or prompt/decoding variations rather than genuine model capability.

major comments (5)

[Table II, Section IV-B] The reported F1 for gemini-1.5-flash is 0.293, but the harmonic mean of the reported precision 0.231 and recall 0.424 is 2*0.231*0.424/(0.231+0.424) = 0.299. This internal inconsistency indicates that at least one of the three metrics is not computed from the same underlying counts, and it undermines confidence in all of the table's numbers. Please verify every F1 value against the raw counts and report the counts (TP, FP, FN) or provide the evaluation script.
[Section III-A, Section IV-A] The evaluation protocol is underspecified to the point of being non-reproducible. The paper shows one example prompt in Figure 1 but does not state the exact prompt template used for all models, nor the decoding parameters (temperature, max_tokens, top_p), nor the number of DuIE 2.0 sentences sampled and the sampling procedure. Because the comparison's validity depends on identical conditions across models, these omissions are load-bearing; different prompts or decoding settings could change the ranking in Table II.
[Section III-B] The semantic matching scorer is not defined. The paper does not specify the embedding model, similarity function, threshold, or whether matching is done per entity, per relation, or per triple. Since the top F1 scores are close (0.367 vs 0.343 vs 0.309), a scoring rule that accepts paraphrased relation names or partial entity overlaps at a different threshold could easily change the ordering. Please describe the scorer precisely and, ideally, release the matching code.
[Section IV-B, Figure 2] The latency measurements are reported as single numbers with no methodology: no number of runs, no indication of API concurrency or queueing, no hardware or date of measurement. API latency is highly variable; a single measurement is insufficient to support the claim that gemini-1.5-flash-8b is the fastest model. Please report mean and standard deviation over repeated runs and describe the measurement conditions.
[Section IV-A] The statement 'We utilize the latest APIs and libraries provided by each model's developers' is too vague. Model versions are not pinned (e.g., gpt-4-turbo has multiple release dates), and API behavior changes over time. Without version identifiers and access dates, the results are not reproducible and may already be outdated.

minor comments (5)

[Section IV-B] The text refers to 'Table IV-B' but the table is numbered II; please correct the cross-reference.
[Section IV-B] The phrase 'zero-shot CN RE task' should be 'zero-shot Chinese RE task' for clarity.
[Figure 1] In Figure 1, the arrow from 'Groundtruth' to the output '黄真真-导演-《闺蜜2》' is confusing: it is unclear whether this is a sample model output or the gold triple. Please clarify the diagram.
[Section II-A] The descriptions of ChatGPT, Gemini, and LLaMA are general and not tied to the specific model checkpoints used; consider adding references to official model documentation or model cards.
[Abstract and Conclusion] The abstract and conclusion state 'ChatGPT achieves the best accuracy' but the table shows gpt-4-turbo, a ChatGPT-family model; consider being more specific about the model version in those statements.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an external benchmark comparison whose claims rest on measured API outputs and the DuIE 2.0 dataset, with no fitting, derivation, or load-bearing self-citation.

full rationale

The paper makes no derivational claim: it reports measured precision, recall, F1, and latency from querying external LLM APIs on DuIE 2.0 sentences. The central ranking (gpt-4-turbo highest F1, gemini-1.5-flash-8b lowest latency, LLaMA underperforming) is an empirical observation, not a quantity constructed from the conclusion. No equation in the paper defines the ranking in terms of its own inputs, and no fitted parameter is renamed as a prediction. The self-references [1]-[5] appear only in the introduction as general citations to the authors' prior work and are not load-bearing for any result; the evaluation method is attributed to an external source [16] and the dataset to an external benchmark [17]. The evaluation pipeline is underspecified: Section IV-A only says the authors 'utilize the latest APIs,' Section III-B describes semantic matching without giving the similarity model or threshold, and Table II contains an internal inconsistency for gemini-1.5-flash (P=0.231, R=0.424 implies F1=0.299, not 0.293). These are legitimate correctness and reproducibility concerns, but they do not constitute circularity because the reported numbers are not equivalent by definition to the conclusions drawn from them. A different scorer or decoding configuration could change the ranking, but that is a measurement-validity issue, not a logical reduction of the conclusion to its evidence.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No explicit fitted parameters appear in the paper. The evaluation settings that function as free choices, such as the semantic-matching threshold and decoding parameters, are not reported, so they are unaccounted rather than fitted values. The paper introduces no new entities, particles, forces, or formal objects.

assumptions (3)

domain assumption DuIE 2.0 ground-truth annotations are correct and representative of Chinese relation extraction.
The entire evaluation is based on this single benchmark (Section IV-A), and its triples are treated as ground truth without discussion of annotation noise or coverage.
domain assumption Semantic similarity between model outputs and ground truth is a valid proxy for correct extraction.
Section III-B introduces semantic matching as the evaluation mechanism, but the paper gives no implementation details, so the assumption cannot be checked.
domain assumption API outputs are stable enough to compare models without repeated sampling or temperature control.
Section IV-A states only that the latest APIs were used; no variance, temperature, or decoding settings are reported, so the comparison implicitly assumes deterministic or representative outputs.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Zero-Shot End-to-End Relation Extraction in Chinese: A Comparative Study of Gemini, LLaMA and ChatGPT." pith.science (2026). https://pith.science/paper/ROBHX4VI

@misc{pith2026250205694,
  author       = {Pith},
  title        = {Pith review of: Zero-Shot End-to-End Relation Extraction in Chinese: A Comparative Study of Gemini, LLaMA and ChatGPT},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ROBHX4VI}},
  note         = {Machine review of arXiv:2502.05694}
}

read the original abstract

This study investigates the performance of various large language models (LLMs) on zero-shot end-to-end relation extraction (RE) in Chinese, a task that integrates entity recognition and relation extraction without requiring annotated data. While LLMs show promise for RE, most prior work focuses on English or assumes pre-annotated entities, leaving their effectiveness in Chinese RE largely unexplored. To bridge this gap, we evaluate ChatGPT, Gemini, and LLaMA based on accuracy, efficiency, and adaptability. ChatGPT demonstrates the highest overall performance, balancing precision and recall, while Gemini achieves the fastest inference speed, making it suitable for real-time applications. LLaMA underperforms in both accuracy and latency, highlighting the need for further adaptation. Our findings provide insights into the strengths and limitations of LLMs for zero-shot Chinese RE, shedding light on trade-offs between accuracy and efficiency. This study serves as a foundation for future research aimed at improving LLM adaptability to complex linguistic tasks in Chinese NLP.

Figures

Figures reproduced from arXiv: 2502.05694 by the authors.

**Figure 2.** Latency Comparison of Models. Latency Comparison Beyond accuracy, inference latency is a crucial factor for practical deployment, as shown in Figure 2. OpenAI’s models generally exhibit higher latency, with gpt-4 reaching 3.98 seconds, reflecting its computational complexity. In contrast, optimized versions such as gpt-4o and gpt-4o-mini achieve lower latencies of 1.86 and 2.31 seconds, respectively, making them mo… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 13 canonical work pages

[1]

Patch elastic deformation: An effective data augmen- tation method,

S. Du, K. Hao, H. Zhang, X.-s. Tang, and B. Wei, “Patch elastic deformation: An effective data augmen- tation method,” in 2022 China Automation Congress (CAC), 2022, pp. 2079–2084

work page 2022
[2]

Image2Text2Image: A Novel Framework for Label-Free Evaluation of Image-to-Text Generation with Text-to-Image Diffusion Models

J.-H. Huang, H. Zhu, Y . Shen, S. Rudinac, and E. Kanoulas, “Image2text2image: A novel framework for label-free evaluation of image-to-text generation with text-to-image diffusion models,” arXiv preprint arXiv:2411.05706, 2024

work page Pith review arXiv 2024
[3]

Financial sentiment analysis on news and reports using large language models and finbert,

Y . Shen and P. K. Zhang, “Financial sentiment analysis on news and reports using large language models and finbert,” 2024. [Online]. Available: https: //arxiv.org/abs/2410.01987

arXiv 2024
[4]

Improving di- versity and discriminability based implicit contrastive learning for unsupervised domain adaptation,

H. Xu, C. Shi, W. Fan, and Z. Chen, “Improving di- versity and discriminability based implicit contrastive learning for unsupervised domain adaptation,” Applied Intelligence, vol. 54, no. 20, pp. 10 007–10 017, 2024

work page 2024
[5]

Nevlp: Noise-robust framework for efficient vision-language pre-training,

Y . Tao, Z. Wang, H. Zhang, and L. Wang, “Nevlp: Noise-robust framework for efficient vision-language pre-training,” arXiv preprint arXiv:2409.09582 , 2024

arXiv 2024
[6]

(2022) Chatgpt

OpenAI. (2022) Chatgpt. [Online]. Available: https: //openai.com/chatgpt

work page 2022
[7]

(2023) Gemini

Google. (2023) Gemini. [Online]. Available: https: //gemini.google.com/app

work page 2023
[8]

(2023) Llama

Meta. (2023) Llama. [Online]. Available: https://ai.meta. com/llama

work page 2023

Show all 17 references

[9]

Exploring various knowledge in relation extraction,

G. Zhou, J. Su, J. Zhang, and M. Zhang, “Exploring various knowledge in relation extraction,” in ACL 2005, 43rd Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, 25-30 June 2005, University of Michigan, USA . The Association for Co...

2005
[10]

REBEL: relation extrac- tion by end-to-end language generation,

P. H. Cabot and R. Navigli, “REBEL: relation extrac- tion by end-to-end language generation,” in Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, M. Moens, X. Huang, L. Specia, and S. W...

2021
[11]

Zero- shot relation extraction via reading comprehension,

O. Levy, M. Seo, E. Choi, and L. Zettlemoyer, “Zero- shot relation extraction via reading comprehension,” in Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), Vancouver, Canada, August 3-4, 2017 , R. Levy and L. Specia, Eds. Associatio...

2017
[12]

Iterative zero-shot LLM prompting for knowledge graph construction,

S. Carta, A. Giuliani, L. Piano, A. S. Podda, L. Pom- pianu, and S. G. Tiddia, “Iterative zero-shot LLM prompting for knowledge graph construction,” CoRR, vol. abs/2307.01128, 2023

2023 arXiv
[13]

Chatie: Zero- shot information extraction via chatting with chatgpt,

X. Wei, X. Cui, N. Cheng, X. Wang, X. Zhang, S. Huang, P. Xie, J. Xu, Y . Chen, M. Zhang et al., “Chatie: Zero- shot information extraction via chatting with chatgpt,” arXiv preprint arXiv:2302.10205 , 2023

2023 arXiv
[14]

Llms for knowledge graph construction and reasoning: recent capabilities and future opportunities,

Y . Zhu, X. Wang, J. Chen, S. Qiao, Y . Ou, Y . Yao, S. Deng, H. Chen, and N. Zhang, “Llms for knowledge graph construction and reasoning: recent capabilities and future opportunities,” World Wide Web (WWW), vol. 27, no. 5, p. 58, 2024

2024
[15]

Zero-shot construction of chinese medical knowledge graph with gpt-3.5-turbo and gpt-4,

L.-I. Wu, Y . Su, and G. Li, “Zero-shot construction of chinese medical knowledge graph with gpt-3.5-turbo and gpt-4,” ACM Transactions on Management Information Systems, 2024

2024
[16]

Semantic matching,

F. Giunchiglia and P. Shvaiko, “Semantic matching,” The Knowledge Engineering Review, vol. 18, no. 3, pp. 265– 280, 2003

2003
[17]

Duie: A large-scale chi- nese dataset for information extraction,

S. Li, W. He, Y . Shi, W. Jiang, H. Liang, Y . Jiang, Y . Zhang, Y . Lyu, and Y . Zhu, “Duie: A large-scale chi- nese dataset for information extraction,” in Natural Lan- guage Processing and Chinese Computing - 8th CCF In- ternational Conference, NLPCC 2019, Dunhuang, China, ...

2019

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

Patch elastic deformation: An effective data augmen- tation method,

S. Du, K. Hao, H. Zhang, X.-s. Tang, and B. Wei, “Patch elastic deformation: An effective data augmen- tation method,” in 2022 China Automation Congress (CAC), 2022, pp. 2079–2084

work page 2022

[2] [2]

Image2Text2Image: A Novel Framework for Label-Free Evaluation of Image-to-Text Generation with Text-to-Image Diffusion Models

J.-H. Huang, H. Zhu, Y . Shen, S. Rudinac, and E. Kanoulas, “Image2text2image: A novel framework for label-free evaluation of image-to-text generation with text-to-image diffusion models,” arXiv preprint arXiv:2411.05706, 2024

work page Pith review arXiv 2024

[3] [3]

Financial sentiment analysis on news and reports using large language models and finbert,

Y . Shen and P. K. Zhang, “Financial sentiment analysis on news and reports using large language models and finbert,” 2024. [Online]. Available: https: //arxiv.org/abs/2410.01987

arXiv 2024

[4] [4]

Improving di- versity and discriminability based implicit contrastive learning for unsupervised domain adaptation,

H. Xu, C. Shi, W. Fan, and Z. Chen, “Improving di- versity and discriminability based implicit contrastive learning for unsupervised domain adaptation,” Applied Intelligence, vol. 54, no. 20, pp. 10 007–10 017, 2024

work page 2024

[5] [5]

Nevlp: Noise-robust framework for efficient vision-language pre-training,

Y . Tao, Z. Wang, H. Zhang, and L. Wang, “Nevlp: Noise-robust framework for efficient vision-language pre-training,” arXiv preprint arXiv:2409.09582 , 2024

arXiv 2024

[6] [6]

(2022) Chatgpt

OpenAI. (2022) Chatgpt. [Online]. Available: https: //openai.com/chatgpt

work page 2022

[7] [7]

(2023) Gemini

Google. (2023) Gemini. [Online]. Available: https: //gemini.google.com/app

work page 2023

[8] [8]

(2023) Llama

Meta. (2023) Llama. [Online]. Available: https://ai.meta. com/llama

work page 2023