Pith. sign in

REVIEW 3 major objections 5 minor 18 references

Tianyi: A Traditional Chinese Medicine all-rounder language model and its Real-World Clinical Practice

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read Tianyi, a 7.6-billion-parameter language model built for Traditional Chinese Medicine, claims to outperform same-scale models on exams, clinical record tasks, doctor-rated diagnosis and prescription quality, and zero-shot prediction of…

desk verdict A serious TCM LLM paper whose real-world clinical claim lacks the baselines needed to support it. read the letter →

arxiv 2505.13156 v1 pith:GMQFUESL submitted 2025-05-19 cs.CL cs.AI

classification cs.CLcs.AI
keywords largelanguagemodelTraditionalChineseMedicineclinicaldecisionsupportzero-shotpredictionmedicalbenchmarkprescriptiongenerationsyndromediagnosisTCMEval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Tianyi, a 7.6-billion-parameter language model trained specifically for Traditional Chinese Medicine, and argues that a model of this scale—far smaller than hundred-billion-parameter generalists—can serve real TCM clinical practice. The authors build the model by progressively pre-training on 3.4 billion TCM-domain tokens drawn from classics, textbooks, research articles, and roughly 30,000 clinical records, then fine-tuning on four correlated instruction types and aligning preferences with a KTO objective. They assemble TCMEval, a benchmark spanning TCM exams, clinical-record tasks, domain QA, doctor-rated syndrome and prescription evaluation, and two real-world clinical trials—one retrospective influenza-prevention study with 108 children and one prospective H1N1 trial with 112 adults. Tianyi reports the best same-scale results on nearly every measure, including 75.38% exam accuracy, 81.21% F1 on clinical-record tasks, and balanced zero-shot predictions of 48-hour fever abatement around 75%. The core claim is that Tianyi can solve unseen, real-world clinical tasks and has the potential to act as a top-tier TCM assistant.

What carries the argument

The load-bearing mechanism is a progressive three-stage training pipeline. First, a general Chinese LLM is continued pre-training on TCMCorpus, a 3.4-billion-token corpus organized in increasing specialty and difficulty—basic medical knowledge, general TCM knowledge, then the reinforced theory and clinical experience of renowned TCM doctors. Second, TCM-LLM-Base is instruction-fine-tuned on four correlated resource types: knowledge-based QA, clinical EHR tasks, TCM research-article analysis, and complete doctor-patient consultations, so the model learns to connect knowledge branches rather than memorize isolated answers. Third, KTO preference alignment shapes responses toward doctor-preferred content and format. The evaluation side is TCMEval, whose real-world component supplies the unseen tasks—human-rated syndrome and prescription chains and zero-shot prediction of two clinical trials—that separate trained knowledge from emergent clinical reasoning.

What would settle it

If a trivial baseline on the 112-patient H1N1 cohort—for example, always predicting the majority outcome on each task, or predicting fever-abatement time from the base-rate distribution alone—achieves balanced accuracy equal to or better than Tianyi's roughly 75% true-positive/true-negative rates once class imbalance is accounted for, the claim that TCM knowledge drives the predictions is falsified; the same test can be pre-registered prospectively on a new trial before outcomes are collected.

Watch

Extended reading notes

Core claim

The central discovery claimed is that a 7.6B-parameter model, deliberately trained on interconnected TCM knowledge, generalizes to clinical situations it was never trained on. In the regular benchmarks, Tianyi reaches 75.38% accuracy on TCM multiple-choice exams, 81.21% F1 on the three-task clinical-record benchmark (syndrome prediction, treatment-principle decision, prescription generation), and 46.83% Rouge-L on domain QA, exceeding all same-scale general models and the existing TCM-specialized comparison model. In the human study, ten experienced TCM doctors rated Tianyi's syndrome diagnoses, prescriptions, prescription analyses, and etiology explanations as the most correct, comprehensive, and legible, with the smallest scoring variance across five cases. In the real-world trials, Tianyi predicts complete fever abatement within 48 hours, the time window of fever abatement, adverse reactions, and 7-day influenza incidence with the most balanced accuracy among the four strongest compared models, despite receiving only zero-shot prompts. The authors therefore assert that the capability to solve unseen clinical tasks indicates Tianyi's potential as a top-tier TCM assistant for daily clinical work.

Load-bearing premise

The load-bearing premise is that the real-world evaluations—108 and 112 patient records plus five human-evaluated cases—are valid tests of clinical competence, meaning patient outcomes such as fever-abatement time and adverse reactions are actually predictable from the provided textual features and the standard TCM prescription rather than dominated by unmeasured factors like viral strain, medication adherence, or host immunity.

Editorial extensions

If this is right

  • A well-designed 7.6B model can carry the whole TCM diagnosis-to-prescription chain, not just answer questions about TCM.
  • Zero-shot prediction of individual patient outcomes, if it holds, means a domain LLM could flag expected fever-abatement delays or adverse reactions before a trial's results are known.
  • At roughly 7.6B parameters, such a model could be deployed in clinics and primary-care settings where hundred-billion-parameter systems are infeasible.
  • The TCMEval structure separates factual TCM knowledge from transferable clinical reasoning, giving later TCM models a common yardstick for both.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because two compared models collapsed into all-'yes' or all-'no' predictions, reporting class-balanced metrics (balanced accuracy and AUC) on the same trial data would give a sharper test of whether the reported accuracies exceed what base rates alone can achieve.
  • Editorial inference: the three-question chain of syndrome diagnosis, prescription analysis, and etiology explanation used in the human study could be turned into a reusable TCM reasoning benchmark, replacing costly per-case doctor panels with a standardized rubric.
  • Editorial inference: if the progressive pretraining recipe transfers, other text-centric traditional medical systems with classical corpora and structured records could be specialized at similar scale, though each would need its own clinical validation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. Tianyi is a 7.6B-parameter decoder-only language model built by continual pre-training of Baichuan2 on a 3.4B-token TCM corpus, followed by instruction fine-tuning and KTO-based alignment on EHR-derived tasks, TCM examination and QA books, doctor-patient consultations, and a TCM knowledge graph. The authors introduce the TCMEval evaluation suite, comprising three regular benchmarks (TCM-MCE, TCM-CRT, TCM-DQA) and two real-world evaluation settings (TCM-SPHE, a human-doctor assessment of syndrome diagnosis and prescription, and TCM-CTP, prediction of outcomes in two TCM clinical trials). The paper reports state-of-the-art performance for Tianyi on the regular benchmarks and argues that zero-shot predictions in the clinical-trial tasks demonstrate Tianyi's real-world clinical reasoning capability and its potential as a top-tier TCM assistant.

Significance. If fully supported, the contribution would be significant: it combines a deployable-scale TCM-specific model, a multi-scenario evaluation suite, a blinded human-doctor evaluation, and ethics-approved prospective and retrospective clinical trials. The paper's strengths are its explicit design goal of a scale-proper TCM LLM, the progressive pre-training strategy, the use of real clinical-trial data rather than synthetic QA alone, and the commitment to release the model. However, the regular benchmark results are compromised by likely training-evaluation overlap, and the real-world clinical-trial predictions are reported without any trivial baseline, so the central claim of demonstrated real-world clinical competence is not yet established at the level claimed.

major comments (3)
  1. [Evaluation Benchmarks / Table 1] The Table 1 results cannot support the claimed SOTA performance on the regular benchmarks because those benchmarks are not independent of Tianyi's training data. TCM-MCE is sampled from TCM examination and QA books, the same resource family used for fine-tuning ('TCM examination and QA books (100K)'); TCM-DQA is drawn from the same QA-book family; and TCM-CRT uses clinical records from the authors' previous works [11-13], which are explicitly listed as part of the pre-training corpus. The paper's own caveat that models 'may be trained and fine-tuned by the similar sample and distribution with samples in Regular TCM Benchmark Tasks' therefore applies directly to Tianyi, not only to external baselines. The authors should report contamination statistics (e.g., n-gram overlap between training and evaluation items) or construct held-out versions of TCM-MCE, TCM-CRT, and TCM-DQA and rerun Table 1 before claiming superiority on these tasks.
  2. [Results / TCM-CTP, Fig. 4] The central claim that Tianyi 'demonstrates the capability on solving the unseen, real-world clinical tasks' rests on the TCM-CTP results in Fig. 4, but no null model or trivial baseline is reported for either clinical trial. Given the 'extreme sparsity of true negative cases' noted for the 48-hour fever-abatement task and the observed all-yes/all-no behavior of the Qwen models, raw per-class accuracy is uninterpretable without the class base rates, a majority-class predictor, and a simple feature-based predictor (e.g., logistic regression on the same symptom, history, and medication items). In the prospective trial, every model receives the same standardized TCM prescription as part of the input (Table 4), so prescription variation cannot contribute to any model's predictive skill; the task reduces to outcome prediction from patient features. Without these baselines and confidence intervals, Fig. 4 does not establish that any LLM, including Tianyi, performs TCM-specific clinical outcome prediction.
  3. [Results / TCM-SPHE, Fig. 2] The human evaluation uses only 5 clinical records and 10 doctors, and although this yields multiple ratings per model per criterion, the case-level sample is very small. The paper reports mean scores and variance across doctors but provides no inter-rater agreement statistic, no test of whether the ranking of models is stable across cases, and no confidence intervals for the average scores. This evidence can support a qualitative demonstration of feasibility, but it cannot independently carry the quantitative claim that Tianyi is a 'top-tier TCM assistant' for syndrome diagnosis and prescription; a larger set of clinical records or explicit case-level uncertainty quantification is needed.
minor comments (5)
  1. [TCM-SPHE, Tables 1 and 2] The text says the three TCM-SPHE questions are listed in Table 1, but the actual question list appears as Table 2; one of the two references should be corrected.
  2. [Fig. 4 caption] The caption uses 'YES' both as a model prediction and as 'the portion that model correctly predicted', which makes the reported counts difficult to interpret; please replace this with explicit true-positive, true-negative, false-positive, and false-negative counts and define accuracy accordingly.
  3. [Table 1 / TCM-CRT metric] The evaluation section states that syndrome prediction and treatment-principle prediction are measured by accuracy while prescription generation is measured by F1, yet Table 1 reports a single F1 value per model for TCM-CRT; the aggregation and label-matching procedure should be specified.
  4. [Discussion, reference [14]] The Discussion attribute the Zhongjing-Llama model to reference [14], but [14] is a paper on tongue colour and coating prediction; the correct citation appears to be [32].
  5. [Throughout] There are several typographical and formatting errors, including 'togue and pulse condition', 'an evaluation HER', and an empty citation in 'Rouge-L[]'; these should be fixed in a revised version.

Circularity Check

2 steps flagged · score 6.0 of 10

The Regular TCM Benchmark Tasks are built from the same exam/QA books and clinical-record collections used to fine-tune Tianyi, and the paper itself concedes this; part of the headline evaluation therefore reduces to re-testing training-distribution material, though the real-world TCM-CTP trials remain independent.

  1. fitted input called prediction [Evaluation Benchmarks / Regular TCM Benchmark Tasks (TCM-MCE/TCM-DQA); Method / Fine-tuning dataset]
    "One thing should be noticed that the LLMs may trained and fine-tuned by the similar sample and distribution with samples in Regular TCM Benchmark Tasks, which shows the ability of solving the issues or tasks they trained"

    The fine-tuning set is built by converting 'TCM examination and QA books (100K)' into instruction samples, and TCM-MCE is then formed by extracting 'all the multiple-choice questions and its answers from the available books and materials' and sampling 20%. TCM-DQA is likewise drawn from the same TCM QA-book corpora. Accuracy on these benchmarks therefore measures the model's ability to answer items from the same distribution on which it was fine-tuned; the reported scores are a renamed training input rather than an independent prediction. The quoted sentence is the authors' own explicit admission of this reduction for the Regular TCM Benchmark Tasks.

  2. fitted input called prediction [Method / Pre-training dataset; Evaluation Benchmarks / TCM-CRT]
    "This benchmark contains various TCM clinical records that we collect during our researches on AI of TCM."

    Pre-training states that the clinical-record corpus comprises 'the real-world TCM clinical records that collected in our previous works[11-13] and the new collections provided by the cooperation TCM hospitals'. The TCM-CRT benchmark is then defined as 'various TCM clinical records that we collect during our researches on AI of TCM', i.e., the same line of authors' own prior data collections, after the model has been instruction-fine-tuned on converted clinical records for exactly the three TCM-CRT tasks: syndrome prediction, treatment-principle decision, and prescription generation.

full rationale

Tianyi's real-world TCM-CTP evaluations (retrospective influenza-prevention and prospective influenza-A trials) are new trial data used zero-shot and are not part of pre-training or fine-tuning; those results are independent and I do not count them as circular. However, the paper's overall claim of being a 'top-tier TCM assistant' is supported by TCMEval as a whole, and the Regular TCM Benchmark Tasks are contaminated by construction: the fine-tuning corpus includes the same examination/QA books and the same clinical-record collections (references [11-13] are the authors' own prior work) that were then sampled to form TCM-MCE, TCM-DQA, and TCM-CRT. The paper's own caveat explicitly acknowledges that the compared LLMs 'may trained and fine-tuned by the similar sample and distribution with samples in Regular TCM Benchmark Tasks'. The human TCM-SPHE evaluation uses only 5 records and 10 doctors, but it is not definitionally circular. The absence of majority-class or simple-feature baselines for the TCM-CTP tasks is an external-validity weakness rather than a circularity. Score 6 reflects that a substantial part of the headline evaluation reduces to re-testing training-distribution material, while the real-world component retains independent content and prevents a higher score.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper's central claims rest on proprietary data whose composition is only partially described, on clinical labels taken as ground truth, and on the assumption that zero-shot outcome predictions from text are meaningful. Because the model and data are not released, these assumptions cannot be checked independently.

free parameters (1)
  • Training hyperparameters (learning rate, batch size, epochs, curriculum split)
    Not reported; the progressive pre-training and fine-tuning procedure depends on these hand-chosen settings, and the paper provides no values or ablation.
assumptions (4)
  • domain assumption TCM theory and textbook content are reliable sources of medical knowledge.
    The model is trained to reproduce this knowledge; if the source texts contain errors, the model inherits them.
  • domain assumption Clinical record labels (syndrome, treatment principle, prescription) are correct and consistent across hospitals.
    These labels are treated as ground truth for TCM-CRT and human evaluation.
  • domain assumption Zero-shot prediction of clinical trial outcomes from textual features is a valid measure of clinical reasoning.
    The real-world evaluation assumes outcomes are predictable from the provided patient information and prescription.
  • domain assumption Evaluation metrics (accuracy, F1, ROUGE-L, human ratings) capture clinically meaningful quality.
    No validation of these metrics against patient outcomes or doctor preferences is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Tianyi: A Traditional Chinese Medicine all-rounder language model and its Real-World Clinical Practice." pith.science (2026). https://pith.science/paper/GMQFUESL

@misc{pith2026250513156,
  author       = {Pith},
  title        = {Pith review of: Tianyi: A Traditional Chinese Medicine all-rounder language model and its Real-World Clinical Practice},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GMQFUESL}},
  note         = {Machine review of arXiv:2505.13156}
}
read the original abstract

Natural medicines, particularly Traditional Chinese Medicine (TCM), are gaining global recognition for their therapeutic potential in addressing human symptoms and diseases. TCM, with its systematic theories and extensive practical experience, provides abundant resources for healthcare. However, the effective application of TCM requires precise syndrome diagnosis, determination of treatment principles, and prescription formulation, which demand decades of clinical expertise. Despite advancements in TCM-based decision systems, machine learning, and deep learning research, limitations in data and single-objective constraints hinder their practical application. In recent years, large language models (LLMs) have demonstrated potential in complex tasks, but lack specialization in TCM and face significant challenges, such as too big model scale to deploy and issues with hallucination. To address these challenges, we introduce Tianyi with 7.6-billion-parameter LLM, a model scale proper and specifically designed for TCM, pre-trained and fine-tuned on diverse TCM corpora, including classical texts, expert treatises, clinical records, and knowledge graphs. Tianyi is designed to assimilate interconnected and systematic TCM knowledge through a progressive learning manner. Additionally, we establish TCMEval, a comprehensive evaluation benchmark, to assess LLMs in TCM examinations, clinical tasks, domain-specific question-answering, and real-world trials. The extensive evaluations demonstrate the significant potential of Tianyi as an AI assistant in TCM clinical practice and research, bridging the gap between TCM knowledge and practical application.

Figures

Figures reproduced from arXiv: 2505.13156 by the authors.

Figure 5
Figure 5. The framework of TCMEval. It contains 2 categories of TCM evaluations: regular TCM benchmark task and the TCM [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. The methods for enrolling patients in the two TCM clinical trials and the model prediction on them. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 2
Figure 2. The a, b, c, d are the average scores of the representative LLMs that evaluated by TCM human doctors from (1) TCM syndrome diagnosis and prescription, (2) TCM prescription analysis, (3) etiology and pathogenesis analysis, and (4) readability and intelligibility ( [PITH_FULL_IMAGE:figures/full_fig_p015_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 15 canonical work pages

  1. [3]

    Z. Shi, T. Song, Y. Wan, J. Xie, Y. Yan, K. Shi, Y. Du, L. Shang, A systemati cr e v i e wa n d meta-analysis of traditional insect Chinese medicines combined chemoth erapy for non-surgical hepatocellular carcinoma therapy, Scientific reports, 7 (2017) 4355

  2. [4]

    Werneke, T

    U. Werneke, T. Turner, S. Priebe, Complementary medicines in psychiat ry: review of effectiveness and safety, The British Journal of Psychiatry, 188 (2006) 1 09-121

  3. [5]

    Lu, H.-W

    A.-P. Lu, H.-W. Jia, C. Xiao, Q.-P. Lu, Theory of traditional Chinese me dicine and therapeutic method of diseases, World journal of gastroenterology: WJG, 10 (2004) 185 4

  4. [6]

    L. Shao, B. Zhang, Traditional Chi nese medicine network pharmacology : theory, methodology and application, Chinese journal of natural medicines, 11 (2013) 110-120 . [ 7 ]X .Z h o u ,S .C h e n ,B .L i u ,R .Z h a n g ,Y .W a n g ,P .L i ,Y .G u o ,H .Z h a n g ,Z .G a o , X. Yan, Development of traditional Chinese medicine clinical data warehouse for me...

  5. [9]

    Y. Hu, G. Wen, H. Liao, C. Wang, D. Da i, Z. Yu, Automatic construction of c hinese herbal prescriptions from tongue images using CNNs and auxiliary latent therapy topics, IEEE transactions on cybernetics, 51 (2019) 708-721. [ 1 0 ]X .L i ,K .C h e n ,J .Y a n g ,C .W a n g ,T .Y a n g ,C .L u o ,N .L i ,Z .L i u ,T L D A :At r ansfer learning based dual-...

  6. [12]

    Z. Liu, J. Yang, K. Chen, T. Yang, X. Li, B. Lu, D. Fu, Z. Zheng, C. Luo, TCM -KDIF: An Information Interaction Framework Driven by Knowledge-Data and Its Clin ical Application in Traditional Chinese Medicine, IEEE Internet of Things Journal, (2024)

  7. [13]

    Z. Liu, Z. Zheng, X. Guo, L. Qi, J. Gui, D. Fu, Q. Yao, L. Jin, Attentivehe rb: a novel method for traditional medicine prescription g eneration, IEEE Access, 7 (2019) 139 069-139085

  8. [14]

    Zhang, J

    D. Zhang, J. Zhang, Z. Wang, M. Sun, Tongue colour and coating predicti on in traditional Chinese medicine based on visible hyperspectral imaging, IET Image Proce ssing, 13 (2019) 2265-2270. [ 1 5 ]J .A c h i a m ,S .A d l e r ,S .A g a r w a l ,L .A h m a d ,I .A k k a y a ,F . L .A l e m a n ,D .A lmeida, J. Altenschmidt, S. Altman, S. Anadkat, Gpt-4 te...

Show all 18 references
  1. [16]

    Dubey, A

    A. Dubey, A. Jauhri, A. Pandey, A. K a d i a n ,A .A l - D a h l e ,A .L e t m a n ,A .M athur, A. Schelten, A. Y a n g ,A .F a n ,T h el l a m a3h e r do fm o d e l s ,a r X i vp r e p r i n ta r X i v : 2 4 0 7 . 2 1 7 8 3 ,(2024)

  2. [17]

    Ravaut, A

    M. Ravaut, A. Sun, N. Chen, S. Joty, On context utilization in summariz ation with large language models, in: Proceedings of the 62nd Annual Meeting of the Associa tion for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 2764-2781. [ 1 8 ]Y .W a n g ,N .L i p k a ...

  3. [22]

    Q. Jin, Y. Yang, Q. Chen, Z. Lu, GeneGPT: augmenting large language mod els with domain tools for improved access to biomedic al information, Bioinformatics, 40 (2024)

  4. [23]

    R. Luo, L. Sun, Y. Xia, T. Qin, S. Zhang, H. Poon, T.-Y. Liu, BioGPT: gene rative pre-trained transformer for biomedical text gene ration and mining, Briefings in Bioi nformatics, 23 (2022). [ 2 4 ]X .L i u ,H .L i u ,G .Y a n g ,Z .J i a n g ,S .C u i ,Z .Z h a n g ,H .W a n ...

  5. [27]

    J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Hu ang, Qwen technical report, arXiv preprint arXiv:2309.16609, (2023)

  6. [28]

    A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang , H. Wei, Qwen2. 5 technical report, arXiv preprin t arXiv:2412.15115, (2024). [ 2 9 ]A .Y a n g ,B .X i a o ,B .W a n g ,B .Z h a n g ,C .B i a n ,C .Y i n ,C .L v ,D .P a n ,D .W a n g,D .Y a n ,B a...

  7. [32]

    S. Yang, H. Zhao, S. Zhu, G. Zhou, H . Xu, Y. Jia, H. Zan, Zhongjing: Enhan cing the chinese medical capabilities of large language model through expert feedback and real-world multi-turn dialogue, in: Proceedings of the AAAI Conference on Artificial Intellige nce, 2024, pp. 1...

  8. [33]

    Ko či s k ý ,J .S c h w a r z ,P .B l u n s o m ,C .D y e r ,K

    T. Ko či s k ý ,J .S c h w a r z ,P .B l u n s o m ,C .D y e r ,K . M .H e r m a n n ,G .M e l i s ,E .G r e f e n s t ette, The narrativeqa reading comprehension c hallenge, Transactions of the Assoc iation for Computational Linguistics, 6 (2018) 317-328

  9. [34]

    Contributors, Opencompass: A universal evaluation platform for f oundation models, GitHub repository, (2023)

    O. Contributors, Opencompass: A universal evaluation platform for f oundation models, GitHub repository, (2023)

  10. [35]

    M. Moor, O. Banerjee, Z.S.H. Abad, H.M. Krumholz, J. Leskovec, E.J. To pol, P. Rajpurkar, Foundation models for generalist med ical artificial intelligence, Natu re, 616 (2023) 259-265. Data availability Due to privacy regulations, the pretraining corpora and EHRs cannot be ma...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.