Pith. sign in

REVIEW 4 major objections 4 minor 20 references

ICH-Qwen: A Large Language Model Towards Chinese Intangible Cultural Heritage

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 7B model beats six general LLMs on Chinese intangible heritage Q&A.

desk verdict Routine domain adaptation for Chinese ICH with a real data contribution but an evaluation that can't support the SOTA claim. read the letter →

arxiv 2505.23831 v1 pith:NQNLNUQK submitted 2025-05-28 cs.CL

Wenhao Ye , Tiansheng Zheng , Yue Qi , Wenhua Zhao , Xiyu Wang , Xue Zhao , Jiacheng He , Yaya Zheng
show 1 more author
Dongbo Wang
This is my paper
classification cs.CL
keywords IntangibleCulturalHeritageChineseLLMdomain-specificlargelanguagemodelinstructionfine-tuningsyntheticdataLoRAknowledgeQ&Adigitalhumanities
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that adapting a general-purpose 7B language model to Chinese intangible cultural heritage (ICH) makes it the best of seven tested models on three ICH tasks: knowledge Q&A, context-aware Q&A, and terminology interpretation. The authors assemble a multi-source corpus from official ICH records and academic abstracts, annotate knowledge entities, continue pre-training, and fine-tune with LoRA on instruction data that includes synthetic Q&A pairs generated by a larger teacher model. Across 100 test questions per task, ICH-Qwen records the highest ROUGE, BLEU, and chrF scores on nearly all metrics. If the evaluation holds, this is evidence that modest domain adaptation can turn a general assistant into a usable heritage-knowledge resource without large-scale retraining.

What carries the argument

The working mechanism is a three-stage domain-adaptation pipeline. First, raw text from the China Intangible Cultural Heritage Network and journal abstracts is cleaned and annotated with ICH-specific entity labels such as ICH-TITLE, ICH-PLACE, and ICH-TERM. Second, the model undergoes continued pre-training on this corpus. Third, instruction fine-tuning with LoRA on Qwen2.5-7B-Chat uses three task templates (knowledge Q&A, context-aware Q&A, terminology interpretation) plus synthetic dialogue data generated by Qwen2-72B-Instruct and manually checked. The synthetic data is the load-bearing ingredient that expands scarce expert-written Q&A into enough training signal for the 7B model.

What would settle it

Re-run the three evaluations on a fresh, independently written set of ICH questions after removing any overlapping text from training and synthetic data; if ICH-Qwen's margin over Qwen2.5-7B-Instruct shrinks or reverses, the claimed state-of-the-art result is an artifact of test-set overlap rather than domain adaptation.

Watch

Extended reading notes

Core claim

On the paper's own terms, ICH-Qwen is a state-of-the-art model for Chinese intangible cultural heritage text generation. In the knowledge Q&A test it reaches ROUGE-1-F 25.04, ROUGE-L-F 20.88, and BLEU-4 6.32 against baselines like Qwen2.5-7B-Instruct and GLM4-9B-Chat; in context-aware Q&A it reaches ROUGE-1-F 37.21 and BLEU-4 11.99, and in terminology interpretation it leads on all eight metrics. The largest gains appear when the model is given reference material, with ROUGE-1-F rising 48.6% and BLEU-4 rising 89.7% over its own no-context performance. The authors interpret these results as showing stronger lexical alignment, sentence-structure similarity, and terminology mastery in the ICH domain.

Load-bearing premise

The 300 test questions fairly measure ICH ability, meaning none of them (or near-duplicates) appeared in the pre-training corpus, the synthetic data, or the base-model selection step that picked Qwen2.5-7B-Chat.

Editorial extensions

If this is right

  • LoRA fine-tuning on a 7B base is enough to reach top scores in a niche cultural domain, implying comparable gains are available for other low-resource heritage topics without retraining a full model.
  • The context-aware task's large improvement suggests that grounding the model with reference passages is a cheap way to improve answer fidelity in knowledge-intensive cultural Q&A.
  • The same synthesize-then-verify data pipeline can be reused whenever expert-authored Q&A is scarce, as long as a larger teacher model and manual validation are available.
  • A model like this could power public-facing heritage chatbots, museum guides, and digital humanities search tools that answer in Chinese about crafts, rituals, and oral traditions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I would add a human evaluation or a factuality test, because ROUGE, BLEU, and chrF measure n-gram overlap and can reward verbose phrasing even when the content is wrong.
  • Because ICH is inherently multimodal, extending the corpus to images, audio, and video of crafts and performances would likely capture knowledge that text-only training misses; the paper itself flags this as future work.
  • The model's lead is evaluated on only 100 questions per task; a larger held-out set with error analysis by heritage experts would tell whether the gains are concentrated in a few predictable question types.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces ICH-Qwen, a 7B-class Chinese LLM adapted to the intangible cultural heritage (ICH) domain via continued pre-training on a multi-source ICH corpus and instruction fine-tuning on synthetic QA data. It evaluates the model on three tasks—knowledge QA, context-aware QA, and terminology interpretation—each with 100 samples, comparing against six open LLMs using ROUGE, BLEU, and chrF, and claims state-of-the-art performance. The core claim is not supported by the reported evidence: the evaluation is not independent of the training data, no contamination check is provided, the base model is selected using the same evaluation tasks, and the reported differences lack any statistical grounding.

Significance. If the claims were validated, a domain-adapted 7B model for Chinese ICH could be a useful digital-humanities resource and a concrete demonstration of low-cost domain adaptation. The paper's strengths include constructing a relatively large ICH corpus, using a synthetic-data pipeline with manual verification, and benchmarking against several modern LLMs. However, the evaluation design has load-bearing flaws that prevent the current results from establishing the claimed advantage. The paper does not provide code, datasets, or an independent test set, so the empirical contribution is currently not reproducible beyond the corresponding-author request.

major comments (4)
  1. [§3.2.1, §3.3, Tables 2 and 6–8] The evaluation set is not shown to be independent of the training data. The pre-training corpus is built from the China Intangible Cultural Heritage Network and academic abstracts (§3.2.1), and the synthetic instruction data are generated from 'pre-acquired ICH domain texts and archival manuscripts' (§3.3). The test references in Table 2—e.g., the Miao Ancient Songs passage—are exactly the style and content of that corpus, and the same reference sentence is reused for Knowledge Q&A and Context-aware Knowledge Q&A. Since ROUGE, BLEU, and chrF are lexical-overlap metrics, a model that memorizes source passages can inflate scores without displaying ICH understanding. No train/test overlap check or held-out validation is reported. This directly undermines the central 'state-of-the-art' claim in §4.4.
  2. [§5 (Discussion) and Tables 6–8] The base-model selection procedure circularly depends on the evaluation tasks. Section 5 states that Qwen2.5-7B-Chat was chosen as the base 'due to its exceptional performance in knowledge-intensive tasks' among seven evaluated models, yet Tables 6–8 do not report Qwen2.5-7B-Chat's scores. If the base model was selected using the same test set that is later used to show ICH-Qwen's advantage, the comparison is biased. The actual chosen baseline is missing from the results, so the reader cannot verify the claimed improvement over the starting point.
  3. [§4.4, Tables 6–8] The evaluation has no statistical support. Each task uses only 100 samples, and no error bars, confidence intervals, or significance tests are reported. Several metric gaps are small—for example, ROUGE-1-F in Table 6 is 25.04 for ICH-Qwen versus 23.77 for Baichuan2-7B-Chat—and cannot be distinguished from sampling noise. The text in §4.4 explicitly says 'statistically significant improvements' without performing any significance test. The claims of superiority are therefore not statistically grounded.
  4. [§4.4 and Table 6] The statement that ICH-Qwen achieves 'state-of-the-art performance across all three evaluation tasks' is not supported even by the reported numbers. In the Knowledge Q&A task (Table 6), Baichuan2-7B-Chat obtains a higher BLEU-1 score (18.44 vs. 18.37) and a substantially higher chrF score (13.43 vs. 10.54). The paper's own narrative acknowledges that Baichuan2-7B-Chat 'shows better performance in BLEU-1 and chrF metrics.' Thus ICH-Qwen is not uniformly superior, and the unqualified SOTA claim should be revised or justified with an explicit aggregation rule.
minor comments (4)
  1. [Abstract and §4.2] The manuscript contains noticeable English errors, e.g., 'threats damage, disappearance and discontinuity' in the abstract and 'chosed' in §4.2; a careful language edit is needed.
  2. [§2] Section numbering is inconsistent: the domain-specific LLM subsection appears as '2.2.2' after '2.1', and '2.3Digital' is missing a space. Please renumber the subsections.
  3. [Figure 2 and §4.3] The loss curve in Figure 2 shows training loss only, with no validation loss or early-stopping criterion; please clarify how convergence was determined and whether any checkpoint selection was based on the evaluation set.
  4. [Data availability] The data availability statement says data are 'available from the corresponding author on reasonable request,' which is insufficient for reproducibility; please release the datasets, code, and model checkpoints with a stable URL.

Circularity Check

0 steps flagged · score 0.0 of 10

No construction-level circularity: the ICH-Qwen pipeline is an empirical train-and-evaluate system, not an equation-level derivation, and the evaluation-source ambiguity is a validity concern rather than a circular step.

full rationale

The paper's central claim is an empirical SOTA comparison in Section 4.4, not a mathematical derivation. The chain is: collect ICH corpus, continue pre-training, synthesize instruction data with Qwen2-72B-Instruct, manually verify it, LoRA fine-tune, and evaluate on 100 samples per task against six baselines. No fitted parameter is renamed as a prediction, no target quantity is defined in terms of the output, and no uniqueness theorem or ansatz is imported from prior work. The only self-citation ([11], GujiBERT/GujiGPT) appears in Related Work as background and does not bear weight on the ICH-Qwen construction. The main weakness is that the paper does not state how the 300 evaluation samples were selected or whether they overlap the pre-training/synthetic corpus, and Section 5's base-model selection could have been influenced by similar tasks; these are threats to external validity and reproducibility, not circularity by construction, because the paper never asserts that the test references are generated from or identical to the training data. Under the hard rule requiring a quoted reduction (Eq. X = Eq. Y, or fitted parameter = prediction), no such reduction exists here.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim (state-of-the-art ICH-domain performance) rests on a set of unverified assumptions about data quality, synthetic-data validity, metric validity, and evaluation independence. The most consequential free parameter is the base-model choice, which was made using the evaluation tasks themselves.

free parameters (6)
  • Learning rate = 2e-4
    Table 5; chosen by hand without sensitivity analysis; affects convergence and final performance.
  • Max epochs = 5
    Table 5; early stopping not described; may overfit the ICH corpus.
  • Batch size = 4
    Table 5; chosen without ablation; interacts with LoRA and learning rate.
  • LoRA rank and alpha = not reported
    Table 5 reports 'Finetuning Type LoRA' but no rank or alpha; these hyperparameters materially affect adaption quality.
  • Base model selection = Qwen2.5-7B-Chat (chosen by eval)
    Discussion Section 5 states the base architecture was selected because of its evaluation performance; this is a model-selection parameter fitted to the test tasks.
  • Evaluation sample size = 100 per task
    Section 4.4; no power analysis or confidence intervals; small samples inflate variance.
assumptions (5)
  • domain assumption The ICH corpus from the China Intangible Cultural Heritage Network and academic abstract databases is authoritative, complete, and clean enough for training.
    Section 3.2.1 describes rule-based cleansing but reports no annotation agreement, error rates, or quality audit.
  • domain assumption Synthetic QA pairs generated by Qwen2-72B-Instruct are valid and error-free after manual verification.
    Section 3.3 states manual verification but gives no protocol, sample size, or correction rate.
  • domain assumption ROUGE, BLEU, and chrF scores are adequate measures of answer quality for ICH tasks.
    Section 4.2 uses only n-gram overlap; no human evaluation, faithfulness, or factual correctness checks are reported.
  • domain assumption The 100 evaluation samples per task are representative and do not overlap the training corpus.
    Section 4.4 gives no details on how evaluation samples were constructed, no contamination check, and no separate validation split.
  • ad hoc to paper Qwen2.5-7B-Chat is a legitimate base for ICH-Qwen and its selection based on evaluation performance does not bias the comparison.
    Discussion Section 5 reveals the base model was chosen from the same set of evaluated models; the paper does not describe a held-out selection procedure.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ICH-Qwen: A Large Language Model Towards Chinese Intangible Cultural Heritage." pith.science (2026). https://pith.science/paper/NQNLNUQK

@misc{pith2026250523831,
  author       = {Pith},
  title        = {Pith review of: ICH-Qwen: A Large Language Model Towards Chinese Intangible Cultural Heritage},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NQNLNUQK}},
  note         = {Machine review of arXiv:2505.23831}
}
read the original abstract

The intangible cultural heritage (ICH) of China, a cultural asset transmitted across generations by various ethnic groups, serves as a significant testament to the evolution of human civilization and holds irreplaceable value for the preservation of historical lineage and the enhancement of cultural self-confidence. However, the rapid pace of modernization poses formidable challenges to ICH, including threats damage, disappearance and discontinuity of inheritance. China has the highest number of items on the UNESCO Intangible Cultural Heritage List, which is indicative of the nation's abundant cultural resources and emphasises the pressing need for ICH preservation. In recent years, the rapid advancements in large language modelling have provided a novel technological approach for the preservation and dissemination of ICH. This study utilises a substantial corpus of open-source Chinese ICH data to develop a large language model, ICH-Qwen, for the ICH domain. The model employs natural language understanding and knowledge reasoning capabilities of large language models, augmented with synthetic data and fine-tuning techniques. The experimental results demonstrate the efficacy of ICH-Qwen in executing tasks specific to the ICH domain. It is anticipated that the model will provide intelligent solutions for the protection, inheritance and dissemination of intangible cultural heritage, as well as new theoretical and practical references for the sustainable development of intangible cultural heritage. Furthermore, it is expected that the study will open up new paths for digital humanities research.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

20 extracted references · 13 canonical work pages

  1. [1]

    A neural probabilistic language model[J]

    Bengio Y, Ducharme R, Vincent P, et al. A neural probabilistic language model[J]. Jo urnal of machine learning research, 2003, 3(Feb): 1137 -1155

    work page 2003

  2. [2]

    Neural networks and physical systems with e mergent collective comp utational abilities[J]

    HOPFIELD J J. Neural networks and physical systems with e mergent collective comp utational abilities[J]. Pro ceedings of the National Academy of Sciences,1982,79(8):255 4-2558

    work page 1982

  3. [3]

    Long short -term memory[J]

    HOCHREITER S ，SCHMIDHUBER J. Long short -term memory[J]. Neural Computatio n,1997,9(8):1735-1780

    work page 1997

  4. [4]

    Neural machine translation by jointly learning to alig n and trans late[J]

    Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to alig n and trans late[J]. arXiv preprint arXiv:1409.0473, 2014

    arXiv 2014

  5. [5]

    Attention is all you need[J]

    Vaswani A, Shazeer N , Parmar N, et al. Attention is all you need[J]. Advances in ne ural info rmation processing systems, 2017, 30

    work page 2017

  6. [6]

    OpenAI,Josh A,Steven A,et al.GPT -4 Technical Report[R].arXiv preprint arXiv,202 3,23 03(08774)

    work page

  7. [7]

    A comprehensive capability analysis of gpt -3 and gpt - 3.5 series models[EB/OL]

    YE J, CHEN X, XU N, et al. A comprehensive capability analysis of gpt -3 and gpt - 3.5 series models[EB/OL]. [2024-12-18]. https://arxiv.org/abs/2303.10420

    arXiv 2024

  8. [8]

    Raffel C,Shazeer N,Roberts A ,et al.Ex ploring the Limits of Transfer Learning with a Unified Text -to-Text Transformer[J/ OL].Machine Learning,2020.https://arxiv.org/abs/19 10.10683

    work page 2020

Show all 20 references
  1. [9]

    arXiv preprint arXiv ：1810.04805，2018.https: //doi.org/10.48550/arXiv.1810.04805

    Devlin J ，Chang M W ，Lee K ，et al.Bert:Pre -training of Deep Bidirectional Transfor mers for Ianguage Underst anding[J/OL]. arXiv preprint arXiv ：1810.04805，2018.https: //doi.org/10.48550/arXiv.1810.04805

    2018 doi arXiv

  2. [10]

    Exploring Ho w LLMs Capture and Repre sent Domain -Specific Knowledge[J]

    Garcia M H, Couturier C, Diaz D M, et al. Exploring Ho w LLMs Capture and Repre sent Domain -Specific Knowledge[J]. arXiv preprint arXiv:2504.16871, 2025

    2025 arXiv

  3. [11]

    GujiBERT and GujiGPT: Construction of intelligent i nformation processing foundation language models for ancient texts[J]

    Wang D, Liu C, Zhao Z, et al. GujiBERT and GujiGPT: Construction of intelligent i nformation processing foundation language models for ancient texts[J]. arXiv preprint arXiv:2307.05354, 2023

    2023 arXiv

  4. [12]

    Ancient Text Translation Model Optimized with GujiB ERT and Entropy- SkipBERT[J]

    Yu F, Han R, Zhang Y, et al. Ancient Text Translation Model Optimized with GujiB ERT and Entropy- SkipBERT[J]. Electronics, 2024, 13(22): 4492

    2024

  5. [13]

    LawGPT: A Chinese Legal Knowledge -Enhanced Large Language Model[J]

    Zhou Z, Shi J X, Song P X, et al. LawGPT: A Chinese Legal Knowledge -Enhanced Large Language Model[J]. arXiv preprint arXiv:2406.04614, 2024

    2024 arXiv

  6. [14]

    ChatGLM -Math: Improving Math Problem-Solving in Lar ge Language Models with a Self -Critique Pipel ine[J]

    Xu Y, Liu X, Liu X, et al. ChatGLM -Math: Improving Math Problem-Solving in Lar ge Language Models with a Self -Critique Pipel ine[J]. arXiv preprint arXiv:2404.02893, 2024

    2024 arXiv

  7. [15]

    Biomedgpt: Open multimodal generative pre -trained tra nsformer for biomedicine[J]

    Luo Y, Zhang J, Fan S, et al. Biomedgpt: Open multimodal generative pre -trained tra nsformer for biomedicine[J]. arXiv preprint arXiv:2308.09442, 2023

    2023 arXiv

  8. [16]

    Application of digital technology in intangible cultural heritage protection[J]

    Liu Y. Application of digital technology in intangible cultural heritage protection[J]. Mobile Information Systems, 2022, 2022(1): 7471121

    2022

  9. [17]

    Preservation of intangible and tangible cultural he ritage using digital te chnology[J]

    Amali L N, Katili M R, Ismail W. Preservation of intangible and tangible cultural he ritage using digital te chnology[J]. Indonesian Journal of Electrical Engineering and Co mputer Science, 2022, 28(2): 980-986

    2022

  10. [18]

    The Digital Protection and Inheritance of Intangible Cult ural Heritage[J/OL]

    YUE M, WANG G S, LI Z. The Digital Protection and Inheritance of Intangible Cult ural Heritage[J/OL]. Highlights in Art and Design, 2022[2025-01-12]

    2022

  11. [19]

    Digital practices to enhance in tangible cul tural heritage[J]

    Giovannini E C, Lo Turco M, Tomalini A. Digital practices to enhance in tangible cul tural heritage[J]. The International Archives of the Photogr ammetry, Remote Sensing a nd Spatial Information Sciences, 2021, 46: 273-278

    2021

  12. [20]

    Discussion on the applica tion of computer digital technology in the protection of i ntangible cultural heritage[C]//Journal of Physics: Conference Series

    Shuai H, Yu W. Discussion on the applica tion of computer digital technology in the protection of i ntangible cultural heritage[C]//Journal of Physics: Conference Series. IO P Publis hing, 2021, 1915(3): 032048

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.