Pith. sign in

REVIEW 5 major objections 5 minor 20 references

WisdomBot: Tuning Large Language Models with Artificial Intelligence Knowledge

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read WisdomBot claims that fine-tuning open Chinese LLMs on Bloom's Taxonomy-structured textbook data, plus local and search-engine retrieval at inference, yields education-specialized assistants that beat their base models on every evaluated…

desk verdict Sensible domain-adaptation recipe undermined by a self-referential evaluation: GPT-4 generated the training data and also judged the responses, so the headline win rates overstate what is demonstrated. read the letter →

arxiv 2501.12877 v1 pith:L53DMJUS submitted 2025-01-22 cs.CL

classification cs.CL
keywords largelanguagemodelseducationBloom'sTaxonomyinstructiontuningself-instructionretrievalaugmentationChineseLLMsC-Eval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that open Chinese large language models can be turned into education-specialized assistants by fine-tuning them on synthetic instruction data organized around Bloom's Taxonomy, then adding retrieval at inference time. The authors build WisdomBot from coarse-grained AI textbook concepts and ChatGPT-generated fine-grained concepts and questions, producing 38,784 instruction-output pairs covering three educational tasks and six cognitive levels. They fine-tune Chinese-Alpaca-7B/13B and Qwen-7B-Chat, and report that WisdomBot wins against the original models on at least 63% of comparisons across every evaluation part, reaching 100% on professional question answering, with gains also on the C-Eval benchmark. If these results hold, the paper offers a practical recipe for adapting general LLMs to education without changing their architecture, using only data construction and retrieval.

What carries the argument

The load-bearing mechanism is a data-generation loop anchored to Bloom's Taxonomy, the revised educational framework that classifies learning objectives along a knowledge dimension (factual, conceptual, procedural, metacognitive) and a cognitive-process dimension (remember, understand, apply, analyse, evaluate, create). The paper turns these dimensions into a production line: coarse textbook concepts seed ChatGPT-generated fine-grained concepts; those concepts fill templates for professional Q&A, test problem generation, and intelligent tutoring; the resulting 38,784 instruction-output pairs are the supervised fine-tuning signal. At inference, a second mechanism, retrieval augmentation, compensates for the model's fixed parametric knowledge by retrieving textbook chunks from a local vector store or search results from a search engine and attaching them to the query.

What would settle it

Have independent human experts score WisdomBot and the original models on a fresh set of AI questions written without ChatGPT or the paper's template pipeline, with response order swapped across trials; if the reported 63-100% win rates shrink to chance, the central claim fails. A second check asks GPT-4 to compare base-model answers rewritten to match WisdomBot's structure; if the advantage vanishes, the judge was rewarding format.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that educational quality in LLMs can be manufactured through data structure, not architecture. The authors manually extract 117 coarse-grained AI knowledge concepts from textbooks, use ChatGPT in a self-instruction loop to expand them into 981 fine-grained concepts and 1,196 learner questions, and then fill 39 instruction templates aligned with Bloom's Taxonomy's six cognitive processes to create 38,784 instruction-output pairs. Fine-tuning these pairs into Chinese-Alpaca-7B, Chinese-Alpaca-13B, and Qwen-7B-Chat yields WisdomBot. During inference, a local vector knowledge base built from textbook chunks and a search engine are used as optional retrieval sources. The paper reports that WisdomBot beats the original models on at least 63% of comparisons in every evaluation part, reaches 100% on professional question answering versus Chinese-Alpaca-7B, and improves C-Eval validation accuracy, with the largest gains in STEM and the Hard subset.

Load-bearing premise

The load-bearing premise is that the self-built 130-question test set and its GPT-4 judge measure real educational quality rather than similarity to the ChatGPT-generated data used in training.

Editorial extensions

If this is right

  • If the win rates generalize outside the test set, any open Chinese LLM can be repurposed for education by generating textbook-anchored, taxonomy-structured synthetic data and fine-tuning; no architectural redesign is required.
  • Retrieval becomes a controllable quality switch: the same model can answer from memory alone or, when the user opts in, with local textbook chunks or fresh search-engine results; the paper reports local retrieval raising professionalism scores from 30% to 70% and search retrieval raising factual correctness from 35% to 93% on its curated sets.
  • The C-Eval results suggest the training data transfers beyond the self-constructed test set: WisdomBot's overall validation score (62.06) exceeds Qwen-7B-Chat's (59.37), with the Hard-subset score rising from 35.14 to 49.26, implying the hardest questions benefit most.
  • A taxonomy-driven data recipe of this kind can be updated cheaply as textbooks or curricula change: regenerating concepts and instructions for new material replaces expensive human-annotated dataset collection.
  • The same fine-tuned model can be deployed in both closed-book and open-book modes, since retrieval is an optional inference-time feature rather than a training requirement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested implication is that the win-rate gap may overstate content gains, since GPT-4, the model family that generated much of the training data, also judges the comparisons; a human-authored exam set would quantify any judge bias.
  • The pipeline's textbook-anchored design suggests a cheap path to other subjects and languages: swap the textbook corpus and retrieval backend without retraining from scratch, but this is not demonstrated beyond Chinese AI instruction.
  • In deployment, retrieval augmentation could be gated by query type, routing factual questions to search, conceptual questions to the local textbook store, and open-ended tutoring to memory; the paper tests the components separately but does not test such a router.
  • The self-constructed test set and the training data share a ChatGPT generation pipeline, so the measured win rates are best read as upper bounds until an independently authored, human-graded evaluation is available.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes WisdomBot, a fine-tuning pipeline for adapting Chinese open-source LLMs (Chinese-Alpaca-7B/13B and Qwen-7B-Chat) to educational tasks. Training data are generated from coarse-grained textbook concepts, expanded into fine-grained concepts and questions via ChatGPT, and organized into instruction-output pairs guided by Bloom's Taxonomy. During inference, local knowledge-base retrieval and search-engine retrieval are optionally used to augment factual accuracy. The authors evaluate the resulting models on a self-constructed 130-question test set (via human and GPT-4 pairwise comparisons) and on the C-Eval validation set, reporting win rates of at least 63% for WisdomBot over baselines and improved C-Eval scores, especially on AI-related subjects.

Significance. If the claims are substantiated, the paper contributes a practical recipe for domain-adapting Chinese LLMs to education, combining pedagogical theory with synthetic instruction data and retrieval. The use of a public benchmark (C-Eval) provides some external evidence, and the reported gains on AI-related subjects are potentially valuable. The retrieval-augmentation results, if properly validated, could also be a useful component. However, the central claim that WisdomBot produces 'more reliable and professional' responses rests mainly on a self-constructed test set and GPT-4 evaluations that share the same generation pipeline as the training data; this circularity currently prevents the conclusion from being accepted without further evidence.

major comments (5)
  1. [Section 4.3.1 and 4.4] The self-constructed test set and the GPT-4 evaluation are too closely tied to the training-data generation pipeline to support the headline claim. Section 3.2 describes how ChatGPT generates and filters instructions, and Section 4.3.1 does not describe how the test questions were constructed; given the same concept/template pipeline, the test set likely overlaps the training distribution. GPT-4, used as the judge in Figures 7-8, is the same model family that produced much of the synthetic data; preference judgments could therefore track stylistic similarity to GPT-4's own generations rather than educational quality. The paper must either provide an independent test set from a disjoint source, a human evaluation with a detailed protocol and inter-rater agreement, or a demonstration that GPT-4 preferences correlate with human judgments on a held-out subset. Without this, the 'at least 63% winning rate' claim is not established.
  2. [Section 4.5, Tables 1-5] The C-Eval results are reported for a single model named 'WisdomBot', but Section 4.2 states that fine-tuning was applied to Chinese-Alpaca-7B, Chinese-Alpaca-13B, and Qwen-7B-Chat. It is unclear which base model underlies the C-Eval row. If the C-Eval WisdomBot is Qwen-7B-Chat, comparing it to Chinese-Alpaca-7B is not a matched comparison; if it is Chinese-Alpaca-7B, the text should state this explicitly. In addition, all subset tables report small numbers of items (e.g., many percentages based on fewer than 20 questions) with no error bars or significance tests, so the claim of improvement 'in most subjects' is not quantified.
  3. [Sections 4.4 and 4.6] The reported win rates and cognitive-ability scores lack per-category sample sizes and uncertainty. For example, the 100% winning rate in Figure 7 for professional question answering vs Chinese-Alpaca-7B could be based on a very small number of items; Figure 4 shows only the total counts for three functions and one aggregate for cognitive capacities, not per-category counts for the 70 and 60 instances. Similarly, the logical reasoning accuracies in Table 6 (8%, 46%, 52%) are reported without the number of samples or a description of the scoring rubric. Report the number of items in each cell and include confidence intervals or significance tests (e.g., binomial tests) to support the claimed superiority.
  4. [Section 4.2 (human evaluation)] The human evaluation protocol is under-specified. The text states that ten AI experts compared responses, but it does not say whether experts were blind to model identity, what instructions or scoring rubric they received, how ties were resolved, or whether disagreements were reconciled. Inter-annotator agreement (e.g., Cohen's kappa) is not reported. Without these details, the aggregate percentages in Figures 5-6 cannot be interpreted as reliable evidence that WisdomBot outperforms the baselines.
  5. [Section 4.7, Table 7] The retrieval-augmentation evaluation uses GPT-4 as the judge for 'professional level' and 'correctness' on small curated sets (50 and 30 questions), but the paper does not specify which model variant is used (presumably WisdomBot), what prompts are given to GPT-4, or whether any human verification was performed. Since the central contribution includes retrieval enhancement, the evaluation should at least include a sample of human judgments or a clearly described scoring rubric, and the sample sizes should be reported per condition.
minor comments (5)
  1. [Abstract] The phrase 'artificial generated intelligence (AGI)' should be corrected to 'artificial general intelligence (AGI)'.
  2. [Section 2.2] Bloom's Taxonomy is cited as reference [15], but reference [15] is the GLM-130B technical report; the correct citation is the Anderson et al. revision (reference [10]).
  3. [Table 3] In the row for 'Education science', the WisdomBot score appears as '72.4 1 81.82', which seems to be a formatting error; it should likely read '72.41'.
  4. [Figure 9 caption] The caption contains a typo: 'retiveval' should be 'retrieval'.
  5. [Section 4.6, Table 6] The caption of Table 6 reads 'Comparisons on retrieval enhancements' but the table reports creativity, personalized ability, and logical reasoning; the caption should be updated to reflect the content.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are supported by human evaluation and an external benchmark, with only a methodological evaluator-overlap caveat.

full rationale

The paper's derivation chain is: textbooks and Bloom's Taxonomy guide the construction of knowledge concepts; ChatGPT-assisted self-instruction generates instructions; human experts with GPT assistance write answers; fine-tuning transforms general LLMs into WisdomBot; retrieval augmentation supplies textbook and search-engine content at inference. The effectiveness claims rest on three evaluations: human expert pairwise comparison (Figures 5-6), GPT-4 pairwise comparison (Figures 7-8), and zero-shot C-Eval scores (Tables 1-5). No step is equivalent to its input by construction. Fine-tuning on GPT-assisted instruction-output pairs does not force a pairwise win against Chinese-Alpaca-7B or Chinese-Alpaca-13B, and the human evaluation provides an independent judgment that the fine-tuned model is preferred. The C-Eval benchmark is external, with fixed questions, and the reported improvements (e.g., overall 59.37 to 62.06, and STEM/Hard gains) do not reduce to the training-data construction. The main caveat is methodological rather than circular: GPT-4 is used both as an assistant in generating training answers (Section 3.2, 'human experts utilizing GPT as an assistant') and as an evaluator (Section 4.4, 'GPT-4 evaluation'), which can introduce style-matching bias; the self-constructed test set construction is also not specified in detail. However, these are validity risks, not construction-level circularity, and the human evaluation plus the external C-Eval benchmark keep the central claim from reducing to the GPT-4 judge. Accordingly, the paper is not significantly circular.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim depends on evaluation validity and on the quality of ChatGPT-generated data. There are no fitted constants in a mathematical derivation, but there are unstated design choices, notably retrieval k and filtering thresholds, plus domain assumptions about Bloom's Taxonomy and GPT-4 as judge.

free parameters (2)
  • Fine-tuning hyperparameters = learning rate 2e-5, LoRA rank 8 for 7B and 32 for 13B, full fine-tuning for Qwen, batch size 16 per GPU
    Reported in Section 4.2; chosen by hand without ablation, so a replication would need to retune on the regenerated dataset.
  • Number of retrieved segments k for local knowledge base = not reported
    Section 3.3 describes top-k retrieval but never reports k; the retrieval gain in Table 7 depends on this unstated choice.
assumptions (4)
  • domain assumption Bloom's Taxonomy's knowledge and cognitive-process dimensions are a valid guide for generating instruction data that improves LLM educational ability.
    The entire data construction in Section 3.2 maps concepts and templates to Bloom's dimensions; no control condition tests whether the taxonomy, rather than the extra domain data, causes the gains.
  • domain assumption ChatGPT-generated fine-grained concepts and instructions are accurate, complete, and diverse enough for training.
    Section 3.1 relies on ChatGPT as an AI learner to produce 981 concepts and 1,196 questions, with only manual cleaning. Errors would propagate into training data and into the self-constructed test set.
  • domain assumption GPT-4 judgments of response quality correlate with human educational quality.
    Section 4.4 uses GPT-4 as evaluator; no agreement statistics with the ten human experts are reported, and GPT-4 generated part of the training data.
  • domain assumption The C-Eval validation subset is a reliable proxy for educational competence at this sample size.
    Section 4.5 uses the validation set without statistical tests; many subject subsets have fewer than twenty questions, so a few correct answers swing percentages by over five points.

how reviews work

0 comments
Cite this review

Pith. "Pith review of WisdomBot: Tuning Large Language Models with Artificial Intelligence Knowledge." pith.science (2026). https://pith.science/paper/L53DMJUS

@misc{pith2026250112877,
  author       = {Pith},
  title        = {Pith review of: WisdomBot: Tuning Large Language Models with Artificial Intelligence Knowledge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L53DMJUS}},
  note         = {Machine review of arXiv:2501.12877}
}
read the original abstract

Large language models (LLMs) have emerged as powerful tools in natural language processing (NLP), showing a promising future of artificial generated intelligence (AGI). Despite their notable performance in the general domain, LLMs have remained suboptimal in the field of education, owing to the unique challenges presented by this domain, such as the need for more specialized knowledge, the requirement for personalized learning experiences, and the necessity for concise explanations of complex concepts. To address these issues, this paper presents a novel LLM for education named WisdomBot, which combines the power of LLMs with educational theories, enabling their seamless integration into educational contexts. To be specific, we harness self-instructed knowledge concepts and instructions under the guidance of Bloom's Taxonomy as training data. To further enhance the accuracy and professionalism of model's response on factual questions, we introduce two key enhancements during inference, i.e., local knowledge base retrieval augmentation and search engine retrieval augmentation during inference. We substantiate the effectiveness of our approach by applying it to several Chinese LLMs, thereby showcasing that the fine-tuned models can generate more reliable and professional responses.

Figures

Figures reproduced from arXiv: 2501.12877 by the authors.

Figure 1
Figure 1. Limitations of general LLMs in education: (a) comprehension ability, (b) out-of-date [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Training pipeline. We collect knowledge concepts and instructions under the guidance [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Demonstrations of the dataset. academic behaviors and manifestations of learning that need to be mastered, including six categories: remember, understand, apply, analyse, evaluate, and create, which are arranged in ascending order of cognitive complexity. This theory has been used to explore the weaknesses of ChatGPT in the field of education [17]. 3 Methods The details of our finetuning method will be introduced in… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Distribution of test data. 4.2 Experiment Detail We adopt AdamW optimizer with an initial learning rate of 2e-5. The models are trained on 8 A100 GPUs and the batch size of each GPU is set to 16. We use Low-Rank Adaptation [19] training strategy to reduce training para…
Figure 5
Figure 5. Figure 5: Human evaluation of whether WisdomBot outperforms Chinese-Alpaca-7B. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Human evaluation of whether WisdomBot outperforms Chinese-Alpaca-13B. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: GPT-4 evaluation of whether WisdomBot outperforms Chinese-Alpaca-7B. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: GPT-4 evaluation of whether WisdomBot outperforms Chinese-Alpaca-13B. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Case examples generated by WisdomBot and baselines: (a) creativity, (b) personalized [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 7 canonical work pages

  1. [1]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023

  2. [2]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023

  3. [3]

    Knowledgeable or educated guess? revisiting language models as knowledge bases

    Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingyong Yan, Meng Liao, Tong Xue, and Jin Xu. Knowledgeable or educated guess? revisiting language models as knowledge bases. arXiv preprint arXiv:2106.09231, 2021

  4. [4]

    Evaluating the logical reasoning ability of chatgpt and gpt-4

    Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. Evaluating the logical reasoning ability of chatgpt and gpt-4. arXiv preprint arXiv:2304.03439, 2023

  5. [5]

    Can Generative Pre-trained Language Models Serve as Knowledge Bases for Closed-book QA?

    Cunxiang Wang, Pai Liu, and Yue Zhang. Can generative pre-trained language models serve as knowledge bases for closed-book qa? arXiv preprint arXiv:2106.01561, 2021

  6. [6]

    Chatgpt is not enough: Enhancing large language models with knowledge graphs for fact-aware language modeling

    Linyao Yang, Hongyang Chen, Zhao Li, Xiao Ding, and Xindong Wu. Chatgpt is not enough: Enhancing large language models with knowledge graphs for fact-aware language modeling. arXiv preprint arXiv:2306.11489, 2023

  7. [7]

    Education in the era of generative artificial intelligence (ai): Understanding the potential benefits of chatgpt in promoting teaching and learning

    David Baidoo-Anu and Leticia Owusu Ansah. Education in the era of generative artificial intelligence (ai): Understanding the potential benefits of chatgpt in promoting teaching and learning. Available at SSRN 4337484, 2023

  8. [8]

    Efficient and effective text encoding for chinese llama and alpaca

    Yiming Cui, Ziqing Yang, and Xin Yao. Efficient and effective text encoding for chinese llama and alpaca. arXiv preprint arXiv:2304.08177, 2023

Show all 20 references
  1. [9]

    Stanford alpaca: An instruction-following llama model, 2023

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following llama model, 2023

  2. [10]

    A revision of bloom’s taxonomy of educational objectives

    Lorin W Anderson, David R Krathwohl, PW Airasian, KA Cruikshank, RE Mayer, PR Pintrich, et al. A revision of bloom’s taxonomy of educational objectives. A Taxonomy for Learning, Teaching and Assessing. Longman, New York, 2001

  3. [11]

    On pedagogy of personality assessment: Application of bloom’s taxonomy of educational objectives

    Tatyana V Ramirez. On pedagogy of personality assessment: Application of bloom’s taxonomy of educational objectives. Journal of personality assessment, 99(2):146–152, 2017. 11

  4. [12]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instruc- tions, 2023

  5. [13]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

  6. [14]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:277...

  7. [15]

    Glm-130b: An open bilingual pre-trained model

    Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414, 2022

  8. [16]

    Handbook i: cognitive domain

    Benjamin S Bloom, Max D Engelhart, EJ Furst, Walker H Hill, and David R Krathwohl. Handbook i: cognitive domain. New York: David McKay, 1956

  9. [17]

    Towards mitigating chatgpt’s negative impact on education: Optimizing question design through bloom’s taxonomy, 2023

    Saber Elsayed. Towards mitigating chatgpt’s negative impact on education: Optimizing question design through bloom’s taxonomy, 2023

  10. [18]

    Finetuned language models are zero-shot learners

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021

  11. [19]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021

  12. [20]

    C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems , 36,...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.