Pith. sign in

REVIEW 4 major objections 6 minor 61 references

Resolving Knowledge Conflicts in Domain-specific Data Selection: A Case Study on Medical Instruction-tuning

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read For domain-specific instruction-tuning, data selection should be driven by knowledge conflicts between the model's pretrained knowledge and the training data, and filtering out such conflicts improves medical QA performance.

desk verdict Practical GPT-4-free data selection with consistent gains, but the 'knowledge conflict' framing rests on an unexamined premise about what instruction tuning can learn. read the letter →

arxiv 2505.21958 v1 pith:2NQAMUUG submitted 2025-05-28 cs.CL

Qihuang Zhong , Liang Ding , Fei Liao , Juhua Liu , Bo Du , Dacheng Tao This is my paper
classification cs.CL
keywords dataselectioninstructiontuningknowledgeconflictmedicalquestionansweringhallucinationLLMdomainadaptationNLIentailmentscoringmultilingualgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Domain-specific instruction-tuning datasets contain examples whose answers contradict what the base LLM learned during pretraining, and the paper argues that training on those examples hurts performance and encourages hallucination. To handle this, it proposes a knowledge-aware data selection method, KDS, that scores every instruction example with two metrics: how often the model's sampled responses are entailed by the reference answer (knowledge alignment, KA) and how consistent the model's multiple responses are with each other (knowledge consistency, KC). Sorting by these scores and filtering low-quality, redundant examples yields a small subset that the model can actually absorb. In medical QA experiments on LLaMA-3 and Qwen2.5 models, KDS-selected 5,000-example subsets outperform random selection, full fine-tuning, and general-domain data selection baselines, with average gains up to +2.56 points over the base model and reduced hallucination on Med-HALT.

What carries the argument

The load-bearing mechanism is a pair of scores computed before training. Knowledge alignment (KA) is the fraction of $m$ sampled responses to a question that an NLI model classifies as entailed by the reference answer. Knowledge consistency (KC) clusters the $m$ responses by entailment, assigns each cluster a probability, and normalizes the cluster entropy so a model that gives the same answer each time scores near 1. The full pipeline sorts the dataset by these scores, then applies a quality filter in which the base LLM rates each answer from 0 to 5, and a diversity filter that drops examples too close to already-selected ones in embedding space.

What would settle it

Take a set of medical questions whose answers reflect recently updated guidelines that the base model has not seen, split them into high-KA/KC and low-KA/KC groups, fine-tune the same base model on each group separately, and test on held-out updated-guideline questions; if the low-KA/KC group learns the new facts and the high-KA/KC group does not improve, the claim that conflicting data must be filtered breaks down.

Watch

Extended reading notes

Core claim

The central discovery is that knowledge conflicts, not data quality or complexity alone, are the main obstacle in domain-specific instruction-tuning, and they can be quantified and filtered out before training. The paper claims that when an LLM's multiple sampled responses fail to match the reference answer (low KA) or disagree among themselves (low KC), the example encodes knowledge the model does not have, and fine-tuning on it damages prior abilities and induces hallucination. Conversely, keeping high-KA and high-KC examples lets instruction-tuning stimulate existing abilities rather than overwrite them. Across three base models and several medical QA benchmarks, the authors report consistent improvements, better multilingual generalization, and up to +9.86 points over random selection on the Med-HALT hallucination test.

Load-bearing premise

The method assumes that instruction-tuning cannot add new knowledge to the model, so any example whose answer the model cannot already produce is treated as a harmful conflict; if instruction-tuning can teach new or updated facts, the method would throw away exactly the examples needed for domain learning.

Editorial extensions

If this is right

  • On medical QA benchmarks, KDS-selected 5,000-example subsets improve average accuracy by up to +2.56 points over the base model across LLaMA-3-8B and Qwen2.5-7B/14B, beating Full-SFT, Random, and the general-domain baselines IFD, Alpagasus, and DEITA.
  • At a 1,000-example budget, KDS-KC reaches an average of 48.77, higher than the best non-KDS method's 48.46 at a 5,000-example budget, showing improved data efficiency.
  • Models tuned with KDS generalize to multilingual medical QA, with up to +4.17 average points over the base on MMedBench for Qwen2.5-7B, including +6.25 points in Russian and +3.79 points in Chinese.
  • On Med-HALT reasoning hallucination tests, KDS-KC improves the average score by up to +9.86 points over Random, countering the hallucination increase that instruction-tuning usually brings.
  • The same KDS pipeline transfers to Chinese finance instruction data, outperforming Full-SFT by up to +1.67 average points on FinanceIQ.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because KDS scores examples by what the base model already knows, its benefit should shrink as the base model's domain knowledge grows; on a model that already answers most medical questions correctly, the filter will have few examples to remove, so gains may come mostly from the quality and diversity filters rather than conflict resolution.
  • The KA metric depends on the NLI model's accuracy; the paper measures 89% accuracy on 100 manually labeled medical pairs for the large DeBERTa model, so a medical-domain NLI model could sharpen KA and KC further, which the authors list as future work.
  • The reference-free KC metric could be applied where no gold answers exist, such as proprietary or private instruction data, and could be combined with other reference-free signals like semantic entropy to flag uncertain examples.
  • If instruction-tuning can in fact update knowledge, KDS would trade away factual updates for stability; a variant that keeps a small budget of high-conflict, externally verified examples might combine both.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes KDS, a knowledge-aware data selection framework for domain-specific instruction tuning. KDS scores each candidate instruction sample by two metrics: knowledge alignment (KA), the fraction of sampled base-model responses that the NLI model judges as entailing the reference answer, and knowledge consistency (KC), a cluster-entropy-based measure of agreement among multiple sampled responses. After quality and diversity filtering, the top-k samples are used for LoRA fine-tuning. On a medical instruction-tuning testbed built from MedAlpaca subtasks, the authors report that KDS outperforms existing data-selection baselines across LLaMA-3-8B-Instruct and Qwen-2.5-7B/14B-Instruct, improves multilingual generalization on MMedBench, reduces hallucination on Med-HALT, and transfers to a Chinese finance QA benchmark.

Significance. The idea of explicitly measuring knowledge conflicts between a base LLM's parametric knowledge and the knowledge required by instruction data is timely and potentially useful for domain-specific tuning. The paper provides a fairly comprehensive suite of experiments: three model families/sizes, multiple medical QA benchmarks, a long-form QA evaluation, hallucination tests, scalability analysis, and a cross-domain finance experiment. The framework is simple and does not require additional trained scorers beyond an off-the-shelf NLI model. If the central premise is accepted, the consistent gains over baselines suggest the approach has practical value. However, the strength of the claims is limited by the issues detailed below: the reported average column in the main table is not reproducible from the displayed numbers, no statistical significance is provided, hyperparameters are tuned on the evaluation set, and the motivating assumption that instruction tuning cannot add new knowledge is neither tested nor properly delimited. These issues are load-bearing for the paper's central claim, so the current version requires substantial revision.

major comments (4)
  1. [Table III] The 'Avg.' column in Table III does not correspond to the macro-average of the 10 listed scores, despite the caption stating 'Avg. denotes the macro-average performance.' For example, in the LLaMA-3-8B-Instruct Base row, the simple average of HoT, MedMCQA, MedQA, PubmedQA, and the six MMLU-Medical subjects is 64.23, whereas the reported Avg. is 47.41; similar discrepancies hold for other rows and models. Because the headline gains (up to +2.56%) are computed from this Avg. column, the aggregation formula needs to be specified precisely and the numbers made internally consistent.
  2. [Section IV-B] All results are reported from what appear to be single runs, with no standard deviations, confidence intervals, or significance tests. Many of the improvements over baselines are small; for instance, on Qwen-2.5-7B-Instruct, KDS-KA+KC achieves 51.40% versus Alpagasus at 50.63%, and on several individual MMLU-Medical subjects the differences are under one point. The claim of 'significant and consistent performance gains' (Section IV-B, item 2) is therefore not statistically supported. Please provide mean and standard deviation over at least three seeds for the main comparisons, and report paired significance tests where appropriate.
  3. [Section IV-C and Figure 5] The threshold hyperparameters τ (quality) and λ (diversity) are chosen by scanning values on the evaluation benchmarks themselves: Figure 5(b) and (c) show average performance on HoT and the multiple-choice QA benchmarks as a function of τ and λ, and the default values τ=3 and λ=0.9 are selected from those maxima. This constitutes selection on the test data and can inflate the reported performance of KDS relative to baselines. The thresholds should be tuned on a separate validation split (or via nested cross-validation) and the resulting performance reported, or the authors should argue credibly that the thresholds are not dataset-adaptive.
  4. [Section I, paragraph 2; Section II-B] The framework's motivation rests on the claim, attributed to Ren et al. [16], that 'instruction-tuning fails to learn additional knowledge.' This is a strong and contested premise. If instruction tuning can in fact impart new or updated facts, then KDS's KA/KC metrics preferentially discard exactly those samples: low KA means the base model cannot already produce the reference answer, and low KC means its sampled responses are inconsistent. The paper does not provide direct evidence for the premise, nor does it separate 'knowledge conflict' from 'novel/unfamiliar knowledge.' A concrete test would be to evaluate KDS on a set of questions whose answers are genuinely absent from the base model's pretraining (e.g., newly updated medical guidelines or a deliberately constructed novel-facts set) and compare KDS against a variant that selects low-KA/low-KC data. Without such an experiment, the reported gains are compatible with a simpler explanation: KDS selects familiar examples the model already knows, improving format and alignment rather than genuinely resolving knowledge conflicts.
minor comments (6)
  1. [Section III-B] The text refers to 'intro-memory conflicts'; this should be 'intra-memory conflicts' to match the standard terminology used elsewhere in the paper.
  2. [Section IV-A1] In the description of the long-form medical QA benchmark, 'bais' should be 'bias'.
  3. [Figure 1 caption] The caption writes 'Complexity deita' and 'Quality deita'; these should presumably read 'Complexity DEITA' and 'Quality DEITA' for consistency with the baseline name.
  4. [Section V-D and Table VIII] The reliability evaluation of the NLI model is based on only 100 manually labeled pairs, with the large NLI model reaching 89% accuracy. No confidence interval or error analysis is provided. Given that KA/KC are the core metrics of the framework, a larger and more systematically sampled evaluation set would strengthen the claim that the NLI judgments are reliable.
  5. [Section IV-A2] The figure near the quality score distribution contains a rendering artifact ('glyph1197umber') that should read 'number.'
  6. [Section IV-B, item 3] The long-form QA evaluation relies on a single LLM judge (GPT-4o-mini) without reporting inter-judge agreement or human validation. Since the win rates in Figure 4 are used to support the hallucination and quality claims, a brief discussion of judge reliability would be useful.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: KDS's selection scores are measured from the base model's own responses, but the claimed performance gains are evaluated on external benchmarks and the key premise about instruction tuning being unable to add knowledge comes from cited external prior work, not from the paper's own derivation.

full rationale

The paper's derivation chain is not circular under the stated rules. KDS defines two knowledge-aware scores, KA (Eq. 1) and KC (Eq. 2), from the base LLM's multiple responses and an external NLI model, then selects top-scoring instruction data and fine-tunes the model. The central claim that filtering high-conflict data improves domain-specific performance is an empirical hypothesis tested on external benchmarks (MedQA, MedMCQA, PubmedQA, MMLU-Medical, Med-HALT, FinanceIQ) that are not used to compute the selection scores. The self-referential aspect—scoring data with the same model that is later fine-tuned—does not make the result equivalent to its input by construction: a high KA/KC sample is defined as one the base model can already answer or answer consistently, not as one that will score well on the evaluation benchmarks. The paper's load-bearing assumption that instruction-tuning cannot add new knowledge is attributed to Ren et al. [16], which is external prior work and not a self-citation. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' own prior work, and no core ansatz is smuggled in via self-citation. The concern that KDS may simply select easy, already-known data and that the gains may not transfer to genuinely new knowledge is a plausible confound or limitation of the method, but it is not a circularity: the paper does not define its evaluation outcome in terms of its selection scores. Therefore the appropriate finding is no significant circularity.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

No new physical or conceptual entities are postulated; 'knowledge conflict' is a framing borrowed from prior work (Xu et al. [13]). The central claim rests mainly on five domain assumptions and on several hand-chosen thresholds and sampling settings.

free parameters (6)
  • m (number of sampled responses) = 10
    Set to 10 for response generation; chosen by hand, not justified by data.
  • temperature = 0.7
    Sampling temperature for multiple response generation; chosen by hand.
  • quality threshold tau = 3
    Selected by scanning tau in {0,1,2,3,4} and picking the value with best average performance on the evaluation set (Figure 5b, Section IV-C.3).
  • diversity threshold lambda = 0.9
    Selected by scanning lambda in {0.7,0.8,0.85,0.9,0.95} and picking the best performer (Figure 5c, Section IV-C.4).
  • data budget k = 5000
    Default training subset size; a study design decision rather than a fitted constant.
  • LoRA fine-tuning epochs = 3
    Number of epochs is set to 3 following common practice; not tuned in the paper.
assumptions (5)
  • domain assumption Instruction-tuning does not add new knowledge to LLMs; it only elicits pretrained knowledge.
    Invoked in Section I (paragraph 2) to justify filtering data whose answers the model cannot already produce; relies on Ren et al. [16]. If false, KDS discards learnable material.
  • domain assumption Reference answers in the training set are correct, so a model response that does not entail the reference indicates a knowledge conflict.
    Core to the KA metric (Eq. 1); the paper acknowledges answers may be low-quality but does not validate this assumption beyond qualitative examples.
  • domain assumption The NLI model's entailment judgments reliably reflect semantic agreement for medical content.
    Used in KA and KC scoring (Section III-B); verified only on 100 manually labeled pairs (Section V-D), which is a small sample.
  • domain assumption Sampling 10 responses at temperature 0.7 adequately probes the model's internal knowledge.
    Section III-B; no sensitivity analysis for m or temperature is provided.
  • domain assumption The base LLM's self-rated quality scores are meaningful for filtering low-quality data.
    Quality filter in Section III-B uses the model itself to rate answer quality on a 0-5 scale; Figure 3 shows distributions but no external validation of the scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Resolving Knowledge Conflicts in Domain-specific Data Selection: A Case Study on Medical Instruction-tuning." pith.science (2026). https://pith.science/paper/2NQAMUUG

@misc{pith2026250521958,
  author       = {Pith},
  title        = {Pith review of: Resolving Knowledge Conflicts in Domain-specific Data Selection: A Case Study on Medical Instruction-tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2NQAMUUG}},
  note         = {Machine review of arXiv:2505.21958}
}
read the original abstract

Domain-specific instruction-tuning has become the defacto standard for improving the performance of large language models (LLMs) in specialized applications, e.g., medical question answering. Since the instruction-tuning dataset might contain redundant or low-quality data, data selection (DS) is usually required to maximize the data efficiency. Despite the successes in the general domain, current DS methods often struggle to select the desired data for domain-specific instruction-tuning. One of the main reasons is that they neglect the impact of knowledge conflicts, i.e., the discrepancy between LLMs' pretrained knowledge and context knowledge of instruction data, which could damage LLMs' prior abilities and lead to hallucination. To this end, we propose a simple-yet-effective Knowledge-aware Data Selection (namely KDS) framework to select the domain-specific instruction-tuning data that meets LLMs' actual needs. The core of KDS is to leverage two knowledge-aware metrics for quantitatively measuring knowledge conflicts from two aspects: context-memory knowledge alignment and intra-memory knowledge consistency. By filtering the data with large knowledge conflicts and sampling the high-quality and diverse data, KDS can effectively stimulate the LLMs' abilities and achieve better domain-specific performance. Taking the medical domain as the testbed, we conduct extensive experiments and empirically prove that KDS surpasses the other baselines and brings significant and consistent performance gains among all LLMs. More encouragingly, KDS effectively improves the model generalization and alleviates the hallucination problem.

Figures

Figures reproduced from arXiv: 2505.21958 by the authors.

Figure 1
Figure 1. Performance comparisons (%) of different DS metrics. Notably, “IFD” means the instruction-following difficulty [9], “Complexitydeita” and “Qualitydeita” are from DEITA [10], and the metrics in red are ours. The y-axis denotes the average performance of tuned LLaMA models on several medical benchmarks, where the details are shown in Section IV. domain, they often struggle to handle the domain-specific tasks, e.g., me… view at source ↗
Figure 2
Figure 2. Overview of our KDS framework, which contains three processes: ❶ obtaining multiple responses of LLM for each question; ❷ scoring the data with the knowledge alignment and consistency metrics; ❸ filtering the low-quality and repetitive data, and sampling the final data. Notably, for ease of illustration, we only show a representative sample and simplified formulation in (b) and (c). n denotes the number of responses… view at source ↗
Figure 3
Figure 3. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Comparative winning rates (%) of KDS-KA+KC v.s. other baselines on the long-form medical QA benchmark [52]. LLaMA-3-8B-Instruct is used as the base model, and GPT-4o-mini is used as the automated evaluator. xsmall base large (a) NLI model sizes 46 47 48 49 50 Average P…
Figure 5
Figure 5. Figure 5: (a) Effect of NLI models with different model sizes, (b) Parameter analysis of quality threshold τ and (c) Parameter analysis of diversity threshold λ. Notably, we use the LLaMA-3-8B-Instruct as the base model and report the average performance of HoT and multiple-choi…
Figure 6
Figure 6. Figure 6: Results at various training data scales. We use the LLaMA-3-8B-Instruct as the base model. and 5) the efficiency of KDS pipeline. Lastly, we provide some case studies to qualitatively analyze the effectiveness of KDS. A. Does KDS still Work at other Data Scales? In the…
Figure 7
Figure 7. Figure 7: Comparative results of LLMs tuned with different methods on the MMedBench. MMedBench [57] is a multilingual medical multiple-choice QA benchmark across six primary languages: English, Chinese, Japanese, French, Russian, and Spanish. KDS method can effectively improve t…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 39 canonical work pages

  1. [16]

    Learning or self-aligning? rethinking instruction fine-tuning,

    M. Ren, B. Cao, H. Lin, C. Liu, X. Han, K. Zeng, G. Wan, X. Cai, and L. Sun, “Learning or self-aligning? rethinking instruction fine-tuning,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2024

    work page 2024

  2. [1]

    Gpt-4 technical report,

    OpenAI, “Gpt-4 technical report,” 2023

    2023

  3. [2]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al., “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    arXiv 2024

  4. [3]

    Qwen2. 5 technical report,

    A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei et al. , “Qwen2. 5 technical report,” arXiv preprint arXiv:2412.15115, 2024

    arXiv 2024

  5. [4]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan et al., “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, 2024

    arXiv 2024

  6. [5]

    A survey of large language models,

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y . Hou, Y . Min, B. Zhang, J. Zhang, Z. Dong et al., “A survey of large language models,” arXiv preprint arXiv:2303.18223, vol. 1, no. 2, 2023. 13

    arXiv 2023

  7. [6]

    Biomistral: A collection of open-source pretrained large language models for medical domains,

    Y . Labrak, A. Bazoge, E. Morin, P.-A. Gourraud, M. Rouvier, and R. Dufour, “Biomistral: A collection of open-source pretrained large language models for medical domains,” in Findings of the Association for Computational Linguistics: ACL 2024 , 2024

    work page 2024

  8. [7]

    Finetuned language models are zero-shot learners,

    J. Wei, M. Bosma, V . Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V . Le, “Finetuned language models are zero-shot learners,” in International Conference on Learning Representations , 2021

    work page 2021

Show all 61 references
  1. [8]

    Lima: Less is more for alignment,

    C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y . Mao, X. Ma, A. Efrat, P. Yu, L. Yu et al., “Lima: Less is more for alignment,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  2. [9]

    From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning,

    M. Li, Y . Zhang, Z. Li, J. Chen, L. Chen, N. Cheng, J. Wang, T. Zhou, and J. Xiao, “From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning,” in Proceedings of the 2024 Conference of the North American Chapter of the Associati...

    2024

  3. [10]

    What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning,

    W. Liu, W. Zeng, K. He, Y . Jiang, and J. He, “What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning,” in The Twelfth International Conference on Learning Representations, 2024

    2024

  4. [11]

    Alpagasus: Training a better alpaca with fewer data,

    L. Chen, S. Li, J. Yan, H. Wang, K. Gunaratna, V . Yadav, Z. Tang, V . Srinivasan, T. Zhou, H. Huang et al., “Alpagasus: Training a better alpaca with fewer data,” in The Twelfth International Conference on Learning Representations, 2024

    2024

  5. [12]

    Selfcheckgpt: Zero-resource black- box hallucination detection for generative large language models,

    P. Manakul, A. Liusie, and M. Gales, “Selfcheckgpt: Zero-resource black- box hallucination detection for generative large language models,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023

    2023

  6. [13]

    Knowledge conflicts for llms: A survey,

    R. Xu, Z. Qi, Z. Guo, C. Wang, H. Wang, Y . Zhang, and W. Xu, “Knowledge conflicts for llms: A survey,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024

    2024

  7. [14]

    Does fine-tuning llms on new knowledge encourage hallucinations?

    Z. Gekhman, G. Yona, R. Aharoni, M. Eyal, A. Feder, R. Reichart, and J. Herzig, “Does fine-tuning llms on new knowledge encourage hallucinations?” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , 2024

    2024

  8. [15]

    Conflictbank: A benchmark for evaluating the influence of knowledge conflicts in llm,

    Z. Su, J. Zhang, X. Qu, T. Zhu, Y . Li, J. Sun, J. Li, M. Zhang, and Y . Cheng, “Conflictbank: A benchmark for evaluating the influence of knowledge conflicts in llm,” arXiv preprint arXiv:2408.12076 , 2024

    2024 arXiv

  9. [17]

    A survey of knowledge enhanced pre-trained language models,

    L. Hu, Z. Liu, Z. Zhao, L. Hou, L. Nie, and J. Li, “A survey of knowledge enhanced pre-trained language models,” IEEE Transactions on Knowledge and Data Engineering , vol. 36, no. 4, pp. 1413–1430, 2023

    2023

  10. [18]

    Large language models on graphs: A comprehensive survey,

    B. Jin, G. Liu, C. Han, M. Jiang, H. Ji, and J. Han, “Large language models on graphs: A comprehensive survey,” IEEE Transactions on Knowledge and Data Engineering , 2024

    2024

  11. [19]

    Towards making the most of chatgpt for machine translation,

    K. Peng, L. Ding, Q. Zhong, L. Shen, X. Liu, M. Zhang, Y . Ouyang, and D. Tao, “Towards making the most of chatgpt for machine translation,” in Findings of the Association for Computational Linguistics: EMNLP 2023, 2023, pp. 5622–5633

    2023

  12. [20]

    Prompting large language model for machine translation: A case study,

    B. Zhang, B. Haddow, and A. Birch, “Prompting large language model for machine translation: A case study,” in International Conference on Machine Learning. PMLR, 2023, pp. 41 092–41 110

    2023

  13. [21]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models,

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . Li, Y . Wuet al., “Deepseekmath: Pushing the limits of mathematical reasoning in open language models,” arXiv preprint arXiv:2402.03300 , 2024

    2024 arXiv

  14. [22]

    Achieving> 97% on gsm8k: Deeply understanding the problems makes llms better solvers for math word problems,

    Q. Zhong, K. Wang, Z. Xu, J. Liu, L. Ding, and B. Du, “Achieving> 97% on gsm8k: Deeply understanding the problems makes llms better solvers for math word problems,” arXiv preprint arXiv:2404.14963 , 2024

    2024 arXiv

  15. [23]

    A survey on aspect-based sentiment analysis: Tasks, methods, and challenges,

    W. Zhang, X. Li, Y . Deng, L. Bing, and W. Lam, “A survey on aspect-based sentiment analysis: Tasks, methods, and challenges,” IEEE Transactions on Knowledge and Data Engineering , vol. 35, no. 11, pp. 11 019–11 038, 2022

    2022

  16. [24]

    Knowledge graph augmented network towards multiview representation learning for aspect-based sentiment analysis,

    Q. Zhong, L. Ding, J. Liu, B. Du, H. Jin, and D. Tao, “Knowledge graph augmented network towards multiview representation learning for aspect-based sentiment analysis,” IEEE Transactions on knowledge and data engineering, vol. 35, no. 10, pp. 10 098–10 111, 2023

    2023

  17. [25]

    Recommender systems in the era of large language models (llms),

    W. Fan, “Recommender systems in the era of large language models (llms),” IEEE Transactions on Knowledge and Data Engineering , pp. 1–20, 2024

    2024

  18. [26]

    Collm: Integrating collaborative embeddings into large language models for rec- ommendation,

    Y . Zhang, F. Feng, J. Zhang, K. Bao, Q. Wang, and X. He, “Collm: Integrating collaborative embeddings into large language models for rec- ommendation,” IEEE Transactions on Knowledge and Data Engineering , 2025

    2025

  19. [27]

    Meditron-70b: Scaling medical pretraining for large language models,

    Z. Chen, A. H. Cano, A. Romanou, A. Bonnet, K. Matoba, F. Salvi, M. Pagliardini, S. Fan, A. Köpf, A. Mohtashami et al., “Meditron-70b: Scaling medical pretraining for large language models,” arXiv preprint arXiv:2311.16079, 2023

    2023 arXiv

  20. [28]

    Zhongjing: Enhancing the chinese medical capabilities of large language model through expert feedback and real-world multi-turn dialogue,

    S. Yang, H. Zhao, S. Zhu, G. Zhou, H. Xu, Y . Jia, and H. Zan, “Zhongjing: Enhancing the chinese medical capabilities of large language model through expert feedback and real-world multi-turn dialogue,” in Proceedings of the AAAI conference on artificial intelligence , vol. 38...

    2024

  21. [29]

    Alpacare: Instruction-tuned large language models for medical application,

    X. Zhang, C. Tian, X. Yang, L. Chen, Z. Li, and L. R. Petzold, “Alpacare: Instruction-tuned large language models for medical application,” arXiv preprint arXiv:2310.14558, 2023

    2023 arXiv

  22. [30]

    Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge,

    Y . Li, Z. Li, K. Zhang, R. Dan, S. Jiang, and Y . Zhang, “Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge,” Cureus, 2023

    2023

  23. [31]

    Survey of hallucination in natural language generation,

    Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y . Xu, E. Ishii, Y . J. Bang, A. Madotto, and P. Fung, “Survey of hallucination in natural language generation,” ACM computing surveys , vol. 55, no. 12, pp. 1–38, 2023

    2023

  24. [32]

    Alpagasus: Training a better alpaca with fewer data,

    L. Chen, S. Li, J. Yan, H. Wang, K. Gunaratna, V . Yadav, Z. Tang, V . Srinivasan, T. Zhou, H. Huang et al., “Alpagasus: Training a better alpaca with fewer data,” arXiv preprint , 2023. [Online]. Available: https://arxiv.org/pdf/2307.08701.pdf

    2023 arXiv

  25. [33]

    Resolving knowledge conflicts in large language models,

    Y . Wang, S. Feng, H. Wang, W. Shi, V . Balachandran, T. He, and Y . Tsvetkov, “Resolving knowledge conflicts in large language models,” in First Conference on Language Modeling , 2024

    2024

  26. [34]

    Knowing what llms do not know: A simple yet effective self- detection method,

    Y . Zhao, L. Yan, W. Sun, G. Xing, C. Meng, S. Wang, Z. Cheng, Z. Ren, and D. Yin, “Knowing what llms do not know: A simple yet effective self- detection method,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisti...

    2024

  27. [35]

    Tug-of-war between knowledge: Exploring and resolving knowledge conflicts in retrieval-augmented language models,

    Z. Jin, P. Cao, Y . Chen, K. Liu, X. Jiang, J. Xu, L. Qiuxia, and J. Zhao, “Tug-of-war between knowledge: Exploring and resolving knowledge conflicts in retrieval-augmented language models,” in Proceedings of the 2024 Joint International Conference on Computational Linguistics...

    2024

  28. [36]

    Characterizing mechanisms for factual recall in language models,

    Q. Yu, J. Merullo, and E. Pavlick, “Characterizing mechanisms for factual recall in language models,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , 2023

    2023

  29. [37]

    3ds: Decomposed difficulty data selection’s case study on llm medical domain adaptation,

    H. Ding, Y . Fang, R. Zhu, X. Jiang, J. Zhang, Y . Xu, X. Chu, J. Zhao, and Y . Wang, “3ds: Decomposed difficulty data selection’s case study on llm medical domain adaptation,” arXiv preprint arXiv:2410.10901 , 2024

    2024

  30. [38]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” in Advances in neural information processing systems, 2020

    2020

  31. [39]

    Rethinking the role of demonstrations: What makes in-context learning work?

    S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer, “Rethinking the role of demonstrations: What makes in-context learning work?” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , 2022

    2022

  32. [40]

    Empirical insights on fine-tuning large language models for question-answering,

    J. Ye, Y . Yang, Q. Zhang, T. Gui, X. Huang, P. Wang, Z. Shi, and J. Fan, “Empirical insights on fine-tuning large language models for question-answering,” arXiv preprint arXiv:2409.15825 , 2024

    2024 arXiv

  33. [41]

    Self-consistency improves chain of thought reasoning in language models,

    X. Wang, J. Wei, D. Schuurmans, Q. V . Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” in The Eleventh International Conference on Learning Representations , 2023

    2023

  34. [42]

    Detecting hallucinations in large language models using semantic entropy,

    S. Farquhar, J. Kossen, L. Kuhn, and Y . Gal, “Detecting hallucinations in large language models using semantic entropy,” Nature, 2024

    2024

  35. [43]

    Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation,

    L. Kuhn, Y . Gal, and S. Farquhar, “Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation,” in The Eleventh International Conference on Learning Representations , 2023

    2023

  36. [44]

    A survey on llm-as-a-judge,

    J. Gu, X. Jiang, Z. Shi, H. Tan, X. Zhai, C. Xu, W. Li, Y . Shen, S. Ma, H. Liu et al. , “A survey on llm-as-a-judge,” arXiv preprint arXiv:2411.15594, 2024

    2024 arXiv

  37. [45]

    M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation,

    J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu, “M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation,” in Findings of the Association for Computational Linguistics: ACL 2024 , 2024

    2024

  38. [46]

    Stanford alpaca: An instruction-following llama model,

    R. Taori, I. Gulrajani, T. Zhang, Y . Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, “Stanford alpaca: An instruction-following llama model,” 2023

    2023

  39. [47]

    Medalpaca–an open-source collection of medical conversational ai models and training data,

    T. Han, L. C. Adams, J.-M. Papaioannou, P. Grundmann, T. Oberhauser, A. Löser, D. Truhn, and K. K. Bressem, “Medalpaca–an open-source collection of medical conversational ai models and training data,” arXiv preprint arXiv:2304.08247, 2023. 14

    2023 arXiv

  40. [48]

    Medmcqa: A large- scale multi-subject multi-choice dataset for medical domain question answering,

    A. Pal, L. K. Umapathi, and M. Sankarasubbu, “Medmcqa: A large- scale multi-subject multi-choice dataset for medical domain question answering,” in Conference on health, inference, and learning , 2022

    2022

  41. [49]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams,

    D. Jin, E. Pan, N. Oufattole, W.-H. Weng, H. Fang, and P. Szolovits, “What disease does this patient have? a large-scale open domain question answering dataset from medical exams,” Applied Sciences, 2021

    2021

  42. [50]

    Pubmedqa: A dataset for biomedical research question answering,

    Q. Jin, B. Dhingra, Z. Liu, W. Cohen, and X. Lu, “Pubmedqa: A dataset for biomedical research question answering,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processin...

    2019

  43. [51]

    Measuring massive multitask language understanding,

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” in International Conference on Learning Representations , 2020

    2020

  44. [52]

    A benchmark for long-form medical question an- swering,

    P. Hosseini, J. M. Sin, B. Ren, B. G. Thomas, E. Nouri, A. Farahanchi, and S. Hassanpour, “A benchmark for long-form medical question an- swering,” in Advancements In Medical Foundation Models: Explainability, Robustness, Security, and Beyond , 2024

    2024

  45. [53]

    Rouge: A package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” in Text summarization branches out , 2004

    2004

  46. [54]

    Large language models encode clinical knowledge,

    K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl et al. , “Large language models encode clinical knowledge,” Nature, 2023

    2023

  47. [55]

    Toward expert-level medical question answering with large language models,

    K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, M. Amin, L. Hou, K. Clark, S. R. Pfohl, H. Cole-Lewis et al., “Toward expert-level medical question answering with large language models,” Nature Medicine, 2025

    2025

  48. [56]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chenet al., “Lora: Low-rank adaptation of large language models,” in International Conference on Learning Representations , 2021

    2021

  49. [57]

    Towards building multilingual language model for medicine,

    P. Qiu, C. Wu, X. Zhang, W. Lin, H. Wang, Y . Zhang, Y . Wang, and W. Xie, “Towards building multilingual language model for medicine,” Nature Communications, 2024

    2024

  50. [58]

    Med-halt: Medical domain hallucination test for large language models,

    A. Pal, L. K. Umapathi, and M. Sankarasubbu, “Med-halt: Medical domain hallucination test for large language models,” in Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), 2023

    2023

  51. [59]

    Xuanyuan 2.0: A large chinese financial chat model with hundreds of billions parameters,

    X. Zhang and Q. Yang, “Xuanyuan 2.0: A large chinese financial chat model with hundreds of billions parameters,” in Proceedings of the 32nd ACM international conference on information and knowledge management, 2023, pp. 4435–4439

    2023

  52. [60]

    Debertav3: Improving deberta using electra- style pre-training with gradient-disentangled embedding sharing,

    P. He, J. Gao, and W. Chen, “Debertav3: Improving deberta using electra- style pre-training with gradient-disentangled embedding sharing,” in The Eleventh International Conference on Learning Representations , 2021

    2021

  53. [61]

    A broad-coverage challenge corpus for sentence understanding through inference,

    A. Williams, N. Nangia, and S. Bowman, “A broad-coverage challenge corpus for sentence understanding through inference,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 ...

    2018

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.