Pith. sign in

REVIEW 4 major objections 6 minor 59 references

Refine Knowledge of Large Language Models via Adaptive Contrastive Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A training strategy that sorts an LLM's knowledge into known, uncertain, and unknown reduces hallucination by teaching each region differently.

desk verdict Adaptive quadrant-aware contrastive SFT is a plausible and well-ablated method, but the reported Truthful Rate gains are not trustworthy until the known/unknown labels used for evaluation are independent of the model's own sampling. read the letter →

arxiv 2502.07184 v1 pith:5RAPN2L4 submitted 2025-02-11 cs.CL cs.AI

Yinghui Li , Haojing Huang , Jiayi Kuang , Yangning Li , Shu-Yu Guo , Chao Qu , Xiaoyu Tan , Hai-Tao Zheng
show 2 more authors
Ying Shen Philip S. Yu
This is my paper · ORCID
classification cs.CLcs.AI
keywords hallucinationmitigationknowledgeboundariescontrastivelearninginstructiontuningtruthfulratelargelanguagemodelsIdon'tknowrefusalquadrants
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models hallucinate when they answer questions beyond their knowledge. This paper argues that the remedy is to help the model act on an accurate picture of what it knows: consolidate correct knowledge, strengthen half-learned knowledge, forget wrong knowledge, and answer "I don't know" when knowledge is absent. To do this it proposes Adaptive Contrastive Learning, which first sorts questions into known, uncertain, and unknown using the model's own repeated-sampling accuracy against two thresholds, $IK$ and $IDK$, and then builds different positive and negative training pairs for each sort. The reported effect is a Truthful Rate—correct answers plus correct refusals—of up to 78.2% versus 73.2% for supervised fine-tuning on TriviaQA with LLaMA-2-7B-chat, and up to 6.9 points over IDK-SFT on the out-of-distribution Natural Questions set.

What carries the argument

The central object is a two-threshold knowledge partition ($IK$, $IDK$) computed from the model's repeated-sampling accuracy, which splits questions into three regions: accuracy $\ge IK$ means the model knows it knows, accuracy between $IDK$ and $IK$ means the model is uncertain, and accuracy $\le IDK$ means the model does not know. On each region the paper builds contrastive instruction triplets (anchor, positive, negative) with quadrant-specific semantics, then optimizes an adaptive loss $L^{Adap} = L^{gen} + \max(\lambda, \text{detach}(L^{gen}/L^{ctr})) L^{ctr}$, where $L^{gen}$ is standard cross-entropy generation loss and $L^{ctr}$ is a cosine-similarity contrastive loss that pulls positive hidden states together and pushes negative hidden states apart.

What would settle it

Build a test set of questions where the base model repeatedly answers correctly with a confident but factually wrong answer (accuracy comfortably above $IK$), then train with IDK-SFT-Adpt-Ctr and check whether it still emits those wrong answers on a gold-labeled test. If Truthful Rate stays high while factual accuracy on those questions does not improve, the method is learning to lock in its own confidence pattern rather than refining knowledge.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a single fine-tuning strategy can simultaneously improve both sides of honesty—giving correct answers when the model knows and refusing when it does not—by adapting contrastive learning to the model's current mastery. The authors construct knowledge quadrants from repeated-sampling accuracy with thresholds $IK$ and $IDK$, and show that treating each quadrant with a purpose-built contrastive loss raises Truthful Rate consistently across two base models and out-of-distribution datasets. For known questions, the correct answer is pulled toward the anchor while "I don't know" and wrong answers are pushed away; for uncertain questions, the golden answer is pulled in and wrong answers are pushed away; for unknown questions, "I don't know" is pulled in and wrong answers are pushed away. They also report that the full multi-quadrant loss beats any single-quadrant or pair combination, and that the method remains effective when combined with retrieval-augmented generation and when scaled to a 13B model.

Load-bearing premise

The load-bearing premise is that a question's knowledge status—known, uncertain, or unknown—is faithfully measured by how often the model itself answers it correctly across repeated samples; if that self-accuracy proxy is wrong, the positive and negative training pairs encode the model's own confidence rather than true knowledge.

Editorial extensions

If this is right

  • On TriviaQA with LLaMA-2-7B-chat, the method lifts Truthful Rate from 73.2% with IDK-SFT to 78.2%, with the IK-IK rate (correct answers on known questions) rising 9.3 points even as correct refusals remain close to the SFT level.
  • On out-of-distribution Natural Questions, Truthful Rate rises 6.1 points over IDK-Prompting and 6.9 points over IDK-SFT for LLaMA, and 15.9 and 6.8 points for Mistral, indicating the learned answer-or-refuse boundary transfers beyond the training data.
  • Loss ablations show that consolidating uncertain knowledge contributes the most to correct answering, the known-knowledge loss makes the model more conservative, and the unknown-knowledge loss encourages forgetting wrong answers; only the full combination reaches the maximum Truthful Rate.
  • Combining with retrieval-augmented generation does not break the method: on RAG-Bench, retrieved context raises the method's Truthful Rate from 66.9% to 71.2%, while shifting some refusals into answers as expected.
  • Scaling from 7B to 13B parameters improves all metrics slightly for every method, and the proposed method keeps its advantage, suggesting the strategy composes with model scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the model's repeated-sampling accuracy is an imperfect mirror of true knowledge—for example, if a model confidently repeats a plausible falsehood—then the training targets inherit that error, so part of the reported gain may be the model learning to reproduce its own confidence pattern rather than correcting facts; a direct test would measure factual precision on a set of consistently-wrong-but
  • Because Truthful Rate counts correct answers and correct refusals as interchangeable, a model that over-refuses can score well on unknown-heavy test sets; the choice of $IK$ and $IDK$ thresholds therefore does real work in the metric, not just in the training loss.
  • The adaptive loss weight $\max(\lambda, \text{detach}(L^{gen}/L^{ctr}))$ implicitly up-weights the contrastive term when it is small relative to the generation loss, acting as a stabilizer against the contrastive signal vanishing—a ratio design that could transfer to other contrastive fine-tuning objectives.
  • Applying the same quadrant construction with an external knowledge oracle (such as retrieval or a trusted fact base) instead of self-sampling would separate "knowing" from "thinking it knows," which is a natural next experiment to isolate where the gains actually come from.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Adaptive Contrastive Learning (IDK-SFT-Adpt-Ctr) to improve the honesty of LLMs by refining their knowledge boundaries. The method defines upper and lower accuracy thresholds (IK and IDK) from repeated sampling of the base model, classifies questions into known, uncertain, and unknown regions, and constructs quadrant-specific positive and negative instances for contrastive instruction tuning. An adaptive loss, Eq. (3), combines generation loss with a contrastive loss whose weight is max(lambda, detach(L_gen/L_ctr)). Experiments on TriviaQA and Natural Questions with LLaMA-2-7B-chat and Mistral-7B-Instruct-v0.1 report Truthful Rate improvements over IDK-Prompting and IDK-SFT, with additional ablations on loss components, IDK threshold values, model size, ALCUNA, and RAG integration.

Significance. If the reported gains are real, the method would be a practical contribution to selective answering and hallucination mitigation, and the quadrant-specific contrastive data construction is a plausible mechanism. The paper has useful strengths: it evaluates on two base models, includes a loss-combination ablation (Table 3), a threshold analysis (Table 2), and an independent ALCUNA experiment (Table 4). However, the evaluation is built on labels derived from the same base-model repeated-sampling procedure used to construct training data, and the reported Truthful Rate may largely measure how well the model reproduces the base model's own confidence pattern. The absence of error bars, the absence of strong baselines, and the very small independent gain on ALCUNA mean the central claim is not yet established.

major comments (4)
  1. [Section 3.1 and Appendix B.1] The test-set known/unknown labels are never specified. Section 3.1 describes generating labels only for training via repeated sampling of the base LLM and thresholding by IK/IDK, while Appendix B.1 defines IK-IK and IK-IDK rates relative to questions "labeled as model knows" or "model-don't-know" in the test set. If the same self-derived labels are used for the 11,313 TriviaQA and 3,610 Natural Questions test instances, then a model that learns to answer exactly where the base model sampled correctly and to refuse exactly where it sampled incorrectly will attain a high Truthful Rate regardless of whether those labels reflect true knowledge. The authors must state how test labels were obtained, and should re-evaluate on externally labeled data (human annotation, ALCUNA-style artificial unknowns, or labels from a different model) to support the hallucination-reduction claim.
  2. [Table 2 and Section 4.4] The IDK threshold is selected on the test set without a validation protocol, and no error bars or significance tests are reported. Table 2 shows IDK=0.7 maximizing Truthful Rate on the test set, which makes the headline numbers optimistic. In Table 1, the Mistral TriviaQA gain of IDK-SFT-Adpt-Ctr over IDK-SFT is 1.3 points (73.5 vs. 72.2); with 11,313 test questions, this difference may be within sampling noise. Multi-seed runs or bootstrap confidence intervals are needed before the 5.0-6.9 point claims can be taken at face value.
  3. [Eq. (3) and Section 5.2] The contribution of the "adaptive" weighting mechanism itself is not ablated. Table 3 ablates the quadrant-specific loss components, but every configuration uses the same max(lambda, detach(L_gen/L_ctr)) weighting; a fixed-weight contrastive loss, or a variant with lambda only, is needed to determine whether the adaptive term causes the improvement or whether the improvement comes from the combination of quadrant-specific contrastive data alone. Without this control, the central causal claim that the adaptive loss drives the gains is unsupported.
  4. [Section 4.4 and Appendix C.1] The comparison set is too narrow. The baselines are only IDK-Prompting and IDK-SFT, both constructed by the authors; established methods for "I don't know" calibration, such as R-tuning (cited as Zhang et al. 2024c), and standard factuality benchmarks (e.g., TruthfulQA) are absent. The only independent evaluation, ALCUNA in Table 4, shows a 0.5 point Truthful Rate gain over IDK-SFT (98.3 vs. 97.8) on questions that are unknown by construction, and no significance test is given. This is a weak basis for the paper's strong conclusions about reducing hallucinations.
minor comments (6)
  1. [Eq. (3) and Eq. (6)] The text says lambda is the "upper bound of the weight," but max(lambda, detach(L_gen/L_ctr)) actually makes lambda a lower bound on the contrastive weight; the terminology should be corrected.
  2. [Section 4.1.2] The sentence "We select LLaMA-2-7B-chat and Mistral-7B-Instruct-v0.1 as base models for testing on TriviaQA and Natural Questions, respectively" is ambiguous, since Table 1 reports both models on both datasets; clarify which model is evaluated on which dataset.
  3. [Section 5.3 and Figure 4] Figure 4 contains four panels whose labels are inconsistent ("Unknown Question" vs. "UnknownQuestions", "Known Question" vs. "KnownQuestions"); clarify how the repeated-sampling accuracy categories are computed and why there are two versions of each plot.
  4. [Table 2] The column header "Knowing Rate" is unclear; it should be "IDK Rate" or "I Don't Know Rate" to match the description in the text.
  5. [Appendix B.1] The definition of IK-IK rate contains a typo: "We calculate the number of correct answers given by the model as a percentage of the total number of questions in the dataset as a fraction of the IK-IDK rate" should read "as the IK-IK rate."
  6. [General] No code or data release link is provided, which hinders reproducibility of the data construction and training pipeline; please include the implementation details or a public repository.

Circularity Check

1 steps flagged · score 6.0 of 10

Truthful Rate improvements are measured against the same base-model repeated-sampling labels used to construct training data, making the central claim partly definitional.

  1. self definitional [Section 3.1 (knowledge boundaries); Section 5.3 (repeated sampling distribution); Appendix B.1 (evaluation metrics)]
    "Following previous studies Cheng et al. (2024); Zhang et al. (2024c), we filter questions from public datasets and query LLMs multiple times to sample responses, calculating accuracy as a measure of the model's confidence in each question. ... Confidence below IDK indicates a lack of knowledge. ... we conduct ten repeated samplings of questions from the TriviaQA test set on LLaMA-2-7B-chat, categorizing them into 'Unknown Questions' and 'Known Questions.' ... IK-IDK rate: ... model refuses to answer a question that is labeled as a 'model-don't-know question' in the test set."

    The paper defines 'known' vs 'unknown' exclusively by the base LLM's repeated-sampling accuracy relative to IK/IDK thresholds (Sec. 3.1). The test-set labels used in Truthful Rate are also generated by repeated sampling of the same base model (Sec. 5.3 shows this for TriviaQA; Appendix B.1 refers to questions 'labeled as a model-don't-know question in the test set'). Training data for IDK-SFT and the adaptive loss are built from the same kind of self-generated labels. Thus the truthfulness metric measures how well the trained model reproduces the base model's own confidence pattern: a model that answers exactly where the base sampled correctly and refuses exactly where the base sampled incorrectly attains high Truthful Rate, even if both behaviors are wrong about external facts.

full rationale

Score 6: the central claim is substantially circular because the evaluation labels are the model's own sampling pattern, which is also what the training optimizes. No significant self-citation circularity: cited prior work (Cheng et al., Zhang et al.) is external and the contrastive loss follows Yan et al. The ALCUNA experiment (Appendix C.1) provides independent external evidence but shows only a 0.5 pt gain over IDK-SFT on artificially unknown questions. Threshold selection in Table 2 appears to be performed on the test set, adding optimism but not changing the circularity diagnosis. If one accepts repeated-sampling accuracy as a definition of 'knowledge', the method is internally consistent; the circularity is that the paper presents this self-consistency as evidence of hallucination reduction.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central lever is the self-referential knowledge labeling: the model's own sampled responses define the training and evaluation categories, and the thresholds and weighting are chosen empirically. No new physical or conceptual entities are introduced beyond the knowledge quadrants, which are borrowed from prior work.

free parameters (6)
  • IK (upper threshold) = 1.0 (default)
    Divides known from uncertain knowledge; affects which negative instances are used. No independent justification is given in the paper.
  • IDK (lower threshold) = 0.7 (validation-optimal)
    Divides uncertain from unknown knowledge. Tuned on validation, Table 2 shows best at 0.7; no theory for the value.
  • tau (temperature in contrastive loss) = 0.01
    Set in implementation details (Section 4.1.2); standard contrastive temperature.
  • lambda (upper bound of contrastive weight) = 1
    Set in implementation details (Section 4.1.2); caps the adaptive loss weight.
  • Number of sampled responses per question = 10 (used in experiments, inferred from Section 5.3)
    Used to compute response accuracy for knowledge labeling; not explicitly given as a hyperparameter choice.
  • Lexical matching correctness threshold = ~90%
    Used to decide whether a sampled response counts as correct (Section 3.1); approximate and task-specific.
assumptions (4)
  • domain assumption Repeated sampling accuracy is a valid proxy for knowledge mastery.
    Section 3.1 uses accuracy over multiple sampled responses to place questions into knowledge quadrants. If sampling is noisy or biased, the whole labeling scheme is unstable.
  • domain assumption The knowledge quadrant framework (Yin et al., 2023b) meaningfully describes LLM knowledge states.
    Section 1 adopts the four-quadrant model of Known Knowns, Known Unknowns, Unknown Knowns, and Unknown Unknowns as the basis for the method.
  • domain assumption Lexical matching with about 90% overlap is a reliable automatic correctness metric.
    Section 3.1 uses lexical matching to determine whether model responses are correct; this can misclassify paraphrases or partial answers.
  • ad hoc to paper The adaptive loss weighting max(lambda, detach(L_gen/L_ctr)) improves training.
    Equation 3 introduces this weighting with no derivation or theoretical justification; it is an empirical heuristic.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Refine Knowledge of Large Language Models via Adaptive Contrastive Learning." pith.science (2026). https://pith.science/paper/5RAPN2L4

@misc{pith2026250207184,
  author       = {Pith},
  title        = {Pith review of: Refine Knowledge of Large Language Models via Adaptive Contrastive Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5RAPN2L4}},
  note         = {Machine review of arXiv:2502.07184}
}
read the original abstract

How to alleviate the hallucinations of Large Language Models (LLMs) has always been the fundamental goal pursued by the LLMs research community. Looking through numerous hallucination-related studies, a mainstream category of methods is to reduce hallucinations by optimizing the knowledge representation of LLMs to change their output. Considering that the core focus of these works is the knowledge acquired by models, and knowledge has long been a central theme in human societal progress, we believe that the process of models refining knowledge can greatly benefit from the way humans learn. In our work, by imitating the human learning process, we design an Adaptive Contrastive Learning strategy. Our method flexibly constructs different positive and negative samples for contrastive learning based on LLMs' actual mastery of knowledge. This strategy helps LLMs consolidate the correct knowledge they already possess, deepen their understanding of the correct knowledge they have encountered but not fully grasped, forget the incorrect knowledge they previously learned, and honestly acknowledge the knowledge they lack. Extensive experiments and detailed analyses on widely used datasets demonstrate the effectiveness of our method.

Figures

Figures reproduced from arXiv: 2502.07184 by the authors.

Figure 1
Figure 1. The illustration of our adaptive contrastive learning. Based on the multiple sampling [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The illustration of the knowledge boundaries and the sample response. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The examples of the contrastive learning data. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Accuracy Distribution for Known and Unknown Questions in TriviaQA. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 40 canonical work pages

  1. [1]

    Actions, not apps: Toward using llms to reshape context aware interactions in mixed reality systems

    Amir - Reza Asadi, Joel Appiah, Siddique Abubakr Muntaka, and Jess Kropczynski. Actions, not apps: Toward using llms to reshape context aware interactions in mixed reality systems. In Constantine Stephanidis, Margherita Antona, Stavroula Ntoa, and Gavriel Salvendy (eds.), HCI International 2024 Posters - 26th International Conference on Human-Computer Int...

    work page 2024

  2. [2]

    Self-rag: Learning to retrieve, generate, and critique through self-reflection

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024

    work page 2024

  3. [3]

    LLM augmented llms: Expanding capabilities through composition

    Rachit Bansal, Bidisha Samanta, Siddharth Dalmia, Nitish Gupta, Sriram Ganapathy, Abhishek Bapna, Prateek Jain, and Partha Talukdar. LLM augmented llms: Expanding capabilities through composition. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024

    work page 2024

  4. [4]

    Semantic parsing on freebase from question-answer pairs

    Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL , pp.\ 1533--...

    work page 2013

  5. [5]

    Discovering latent knowledge in language models without supervision

    Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023

    work page 2023

  6. [6]

    INSIDE: llms' internal states retain the power of hallucination detection

    Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. INSIDE: llms' internal states retain the power of hallucination detection. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024

    work page 2024

  7. [7]

    Can AI assistants know what they don't know? In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024

    Qinyuan Cheng, Tianxiang Sun, Xiangyang Liu, Wenwei Zhang, Zhangyue Yin, Shimin Li, Linyang Li, Zhengfu He, Kai Chen, and Xipeng Qiu. Can AI assistants know what they don't know? In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024

    work page 2024

  8. [8]

    Glass, and Pengcheng He

    Yung - Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024

    work page 2024

Show all 59 references
  1. [9]

    Zhao, Yanping Huang, Andrew M

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent...

    arXiv

  2. [10]

    Yu, and Wenpeng Yin

    Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Peng Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, Haoran Zhang, Vipul Gupta, Yinghui Li, Tao Li, Fei Wang, Qin Liu, Tianlin Liu, Pengzhi Gao, Congying Xia, Chen Xing, Cheng Jiayang, ...

    2024

  3. [11]

    Enhancing noise robustness of retrieval-augmented language models with adaptive adversarial training

    Feiteng Fang, Yuelin Bai, Shiwen Ni, Min Yang, Xiaojun Chen, and Ruifeng Xu. Enhancing noise robustness of retrieval-augmented language models with adaptive adversarial training. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of...

    2024 doi

  4. [12]

    Does fine-tuning llms on new knowledge encourage hallucinations? arXiv preprint arXiv:2405.05904, 2024

    Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning llms on new knowledge encourage hallucinations? arXiv preprint arXiv:2405.05904, 2024

    2024 arXiv

  5. [13]

    Ctooleval: A chinese benchmark for llm-powered agent evaluation in real-world API interactions

    Zishan Guo, Yufei Huang, and Deyi Xiong. Ctooleval: A chinese benchmark for llm-powered agent evaluation in real-world API interactions. In Lun - Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Tha...

    2024

  6. [14]

    Lateval: An interactive llms evaluation benchmark with incomplete information from lateral thinking puzzles

    Shulin Huang, Shirong Ma, Yinghui Li, Mengzuo Huang, Wuhe Zou, Weidong Zhang, and Hai - Tao Zheng. Lateval: An interactive llms evaluation benchmark with incomplete information from lateral thinking puzzles. CoRR, abs/2308.10855, 2023. doi:10.48550/ARXIV.2308.10855. URL https:...

    2023 doi arXiv

  7. [15]

    Weld, and Luke Zettlemoyer

    Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min - Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational L...

    2017

  8. [16]

    Natural language understanding and inference with MLLM in visual question answering: A survey

    Jiayi Kuang, Jingyou Xie, Haohao Luo, Ronghao Li, Zhe Xu, Xianfeng Cheng, Yinghui Li, Xika Lin, and Ying Shen. Natural language understanding and inference with MLLM in visual question answering: A survey. CoRR, abs/2411.17558, 2024. doi:10.48550/ARXIV.2411.17558. URL https://...

    2024 doi arXiv

  9. [17]

    Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming - Wei Chang, Andrew M

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming - Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and...

    2019

  10. [18]

    The dawn after the dark: An empirical study on factuality hallucination in large language models

    Junyi Li, Jie Chen, Ruiyang Ren, Xiaoxue Cheng, Xin Zhao, Jian - Yun Nie, and Ji - Rong Wen. The dawn after the dark: An empirical study on factuality hallucination in large language models. In Lun - Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Ann...

    2024

  11. [19]

    Yu, Fei Huang, and Jingren Zhou

    Yangning Li, Yinghui Li, Xinyu Wang, Yong Jiang, Zhen Zhang, Xinran Zheng, Hui Wang, Hai - Tao Zheng, Pengjun Xie, Philip S. Yu, Fei Huang, and Jingren Zhou. Benchmarking multimodal retrieval augmented generation with dynamic VQA dataset and self-adaptive planning agent. CoRR,...

    2024 doi arXiv

  12. [20]

    MESED: A multi-modal entity set expansion dataset with fine-grained semantic classes and hard negative entities

    Yangning Li, Tingwei Lu, Hai - Tao Zheng, Yinghui Li, Shulin Huang, Tianyu Yu, Jun Yuan, and Rui Zhang. MESED: A multi-modal entity set expansion dataset with fine-grained semantic classes and hard negative entities. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natara...

    2024

  13. [21]

    The past mistake is the future wisdom: Error-driven contrastive probability optimization for chinese spell checking

    Yinghui Li, Qingyu Zhou, Yangning Li, Zhongli Li, Ruiyang Liu, Rongyi Sun, Zizhen Wang, Chao Li, Yunbo Cao, and Hai - Tao Zheng. The past mistake is the future wisdom: Error-driven contrastive probability optimization for chinese spell checking. In Smaranda Muresan, Preslav Na...

    2022 doi

  14. [22]

    On the (in)effectiveness of large language models for chinese text correction

    Yinghui Li, Haojing Huang, Shirong Ma, Yong Jiang, Yangning Li, Feng Zhou, Hai - Tao Zheng, and Qingyu Zhou. On the (in)effectiveness of large language models for chinese text correction. CoRR, abs/2307.09007, 2023 a . doi:10.48550/ARXIV.2307.09007. URL https://doi.org/10.4855...

    2023 doi arXiv

  15. [23]

    Bidirectional end-to-end learning of retriever-reader paradigm for entity linking

    Yinghui Li, Yong Jiang, Shen Huang, Xingyu Lu, Yangning Li, Pengjun Xie, Fei Huang, and Hai - Tao Zheng. Bidirectional end-to-end learning of retriever-reader paradigm for entity linking. CoRR, abs/2306.12245, 2023 b . doi:10.48550/ARXIV.2306.12245. URL https://doi.org/10.4855...

    2023 doi arXiv

  16. [24]

    Yinghui Li, Shang Qin, Jingheng Ye, Shirong Ma, Yangning Li, Libo Qin, Xuming Hu, Wenhao Jiang, Hai - Tao Zheng, and Philip S. Yu. Rethinking the roles of large language models in chinese grammatical error correction. CoRR, abs/2402.11420, 2024 d . doi:10.48550/ARXIV.2402.1142...

    2024 doi arXiv

  17. [25]

    Towards real-world writing assistance: A chinese character checking benchmark with faked and misspelled characters

    Yinghui Li, Zishan Xu, Shaoshen Chen, Haojing Huang, Yangning Li, Shirong Ma, Yong Jiang, Zhongli Li, Qingyu Zhou, Hai - Tao Zheng, and Ying Shen. Towards real-world writing assistance: A chinese character checking benchmark with faked and misspelled characters. In Lun - Wei K...

    2024

  18. [26]

    When llms meet cunning texts: A fallacy understanding benchmark for large language models

    Yinghui Li, Qingyu Zhou, Yuanzhen Luo, Shirong Ma, Yangning Li, Hai-Tao Zheng, Xuming Hu, and S Yu Philip. When llms meet cunning texts: A fallacy understanding benchmark for large language models. In The Thirty-eight Conference on Neural Information Processing Systems Dataset...

    2024

  19. [27]

    Correct like humans: Progressive learning framework for chinese text error correction

    Yinghui Li, Shirong Ma, Shaoshen Chen, Haojing Huang, Shulin Huang, Yangning Li, Hai - Tao Zheng, and Ying Shen. Correct like humans: Progressive learning framework for chinese text error correction. Expert Syst. Appl., 265: 0 126039, 2025. doi:10.1016/J.ESWA.2024.126039. URL ...

    2025

  20. [28]

    Mitigating hallucination in large multi-modal models via robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 ....

    2024

  21. [29]

    Are we ready for a new paradigm shift? A survey on visual deep MLP

    Ruiyang Liu, Yinghui Li, Linmi Tao, Dun Liang, and Hai - Tao Zheng. Are we ready for a new paradigm shift? A survey on visual deep MLP . Patterns, 3 0 (7): 0 100520, 2022 a . doi:10.1016/J.PATTER.2022.100520. URL https://doi.org/10.1016/j.patter.2022.100520

    2022

  22. [30]

    What makes good data for alignment? A comprehensive study of automatic data selection in instruction tuning

    Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? A comprehensive study of automatic data selection in instruction tuning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, ...

    2024

  23. [31]

    Radev, and Graham Neubig

    Yixin Liu, Pengfei Liu, Dragomir R. Radev, and Graham Neubig. BRIO: bringing order to abstractive summarization. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume ...

    2022

  24. [32]

    Le, Barret Zoph, Jason Wei, and Adam Roberts

    Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara ...

    2023

  25. [33]

    Coarse-to-fine highlighting: Reducing knowledge hallucination in large language models

    Qitan Lv, Jie Wang, Hanzhu Chen, Bin Li, Yongdong Zhang, and Feng Wu. Coarse-to-fine highlighting: Reducing knowledge hallucination in large language models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview....

    2024

  26. [34]

    Explicitly stating assumptions reduces hallucinations in natural language inference

    Wenchuan Mu and Kwan Hui Lim. Explicitly stating assumptions reduces hallucinations in natural language inference. In The Second Tiny Papers Track at ICLR 2024, Tiny Papers @ ICLR 2024, Vienna, Austria, May 11, 2024 . OpenReview.net, 2024

    2024

  27. [35]

    Octopack: Instruction tuning code large language models

    Niklas Muennighoff, Qian Liu, Armel Randy Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. In The Twelfth International Conference on Learning Represen...

    2024

  28. [36]

    Niels M \" u ndler, Jingxuan He, Slobodan Jenko, and Martin T. Vechev. Self-contradictory hallucinations of large language models: Evaluation, detection and mitigation. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2...

    2024

  29. [37]

    Ragtruth: A hallucination corpus for developing trustworthy retrieval-augmented language models

    Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, Kashun Shum, Randy Zhong, Juntong Song, and Tong Zhang. Ragtruth: A hallucination corpus for developing trustworthy retrieval-augmented language models. In Lun - Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 6...

    2024

  30. [38]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leik...

    2022

  31. [39]

    Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, D...

    2022

  32. [40]

    Arik, and Tomas Pfister

    Pritam Sarkar, Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan \" O . Arik, and Tomas Pfister. Mitigating object hallucination via data augmented contrastive tuning. CoRR, abs/2405.18654, 2024

    2024 arXiv

  33. [41]

    Zhao, Hongkun Yu, Kurt Keutzer, Trevor Darrell, and Denny Zhou

    Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, Tu Vu, Yuexin Wu, Wuyang Chen, Albert Webson, Yunxuan Li, Vincent Y. Zhao, Hongkun Yu, Kurt Keutzer, Trevor Darrell, and Denny Zhou. Flan-moe: Scaling i...

    2023 arXiv

  34. [42]

    DAMO-NLP at semeval-2023 task 2: A unified retrieval-augmented system for multilingual named entity recognition

    Zeqi Tan, Shen Huang, Zixia Jia, Jiong Cai, Yinghui Li, Weiming Lu, Yueting Zhuang, Kewei Tu, Pengjun Xie, Fei Huang, and Yong Jiang. DAMO-NLP at semeval-2023 task 2: A unified retrieval-augmented system for multilingual named entity recognition. CoRR, abs/2305.03688, 2023. do...

    2023 doi arXiv

  35. [43]

    MINT: evaluating llms in multi-turn interaction with tools and language feedback

    Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. MINT: evaluating llms in multi-turn interaction with tools and language feedback. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2...

    2024

  36. [44]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

    Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29,...

    2022

  37. [45]

    Let llms take on the latest challenges! A chinese dynamic question answering benchmark

    Zhikun Xu, Yinghui Li, Ruixue Ding, Xinyu Wang, Boli Chen, Yong Jiang, Haitao Zheng, Wenlian Lu, Pengjun Xie, and Fei Huang. Let llms take on the latest challenges! A chinese dynamic question answering benchmark. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al - Khali...

    2025

  38. [46]

    Huang, Wenxuan Zhou, Fan Yin, Aram Galstyan, Wenpeng Yin, and Muhao Chen

    Tianyi Yan, Fei Wang, James Y. Huang, Wenxuan Zhou, Fan Yin, Aram Galstyan, Wenpeng Yin, and Muhao Chen. Contrastive instruction tuning. In Lun - Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Tha...

    2024

  39. [47]

    ALCUNA : Large language models meet new knowledge

    Xunjian Yin, Baizhou Huang, and Xiaojun Wan. ALCUNA : Large language models meet new knowledge. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 1397--1414, Singapore, December 202...

    2023 doi

  40. [48]

    Do large language models know what they don't know? In Anna Rogers, Jordan L

    Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. Do large language models know what they don't know? In Anna Rogers, Jordan L. Boyd - Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, Toronto...

    2023

  41. [49]

    Seqgpt: An out-of-the-box large language model for open domain sequence understanding

    Tianyu Yu, Chengyue Jiang, Chao Lou, Shen Huang, Xiaobin Wang, Wei Liu, Jiong Cai, Yangning Li, Yinghui Li, Kewei Tu, Hai - Tao Zheng, Ningyu Zhang, Pengjun Xie, Fei Huang, and Yong Jiang. Seqgpt: An out-of-the-box large language model for open domain sequence understanding. I...

    2024

  42. [50]

    When scaling meets LLM finetuning: The effect of data, model and finetuning method

    Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets LLM finetuning: The effect of data, model and finetuning method. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024 a

    2024

  43. [51]

    Enhancing hallucination detection through perturbation-based synthetic data generation in system responses

    Dongxu Zhang, Varun Gangal, Barrett Martin Lattimer, and Yi Yang. Enhancing hallucination detection through perturbation-based synthetic data generation in system responses. In Lun - Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computationa...

    2024

  44. [52]

    Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang

    Hanning Zhang, Shizhe Diao, Yong Lin, Yi R. Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. R-tuning: Instructing large language models to say 'i don't know'. In Kevin Duh, Helena G \' o mez - Adorno, and Steven Bethard (eds.), Proceedings of the 2024 Conf...

    2024

  45. [53]

    Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A. Smith. How language model hallucinations can snowball. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024 d

    2024

  46. [54]

    Self-alignment for factuality: Mitigating hallucinations in llms via self-evaluation

    Xiaoying Zhang, Baolin Peng, Ye Tian, Jingyan Zhou, Lifeng Jin, Linfeng Song, Haitao Mi, and Helen Meng. Self-alignment for factuality: Mitigating hallucinations in llms via self-evaluation. In Lun - Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Ann...

    2024

  47. [55]

    A survey on model compression for large language models

    Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey on model compression for large language models. CoRR, abs/2308.07633, 2023

    2023 arXiv

  48. [56]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  49. [57]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  50. [58]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  51. [59]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.