pith. sign in

arxiv: 2606.07521 · v1 · pith:NF7QUN4Rnew · submitted 2026-04-19 · 💻 cs.CL · cs.AI

Evaluating Hallucinations in Domain-Adapted Large Language Models

Sanchita Porwal , Sai Prasath S , Xingjian Bi , Madelyn Scandlen This is my paper

Pith reviewed 2026-07-05 16:50 UTC · model glm-5.2

classification 💻 cs.CL cs.AI
keywords llmsdomain-specifichallucinationsinformationmodelmodelsadaptingdata
0
0 comments X

The pith

Fine-tuning teaches LLMs to parrot, not reason, over new facts

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether fine-tuning alone is enough to inject genuinely new knowledge into a large language model, or whether it merely teaches the model to mimic surface patterns from the training data. The authors fine-tune Llama-2 on 1,260 question-answer pairs about the Lamini library, a documentation set chosen because its function names and API calls were likely unseen during the model's original pre-training. They then run a four-stage evaluation: memorization (can the model answer questions similar to those it saw during fine-tuning?), recall (can it synthesize information across multi-hop questions built from test-set pairs?), understanding (can it describe what a newly introduced function does?), and reasoning (can it recognize that a newly learned function is analogous to a function from a different platform it already knows?). The central finding is a sharp split. On memorization, the model answers about 80 percent of test questions correctly, though it tends to over-generate, producing answers roughly twice as long as the gold answers and frequently padding correct responses with extra details drawn from pre-training. But as the evaluation probes deeper understanding of the newly introduced entities themselves, performance collapses. Of 39 filtered new function names, only one passes a stress test that checks whether the model can pick the correct function out of a lineup of similar names. Zero entities pass the advanced reasoning test, which asks the model to draw an analogy between a newly learned Lamini function and an equivalent function from a platform like H2O.ai or Google Cloud AI. The model can recall the descriptions of both functions but cannot reason about their similarity. The paper concludes that fine-tuning alone is insufficient for robust knowledge injection: it produces surface-level memorization without the deeper comprehension needed to avoid hallucination on novel domain-specific material.

Core claim

The paper's central discovery is that fine-tuning a large language model on domain-specific question-answer pairs produces a model that performs well on questions closely resembling its training data (roughly 80 percent correct on memorization tasks) but fails almost completely when tested on genuine understanding, retrieval, and reasoning over the newly introduced entities themselves. The four-stage evaluation pipeline reveals a cascading failure: the model passes understanding for common-sounding function names but not for specific ones, achieves near-zero retrieval on specific entities, passes only one entity through a stress test, and passes zero entities through an advanced reasoning类比.

What carries the argument

The paper's argument is carried by a four-stage sequential evaluation pipeline applied to 39 newly introduced entity names (Lamini-specific function calls filtered out of the fine-tuning dataset via TF-IDF, GloVe removal, and manual selection). Stage A (Understanding) asks the fine-tuned model to describe each function. Stage B (Retrieval) asks it to select the correct function for a task from an inverted question. Stage C (Stress Test) asks it to pick the correct function from a shuffled list of the real function plus four plausible distractors generated by GPT-4. Stage D (Advanced Reasoning) asks the model to recognize an analogy between the newly learned function and an equivalent from a

If this is right

  • Organizations that fine-tune LLMs on internal documentation should not assume the model has learned the meaning of new terms; it may be relying on surface-level pattern matching that breaks when queries are phrased differently or require multi-step reasoning.
  • The over-generation tendency (correct answers padded with extra details) suggests that fine-tuned models may confidently mix accurate fine-tuned knowledge with plausible-sounding but unverified pre-training content, a specific hallucination risk profile for domain-adapted systems.
  • The cascading evaluation pipeline (understanding → retrieval → stress test → reasoning) could serve as a diagnostic template for auditing any knowledge-injection method, including retrieval-augmented generation, to distinguish genuine learning from memorization.
  • The zero-percent advanced reasoning result implies that current fine-tuning does not build the kind of relational knowledge representations needed for cross-domain analogical reasoning, pointing to a structural limitation in how gradient-based updates encode new factual information.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The sharp performance drop between memorization and reasoning suggests that fine-tuning may primarily adjust the model's output distribution for familiar question patterns rather than updating its internal knowledge graph with new nodes, consistent with the hypothesis cited in the paper that fine-tuning creates weak links rather than robust new connections.
  • If the over-generation behavior is a symptom of the model interpolating between old pre-training knowledge and new fine-tuned patterns, then mixing retrieval-augmented generation with fine-tuning might reduce hallucination by providing explicit context that anchors the model's response, but this would address recall rather than reasoning.
  • The finding that only common-sounding function names pass the understanding test raises the possibility that the model is leveraging semantic priors from pre-training (recognizing that 'add data' likely means adding data) rather than learning anything specific from the fine-tuning corpus, which would mean the fine-tuning signal is largely redundant for these entities.
  • Testing whether larger models or larger fine-tuning datasets close the reasoning gap would help distinguish whether the failure is a fundamental limitation of gradient-based knowledge injection or an artifact of the small model and dataset used here; the paper's own limitations section gestures at this but does not resolve it.

Load-bearing premise

The evaluation pipeline relies on GPT-4 to categorize model responses as Correct, Wrong, or Correct + Extra Details, and to generate the multi-hop questions and gold answers used in the recall experiments. If GPT-4 introduces its own errors or biases into these labels, the reported metrics are compromised.

What would settle it

If a model fine-tuned on domain-specific data could pass the advanced reasoning test (recognizing analogies between newly learned functions and equivalent functions from other platforms), the paper's claim that fine-tuning produces only memorization without reasoning would be falsified. Conversely, if a different knowledge-injection method (e.g., RAG) also fails the same four-stage pipeline, the limitation would extend beyond fine-tuning specifically.

Figures

Figures reproduced from arXiv: 2606.07521 by Madelyn Scandlen, Sai Prasath S, Sanchita Porwal, Xingjian Bi.

Figure 1
Figure 1. Figure 1: The fine-tuned model outputs longer answers than expected [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The fine-tuned model uses more domain specific words than expected [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
read the original abstract

This study investigates the phenomenon of hallucinations in domain-adapted Large Language Models (LLMs), focusing on the fine-tuning of the Llama-2 model with the Lamini dataset. Hallucinations, or the generation of nonsensical or unfaithful content by LLMs, pose a significant challenge, especially when these models are fine-tuned with domain-specific data. Our methodology involves a series of experiments testing memorization, recall, and reasoning capabilities of the fine-tuned LLM, comparing its performance on novel question-answer pairs and domain-specific information. We found that while the model shows proficiency in tasks similar to its training data, its capability to accurately reason about and recall new domain-specific information remains limited, leading to instances of hallucination. The model demonstrates a tendency to provide correct answers with extra information, suggesting an inclination toward over-generation. These results suggest important limitations of fine-tuning-only approaches for mitigating hallucinations when adapting LLMs to specialized domains and underscore the need for more robust methods in adapting LLMs to specialized domains. The study also provides insights into the varying performance of LLMs on different types of information, revealing a comparative weakness in handling domain-specific queries.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 7 minor

Summary. This paper investigates whether fine-tuning Llama-2-7B-chat on the Lamini documentation dataset leads to genuine learning of new domain-specific entities or to hallucination. The authors evaluate three capabilities: memorization (performance on test-set questions similar to training questions), recall (performance on GPT-4-generated multi-hop questions), and reasoning (a four-stage sequential pipeline testing understanding, retrieval, stress resistance, and advanced reasoning on 39 filtered domain-specific entities). The main findings are that the model achieves ~80% correct responses on memorization tasks (including extra-detail cases), ~76% on recall, and 0% on advanced reasoning. The authors conclude that fine-tuning alone is insufficient for robust knowledge injection.

Significance. The question of whether fine-tuning effectively injects new knowledge into LLMs or merely teaches surface-level patterns is practically important. The paper's multi-stage reasoning evaluation (understanding → retrieval → stress test → advanced reasoning) is a thoughtful design that attempts to separate genuine learning from name-based deduction. The memorization and recall results (n=140 and n=38 respectively) provide a reasonable empirical basis for the over-generation and surface-pattern observations. The use of TF-IDF + GloVe filtering to isolate genuinely novel entities is a reasonable heuristic for identifying newly introduced terms.

major comments (3)
  1. §4.4.2 and Table 3: The headline claim of '0% advanced reasoning' is based on a single entity. The sequential filtering pipeline funnels 39 entities → 23 pass understanding → 9 pass retrieval → 1 passes stress test → 0 passes advanced reasoning. A 0/1 result carries no statistical signal—it is equally consistent with a true 0% pass rate and a 50% pass rate. The conclusion (§6) states '0% performance on our advanced reasoning evaluation across both common and specific entities' without flagging that this is one observation. This is load-bearing because the paper's central claim rests on two quantitative pillars (80% memorization, 0% reasoning), and the reasoning pillar has no inferential weight. The authors should either (a) restructure the evaluation so that advanced reasoning is tested on all entities that pass understanding (not only those surviving all prior stages), or (b) explicitly
  2. §5.1: The '80% correct' memorization claim combines 'Correct' (43.88%) and 'Correct + Extra Details' (37.75%). However, the 'Correct + Extra Details' category is defined in §4.1 as answers that 'include additional relevant information which is not necessary.' The paper does not verify whether this extra information is factually accurate or hallucinated. If some extra details are fabricated, counting these as 'correct' inflates the memorization rate. The paper should clarify whether extra details were checked for factual accuracy, or else report the two categories separately in the headline figure.
  3. §4.4.2D: The advanced reasoning evaluation uses only a single manually selected equivalent entity per Lamini function, with a single justification. The model is asked two yes/no questions. The example in §5.3.2 shows the model correctly answering Q1 (recall) but incorrectly answering Q2 (reasoning), but with n=1 this could reflect ambiguity in the specific entity pair chosen rather than a general reasoning deficit. The paper should acknowledge this design limitation and either test multiple equivalent-entity pairings per function or frame the finding as illustrative rather than quantitative.
minor comments (7)
  1. §4.1: The GPT-4 annotation prompt is minimal and does not specify how to handle partial matches or near-synonyms. A brief discussion of inter-annotator agreement (e.g., comparing GPT-4 labels against human labels on a sample) would strengthen the evaluation.
  2. §4.3: The recall evaluation uses 38 MHQA pairs generated by GPT-4. The paper notes that the underlying test questions are very similar to training questions (BERT similarity 0.8856), which means the multi-hop questions may also be close to training data. This caveat should be stated more prominently in the recall results section, not only in the memorization section.
  3. Table 1: LoRA Alpha is set to 16 with Rank 64. The conventional heuristic is Alpha = 2×Rank. The paper should justify this choice or note it as a potential factor in training quality.
  4. §4.4.1: The manual filtering stage (stage 3) selects only 'function calls, variable names, parameters, and Python scripts' but the criteria for what counts as a function call vs. a domain-specific word are not fully specified. The boundary between 'common entities' and 'specific entities' (§5.3.1) is also subjective. A brief coding guide would improve reproducibility.
  5. §5.1: The top-5 domain-specific words include 'llms2' and 'chatbots' which appear to be general terms rather than Lamini-specific entities. This suggests the filtering pipeline may not fully isolate novel entities.
  6. Figure 1 and Figure 2 are referenced but not labeled with axis titles or legends in the text provided. Ensure these are clearly labeled in the final version.
  7. §6: The conclusion repeats 'specialized domains' twice in the same sentence ('...adapting LLMs to specialized domains and underscores the need for more robust methods in adapting LLMs to specialized domains').

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive reading of our manuscript. The referee raises three substantive points, all of which we take seriously. We agree fully with the first and third comments and will revise accordingly. On the second comment, we agree the manuscript should have been more explicit and will partially revise. We address each point below.

read point-by-point responses

  1. Referee: §4.4.2 and Table 3: The headline claim of '0% advanced reasoning' is based on a single entity. The sequential filtering pipeline funnels 39 entities → 23 pass understanding → 9 pass retrieval → 1 passes stress test → 0 passes advanced reasoning. A 0/1 result carries no statistical signal—it is equally consistent with a true 0% pass rate and a 50% pass rate. The conclusion (§6) states '0% performance on our advanced reasoning evaluation across both common and specific entities' without flagging that this is one observation. This is load-bearing because the paper's central claim rests on two quantitative pillars (80% memorization, 0% reasoning), and the reasoning pillar has no inferential weight. The authors should either (a) restructure the evaluation so that advanced reasoning is tested on all entities that pass understanding (not only those surviving all prior stages), or (b) explicitly

    Authors: The referee is correct, and we will revise the manuscript accordingly. The 0/1 result at the advanced reasoning stage carries no inferential weight, and we should not have presented it as a quantitative pillar of our findings. We accept both of the referee's proposed remedies. First, we will restructure the evaluation so that advanced reasoning is tested on all entities that pass the understanding stage (n=23), rather than only the single entity surviving all prior stages. This will provide a meaningful sample size. Second, we will explicitly flag in both §4.4.2 and §6 that the original result was based on a single observation and that the funnel structure of the sequential pipeline was a design limitation that precluded statistical inference. We will also temper the conclusion in §6 to reflect that the advanced reasoning finding is illustrative rather than statistically grounded, pending the expanded evaluation. revision: yes

  2. Referee: §5.1: The '80% correct' memorization claim combines 'Correct' (43.88%) and 'Correct + Extra Details' (37.75%). However, the 'Correct + Extra Details' category is defined in §4.1 as answers that 'include additional relevant information which is not necessary.' The paper does not verify whether this extra information is factually accurate or hallucinated. If some extra details are fabricated, counting these as 'correct' inflates the memorization rate. The paper should clarify whether extra details were checked for factual accuracy, or else report the two categories separately in the headline figure.

    Authors: The referee raises a valid concern. In our original evaluation, the GPT-4 annotation prompt categorized responses as 'Correct + Extra Details' when the core answer aligned with the gold response, but we did not systematically verify whether the extra details were factually accurate or hallucinated. We acknowledge that if some extra details are fabricated, combining the two categories inflates the headline memorization rate. We will partially revise by taking two steps. First, we will report the two categories separately in the headline figure so that readers can see the breakdown (43.88% strictly correct vs. 37.75% correct with extra details). Second, we will add an explicit caveat in §5.1 noting that the factual accuracy of the extra details was not verified and that the combined 80% figure should be interpreted as an upper bound on memorization accuracy. We agree that a full factual audit of the extra details would be ideal, but given the scope of the revision, we will at minimum make the limitation transparent and report the categories separately. revision: partial

  3. Referee: §4.4.2D: The advanced reasoning evaluation uses only a single manually selected equivalent entity per Lamini function, with a single justification. The model is asked two yes/no questions. The example in §5.3.2 shows the model correctly answering Q1 (recall) but incorrectly answering Q2 (reasoning), but with n=1 this could reflect ambiguity in the specific entity pair chosen rather than a general reasoning deficit. The paper should acknowledge this design limitation and either test multiple equivalent-entity pairings per function or frame the finding as illustrative rather than quantitative.

    Authors: We agree with the referee. The use of a single manually selected equivalent entity per Lamini function, with a single justification, is a design limitation that we did not adequately acknowledge. The referee is correct that with n=1, the observed failure could reflect ambiguity in the specific entity pair rather than a general reasoning deficit. We will revise the manuscript in two ways. First, we will explicitly acknowledge this design limitation in §4.4.2D, noting that the single-pair design limits generalizability. Second, in conjunction with the restructuring described in our response to the first comment, we will expand the advanced reasoning evaluation to include multiple equivalent-entity pairings per function where feasible, so that the finding is supported by more than one observation. Where multiple pairings are not feasible, we will frame the finding as illustrative rather than quantitative, as the referee suggests. revision: yes

Circularity Check

0 steps flagged

No circularity found: empirical evaluation paper with no derivation chain or self-citation load-bearing structure.

full rationale

This is an empirical evaluation paper, not a theoretical derivation. The paper does not claim to derive any result from first principles, fit a parameter to data and then predict the same data, or invoke self-citations as load-bearing evidence. There are no self-citations visible in the reference list — all cited works are by different author groups. The GPT-4-as-judge concern (raised by the reader) is a validity and reliability issue, not a circularity issue: GPT-4 generates questions/answers and separately evaluates model responses against gold answers, but these are distinct roles and the evaluation is not tautologically defined by the generation step. The memorization gold answers come from the test set itself (not GPT-4), and the recall gold answers (A3) are generated by GPT-4 from existing test pairs (Q1-A1, Q2-A2) — a potential bias concern but not a case where the prediction equals the input by construction. The n=1 advanced reasoning result is a statistical power problem, not a circularity problem. None of the seven enumerated circularity patterns apply.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The paper uses standard NLP/ML methods and does not invent new entities. The free parameters are standard fine-tuning hyperparameters and filtering thresholds. The axioms rely on GPT-4 as a capable evaluator and generator, which is a common but unverified domain assumption in this study.

free parameters (4)
  • LoRA Rank = 64
    Chosen hyperparameter for QLoRA fine-tuning.
  • Learning Rate = 5e-5
    Chosen hyperparameter for fine-tuning.
  • Epochs = 4
    Chosen hyperparameter for fine-tuning.
  • TF-IDF threshold = 1000
    Manual cutoff for top tf-idf words used in entity filtering.
axioms (4)
  • domain assumption GPT-4 is a reliable evaluator for categorizing LLM responses as correct, wrong, or correct with extra details.
    Used throughout §4.1 for annotating model responses against gold answers.
  • domain assumption GPT-4 can generate valid multi-hop questions and gold answers from existing QA pairs.
    Used in §4.3 to create the recall evaluation dataset.
  • domain assumption The Lamini dataset was not encountered by Llama-2 during pre-training.
    Stated in §3.1 as the rationale for dataset selection; assumed but not proven.
  • domain assumption RAG + base model generates accurate gold function descriptions.
    Used in §4.4.2.A to create gold labels for the understanding test.

pith-pipeline@v1.1.0-glm · 14233 in / 2207 out tokens · 173897 ms · 2026-07-05T16:50:42.382346+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 27 canonical work pages · 9 internal anchors

  1. [1]

    Leancontext: Cost-efficient domain-specific question answering using llms, 2023

    Md Adnan Arefeen, Biplob Debnath, and Srimat Chakradhar. Leancontext: Cost-efficient domain-specific question answering using llms, 2023

    work page 2023

  2. [2]

    Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, ...

    work page 2022

  3. [3]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    work page 2020

  4. [4]

    QLoRA: Efficient Finetuning of Quantized LLMs

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  5. [5]

    Chain-of-verification reduces hallucination in large language models, 2023

    Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. Chain-of-verification reduces hallucination in large language models, 2023

    work page 2023

  6. [6]

    The false promise of imitating proprietary llms, 2023

    Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary llms, 2023

    work page 2023

  7. [7]

    Don't Stop Pretraining: Adapt Language Models to Domains and Tasks

    Suchin Gururangan, Ana Marasovi \'c , Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2004

  8. [8]

    Survey of hallucination in natural language generation

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55 0 (12): 0 1--38, 2023

    work page 2023

  9. [9]

    Lamini docs dataset at hugging face, 2023

    Lamini. Lamini docs dataset at hugging face, 2023. URL https://huggingface.co/datasets/lamini/lamini_docs/tree/main

    work page 2023

  10. [10]

    Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021

    work page 2021

  11. [11]

    ChipNeMo: Domain-Adapted LLMs for Chip Design

    Mingjie Liu, Teo Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjian Liang, Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktaroglu, et al. Chipnemo: Domain-adapted llms for chip design. arXiv preprint arXiv:2311.00176, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  12. [12]

    Sources of hallucination by large language models on inference tasks, 2023

    Nick McKenna, Tianyi Li, Liang Cheng, Mohammad Javad Hosseini, Mark Johnson, and Mark Steedman. Sources of hallucination by large language models on inference tasks, 2023

    work page 2023

  13. [13]

    Chatgpt, 2023a

    OpenAI. Chatgpt, 2023a. URL https://openai.com/blog/chatgpt/

    work page

  14. [14]

    Gpt-4, 2023b

    OpenAI. Gpt-4, 2023b. URL https://openai.com/gpt-4/

    work page

  15. [15]

    Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1908

  16. [16]

    Reinforcement learning from human feedback: Progress and challenges, 2023

    John Schulman. Reinforcement learning from human feedback: Progress and challenges, 2023

    work page 2023

  17. [17]

    Yixuan Tang, Hwee Tou Ng, and Anthony Tung. Do multi-hop question answering systems know how to answer the single-hop sub-questions? In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp.\ 3244--3249, Online, April 2021. Associa...

    work page doi:10.18653/v1/2021.eacl-main.283 2021

  18. [18]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Harts...

    work page 2023

  19. [19]

    Med-HALT: Medical Domain Hallucination Test for Large Language Models

    Logesh Kumar Umapathi, Ankit Pal, and Malaikannan Sankarasubbu. Med-halt: Medical domain hallucination test for large language models. arXiv preprint arXiv:2307.15343, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  20. [20]

    Benchmarking Large Language Models for News Summarization

    Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. Benchmarking large language models for news summarization. arXiv preprint arXiv:2301.13848, 2023 a

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  21. [21]

    Siren's song in the ai ocean: A survey on hallucination in large language models, 2023 b

    Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. Siren's song in the ai ocean: A survey on hallucination in large language models, 2023 b

    work page 2023

  22. [22]

    TrafficSafetyGPT: Tuning a Pre-trained Large Language Model to a Domain-Specific Expert in Transportation Safety

    Ou Zheng, Mohamed Abdel-Aty, Dongdong Wang, Chenzhu Wang, and Shengxuan Ding. Trafficsafetygpt: Tuning a pre-trained large language model to a domain-specific expert in transportation safety. arXiv preprint arXiv:2307.15311, 2023 a

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  23. [23]

    Why Does ChatGPT Fall Short in Providing Truthful Answers?

    Shen Zheng, Jie Huang, and Kevin Chen-Chuan Chang. Why does chatgpt fall short in providing truthful answers. ArXiv preprint, abs/2304.10513, 2023 b

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  24. [24]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

    work page

  25. [25]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

    work page

  26. [26]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

    work page

  27. [27]

    Chain-of-Verification Reduces Hallucination in Large Language Models

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.18653/v1/2020.acl-main.173 2023