Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Bridging the Safety Gap: A Guardrail Pipeline for Trustworthy LLM Inferences

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Wildflare GuardRail claims that a coordinated pipeline of small fine-tuned detectors, vector grounding, rule-based wrappers, and a repair model can make LLM inference safer end to end, with the moderation model matching OpenAI's API and…

desk verdict Useful modular guardrail pipeline with a promising explainable repair loop, but the headline claims—especially 80.7% fix rate—rest on in-family evaluations that need out-of-distribution support. read the letter →

arxiv 2502.08142 v1 pith:FYYFKCUP submitted 2025-02-12 cs.AI

Shanshan Han , Salman Avestimehr , Chaoyang He This is my paper · ORCID
classification cs.AI
keywords LLMguardrailshallucinationdetectionunsafecontentmoderationretrieval-augmentedgenerationpromptinjectiondefenseexplainablesafetylightweightwrappersedgedeployment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that LLM safety is a pipeline problem, not a single-model problem, and introduces Wildflare GuardRail to demonstrate the point. The system places a fine-tuned Safety Detector before and after the LLM, grounds user queries with retrieved knowledge, edits outputs with lightweight wrappers, and repairs hallucinated answers using explanations produced by the detector. Its reported results are that the unsafe-input detector performs comparably to OpenAI's moderation API, that the URL-handling wrapper works at 1.06 seconds per query with 100% detection accuracy, and that the repair model fixes 80.7% of hallucinated test outputs. This matters because isolated safety tools leave gaps elsewhere in inference, whereas a coordinated pipeline can be deployed flexibly and run on edge devices.

What carries the argument

The load-bearing mechanism is the explanation loop between Safety Detector and Repairer. The hallucination detector is trained on HaluEval prompts to answer with 'Yes' or 'No' as its first token, and its hallucination probability is read from the top-k first-token logits rather than from a full generation; when the answer is 'Yes', the model also produces a root-cause explanation. The Repairer then receives the original question, the retrieved context, the hallucinated answer, and that explanation in a templated prompt, and is trained to output the corrected answer. Around this loop, Grounding supplies context through two indexing schemes (Whole Knowledge Index and Key Information Index, measured by callback), and Customizer supplies deterministic regex- and API-based wrappers such as Google SafeBrowsing for URL warnings that cannot be done reliably by prompt engineering.

What would settle it

Run the Repairer on hallucinated outputs produced by the target LLM on real user queries from a domain outside HaluEval, such as medical or financial questions, and have human annotators check each corrected answer against the provided ground-truth context; if the rate of factually consistent corrections is substantially below 80.7%, or if the corrections introduce new false statements, the claimed effectiveness comes from the benchmark rather than from the pipeline itself.

Watch

Extended reading notes

Core claim

The central claim is that the safety of an LLM inference should be managed as a single workflow rather than as isolated fixes, and that Wildflare GuardRail realizes this workflow by combining a fine-tuned Safety Detector, a Grounding module for vector retrieval, rule-based Customizer wrappers, and a Repairer that uses hallucination explanations. All three fine-tuned models are built from the same 1.6B base model, which the paper says makes them light enough for edge deployment. The paper's headline empirical assertions are that the unsafe-input detector performs comparably to OpenAI's moderation API, that the Customizer wrapper detects and validates URLs in 1.06 seconds per query with 100% detection accuracy, that hallucination detection reaches 0.78 accuracy on HaluEval, and that the Repairer fixes 80.7% of hallucinated answers in the HaluEval-QA test set, judged by the Vectara consistency model.

Load-bearing premise

The results stand on the assumption that HaluEval's questions, contexts, and hallucinated answers—and the Vectara consistency judge—represent how hallucinations and fixes behave for real user queries; if that distribution does not transfer, the 80.7% fix rate and 0.78 detection accuracy are benchmark artifacts rather than real-world evidence.

Editorial extensions

If this is right

  • A single Wildflare GuardRail deployment covers pre-inference and post-inference safety: reject unsafe prompts, ground queries with retrieved context, edit outputs with wrappers, and repair hallucinations without switching between separate tools.
  • A 1.6B fine-tuned model can match OpenAI's moderation API for unsafe-input detection, so content moderation can run on edge devices without sending user queries to external services.
  • Deterministic wrappers are faster and more reliable than LLM calls for deterministic output edits such as URL warning placement: 1.06s per query with 100% URL detection versus 9-35s and lower or zero accuracy.
  • The explanation produced by hallucination detection is informative enough to drive a repair stage, which fixes 80.7% of hallucinated answers in the reported test set.
  • Key Information Index gives higher retrieval callback than Whole Knowledge Index, recommending question-focused indexing for grounding in FAQ-style knowledge bases.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test reusing the detector's hallucination explanation for anything except repair; a natural extension is to make the LLM abstain, add citations, or flag low-confidence answers when the hallucination probability is high.
  • Because the Repairer and hallucination detector are both trained and tested on HaluEval, the 80.7% fix rate is likely optimistic; a deployment test on hallucinated outputs produced by the serving LLM under real user traffic would measure the true gap.
  • The top-k first-token probability method is a general lightweight classification trick and could be transferred to other yes/no guardrail decisions, such as prompt-injection detection, without retraining the whole pipeline.
  • The callback advantage of Key Information Index over Whole Knowledge Index suggests that vector databases for grounding can be built from question-style keys rather than full entries, reducing storage and retrieval cost for FAQ-style knowledge bases.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Wildflare GuardRail, a guardrail pipeline for LLM inference that combines a Safety Detector (unsafe input moderation plus explainable hallucination detection), a Grounding module (vector retrieval), a Customizer (lightweight rule-based wrappers), and a Repairer (hallucination fixing). The headline empirical claims are that the unsafe-input detector is comparable to the OpenAI moderation API, that the URL wrapper processes a query in 1.06 s with 100% detection accuracy and 83.33% validation accuracy, and that the Repairer fixes 80.7% of hallucinated outputs. The paper also contributes two indexing schemes for grounding and a two-stage explainable hallucination mitigation design.

Significance. If the quantitative claims were validated, Wildflare GuardRail would be a useful contribution: it addresses safety across the full inference workflow rather than in isolation, uses a small open base model (Fox-1) that is plausibly edge-deployable, and the two-stage detection-and-repair design with root-cause explanations is a sensible direction. The paper is also honest about using a compact training set for moderation and about the lightweight nature of the Customizer. However, the headline numbers currently rest on a very small URL test set and on an in-family HaluEval evaluation loop, and the moderation comparison is not reported with numbers; the evidence in the manuscript is therefore not yet sufficient to support the strength of the abstract claims.

major comments (4)
  1. [§8, Exp 1] The claim that the Safety Detector's unsafe-input model 'achieves comparable performance with OpenAI API' is not supported by any quantitative result in the text: Figure 4 is referenced, but no accuracy, precision, recall, F1, ROC, or test-set statistics are reported, and the comparison protocol is undefined. Since this is one of the paper's three headline claims, the comparison must be reported with concrete numbers, error bars, and a description of how the baselines were invoked.
  2. [§8, Exp 4 and Table 3] The URL evaluation is built from only 30 constructed texts (15 from each of two datasets) with a 20% probability of inserting malicious URLs; at that rate the expected number of malicious URL instances is about six, so the reported 100% detection accuracy and 83.33% validation accuracy rest on a handful of positive cases. The abstract's '100% accuracy' should be qualified, and the experiment should be repeated on a larger and more realistic URL corpus before the efficiency claim is used as a headline result.
  3. [§8, Exp 5 and §7] The 80.7% fix rate is an in-family, closed-loop number and is not evidence of real-world repair: the Repairer is fine-tuned on HaluEval, the hallucination_reason column is derived from Safety Detector (also trained on HaluEval), the evaluation uses only 100 HaluEval-QA test records, and success is judged by the Vectara consistency model. In addition, since the Safety Detector's hallucination accuracy is 0.78, the pipeline-level fraction of all hallucinated outputs actually repaired is at most about 0.63, a quantity the paper does not state; the Repairer evaluation should be reported conditional on detector errors, and an out-of-distribution evaluation is needed to support the 'safety gap' claim.
  4. [Definition 1] The probability of hallucination in Definition 1 is not well-defined as written: P(t_i | t_i in T(Yes)) is undefined for tokens that are neither Yes nor No tokens, and the denominator sums only the Yes and No contributions, so the ratio is not normalized over the top-k distribution and may not be a probability. The estimator should be written as the sum of probabilities of Yes-matching tokens divided by the sum over all top-k tokens, or the conditional form should be justified.
minor comments (4)
  1. [§5] There are several typos, including 'ensurers' for 'ensures' in the first paragraph of §5 and 'founded' for 'found' in Example 1 of §6.
  2. [§6, Example 1] The sentence 'Customizer should check the safety of the URLs founded' should read 'URLs found'; the paper would also benefit from a pass to remove the preposition duplication in the Exp 4 sentence 'evaluated the efficiency of Customizer in with the URL detection task'.
  3. [§8, Exp 3] The Grounding evaluation uses only 50 randomly selected questions per dataset and reports callback values without standard deviations or significance tests; the variability of the callback estimates should be reported.
  4. [General] The manuscript uses placeholder venue and DOI information ('Conference'17', '10.1145/nnnnnnn.nnnnnnn') and appears to be formatted as a conference submission; these should be updated for a journal submission.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported accuracy and fix-rate numbers are held-out benchmark measurements, not inputs renamed as predictions; the HaluEval-based loop is an external-validity caveat, not a definitional reduction.

full rationale

The paper's central quantitative claims are empirical benchmark results rather than quantities derived from their own inputs by construction. Exp 2 reports hallucination detection accuracy 0.78 on the held-out HaluEval test split, and Exp 5 reports an 80.7% fix rate using 100 HaluEval-QA test records with Vectara's hallucination-evaluation model as an independent judge of output consistency. Although the Safety Detector, its hallucination explanations, and the Repairer all originate from HaluEval, the test split is not used in training, and the success criterion is external to the fine-tuned models. The fact that the hallucination_reason column is derived from Safety Detector outputs means the Repairer is trained and evaluated within the same family of model-generated explanations, which is a legitimate generalization and validity concern, but it is not a circular reduction: the Repairer could still fail to fix outputs, and the 80.7% number is not forced by the training objective. The only notable self-citation is Fox-1 [68], a blog by the authors' organization used as the base model; this is a reproducibility caveat rather than a load-bearing circular argument, because the fine-tuned models are judged against external benchmarks and baselines. No equation, fitted parameter, or prediction in the paper reduces by construction to its own input, so no circularity is identified.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The pipeline introduces no physical or theoretical entities; all modules are implemented with standard LLM, RAG, and rule-based components. The central claims rest on hand-set thresholds, small evaluation samples, and the assumed validity of public dataset labels and external judge models. The in-family HaluEval evaluation loop is the main structural weakness.

free parameters (5)
  • Hallucination probability threshold = 0.5
    Algorithm 2 sets P_halu >= 0.5 to call an answer hallucinated; no sensitivity analysis is provided.
  • Top-k first-token window = 10
    Algorithm 2 uses default k=10 for the first-token probabilities that define P_halu.
  • Grounding evaluation sample size = 50 questions per dataset
    Exp 3 computes callback on 50 randomly selected questions per dataset; no error bars are given.
  • Malicious URL injection probability = 20%
    Exp 4 inserts malicious URLs with 20% probability into 30 constructed texts, producing a small positive set.
  • Repairer evaluation sample = 100 HaluEval-QA records
    Exp 5 evaluates on 100 test records although Table 2 lists 1,000 test records for the Repairer subsets.
assumptions (5)
  • domain assumption Public dataset labels (HEx-PHI, Jigsaw, HaluEval, etc.) are accurate enough for training and evaluation.
    Fine-tuning and metrics rely on these labels as ground truth without auditing them (§8, Table 2).
  • domain assumption GPT-4 generated hallucination explanations are valid training targets.
    Algorithm 1 uses GPT-4 API to write hallucination reasons that become labels for the detection model (§4.2).
  • domain assumption Vectara hallucination evaluation model provides trustworthy consistency scores.
    Repairer accuracy is judged by Vectara's model (§8 Exp 5); its own error rate is not discussed.
  • domain assumption Binary safe/unsafe classification is sufficient to protect LLM services.
    §4.1 argues a single binary label is sufficient; this collapses distinct risk categories used by comparable tools.
  • standard math Autoregressive LLMs cannot anticipate future tokens, so post-hoc wrappers are necessary.
    Example 2 in §6 relies on this property to motivate wrappers for jobs like counting words.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bridging the Safety Gap: A Guardrail Pipeline for Trustworthy LLM Inferences." pith.science (2026). https://pith.science/paper/FYYFKCUP

@misc{pith2026250208142,
  author       = {Pith},
  title        = {Pith review of: Bridging the Safety Gap: A Guardrail Pipeline for Trustworthy LLM Inferences},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FYYFKCUP}},
  note         = {Machine review of arXiv:2502.08142}
}
read the original abstract

We present Wildflare GuardRail, a guardrail pipeline designed to enhance the safety and reliability of Large Language Model (LLM) inferences by systematically addressing risks across the entire processing workflow. Wildflare GuardRail integrates several core functional modules, including Safety Detector that identifies unsafe inputs and detects hallucinations in model outputs while generating root-cause explanations, Grounding that contextualizes user queries with information retrieved from vector databases, Customizer that adjusts outputs in real time using lightweight, rule-based wrappers, and Repairer that corrects erroneous LLM outputs using hallucination explanations provided by Safety Detector. Results show that our unsafe content detection model in Safety Detector achieves comparable performance with OpenAI API, though trained on a small dataset constructed with several public datasets. Meanwhile, the lightweight wrappers can address malicious URLs in model outputs in 1.06s per query with 100% accuracy without costly model calls. Moreover, the hallucination fixing model demonstrates effectiveness in reducing hallucinations with an accuracy of 80.7%.

Figures

Figures reproduced from arXiv: 2502.08142 by the authors.

Figure 1
Figure 1. Overview. with Safety Detector and ground the queries with context knowl￾edge with Grounding. Safety Detector monitors user inputs to identify and reject queries that might be unsafe. The monitoring in￾cludes typical safety checks, including toxicity, stereotypes, threats, obscenities, prompt injection attacks, etc. Any form of unsafe con￾tent will lead to the queries being rejected. Inputs that pass this initial sa… view at source ↗
Figure 2
Figure 2. Prompt templates and sample training data for hallucination detection and reasoning. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Prompt templates and sample training data for [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Whole index [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Get Experience from Practice: LLM Agents with Record & Replay

    cs.LG 2025-05 reject novelty 4.0 of 10

    AgentRR is a proposed paradigm that records agent traces, generalizes them into multi-level experiences, and replays them under safety checks to make LLM agents cheaper, faster, and more reliable.

Reference graph

Works this paper leans on

85 extracted references · 36 canonical work pages · cited by 1 Pith paper

  1. [1]

    Fatih Kadir Akın. 2023. Awesome ChatGPT Prompts Dataset. (2023). https: //huggingface.co/datasets/fka/awesome-chatgpt-prompts

    2023

  2. [2]

    Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-40B: an open large language model with state-of-the-art performance. (2023)

    2023

  3. [3]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv 2023

  4. [4]

    Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung, Maksym Zhu- ravinskyi, Reshinth Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, et al. 2024. Stable LM 2 1.6 B Technical Report. arXiv preprint arXiv:2402.17834 (2024)

    arXiv 2024

  5. [5]

    Zhiyuan Chang, Mingyang Li, Yi Liu, Junjie Wang, Qing Wang, and Yang Liu

  6. [6]

    Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2024. Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 17754–17762

    2024

  7. [7]

    Junjie Chu, Yugeng Liu, Ziqing Yang, Xinyue Shen, Michael Backes, and Yang Zhang. 2024. Comprehensive Assessment of Jailbreak Attacks Against LLMs. ArXiv abs/2402.05668 (2024). https://api.semanticscholar.org/CorpusID: 267547966

    arXiv 2024

  8. [8]

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. 2023. Dola: Decoding by contrasting layers improves factuality in large language models. arXiv preprint arXiv:2309.03883 (2023)

    arXiv 2023

Show all 85 references
  1. [9]

    Data Is Better Together Community. 2024. 10k Prompts Ranked Dataset. (2024). https://huggingface.co/datasets/DIBT/10k_prompts_ranked

    2024

  2. [10]

    Data Is Better Together Community. 2024. Iterative Prompt 20K Dataset. (2024). https://huggingface.co/datasets/RLHFlow/iterative-prompt-v1-iter1-20K

    2024

  3. [11]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805 (2018)

    2018 arXiv

  4. [12]

    Antareep Dey. 2023. Patient Doctor Chat Dataset. (2023). https://huggingface. co/datasets/antareepdey/Patient_doctor_chat

    2023

  5. [13]

    Antareep Dey. 2023. Reddit SYAC URL Dataset. (2023). https://huggingface.co/ datasets/marksverdhei/reddit-syac-urls

    2023

  6. [14]

    Hayder Elesedy, Pedro M Esperança, Silviu Vlad Oprea, and Mete Ozay. 2024. LoRA-Guard: Parameter-Efficient Guardrail Adaptation for Content Moderation of Large Language Models. arXiv preprint arXiv:2407.02987 (2024)

    2024 arXiv

  7. [15]

    Falconsai. 2023. Text Summarization Model. (2023). https://huggingface.co/ Falconsai/text_summarization

    2023

  8. [16]

    Mingyuan Fan, Chengyu Wang, Cen Chen, Yang Liu, and Jun Huang. 2023. On the Trustworthiness Landscape of State-of-the-art Generative Models: A Survey and Outlook. https://api.semanticscholar.org/CorpusID:266149416

    2023

  9. [17]

    Katja Filippova. 2020. Controlled hallucinations: Learning to generate faithfully from noisy data. arXiv preprint arXiv:2010.05873 (2020)

    2020 arXiv

  10. [18]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 (2023)

    2023 arXiv

  11. [19]

    Google. 2005. Google SafeBrowsing. (2005). https://developers.google.com/safe- browsing

    2005

  12. [20]

    Sagar Goyal, Eti Rastogi, Sree Prasanna Rajagopal, Dong Yuan, Fen Zhao, Jai Chintagunta, Gautam Naik, and Jeff Ward. 2024. Healai: A healthcare llm for effective medical documentation. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining . 1167–1168

    2024

  13. [21]

    Laura Hanu and Unitary team. 2020. Detoxify. Github. https://github.com/unitaryai/detoxify

    2020

  14. [22]

    Jack Hao. 2023. Jailbreak Classification Dataset. (2023). https://huggingface.co/ datasets/jackhhao/jailbreak-classification

    2023

  15. [24]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ArXiv abs/2311.05...

    2023 arXiv

  16. [25]

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. 2023. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674 (2023)

    2023 arXiv

  17. [26]

    Kenneth E Iverson. 1962. A programming language. In Proceedings of the May 1-3, 1962, spring joint computer conference . 345–351

    1962

  18. [27]

    Prince Jha, Raghav Jain, Konika Mandal, Aman Chadha, Sriparna Saha, and Pushpak Bhattacharyya. 2024. MemeGuard: An LLM and VLM-based Frame- work for Advancing Content Moderation via Meme Intervention. arXiv preprint arXiv:2406.05344 (2024)

    2024 arXiv

  19. [28]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, De- vendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al . 2023. Mistral 7B. arXiv preprint arXiv:2310.06825 (2023)

    2023 arXiv

  20. [29]

    JIGSAW. 2018. Jigsaw Toxic Comment Classification Dataset. (2018). https: //www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/data

    2018

  21. [30]

    JIGSAW. 2019. Jigsaw Unintended Bias in Toxicity Classification Dataset. (2019). https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification

    2019

  22. [31]

    JIGSAW. 2020. Jigsaw Multilingual Toxic Comment Classification Dataset. (2020). https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment- classification/data

    2020

  23. [32]

    Cheongwoong Kang and Jaesik Choi. 2023. Impact of Co-occurrence on Factual Knowledge of Large Language Models. arXiv preprint arXiv:2310.08256 (2023)

    2023 arXiv

  24. [33]

    Wis Kojohnjaratkul. 2023. Instruction Following Dataset. (2023). https:// huggingface.co/datasets/wis-k/instruction-following-eval

    2023

  25. [34]

    Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Soheil Feizi, and Himabindu Lakkaraju. 2023. Certifying LLM Safety against Adversarial Prompting. ArXiv abs/2309.02705 (2023). https://api.semanticscholar.org/CorpusID:261557007

    2023 arXiv

  26. [35]

    Deepak Kumar, Yousef Anees AbuHashem, and Zakir Durumeric. 2024. Watch Your Language: Investigating Content Moderation with Large Language Models. In Proceedings of the International AAAI Conference on Web and Social Media , Vol. 18. 865–878

    2024

  27. [36]

    Alyssa Lees, Vinh Q Tran, Yi Tay, Jeffrey Sorensen, Jai Gupta, Donald Metzler, and Lucy Vasserman. 2022. A new generation of perspective api: Efficient multilingual character-level transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data M...

    2022

  28. [37]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing S...

    2020

  29. [38]

    Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023. Halueval: A large-scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 6449–6464

    2023

  30. [39]

    Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, and You Zhang. 2023. ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge. Cureus 15, 6 (2023)

    2023

  31. [40]

    Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. 2023. Large language models in finance: A survey. In Proceedings of the fourth ACM international conference on AI in finance . 374–382

    2023

  32. [41]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. TruthfulQA: Measuring How Models Mimic Human Falsehoods. arXiv:2109.07958 [cs.CL]

    2021 arXiv

  33. [42]

    Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, and Jingbo Shang. 2023. ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation. arXiv:2310.17389 [cs.CL]

    2023 arXiv

  34. [43]

    Tianyu Liu, Yizhe Zhang, Chris Brockett, Yi Mao, Zhifang Sui, Weizhu Chen, and Bill Dolan. 2021. A token-level reference-free hallucination detection benchmark for free-form text generation. arXiv preprint arXiv:2104.08704 (2021)

    2021 arXiv

  35. [44]

    Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yanhong Zheng, and Yang Liu. 2023. Prompt Injection attack against LLM-integrated Applications. ArXiv abs/2306.05499 (2023). https://api. semanticscholar.org/CorpusID:259129807

    2023 arXiv

  36. [45]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019)

    2019 arXiv

  37. [46]

    Huan Ma, Changqing Zhang, Huazhu Fu, Peilin Zhao, and Bingzhe Wu. 2023. Adapting large language models for content moderation: Pitfalls in data engi- neering and supervised fine-tuning. arXiv preprint arXiv:2310.03400 (2023)

    2023 arXiv

  38. [47]

    Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgpt: Zero- resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896 (2023)

    2023 arXiv

  39. [48]

    Todor Markov, Chong Zhang, Sandhini Agarwal, Florentine Eloundou Nekoul, Theodore Lee, Steven Adler, Angela Jiang, and Lilian Weng. 2023. A holistic approach to undesired content detection in the real world. In Proceedings of the AAAI Conference on Artificial Intelligence , Vo...

    2023

  40. [49]

    Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. arXiv preprint arXiv:2005.00661 (2020)

    2020 arXiv

  41. [50]

    Sachin Mehta, Farzad Abdolhosseini, and Mohammad Rastegari. 2022. CVNets: High Performance Library for Computer Vision. In Proceedings of the 30th ACM International Conference on Multimedia (MM ’22)

    2022

  42. [51]

    Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, and Mohammad Rastegari. 2024. OpenELM: An Efficient Language Model Family with Open Training and Inference Framework. a...

    2024 arXiv

  43. [52]

    Safeer Mohiuddin. 2024. Guardrails AI’s Commitment to Responsible Vulnerabil- ity Disclosure. https://www.guardrailsai.com/blog/commitment-to-responsible- vulnerability

    2024

  44. [53]

    OWASP. 2023. Model Denial of Service in OWASP Top 10 List for Large Language Models. https://llmtop10.com/llm04/. Conference’17, July 2017, Washington, DC, USA Han et al

    2023

  45. [54]

    OWASP. 2023. Prompt Injection in OWASP Top 10 List for Large Language Models. https://llmtop10.com/llm01/

    2023

  46. [55]

    OWASP. 2023. Supply Chain Vulnerabilities in OWASP Top 10 List for Large Language Models. https://llmtop10.com/llm05/

    2023

  47. [56]

    OWASP. 2023. Training Data Poisoning in OWASP Top 10 List for Large Language Models. https://llmtop10.com/llm03/

    2023

  48. [57]

    Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Pe- ter Henderson. 2024. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!. In The Twelfth International Conference on Learning Representations. https://openre...

    2024

  49. [58]

    Jianing Qiu, Kyle Lam, Guohao Li, Amish Acharya, Tien Yin Wong, Ara Darzi, Wu Yuan, and Eric J Topol. 2024. LLM-based agentic systems in medicine and healthcare. Nature Machine Intelligence 6, 12 (2024), 1418–1420

    2024

  50. [59]

    Mohamed Rashad. 2023. ChatGPT Prompts Dataset. (2023). https://huggingface. co/datasets/MohamedRashad/ChatGPT-prompts

    2023

  51. [60]

    Vipula Rawte, Amit Sheth, and Amitava Das. 2023. A survey of hallucination in large foundation models. arXiv preprint arXiv:2309.05922 (2023)

    2023 arXiv

  52. [61]

    Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, and Jonathan Cohen. 2023. Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails. arXiv preprint arXiv:2310.10501 (2023)

    2023 arXiv

  53. [62]

    Johann Rehberger. 2023. LLM Apps: Don’t Get Stuck in an Infinite Loop! https: //embracethered.com/blog/posts/2023/llm-cost-and-dos-threat/

    2023

  54. [63]

    Rubén Darío Jaramillo Romero. 2023. ChatGPT Jailbreak Prompts Dataset. (2023). https://huggingface.co/datasets/rubend18/ChatGPT-Jailbreak-Prompts

    2023

  55. [64]

    Mark Russinovich, Ahmed Salem, and Ronen Eldan. 2024. Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack. ArXiv abs/2404.01833 (2024). https://api.semanticscholar.org/CorpusID:268856920

    2024 arXiv

  56. [65]

    Andrew Siah. 2024. Filtered Personalization Prompt Response Dataset. (2024). https://huggingface.co/datasets/andrewsiah/filtered_personalization_ prompt_response

    2024

  57. [66]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupati- raju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295 (2024)

    2024 arXiv

  58. [67]

    Perspective API team. 2008. Perspective API Case Studies. (2008). https: //perspectiveapi.com/case-studies/

    2008

  59. [68]

    TensorOpera AI Team. 2024. TensorOpera Unveils Fox Foundation Model: A Pioneering Small Language Model (SLM) for Cloud and Edge. https://blog.tensoropera.ai/tensoropera-unveils-fox-foundation-model-a- pioneering-open-source-slm-leading-the-way-against-tech-giants/

    2024

  60. [69]

    Simone Tedeschi, Felix Friedrich, Patrick Schramowski, Kristian Kersting, Roberto Navigli, Huu Nguyen, and Bo Li. 2024. ALERT: A Comprehensive Benchmark for Assessing Large Language Models’ Safety through Red Teaming. ArXiv abs/2404.08676 (2024). https://api.semanticscholar.or...

    2024 arXiv

  61. [70]

    NM Testing. 2024. QA Chat Prompts Dataset. (2024). https://huggingface.co/ datasets/nm-testing/qa-chat-prompts

    2024

  62. [71]

    Xing Tian. 2023. E Commerce Customer Service Dataset. https://huggingface. co/datasets/qgyd2021/e_commerce_customer_service

    2023

  63. [72]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  64. [73]

    Vectara. 2023. Hallucination Evaluation Model. (2023). https://huggingface.co/ vectara/hallucination_evaluation_model

    2023

  65. [74]

    Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zi-Han Lin, Yuk-Kit Cheng, Sanmi Koyejo, Dawn Xiaodong Song, and Bo Li. 2023. Decoding...

    2023 arXiv

  66. [75]

    Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebas- tian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann

  67. [76]

    Kankanhalli

    Ziwei Xu, Sanjay Jain, and Mohan S. Kankanhalli. 2024. Hallucination is Inevitable: An Innate Limitation of Large Language Models. ArXiv abs/2401.11817 (2024). https://api.semanticscholar.org/CorpusID:267069207

    2024 arXiv

  68. [77]

    Zihao Xu, Yi Liu, Gelei Deng, Yuekang Li, and Stjepan Picek. 2024. LLM Jailbreak Attack versus Defense Techniques–A Comprehensive Study. arXiv preprint arXiv:2402.13457 (2024)

    2024 arXiv

  69. [78]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. HotpotQA: A dataset for di- verse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600 (2018)

    2018 arXiv

  70. [79]

    Ziqi Yang, Xuhai Xu, Bingsheng Yao, Ethan Rogers, Shao Zhang, Stephen Intille, Nawar Shara, Guodong Gordon Gao, and Dakuo Wang. 2024. Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults. Proceedings of the ACM on Interactive,...

    2024

  71. [80]

    Jiang Zhang, Qiong Wu, Yiming Xu, Cheng Cao, Zheng Du, and Konstantinos Psounis. 2024. Efficient toxic content detection by bootstrapping and distill- ing large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 21779–21787

    2024

  72. [81]

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. TinyLlama: An Open-Source Small Language Model. arXiv:2401.02385 [cs.CL]

    2024 arXiv

  73. [82]

    Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. 2023. Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models. ArXiv abs/...

    2023 arXiv

  74. [83]

    Xuandong Zhao, Xianjun Yang, Tianyu Pang, Chao Du, Lei Li, Yu-Xiang Wang, and William Yang Wang. 2024. Weak-to-Strong Jailbreaking on Large Language Models. ArXiv abs/2401.17256 (2024). https://api.semanticscholar.org/CorpusID: 267320277

    2024 arXiv

  75. [84]

    Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Weirong Ye, Neil Zhenqiang Gong, Yue Zhang, and Xingxu Xie. 2023. PromptBench: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts. ArXiv abs/2306.04528 (2023)...

    2023 arXiv

  76. [2023]

    arXiv preprint arXiv:2303.17564 (2023)

    Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564 (2023)

    2023 arXiv

  77. [2024]

    arXiv preprint arXiv:2402.09091 (2024)

    Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Clues. arXiv preprint arXiv:2402.09091 (2024)

    2024 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.