A Flexible Large Language Models Guardrail Development Methodology Applied to Off-Topic Prompt Detection

Gabriel Chua; Shaun Khoo; Shing Yee Chan

REVIEW 3 major objections 5 minor 1 cited by

A Flexible Large Language Models Guardrail Development Methodology Applied to Off-Topic Prompt Detection

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a guardrail for off-topic LLM prompts can be trained entirely on LLM-generated synthetic data, defined qualitatively, and that the same classifier catches jailbreak and harmful prompts when paired with a narrow…

desk verdict A practical data-free recipe for off-topic guardrails with open assets, but the transfer-to-deployment claim is stronger than the evidence. read the letter →

arxiv 2411.12946 v2 pith:AVBG36BE submitted 2024-11-20 cs.CL cs.LG

Gabriel Chua , Shing Yee Chan , Shaun Khoo This is my paper · ORCID

classification cs.CLcs.LG

keywords off-topicpromptdetectionLLMguardrailssyntheticdatagenerationclassificationjailbreakharmfulcontentmoderationbi-encoderclassifierdata-freemethodology

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models deployed in narrow domains will inevitably be asked to do things outside their scope. This paper proposes a way to build a guardrail for that problem before any real user data exists: write a qualitative description of the intended scope and its boundary cases, hand it to a powerful LLM to invent millions of synthetic (system prompt, user prompt) pairs labeled on-topic or off-topic, and fine-tune a small classifier on those pairs. On a synthetic hold-out, the fine-tuned cross-encoder and bi-encoder reach ROC-AUC and F1 around 0.99 and 0.97 respectively, beating cosine-similarity, nearest-neighbor, and prompt-engineering baselines. When the same classifiers are paired with a specialized system prompt, they flag a large fraction of jailbreak and harmful prompts as off-topic, with the bi-encoder reaching recall 0.99 on HarmBench and 0.97 on TrustLLM. The method matters because it gives pre-production teams a deployable safety baseline without collecting sensitive user data, and it comes with open-sourced data and models.

What carries the argument

The mechanism is a three-step pipeline: (1) qualitative problem analysis of the intended use cases and off-topic edge cases, expressed in natural language; (2) LLM-powered synthetic data generation via a carefully crafted meta-prompt that asks for diverse (system, user) pairs with balanced labels, varying domain, style, complexity, language, and random seeds; (3) fine-tuning a small bi-encoder or cross-encoder on these pairs with a standard binary classification head. The crux is that the training signal is the binary label 'is this user prompt off-topic relative to this system prompt,' which forces the model to learn a comparative relevance judgment rather than a topic taxonomy. This formulation is what allows the same guardrail to treat jailbreaks and harmful requests as off-topic when the system prompt defines a narrow, benign scope.

What would settle it

Collect real user prompts from a deployed narrow-domain application, label them for off-topicness with human raters, and measure the fine-tuned bi-encoder's recall and precision; if recall on these real prompts falls well below the 0.95-level seen on synthetic hold-outs (e.g., below 0.8) while precision stays high, the core claim that synthetic data suffices for pre-production guardrails would be disproven.

Watch

Extended reading notes

Core claim

The central claim is that off-topicness, defined as 'user prompt irrelevant to the domain or scope specified in the system prompt,' is a learnable, domain-invariant relation that can be taught entirely with synthetic data. The authors formalize this as a binary function F(S, U) and generate training pairs by prompting GPT-4o with a 'meta-prompt' that encodes the qualitative problem analysis and edge cases (short, vague, multilingual, adversarial prompts). Two lightweight architectures are fine-tuned on roughly 17k such pairs: a cross-encoder (stsb-roberta-base) and a bi-encoder (jina-embeddings-v2-small-en with adapter and attention pooling). On the synthetic hold-out both outperform all baselines, and the bi-encoder transfers strongly to JailbreakBench and HarmBench, indicating that the relation learned is not specific to the synthetic distribution but captures a generalized notion of relevance. The paper's own limitations are acknowledged: synthetic data may carry distributional biases, the method suits well-scoped system prompts, and English-only experiments leave multilingual generalization open.

Load-bearing premise

The whole methodology depends on LLM-generated synthetic prompts being representative enough of real user behavior that a classifier trained on them performs at deployment, an assumption the paper itself flags as potentially fragile.

Editorial extensions

If this is right

Pre-production systems can ship with a guardrail before the first real user query arrives, turning a qualitative spec into a trained filter in hours.
A single guardrail trained on relevance can cover multiple misuse categories—jailbreakers and harmful prompts—by pairing it with a specialized system prompt, reducing the need for category-specific classifiers.
The high throughput (about 2,200 prompt pairs per minute on a T4 GPU) makes the fine-tuned bi-encoder practical as an inline filter for interactive applications.
The released synthetic dataset (2M+ pairs) and model weights give other teams a starting point to create their own domain-specific off-topic detectors without data collection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves implicit that active learning after deployment—mixing a small stream of labeled real queries into the synthetic pool—could directly correct the distributional biases it acknowledges, turning the method into a continuous-improvement loop.
Because the classifier outputs a continuous probability, the same trained model could be reused across multiple applications by adjusting the refusal threshold per domain, though the paper only demonstrates threshold tuning within one domain.
The framing might extend to multi-turn conversations by applying the relevance check per user turn against the accumulated context; the paper does not test this, but the pairwise (system, user) formulation is readily adapted.
If the binding constraint is synthetic-data quality, the method's ceiling is likely set by the generative LLM's ability to imagine realistic off-topic probes; teams with a stronger generator or a small seed set of real edge cases would get proportionally better guardrails.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper does something concrete: a three-step recipe (qualitative scope definition, LLM-generated synthetic (system, user) pairs, fine-tuned lightweight relevance classifiers) and it ships the dataset and models. That is genuinely useful for pre-production guardrail work, and the head-to-head against six baselines is the right kind of comparison. The cross-generator annex (Gemini, Claude, Llama) is a good robustness check. So the core engineering claim—that you can build an off-topic detector with no real user data and beat heuristics—is supported on synthetic holdouts.

The soft spots are mostly about what is not measured. The Table 1 holdout comes from the same generator and meta-prompt as training, so high AUC is partly fit to the generator's labeling function. The annex mitigates this by testing on other generators, but the prompt template stays the same, so style artifacts shared across LLMs could still leak. More importantly, the external generalization numbers are recall-only on harmful sets paired with random narrow system prompts; we don't get precision on benign prompts in that same setting, so we can't judge false-positive costs at deployment. The paper's own Section 5.1 concedes that real-world usage might differ significantly, and the "deployed internally" sentence is not accompanied by an evaluation. That said, the authors are upfront about these gaps and propose active learning as a fix, which is a reasonable path.

The central premise—synthetic data represents real user prompts—is plausible but untested in this paper. I don't think that's a fatal flaw; it's a standard limitation of data-free approaches, and the paper labels it as such. For a workshop paper, this is solid. For a journal or main conference, I'd want error bars, a precision-recall analysis on harmful sets, and at least a small real-user pilot.

Who's it for: practitioners building guardrails for narrow-domain LLM apps, and researchers working on synthetic data for safety. I'd send it to review rather than desk-reject; a serious referee would push for clearer claims about generalization and would ask what a real-deployment evaluation would look like. My own verdict would be conditional: accept as a practical recipe, with the deployment claim softened.

Referee Report

3 major / 5 minor

Summary. The paper proposes a three-step, 'data-free' methodology for building LLM off-topic guardrails: (1) qualitatively define the problem scope and edge cases, (2) use an LLM (GPT-4o) to generate a large synthetic dataset of (system prompt, user prompt) pairs with on/off-topic labels, and (3) fine-tune a lightweight bi-encoder or cross-encoder classifier on this synthetic data. The authors report that their fine-tuned classifiers achieve ROC-AUC 0.99 and F1 0.99 on a held-out synthetic set, outperform heuristic baselines, and generalize to jailbreak and harmful prompt benchmarks (e.g., recall 0.99 on HarmBench and 0.97 on TrustLLM for the bi-encoder). They also release the synthetic dataset and trained models. The central claim is that this procedure enables deployment-ready guardrails before real user data is available.

Significance. If the central claim were established, the methodology would be practically valuable for pre-production guardrail development, and the open-sourced dataset and model weights are useful concrete contributions to the LLM-safety community. The paper is also transparent about several limitations, including synthetic-data bias and the challenge of open-ended system prompts, which is commendable. However, the evidence as presented does not yet establish transfer to real user prompts, because the main evaluation is on synthetic holdout data generated with the same prompt template and generator, and the external benchmarks use a protocol that makes the detection task artificially easy. The deployment claim in Section 6 is asserted without supporting data.

major comments (3)

[§4.4, Table 1; Annex Tables 5–6] The central claim that the methodology produces deployable guardrails rests on the representativeness of LLM-generated synthetic data. The primary evaluation is a held-out set drawn from the same GPT-4o generation process and the same meta-prompt used to create the training data, so Table 1 largely measures how well the classifier fits the generator's labeling function. The annex replaces the generator LLM (Gemini Pro 1.5, Claude 3.5 Sonnet, Llama 3.1 405B) but retains the same prompt template and qualitative domain definition, so shared style artifacts and coverage gaps across LLMs are not exposed. Section 5.1 concedes that 'real-world usage might differ significantly.' No evaluation on genuine user prompts is provided, despite Section 6 stating that the guardrails have been deployed internally since September 2024. This is a load-bearing omission for the 'deployable' claim and should be addressed, for example by adding a real-user evaluation from the internal deployment or by substantially softening the deployment claim.
[§4.5, Tables 2–3] The external generalization evaluation pairs harmful and jailbreak prompts with random specialized system prompts. A narrow system prompt (for example, a healthcare policy Q&A bot) makes almost any harmful request trivially off-topic by construction, so high recall on HarmBench, TrustLLM, and the localized harmful dataset does not demonstrate that the model learns a robust, general notion of relevance. Table 3 reports recall only; without precision or a balanced negative set of benign prompts under the same pairing protocol, the reader cannot assess whether the guardrail would over-block legitimate user requests. To support the generalization claim, the authors should report precision on a balanced sample and evaluate on benign user prompts paired with the same specialized system prompts.
[§4.4, Table 1] No confidence intervals, error bars, or significance tests are reported for any metric. The differences between the fine-tuned models and the baselines on the synthetic holdout are large, which mitigates this concern, but a journal-level revision should quantify variability across random seeds and, ideally, across independent synthetic data generation runs. This is necessary to support the comparative claim that the fine-tuned models 'outperform' the baselines beyond a single run.

minor comments (5)

[§1, last paragraph] There is a typo: 'repsectively' should be 'respectively'.
[§3.1, Step 2] The term 'data-free' is used, but the pipeline includes 'light heuristics or human verification' to clean the synthetic data; please clarify that 'data-free' means no real user data, not no human effort or curation.
[§4.1] Minor formatting: 'GPT 4o' should be 'GPT-4o', and the claim of 'more than 2M pairs' is not further disaggregated; please state how many examples were used for training versus validation after filtering.
[References] Several reference entries are malformed, e.g., entries beginning with 'et al., 2024b' and 'Gemini Team et al.' should follow a consistent author-name style. Please harmonize with the venue's bibliography format.
[§4.4, Figure 4] The calibration plot is described as 'near-diagonal,' but no quantitative calibration metric (e.g., expected calibration error) is reported; adding such a number would make the claim more precise.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the core training/evaluation loop is conventional supervised learning on synthetic data, and the one author-overlapping benchmark is not load-bearing.

full rationale

The paper's derivation chain is: (1) define off-topic qualitatively, (2) use GPT-4o to generate (system prompt, user prompt, label) triples, (3) train bi-encoder and cross-encoder classifiers on a subset, (4) evaluate on a held-out subset from the same generator, and (5) evaluate on external benchmarks. Step (4) is an in-distribution holdout, not a circular prediction: the classifier never sees holdout labels, and the claim is only that it outperforms baselines on synthetic data. The paper explicitly concedes the representativeness limitation in Section 5.1: 'LLMs may introduce distributional biases or style artifacts when generating synthetic data. Although we mitigate this via temperature tuning and random seed words, real-world usage might differ significantly,' which frames the synthetic holdout as a proxy rather than a disguised restatement of the training objective. The external evaluations use JailbreakBench, HarmBench, and TrustLLM, which are independent of the authors, and the classifier was not fitted to them. The only author-overlapping item is the 'Localized Harmful dataset (Foo & Khoo, 2025)' in Table 3, contributed by co-author Shaun Khoo; because the same table reports independent HarmBench and TrustLLM results and the paper's headline generalization claim does not rest on this dataset, the self-citation is not load-bearing. No equation reduces a predicted quantity to a fitted input, and no uniqueness theorem or ansatz is imported from the authors' prior work. The methodological weakness is external validity, not circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The method rests on the assumption that LLM-generated synthetic data is representative enough of real-world prompts, and that off-topic is a learnable relevance relation between system and user prompts. These are domain assumptions acknowledged by the authors as limitations rather than independently verified facts. No new physical entities are introduced; the only user-chosen numbers affecting reported metrics are the classification threshold and generation diversity settings, both discussed as deployment or data-creation choices.

free parameters (2)

classification threshold t = not reported (default 0.5 assumed; internal deployments use 0.4 to 0.6)
The reported precision, recall, and F1 values in Tables 1 and 2 depend on the threshold used to binarize the off-topic probability. The paper does not state the threshold for Table 1, so the exact operating point is a user-chosen value rather than a fitted parameter.
LLM generation temperature and top-k = not specified
Selected by hand to increase prompt diversity during synthetic data generation; these settings shape the training distribution and therefore influence results, but they are not fitted to any target metric.

assumptions (3)

domain assumption LLM-generated synthetic examples are representative of real-world user prompts for specialized LLM applications.
The entire methodology replaces real data with generated data; Section 5.1 acknowledges that 'LLMs may introduce distributional biases or style artifacts' and 'real-world usage might differ significantly.' This is load-bearing but unverified.
domain assumption Off-topic is well-defined by relevance between system prompt and user prompt, and this relation is learnable from paired examples.
Section 3.1 defines off-topic as 'Any user prompt that is irrelevant to the domain or scope specified in the system prompt.' The entire classification task depends on this framing being both meaningful and learnable.
domain assumption Harmful and jailbreak prompts, when paired with a narrow specialized system prompt, behave like off-topic prompts.
Section 4.1 states that from the standpoint of a specialized system prompt with a narrow domain, these prompts are 'effectively off-topic or disallowed.' This assumption underlies the generalization results in Tables 2 and 3.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Flexible Large Language Models Guardrail Development Methodology Applied to Off-Topic Prompt Detection." pith.science (2026). https://pith.science/paper/AVBG36BE

@misc{pith2026241112946,
  author       = {Pith},
  title        = {Pith review of: A Flexible Large Language Models Guardrail Development Methodology Applied to Off-Topic Prompt Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AVBG36BE}},
  note         = {Machine review of arXiv:2411.12946}
}

read the original abstract

Large Language Models (LLMs) are prone to off-topic misuse, where users may prompt these models to perform tasks beyond their intended scope. Current guardrails, which often rely on curated examples or custom classifiers, suffer from high false-positive rates, limited adaptability, and the impracticality of requiring real-world data that is not available in pre-production. In this paper, we introduce a flexible, data-free guardrail development methodology that addresses these challenges. By thoroughly defining the problem space qualitatively and passing this to an LLM to generate diverse prompts, we construct a synthetic dataset to benchmark and train off-topic guardrails that outperform heuristic approaches. Additionally, by framing the task as classifying whether the user prompt is relevant with respect to the system prompt, our guardrails effectively generalize to other misuse categories, including jailbreak and harmful prompts. Lastly, we further contribute to the field by open-sourcing both the synthetic dataset and the off-topic guardrail models, providing valuable resources for developing guardrails in pre-production environments and supporting future research and development in LLM safety.

Figures

Figures reproduced from arXiv: 2411.12946 by the authors.

**Figure 2.** Our Guardrail Development Methodology 3 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Summary of the two modeling approaches for off-topic prompt detection [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Calibration Plot on the Synthetic Hold-Out Set. A near-diagonal plot indicates good [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

kNNGuard: Turning LLM Hidden Activations into a Training-Free Configurable Guardrail
cs.LG 2026-07 unverdicted novelty 6.0 of 10

Multi-layer Fisher-weighted kNN over frozen-LLM activations, fused with embedding kNN, yields competitive F1 guardrails from a 50-example bank with no fine-tuning and sub-10-second domain adaptation.

Reference graph

Works this paper leans on

31 extracted references · 6 canonical work pages · cited by 1 Pith paper

[1]

Block denied topics to help remove harmful content - amazon bedrock --- docs.aws.amazon.com

AWS. Block denied topics to help remove harmful content - amazon bedrock --- docs.aws.amazon.com. https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-denied-topics.html. [Accessed 12-11-2024]

work page 2024
[2]

Custom categories in azure ai content safety --- learn.microsoft.com

Azure. Custom categories in azure ai content safety --- learn.microsoft.com. https://learn.microsoft.com/en-us/azure/ai-services/content-safety/concepts/custom-categories?tabs=standard. [Accessed 12-11-2024]

work page 2024
[3]

How knowledge workers use and want to use llms in an enterprise context

Michelle Brachman, Amina El-Ashry, Casey Dugan, and Werner Geyer. How knowledge workers use and want to use llms in an enterprise context. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, CHI EA '24, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400703317. doi:10.1145/3613905.3650841. URL https://...

arXiv 2024
[4]

Pappas, Florian Tramer, Hamed Hassani, and Eric Wong

Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramer, Hamed Hassani, and Eric Wong. Jailbreakbench: An open robustness benchmark for jailbreaking large language models, 2024. URL https://arxiv.org/abs/2404.01318

arXiv 2024
[5]

Deep reinforcement learning from human preferences

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings...

work page 2017
[6]

Safeguarding large language models: A survey, 2024

Yi Dong, Ronghui Mu, Yanghao Zhang, Siqi Sun, Tianle Zhang, Changshun Wu, Gaojie Jin, Yi Qi, Jinwei Hu, Jie Meng, Saddek Bensalem, and Xiaowei Huang. Safeguarding large language models: A survey, 2024. URL https://arxiv.org/abs/2406.02622

arXiv 2024
[7]

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024 a

Gemini Team et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024 a . URL https://arxiv.org/abs/2403.05530

arXiv 2024
[8]

Gpt-4o system card, 2024 b

OpenAI et al. Gpt-4o system card, 2024 b . URL https://arxiv.org/abs/2410.21276

arXiv 2024

Show all 31 references

[9]

Trustllm: Trustworthiness in large language models, 2024 c

Yue Huang et al. Trustllm: Trustworthiness in large language models, 2024 c . URL https://arxiv.org/abs/2401.05561

2024 arXiv
[10]

L ion G uard: A contextualized moderation classifier to tackle localized unsafe content

Jessica Foo and Shaun Khoo. L ion G uard: A contextualized moderation classifier to tackle localized unsafe content. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, Steven Schockaert, Kareem Darwish, and Apoorv Agarwal (eds.), Proceedings ...

2025
[11]

Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned, 2022

Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfi...

2022 arXiv
[12]

Jina embeddings 2: 8192-token general-purpose text embeddings for long documents, 2024

Michael Günther, Jackmin Ong, Isabelle Mohr, Alaeddine Abdessalem, Tanguy Abel, Mohammad Kalim Akram, Susana Guzman, Georgios Mastrapas, Saba Sturua, Bo Wang, Maximilian Werk, Nan Wang, and Han Xiao. Jina embeddings 2: 8192-token general-purpose text embeddings for long docume...

2024 arXiv
[13]

Llama guard: Llm-based input-output safeguard for human-ai conversations, 2023

Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama guard: Llm-based input-output safeguard for human-ai conversations, 2023. URL https://arxiv.org/abs/2312.06674

2023 arXiv
[14]

Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, and Andrew M. Dai. Best practices and lessons learned on synthetic data, 2024. URL https://arxiv.org/abs/2404.07503

2024 arXiv
[15]

Llama Team

AI@Meta et al. Llama Team. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

2024 arXiv
[16]

On llms-driven synthetic data generation, curation, and evaluation: A survey, 2024

Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. On llms-driven synthetic data generation, curation, and evaluation: A survey, 2024. URL https://arxiv.org/abs/2406.15126

2024 arXiv
[17]

Chan, Jakob Foerster, Mihaela van der Schaar, and Robert Tjarko Lange

Chris Lu, Samuel Holt, Claudio Fanconi, Alex J. Chan, Jakob Foerster, Mihaela van der Schaar, and Robert Tjarko Lange. Discovering preference optimization algorithms with and for large language models, 2024. URL https://arxiv.org/abs/2406.08414

2024 arXiv
[18]

Harmbench: A standardized evaluation framework for automated red teaming and robust refusal, 2024

Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal, 2024. URL https://arxiv.org...

2024 arXiv
[19]

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

2024
[20]

Manning, and Chelsea Finn

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: your language model is secretly a reward model. In Proceedings of the 37th International Conference on Neural Information Processing Systems,...

2024
[21]

Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails, 2023

Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, and Jonathan Cohen. Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails, 2023. URL https://arxiv.org/abs/2310.10501

2023 arXiv
[22]

Colbertv2: Effective and efficient retrieval via lightweight late interaction, 2022

Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. Colbertv2: Effective and efficient retrieval via lightweight late interaction, 2022. URL https://arxiv.org/abs/2112.01488

2022 arXiv
[23]

Mrinank Sharma, Meg Tong, Jesse Mu, Jerry Wei, Jorrit Kruthoff, Scott Goodfriend, Euan Ong, Alwin Peng, Raj Agarwal, Cem Anil, Amanda Askell, Nathan Bailey, Joe Benton, Emma Bluemke, Samuel R. Bowman, Eric Christiansen, Hoagy Cunningham, Andy Dau, Anjali Gopal, Rob Gilson, Log...

2025 arXiv
[24]

do anything now

Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. "do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models, 2024. URL https://arxiv.org/abs/2308.03825

2024 arXiv
[25]

Smith, Daniel Khashabi, and Hannaneh Hajishirzi

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions, 2023. URL https://arxiv.org/abs/2212.10560

2023 arXiv
[26]

C-pack: Packed resources for general chinese embeddings, 2024

Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. C-pack: Packed resources for general chinese embeddings, 2024. URL https://arxiv.org/abs/2309.07597

2024 arXiv
[27]

Ho, Carl Yang, and Qi He

Ran Xu, Hui Liu, Sreyashi Nag, Zhenwei Dai, Yaochen Xie, Xianfeng Tang, Chen Luo, Yang Li, Joyce C. Ho, Carl Yang, and Qi He. Simrag: Self-improving retrieval-augmented generation for adapting large language models to specialized domains, 2024. URL https://arxiv.org/abs/2410.17952

2024 arXiv
[28]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
[29]

@esa (Ref

\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
[30]

\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
[31]

5 ZE ],`CWE ;͎Y ł. ʛ u_|?gsܙ q# 壚 aA1itR7 c s 366 2 Y IZx| L[!JH\ Q[N. 0@G x g+p g*>Mb< T8 q6W O/fC >]< : y g@l mD ?dM3sH ȋJ@

@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

1999

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Block denied topics to help remove harmful content - amazon bedrock --- docs.aws.amazon.com

AWS. Block denied topics to help remove harmful content - amazon bedrock --- docs.aws.amazon.com. https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-denied-topics.html. [Accessed 12-11-2024]

work page 2024

[2] [2]

Custom categories in azure ai content safety --- learn.microsoft.com

Azure. Custom categories in azure ai content safety --- learn.microsoft.com. https://learn.microsoft.com/en-us/azure/ai-services/content-safety/concepts/custom-categories?tabs=standard. [Accessed 12-11-2024]

work page 2024

[3] [3]

How knowledge workers use and want to use llms in an enterprise context

Michelle Brachman, Amina El-Ashry, Casey Dugan, and Werner Geyer. How knowledge workers use and want to use llms in an enterprise context. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, CHI EA '24, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400703317. doi:10.1145/3613905.3650841. URL https://...

arXiv 2024

[4] [4]

Pappas, Florian Tramer, Hamed Hassani, and Eric Wong

Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramer, Hamed Hassani, and Eric Wong. Jailbreakbench: An open robustness benchmark for jailbreaking large language models, 2024. URL https://arxiv.org/abs/2404.01318

arXiv 2024

[5] [5]

Deep reinforcement learning from human preferences

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings...

work page 2017

[6] [6]

Safeguarding large language models: A survey, 2024

Yi Dong, Ronghui Mu, Yanghao Zhang, Siqi Sun, Tianle Zhang, Changshun Wu, Gaojie Jin, Yi Qi, Jinwei Hu, Jie Meng, Saddek Bensalem, and Xiaowei Huang. Safeguarding large language models: A survey, 2024. URL https://arxiv.org/abs/2406.02622

arXiv 2024

[7] [7]

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024 a

Gemini Team et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024 a . URL https://arxiv.org/abs/2403.05530

arXiv 2024

[8] [8]

Gpt-4o system card, 2024 b

OpenAI et al. Gpt-4o system card, 2024 b . URL https://arxiv.org/abs/2410.21276

arXiv 2024