Pith. sign in

REVIEW 4 major objections 5 minor 25 references

When Meaning Stays the Same, but Models Drift: Evaluating Quality of Service under Token-Level Behavioral Instability in LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLMs drift under rewording even when meaning is fixed.

desk verdict A useful descriptive sweep undermined by a missing control: the claim that PBSS drift is 'not noise' is unsupported because no same-prompt repeated-generation baseline exists. read the letter →

arxiv 2506.10095 v1 pith:XKORP3UC submitted 2025-06-11 cs.CL

Xiao Li , Joel Kreuzwieser , Alan Peters This is my paper
classification cs.CL
keywords promptvariancebehavioraldriftsemanticembeddingsqualityofserviceLLMevaluationstabilitytoken-levelsensitivityinstruction-tunedmodelsparaphraserobustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that large language models systematically change their behavior when a prompt is reworded in ways that keep its meaning identical—what it calls prompt variance—and that this drift is structured and model-specific rather than random noise. It proposes a diagnostic, Prompt-Based Semantic Shift (PBSS), that measures how far apart the model's answers to semantically equivalent prompts are in embedding space. Across ten tasks, five to ten models, and three embedding encoders, the paper finds a sharp divide: instruction-tuned models stay stable under rephrasing, while older smaller models drift widely. If true, this gives deployers a cheap, black-box way to screen models for behavioral reliability before putting them in high-stakes settings such as clinical or legal workflows, where a rephrased question could change the answer.

What carries the argument

The load-bearing object is Prompt-Based Semantic Shift (PBSS), defined as the cosine distance between sentence-embedding vectors of a model's outputs to two paraphrased prompts that share intent. Pairwise PBSS scores are assembled into a drift matrix and a cumulative distribution function, with z-score heatmaps isolating outlier prompt variants; the steepness and location of the CDF becomes the model's stability fingerprint. The mechanism that carries the argument is the comparison of these fingerprints across models, temperatures, tasks, and three independent embedding encoders, which the paper argues isolates model-internal sensitivity to token-level surface form.

What would settle it

Run each of the 15 paraphrase prompts through the same model many times at the same temperature and compute the PBSS distance between repeated outputs of the identical prompt; if that repeat-distance distribution overlaps the between-paraphrase distance distribution, the reported drift could be sampling noise rather than prompt-induced sensitivity.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that surface-level rephrasing—different token sequences that preserve intent—produces measurable, consistent response shifts in LLMs, and the size of that shift is a stable property of the model, not of the prompt set or the embedding encoder. Grouping models by parameter scale and alignment stage, PBSS cumulative distributions separate cleanly into behavioral tiers: GPT-3.5, LLaMA-2, and Mistral cluster tightly with low drift, while GPT-2 and GPT-Neo show broad dispersion, a divide the paper calls a phase boundary between alignment-aware and legacy architectures. The claim that this is structure rather than noise rests on three consistency observations: the same model rankings appear across three different sentence-embedding encoders, the CDF shapes survive the change in decoding temperature from 0.2 to 1.3, and Kruskal–Wallis tests return highly significant differences between model tiers.

Load-bearing premise

The paper's structural claims stand on the assumption that the measured output divergence is caused by the prompt paraphrase and not by the model's own random sampling, since identical prompts are never repeated to measure a noise floor.

Editorial extensions

If this is right

  • PBSS can act as a pre-deployment quality-of-service screen: a model whose PBSS drift is high on benign rephrasings can be flagged before it reaches clinical, legal, or financial workflows.
  • Model rankings by drift are consistent across encoder choices and temperatures, so the diagnostic does not depend on a particular embedding model to expose the behavioral tiers.
  • The observed phase boundary implies that instruction-tuned models have acquired rhetorical consistency as a side effect of alignment, making drift resistance a property that could be tracked over successive fine-tuning generations.
  • Because jailbreaking is reframed as an extreme case of prompt-induced drift, high-PBSS prompt regions become candidate early-warning zones for jailbreak susceptibility, a direction the paper explicitly opens.
  • For safety-sensitive users, the method gives a concrete number—mean pairwise drift per model—that can be monitored over model updates to detect regressions in behavioral stability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 'phase boundary' may partly be a decoding-entropy effect: the legacy small models in the sample are also highly stochastic samplers at temperature 1.3, so a same-prompt repetition baseline would be needed to separate prompt sensitivity from sampling variance; building that baseline is a natural next experiment.
  • If the drift rankings are stable across embedding encoders, the same experimental design could be used as a cheap calibration probe for testing whether alignment techniques monotonically reduce prompt sensitivity, without needing downstream task labels.
  • A testable extension: check whether PBSS drift on benign rephrasing predicts refusal-rate variance or jailbreak success on the same prompts; a positive correlation would turn the diagnostic into a security screening tool.
  • The paper's encoder-agnostic 'semantic resonance' suggests PBSS might measure something about the model's hidden representations rather than its output distribution; probing internal activations for the same paraphrases could reveal whether the drift originates in tokenization or in later decoding decisions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript introduces Prompt-Based Semantic Shift (PBSS), a cosine-distance measure between SBERT embeddings of a model's outputs to semantically equivalent paraphrased prompts. The authors evaluate five to ten LLMs on ten constrained tasks with fifteen prompt variants per prompt set at two temperatures, and report model-specific drift patterns, a behavioral phase boundary between instruction-tuned and legacy models, and robustness across three SBERT encoders. They propose PBSS as a lightweight QoS screening diagnostic for high-stakes deployment.

Significance. If the central claim holds, PBSS would be a cheap, black-box diagnostic for surface-form sensitivity, with potential value for clinical and legal deployment screening. The paper has strengths: it releases code, prompts, and outputs; it reports Kruskal-Wallis statistics across three encoders and two temperatures; and the descriptive ordering (instruction-tuned models more stable than legacy models under rephrasing) is plausible from the CDFs. However, the signature inference that measured drift is 'not noise' (Section 5.9) currently lacks a same-prompt repeated-generation control, so the paper's main contribution is not yet established.

major comments (4)
  1. [§5.9 and Appendix B.1] The claim that 'this drift is not noise' is not supported by the experimental design. PBSS compares two different prompts, each sampled once per temperature; at T=1.3, sampling noise alone can shift sentence embeddings substantially, and the paper reports no same-prompt repeated-generation baseline anywhere. Without such a baseline, the observed cross-prompt divergence cannot be separated from decoding stochasticity, and the small-vs-large model ordering is confounded with sampling entropy and response stereotypy. Please add a condition where the same prompt is generated multiple times at T=1.3 and compare the within-prompt embedding distance to the cross-prompt PBSS distribution.
  2. [§5.5 and §5.7] The 'behavioral phase boundary' between alignment-aware and legacy architectures is not established because model scale and alignment stage are fully confounded. GPT-3.5 is both the largest and most heavily instruction-tuned model, while GPT-2 and GPT-Neo are both the smallest and least aligned. The model tiers in Table 1 are also assigned post hoc after observing the CDF ordering, and the cutoffs (e.g., SmolLM-360M as Small, Phi-2 2.7B as Medium, MythoMax-13B as Large) appear arbitrary. A matched comparison—same architecture with and without instruction tuning, or a regression with parameter count as a covariate—is needed before the drift patterns can be attributed to alignment stage rather than scale.
  3. [§4.2] The semantic-equivalence validation uses all-mpnet-base-v2, which is also one of the three encoders used to compute PBSS. Because prompt variants are selected or filtered for high similarity under this encoder, the measured drift for that encoder is not independent of the selection criterion. The authors should repeat the prompt validation with an independent embedding model or with human ratings, and confirm that the PBSS findings survive when the validation encoder is excluded from the analysis.
  4. [§3.2 and Appendix A] The interpretation of PBSS as capturing 'tone, structure, or rhetorical differences' is asserted rather than validated. Cosine distance in SBERT space is sensitive to many output properties, including length, lexical overlap, and topic. The paper provides no evidence that the measured distances align with the claimed behavioral axis, nor any comparison of PBSS to lexical diversity measures, reference-free text similarity metrics, or human judgments of rhetorical stability. Adding such a validation would strengthen the claim that PBSS measures behavioral drift rather than generic output variability.
minor comments (5)
  1. [§5.9] The statement 'The 750 unique instructions per model yield 300 distinct prompt pairs' is numerically inconsistent with the stated design: 50 prompt sets (10 tasks x 5 origins) each contribute C(15,2)=105 pairs, giving 5,250 pairs. Please clarify the counting or correct the number.
  2. [Figures 5, 6, 25, 26] Several figure captions contain the typo 'T emp' instead of 'Temp', and the manuscript inconsistently uses 'S-BERT', 'SBERT', and 'S-BERT variants'. Please standardize notation throughout.
  3. [Appendix C] The main text focuses on five core models while Appendix C introduces five additional models and 'alt case' figures; please clarify in the main text which analyses are primary and which are supplementary, and unify the model naming (e.g., 'Mistral-7B-instruct-v0.2' vs 'Mistral-7B-Instruct-v0.2').
  4. [Table 3] The Kruskal-Wallis results report H statistics and p-values but no effect sizes; for the extremely large H values, a rank-based effect size such as epsilon-squared would help readers judge the practical magnitude of the between-group differences.
  5. [§6.2] The sentence 'This has significant implications...' overstates the direct clinical utility of an embedding-distance diagnostic; please temper the claim or provide a specific validation scenario in a clinical NLP setting.

Circularity Check

1 steps flagged · score 2.0 of 10

No derivation-equals-input circularity: PBSS is computed, not fitted, and no self-citation chain is load-bearing. The only partly circular step is the tier-based Kruskal-Wallis test, whose 'empirically motivated' groups are described by the same drift scores they are used to confirm.

  1. fitted input called prediction [Section 5.7-5.8 (model tier taxonomy and Table 3)]
    "We organize models into three empirically motivated groups: Legacy Small Models (e.g., GPT-2, GPT-Neo-1.3B, SmolLM) — pre-alignment or lightly tuned models with sub-2B parameter counts, showing high variance and poor consistency under paraphrastic perturbation ... Kruskal–Wallis tests confirm statistically significant divergence between these groups."

    The three tiers are 'empirically motivated' and are defined in the same paragraphs by the very PBSS outcome later tested: Legacy Small models are those 'showing high variance and poor consistency under paraphrastic perturbation,' Mid-Sized systems show 'intermediate drift behaviors,' and instruction-tuned models show 'strong consistency across rewordings.' Running Kruskal-Wallis on PBSS scores grouped by these tiers and presenting 'statistically significant divergence' as confirmation of a phase boundary partially re-detects the classification criterion, since both the group descriptions and the test statistic draw on the same drift distribution.

full rationale

This paper is a measurement study rather than a derivation: Appendix A defines PBSS as a cosine distance between output embeddings, and no parameter is fitted to data and then renamed a prediction. There are no self-citations and no imported uniqueness theorems, so the self-citation circularity patterns do not apply. The central claims, model-specific CDF ordering, t-SNE clustering by model, and 'semantic resonance' across three SBERT encoders, are computed quantities and are not equivalent to their inputs by construction. Two design features temper independence but do not rise to definitional circularity. First, the tier-based significance testing (Sections 5.7-5.8) is partially self-referential: the 'empirically motivated' tiers are described by the drift they exhibit, so the Kruskal-Wallis confirmation partly validates the grouping criterion; this is mitigated because groups follow parameter scale and alignment stage and the all-models test is tier-free. Second, prompt-equivalence validation uses all-mpnet-base-v2 (Section 4.2), one of the three drift encoders, so the claimed cross-encoder 'semantic resonance' is partially built in for one of three encoders; however, two encoders are validation-independent and the drift score is computed on outputs, not defined as the validation similarity, making this a robustness confound rather than a reduction. Separately, Section 5.9's 'this drift is not noise' claim lacks a same-prompt repeated-generation baseline (Appendix B.1 samples each prompt once per temperature), so decoding stochasticity is not excluded; under the stated review rules that is a correctness and validity gap to be assessed against external controls, not a circularity. Score 2 reflects one partial, non-load-bearing circular step in supporting analysis, while the core PBSS measurement retains independent content.

Assumptions & free parameters 4 free parameters · 5 assumptions · 3 invented entities

No fitted constants appear in the PBSS definition itself; the load-bearing choices are the 0.5 semantic-equivalence cutoff, the unused hybrid weight lambda, the post hoc model tiers, and the two hand-chosen temperatures. The SBERT encoder family is the measurement instrument, so encoder choice is not an independent benchmark. The invented entities are interpretive labels and a definitional metric, none with independent falsifiable handles.

free parameters (4)
  • Semantic-equivalence sanity threshold = 0.5
    The pipeline diagram (Figure 2) applies 'S-BERT similarity > 0.5 (ranked)' as the sanity check that declares prompt variants semantically equivalent. The cutoff and the encoder family that computes it shape which prompts enter the drift measurement.
  • Hybrid PBSS weighting lambda = not set, hybrid score never used
    Appendix A introduces PBSShybrid = lambda * Sim_sem + (1 - lambda) * PBSS as an optional formulation. If instantiated, lambda is a free trade-off parameter; it is not used in any reported result.
  • Model tier cutoffs for Small, Mid, Large groups = post hoc, roughly <2B, 2B to 13B, >=13B
    Sections 5.7 and 5.8 define the tiers after inspecting PBSS distributions, and the text says the taxonomy 'recapitulates clear behavioral boundaries observed in PBSS distributions'. Significance tests in Table 3 then run on those same tiers.
  • Decoding temperatures = 0.2 and 1.3
    Two hand-chosen sampling conditions. All drift statistics are conditional on these temperatures, and no other sampling configurations are reported.
assumptions (5)
  • domain assumption Cosine distance in SBERT embedding space is a valid proxy for behavioral instability and quality-of-service inconsistency.
    PBSS is defined as 1 minus cosine(s(y_i), s(y_j)) in Appendix A and is described as detecting tone, structure, and rhetorical differences. No evidence ties embedding-space distance to the QoS construct the paper claims to measure.
  • domain assumption The 15 prompt variants per set are semantically equivalent and differ only at the token level.
    Section 4.2 validates equivalence with manual review, rule-based filtering, and a semantic threshold computed by all-mpnet-base-v2, the same SBERT family later used to measure drift. The equivalence claim is therefore partly built on the measurement instrument.
  • domain assumption Observed output divergence is caused by the prompt paraphrase and not by decoding stochasticity.
    Section 5.9 asserts 'this drift is not noise' without any same-prompt repeated-generation baseline. At T=1.3, sampling alone produces large output differences, so the assertion is unverified.
  • ad hoc to paper Between-model drift differences are attributable to tokenization and decoding dynamics rather than to model scale, training data, or alignment stage.
    The abstract attributes drift to tokenization and decoding, but no tokenizer or decoding variable is manipulated directly. The models differ jointly in scale, data, and alignment, so the design cannot separate these causes.
  • standard math Kruskal-Wallis H-test is an appropriate test for comparing PBSS score distributions across model groups.
    Table 3 uses the test across tiers. The test itself is standard, though the post hoc tier definition in Section 5.7 compromises the inference.
invented entities (3)
  • PBSS drift metric
    purpose: One minus cosine distance between SBERT embeddings of model outputs to paired paraphrased prompts; used as the QoS diagnostic centerpiece.
    A definitional score, not a falsifiable entity. Its validity depends entirely on the chosen encoder family, and no external benchmark shows drift scores predict deployment-relevant outcomes such as task failure or safety incidents.
  • Semantic resonance
    purpose: Name for the observation that model rankings are consistent across the three SBERT encoders.
    Coined in Section 5.8 to describe encoder agreement. The three encoders share the same contrastive sentence-embedding paradigm, so agreement is expected and provides no independent handle outside this paper.
  • Behavioral phase boundary
    purpose: Label for the CDF discontinuity separating instruction-tuned from legacy models.
    Section 5.5 reads the CDF shapes as a phase shift. The label is an interpretation of the same data it purports to explain and is not tied to any mechanistic or independent measurement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When Meaning Stays the Same, but Models Drift: Evaluating Quality of Service under Token-Level Behavioral Instability in LLMs." pith.science (2026). https://pith.science/paper/XKORP3UC

@misc{pith2026250610095,
  author       = {Pith},
  title        = {Pith review of: When Meaning Stays the Same, but Models Drift: Evaluating Quality of Service under Token-Level Behavioral Instability in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XKORP3UC}},
  note         = {Machine review of arXiv:2506.10095}
}
read the original abstract

We investigate how large language models respond to prompts that differ only in their token-level realization but preserve the same semantic intent, a phenomenon we call prompt variance. We propose Prompt-Based Semantic Shift (PBSS), a diagnostic framework for measuring behavioral drift in LLMs under semantically equivalent prompt rewordings. Applied to ten constrained tasks, PBSS reveals consistent, model-specific response shifts, suggesting statistical regularities linked to tokenization and decoding. These results highlight an overlooked dimension of model evaluation stability under rephrasing and suggest that tokenization strategies and decoding dynamics may contribute to post-training quality of service instability.

Figures

Figures reproduced from arXiv: 2506.10095 by the authors.

Figure 1
Figure 1. PBSS heatmap showing output divergence across five semantically equivalent prompts. Red = less semantic similarity. See Section 5. The same model may hedge, speculate, or add cautionary notes absent from the original. Al￾though the task remains constant, subtle shifts in tone, emphasis, and even factual framing emerge. These inconsistencies raise concerns not only for trust and alignment, but for safety—especially i… view at source ↗
Figure 2
Figure 2. Overview of our experimental framework. Structured prompt sets are constructed across ten tasks, undergo [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 5
Figure 5. PBSS CDF for MiniLM-L6 under T = 0.2. 0.3 0.4 0.5 0.6 0.7 PBSS 0.0 0.2 0.4 0.6 0.8 1.0 Cumulative % PBSS Cumulative Distribution (Temperature=1.3) gpt2-large gpt-neo-1.3B gpt-3.5-turbo Llama-2-7b-chat-hf Mistral-7B-instruct-v0.2 [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Origin-space t-SNE (10 tasks, 5 models) - 1 [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 7
Figure 7. Figure 7: MiniLM-L6: Shows diver￾gent structure and weaker clustering. V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12 V13 V14 V15 V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12 V13 V14 V15 0.00 -0.84 -0.98 -0.04 -0.91 -0.62 -1.33 -1.58 0.63 -1.29 -1.21 -1.17 0.30 0.74 1.24 -0.84 0.00 -1.46 0.36…
Figure 10
Figure 10. Figure 10: S-BERT-based similarity distribution for prompt variants across 10 tasks. Each prompt set per task [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Each point reflects a prompt variant, plotted by S-BERT similarity (x-axis) and shallow syntax distance [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Model-space t-SNE (3 tasks, 5 models) 20 10 0 10 20 30 40 tsne_x 40 30 20 10 0 10 20 30 tsne_y Model Semantic Behavior Map (T-SNE) origin1 origin2 origin3 [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 14
Figure 14. Figure 14: Model-space t-SNE (10 tasks, 5 new models) [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 16
Figure 16. Figure 16: PBSS CDF for MiniLM￾L6: 3 tasks. 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 PBSS 0.0 0.2 0.4 0.6 0.8 1.0 Cumulative % PBSS Cumulative Distribution (CDF) gpt2-large gpt-neo-1.3B gpt-3.5-turbo Llama-2-7b-chat-hf Mistral-7B-Instruct-v0.2 [PITH_FULL_IMAGE:figures/full_fig_p017_16.png]
Figure 19
Figure 19. Figure 19: PBSS CDF for MiniLM￾L6: 10 tasks. 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 PBSS 0.0 0.2 0.4 0.6 0.8 1.0 Cumulative % PBSS Cumulative Distribution (CDF) gpt2-large EleutherAI_gpt-neo-1.3B gpt-3.5-turbo meta-llama_Llama-2-7b-chat-hf Mistral-7B-Instruct-v0.2 [PITH_FULL_IMAGE…
Figure 22
Figure 22. Figure 22: PBSS CDF for MiniLM￾L6 (alt case). 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 PBSS 0.0 0.2 0.4 0.6 0.8 1.0 Cumulative % PBSS Cumulative Distribution (CDF) gpt2-large gpt-neo-1.3B gpt-3.5-turbo Llama-2-7b-chat-hf Mistral-7B-instruct-v0.2 Hugging-Face_SmolLM-360M Microsoft_phi…
Figure 25
Figure 25. Figure 25: PBSS CDF for MiniLM-L6, T = 0.2. 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 PBSS 0.0 0.2 0.4 0.6 0.8 1.0 Cumulative % PBSS Cumulative Distribution (Temperature=1.3) gpt2-large EleutherAI_gpt-neo-1.3B gpt-3.5-turbo meta-llama_Llama-2-7b-chat-hf Mistral-7B-Instruct-v0.2 HuggingFac…
Figure 27
Figure 27. Figure 27: GPT-2: Global Z-Score V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12 V13 V14 V15 V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12 V13 V14 V15 0.00 -0.15 -0.37 0.17 -0.65 -1.28 1.16 0.76 -1.11 1.84 1.04 -1.24 -1.20 1.07 -0.04 1.17 0.00 -1.18 -0.85 -1.10 -0.15 -0.44 -0.26 -0.67 1.88 -0.8…
Figure 30
Figure 30. Figure 30: GPT-3.5: Global Z￾Score V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12 V13 V14 V15 V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12 V13 V14 V15 0.00 -1.83 0.33 0.40 -0.97 -1.02 -0.94 -0.70 0.05 0.29 0.22 0.61 -0.05 1.75 1.86 -1.85 0.00 0.44 0.35 -1.17 -0.76 -0.79 -0.69 -0.00 0.05 0.19 …
Figure 33
Figure 33. Figure 33: LLaMA-2: Global Z￾Score V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12 V13 V14 V15 V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12 V13 V14 V15 0.00 -0.89 0.24 -1.09 -1.30 -0.11 -0.70 -0.79 -0.28 0.15 0.52 0.37 -0.26 2.06 2.08 -1.23 0.00 0.21 -1.29 -1.31 -0.11 -0.55 -0.98 -0.23 0.45 0.…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

25 extracted references · 14 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Shengyuan Bai et al. 2025. Enhancing nlu in large language models using adversarial noisy instruction tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39

    work page 2025

  4. [4]

    Mingyang Chen et al. 2024. Facilitating multi-turn function calling for llms via compositional instruction tuning. arXiv preprint arXiv:2410.12952

    arXiv 2024

  5. [5]

    Yuyan Chen et al. 2025. Attributive reasoning for hallucination diagnosis of large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39

    work page 2025

  6. [6]

    Xiaohu Du et al. 2025. Multi-turn jailbreaking large language models via attention shifting. Preprint

    work page 2025

  7. [7]

    Deep Ganguli et al. 2022. Predictability and surprise in large generative models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency

    work page 2022

  8. [8]

    J. Gao, D. Tang, Y. Xu, X. Feng, Z. Zhang, L. Shou, B. Qin, D. Jiang, and T. Liu. 2021. Simcse: Simple contrastive learning of sentence embeddings. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 689--707. https://arxiv.org/abs/2104.08821

    arXiv 2021

Show all 25 references
  1. [9]

    Ziwei Ji et al. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1--38

    2023

  2. [10]

    Xiaojun Jia et al. 2024. Improved techniques for optimization-based jailbreaking on large language models. arXiv preprint arXiv:2405.21018

    2024 arXiv

  3. [11]

    Zekun Li et al. 2023. Evaluating the instruction-following robustness of large language models to prompt injection. arXiv preprint arXiv:2308.10819

    2023 arXiv

  4. [12]

    Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896

    2023 arXiv

  5. [13]

    Joshua Maynez et al. 2020. On faithfulness and factuality in abstractive summarization. arXiv preprint arXiv:2005.00661

    2020 arXiv

  6. [14]

    OpenAI. 2023. Gpt-4 system card. Technical report, OpenAI. https://cdn.openai.com/papers/gpt-4-system-card.pdf

    2023

  7. [15]

    Long Ouyang et al. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730--27744

    2022

  8. [16]

    Ethan Perez et al. 2023. Discovering language model behaviors with model-written evaluations. In Findings of the Association for Computational Linguistics: ACL 2023

    2023

  9. [17]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084

    2019 arXiv

  10. [18]

    Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems

    2021

  11. [19]

    Amrith Setlur et al. 2024. Prompting is a double-edged sword: Improving worst-group robustness of foundation models. In Proceedings of the 41st International Conference on Machine Learning

    2024

  12. [20]

    Albert Webson and Ellie Pavlick. 2022. Do prompt-based models really understand the meaning of their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

    2022

  13. [21]

    Jason Wei et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824--24837

    2022

  14. [22]

    Zhuosheng Zhang et al. 2025. Igniting language intelligence: The hitchhiker’s guide from chain-of-thought reasoning to language agents. ACM Computing Surveys, 57(8):1--39

    2025

  15. [23]

    Lexin Zhou et al. 2024. Larger and more instructable language models become less reliable. Nature, 634(8032):61--68

    2024

  16. [24]

    Runchuan Zhu et al. 2025. Utilize the flow before stepping into the same river twice: Certainty represented knowledge flow for refusal-aware instruction tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39

    2025

  17. [25]

    Andy Zou et al. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.