Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

GaussMark: A Practical Approach for Structural Watermarking of Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding a small Gaussian perturbation to a language model's weights produces a watermark that a provider can detect with a valid statistical test and no generation latency.

desk verdict Sound statistical validity for a genuinely new weight-perturbation watermark, but the missing key-recovery protocol leaves the practical deployment claim unsupported. read the letter →

arxiv 2501.13941 v1 pith:DGEBA7MP submitted 2025-01-17 cs.CR cs.AIcs.CLcs.LG

Adam Block , Ayush Sekhari , Alexander Rakhlin This is my paper · ORCID
classification cs.CRcs.AIcs.CLcs.LG MSC 62F0362G1068T50
keywords watermarkinglanguagemodelsstructuralwatermarkGaussianperturbationhypothesistestingstatisticalvaliditysoftmaxmodelrobustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GaussMark claims that a language model can be watermarked structurally by adding a small Gaussian perturbation to one of its weight matrices, with no change to the sampling process and therefore no generation latency. The watermark is detected by testing whether a secret key vector is correlated with the gradient of the candidate text's log-likelihood; because the key is Gaussian, the test statistic is standard normal under the null hypothesis, giving valid p-values for any text distribution. Under a linear softmax approximation of the model, the paper proves the test has high power that grows with the dimension of the watermarked layer and the length of the text. Empirically the scheme is reported to be detectable across three modern LLMs with essentially no loss on SuperGLUE, GSM-8K, and AlpacaEval, and to retain nontrivial power after token-level corruptions and roundtrip translation.

What carries the argument

The carrying object is the normalized inner product $\psi = \langle \xi, \nabla_\theta \log p_\theta(y \mid x) \rangle / (\sigma \|\nabla_\theta \log p_\theta(y \mid x)\|)$, a one-dimensional projection of the Gaussian key onto the gradient direction. Its validity comes from Gaussian rotation invariance: for any fixed gradient direction, $\langle \xi, g \rangle / (\sigma \|g\|)$ is standard normal when $\xi$ is isotropic Gaussian and independent of the text. The power analysis additionally uses the identity that under a linear softmax model the likelihood ratio equals $\exp(\langle \xi, \nabla \log p_\theta \rangle)$ renormalized, which tilts the sampling distribution toward text whose gradient aligns with the key.

What would settle it

Run Algorithm 2 on watermarked outputs while feeding in a guessed key instead of the true per-generation key: if the returned p-values are uniform, then the scheme has no detection power without an explicit key-recovery mechanism.

Watch

Extended reading notes

Core claim

The central claim is that one can watermark the weights rather than the tokens: sample $\xi \sim N(0, \sigma^2 I)$, generate with parameters $\theta + \xi$, and then test a candidate text by computing $\psi = \langle \xi, \nabla_\theta \log p_\theta(y \mid x) \rangle / (\sigma \|\nabla_\theta \log p_\theta(y \mid x)\|)$. Under the null that the text is generated independently of the key, the normalized inner product is exactly standard normal, so the test has level $\alpha$ and returns valid p-values (Proposition 3.2). Under the linear softmax model, the power is controlled by the exponential tilting of the gradient-aligned responses and grows with model dimension and text length (Proposition 3.4).

Load-bearing premise

Detection requires the provider to have or reconstruct the exact per-generation Gaussian key used when the text was produced, and the paper does not describe a procedure for mapping an arbitrary candidate text back to its key.

Editorial extensions

If this is right

  • Providers can watermark generated text with no impact on generation latency, because watermarking changes weights rather than the sampling procedure.
  • The detector returns statistically valid p-values for any candidate text distribution under the null, so false positive claims of authorship are controlled at level $\alpha$.
  • Detection power grows with the number of tokens and with the dimension of the watermarked layer, matching the theory's gradient-norm and Gaussian-width analysis.
  • Restricting the perturbation to low-rank principal components lets larger noise be added without measurable quality loss, while preserving detectability.
  • GaussMark can run inside standard inference accelerators, so deployment requires no bespoke sampling or decoding code.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves unspecified how a provider maps an arbitrary candidate text back to the exact per-generation key; until that mapping exists, the detector only works when the provider stores or regenerates the key alongside the prompt.
  • Because detection requires white-box access to the full model and a gradient back-pass, GaussMark is naturally a provider-side service rather than a public verifier.
  • A natural testable extension is to watermark the embedding or attention layers instead of a single MLP matrix, and to measure whether low-entropy instruction-tuned outputs become detectable when the system prompt is included in the gradient computation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. GaussMark proposes a structural watermark for LLMs, embedded in the weights rather than in token sampling. At generation (Algorithm 1), the provider samples a fresh Gaussian key ξ∼N(0,σ²I) and generates text from θ+ξ. At detection (Algorithm 2), the provider computes the score statistic ψ(y,ξ|x)=⟨ξ,∇_θ log p_θ(y|x)⟩/(σ‖∇_θ log p_θ(y|x)‖) and rejects the null that key and text are independent when ψ exceeds the (1−α)-quantile of the standard Gaussian. The paper proves (Proposition 3.2) that the test has exact level α and yields valid p-values for the composite null, with no assumption on the text distribution, and develops a power analysis for linear softmax models (Proposition 3.4 and Corollary 3.5). Experiments on Mistral-7B, Llama3.1-8B, and Phi3.5-Mini report high detectability at 1K tokens, essentially no quality loss on SuperGLUE, GSM-8K, and AlpacaEval-2.0, robustness to token-level corruptions and roundtrip translation, and large generation-speed advantages over the KGW scheme.

Significance. The validity theorem is the strongest asset: the level-α guarantee is essentially unconditional (up to differentiability and nonzero gradient), covers the composite null with arbitrary text distributions, and therefore gives calibrated p-values even for human text. The scheme is simple, has no generation latency, and is compatible with standard inference engines; the empirical study is broad (three models, three quality benchmarks, extensive ablation grids, a KGW comparison), and the limitations are stated candidly. However, the power analysis delivers less than the abstract promises: Proposition 3.4 is an exact error expression rather than a bound, and Corollary 3.5 rests on an unverified condition. More importantly, the deployment protocol is incomplete: Algorithm 2 requires the per-generation key as input, and the paper never specifies how the key for an arbitrary candidate text is recovered, so the headline 'practical' claim currently outruns the specified method. The underlying idea is sound and worth publishing after the protocol and claims are fixed.

major comments (3)
  1. [§2.3, Algorithms 1–2, §4.3, §6] The central practical claim, that a provider holding a secret key can determine whether arbitrary text was generated by its watermarked model, is not supported as specified because the paper never says how the provider obtains the per-generation key ξ for a candidate text. Algorithm 1 samples a fresh ξ∼N(0,σ²I) for every generation, and Algorithm 2 requires that exact ξ (and a prompt x) as input. If the provider instead tests an independent key ξ′, the statistic ψ(y,ξ′|x) is exactly standard normal conditional on y, so the test has level α but no power and detection fails. The robustness experiments in §4.3 (Figures 4 and 5) always use the correct oracle key and effectively the original prompt, even for corrupted or translated text, so they test resilience to edits but bypass the key-matching step that any deployment would face. Section 6's limitations list (white-box access, token requirements, robustness) does not mention key management. In addition, if the provider must search over many stored (prompt, key) pairs to match a candidate text, the per-test level does not control the error over that search. The revision should specify a key-management and matching protocol (per-request key storage, a deterministic key schedule, or an explicit assumption that the text arrives with its generation key) and should either evaluate the end-to-end pipeline or state the matching assumption as part of the method.
  2. [§3.2, Proposition 3.4 and Corollary 3.5] The power guarantees are weaker than advertised. Proposition 3.4 is a change-of-measure identity for the Type II error β, not a concrete bound: it re-expresses β as an exponentially tilted expectation under y∼p_θ, and the subsequent discussion converts that into a Gaussian-width and condition-number heuristic (the R/r reasoning) without proving a bound. The only concrete statement, Corollary 3.5, is conditional on Λ(θ,x)≥0 in Eq. (6), a condition that is never verified for the linear softmax family nor for any transformer. Moreover, the proof of the corollary does not line up with its statement: the proof bounds e^{σ τ_α‖∇ log p_θ‖ − Γ̃} while Eq. (6) omits the quantile threshold τ_α, and the symbol σ is used both for the watermark's noise variance and as the dummy variable inside the supremum defining Λ. As written, the abstract's claim of 'formal statistical bounds on the power' is not delivered. The authors should either prove a bound with a checkable hypothesis for a concrete model class, or reframe the theoretical contribution as an exact error expression plus a heuristic power analysis.
  3. [§3.2.1, Lemma G.4, §4.1] Proposition 3.4 is proved for the sequence-level linear softmax model (Definition 3.3), but the empirical implementation (Table 2) adds noise to a single MLP weight matrix inside a transformer, and Lemma G.4 shows that the resulting model is a nonlinear softmax whose feature map χ(θ;x) depends on θ in a complicated way. The paper bridges this gap with a local-linearity heuristic (modern transformers are 'approximately linear softmax models' for sufficiently small ‖ξ‖) that is never tested, and the experiments in Figures 10–12 suggest that the operating regime of interest (intermediate σ) is not an asymptotic regime in σ. Consequently, the power guarantee applies only to the idealized family; for real transformers the power claim rests on the empirical TPR/AUC results. The paper should state clearly that Proposition 3.4 holds for the linear softmax family, and should either verify the Jacobian approximation as a function of σ or label the power analysis for transformers as heuristic.
minor comments (6)
  1. [§3, before Algorithm 2] The text says the null is rejected when ψ(y, ξ|x) ≥ Φ(1−α); this should be the inverse CDF Φ⁻¹(1−α), matching Algorithm 2 and Proposition 3.2.
  2. [Corollary 3.5, Eq. (6)] The symbol σ is used both for the watermark's noise variance and as the dummy variable in the supremum defining Λ(θ,x), which makes the condition difficult to parse; the statement's exponent σ‖∇ log pθ(y|x)‖ also omits the threshold τ_α that appears in the proof's analogous term, so the statement and proof should be reconciled.
  3. [§4] The sentence 'Appendix G.4 explores the robustness of GaussMark to various kinds of corruptions' is a mis-reference: the robustness experiments are in Section 4.3 and Appendix D, while Appendix G.4 contains the theoretical robustness analysis.
  4. [Proposition 3.2 and its proof] The proof divides by ‖∇ log pθ(y|x)‖ without treating the case where the gradient vanishes; since the null distribution q is arbitrary, it can place mass on such y, and the paper should specify a convention (e.g., define the statistic to be zero or randomize at level α) for the degenerate case.
  5. [Appendix A and Section 4.1] The watermarking hyperparameters (layer, weight, variance) are selected using p-values on the same 1K C4 prompts that are later used to report detectability in Section 4.1, so the reported detection rates may be optimistic; using a held-out set for evaluation would strengthen the empirical claims.
  6. [§3, Practical Implementation] The claim that GaussMark.Generate 'imposes no additional computational cost' and integrates 'seamlessly into ... vLLM' should be reconciled with the need to instantiate θ+ξ and track ξ per generation; for concurrent requests with distinct keys, the provider must maintain per-request weight versions, which is an engineering burden the paper does not discuss.

Circularity Check

2 steps flagged · score 6.0 of 10

Headline detectability results are in-sample fits (hyperparameters tuned on the same 1K prompts); the power theorem is a definitional restatement of the test's error. The validity proof is self-contained.

  1. fitted input called prediction [Appendix A (hyperparameter selection) and Section 4.1 (detectability evaluation)]
    "These parameters were selected by sweeping over many choices of layer, weight, and variance and for each watermarking parameter evaluating the following: (1) finding the p-values of watermark detection on completions of the 1K prompts ... We then chose the watermark parameters for each model as those that maximized detectability subject to not hurting performance. ... We use the same 1K prompts for all models and all watermarking keys in order to make the comparison fair."

    The reported detection rates (Figures 1(a), 2(a), and the ROC/TPR plots) are computed on the same 1K C4 prompts that were used to select sigma, layer, and weight matrix. Choosing parameters to maximize p-value significance on a set and then presenting detection performance on that same set is in-sample fitting, not prediction: the headline TPR and p-values are forced by the selection procedure rather than constituting independent evidence of detectability.

  2. other [Proposition 3.4 (Section 3.2.1)]
    "β = E_{y∼pθ,ξ∼N(0,σ2I)}[γ(y, ξ∣ x)⋅ I{ψ(y, ξ∣ x)≤ τα}], with the multiplicative factor γ(y, ξ∣ x):= e^{⟨ξ,∇ log pθ(y∣x)⟩}/E_{y∼pθ}[e^{⟨ξ,∇ log pθ(y∣x)⟩}]."

    By definition, the Type II error under HA is P_{y∼p_{θ+ξ}, ξ}(ψ≤τα). Changing measure to y∼pθ and inserting the linear-softmax density ratio (Lemma G.3) gives exactly this expression for beta. Thus (5) is the test's own error restated, not a quantitative bound showing that beta is small. The only concrete power bound, Corollary 3.5, requires the uncomputed condition (6) (Λ(θ,x)≥0), so the promised formal power guarantee is conditional on an assumption that is neither derived nor verified.

full rationale

GaussMark's validity proof (Proposition 3.2) is self-contained: under H0, ξ and y are independent, and for any fixed y the normalized inner product ξᵀ∇/σ∥∇∥ is exactly standard normal, giving an unconditional level guarantee. This theoretical core does not depend on fitted constants or self-citations. The power analysis is weaker: Proposition 3.4's beta expression is an exact change-of-measure reformulation of the Type II error under the linear-softmax model, not a bound; the subsequent Corollary 3.5 supplies a bound only under the uncomputed condition (6). Separately, the main empirical detectability claim is partially circular because σ, layer, and weight are selected by maximizing p-value significance on the same 1K prompts used for the headline detection plots. Quality benchmarks (SuperGLUE, GSM-8K, AlpacaEval) and comparisons to KGW provide some external signal, but they do not rescue the in-sample nature of the primary detectability evidence. Finally, the scheme as written lacks a key-recovery procedure: Algorithm 2 requires the exact key ξ as input, Algorithm 1 samples a fresh ξ per generation, and the paper never specifies how a provider obtains the correct ξ for an arbitrary candidate text; this is a completeness gap rather than a circular step and is not listed among the Section 6 limitations. Overall, the central validity theorem remains independent, but two load-bearing claims—empirical detectability and the power bound—reduce to fitted or definitional content, yielding partial circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The only new mathematical object is the per-generation Gaussian perturbation itself, which is not an invented physical entity but a sampling mechanism. The load-bearing free parameters are the noise variance, the choice of weight matrix, and the rank-reduction count. The key-recovery premise is an unflagged operational axiom that the paper does not justify.

free parameters (3)
  • Gaussian noise variance sigma^2 = 1e-05 (Mistral), 3e-04 (Llama), 1e-03 (Phi)
    Chosen by sweeping to maximize detectability while keeping quality on SuperGLUE and GSM-8K. Detectability and model quality both depend on it.
  • Watermarked layer and weight matrix = Mistral layer 20 up_proj; Llama layer 28 up_proj; Phi layer 20 down_proj
    Selected from a sweep over layers and weights to balance detectability and quality; not derived from theory.
  • Rank-reduction k (number of principal components dropped) = 1024 (Mistral, Phi), 512 (Llama)
    Selected for the rank-reduced variant to trade off detectability against model quality.
assumptions (4)
  • standard math Assumption 3.1: theta -> log p_theta(y|x) is differentiable for all x and y.
    Required for computing the gradient statistic; ReLU models only satisfy this via the subgradient extension mentioned by the authors.
  • domain assumption Local linear-softmax approximation: the map from perturbed feedforward weights to logits is approximately linear in a small neighborhood of trained weights.
    Used to justify the power analysis in Section 3.2.1; supported by model-merging observations, not by a formal bound for transformers.
  • domain assumption Gradient norms concentrate in a well-conditioned annulus r <= ||grad log p_theta(y|x)|| <= R.
    Used in the heuristic power argument and in Corollary 3.5 through the condition Lambda(theta,x) >= 0; reported empirically in Figure 9 but not guaranteed.
  • ad hoc to paper The provider can obtain the exact per-generation key xi used for a candidate text.
    Algorithm 2 takes xi as input but no key storage or derivation is specified; without this, the detection procedure cannot be run.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GaussMark: A Practical Approach for Structural Watermarking of Language Models." pith.science (2026). https://pith.science/paper/DGEBA7MP

@misc{pith2026250113941,
  author       = {Pith},
  title        = {Pith review of: GaussMark: A Practical Approach for Structural Watermarking of Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DGEBA7MP}},
  note         = {Machine review of arXiv:2501.13941}
}
read the original abstract

Recent advances in Large Language Models (LLMs) have led to significant improvements in natural language processing tasks, but their ability to generate human-quality text raises significant ethical and operational concerns in settings where it is important to recognize whether or not a given text was generated by a human. Thus, recent work has focused on developing techniques for watermarking LLM-generated text, i.e., introducing an almost imperceptible signal that allows a provider equipped with a secret key to determine if given text was generated by their model. Current watermarking techniques are often not practical due to concerns with generation latency, detection time, degradation in text quality, or robustness. Many of these drawbacks come from the focus on token-level watermarking, which ignores the inherent structure of text. In this work, we introduce a new scheme, GaussMark, that is simple and efficient to implement, has formal statistical guarantees on its efficacy, comes at no cost in generation latency, and embeds the watermark into the weights of the model itself, providing a structural watermark. Our approach is based on Gaussian independence testing and is motivated by recent empirical observations that minor additive corruptions to LLM weights can result in models of identical (or even improved) quality. We show that by adding a small amount of Gaussian noise to the weights of a given LLM, we can watermark the model in a way that is statistically detectable by a provider who retains the secret key. We provide formal statistical bounds on the validity and power of our procedure. Through an extensive suite of experiments, we demonstrate that GaussMark is reliable, efficient, and relatively robust to corruptions such as insertions, deletions, substitutions, and roundtrip translations and can be instantiated with essentially no loss in model quality.

Figures

Figures reproduced from arXiv: 2501.13941 by the authors.

Figure 1
Figure 1. Demonstration of the efficacy of GaussianMark on C4 prompts. (a) The median p-value of our detection procedure on 1K watermarked generations for different numbers of generated tokens averaged across 3 seeds. (b) Examples of watermarked text generated via GaussMark.Generate on Llama3.1-8B model. The full text completions are given in Appendix H.1 pose significant risks to social, political, and economic institutions … view at source ↗
Figure 2
Figure 2. Effect of length of generated token sequence on p-values for [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. Latency of GaussMark in seconds for generation (a) and detection (b) as number of generated tokens increases. Note that both generation and detection processes are highly efficient, ensuring practical applicability of our approach. three recent LMs: Llama3.1–8B [Dubey et al., 2024], Mistral–7B [Jiang et al., 2023], and Phi3.5–Mini [Abdin et al., 2024]. Note that versions of Llama3.1–8B and Mistral–7B have been used … view at source ↗
Figures from the paper (26 more)
Figure 4
Figure 4. Figure 4: Demonstration of robustness of GaussMark to four kinds of corruptions. We demonstrate the effects of (a) random insertions of tokens, (b) random deletions of tokens, and (c) random substitutions of tokens on the rate of detection of true positives of watermarked text a…
Figure 5
Figure 5. Figure 5: Demonstration of robustness of GaussMark to ignorance of prompt. We demonstrate the effects of (a) inserting, (b) deleting, and (c) substituting parts of the prompt on the fraction of detected sequences at the p = 0.05 level. Note that GaussMark is robust to these corr…
Figure 6
Figure 6. Figure 6: Demonstration of effect of of rank-reduced [PITH_FULL_IMAGE:figures/full_fig_p030_6.png]
Figure 7
Figure 7. Figure 7: Effect of number of tokens on the detectability of [PITH_FULL_IMAGE:figures/full_fig_p032_7.png]
Figure 8
Figure 8. Figure 8: Effect of number of tokens on the detectability of [PITH_FULL_IMAGE:figures/full_fig_p050_8.png]
Figure 9
Figure 9. Figure 9: Distribution of ∥∇log pθ(y ∣ x)∥ for 5 different prompts for (a) Mistral–7B, (b) Llama3.1–8B, and (c) Phi3.5–Mini. The relatively tight concentration of the gradient norms serves to justify the linear softmax approximations as per the discussion following Proposition 3…
Figure 10
Figure 10. Figure 10: Detectability of watermarked Mistral–7B models with GaussMark for different layers and variances as measured by the median p-value over 1K prompts for (a) down_proj, (b) up_proj, and (c) gate_proj, colored by value. 52 [PITH_FULL_IMAGE:figures/full_fig_p052_10.png]
Figure 11
Figure 11. Figure 11: Detectability of watermarked Llama3.1–8B models with GaussMark for different layers and variances as measured by the median p-value over 1K prompts for (a) down_proj, (b) up_proj, and (c) gate_proj, colored by value. 1e-05 3e-05 1e-04 3e-04 1e-03 Layer 31 Layer 30 Lay…
Figure 12
Figure 12. Figure 12: Detectability of watermarked Phi3.5–Mini models with GaussMark for different layers and variances as measured by the median p-value over 1K prompts for (a) down_proj and (b) gate_up_proj, colored by value. 53 [PITH_FULL_IMAGE:figures/full_fig_p053_12.png]
Figure 13
Figure 13. Figure 13: Effect that GaussMark has on Mistral–7B’s performance on GSM–8K for different layers and variances for (a) down_proj, (b) up_proj, and (c) gate_proj, colored by whether the watermarked model’s performance is statistically indistinguishable from that of the base model …
Figure 14
Figure 14. Figure 14: Effect that GaussMark has on Llama3.1–8B’s performance on GSM–8K for different layers and variances for (a) down_proj, (b) up_proj, and (c) gate_proj, colored by whether the watermarked model’s performance is statistically indistinguishable from that of the base model…
Figure 15
Figure 15. Figure 15: Effect that GaussMark has on Phi3.5–Mini’s performance on GSM–8K for different layers and variances for (a) down_proj and (b) gate_up_proj, a, colored by whether the watermarked model’s performance is statistically indistinguishable from that of the base model (yellow…
Figure 16
Figure 16. Figure 16: Effect of token-level corruptions on the median p-values of [PITH_FULL_IMAGE:figures/full_fig_p057_16.png]
Figure 17
Figure 17. Figure 17: Effect of token-level corruptions on the AUC of [PITH_FULL_IMAGE:figures/full_fig_p058_17.png]
Figure 18
Figure 18. Figure 18: Detectability of rank-reduced GaussMark on all three models as measured by (a) TPR at FPR 0.01 and (b) median p-values as a function of the number of watermarked tokens. The watermark is detectable for a substantial fraction of the text in all three models, with more …
Figure 19
Figure 19. Figure 19: Effect of rank reduction on the median p-values of [PITH_FULL_IMAGE:figures/full_fig_p060_19.png]
Figure 20
Figure 20. Figure 20: Effect of rank reduction in GaussMark on model performance on GSM–8K on (a) Llama3.1–8B, (b) Mistral–7B, and (c) Phi3.5–Mini. As the rank of the quotient space increases, the model quality increases as well, reflecting the decreased influence of the perturbation on th…
Figure 21
Figure 21. Figure 21: Effect of roundtrip translation on rank-reduced instantiation of [PITH_FULL_IMAGE:figures/full_fig_p062_21.png]
Figure 22
Figure 22. Figure 22: Effect of token-level corruptions on Rank-reduced [PITH_FULL_IMAGE:figures/full_fig_p063_22.png]
Figure 23
Figure 23. Figure 23: Effect of token-level corruptions on rank-reduced [PITH_FULL_IMAGE:figures/full_fig_p064_23.png]
Figure 24
Figure 24. Figure 24: Effect of token-level corruptions on rank-reduced [PITH_FULL_IMAGE:figures/full_fig_p065_24.png]
Figure 25
Figure 25. Figure 25: Comparison of detectability of GaussMark with KGW-1 and KGW-2 as measured by median detection times and AUCs for Llama3.1–8B (a-b), Mistral–7B (c-d), and Phi3.5–Mini (e-f). 66 [PITH_FULL_IMAGE:figures/full_fig_p066_25.png]
Figure 26
Figure 26. Figure 26: Comparison of detectability of GaussMark with KGW-1 and KGW-2 as measured by the TPR@FPR 0.05 and 0.01 for Llama3.1–8B (a-b), Mistral–7B (c-d), and Phi3.5–Mini (e-f). 67 [PITH_FULL_IMAGE:figures/full_fig_p067_26.png]
Figure 27
Figure 27. Figure 27: Comparison between GaussMark and KGW generation and detection times averaged across 3 seeds and 100 generations for Llama3.1–8B (a-b), Mistral–7B (c-d), and Phi3.5–Mini (e-f). 68 [PITH_FULL_IMAGE:figures/full_fig_p068_27.png]
Figure 28
Figure 28. Figure 28: Comparison between GaussMark and KGW robustness to token level corruptions (adding and removing at random points) as measured by detection at FPR = 0.05 for Llama3.1–8B (a-b), Mistral–7B (c-d), and Phi3.5–Mini (e-f). 69 [PITH_FULL_IMAGE:figures/full_fig_p069_28.png]
Figure 29
Figure 29. Figure 29: Comparison between GaussMark and KGW robustness to token level corruptions (substi￾tuting tokens) and roundtrip translation through French for Llama3.1–8B (a-b), Mistral–7B (c-d), and Phi3.5–Mini (e-f). 70 [PITH_FULL_IMAGE:figures/full_fig_p070_29.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. More Human or More AI? Visualizing Human-AI Collaboration Disclosures in Journalistic News Production

    cs.HC 2026-01 conditional novelty 6.0 of 10

    Disclosure visualization format systematically shifts readers' perceptions of human vs AI contribution: role-based timelines amplify perceived AI role in mostly human articles, while task-based timelines make mostly A...

Reference graph

Works this paper leans on

107 extracted references · 22 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Membership inference attacks from first principles

    Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., and Tramer, F. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pp.\ 1897--1914. IEEE, 2022

    2022

  3. [3]

    and Berger, R

    Casella, G. and Berger, R. Statistical inference. Duxbury Resource Center , June 2001. ISBN 0534243126

    2001

  4. [4]

    Scalable watermarking for identifying large language model outputs

    Dathathri, S., See, A., Ghaisas, S., Huang, P.-S., McAdam, R., Welbl, J., Bachani, V., Kaskasoli, A., Stanforth, R., Matejovicova, T., et al. Scalable watermarking for identifying large language model outputs. Nature, 634 0 (8035): 0 818--823, 2024

    2024

  5. [6]

    A watermark for large language models

    Kirchenbauer, J., Geiping, J., Wen, Y., Katz, J., Miers, I., and Goldstein, T. A watermark for large language models. In International Conference on Machine Learning, pp.\ 17061--17084. PMLR, 2023 a

    2023

  6. [9]

    H., Gonzalez, J

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  7. [10]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  8. [12]

    High-dimensional probability: An introduction with applications in data science, volume 47

    Vershynin, R. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018

    2018

Show all 107 references
  1. [13]

    Wolf, T. et al. Huggingface's transformers: State-of-the-art natural language processing. https://github.com/huggingface/transformers, 2020

    2020

  2. [14]

    Watermarking gpt outputs

    Scott Aaronson and Hendrik Kirchner. Watermarking gpt outputs. 2022. URL https://www.scottaaronson.com/talks/watermark.ppt

    2022

  3. [15]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024

    2024 arXiv

  4. [16]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  5. [17]

    Large language models for mathematical reasoning: Progresses and challenges

    Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges. arXiv preprint arXiv:2402.00157, 2024

    2024 arXiv

  6. [18]

    Mechanistic interpretability for ai safety--a review

    Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety--a review. arXiv preprint arXiv:2404.14082, 2024

    2024 arXiv

  7. [19]

    Multi-bit distortion-free watermarking for large language models

    Massieh Kordi Boroujeny, Ya Jiang, Kai Zeng, and Brian Mark. Multi-bit distortion-free watermarking for large language models. arXiv preprint arXiv:2402.16578, 2024

    2024 arXiv

  8. [20]

    Membership inference attacks from first principles

    Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1897--1914. IEEE, 2022

    2022

  9. [21]

    Statistical inference

    George Casella and Roger Berger. Statistical inference. Duxbury Resource Center , June 2001. ISBN 0534243126

    2001

  10. [22]

    Watermark smoothing attacks against language models

    Hongyan Chang, Hamed Hassani, and Reza Shokri. Watermark smoothing attacks against language models. arXiv preprint arXiv:2407.14206, 2024

    2024 arXiv

  11. [23]

    Undetectable watermarks for language models

    Miranda Christ, Sam Gunn, and Or Zamir. Undetectable watermarks for language models. In The Thirty Seventh Annual Conference on Learning Theory, pages 1125--1139. PMLR, 2024

    2024

  12. [24]

    Adaptersoup: Weight averaging to improve generalization of pretrained language models

    Alexandra Chronopoulou, Matthew E Peters, Alexander Fraser, and Jesse Dodge. Adaptersoup: Weight averaging to improve generalization of pretrained language models. arXiv preprint arXiv:2302.07027, 2023

    2023 arXiv

  13. [25]

    All that's 'human' is not gold: Evaluating human evaluation of generated text

    Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A Smith. All that's 'human' is not gold: Evaluating human evaluation of generated text. arXiv preprint arXiv:2107.00061, 2021

    2021 arXiv

  14. [26]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  15. [27]

    Scalable watermarking for identifying large language model outputs

    Sumanth Dathathri, Abigail See, Sumedh Ghaisas, Po-Sen Huang, Rob McAdam, Johannes Welbl, Vandana Bachani, Alex Kaskasoli, Robert Stanforth, Tatiana Matejovicova, et al. Scalable watermarking for identifying large language model outputs. Nature, 634 0 (8035): 0 818--823, 2024

    2024

  16. [28]

    Qlora: Efficient finetuning of quantized llms

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024

    2024

  17. [29]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  18. [30]

    Length-controlled alpacaeval: A simple way to debias automatic evaluators

    Yann Dubois, Bal \'a zs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024

    2024 arXiv

  19. [31]

    Real or fake text?: Investigating human ability to detect boundaries between human-written and machine-generated text

    Liam Dugan, Daphne Ippolito, Arun Kirubarajan, Sherry Shi, and Chris Callison-Burch. Real or fake text?: Investigating human ability to detect boundaries between human-written and machine-generated text. In Proceedings of the AAAI Conference on Artificial Intelligence, volume ...

    2023

  20. [32]

    Considers-the-human evaluation framework: Rethinking human evaluation for generative large language models

    Aparna Elangovan, Ling Liu, Lei Xu, Sravan Bodapati, and Dan Roth. Considers-the-human evaluation framework: Rethinking human evaluation for generative large language models. arXiv preprint arXiv:2405.18638, 2024

    2024 arXiv

  21. [33]

    Publicly detectable watermarking for language models

    Jaiden Fairoze, Sanjam Garg, Somesh Jha, Saeed Mahloujifar, Mohammad Mahmoody, and Mingyuan Wang. Publicly detectable watermarking for language models. arXiv preprint arXiv:2310.18491, 2023

    2023 arXiv

  22. [34]

    Functional invariants to watermark large transformers

    Pierre Fernandez, Guillaume Couairon, Teddy Furon, and Matthijs Douze. Functional invariants to watermark large transformers. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4815--4819. IEEE, 2024

    2024

  23. [35]

    On pushing deepfake tweet detection capabilities to the limits

    Margherita Gambini, Tiziano Fagni, Fabrizio Falchi, and Maurizio Tesconi. On pushing deepfake tweet detection capabilities to the limits. In Proceedings of the 14th ACM Web Science Conference 2022, pages 154--163, 2022

    2022

  24. [36]

    A framework for few-shot language model evaluation, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024

  25. [37]

    Gltr: Statistical detection and visualization of generated text

    Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. Gltr: Statistical detection and visualization of generated text. arXiv preprint arXiv:1906.04043, 2019

    1906 arXiv

  26. [38]

    Fundamentals of nonparametric Bayesian inference, volume 44

    Subhashis Ghosal and Aad W van der Vaart. Fundamentals of nonparametric Bayesian inference, volume 44. Cambridge University Press, 2017

    2017

  27. [39]

    The problem with false positives: Ai detection unfairly accuses scholars of ai plagiarism

    Louie Giray. The problem with false positives: Ai detection unfairly accuses scholars of ai plagiarism. The Serials Librarian, pages 1--9, 2024

    2024

  28. [40]

    Arcee ' s M erge K it: A toolkit for merging large language models

    Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee ' s M erge K it: A toolkit for merging large language models. In Franck Dernoncourt, Daniel Preo t iuc-Pietro, and Anastasia Shimor...

    2024

  29. [41]

    Edit distance robust watermarks for language models

    Noah Golowich and Ankur Moitra. Edit distance robust watermarks for language models. arXiv preprint arXiv:2406.02633, 2024

    2024 arXiv

  30. [42]

    What makes quantization for large language model hard? an empirical study from the lens of perturbation

    Zhuocheng Gong, Jiahao Liu, Jingang Wang, Xunliang Cai, Dongyan Zhao, and Rui Yan. What makes quantization for large language model hard? an empirical study from the lens of perturbation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18082-...

    2024

  31. [43]

    A kernel two-sample test

    Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Sch \"o lkopf, and Alexander J Smola. A kernel two-sample test. Journal of Machine Learning Research, 13 0 (Mar): 0 723--773, 2012

    2012

  32. [44]

    On the learnability of watermarks for language models

    Chenchen Gu, Xiang Lisa Li, Percy Liang, and Tatsunori Hashimoto. On the learnability of watermarks for language models. arXiv preprint arXiv:2312.04469, 2023

    2023 arXiv

  33. [45]

    Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning

    Han Guo, Philip Greengard, Eric P Xing, and Yoon Kim. Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. arXiv preprint arXiv:2311.12023, 2023

    2023 arXiv

  34. [46]

    Parameter-efficient fine-tuning for large models: A comprehensive survey, 2024

    Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey, 2024. URL https://arxiv.org/abs/2403.14608

    2024 arXiv

  35. [47]

    Wang, and Noah D

    Joy He-Yueya, Gabriel Poesia, Rose E. Wang, and Noah D. Goodman. Solving math word problems by combining language models with symbolic solvers, 2023. URL https://arxiv.org/abs/2304.09102

    2023 arXiv

  36. [48]

    Semstamp: A semantic watermark with paraphrastic robustness for text generation

    Abe Bohan Hou, Jingyu Zhang, Tianxing He, Yichen Wang, Yung-Sung Chuang, Hongwei Wang, Lingfeng Shen, Benjamin Van Durme, Daniel Khashabi, and Yulia Tsvetkov. Semstamp: A semantic watermark with paraphrastic robustness for text generation. arXiv preprint arXiv:2310.03991, 2023

    arXiv

  37. [49]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  38. [50]

    Towards optimal statistical watermarking

    Baihe Huang, Hanlin Zhu, Banghua Zhu, Kannan Ramchandran, Michael I Jordan, Jason D Lee, and Jiantao Jiao. Towards optimal statistical watermarking. arXiv preprint arXiv:2312.07930, 2023

    2023 arXiv

  39. [51]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022

    2022 arXiv

  40. [52]

    Mathprompter: Mathematical reasoning using large language models

    Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models. arXiv preprint arXiv:2303.05398, 2023

    2023 arXiv

  41. [53]

    Automatic detection of generated text is easiest when humans are fooled

    Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. Automatic detection of generated text is easiest when humans are fooled. arXiv preprint arXiv:1911.00650, 2019

    1911 arXiv

  42. [54]

    Deep learning for misinformation detection on online social networks: a survey and new perspectives

    Md Rafiqul Islam, Shaowu Liu, Xianzhi Wang, and Guandong Xu. Deep learning for misinformation detection on online social networks: a survey and new perspectives. Social Network Analysis and Mining, 10 0 (1): 0 82, 2020

    2020

  43. [55]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  44. [56]

    A survey on large language models for code generation

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. arXiv preprint arXiv:2406.00515, 2024

    2024 arXiv

  45. [57]

    Professors are using chatgpt detector tools to accuse students of cheating

    Kayla Jimenez. Professors are using chatgpt detector tools to accuse students of cheating. but what if the software is wrong. USA Today, 2023

    2023

  46. [58]

    Watermark stealing in large language models

    Nikola Jovanovi \'c , Robin Staab, and Martin Vechev. Watermark stealing in large language models. arXiv preprint arXiv:2402.19361, 2024

    2024 arXiv

  47. [59]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  48. [60]

    u chemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan G \

    Enkelejda Kasneci, Kathrin Se ler, Stefan K \"u chemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan G \"u nnemann, Eyke H \"u llermeier, et al. Chatgpt for good? on opportunities and challenges of large language models for education. Learn...

    2023

  49. [61]

    A watermark for large language models

    John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. In International Conference on Machine Learning, pages 17061--17084. PMLR, 2023 a

    2023

  50. [62]

    On the reliability of watermarks for large language models

    John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. On the reliability of watermarks for large language models. arXiv preprint arXiv:2306.04634, 2023 b

    2023 arXiv

  51. [63]

    She was falsely accused of cheating with ai--and she won’t be the last

    M Klee. She was falsely accused of cheating with ai--and she won’t be the last. Rolling Stone, 2023

    2023

  52. [64]

    Robust distortion-free watermarks for language models

    Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang. Robust distortion-free watermarks for language models. arXiv preprint arXiv:2307.15593, 2023

    2023 arXiv

  53. [65]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems...

    2023

  54. [66]

    Waterfall: Scalable framework for robust text watermarking and provenance for llms

    Gregory Kang Ruey Lau, Xinyuan Niu, Hieu Dao, Jiangwei Chen, Chuan-Sheng Foo, and Bryan Kian Hsiang Low. Waterfall: Scalable framework for robust text watermarking and provenance for llms. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processin...

    2024

  55. [67]

    Asymptotic methods in statistical decision theory

    Lucien Le Cam. Asymptotic methods in statistical decision theory. Springer Science & Business Media, 2012

    2012

  56. [68]

    Evaluating human-language model interaction

    Mina Lee, Megha Srivastava, Amelia Hardy, John Thickstun, Esin Durmus, Ashwin Paranjape, Ines Gerard-Ursin, Xiang Lisa Li, Faisal Ladhak, Frieda Rong, et al. Evaluating human-language model interaction. arXiv preprint arXiv:2212.09746, 2022

    2022 arXiv

  57. [69]

    Testing statistical hypotheses, volume 3

    Erich Leo Lehmann, Joseph P Romano, and George Casella. Testing statistical hypotheses, volume 3. Springer, 1986

    1986

  58. [70]

    Watermarking llms with weight quantization

    Linyang Li, Botian Jiang, Pengyu Wang, Ke Ren, Hang Yan, and Xipeng Qiu. Watermarking llms with weight quantization. arXiv preprint arXiv:2310.11237, 2023 a

    2023 arXiv

  59. [71]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 5 2023 b

    2023

  60. [72]

    Watermarking techniques for large language models: A survey

    Yuqing Liang, Jiancheng Xiao, Wensheng Gan, and Philip S Yu. Watermarking techniques for large language models: A survey. arXiv preprint arXiv:2409.00089, 2024

    2024 arXiv

  61. [73]

    A semantic invariant robust watermark for large language models, 2024 a

    Aiwei Liu, Leyi Pan, Xuming Hu, Shiao Meng, and Lijie Wen. A semantic invariant robust watermark for large language models, 2024 a . URL https://arxiv.org/abs/2310.06356

    2024 arXiv

  62. [74]

    A survey of text watermarking in the era of large language models

    Aiwei Liu, Leyi Pan, Yijian Lu, Jingjing Li, Xuming Hu, Xi Zhang, Lijie Wen, Irwin King, Hui Xiong, and Philip Yu. A survey of text watermarking in the era of large language models. ACM Computing Surveys, 2024 b

    2024

  63. [75]

    Ai detection tools falsely accuse international students of cheating

    TG Mathewson. Ai detection tools falsely accuse international students of cheating. The Markup, 2023

    2023

  64. [76]

    A survey on knowledge editing of neural networks

    Vittorio Mazzia, Alessandro Pedrani, Andrea Caciolai, Kay Rottmann, and Davide Bernardi. A survey on knowledge editing of neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2024

    2024

  65. [77]

    The threat of offensive ai to organizations

    Yisroel Mirsky, Ambra Demontis, Jaidip Kotak, Ram Shankar, Deng Gelei, Liu Yang, Xiangyu Zhang, Maura Pintor, Wenke Lee, Yuval Elovici, et al. The threat of offensive ai to organizations. Computers & Security, 124: 0 103006, 2023

    2023

  66. [78]

    Detectgpt: Zero-shot machine-generated text detection using probability curvature

    Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. Detectgpt: Zero-shot machine-generated text detection using probability curvature. In International Conference on Machine Learning, pages 24950--24962. PMLR, 2023

    2023

  67. [79]

    On the problem of the most efficient tests of statistical hypotheses

    Jerzy Neyman and Egon Sharpe Pearson. On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 231 0 (694-706): 0 289--337, 1933

    1933

  68. [80]

    New ai classifier for indicating ai-written text

    OpenAI. New ai classifier for indicating ai-written text. OpenAI blog, 2023. URL https://openai.com/index/new-ai-classifier-for-indicating-ai-written-text/

    2023

  69. [81]

    Markllm: An open-source toolkit for llm watermarking

    Leyi Pan, Aiwei Liu, Zhiwei He, Zitian Gao, Xuandong Zhao, Yijian Lu, Binglin Zhou, Shuliang Liu, Xuming Hu, Lijie Wen, et al. Markllm: An open-source toolkit for llm watermarking. arXiv preprint arXiv:2405.10051, 2024

    2024 arXiv

  70. [82]

    Attacking llm watermarks by exploiting their strengths

    Qi Pang, Shengyuan Hu, Wenting Zheng, and Virginia Smith. Attacking llm watermarks by exploiting their strengths. arXiv preprint arXiv:2402.16187, 2024

    2024 arXiv

  71. [83]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019

    2019

  72. [84]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  73. [85]

    Revisiting the robustness of watermarking to paraphrasing attacks

    Saksham Rastogi and Danish Pruthi. Revisiting the robustness of watermarking to paraphrasing attacks. arXiv preprint arXiv:2411.05277, 2024

    2024 arXiv

  74. [86]

    Can ai-generated text be reliably detected? arXiv preprint arXiv:2303.11156, 2023

    Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. Can ai-generated text be reliably detected? arXiv preprint arXiv:2303.11156, 2023

    2023 arXiv

  75. [87]

    Chatgpt: Optimizing language models for dialogue

    John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Felipe Ceron Uribe, Liam Fedus, Luke Metz, Michael Pokorny, et al. Chatgpt: Optimizing language models for dialogue. OpenAI blog, 2 0 (4), 2022

    2022

  76. [88]

    Automatic fake news detection with pre-trained transformer models

    Mina Sch \"u tz, Alexander Schindler, Melanie Siegel, and Kawa Nazemi. Automatic fake news detection with pre-trained transformer models. In Pattern Recognition. ICPR International Workshops and Challenges: Virtual Event, January 10-15, 2021, Proceedings, Part VII, pages 627--...

    2021

  77. [89]

    The truth is in there: Improving reasoning in language models with layer-selective rank reduction

    Pratyusha Sharma, Jordan T Ash, and Dipendra Misra. The truth is in there: Improving reasoning in language models with layer-selective rank reduction. The Twelfth International Conference on Learning Representations, 2024

    2024

  78. [90]

    A simple and effective pruning approach for large language models

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695, 2023

    2023 arXiv

  79. [91]

    Gemma: Open models based on gemini research and technology

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  80. [92]

    Daniel Freeman, Theodore R

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua ...

    2024

  81. [93]

    Parents sue son’s high school history teacher over ai ‘cheating’ punishment, October 2024

    Kat Tenbarge. Parents sue son’s high school history teacher over ai ‘cheating’ punishment, October 2024. URL https://www.nbcnews.com/tech/tech-news/ai-paper-write-cheating-lawsuit-massachusetts-help-rcna175669. Accessed: 2024-12-24

    2024

  82. [94]

    o rg Tiedemann, Mikko Aulamo, Daria Bakshandaeva, Michele Boggia, Stig-Arne Gr \

    J \"o rg Tiedemann, Mikko Aulamo, Daria Bakshandaeva, Michele Boggia, Stig-Arne Gr \"o nroos, Tommi Nieminen, Alessandro Raganato, Yves Scherrer, Raul Vazquez, and Sami Virpioja. Democratizing machine translation with opus-mt. arXiv preprint arXiv:2212.01936, 2022

    2022 arXiv

  83. [95]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  84. [96]

    A professor accused his class of using chatgpt, putting diplomas in jeopardy

    Prashnu Verma. A professor accused his class of using chatgpt, putting diplomas in jeopardy. The Washington Post, May 18 2023. URL https://www.washingtonpost.com. Retrieved July 10, 2023

    2023

  85. [97]

    High-dimensional probability: An introduction with applications in data science, volume 47

    Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018

    2018

  86. [98]

    Superglue: A stickier benchmark for general-purpose language understanding systems

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019

    2019

  87. [99]

    Milora: Harnessing minor singular components for parameter-efficient llm finetuning

    Hanqing Wang, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. Milora: Harnessing minor singular components for parameter-efficient llm finetuning. arXiv preprint arXiv:2406.09044, 2024

    2024 arXiv

  88. [100]

    Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning

    Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning. arXiv preprint arXiv:2310.03731, 2023

    arXiv

  89. [101]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020

  90. [102]

    Huggingface's transformers: State-of-the-art natural language processing

    Thomas Wolf et al. Huggingface's transformers: State-of-the-art natural language processing. https://github.com/huggingface/transformers, 2020 b

    2020

  91. [103]

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing infe...

    2022

  92. [104]

    Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities

    Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities. arXiv preprint arXiv:2408.07666, 2024

    2024 arXiv

  93. [105]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning, 2024

    2024

  94. [106]

    Excuse me, sir? your language model is leaking (information)

    Or Zamir. Excuse me, sir? your language model is leaking (information). arXiv preprint arXiv:2401.10360, 2024

    2024 arXiv

  95. [107]

    Defending against neural fake news

    Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. Advances in neural information processing systems, 32, 2019

    2019

  96. [108]

    Watermarks in the sand: Impossibility of strong watermarking for generative models

    Hanlin Zhang, Benjamin L Edelman, Danilo Francati, Daniele Venturi, Giuseppe Ateniese, and Boaz Barak. Watermarks in the sand: Impossibility of strong watermarking for generative models. arXiv preprint arXiv:2311.04378, 2023

    2023 arXiv

  97. [109]

    Provable robust watermarking for ai-generated text

    Xuandong Zhao, Prabhanjan Ananth, Lei Li, and Yu-Xiang Wang. Provable robust watermarking for ai-generated text. arXiv preprint arXiv:2306.17439, 2023 a

    2023 arXiv

  98. [110]

    Protecting language generation models via invisible watermarking

    Xuandong Zhao, Yu-Xiang Wang, and Lei Li. Protecting language generation models via invisible watermarking. In International Conference on Machine Learning, pages 42187--42199. PMLR, 2023 b

    2023

  99. [111]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.