Pith. sign in

REVIEW 34 references

Can adversarial attacks by large language models be attributed?

T0 review · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Under a worst-case formal model, attributing LLM outputs to a specific model is provably impossible except in the narrow case of finitely many deterministic models.

arxiv 2411.08003 v3 pith:MYUBK3FI submitted 2024-11-12 cs.AI cs.CLcs.CYcs.FL

Manuel Cebrian , Andres Abeliuk , Jan Arne Telle This is my paper · ORCID
classification cs.AIcs.CLcs.CYcs.FL
keywords llmsmodelsclasslanguageattributiondeterministicfiniteidentifiable
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper models each language model as a formal language, meaning the set of all texts it can possibly produce. Attribution is then the task of deciding which language produced an observed set of texts. The authors apply Gold's classical result on learning languages from examples: when the set of possible models is infinite and one model's output language contains another's, no finite or even infinite stream of observed texts can uniquely identify the source, because the same texts are consistent with many models. They extend this to probabilistic models. Since the formal identification rule only looks at which strings appear, not how often they appear, even two models that can produce exactly the same texts but with different probabilities cannot be separated in the worst case. The paper gives an explicit example with two probability distributions over the same single-letter strings.
Extended reading notes

Core claim

Even a finite class of probabilistic LLMs can be non-identifiable: Theorem 7 states that 'even for a finite number of probabilistic formal languages (as few as two), identification in the limit does not always hold.' More broadly, the paper claims 'the only regime that avoids impossibility is the one with a finite, discrete set of hypotheses,' and that 'under mild assumptions of overlapping capabilities among models, certain classes of LLMs are fundamentally non-identifiable from their outputs alone.' If the paper is correct, any attribution method that only uses the set of observed outputs and not their frequencies cannot guarantee source identification, and the LLM hypothesis space is growing so fast that exhaustive search is infeasible.

Load-bearing premise

The load-bearing modeling premise is Definition 5: identification in the limit for probabilistic languages is defined over an adversarial infinite sequence in which every string in the support appears at least once, but the learner is not told the sampling probabilities or frequencies. Since the model's probabilities never enter the data, two distributions with identical support are indistinguishable by construction. If instead attribution is allowed to use repeated independent samples drawn from the model's distribution, or to query models, the finite probabilistic counterexample no longer applies. This is acknowledged in Section 5 ('if instead we did assume the samples were drawn from the model's distribution, then over time one might detect the slight differences in frequencies'). The entire negative claim for finite classes rests on this choice of presentation model.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central theoretical results borrow standard learning-theory theorems, while the empirical claims rely on two modeling choices: the adversarial presentation model that discards frequencies, and the assumption that any dataset fine-tune yields a distinct model. The fitted growth rates are free parameters derived from the constructed N(t) series.

free parameters (4)
  • Exponential growth rate b for N_single = 1.39 yr^-1
    Least-squares exponential fit to the constructed N_single(t) (Section 6.1, Table 1).
  • Exponential growth rate b for N_k<=2 = 1.95 yr^-1
    Fitted to N_k<=2(t) (Table 1).
  • Exponential growth rate b for N_k<=3 = 2.51 yr^-1
    Fitted to N_k<=3(t) (Table 1).
  • Imputed parameter sizes = Average of known model sizes
    Used to compute cumulative parameter counts; missing sizes replaced with average (Section 7).
assumptions (4)
  • standard math Gold's identification-in-the-limit framework and Angluin's tell-tale theorem.
    Used as the formal basis for all four regimes (Section 2, Theorems 1-2).
  • domain assumption An LLM's set of possible outputs can be represented as a formal language (support of a distribution over strings).
    Section 1 frames attribution as language identification; ignores prompt-conditioning and interface details.
  • ad hoc to paper Attribution is modeled by adversarial infinite presentations containing each support string at least once, without frequency information.
    Definition 5 in Section 5; this choice makes same-support probabilistic models indistinguishable.
  • ad hoc to paper Each open model can be fine-tuned on each dataset to yield a distinct candidate model.
    Section 6 formulas N_single, N_k<=2, N_k<=3 assume arbitrary dataset combinations produce distinct variants.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can adversarial attacks by large language models be attributed?." pith.science (2026). https://pith.science/paper/MYUBK3FI

@misc{pith2026241108003,
  author       = {Pith},
  title        = {Pith review of: Can adversarial attacks by large language models be attributed?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MYUBK3FI}},
  note         = {Machine review of arXiv:2411.08003}
}
read the original abstract

Attributing outputs from Large Language Models (LLMs) in adversarial settings-such as cyberattacks and disinformation campaigns-presents significant challenges that are likely to grow in importance. We approach this attribution problem from both a theoretical and an empirical perspective, drawing on formal language theory (identification in the limit) and data-driven analysis of the expanding LLM ecosystem. By modeling an LLM's set of possible outputs as a formal language, we analyze whether finite samples of text can uniquely pinpoint the originating model. Our results show that, under mild assumptions of overlapping capabilities among models, certain classes of LLMs are fundamentally non-identifiable from their outputs alone. We delineate four regimes of theoretical identifiability: (1) an infinite class of deterministic (discrete) LLM languages is not identifiable (Gold's classical result from 1967); (2) an infinite class of probabilistic LLMs is also not identifiable (by extension of the deterministic case); (3) a finite class of deterministic LLMs is identifiable (consistent with Angluin's tell-tale criterion); and (4) even a finite class of probabilistic LLMs can be non-identifiable (we provide a new counterexample establishing this negative result). Complementing these theoretical insights, we quantify the explosion in the number of plausible model origins (hypothesis space) for a given output in recent years. Even under conservative assumptions-each open-source model fine-tuned on at most one new dataset-the count of distinct candidate models doubles approximately every 0.5 years, and allowing multi-dataset fine-tuning combinations yields doubling times as short as 0.28 years. This combinatorial growth, alongside the extraordinary computational cost of brute-force likelihood attribution across all models and potential users, renders exhaustive attribution infeasible in practice.

Figures

Figures reproduced from arXiv: 2411.08003 by the authors.

Figure 1
Figure 1. Growth of the combinatorial lower bound Nsingle(t) (single-dataset fine-tunes per open model) from 2019 to 2025 on a logarithmic scale. The dashed line shows a least-squares exponential fit with estimated growth rate b = 1.39 yr−1 , corresponding to a doubling time of τ = 0.50 yr [PITH_FULL_IMAGE:figures/full_fig_p013_1.png] view at source ↗
Figure 2
Figure 2. Projected growth of the conservative lower bound [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. Sub-population breakdowns of the attribution hypothesis space. (a) Growth of [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Cumulative total of model parameters (log scale) from 2019 to 2025, with discrete year [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Compute wall for exhaustive attribution. (a) Log [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 29 canonical work pages

  1. [1]

    Proc Natl Acad Sci USA 111(4):1298– 1303

    Axelrod R, Iliev R (2014) Timing of cyber conflict. Proc Natl Acad Sci USA 111(4):1298– 1303

    work page 2014

  2. [2]

    Proc Natl Acad Sci USA 114(11):2825–2830

    Edwards B, Furnas T, Forrest S, Axelrod R (2017) Strategic aspects of cyberattack, attri- bution, and blame. Proc Natl Acad Sci USA 114(11):2825–2830

    work page 2017

  3. [3]

    Nat Mach Intell 4(3):189–191

    Urbina F, Lentzos F, Invernizzi C, Ekins S (2022) Dual use of AI-powered drug discovery. Nat Mach Intell 4(3):189–191

    work page 2022

  4. [4]

    Perlroth N (2021) This Is How They Tell Me the World Ends: The Cyberweapons Arms Race (Bloomsbury, New York). 20

    work page 2021

  5. [5]

    Xu J, Stokes JW, McDonald G, Bai X, Marshall D, Wang S, Swaminathan A, Li Z (2024) AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber- attacks (arXiv:2403.01038)

    arXiv 2024

  6. [6]

    (2019) Machine behaviour

    Rahwan I, Cebrian M et al. (2019) Machine behaviour. Nature 568(7753):477–486

    work page 2019

  7. [7]

    (OpenAI, https://cdn.openai.com/papers/Governing Agentic AI.pdf)

    Shavit L, Siddarth D, Trager R, Wolf K (2023) Practices for governing agentic AI systems. (OpenAI, https://cdn.openai.com/papers/Governing Agentic AI.pdf)

    work page 2023

  8. [8]

    (2024) Foundational challenges in assuring alignment and safety of large language models

    Anwar U, Saparov A, Rando J, Paleka D et al. (2024) Foundational challenges in assuring alignment and safety of large language models. (arXiv:2307.13744)

    arXiv 2024

Show all 34 references
  1. [9]

    Inf Control 10(5):447–474

    Gold EM (1967) Language identification in the limit. Inf Control 10(5):447–474

    1967

  2. [10]

    Inf Control 45(2):117–135

    Angluin D (1980) Inductive inference of formal languages from positive data. Inf Control 45(2):117–135

    1980

  3. [11]

    Philos Sci 71(4):571–592

    Johnson K (2004) Gold’s theorem and cognitive science. Philos Sci 71(4):571–592

    2004

  4. [12]

    Trans Assoc Comput Linguist 12:543–561

    Strobl L, Merrill W, Weiss G, Chiang D, Angluin D (2024) What formal languages can transformers express? A survey. Trans Assoc Comput Linguist 12:543–561

    2024

  5. [13]

    In Proc EMNLP 2020 , pp

    Bhattamishra S, Ahuja K, Goyal N (2020) On the ability and limitations of transformers to recognize formal languages. In Proc EMNLP 2020 , pp. 7096–7116

    2020

  6. [14]

    In Proc ACL 2020 Workshop on Deep Learning and Formal Languages

    Merrill W (2020) Linear transformations and the capacity of recurrent neural networks. In Proc ACL 2020 Workshop on Deep Learning and Formal Languages

    2020

  7. [15]

    Peng B, Narayanan S, Papadimitriou C (2024) On the limitations of the transformer archi- tecture. In Proc. 1st Conf. on AI for Language Modeling , (to appear)

    2024

  8. [16]

    (Oak Ridge National Lab news release, May 30, 2022)

    ORNL (2022) Frontier supercomputer debuts as world’s fastest, breaking exascale barrier. (Oak Ridge National Lab news release, May 30, 2022)

    2022

  9. [17]

    (NBER Working Paper No

    Bick A, Blandin A, Mallen J (2024) The rapid adoption of generative AI. (NBER Working Paper No. 32966)

    2024

  10. [18]

    Newman MEJ, Barab´ asi AL, Watts DJ (2006) The Structure and Dynamics of Networks (Princeton Univ. Press)

    2006

  11. [19]

    Christakis NA, Fowler JH (2009) Connected: The Surprising Power of Our Social Networks and How They Shape Our Lives (Little, Brown)

    2009

  12. [20]

    (2022) Hiding individuals and communities in a social network.IEEE Trans Netw Sci Eng 9(1):196–209

    Waniek M et al. (2022) Hiding individuals and communities in a social network.IEEE Trans Netw Sci Eng 9(1):196–209

    2022

  13. [21]

    (2022) Trading contact tracing precision for privacy via structure-preserving anonymization

    Waniek M et al. (2022) Trading contact tracing precision for privacy via structure-preserving anonymization. Sci Rep 12:4

    2022

  14. [22]

    (2021) A time-critical crowdsourced computational search for the origins of COVID-19

    Cebrian M et al. (2021) A time-critical crowdsourced computational search for the origins of COVID-19. Nat Electron 4(1):2–4

    2021

  15. [23]

    (arXiv:2404.06757)

    Kleinberg J, Mullainathan S (2024) Language generation in the limit. (arXiv:2404.06757)

    2024 arXiv

  16. [24]

    Bommasani R, Soylu D, Liao TI, Creel KA, Liang P (2024) Ecosystem Graphs: Document- ing the foundation model supply chain. In Proc. 7th AAAI/ACM Conf. AI, Ethics, and Society (AIES 2024) , pp. 196–207

    2024

  17. [25]

    Available at: https://eur-lex.europa.eu/eli/reg/2024/1689/oj

    Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending certain legislative acts 21 (Artificial Intelligence Act), Official Journal of the European Union , OJ L — 12 July 2024....

    2024

  18. [26]

    Order No

    Exec. Order No. 14 110, Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence , 88 Fed. Reg. 75191–75212 (Oct. 30, 2023). Available at: https://www.federalregister.gov/documents/2023/11/02/2023-24766/ safe-secure-and-trustworthy-development-and-use-of-ar...

    2023

  19. [27]

    AI Regulation: A Pro- Innovation Approach

    UK Department for Science, Innovation & Technology. AI Regulation: A Pro- Innovation Approach . White Paper CP 815, 29 March 2023 (updated 3 Au- gust 2023). Available at https://www.gov.uk/government/publications/ ai-regulation-a-pro-innovation-approach/white-paper

    2023

  20. [28]

    Recommendation of the Council on Artificial Intelligence (OECD/LEGAL/0449)

    Organisation for Economic Co-operation and Development (OECD). Recommendation of the Council on Artificial Intelligence (OECD/LEGAL/0449). Adopted 21 May 2019; revised 8 November 2023 and 3 May 2024. Available at https://legalinstruments.oecd.org/ en/instruments/oecd-legal-0449

    2019

  21. [29]

    Global Framework to Monitor Application of the G7 Hiroshima AI Code of Conduct

    Organisation for Economic Co-operation and Development (OECD). Global Framework to Monitor Application of the G7 Hiroshima AI Code of Conduct

  22. [31]

    Here Come the AI Worms

    Matt Burgess. “Here Come the AI Worms.” WIRED, 1 Mar. 2024. Available at https: //www.wired.com/story/here-come-the-ai-worms/

    2024

  23. [32]

    Researchers Say the Deepfake Biden Robocall Was Likely Made With Tools From AI Startup ElevenLabs

    Kate Knibbs. “Researchers Say the Deepfake Biden Robocall Was Likely Made With Tools From AI Startup ElevenLabs.” WIRED, 26 Jan 2024. Available at https://www.wired. com/story/biden-robocall-deepfake-elevenlabs/

    2024

  24. [33]

    Pindrop Reveals TTS Engine Behind Biden AI Robocall

    Vijay Balasubramaniyan. “Pindrop Reveals TTS Engine Behind Biden AI Robocall.” Pin- drop Research Blog, 25 Jan 2024 (updated 16 July 2025). Available at https://www. pindrop.com/article/pindrop-reveals-tts-engine-behind-biden-ai-robocall/

    2024

  25. [34]

    New Hampshire Opens Criminal Probe Into AI Calls Impersonating Biden

    Cat Zakrzewski and Pranshu Verma. “New Hampshire Opens Criminal Probe Into AI Calls Impersonating Biden.” The Washington Post , 6 Feb 2024. Available at https: //www.washingtonpost.com/technology/2024/02/06/nh-robocalls-ai-biden/ . Code availability. All analyses and figures i...

    2024

  26. [2024]

    Available at https://arxiv.org/abs/2403.02817

    arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.