Pith. sign in

REVIEW 12 cited by

Multi-step Jailbreaking Privacy Attacks on ChatGPT

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2304.05197 v3 pith:JT2WLHHR submitted 2023-04-11 cs.CL cs.CR

Haoran Li , Dadi Guo , Wei Fan , Mingshi Xu , Jie Huang , Fanpu Meng , Yangqiu Song This is my paper
classification cs.CLcs.CR
keywords llmsprivacychatgptthreatscontentdatadownstreamai-generated
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

With the rapid progress of large language models (LLMs), many downstream NLP tasks can be well solved given appropriate prompts. Though model developers and researchers work hard on dialog safety to avoid generating harmful content from LLMs, it is still challenging to steer AI-generated content (AIGC) for the human good. As powerful LLMs are devouring existing text data from various domains (e.g., GPT-3 is trained on 45TB texts), it is natural to doubt whether the private information is included in the training data and what privacy threats can these LLMs and their downstream applications bring. In this paper, we study the privacy threats from OpenAI's ChatGPT and the New Bing enhanced by ChatGPT and show that application-integrated LLMs may cause new privacy threats. To this end, we conduct extensive experiments to support our claims and discuss LLMs' privacy implications.

Discussion (0). Sign in to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 23 citations worldwide. Full citation record

  1. MOCHA: Are Code Language Models Robust Against Multi-Turn Malicious Coding Prompts?

    cs.CL 2025-07 conditional novelty 7.0 of 10

    MOCHA is a benchmark of 10.5K malicious coding prompts, including multi-turn decomposition attacks, showing code LLMs reject these incremental attacks at much lower rates and that fine-tuning on the benchmark improves...

  2. Evaluating the Robustness of Retrieval-Augmented Generation to Adversarial Evidence in the Health Domain

    cs.IR 2025-09 conditional novelty 6.0 of 10

    Misleading health documents in RAG context sharply lower LLM accuracy, and heavily helpful-biased retrieval pools restore it.

  3. ASSURE: Metamorphic Testing for AI-powered Browser Extensions

    cs.SE 2025-07 conditional novelty 6.0 of 10

    A modular metamorphic testing framework for LLM-based browser extensions reports 531 automatically detected issues across six real-world extensions.

  4. `For Argument's Sake, Show Me How to Harm Myself!': Jailbreaking LLMs in Suicide and Self-Harm Contexts

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Academic-framing prompts bypass safety filters in most tested LLMs, turning prior self-harm and suicide intent into detailed actionable instructions.

  5. Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation

    cs.CL 2025-08 reject novelty 5.0 of 10

    Sparse autoencoder activation perturbation (SFPF) applied on top of existing jailbreak prompts raises attack success rate on Qwen3-32B, but with no defense evaluation and weak reproducibility.

  6. PII Jailbreaking in LLMs via Activation Steering Reveals Personal Information Leakage

    cs.CR 2025-07 reject novelty 5.0 of 10

    Activation steering on probe-selected attention heads flips LLM privacy refusals into disclosures, with claims of high rates of factually correct personal information leakage.

  7. Context-Aware CodeLLM Eviction for AI-assisted Coding

    cs.SE 2025-06 reject novelty 5.0 of 10

    CACE, a context-aware eviction policy, cuts code-model reloads and response latency in self-hosted AI-assistant serving compared with LRU, though its future-demand factor reads the actual test workload.

  8. SoK: A Comprehensive Security Analysis of Jailbreak Resilience in GPT and DeepSeek Models

    cs.CR 2025-06 conditional novelty 4.0 of 10

    Across 510 HarmBench behaviors and seven attack methods, GPT-4 models show more consistent jailbreak resilience than DeepSeek models, whose vulnerability grows with scale.

  9. System Prompt Extraction Attacks and Defenses in Large Language Models

    cs.CR 2025-05 conditional novelty 4.0 of 10

    A benchmarking study shows that chain-of-thought, few-shot, and modified sandwich queries can recover LLM system prompts with high similarity-based success, and output filtering is the most reliable tested defense.

  10. The Resurgence of GCG Adversarial Attacks on Large Language Models

    cs.CL 2025-08 conditional novelty 3.0 of 10

    The paper reports that GCG attack success drops with model size, prefix-based metrics overestimate harm, and coding prompts are more vulnerable, but these findings rest on a small model set and a GPT-4o pipeline that ...

  11. SoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigation

    cs.CR 2025-06 conditional novelty 3.0 of 10

    A systematization-of-knowledge survey that categorizes LLM privacy risks into training data, prompts, outputs, and agents, and reviews limitations of current mitigations.

  12. From Hallucinations to Jailbreaks: Rethinking the Vulnerability of Large Foundation Models

    cs.CV 2025-05 reject novelty 3.0 of 10

    The paper argues hallucinations and jailbreaks share the same optimization dynamics and shows that defenses for one also reduce the other, but the theoretical support is largely circular.

Pith tools