Pith. sign in

REVIEW 5 major objections 6 minor 65 references

Automated Privacy Information Annotation in Large Language Model Interactions

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper constructs the first large-scale multilingual dataset for privacy detection in real-name LLM interactions — 249,683 queries with 154,230 annotated privacy phrases — using an automated GPT-4o pipeline, and shows fine-tuned local…

desk verdict A genuinely useful bilingual privacy-leakage dataset and reproducible pipeline, with benchmark numbers that rest on thin GPT-4o validation and should be treated as provisional. read the letter →

arxiv 2505.20910 v2 pith:SZOPHXOL submitted 2025-05-27 cs.CL

Hang Zeng , Xiangyu Liu , Yong Hu , Chaoyue Niu , Fan Wu , Shaojie Tang , Guihai Chen This is my paper · ORCID
classification cs.CL
keywords privacyleakagedetectionLLMinteractionautomatedannotationpipelinephraseextractioninformationsummarizationmultilingualdialoguedatasetlocalreal-nameuserinteractions
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that privacy detection for real-name LLM interactions is a new task distinct from PII anonymization: when a user is logged in, a query's real leak is user-linked personal detail — opinions, preferences, intentions, activities — not just identifiers. To enable local, on-device detection, the authors build the first large-scale multilingual dataset (249,683 user queries; 154,230 annotated privacy phrases) using an automated four-step GPT-4o pipeline over public dialogue corpora, with human checks on small samples. They define hierarchical metrics at query, phrase, and information levels, and benchmark locally deployable models. The central empirical claim: fine-tuning on this dataset lets a 1B-parameter model outperform a 72B model used zero-shot, yet the best fine-tuned model still reaches only 44.7% information-level F1, so the problem is far from solved.

What carries the argument

The load-bearing object is the automated four-step annotation pipeline, run with GPT-4o over public dialogue corpora: (1) leakage-or-not classification filters the data; (2) per-sample privacy categories are extracted and iteratively merged into an extensive category set (325 English, 149 Chinese); (3) category-assisted phrase extraction is run in blocks, then deduplicated and filtered by two rules — the phrase must link directly to the user or their close associates, and it must have an explicit reference; (4) a summary statement is generated for each surviving phrase. The evaluation machinery is a three-level metric stack: accuracy for leakage classification, phrase-level precision/recall/F1 with matching by exact containment or Rouge-L>0.5, and information-level F1 built from Rouge-L between generated and reference summaries.

What would settle it

Re-annotate a random sample of about 2,000 queries from the corpus with trained human annotators using the paper's own guidelines, and compare leakage judgments, phrase sets, and information summaries against the GPT-4o labels; if agreement falls materially below the reported small-sample numbers (leakage accuracy about 96%, phrase F1 about 93.7%, information quality about 0.93) or if model rankings on that human-labeled subset differ from rankings on the full corpus, the dataset's role as ground truth is weakened.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the disclosure that matters in real-name LLM interaction is broader than PII: it is any query-borne fact about the user or their close associates that a service provider could newly learn, including opinions, preferences, intentions, health concerns, travel plans, and consumption ability. Constructing a dataset for this requires controlling what counts as user-related: statements about third parties are excluded unless directly linked to the user. The four-step pipeline achieves human-level agreement on the sampled evaluations (96% leakage-classification accuracy; 93.7% phrase F1 after filtering; 0.93 average information-annotation score), and the resulting benchmark shows that locally fine-tuned models clearly beat both prompting-based baselines and traditional PII detectors, while leaving a large gap at information summarization.

Load-bearing premise

The load-bearing premise is that GPT-4o's labels over the full 249K corpus are reliable enough to serve as ground truth, even though the agreement with human annotators was checked on only a few hundred samples.

Editorial extensions

If this is right

  • Fine-tuned 1B-parameter local models beat direct prompting of a 72B model, so a practical on-device privacy alarm is within reach of small models.
  • With 87.6% leakage-classification accuracy, local detection can already decide whether a query needs scrutiny; the bottleneck moves to phrase extraction (74.3% F1) and especially information summarization (44.7% F1).
  • Traditional PII detectors such as Presidio reach only 17–21% phrase F1 on this data, which corroborates that LLM-interaction privacy is a different object from PII in documents.
  • The dataset and metrics supply a training ground for future local privacy detectors, and the reported open gap defines the headroom for that research.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the pipeline transfers to new corpora at the reported cost (about $0.015 per sample), the reusable annotation recipe may matter more than the fixed benchmark: an organization could label its own interaction logs and train a bespoke local detector.
  • The information-level F1 ceiling near 45% may partly reflect Rouge-L's sensitivity to surface wording rather than true semantic failure; an evaluation using multiple acceptable paraphrases could revise the apparent headroom.
  • The definitional exclusion of third-party facts (e.g., 'David works in Shanghai' when David is not linked to the user) is a policy choice; a deployable tool might need tunable relatedness, which would change label distributions and downstream metrics.
  • A direct behavioral test of the framework: deploy a fine-tuned detector as a rewrite prompt and measure how often users accept and send rewritten queries, connecting detection accuracy to actual leakage reduction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper addresses privacy leakage detection for real-name user interactions with LLMs. It proposes a four-step automated annotation pipeline using GPT-4o: leakage classification, privacy-category extraction, privacy-phrase extraction with filtering and deduplication, and privacy-information summarization. Applying this pipeline to ShareGPT and three Chinese dialogue corpora yields a multilingual dataset of 249,647 user queries and 154,230 annotated privacy phrases across three granularity levels. The authors define query-level, phrase-level, and information-level metrics, and benchmark zero-shot prompting, in-context learning, and LoRA-based supervised fine-tuning over Llama-3.2 and Qwen2.5 models. The best fine-tuned Qwen2.5-7B reports 87.56% leakage-classification accuracy, 74.27% phrase-level F1, and 44.70% information-level F1 on the Chinese test set, with slightly lower English numbers, and the paper concludes that local privacy detection remains challenging.

Significance. If the dataset's annotation quality is confirmed, this is a potentially valuable resource: it is much larger than existing PII-oriented privacy datasets, it targets the underexplored setting of real-name LLM interactions, and it ships public code, data, and detailed prompts. The three-level evaluation design and the systematic comparison of zero-shot, in-context learning, and fine-tuned local models are useful for practitioners. The human evaluation is a genuine strength but covers only 800 samples, and the central claims rest on the assumption that GPT-4o labels are reliable over the full 249K corpus. The paper would be significantly stronger if the authors provided a larger, independent, human-validated test set and clarified the data-splitting protocol.

major comments (5)
  1. [Section 4.3; Tables 3-4 and Figure 6] The human validation is too limited and self-referential to support the use of GPT-4o labels as ground truth for the full benchmark. Only 400 classification, 200 phrase-extraction, and 200 information samples are compared with human annotation, with no inter-annotator agreement, no per-language stratification for the phrase and information sets, no confidence intervals, and no adjudication of disagreements. The information-level check is a subjective 'semantically coherent' score rather than a match against independent ground truth. Because the same GPT-4o labels are the reference for every number in Tables 3-4 and Figure 6, any systematic GPT-4o bias propagates directly into the headline 87.6% accuracy, 74.3% phrase F1, and 44.7% information F1. I recommend constructing a held-out human-annotated test set with per-language and per-category breakdowns, reporting agreement metrics (e.g., Cohen's kappa) and confidence intervals, and scoring the final baselines on that set as well as on the large automatically labeled test set.
  2. [Section 4.3 (raw datasets); Section 3 (problem definition)] The source corpora do not match the claimed real-name LLM interaction scenario. Three of the four corpora are human-human dialogues, and the adaptation is to arbitrarily designate the first speaker as the target user and the second speaker as the assistant, while none of the corpora contain a real unique identifier or an LLM in the loop. This weakens the claim of constructing the first large-scale dataset for privacy detection in real-name LLM interactions and could systematically bias both the leakage rate and the privacy-category distribution. I ask the authors to either validate the pipeline on genuinely real-name LLM interaction data (even a small sample) or explicitly scope the dataset as simulated LLM interactions and analyze the domain shift.
  3. [Section 5.1 (dataset splitting)] The English data split is described only as an 8:2 ratio, without specifying whether the split is performed at the dialogue level or at the individual user-query level. Since each dialogue contains multiple user queries that share conversational context, a random query-level split can place near-duplicate or topically identical content in both training and test sets, which would inflate the reported baseline scores. The Chinese split, by contrast, follows the original dataset splits and is presumably dialogue-level. The authors should clarify the English splitting unit and, if it is query-level, re-run the evaluation with a dialogue-level split.
  4. [Section 5.3 and Table 3] The privacy leakage classification evaluation reports only accuracy, which is insufficient for this imbalanced binary task. The test sets are roughly 2:1 non-leak to leak (e.g., 12,792 vs. 6,658 English samples in Table 6), so an all-non-leak classifier already achieves about 66-67% accuracy. More importantly, for a privacy-alerting application, false negatives are the critical failure mode, and accuracy alone can hide large recall differences between models. I request precision, recall, F1, and ideally an operating-characteristic analysis for the leakage classification task.
  5. [Section 4.4 (phrase matching)] The phrase-level metrics depend on a matching rule that counts two phrases as a match if one is a substring of the other or if their Rouge-L score exceeds 0.5. This threshold is a free parameter, and no sensitivity analysis or empirical justification is provided. Since every phrase-level number in Table 4 and the derived F1 values depend on this choice, the authors should report how RP, PP, and F1P vary with the threshold (e.g., 0.4, 0.5, 0.6) or justify the selected value using independent human judgments.
minor comments (6)
  1. [Abstract vs. Section 4.3/Table 6] The abstract reports 249,683 user queries, but the numbers in Section 4.3 and Table 6 sum to 249,647 (97,659 English + 151,988 Chinese). Please correct the discrepancy.
  2. [Section 4.4 and references] The citation for the Rouge-L matching rule is given as [18], which is the paper by Huang et al. on retrieval-based language models rather than the original ROUGE paper. Also, references [29] and [30] are the same Lison et al. work and should be deduplicated.
  3. [Section 5.3] The sentence stating that SFT with Qwen2.5-7B outperforms ZG and ICL by 14.59% and 15.35% mixes English and Chinese numbers; the English improvements are 14.59% over ZG and 15.29% over ICL, while the Chinese improvements are 17.89% and 15.35%. Please report both languages explicitly.
  4. [Section 4.3] The initial leakage classification counts (over 41K leaked / 56K non-leaked for English, 64K/88K for Chinese) are not reconciled with the final dataset counts (32,814 leaked / 64,845 non-leaked for English, 43,255/108,733 for Chinese). Please state clearly that the difference comes from phrase-based reclassification of samples with no extracted phrases.
  5. [Figure 6] The bar chart is difficult to read: each model has three bars with numeric labels that are not clearly separated, and there is no legend or axis label. Please add a legend, axis labels, and consider providing the exact numerical values in a table as well.
  6. [Section 4.2] The claim that using only one-tenth of the data for privacy-category extraction yields nearly saturated diversity is unsupported by any quantitative evidence; please provide a saturation curve or temper the claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the dataset labels are externally human-validated on small samples, and the SFT evaluation uses held-out labels rather than the training labels.

full rationale

The paper constructs a privacy annotation dataset using GPT-4o and then evaluates local models against held-out GPT-4o labels. This is not circular in the derivation sense: the evaluation labels are not identical to the training labels, and the local models are distinct systems that can and do make errors. The paper provides independent human-annotation checks on 400 classification, 200 phrase-extraction, and 200 information-annotation samples (Section 4.3), which ground the GPT-4o labels externally. The concern that GPT-4o bias may propagate into the benchmark is a data-quality and validity risk, not a circularity of the paper's derivation chain. There are no load-bearing self-citations, no uniqueness theorems imported from the authors, no fitted parameters renamed as predictions, and no ansatz smuggled in via citation. The phrase-level and information-level metrics compare model outputs to the annotated ground truth in a standard supervised-evaluation setup; the reported scores are conditional on label quality, which is an explicit and testable assumption rather than a circular reduction.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central contribution is a dataset, so the main ledger entries are the assumptions that make GPT-4o labels trustworthy and that make the source corpora representative of real-name LLM interactions. The only hand-chosen evaluation parameter is the Rouge-L phrase-matching threshold; training hyperparameters are standard and not load-bearing for the dataset claim. No new physical or formal entities are introduced.

free parameters (1)
  • Phrase matching Rouge-L threshold = 0.5
    Hand-chosen threshold in Section 4.4 for treating two phrases as matching; directly affects reported phrase-level precision, recall, and F1 scores.
assumptions (4)
  • domain assumption Privacy leakage in real-name LLM interactions is correctly defined as any user-related private detail such as opinions, preferences, activities, and intentions, beyond traditional PII.
    Section 3 sets this definition and the annotation prompts operationalize it; the paper does not validate this definition against user expectations, legal privacy norms, or service-provider data practices.
  • ad hoc to paper Human-human dialogue corpora can stand in for real-name LLM interactions by designating the first speaker as the target user and the second speaker as the assistant.
    Section 4.3 and Appendix B.1 apply this mapping to CrossWOZ, DuConv, and LCCC; no evidence shows these dialogues resemble real-name user-LLM query behavior.
  • domain assumption GPT-4o annotations are accurate enough on the full corpus to serve as ground truth for training and evaluating local detectors.
    The entire pipeline in Section 4.2 relies on GPT-4o for leakage classification, category extraction, phrase extraction, filtering, and information summarization; human agreement is checked on only 800 samples in Section 4.3 and Table 2.
  • domain assumption Exact overlap, substring containment, and Rouge-L score are adequate proxies for semantic equivalence of privacy phrases and privacy information.
    Section 4.4 defines matching using set intersection, substring containment, and Rouge-L greater than 0.5; this may overcount semantically different or undercount paraphrased phrases.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automated Privacy Information Annotation in Large Language Model Interactions." pith.science (2026). https://pith.science/paper/SZOPHXOL

@misc{pith2026250520910,
  author       = {Pith},
  title        = {Pith review of: Automated Privacy Information Annotation in Large Language Model Interactions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SZOPHXOL}},
  note         = {Machine review of arXiv:2505.20910}
}
read the original abstract

Users interacting with large language models (LLMs) under their real identifiers often unknowingly risk disclosing private information. Automatically notifying users whether their queries leak privacy and which phrases leak what private information has therefore become a practical need. Existing privacy detection methods, however, were designed for different objectives and application domains, typically tagging personally identifiable information (PII) in anonymous content, which is insufficient in real-name interaction scenarios with LLMs. In this work, to support the development and evaluation of privacy detection models for LLM interactions that are deployable on local user devices, we construct a large-scale multilingual dataset with 249K user queries and 154K annotated privacy phrases. In particular, we build an automated privacy annotation pipeline with strong LLMs to automatically extract privacy phrases from dialogue datasets and annotate leaked information. We also design evaluation metrics at the levels of privacy leakage, extracted privacy phrase, and privacy information. We further establish baseline methods using light-weight LLMs with both tuning-free and tuning-based methods, and report a comprehensive evaluation of their performance. Evaluation results reveal a gap between current performance and the requirements of real-world LLM applications, motivating future research into more effective local privacy detection methods grounded in our dataset.

Figures

Figures reproduced from arXiv: 2505.20910 by the authors.

Figure 1
Figure 1. An illustration of privacy exposure and annotation [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Automated privacy annotation pipeline with strong [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of our automated pipeline to extract privacy phrase and annotate privacy information over dialogue [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (23 more)
Figure 4
Figure 4. Figure 4: Analyses of privacy information in ShareGPT [ [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Analyses of privacy information in CrossWOZ [ [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Performance of different privacy information summarization baselines from information-level F1 (%). [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Case study with Qwen2.5-7B-Instruct. SFT success [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: A failure case of Qwen2.5-7B-Instruct with SFT. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Prompt template for privacy leakage classification for English and Chinese dataset, where <|INPUT|> is the input of [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Prompt template for privacy categories extraction in step 2 for English and Chinese datasets, where <|INPUT|> is the [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Prompt template for categories deduplication in step 2 for English and Chinese dataset, where <|INPUT|> is the [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Prompt template for privacy phrase extraction with categories in step 3 for English and Chinese datasets, where [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Prompt template for privacy phrase deduplication in step 3 for English and Chinese dataset, where <|INPUT|> is the [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: Prompt template for filtering privacy phrase with rule 1 in step 3 for English and Chinese dataset, where <|INPUT|> [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: Prompt template for filtering privacy phrase with rule 2 in step 3 for English and Chinese dataset, where <|INPUT|> [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: Prompt template for privacy information annotation in step 4 for English and Chinese dataset, where <|INPUT|> is [PITH_FULL_IMAGE:figures/full_fig_p021_16.png]
Figure 17
Figure 17. Figure 17: Data samples with privacy phrase and privacy information. The samples come from one raw dialogue. [PITH_FULL_IMAGE:figures/full_fig_p022_17.png]
Figure 18
Figure 18. Figure 18: Prompt template for zero-shot generation of privacy leakage classification task in English and Chinese dataset, [PITH_FULL_IMAGE:figures/full_fig_p023_18.png]
Figure 19
Figure 19. Figure 19: Prompt template for zero-shot generation of privacy phrase extraction task in English and Chinese dataset, where [PITH_FULL_IMAGE:figures/full_fig_p024_19.png]
Figure 20
Figure 20. Figure 20: Prompt template for zero-shot generation of privacy information summarization task in English and Chinese dataset, [PITH_FULL_IMAGE:figures/full_fig_p025_20.png]
Figure 21
Figure 21. Figure 21: Prompt template for in-context learning of privacy leakage classification task in English and Chinese dataset, where [PITH_FULL_IMAGE:figures/full_fig_p026_21.png]
Figure 22
Figure 22. Figure 22: Prompt template for in-context learning of privacy phrase extraction task in English and Chinese dataset, where [PITH_FULL_IMAGE:figures/full_fig_p026_22.png]
Figure 23
Figure 23. Figure 23: Prompt template for in-context learning of privacy information summarization task in English and Chinese dataset, [PITH_FULL_IMAGE:figures/full_fig_p027_23.png]
Figure 24
Figure 24. Figure 24: Instructions for training models for privacy leakage classification task, where <|INPUT|> is the input of the user [PITH_FULL_IMAGE:figures/full_fig_p027_24.png]
Figure 25
Figure 25. Figure 25: Instructions for training models for privacy phrase extraction task, where <|INPUT|> is the input of user query. [PITH_FULL_IMAGE:figures/full_fig_p028_25.png]
Figure 26
Figure 26. Figure 26: Instructions for training models for privacy information summarization task, where <|INPUT|> is the input of the [PITH_FULL_IMAGE:figures/full_fig_p028_26.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 35 canonical work pages

  1. [1]

    Chandan Akiti, Anna Cinzia Squicciarini, and Sarah Michele Rajtmajer. 2020. A Semantics-based Approach to Disclosure Classification in User-Generated Online Content. InFindings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020 (Findings of ACL, Vol. EMNLP 2020), Trevor Cohn, Yulan He, and Yang Liu (Eds.). Asso...

    work page doi:10.18653/v1/2020.findings-emnlp.312 2020

  2. [2]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    2020

  3. [3]

    Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. 2023. Quantifying Memorization Across Neu- ral Language Models. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. https://openreview.net/forum?id=TatRHT_1cK

    2023

  4. [4]

    Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel

    Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting Training Data from Large Lan- guage Models. In30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021, Michael D. Bailey and Rachel ...

    2021

  5. [5]

    Franck Dernoncourt, Ji Young Lee, Özlem Uzuner, and Peter Szolovits. 2017. De-identification of patient notes with recurrent neural networks.J. Am. Medical Informatics Assoc.24, 3 (2017), 596–606. doi:10.1093/JAMIA/OCW156

    doi:10.1093/jamia/ocw156 2017

  6. [6]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023. A Survey for In-context Learning. CoRRabs/2301.00234 (2023). arXiv:2301.00234 doi:10.48550/ARXIV.2301.00234

    arXiv doi:10.48550/arxiv.2301.00234 2023

  7. [7]

    Yao Dou, Isadora Krsek, Tarek Naous, Anubha Kabra, Sauvik Das, Alan Ritter, and Wei Xu. 2024. Reducing Privacy Risks in Online Self-Disclosures with Language Models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, Lun-Wei Ku, Andre Martins, ...

    doi:10.18653/v1/2024 2024

  8. [8]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ah- mad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sra- vankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bet...

    arXiv doi:10.48550/arxiv.2407.21783 2024

Show all 65 references
  1. [9]

    Elisabeth Eder, Ulrike Krieg-Holz, and Udo Hahn. 2020. CodE Alltag 2.0 - A Pseudonymized German-Language Email Corpus. InProceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020, Nicoletta Calzolari, Frédéric Béchet, P...

    2020

  2. [10]

    Polra Victor Falade. 2023. Decoding the Threat Landscape : ChatGPT, FraudGPT, and WormGPT in Social Engineering Attacks.CoRRabs/2310.05595 (2023). arXiv:2310.05595 doi:10.48550/ARXIV.2310.05595

    2023 doi arXiv

  3. [11]

    Symptom severity prediction from neuropsychiatric clinical records: Overview of 2016 CEGS N-GRID shared tasks Track 2

    Michele Filannino, Amber Stubbs, and Özlem Uzuner. 2018. Corrigendum to "Symptom severity prediction from neuropsychiatric clinical records: Overview of 2016 CEGS N-GRID shared tasks Track 2" [J Biomed Inform. 2017 Nov;75S: S62-S70].J. Biomed. Informatics85 (2018), 204. doi:10...

    2018 doi

  4. [12]

    Rajitha Hathurusinghe, Isar Nejadgholi, and Miodrag Bolic. 2021. A Privacy- Preserving Approach to Extraction of Personal Information through Auto- matic Annotation and Federated Learning.CoRRabs/2105.09198 (2021). arXiv:2105.09198 https://arxiv.org/abs/2105.09198

    2021 arXiv

  5. [13]

    Jiyan He, Xuechen Li, Da Yu, Huishuai Zhang, Janardhan Kulkarni, Yin Tat Lee, Arturs Backurs, Nenghai Yu, and Jiang Bian. 2023. Exploring the Limits of Differentially Private Deep Learning with Group-wise Clipping. InThe Eleventh International Conference on Learning Representa...

    2023

  6. [14]

    Guntur Budi Herwanto, Gerald Quirchmayr, and A Min Tjoa. 2021. A Named Entity Recognition Based Approach for Privacy Requirements Engineering. In 29th IEEE International Requirements Engineering Conference Workshops, RE 2021 Workshops, Notre Dame, IN, USA, September 20-24, 202...

    2021

  7. [15]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 202...

    2022

  8. [17]

    Qiushi Huang, Shuai Fu, Xubo Liu, Wenwu Wang, Tom Ko, Yu Zhang, and Lil- ian Tang. 2023. Learning Retrieval Augmentation for Personalized Dialogue Generation. InProceedings of the 2023 Conference on Empirical Methods in Nat- ural Language Processing, EMNLP 2023, Singapore, Dec...

    2023 doi

  9. [18]

    Yangsibo Huang, Samyak Gupta, Zexuan Zhong, Kai Li, and Danqi Chen. 2023. Privacy Implications of Retrieval-Based Language Models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, Houda Bouamo...

    2023 doi

  10. [19]

    Shotaro Ishihara. 2023. Training Data Extraction From Pre-trained Language Models: A Survey.CoRRabs/2305.16157 (2023). arXiv:2305.16157 doi:10.48550/ ARXIV.2305.16157

    2023 doi arXiv

  11. [20]

    Kristian Nørgaard Jensen, Mike Zhang, and Barbara Plank. 2021. De-identification of Privacy-related Entities in Job Postings. InProceedings of the 23rd Nordic Con- ference on Computational Linguistics, NoDaLiDa 2021, Reykjavik, Iceland (Online), May 31 - June 2, 2021, Simon Do...

    2021

  12. [21]

    Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022. Deduplicating Training Data Mitigates Privacy Risks in Language Models. InInternational Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA (Proceedings of Machine Learning Research, Vol. 16...

    2022

  13. [22]

    Siwon Kim, Sangdoo Yun, Hwaran Lee, Martin Gubri, Sungroh Yoon, and Seong Joon Oh. 2023. ProPILE: Probing Privacy Leakage in Large Language Mod- els. InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIP...

    2023

  14. [23]

    Klein, Abeed Sarker, Masoud Rouhizadeh, Karen O’Connor, and Graciela Gonzalez

    Ari Z. Klein, Abeed Sarker, Masoud Rouhizadeh, Karen O’Connor, and Graciela Gonzalez. 2017. Detecting Personal Medication Intake in Twitter: An Anno- tated Corpus and Baseline Classification System. InBioNLP 2017, Vancouver, Canada, August 4, 2017, Kevin Bretonnel Cohen, Dina ...

    2017 doi

  15. [24]

    Jooyoung Lee, Sarah Rajtmajer, Eesha Srivatsavaya, and Shomir Wilson. 2023. Online Self-Disclosure, Social Support, and User Engagement During the COVID- 19 Pandemic.ACM Trans. Soc. Comput.6, 3-4 (2023), 1–31. doi:10.1145/3617654

    2023 doi

  16. [25]

    Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022. Deduplicating Training KDD ’26, August 9-13, 2026, Jeju, Korea Zeng et al. Data Makes Language Models Better. InProceedings of the 60th Annual Meeting ...

    2022

  17. [26]

    Marvin Li, Jason Wang, Jeffrey Wang, and Seth Neel. 2023. MoPe: Model Per- turbation based Privacy Attacks on Language Models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Associatio...

    2023 doi

  18. [27]

    Siyan Li, Vethavikashini Chithrra Raghuram, Omar Khattab, Julia Hirschberg, and Zhou Yu. 2025. PAPILLON: Privacy Preservation from Internet-based and Local Language Model Ensembles. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association ...

    2025

  19. [28]

    Xuechen Li, Florian Tramèr, Percy Liang, and Tatsunori Hashimoto. 2022. Large Language Models Can Be Strong Differentially Private Learners. InThe Tenth In- ternational Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. https:/...

    2022

  20. [30]

    Pierre Lison, Ildikó Pilán, David Sánchez, Montserrat Batet, and Lilja Øvrelid

  21. [31]

    Microsoft. 2020. Presidio: An open-source framework for PII data detection and anonymization. https://github.com/microsoft/presidio. Accessed: 2025-07-26

    2020

  22. [32]

    Anonymisation Models for Text Data: State of the art, Challenges and Future Directions. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volum...

    2021 doi

  23. [33]

    Anthi Papadopoulou, Yunhao Yu, Pierre Lison, and Lilja Øvrelid. 2022. Neural Text Sanitization with Explicit Measures of Privacy Risk. InProceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Jo...

    2022

  24. [34]

    Niloofar Mireshghallah, Maria Antoniak, Yash More, Yejin Choi, and Golnoosh Farnadi. 2024. Trust No Bot: Discovering Personal Disclosures in Human-LLM Conversations in the Wild.CoRRabs/2407.11438 (2024). arXiv:2407.11438 doi:10. 48550/ARXIV.2407.11438

    2024 doi arXiv

  25. [35]

    Daniel Preotiuc-Pietro, Vasileios Lampos, and Nikolaos Aletras. 2015. An analysis of the user occupational class through Twitter content. InProceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Nat...

    2015

  26. [36]

    Ildikó Pilán, Pierre Lison, Lilja Øvrelid, Anthi Papadopoulou, David Sánchez, and Montserrat Batet. 2022. The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymization.Comput. Linguistics 48, 4 (2022), 1053–1101. doi:10.1162/COLI_A_00458

    2022 doi

  27. [37]

    Nafis Sadeq, Zhouhang Xie, Byungkyu Kang, Prarit Lamba, Xiang Gao, and Julian J. McAuley. 2024. Mitigating Hallucination in Fictional Character Role-Play. CoRRabs/2406.17260 (2024). arXiv:2406.17260 doi:10.48550/ARXIV.2406.17260

    2024 doi arXiv

  28. [38]

    Rahime Belen Saglam, Jason R. C. Nurse, and Duncan Hodges. 2021. Privacy Concerns in Chatbot Interactions: When to Trust and When to Worry. InHCI International 2021 - Posters - 23rd HCI International Conference, HCII 2021, Virtual Event, July 24-29, 2021, Proceedings, Part II ...

    2021 doi

  29. [39]

    Ann-Katrin Reuel, Sebastian Peralta, João Sedoc, Garrick Sherman, and Lyle H. Ungar. 2022. Measuring the Language of Self-Disclosure across Corpora. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, Smaranda Muresan, Pres...

    2022 doi

  30. [40]

    ShareGPT. 2023. ShareGPT Dataset. https://huggingface.co/datasets/shareAI/ ShareGPT-Chinese-English-90k

    2023

  31. [41]

    Victoria Smith, Ali Shahin Shamsabadi, Carolyn Ashurst, and Adrian Weller. 2023. Identifying and Mitigating Privacy Risks Stemming from Language Models: A Survey.CoRRabs/2310.01424 (2023). arXiv:2310.01424 doi:10.48550/ARXIV.2310. 01424

    2023 doi arXiv

  32. [42]

    Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. 2023. Character-LLM: A Trainable Agent for Role-Playing. InProceedings of the 2023 Conference on Empir- ical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, Houda Bouamor, Juan Pino, and Ka...

    2023 doi

  33. [43]

    Robin Staab, Mark Vero, Mislav Balunovic, and Martin T. Vechev. 2024. Beyond Memorization: Violating Privacy via Inference with Large Language Models. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net...

    2024

  34. [44]

    Amber Stubbs and Özlem Uzuner. 2015. Annotating longitudinal clinical narra- tives for de-identification: The 2014 i2b2/UTHealth corpus.J. Biomed. Informatics 58 (2015), S20–S29. doi:10.1016/J.JBI.2015.07.020

    2015 doi

  35. [45]

    Haoyu Song, Weinan Zhang, Yiming Cui, Dong Wang, and Ting Liu. 2019. Exploit- ing Persona Information for Diverse Generation of Conversational Responses. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intel- ligence, IJCAI 2019, Macao, China, ...

    2019 doi

  36. [46]

    Albert Yu Sun, Eliott Zemour, Arushi Saxena, Udith Vaidyanathan, Eric Lin, Christian Lau, and Vaikkunth Mugunthan. 2023. Does fine-tuning GPT-3 with the OpenAI API leak personally-identifiable information?CoRRabs/2307.16382 (2023). arXiv:2307.16382 doi:10.48550/ARXIV.2307.16382

    2023 doi arXiv

  37. [47]

    Feiyang Tang and Bjarte M. Østvold. 2024. Finding Privacy-Relevant Source Code. InIEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2024 - Companion, Rovaniemi, Finland, March 12, 2024. IEEE, 111–118. doi:10.1109/SANER-C62648.2024.00020

    2024

  38. [48]

    Nishant Subramani, Sasha Luccioni, Jesse Dodge, and Margaret Mitchell. 2023. Detecting Personal Information in Training Corpora: an Analysis. InProceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023), Anaelia Ovalle, Kai-Wei Chang, Ninareh Me...

    2023

  39. [49]

    Tesfay, Jetzabel M

    Welderufael B. Tesfay, Jetzabel M. Serna, and Kai Rannenberg. 2019. PrivacyBot: Detecting Privacy Sensitive Information in Unstructured Texts. InSixth Interna- tional Conference on Social Networks Analysis, Management and Security, SNAMS 2019, Granada, Spain, October 22-25, 20...

    2019

  40. [50]

    Manuel Tonneau, Dhaval Adjodah, João Palotti, Nir Grinberg, and Samuel Fraiberger. 2022. Multilingual Detection of Personal Employment Status on Twit- ter. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 20...

    2022 doi

  41. [51]

    Mina Valizadeh, Xing Qian, Pardis Ranjbar-Noiey, Cornelia Caragea, and Natalie Parde. 2023. What Clued the AI Doctor In? On the Influence of Data Source and Quality for Transformer-Based Medical Self-Disclosure Detection. InProceedings of the 17th Conference of the European Ch...

    2023 doi

  42. [52]

    Inan, Janardhan Kulkarni, and Xia Hu

    Ruixiang Tang, Gord Lueck, Rodolfo Quispe, Huseyin A. Inan, Janardhan Kulkarni, and Xia Hu. 2023. Assessing Privacy Risks in Language Models: A Case Study on Summarization Tasks. InFindings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10,...

    2023

  43. [53]

    Yida Wang, Pei Ke, Yinhe Zheng, Kaili Huang, Yong Jiang, Xiaoyan Zhu, and Minlie Huang. 2020. A Large-Scale Chinese Short-Text Conversation Dataset. InNatural Language Processing and Chinese Computing - 9th CCF International Conference, NLPCC 2020, Zhengzhou, China, October 14...

    2020 doi

  44. [54]

    Wenquan Wu, Zhen Guo, Xiangyang Zhou, Hua Wu, Xiyuan Zhang, Rongzhong Lian, and Haifeng Wang. 2019. Proactive Human-Machine Conversation with Explicit Conversation Goals.CoRRabs/1906.05572 (2019). arXiv:1906.05572 http://arxiv.org/abs/1906.05572

    2019 arXiv

  45. [55]

    Qiongkai Xu, Lizhen Qu, Zeyu Gao, and Gholamreza Haffari. 2020. Personal Information Leakage Detection in Conversations. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, Bonnie Webber, Trevor Co...

    2020

  46. [56]

    Mina Valizadeh, Pardis Ranjbar-Noiey, Cornelia Caragea, and Natalie Parde. 2021. Identifying Medical Self-Disclosure in Online Communities. InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn...

    2021

  47. [57]

    Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Eric Sun, and Yue Zhang. 2023. A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly.CoRRabs/2312.02003 (2023). arXiv:2312.02003 doi:10.48550/ ARXIV.2312.02003

    2023 doi arXiv

  48. [58]

    Inan, Gau- tam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang

    Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A. Inan, Gau- tam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. 2022. Differentially Private Fine-tuning of Language Models. InThe Tenth International Co...

    2022

  49. [59]

    Hanna Yukhymenko, Robin Staab, Mark Vero, and Martin Vechev. 2024. A Synthetic Dataset for Personal Attribute Inference. InAdvances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Bel- grave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37. Cu...

    2024

  50. [60]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Cheng- peng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzhe...

    2024 arXiv

  51. [61]

    Ghost of the past

    Shuning Zhang, Lyumanshan Ye, Xin Yi, Jingyu Tang, Bo Shui, Haobin Xing, Pengfei Liu, and Hewu Li. 2024. "Ghost of the past": identifying and resolving privacy leakage from LLM’s memory through proactive user interaction.CoRR abs/2410.14931 (2024). arXiv:2410.14931 doi:10.4855...

    2024 doi arXiv

  52. [62]

    Xinyan Zhao, Deahan Yu, and V. G. Vinod Vydiswaran. 2019. Identifying Adverse Drug Events Mentions in Tweets Using Attentive, Collocated, and Aggregated Medical Representation. InProceedings of the Fourth Social Media Mining for Health Application Workshop & Shared Task, SMM4H...

    2019 doi

  53. [63]

    Qi Zhu, Kaili Huang, Zheng Zhang, Xiaoyan Zhu, and Minlie Huang. 2020. Cross- WOZ: A Large-Scale Chinese Cross-Domain Task-Oriented Dialogue Dataset. Trans. Assoc. Comput. Linguistics8 (2020), 281–295. doi:10.1162/TACL_A_00314 KDD ’26, August 9-13, 2026, Jeju, Korea Zeng et al...

    2020 doi

  54. [64]

    Hanna Yukhymenko, Robin Staab, Mark Vero, and Martin T. Vechev. 2024. A Synthetic Dataset for Personal Attribute Inference. InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Can...

    2024

  55. [68]

    true” or “false

    is a task-oriented dialogue dataset covering five domains, con- taining 6K dialogues and 47K user queries. DuConv [54] focuses on movie and film star topics, with 22K dialogues and 99K user queries. Both CrossWOZ and DuConv data are collected through dialogues conducted by two...

    2026

  56. [220]

    doi:10.18653/v1/2023.trustnlp-1.18

    2023 doi

  57. [1764]

    doi:10.3115/V1/P15-1169

    doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.