Pith. sign in

REVIEW 3 major objections 4 minor 101 references

Synthetic Artifact Auditing: Tracing LLM-Generated Synthetic Data Usage in Downstream Applications

T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that an auditor can determine whether a classifier, generator, or statistical plot was built from LLM-generated synthetic data using query access and independently collected reference data, with no proprietary training…

desk verdict Solid new-task paper with credible in-domain results; the headline claims outrun the evidence on metric selection and cross-architecture transfer. read the letter →

arxiv 2502.00808 v1 pith:RY2TCRTU submitted 2025-02-02 cs.LG cs.CRcs.CY

Yixin Wu , Ziqing Yang , Yun Shen , Michael Backes , Yang Zhang This is my paper · ORCID
classification cs.LGcs.CRcs.CY
keywords syntheticartifactauditingLLM-generateddataprovenancemodelmembershipinferencet-SNEvisualizationtextclassificationsummarization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces synthetic artifact auditing: a third party asks whether a given artifact—a text classifier, a summarization generator, or a t-SNE plot—was trained on or derived from LLM-generated synthetic data, and answers without the artifact owner disclosing training data or hyperparameters. It proposes three methods: metric-based auditing, which thresholds the artifact's average confidence, entropy, or accuracy on real and synthetic queries; tuning-based auditing, which optimizes a small set of embedding queries and feeds the outputs to a meta-classifier; and classification-based auditing, which trains an image classifier on t-SNE plots. Across three text classification tasks, two summarization tasks, two visualization tasks, and three training-data scenarios, the paper reports average auditing accuracy of $0.868 \pm 0.071$ for classifiers and $0.880 \pm 0.052$ for generators with 200 black-box queries, and $0.966 \pm 0.003$ for t-SNE plots. The motivation is that synthetic data can carry bias, hallucination, and unauthorized-use risks, so knowing whether an artifact was built from it matters for transparency and regulatory compliance.

What carries the argument

The load-bearing object is the reference artifact set: real and synthetic classifiers, generators, or t-SNE plots that the auditor builds independently from a reference real dataset and an LLM-generated synthetic dataset. This set plays three roles: it fixes the threshold in metric-based auditing, it trains the meta-classifier over output posteriors in tuning-based auditing, and it trains the image classifier in classification-based auditing. The mechanism that makes the references transferable to the target is the assumed behavioral disparity between synthetic-trained and real-trained artifacts—confidence and entropy differences on synthetic queries for classifiers, summarization-score differences on real queries for generators, and decision-boundary sharpness in t-SNE scatter for plots.

What would settle it

Train a target classifier on synthetic data from a domain the auditor did not anticipate, such as a legal-document task, while the auditor's reference classifiers are trained on sentiment reviews; if the auditing accuracy falls to chance, the reference-transfer assumption fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that synthetic-data involvement leaves a detectable trace in the artifact itself, not just in the data. A classifier fine-tuned on LLM-generated text tends to be more confident on synthetic inputs and less confident on real inputs than a classifier trained on real text, so its average confidence on a 200-item synthetic query set separates the two populations well enough for a threshold to decide. The same logic transfers to generators by substituting summarization quality metrics such as ROUGE-L, BLEU, and BERTScore for confidence. For t-SNE plots, the trace is visual: synthetic points from different classes form sharper decision boundaries, and an image classifier on grayscale plots distinguishes synthetic-derived from real-derived plots at high accuracy. The paper argues this holds in three training scenarios—100 percent synthetic data, a controlled mix of real and synthetic data, and a randomized multi-LLM mix—and that the auditor needs no information from the artifact owner beyond either black-box query access or the plot image itself.

Load-bearing premise

The whole method depends on the auditor being able to build reference real and synthetic artifacts that behave like the target artifact, because the audit threshold or meta-classifier is learned from those references and then applied to the target.

Editorial extensions

If this is right

  • A regulator or downstream user can check a deployed small model for synthetic-data use with only API queries and independently gathered data, without the owner's training set.
  • The same metric-based recipe covers both discriminative and generative artifacts: swapping the performance metric adapts the audit from classifiers to summarization generators.
  • Published statistical analyses can be audited from their t-SNE plots alone, which extends provenance checking to results where no model is released.
  • The reference-based design can also identify which specific LLM generated the training data, supporting unauthorized-use investigations rather than only a synthetic-versus-real label.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the confidence-disparity signal should weaken as synthetic data is mixed with more real text or generated by models that closely mimic human style; the reported numbers likely give the upper end of achievable accuracy.
  • I infer the reference-transfer design could be adapted to other modalities, such as images or code, but each domain would need its own query metric and calibration before the threshold transfers.
  • Because the method already distinguishes four source LLMs, it could grow into a broader provenance fingerprinting tool that attributes an artifact to a specific generator.
  • A natural testable extension is active query selection: choosing the most informative examples might preserve the reported accuracy with far fewer than 200 queries.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces synthetic artifact auditing: given a downstream artifact (a text classifier, a text summarization generator, or a t-SNE statistical plot), determine whether it was trained on or derived from LLM-generated synthetic data, without requiring the artifact owner to disclose training details. Three auditing methods are proposed: metric-based auditing (black-box, using classifier confidence or entropy thresholds, or summarization quality scores), tuning-based auditing (white-box, learning a small set of continuous query embeddings and a meta-classifier on reference classifiers), and classification-based auditing (an image classifier trained on reference t-SNE plots). The evaluation covers three text classification tasks, two summarization tasks, and two t-SNE visualization tasks, using GPT-3.5, GPT-4, Mistral, and ChatGLM3 as synthetic data sources, across three training scenarios (100% synthetic, mixed real/synthetic, and multi-LLM mixed). Headline results include 0.868 +/- 0.071 average accuracy for black-box metric-based classifier auditing with 200 queries, 0.880 +/- 0.052 for generator auditing, and 0.966 +/- 0.003 for t-SNE plot auditing.

Significance. If the proposed methods transfer from reference artifacts to real-world targets, this is a practically useful contribution to model transparency and regulatory compliance, especially in light of recent disclosure laws for synthetic data. The paper is empirically extensive and includes several good practices: balanced target/reference sets, multiple LLMs, three data-mix scenarios, an open-source code release, and a robustness study in Section 8 that varies the synthetic-data source LLM and the real-data distribution. However, the central inference of the framework is that thresholds and meta-classifiers learned on reference artifacts transfer to targets, and this transfer is never tested across architecture, task domain, fine-tuning recipe, or visualization pipeline. The reported accuracies are therefore credible only for the matched-reference conditions actually evaluated. The work is a promising first step, but the claimed generality is not yet established.

major comments (3)
  1. [Section 4.5 (Table 1) and Section 4.6] The load-bearing premise is that a threshold or meta-classifier learned on reference artifacts transfers to the target artifact. Every evaluation keeps the artifact family identical between references and targets: BERT/DistilBERT for classifiers, BART for generators, and the same Word2Vec/GloVe + t-SNE + 300x300 PNG pipeline for plots. Section 8 relaxes only the synthetic-data source LLM and the real-data distribution (Rotten Tomatoes vs. IMDB); it never tests a target with a different base architecture, a different task domain, a different fine-tuning recipe, or a different plot-generation pipeline. Section 9 explicitly states that LLM targets and additional tasks were not evaluated, yet Section 10 concludes general feasibility. Without a cross-architecture or cross-task transfer experiment (e.g., references trained on DistilBERT and targets fine-tuned from RoBERTa, or references built from Word2Vec and targets from UMAP), the reported 0.868/0.880/0.966 accuracies do not support the claimed black-box auditor capability in Section 3.
  2. [Section 9 and Section 10] The choice of Confidence for Qsyn and Entropy for Qreal is made on the same target classifier sets that are later used for the main evaluation. Section 4.5.1 and Section 4.5.2 average results over 'three tasks and two pre-trained models in S1 and S2' to select these metrics, and Section 4.6 then uses exactly those choices to report the headline classifier-auditing accuracy of 0.868 +/- 0.071. Because the metric selection and the final evaluation share the same target instances, the reported numbers are optimistically biased. An independent validation split (or nested selection) should be used to support the headline claim.
  3. [Section 4.3.2] Section 9 concedes that evaluations on LLM targets and additional tasks are missing due to infrastructure constraints, but then asserts that the methods 'are expected to be feasible for LLMs and generalizable to other tasks.' This expectation is not evidence, and the conclusion restates it as a general capability. Since the methods rely on fine-tuned PLM confidence calibration, synthetic-data lexical structure, and t-SNE geometry, all of which may differ for LLMs, non-English tasks, or other domains, the generalizability claim in Section 10 should be either removed or supported by at least one out-of-domain experiment, such as a smaller LLM target or a task outside the three evaluated domains.
minor comments (4)
  1. [Figure 12 and Figure 13 captions] The captions contain a typo: 'Auditing Acuuracy' should be 'Auditing Accuracy'.
  2. [Section 4.3.2] The sentence 'We also leave out 1,000 samples as Qaux from the retrieved movies as additional information to constructing Qsyn' should read 'for constructing Qsyn'.
  3. [Appendix B.2] In the TC1 data-split description, the phrase 'The rest of the retrieved movies are evenly split into Daux target and Daux target' appears to contain a typo; the second split is presumably Daux_ref.
  4. [Equation (4)] The notation in Equation (4), specifically 'Cre f,theta', is unclear and appears to be a rendering artifact for 'C_ref,theta'; please clean up the subscript formatting.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: thresholds and meta-classifiers are calibrated on reference artifacts and applied to independently constructed targets; the matched-reference design is an external-validity concern, not a circularity.

full rationale

The paper's derivation chain is not circular. Metric-based auditing fits a threshold on reference classifiers and then applies that threshold to target classifiers that were not used in fitting; tuning-based auditing trains a meta-classifier on reference artifacts and evaluates it on held-out targets; plot auditing trains an image classifier on reference plots and tests it on target plots. These are standard supervised generalization procedures, not reductions of the prediction to the fit. The matched-reference design (same BERT/DistilBERT/BART base models, same tasks, same synthetic-data settings, and same t-SNE pipeline for references and targets) limits external validity, but it is a threat to generalization, not a logical loop: the paper does not define the target label in terms of the reference fit, nor does it rename a fitted parameter as a prediction. Section 8 tests some mismatches (different source LLM, different real-data distribution), and Section 9 explicitly concedes that LLM-scale targets and additional tasks were not evaluated, which is an honest limitation rather than a circular dependence. The self-citations present, such as [38] (MGTBench) and [94] (t-SNE plotting procedure), are background or procedural: [38] supports the intuition that synthetic text has detectable features but the auditing hypothesis is empirically tested here, and [94] only standardizes the plot-rendering pipeline. No uniqueness theorem, ansatz, or defining equation is imported from the authors' prior work in a load-bearing way. Hence no circular step can be exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests mainly on the existence of a synthetic/real behavioral gap and on the auditor's ability to construct matched reference artifacts. The only fitted numeric quantity called out in the method is the threshold tau; the other entries are design choices (query budgets, temperatures, reference counts) that affect performance but are not hidden 'predictions'. No new entities are postulated.

free parameters (5)
  • Threshold tau for metric-based auditing = task/metric-dependent, not reported as a global number
    Equations (2) and (3) classify by comparing mean confidence to tau; tau is chosen to maximize accuracy on reference classifiers, making it a fitted calibration parameter.
  • Number of reference artifacts = 20 for metric-based, 100 for tuning-based
    Selected after preliminary experiments on the target sets; more references improve stability but increase cost.
  • Synthetic data generation temperature = 0.5 for TC1, 1.0 for other tasks
    Set by hand to balance diversity and quality; the resulting data distribution shapes the audit signal.
  • Query budgets = 200 random queries for metric-based, 5 tuned queries for tuning-based
    Chosen from preliminary results showing diminishing returns beyond these values.
  • t-SNE and embedding hyperparameters = not specified
    The paper says standard procedures are adopted but does not report perplexity, iterations, or Word2Vec/GloVe dimension; plot auditing results may depend on these choices.
assumptions (4)
  • domain assumption LLM-generated synthetic text differs from real text in features that measurably affect downstream artifact behavior.
    Section 4.1 invokes prior detection results and Figure 15 shows separable t-SNE clusters; the audit signal depends entirely on this gap.
  • domain assumption The auditor can build reference artifacts that match the target artifact in task, architecture, and data distribution.
    Sections 3 and 4.1 require reference real data from a similar distribution and reference synthetic data from the same source LLM or a similar one; all evaluation scenarios satisfy this by construction.
  • domain assumption Pre-trained backbones (BERT, DistilBERT, BART, RN18) used for references are representative of real-world targets and were not contaminated with synthetic data.
    Section 8 argues contamination is improbable because pretraining predates ChatGPT, but the method cannot distinguish a target's synthetic-data usage from pretraining contamination if that premise fails.
  • standard math Standard fine-tuning hyperparameters and random seeds produce reference artifacts whose behavior distribution is close to plausible target artifacts.
    Appendices B.5-B.7 specify the training protocol; the threshold transfer assumes these references are a valid stand-in for the unknown target's training regime.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Synthetic Artifact Auditing: Tracing LLM-Generated Synthetic Data Usage in Downstream Applications." pith.science (2026). https://pith.science/paper/RY2TCRTU

@misc{pith2026250200808,
  author       = {Pith},
  title        = {Pith review of: Synthetic Artifact Auditing: Tracing LLM-Generated Synthetic Data Usage in Downstream Applications},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RY2TCRTU}},
  note         = {Machine review of arXiv:2502.00808}
}
abstract

Large language models (LLMs) have facilitated the generation of high-quality, cost-effective synthetic data for developing downstream models and conducting statistical analyses in various domains. However, the increased reliance on synthetic data may pose potential negative impacts. Numerous studies have demonstrated that LLM-generated synthetic data can perpetuate and even amplify societal biases and stereotypes, and produce erroneous outputs known as ``hallucinations'' that deviate from factual knowledge. In this paper, we aim to audit artifacts, such as classifiers, generators, or statistical plots, to identify those trained on or derived from synthetic data and raise user awareness, thereby reducing unexpected consequences and risks in downstream applications. To this end, we take the first step to introduce synthetic artifact auditing to assess whether a given artifact is derived from LLM-generated synthetic data. We then propose an auditing framework with three methods including metric-based auditing, tuning-based auditing, and classification-based auditing. These methods operate without requiring the artifact owner to disclose proprietary training details. We evaluate our auditing framework on three text classification tasks, two text summarization tasks, and two data visualization tasks across three training scenarios. Our evaluation demonstrates the effectiveness of all proposed auditing methods across all these tasks. For instance, black-box metric-based auditing can achieve an average accuracy of $0.868 \pm 0.071$ for auditing classifiers and $0.880 \pm 0.052$ for auditing generators using only 200 random queries across three scenarios. We hope our research will enhance model transparency and regulatory compliance, ensuring the ethical and responsible use of synthetic data.

Figures

Figures reproduced from arXiv: 2502.00808 by the authors.

Figure 1
Figure 1. Overview of the synthetic artifact auditing. The au [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the metric-based auditing. cally fine-tune pre-trained language models (PLMs) for spe￾cific tasks. Therefore, we consider that our artifacts are fine￾tuned from PLMs, such as BERT [26] and BART [49]. More￾over, we do not consider the scenario where auditing targets are LLMs for the following reasons. First, in real-world ap￾plications, efficiency, cost-effectiveness, and customizability are essential. Sm… view at source ↗
Figure 3
Figure 3. Overview of the tuning-based auditing. reference classifiers, the auditor leverages a simple gradient￾based approach where they directly optimize the query set Qφ parameterized by φ and a meta-classifier Mω1 parametrized by ω1 via backpropagation. The meta-classifier uses the out￾put probabilities (posteriors) from a given classifier to predict its assigned label. More formally, the auditor aims to maxi￾mize the lik… view at source ↗
Figures from the paper (24 more)
Figure 4
Figure 4. Figure 4: Overview of target/reference classifier setup. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Metric-based auditing performance for target classi [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Metric-based auditing performance for target classi [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Auditing performance for target classifiers fine-tuned on the pre-trained DistilBERT model using metric-based auditing with [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Auditing performance for target classifiers fine-tuned on the pre-trained DistilBERT model using metric-based auditing with [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Auditing performance for target classifiers fine [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 11
Figure 11. Figure 11: Metric-based auditing performance for target gener [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]
Figure 12
Figure 12. Figure 12: Auditing performance for target generators fine [PITH_FULL_IMAGE:figures/full_fig_p010_12.png]
Figure 14
Figure 14. Figure 14: Overview of the classification-based auditing. [PITH_FULL_IMAGE:figures/full_fig_p011_14.png]
Figure 16
Figure 16. Figure 16: Auditing performance on TC1 with different source LLMs in S2. Dreal target is derived from IMDB. Dreal ref is (a) disjoint data from IMDB, and (b) data from Rotten Tomatoes. or random (metric-based) queries to audit target artifacts. 8 Discussion Data Contamination. T…
Figure 19
Figure 19. Figure 19: Metric-based auditing performance for target clas [PITH_FULL_IMAGE:figures/full_fig_p020_19.png]
Figure 20
Figure 20. Figure 20: Metric-based auditing performance for target clas [PITH_FULL_IMAGE:figures/full_fig_p020_20.png]
Figure 17
Figure 17. Figure 17: Metric-based auditing performance for target clas [PITH_FULL_IMAGE:figures/full_fig_p020_17.png]
Figure 18
Figure 18. Figure 18: Metric-based auditing performance for target clas [PITH_FULL_IMAGE:figures/full_fig_p020_18.png]
Figure 27
Figure 27. Figure 27: Tuning-based auditing performance for target clas [PITH_FULL_IMAGE:figures/full_fig_p021_27.png]
Figure 28
Figure 28. Figure 28: Auditing performance for target classifiers fine [PITH_FULL_IMAGE:figures/full_fig_p021_28.png]
Figure 29
Figure 29. Figure 29: Auditing performance for target classifiers fine [PITH_FULL_IMAGE:figures/full_fig_p021_29.png]
Figure 26
Figure 26. Figure 26: Tuning-based auditing performance for target clas [PITH_FULL_IMAGE:figures/full_fig_p021_26.png]
Figure 32
Figure 32. Figure 32: Metric-based auditing performance for target gen [PITH_FULL_IMAGE:figures/full_fig_p022_32.png]
Figure 33
Figure 33. Figure 33: T-SNE plots using GloVe with different synthetic [PITH_FULL_IMAGE:figures/full_fig_p022_33.png]
Figure 34
Figure 34. Figure 34: Average classification performance of reference real [PITH_FULL_IMAGE:figures/full_fig_p022_34.png]
Figure 35
Figure 35. Figure 35: Average target performance of real and synthetic classifiers ( [PITH_FULL_IMAGE:figures/full_fig_p023_35.png]
Figure 36
Figure 36. Figure 36: Average target performance of real and synthetic classifiers ( [PITH_FULL_IMAGE:figures/full_fig_p024_36.png]
Figure 37
Figure 37. Figure 37: Average target performance of real and synthetic generators fine-tuned on pre-trained BART for (a) [PITH_FULL_IMAGE:figures/full_fig_p024_37.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

101 extracted references · 55 canonical work pages

  1. [1]

    https://www.theverge.com/2023/12/15/ 24003151/bytedance- china- openai- microsoft- competitor-llm

    ByteDance is Secretly Using OpenAI’s Tech to Build a Com- petitor. https://www.theverge.com/2023/12/15/ 24003151/bytedance- china- openai- microsoft- competitor-llm. 2

    2023

  2. [2]

    https://www.maye rbrown.com/en/insights/publications/2024/09/ca lifornia-passes-new-generative-artificial-inte lligence-law-requiring-disclosure-of-training- data

    California Passes New Generative Artificial Intelligence Law Requiring Disclosure of Training Data. https://www.maye rbrown.com/en/insights/publications/2024/09/ca lifornia-passes-new-generative-artificial-inte lligence-law-requiring-disclosure-of-training- data. 2

    2024

  3. [3]

    https://openai.com/index/introducing- canvas/

    Canvas. https://openai.com/index/introducing- canvas/. 1

  4. [4]

    https://github.com/THUDM/ChatGLM3

    ChatGLM3. https://github.com/THUDM/ChatGLM3. 5

  5. [5]

    https://platform.openai.com/docs/m odels/gpt-3-5-turbo

    GPT-3.5-Turbo. https://platform.openai.com/docs/m odels/gpt-3-5-turbo . 5

  6. [6]

    https://platform.openai.com/docs/models/ gpt-4-turbo-and-gpt-4

    GPT-4. https://platform.openai.com/docs/models/ gpt-4-turbo-and-gpt-4 . 5

  7. [7]

    https://hazy.com/

    Hazy Synthetic Data Framework. https://hazy.com/. 1

  8. [8]

    https://www.imdb.com/

    IMDB. https://www.imdb.com/. 4, 17

Show all 101 references
  1. [9]

    https://leginfo.legislature.ca.gov /faces/billNavClient.xhtml?bill_id=202320240AB

    Law AB2013. https://leginfo.legislature.ca.gov /faces/billNavClient.xhtml?bill_id=202320240AB

  2. [10]

    https://ai.meta.com/llama/licens e/

    LLaMA License. https://ai.meta.com/llama/licens e/. 2, 12

  3. [11]

    https://huggingface.co/mis tralai/Mistral-7B-Instruct-v0.2

    Mistral-7B-Instruct-v0.2. https://huggingface.co/mis tralai/Mistral-7B-Instruct-v0.2 . 5

  4. [12]

    https://openai.com/policie s/business-terms/

    OpenAI Business License. https://openai.com/policie s/business-terms/. 2

  5. [13]

    https://openai.com/policies/row- terms-of-use/

    OpenAI License. https://openai.com/policies/row- terms-of-use/. 12

  6. [14]

    UK ICO. https://ico.org.uk/for- organisation s/uk-gdpr-guidance-and-resources/artificial- intelligence/explaining-decisions-made-with-ar tificial-intelligence/part-3-what-explaining- ai-means-for-your-organisation/documentation/ . 2

  7. [15]

    An Empirical Study of Clinical Note Generation from Doctor-Patient Encounters

    Asma Ben Abacha, Wen-wai Yim, Yadan Fan, and Thomas Lin. An Empirical Study of Clinical Note Generation from Doctor-Patient Encounters. In Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 2283–2294. ACL, 2023. 1 13

    2023

  8. [16]

    On the difficulty of training Recurrent Neural Networks

    Razvan Pascanu andTomás Mikolov and Yoshua Bengio. On the difficulty of training Recurrent Neural Networks. In In- ternational Conference on Machine Learning (ICML), pages 1310–1318. JMLR, 2013. 2

    2013

  9. [17]

    METEOR: An Auto- matic Metric for MT Evaluation with Improved Correlation with Human Judgments

    Satanjeev Banerjee and Alon Lavie. METEOR: An Auto- matic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Annual Meeting of the Asso- ciation for Computational Linguistics (ACL) , pages 65–72. ACL, 2005. 13

    2005

  10. [18]

    André Bauer, Simon Trapp, Michael Stenger, Robert Lep- pich, Samuel Kounev, Mark Leznik, Kyle Chard, and Ian T. Foster. Comprehensive Exploration of Synthetic Data Gen- eration: A Survey. CoRR abs/2401.02524, 2024. 2

    2024 arXiv

  11. [19]

    Deciphering Tex- tual Authenticity: A Generalized Strategy through the Lens of Large Language Semantics for Detecting Human vs

    Mazal Bethany, Brandon Wherry, Emet Bethany, Nishant Vishwamitra, and Peyman Najafirad. Deciphering Tex- tual Authenticity: A Generalized Strategy through the Lens of Large Language Semantics for Detecting Human vs. Machine-Generated Text. CoRR abs/2401.09407, 2024. 11, 12

    2024 arXiv

  12. [20]

    Membership Inference Attacks From First Principles

    Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramèr. Membership Inference Attacks From First Principles. In IEEE Symposium on Secu- rity and Privacy (S&P), pages 1897–1914. IEEE, 2022. 12

    1914

  13. [21]

    Dif- ferentially private sequential data publication via variable- length n-grams

    Rui Chen, Gergely Acs, and Claude Castelluccia. Dif- ferentially private sequential data publication via variable- length n-grams. In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 638–649. ACM,

  14. [22]

    GPT-Sentinel: Distinguishing Human and ChatGPT Generated Content

    Yutian Chen, Hao Kang, Vivian Zhai, Liangze Li, Rita Singh, and Bhiksha Raj. GPT-Sentinel: Distinguishing Human and ChatGPT Generated Content. CoRR abs/2305.07969, 2023. 12

    2023 arXiv

  15. [23]

    Medically Aware GPT-3 as a Data Genera- tor for Medical Dialogue Summarization

    Bharath Chintagunta, Namit Katariya, Xavier Amatriain, and Anitha Kannan. Medically Aware GPT-3 as a Data Genera- tor for Medical Dialogue Summarization. In Proceedings of the Machine Learning for Healthcare Conference (MLHC) , pages 354–372. PMLR, 2021. 5

    2021

  16. [24]

    What Is Synthetic Data? The Good, The Bad, and The Ugly

    Emiliano De Cristofaro. What Is Synthetic Data? The Good, The Bad, and The Ugly. CoRR abs/2303.01230, 2023. 1

    2023 arXiv

  17. [25]

    Under the Surface: Tracking the Artifactuality of LLM-Generated Data

    Debarati Das, Karin de Langis, Anna Martin-Boyle, Jae- hyung Kim, Minhwa Lee, Zae Myung Kim, Shirley Anu- grah Hayati, Risako Owan, Bin Hu, Ritik Parkar, Ryan Koo, Jong Inn Park, Aahan Tyagi, Libby Ferland, Sanjali Roy, Vincent Liu, and Dongyeop Kang. Under the Surface: Tracki...

    2024 arXiv

  18. [26]

    BERT: Pre-training of Deep Bidirectional Trans- formers for Language Understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Trans- formers for Language Understanding. In Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (NA...

    2019

  19. [27]

    Data Augmentation using LLMs: Data Perspectives, Learning Paradigms and Challenges

    Bosheng Ding, Chengwei Qin, Ruochen Zhao, Tianze Luo, Xinze Li, Guizhen Chen, Wenhan Xia, Junjie Hu, Anh Tuan Luu, and Shafiq Joty. Data Augmentation using LLMs: Data Perspectives, Learning Paradigms and Challenges. CoRR abs/2403.02990, 2024. 1

    2024 arXiv

  20. [28]

    Feature-based detection of automated language models: tackling GPT-2, GPT-3 and Grover

    Leon Fröhling and Arkaitz Zubiaga. Feature-based detection of automated language models: tackling GPT-2, GPT-3 and Grover. PeerJ Computer Science, 2021. 12

    2021

  21. [29]

    Gallegos, Ryan A

    Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md. Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. Bias and Fairness in Large Language Models: A Survey. CoRR abs/2309.00770, 2023. 1

    2023 arXiv

  22. [30]

    Gunter, and Nikita Borisov

    Karan Ganju, Qi Wang, Wei Yang, Carl A. Gunter, and Nikita Borisov. Property Inference Attacks on Fully Con- nected Neural Networks using Permutation Invariant Rep- resentations. In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 619–633. ACM,

  23. [31]

    Self-Guided Noise-Free Data Genera- tion for Efficient Zero-Shot Learning

    Jiahui Gao, Renjie Pi, Yong Lin, Hang Xu, Jiacheng Ye, Zhiyong Wu, Weizhong Zhang, Xiaodan Liang, Zhenguo Li, and Lingpeng Kong. Self-Guided Noise-Free Data Genera- tion for Efficient Zero-Shot Learning. In International Con- ference on Learning Representations (ICLR), 2023. 5

    2023

  24. [32]

    Sebastian Gehrmann, Hendrik Strobelt, and Alexander M. Rush. GLTR: Statistical Detection and Visualization of Gen- erated Text. In Annual Meeting of the Association for Com- putational Linguistics (ACL), pages 111–116. ACL, 2019. 12

    2019

  25. [33]

    Deep Learning

    Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. The MIT Press, 2016. 2

    2016

  26. [34]

    How Close is ChatGPT to Human Experts? Comparison Corpus, Evalu- ation, and Detection

    Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. How Close is ChatGPT to Human Experts? Comparison Corpus, Evalu- ation, and Detection. CoRR abs/2301.07597, 2023. 12

    2023 arXiv

  27. [35]

    Generative AI for Synthetic Data Generation: Methods, Challenges and the Future

    Xu Guo and Yiqiang Chen. Generative AI for Synthetic Data Generation: Methods, Challenges and the Future. CoRR abs/2403.04190, 2024. 1, 3

    2024 arXiv

  28. [36]

    Eval- uating Large Language Models in Generating Synthetic HCI Research Data: a Case Study

    Perttu Hämäläinen, Mikke Tavast, and Anton Kunnari. Eval- uating Large Language Models in Generating Synthetic HCI Research Data: a Case Study. InAnnual ACM Conference on Human Factors in Computing Systems (CHI). ACM, 2023. 1

    2023

  29. [37]

    Deep Residual Learning for Image Recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778. IEEE, 2016. 11

    2016

  30. [38]

    MGTBench: Benchmarking Machine- Generated Text Detection

    Xinlei He, Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. MGTBench: Benchmarking Machine- Generated Text Detection. In ACM SIGSAC Conference on Computer and Communications Security (CCS) . ACM,

  31. [39]

    Tar- geted Data Generation: Finding and Fixing Model Weak- nesses

    Zexue He, Marco Túlio Ribeiro, and Fereshte Khani. Tar- geted Data Generation: Finding and Fixing Model Weak- nesses. In Annual Meeting of the Association for Compu- tational Linguistics (ACL), pages 8506–8520. ACL, 2023. 1, 3, 5

    2023

  32. [40]

    Synthetic data generation for tab- ular health records: A systematic review

    Mikel Hernandez, Gorka Epelde, Ane Alberdi, Rodrigo Cilla, and Debbie Rankin. Synthetic data generation for tab- ular health records: A systematic review. Neurocomputing,

  33. [41]

    On the Utility of Synthetic Data: An Empirical Evaluation on Machine Learning Tasks

    Markus Hittmeir, Andreas Ekelhart, and Rudolf Mayer. On the Utility of Synthetic Data: An Empirical Evaluation on Machine Learning Tasks. In Conference on Availability, Re- liability and Security (ARES). ACM, 2019. 1

    2019

  34. [42]

    Yu, and Xuyun Zhang

    Hongsheng Hu, Zoran Salcic, Lichao Sun, Gillian Dobbie, Philip S. Yu, and Xuyun Zhang. Membership Inference At- tacks on Machine Learning: A Survey.ACM Computing Sur- veys, 2021. 3

    2021

  35. [43]

    Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhut- dinov, and Eric P. Xing. Toward Controlled Generation of 14 Text. In International Conference on Machine Learning (ICML), pages 1587–1596. PMLR, 2017. 2

    2017

  36. [44]

    Survey of Hallucination in Natural Language Generation

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pas- cale Fung. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 2023. 2

    2023

  37. [45]

    Exploiting Asymmetry for Synthetic Train- ing Data Generation: SynthIE and the Case of Information Extraction

    Martin Josifoski, Marija Sakota, Maxime Peyrard, and Robert West. Exploiting Asymmetry for Synthetic Train- ing Data Generation: SynthIE and the Case of Information Extraction. In Conference on Empirical Methods in Natu- ral Language Processing (EMNLP), pages 1555–1574. ACL,

  38. [46]

    Analyzing and Reducing the Damage of Dataset Bias to Face Recognition With Synthetic Data

    Adam Kortylewski, Bernhard Egger, Andreas Schneider, Thomas Gerig, Andreas Morel-Forster, and Thomas Vetter. Analyzing and Reducing the Damage of Dataset Bias to Face Recognition With Synthetic Data. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages ...

    2019

  39. [47]

    Hadas Kotek, Rikker Dockum, and David Q. Sun. Gender bias and stereotypes in Large Language Models. In ACM Collective Intelligence Conference (CI). ACM, 2023. 1

    2023

  40. [48]

    Tahmid Rahman Laskar, Mizanur Rahman, Israt Jahan, Enamul Hoque, and Jimmy Huang

    Md. Tahmid Rahman Laskar, Mizanur Rahman, Israt Jahan, Enamul Hoque, and Jimmy Huang. CQSumDP: A ChatGPT- Annotated Resource for Query-Focused Abstractive Sum- marization Based on Debatepedia. CoRR abs/2305.06147,

    arXiv

  41. [49]

    BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvinine- jad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In Annual Meeting of the Associa-...

    2020

  42. [50]

    Membership Leakage in Label- Only Exposures

    Zheng Li and Yang Zhang. Membership Leakage in Label- Only Exposures. In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 880–895. ACM,

  43. [51]

    Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations

    Zhuoyan Li, Hangxiao Zhu, Zhuoran Lu, and Ming Yin. Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations. In Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pages 10443–10461. ACL, 2023. 2, 5

    2023

  44. [52]

    ROUGE: A Package for Automatic Evalua- tion of Summaries

    Chin-Yew Lin. ROUGE: A Package for Automatic Evalua- tion of Summaries. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 74–81. ACL, 2004. 9

    2004

  45. [53]

    Is Your Code Generated by ChatGPT Really Cor- rect? Rigorous Evaluation of Large Language Models for Code Generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is Your Code Generated by ChatGPT Really Cor- rect? Rigorous Evaluation of Large Language Models for Code Generation. In Annual Conference on Neural Informa- tion Processing Systems (NeurIPS). NeurIPS, 2023. 2

    2023

  46. [54]

    Low-Resource Court Judgment Summariza- tion for Common Law Systems

    Shuaiqi Liu, Jiannong Cao, Yicong Li, Ruosong Yang, and Zhiyuan Wen. Low-Resource Court Judgment Summariza- tion for Common Law Systems. CoRR abs/2403.04454 ,

    arXiv

  47. [55]

    Check Me If You Can: Detecting ChatGPT-Generated Academic Writing using CheckGPT

    Zeyan Liu, Zijun Yao, Fengjun Li, and Bo Luo. Check Me If You Can: Detecting ChatGPT-Generated Academic Writing using CheckGPT. CoRR abs/2306.05524, 2023. 3, 5, 10, 11, 12

    2023 arXiv

  48. [56]

    Zero-Resource Hal- lucination Prevention for Large Language Models

    Junyu Luo, Cao Xiao, and Fenglong Ma. Zero-Resource Hal- lucination Prevention for Large Language Models. In Con- ference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 3586–3602. Association for Computa- tional Linguistics, 2024. 2

    2024

  49. [57]

    Maas, Raymond E

    Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y . Ng, and Christopher Potts. Learning Word Vectors for Sentiment Analysis. In Annual Meeting of the Association for Computational Linguistics (ACL) , pages 142–150. ACL, 2011. 17

    2011

  50. [58]

    JOBSKAPE: A Framework for Generating Synthetic Job Postings to Enhance Skill Match- ing

    Antoine Magron, Anna Dai, Mike Zhang, Syrielle Montar- iol, and Antoine Bosselut. JOBSKAPE: A Framework for Generating Synthetic Job Postings to Enhance Skill Match- ing. CoRR abs/2402.03242, 2024. 1

    2024 arXiv

  51. [59]

    Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. Self- CheckGPT: Zero-Resource Black-Box Hallucination De- tection for Generative Large Language Models. CoRR abs/2303.08896, 2023. 2

    2023 arXiv

  52. [60]

    Spam Filtering with Naive Bayes - Which Naive Bayes? In Conference on Email and Anti-Spam (CEAS)

    Vangelis Metsis, Ion Androutsopoulos, and Georgios Paliouras. Spam Filtering with Naive Bayes - Which Naive Bayes? In Conference on Email and Anti-Spam (CEAS) . CEAS, 2006. 4, 17

    2006

  53. [61]

    Efficient Estimation of Word Representations in Vec- tor Space

    Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vec- tor Space. In International Conference on Learning Repre- sentations (ICLR), 2013. 11

    2013

  54. [62]

    Manning, and Chelsea Finn

    Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christo- pher D. Manning, and Chelsea Finn. DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curva- ture. CoRR abs/2301.11305, 2023. 11, 12

    2023 arXiv

  55. [63]

    AgentInstruct: Toward Generative Teaching with Agentic Flows

    Arindam Mitra, Luciano Del Corro, Guoqing Zheng, Shweti Mahajan, Dany Rouhana, Andrés Codas, Yadong Lu, Weige Chen, Olga Vrousgos, Corby Rosset, Fillipe Silva, Hamed Khanpour, Yash Lara, and Ahmed Awadallah. AgentInstruct: Toward Generative Teaching with Agentic Flows. CoRR ab...

    2024 arXiv

  56. [64]

    The Parrot Dilemma: Human-Labeled vs

    Anders Møller, Arianna Pera, Jacob Aarup Dalsgaard, and Luca Maria Aiello. The Parrot Dilemma: Human-Labeled vs. LLM-augmented Data in Classification Tasks. InConfer- ence of the European Chapter of the Association for Compu- tational Linguistics (EACL), pages 179–192. ACL, 2024. 3, 5

    2024

  57. [65]

    Enhanc- ing Automated Scoring of Math Self-Explanation Quality Using LLM-Generated Datasets: A Semi-Supervised Ap- proach

    Ryosuke Nakamoto, Brendan Flanagan, Taisei Yamauchi, Yiling Dai, Kyosuke Takami, and Hiroaki Ogata. Enhanc- ing Automated Scoring of Math Self-Explanation Quality Using LLM-Generated Datasets: A Semi-Supervised Ap- proach. Computers, 2023. 1, 3

    2023

  58. [66]

    Cohen, and Mirella Lapata

    Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don’t Give Me the Details, Just the Summary! Topic-Aware Con- volutional Neural Networks for Extreme Summarization. In Conference on Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 1797–1807. ACL, 2018. 9

    2018

  59. [67]

    Ma- chine Learning with Membership Privacy using Adversarial Regularization

    Milad Nasr, Reza Shokri, and Amir Houmansadr. Ma- chine Learning with Membership Privacy using Adversarial Regularization. In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 634–646. ACM,

  60. [68]

    Tieu, Huy H

    Hoang-Quoc Nguyen-Son, Ngoc-Dung T. Tieu, Huy H. Nguyen, Junichi Yamagishi, and Isao Echizen. Identifying computer-generated text using statistical analysis. In Asia- Pacific Signal and Information Processing Association An- nual Summit and Conference (APSIPA) , pages 1504–151...

    2017

  61. [69]

    Seeing Stars: Exploiting Class Relationships for Sentiment Categorization with Respect to Rating Scales

    Bo Pang and Lillian Lee. Seeing Stars: Exploiting Class Relationships for Sentiment Categorization with Respect to Rating Scales. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 115–124. ACL,

  62. [70]

    Bleu: a Method for Automatic Evaluation of Machine Translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a Method for Automatic Evaluation of Machine Translation. In Annual Meeting of the Association for Com- putational Linguistics (ACL), pages 311–318. ACL, 2002. 9

    2002

  63. [71]

    Smith, Nima M

    Cheng Peng, Xi Yang, Aokun Chen, Kaleb E. Smith, Nima M. Pournejatian, Anthony B. Costa, Cheryl Martin, Mona G. Flores, Ying Zhang, Tanja Magoc, Gloria P. Li- pori, Duane A. Mitchell, Naykky Singh Ospina, Mustafa M. Ahmed, William R. Hogan, Elizabeth A. Shenkman, Yi Guo, Jiang...

    2023

  64. [72]

    Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global Vectors for Word Representation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. ACL, 2014. 11

    2014

  65. [73]

    Boost- ing Instance Segmentation with Synthetic Data: A study to overcome the limits of real world data sets

    Florentin Poucin, Andrea Kraus, and Martin Simon. Boost- ing Instance Segmentation with Synthetic Data: A study to overcome the limits of real world data sets. InIEEE Interna- tional Conference on Computer Vision Workshops (ICCVW), pages 945–953. IEEE, 2021. 2

    2021

  66. [74]

    Language Models are Unsuper- vised Multitask Learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsuper- vised Multitask Learners. OpenAI blog, 2019. 9

    2019

  67. [75]

    Sheth, and Amitava Das

    Vipula Rawte, Amit P. Sheth, and Amitava Das. A Sur- vey of Hallucination in Large Foundation Models. CoRR abs/2309.05922, 2023. 2

    2023 arXiv

  68. [76]

    ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models

    Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models. In Network and Distributed System Security Symposium (NDSS). Internet So- ciety...

    2019

  69. [77]

    DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR abs/1910.01108, 2019. 5

    1910 arXiv

  70. [78]

    Liu, and Christopher D

    Abigail See, Peter J. Liu, and Christopher D. Manning. Get To The Point: Summarization with Pointer-Generator Net- works. In Annual Meeting of the Association for Computa- tional Linguistics (ACL), pages 1073–1083. ACL, 2017. 9

    2017

  71. [79]

    In ChatGPT We Trust? Measuring and Character- izing the Reliability of ChatGPT

    Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. In ChatGPT We Trust? Measuring and Character- izing the Reliability of ChatGPT. CoRR abs/2304.08979 ,

    arXiv

  72. [80]

    Membership Inference Attacks Against Machine Learning Models

    Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership Inference Attacks Against Machine Learning Models. In IEEE Symposium on Security and Pri- vacy (S&P), pages 3–18. IEEE, 2017. 3, 12

    2017

  73. [81]

    Systematic Evaluation of Privacy Risks of Machine Learning Models

    Liwei Song and Prateek Mittal. Systematic Evaluation of Privacy Risks of Machine Learning Models. In USENIX Se- curity Symposium (USENIX Security). USENIX, 2021. 3

    2021

  74. [82]

    Does Synthetic Data Generation of LLMs Help Clinical Text Mining? CoRR abs/2303.04360, 2023

    Ruixiang Tang, Xiaotian Han, Xiaoqian Jiang, and Xia Hu. Does Synthetic Data Generation of LLMs Help Clinical Text Mining? CoRR abs/2303.04360, 2023. 1, 3, 5

    2023 arXiv

  75. [83]

    Large language models in medicine

    Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabi- lan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature Medicine, 2023. 1

    2023

  76. [84]

    Factoring Variations in Natural Images with Deep Gaussian Mixture Models

    Aäron van den Oord and Benjamin Schrauwen. Factoring Variations in Natural Images with Deep Gaussian Mixture Models. In Annual Conference on Neural Information Pro- cessing Systems (NIPS), pages 3518–3526. NIPS, 2014. 2

    2014

  77. [85]

    Visualizing Data using t-SNE

    Laurens van der Maaten and Geoffrey Hinton. Visualizing Data using t-SNE. Journal of Machine Learning Research ,

  78. [86]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Annual Con- ference on Neural Information Processing Systems (NIPS) , pages 5998–6008. NIPS, 2017. 2

    2017

  79. [87]

    Junda Wang, Zonghai Yao, Avijit Mitra, Samuel Os- ebe, Zhichao Yang, and Hong Yu. UMASS_BioNLP at MEDIQA-Chat 2023: Can LLMs generate high-quality syn- thetic note-oriented doctor-patient conversations? In Clini- cal Natural Language Processing Workshop (ClinicalNLP), pages 46...

    2023

  80. [88]

    M4GT-Bench: Evaluation Benchmark for Black-Box Machine-Generated Text Detection

    Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Osama Mohanned Afzal, Tarek Mahmoud, Giovanni Puccetti, Thomas Arnold, Al- ham Fikri Aji, Nizar Habash, Iryna Gurevych, and Preslav Nakov. M4GT-Bench: Evaluation Benchmark for Black-Box Machi...

    arXiv

  81. [89]

    Quantifying Privacy Risks of Prompts in Visual Prompt Learning

    Yixin Wu, Rui Wen, Michael Backes, Pascal Berrang, Math- ias Humbert, Yun Shen, and Yang Zhang. Quantifying Privacy Risks of Prompts in Visual Prompt Learning. In USENIX Security Symposium (USENIX Security). USENIX,

  82. [90]

    Membership Inference Attacks Against Text-to- image Generation Models

    Yixin Wu, Ning Yu, Zheng Li, Michael Backes, and Yang Zhang. Membership Inference Attacks Against Text-to- image Generation Models. CoRR abs/2210.00968, 2022. 3

    2022 arXiv

  83. [91]

    Towards Auditing Large Language Models: Improving Text-based Stereotype

    Zekun Wu, Sahan Bulathwela, and Adriano Soares Koshiyama. Towards Auditing Large Language Models: Improving Text-based Stereotype. CoRR abs/2311.14126 ,

    arXiv

  84. [92]

    Fairness Feedback Loops: Training on Synthetic Data Am- plifies Bias

    Sierra Calanda Wyllie, Ilia Shumailov, and Nicolas Papernot. Fairness Feedback Loops: Training on Synthetic Data Am- plifies Bias. In Conference on Fairness, Accountability, and Transparency (FAccT), pages 2113–2147. ACM, 2024. 1

    2024

  85. [93]

    Woodpecker: Hallucination Correction for Multi- modal Large Language Models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination Correction for Multi- modal Large Language Models. CoRR abs/2310.16045 ,

    arXiv

  86. [94]

    A Plot is Worth a Thousand Words: Model Information Stealing Attacks via Scientific Plots

    Boyang Zhang, Xinlei He, Yun Shen, Tianhao Wang, and Yang Zhang. A Plot is Worth a Thousand Words: Model Information Stealing Attacks via Scientific Plots. InUSENIX Security Symposium (USENIX Security). USENIX, 2023. 11

    2023

  87. [95]

    Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiao- gang Wang, Xiaolei Huang, and Dimitris N. Metaxas. Stack- GAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks. IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 2019. 2

    2019

  88. [96]

    Wein- berger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Wein- berger, and Yoav Artzi. BERTScore: Evaluating Text Gener- ation with BERT. In International Conference on Learning Representations (ICLR), 2020. 9 16

    2020

  89. [97]

    Character- level Convolutional Networks for Text Classification

    Xiang Zhang, Junbo Zhao, and Yann LeCun. Character- level Convolutional Networks for Text Classification. InAn- nual Conference on Neural Information Processing Systems (NIPS), pages 649–657. NIPS, 2015. 4, 17

    2015

  90. [98]

    Gupta, and Jingbo Shang

    Xiyuan Zhang, Ranak Roy Chowdhury, Rajesh K. Gupta, and Jingbo Shang. Large Language Models for Time Series: A Survey. CoRR abs/2402.01801, 2024. 2

    2024 arXiv

  91. [99]

    Yizhen Zheng, Huan Yee Koh, Jiaxin Ju, Anh T. N. Nguyen, Lauren T. May, Geoffrey I. Webb, and Shirui Pan. Large Language Models for Scientific Synthesis, Inference and Ex- planation. CoRR abs/2310.07984, 2023. 1

    2023 arXiv

  92. [100]

    A Survey on Data Augmentation in Large Model Era

    Yue Zhou, Chenlu Guo, Xu Wang, Yi Chang, and Yuan Wu. A Survey on Data Augmentation in Large Model Era. CoRR abs/2401.15422, 2024. 1 A Metric-Based Auditing Using Entropy Formally, we define the metric-based auditing using Entropy as the performance metric as follows: Ientr(Ct...

    2024 arXiv

  93. [101]

    We observe that tuning-based auditing also achieves strong performance on these two tasks, and more reference classifiers, meaning a larger training dataset leads to bet- ter tuning-based auditing performance. Meanwhile, as illus- trated in Figure 27, we demonstrate that five ...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.