Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Contrastive Private Data Synthesis via Weighted Multi-PLM Fusion

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A weighted privacy-protected voting scheme lets a hundred private examples supervise the creation of a strong synthetic text dataset.

desk verdict WASP is a sensible incremental synthesis framework with a noise-dominated normalization step that can flip its selection; needs a fix and error bars before the empirical claims can be trusted. read the letter →

arxiv 2502.00245 v1 pith:CBOF7EBJ submitted 2025-02-01 cs.LG

Tianyuan Zou , Yang Liu , Peng Li , Yufei Xiong , Jianqing Zhang , Jingjing Liu , Xiaozhou Ye , Ye Ouyang
show 1 more author
Ya-Qin Zhang
This is my paper · ORCID
classification cs.LG
keywords differentialprivacysynthetictextdataprivateevolutionmulti-PLMfusioncontrastivein-contextlearningdata-scarcefederatedlanguagemodelAPIs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a few hundred private examples can guide the creation of a much larger differentially private synthetic text dataset, if the guidance is pooled across several pre-trained language models (PLMs) instead of trusting any one of them. The proposed framework, WASP, lets every private example vote for its closest and its farthest synthetic candidates, gives those votes exponentially decreasing weight, and adds Gaussian noise to keep the whole process differentially private. The best and worst candidates are then shown to every model in a contrastive prompt, and each model's share of the next generation round is set by how often its outputs were voted as close to the private data. Across six text classification tasks, the paper reports that this consistently beats single-model Private Evolution baselines and a zero-shot fusion baseline, using only 100 private samples at a privacy budget of $(4.0, 1\times 10^{-5})$-DP.

What carries the argument

The load-bearing object is the differentially private Top-$Q$ voting histogram. Each private sample casts weighted votes $1, 1/2, \ldots, 1/2^{Q-1}$ for the $Q$ nearest and $Q$ furthest synthetic samples of the same label, using $\ell_2$ distance in the sentence-t5-base embedding space; Gaussian noise with $\sigma = \frac{4\sqrt{2\log(1.25/\delta)}\sqrt{T-1}}{\epsilon}$ is added to both the nearest and furthest histograms. The nearest histogram drives sample selection, ranks the quality of each PLM's outputs, and sets the next-round weight $w_k = \frac{\sum_{(x_i,y_i)\in D_k} s_i}{|D_k|/|D|}$, where $s_i$ is the normalized nearest-vote share of sample $i$. The contrastive prompt then feeds selected high- and low-quality samples to all PLMs, instructing the models to generate examples closer to the high-quality set and farther from the low-quality set.

What would settle it

Take a task where sample quality can be scored independently, such as human ratings or a task-specific validator, run WASP's DP voting with sentence-t5-base, and compute the correlation between its nearest-neighbor vote scores and the external quality scores. If the correlation is near zero or negative for samples that still pass the voting threshold, the mechanism's quality signal is not what drives the improvements. A cleaner experiment would replace the embedding model with a random or deliberately wrong embedding while keeping all other components fixed: if the accuracy gain over single-PLM baselines persists, the claimed mechanism is not the cause.

Watch

Extended reading notes

Core claim

The central claim is that distribution estimation for private data synthesis should not stop at one vote per private sample. WASP's Top-$Q$ voting mechanism has each private sample select the $Q$ nearest and $Q$ furthest synthetic samples of the same label, scoring them with weights $1, 1/2, \ldots, 1/2^{Q-1}$, and adding Gaussian noise calibrated to a sensitivity of 4. The noisy histograms identify high- and low-quality synthetic examples, which are used as cross-PLM contrastive in-context demonstrations, and also determine each PLM's weight for the next generation round. The reported result is that this combination yields better downstream small-model accuracy than any single best PLM's Aug-PE baseline, than FuseGen's equal-weight fusion, and than training directly on the private data, on all six tested tasks under $(4.0, 1\times 10^{-5})$-DP for five iterations and 100 private samples.

Load-bearing premise

The whole pipeline assumes that distances computed by the fixed sentence-embedding model (sentence-t5-base) faithfully measure how good a synthetic sample is and how close it is to the real distribution for every downstream task; if that embedding misses the features that matter, the votes, the contrastive examples, and the PLM weights would all be rewarding the wrong outputs, and the reported gains would not survive.

Editorial extensions

If this is right

  • With only 100 private samples, WASP generates 6,000 synthetic texts that train a BERT classifier to higher accuracy than the best available single PLM's Aug-PE baseline on all six tasks, at $(4.0, 1\times 10^{-5})$-DP.
  • Adding more PLMs improves accuracy and reduces run-to-run variance; every pair of closed-source PLMs tested beats either member alone, so users do not need to know in advance which generator is best for their task.
  • Using more votes per private sample (up to $Q=8$) helps more than one vote, with diminishing returns beyond $Q=8$; the benefit is larger on harder tasks such as Yelp-Rating and Openreview-Rating.
  • The method's advantage over Aug-PE grows as the private set shrinks below 1,000 examples, so the framework is aimed precisely at the data-scarce regime, and it extends to federated settings with secure aggregation at small communication overhead.
  • Performance degrades smoothly as the privacy budget tightens, retaining most of its accuracy at $\epsilon=1.0$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the experiments reported here, the same Top-$Q$ vote-and-contrast recipe could transfer to images, code, or structured records by swapping in a suitable embedding model and rescaling the noise by that embedding's sensitivity.
  • The paper's ablation removes contrastive prompting as a whole; a follow-up could hold the prompt length constant and swap the roles of good and bad examples to tell whether the contrast direction or the mere presence of extra demonstrations is doing the work.
  • Because the PLM weights are recomputed from the same votes that select the in-context examples, WASP converts model-selection risk into a learned allocation; the same idea could improve other API-only generation or self-improvement pipelines without extra API calls.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes WASP, a differentially private synthetic-text generation framework for the data-scarce regime (M=100 private samples). WASP iteratively generates samples from several black-box PLMs, uses a Gaussian-noised Top-Q voting scheme over sentence-embedding distances to label synthetic samples as high- or low-quality, builds contrastive in-context prompts from those samples, and reweights the PLMs according to aggregated nearest-neighbor votes. The method is evaluated on six text classification tasks with six open-source and three closed-source PLMs, reporting downstream STM accuracy gains over single-PLM PE baselines and FuseGen under (4.0, 1e-5)-DP. The appendix adds federated and user-level DP variants, ablations over Q, K, and M, and a formal privacy proof.

Significance. If the proposed method performs as described, it is a practical and relevant contribution: it extends PE-style API-based synthetic data generation to very small private sets, enables multi-PLM collaboration without extra API queries, and comes with a DP guarantee. The paper is clearly written and includes useful ablations, a broad model/task suite, and a code link. The downstream STM accuracy evaluation is independent of the selection mechanism, so the main claim is not circular. However, the published algorithm contains a load-bearing normalization instability, and the main experimental tables lack error bars, so the empirical claims are not yet established at the level required by the manuscript's conclusions.

major comments (3)
  1. [Algorithm 3; Eq. (8)] The normalization of the noisy voting histograms is not well-defined. With M=100, Q=8, the un-noised vote mass in H_n is M·Σ_{q=1}^Q 2^{-(q-1)} ≈ 199.2. For ε=4, δ=1e-5, T=5, the per-entry noise is σ ≈ 4·sqrt(2 ln(1.25/δ))·sqrt(T−1)/ε ≈ 9.7. The sum of the noise over the |D| histogram entries therefore has standard deviation σ√|D|, about 336 at the first iteration (|D|=1200) and about 751 at the last (|D|=6000). Hence the denominator Σ_i H_n[i] in Eq. (8) and in SampleSelection in Algorithm 3 is negative with non-negligible probability (roughly 28% and 40%, respectively). When the denominator is negative, normalized values change sign, the top-S selection picks the lowest-scoring examples rather than the highest-scoring ones, and the PLM weights w_k in Eq. (8) can become negative so that the next round's count [(N/T)·w_k] is undefined. The paper does not clip the histograms, use absolute-value normalization, or specify a fallback. Additionally, Section 4.4 Eq. (7) selects on raw H_n, whereas Algorithm 3 first normalizes H_n, so the two specifications disagree. This is a load-bearing issue that must be fixed and the experiments redone or the implementation must be shown to avoid it.
  2. [Tables 1–3, Section 5.2] The main superiority claim is based on single accuracy numbers without standard deviations or number of seeds. Several WASP gains over the best single-PLM baseline are small: Table 1 shows +0.52 on IMDb (89.52 vs 89.00), +0.63 on Yelp-Category, +0.35 on Openreview-Area, and +0.25 on Banking. Since downstream STM fine-tuning is stochastic, these margins may be within run-to-run noise. The paper already reports STD in Figure 3 for the K ablation, so the authors should add error bars or significance measures to Tables 1–3 and the related ablation tables before claiming consistent superiority.
  3. [Table 2 vs. Section 5.1] Section 5.1 states that the federated setting uses L=10 parties with a total of M=Σ|B_l|=300 private samples, but Table 2 lists |B|=100 for every row. This contradiction makes the federated experiments impossible to interpret as reported. Please correct the table or the prose and state explicitly whether the federated results use 100 or 300 private samples in total.
minor comments (5)
  1. [Figure 5, Appendix E.1] The FID measure uses the same sentence-T5 embedding model that powers the Top-Q voting, so the decreasing FID partly reflects optimization of the selection objective rather than an independent distributional-quality measure; report FID with a different encoder or qualify the claim.
  2. [Appendix D, Theorem D.2] The proof states Δ=4, but the argument establishes an upper bound; the L2 sensitivity of the concatenated H_n and H_f is at most 2√2, so Δ=4 is conservative. The statement should say 'at most 4' rather than 'is 4'.
  3. [Appendix D and Algorithm 1] The proof composes T−1 noisy releases, but Algorithm 1 performs DP_PrivateVoting in all T iterations. Clarify that the last iteration's voting is not used in any released output (or remove it from the pseudocode) so the composition count is unambiguous.
  4. [Table 7] Several contrastive prompt templates are missing the closing bracket in '<sample_S' (e.g., the IMDb, Yelp-Category, and Banking rows), which complicates exact reproduction.
  5. [Section 5.1] The text says 'Close-source PLMs' and should read 'closed-source PLMs'.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor, localized circularity: the FID resemblance metric uses the same sentence-T5 embedding that defines WASP's quality signal; central STM-accuracy claims remain independently benchmarked.

  1. self definitional [Section 4.4 (Eq. 3), Section 5.1 (embedding choice), Appendix E.1 / Figure 5]
    "We use sentence-t5-base [33] as the embedding model φ. ... we additionally use FID between the generated DP synthetic datasetD and the real private datasetB to evaluate the resemblance ... WASP results in a consistently decreasing FID value over iteration, demonstrating it effectiveness in improving the resemblance of D to B."

    The quality signal in Eq. (3) is d(z_j,x_i)=||φ(z_j)-φ(x_i)||_2; Eq. (5) builds H_n/H_f from these distances, Eq. (7) selects 'high-quality' and 'low-quality' samples from them, and Eq. (8) derives PLM weights from H_n. The FID in Figure 5 is computed between D and B in the same sentence-t5-base φ-space (no separate feature extractor is specified). Because WASP's selection, contrastive prompting, and PLM weighting all push D toward B in that space, the reported decreasing FID is largely a re-statement of the algorithm's own selection objective rather than an independent confirmation that D resembles B. Held-out STM accuracies are external and do not inherit this circularity.

full rationale

The central claim of WASP is downstream STM accuracy on held-out test sets, which is an external benchmark independent of the φ-embedding used inside the algorithm; no parameter is fitted to that benchmark and then reported as a prediction. FuseGen, the one baseline by overlapping authors, is used only as a comparison method and does not carry any load-bearing justification for WASP. No uniqueness theorem or ansatz is imported from the authors' prior work. The only identifiable reduction is the supporting FID-resemblance result: the same sentence-t5-base embedding φ defines both the Top-Q voting quality signal and the FID metric, so the Figure 5 improvement is partly by construction. This does not force the main performance conclusions, which survive on the independent STM numbers. Separately, the reviewer's concern about the noise-dominated normalizing sum in Eq. (8)/Algorithm 3 is an internal-consistency and correctness issue, not a circularity, and does not enter this score.

Assumptions & free parameters 6 free parameters · 7 assumptions · 0 invented entities

The central claim rests on standard differential privacy machinery (Gaussian mechanism, composition, post-processing) plus two domain assumptions: representativeness of the small private sample and suitability of the chosen sentence embedding model. The hand-chosen hyperparameters (Q, T, N, S) affect the empirical results but are not fitted constants in a derivation. No new physical or formal entities are introduced; the framework is a combination of existing mechanisms.

free parameters (6)
  • Q (number of votes per private sample) = 8 (default)
    Chosen by hand after sensitivity analysis in Table 5; larger Q gives diminishing returns and Q=8 is used in main experiments.
  • T (number of iterations) = 5
    Fixed in the algorithm with N=6000 total synthetic samples; results may vary with iteration count.
  • N (total synthetic samples) = 6000
    Chosen for the experiments; downstream accuracy could change with dataset size.
  • M (private sample count) = 100 (300 or 500 in federated and user-level settings)
    The paper targets the data-scarce regime; experimental results vary with M.
  • S (number of in-context samples per category) = not reported
    Used in Algorithms 1 and 3 but its value is not specified in the experiments, leaving a reproducibility gap.
  • Voting decay weights (1, 1/2, ..., 1/2^(Q-1)) = geometric sequence
    Hand-chosen to keep the sensitivity sum below 2 per histogram; not derived from data.
assumptions (7)
  • standard math Gaussian mechanism guarantees (epsilon, delta)-DP when noise scale sigma = Delta*sqrt(2 log(1.25/delta))/epsilon is added to a statistic with L2 sensitivity Delta (Theorem D.1).
    Invoked in Section 3 and Appendix D to calibrate noise for the voting histograms.
  • standard math Composition of T Gaussian mechanisms grows the privacy budget like sqrt(T)*epsilon (Lemma D.3).
    Used in the proof of Theorem D.4 to split the privacy budget across iterations.
  • standard math Post-processing of a DP output is free (no additional privacy loss).
    Used implicitly to argue that sample selection, weighted prompting, and STM training on the synthetic set do not consume privacy budget.
  • domain assumption The private sample set B is representative of the true data distribution.
    All PE-style methods estimate the private distribution from B; if B is biased, the synthetic data inherits the bias.
  • domain assumption Sentence-t5-base embeddings phi give a meaningful similarity measure for text quality on all six tasks.
    Equation (3) uses phi for all pairwise distances; the Top-Q voting, sample selection, and PLM weighting rest on this choice.
  • domain assumption The PLMs are capable of following the contrastive few-shot prompt and generating task-related samples.
    The iterative generation loop in Algorithm 1 assumes black-box PLMs can produce usable samples from the provided prompts.
  • domain assumption Honest-but-curious server and secure aggregation in the federated extension.
    Section 4.7 states the threat model; the DP proof for the federated setting relies on aggregated histograms being the only server-visible outputs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Contrastive Private Data Synthesis via Weighted Multi-PLM Fusion." pith.science (2026). https://pith.science/paper/CBOF7EBJ

@misc{pith2026250200245,
  author       = {Pith},
  title        = {Pith review of: Contrastive Private Data Synthesis via Weighted Multi-PLM Fusion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CBOF7EBJ}},
  note         = {Machine review of arXiv:2502.00245}
}
read the original abstract

Substantial quantity and high quality are the golden rules of making a good training dataset with sample privacy protection equally important. Generating synthetic samples that resemble high-quality private data while ensuring Differential Privacy (DP), a formal privacy guarantee, promises scalability and practicality. However, existing methods relying on pre-trained models for data synthesis %that avoid fine-tuning large pre-trained generative models often struggle in data-deficient scenarios, suffering from limited sample size, inevitable generation noise and existing pre-trained model bias. To address these challenges, we propose a novel contrAstive private data Synthesis via Weighted multiple Pre-trained language models (PLM) framework, named as WASP. WASP utilizes limited private samples for more accurate private data distribution estimation via a Top-Q voting mechanism, and leverages low-quality synthetic samples for contrastive generation via collaboration among dynamically weighted multiple pre-trained models.Extensive experiments on 6 well-developed datasets with 6 open-source and 3 closed-source PLMs demonstrate the superiority of WASP in improving model performance over diverse downstream tasks. Code is available at https://anonymous.4open.science/r/WASP.

Figures

Figures reproduced from arXiv: 2502.00245 by the authors.

Figure 1
Figure 1. (𝑎) Comparison of the similarity of synthetic dataset to real private dataset (measured by FID [18]) and STM perfor￾mance (numbers within parenthesis) of Aug-PE [46] (dotted lines) and our refinement (dashed lines) under (4.0, 1 × 10−5 )-DP with IMDb dataset. Lower FID indicates higher similarity. (𝑏) Results of Aug-PE using 100 private samples and (4.0, 1 × 10−5 )-DP. in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of WASP framework. 3 PRELIMINARIES Differential Privacy (DP). If two datasets D and D′ differ in a single entry, they are referred to as Neighboring Datasets. A mecha￾nism M satisfies (𝜖, 𝛿)-DP if for any neighboring datasets D, D′ and any output subset 𝐸 of M, it holds that [13]: Pr[M (D) ∈ 𝐸] ≤ 𝑒 𝜖 · Pr[M (D′ ) ∈ 𝐸] + 𝛿. (1) Note that post-processing on the output of (𝜖, 𝛿)-DP does not incur additional pr… view at source ↗
Figure 3
Figure 3. Evaluation of downstream STM accuracy using Yelp [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Comparison of downstream STM accuracy using [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Comparison of the resemblance of synthetic dataset [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Comparison of the resemblance of synthetic dataset to real private dataset (Fréchet Inception Distance, FID) and [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Comparison of the similarity of synthetic dataset to real private dataset (FID) and trained downstream model [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MAPLE: Metadata Augmented Private Language Evolution

    cs.CL 2026-02 conditional novelty 5.0 of 10

    MAPLE boosts Private Evolution's initial prompt with DP tabular metadata and donated in-context examples, improving synthetic-text utility, convergence, and API cost on specialized corpora.

  2. Differentially Private Synthetic Data via APIs 3: Using Simulators Instead of Foundation Model

    cs.LG 2025-02 conditional novelty 4.0 of 10

    Sim-PE plugs non-neural simulators into Private Evolution to create differentially private images, improving downstream accuracy over foundation-model PE by up to 3x on MNIST.

Reference graph

Works this paper leans on

60 extracted references · 44 canonical work pages · cited by 2 Pith papers

  1. [1]

    Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep Learning with Differential Privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security. 308–318

    work page 2016

  2. [2]

    Borja Balle and Yu-Xiang Wang. 2018. Improving the Gaussian Mechanism for Differential Privacy: Analytical Calibration and Optimal Denoising. In Interna- tional Conference on Machine Learning . PMLR, 394–403

    work page 2018

  3. [3]

    Nikolija Bojkovic and Po-Ling Loh. 2024. Differentially Private Synthetic Data with Private Density Estimation. arXiv preprint arXiv:2405.04554 (2024)

    work page Pith review arXiv 2024

  4. [4]

    Rishi Bommasani, Steven Wu, and Xanda Schofield. 2019. Towards Private Synthetic Text Generation. In NeurIPS 2019 Machine Learning with Guarantees Workshop

    work page 2019

  5. [5]

    Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. 2016. Prac- tical Secure Aggregation for Federated Learning on User-Held Data. InNIPS 2016 Workshop on Private Multi-Party Machine Learning

    work page 2016

  6. [6]

    Lukas Budach, Moritz Feuerpfeil, Nina Ihde, Andrea Nathansen, Nele Sina Noack, Hendrik Patzlaff, Hazar Harmouch, and Felix Naumann. 2022. The Effects of Data Quality on ML-Model Performance. CoRR abs/2207.14529 (2022)

    arXiv 2022

  7. [7]

    Iñigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic. 2020. Efficient Intent Detection with Dual Sentence Encoders. In Proceedings of the 2nd Workshop on NLP for ConvAI - ACL 2020 . https: //arxiv.org/abs/2003.04807 Data available at https://github.com/PolyAI-LDN/task- specific-datasets

    arXiv 2020

  8. [8]

    Han Shi Jocelyn Chew. 2022. The Use of Artificial Intelligence–based Con- versational Agents (Chatbots) for Weight Loss: Scoping Review and Practical Recommendations. JMIR medical informatics 10, 4 (2022), e32578

    work page 2022

Show all 60 references
  1. [9]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. https://lmsys.org/blog/2023-...

    2023

  2. [10]

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling Instruction-finetuned Language Models. arXiv preprint arXiv:2210.11416 (2022)

    2022 arXiv

  3. [11]

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mor- datch. 2023. Improving Factuality and Reasoning in Language Models through Multiagent Debate. arXiv preprint arXiv:2305.14325 (2023)

    2023 arXiv

  4. [12]

    Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long...

    2022

  5. [13]

    Cynthia Dwork. 2006. Differential Privacy. In International colloquium on au- tomata, languages, and programming . Springer, 1–12

    2006

  6. [14]

    Cynthia Dwork, Aaron Roth, et al. 2014. The Algorithmic Foundations of Differ- ential Privacy. Foundations and Trends® in Theoretical Computer Science 9, 3–4 (2014), 211–407

    2014

  7. [15]

    James Flemings and Murali Annavaram. 2024. Differentially Private Knowledge Distillation via Synthetic Text Generation.arXiv preprint arXiv:2403.00932 (2024)

    2024

  8. [16]

    Jiahui Gao, Renjie Pi, Lin Yong, Hang Xu, Jiacheng Ye, Zhiyong Wu, Weizhong Zhang, Xiaodan Liang, Zhenguo Li, and Lingpeng Kong. 2023. Self-guided Noise- free Data Generation for Efficient Zero-shot Learning. In Proceedings of The Eleventh International Conference on Learning ...

    2023

  9. [17]

    Xiang Gao and Kamalika Das. 2024. Customizing Language Model Responses with Contrastive In-Context Learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 18039–18046

    2024

  10. [18]

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. Gans Trained by a Two Time-scale Update Rule Converge to a Local Nash Equilibrium. Advances in Neural Information Processing Systems 30 (2017)

    2017

  11. [19]

    Charlie Hou, Akshat Shrivastava, Hongyuan Zhan, Rylan Conway, Trang Le, Adithya Sagar, Giulia Fanti, and Daniel Lazar. 2024. PrE-Text: Training Language Models on Private Federated Data in the Age of LLMs. In Forty-first International Conference on Machine Learning

    2024

  12. [20]

    Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. 2019. Measuring the Effects of Non-identical Data Distribution for Federated Visual Classification. arXiv preprint arXiv:1909.06335 (2019)

    2019 arXiv

  13. [21]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. GPT-4o System Card. arXiv preprint arXiv:2410.21276 (2024)

    2024 arXiv

  14. [22]

    Inc. Yelp. 2015. Yelp Open Dataset. https://www.yelp.com/dataset

    2015

  15. [23]

    Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye. 2024. More Agents is All You Need. arXiv preprint arXiv:2402.05120 (2024)

    2024 arXiv

  16. [24]

    Chao Liang, Wei Xiang, and Bang Wang. 2024. In-context Contrastive Learning for Event Causality Identification. arXiv preprint arXiv:2405.10512 (2024)

    2024 arXiv

  17. [25]

    Zinan Lin, Sivakanth Gopi, Janardhan Kulkarni, Harsha Nori, and Sergey Yekhanin. 2024. Differentially Private Synthetic Data via Foundation Model APIs 1: Images. In The Twelfth International Conference on Learning Representa- tions

    2024

  18. [26]

    Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. 2024. A Dynamic LLM- Powered Agent Network for Task-Oriented Agent Collaboration. In Proceedings of the 1st Conference on Language Modeling (COLM 2024)

    2024

  19. [27]

    Maas, Raymond E

    Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning Word Vectors for Sentiment Analysis. In Pro- ceedings of the 49th Annual Meeting of the Association for Computational Linguis- tics: Human Language Technologies. Asso...

    2011

  20. [28]

    Justus Mattern, Zhijing Jin, Benjamin Weggenmann, Bernhard Schoelkopf, and Mrinmaya Sachan. 2022. Differentially Private Language Models for Secure Data Sharing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 4860–4873

    2022

  21. [29]

    Costas Mavromatis, Petros Karypis, and George Karypis. 2024. Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimization. arXiv preprint arXiv:2404.11531 (2024)

    2024 arXiv

  22. [30]

    Yosuke Miyanishi and Minh Le Nguyen. 2024. Multimodal Contrastive In-Context Learning. arXiv preprint arXiv:2408.12959 (2024)

    2024 arXiv

  23. [31]

    Ying Mo, Jiahao Liu, Jian Yang, Qifan Wang, Shun Zhang, Jingang Wang, and Zhoujun Li. 2024. C-ICL: Contrastive In-Context Learning for Information Extraction. arXiv preprint arXiv:2402.11254 (2024)

    2024 arXiv

  24. [32]

    Supriya Nagesh, Justin Y Chen, Nina Mishra, and Tal Wagner. 2024. Private Text Generation by Seeding Large Language Model Prompts. In GenAI for Health: Potential, Trust and Policy Compliance

    2024

  25. [33]

    Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. 2022. Sentence-T5: Scalable Sentence Encoders from Pre- trained Text-to-Text Models. In Findings of the Association for Computational Contrastive Private Data Synthesis via Weig...

    2022

  26. [34]

    OpenAI. 2021. GPT-3.5-Turbo. https://platform.openai.com/docs/models/gpt-3- 5-turbo

    2021

  27. [35]

    OpenAI. 2023. GPT-4-Turbo and GPT-4 . https://platform.openai.com/docs/ models/gpt-4-turbo-and-gpt-4

    2023

  28. [36]

    Pranav Putta, Ander Steele, and Joseph W Ferrara. 2023. Differentially Private Conditional Text Generation For Synthetic Data Production. https://openreview. net/forum?id=LUql3ZOFwFD

    2023

  29. [37]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language Models are Unsupervised Multitask Learn- ers. OpenAI blog 1, 8 (2019), 9

    2019

  30. [38]

    Ruifeng Ren and Yong Liu. 2024. Towards Understanding How Transformers Learn In-context Through a Representation Learning Lens. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. https://openreview. net/forum?id=dB6gwSDXKL

    2024

  31. [39]

    Yi Ren, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2019. Almost Unsupervised Text to Speech and Automatic Speech Recognition. InInternational conference on machine learning . PMLR, 5410–5419

    2019

  32. [40]

    Anna Rumshisky, Marzyeh Ghassemi, Tristan Naumann, Peter Szolovits, VM Castro, TH McCoy, and RH Perlis. 2016. Predicting Early Psychiatric Read- mission with Natural Language Processing of Narrative Discharge Summaries. Translational psychiatry 6, 10 (2016), e921–e921

    2016

  33. [41]

    Thomas Steinke. 2022. Composition of Differential Privacy & Privacy Amplifica- tion by Subsampling. arXiv:2210.00597 [cs.CR] https://arxiv.org/abs/2210.00597

    2022 arXiv

  34. [42]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. 2023. Llama 2: Open Foundation and Fine-tuned Chat Models, 2023. URL https://arxiv. org/abs/2307.09288 (2023)

    2023 arXiv

  35. [43]

    Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi

  36. [44]

    Fanqi Wan, Ziyi Yang, Longguang Zhong, Xiaojun Quan, Xinting Huang, and Wei Bi. 2024. FuseChat: Knowledge Fusion of Chat Models. arXiv preprint arXiv:2402.16107 (2024)

    2024 arXiv

  37. [45]

    Jiahao Wang, Bolin Zhang, Qianlong Du, Jiajun Zhang, and Dianhui Chu

  38. [46]

    Chulin Xie, Zinan Lin, Arturs Backurs, Sivakanth Gopi, Da Yu, Huseyin A Inan, Harsha Nori, Haotian Jiang, Huishuai Zhang, Yin Tat Lee, et al. 2024. Differen- tially Private Synthetic Data via Foundation Model APIs 2: Text. In Forty-first International Conference on Machine Learning

    2024

  39. [47]

    arXiv preprint arXiv:2402.05123 (2024)

    A Survey on Data Selection for LLM Instruction Tuning. arXiv preprint arXiv:2402.05123 (2024)

    2024 arXiv

  40. [48]

    Da Yu, Peter Kairouz, Sewoong Oh, and Zheng Xu. 2024. Privacy-Preserving Instructions for Aligning Large Language Models. In Forty-first International Conference on Machine Learning . https://openreview.net/forum?id=mUT1biz09t

    2024

  41. [49]

    Jiacheng Ye, Jiahui Gao, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2022. ProGen: Progressive Zero-shot Dataset Generation via In-context Feedback. In Findings of the Association for Computational Linguistics: EMNLP

    2022

  42. [50]

    Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. 2019. Bayesian Nonparametric Federated Learning of Neural Networks. In International conference on machine learning . PMLR, 7252–7261

    2019

  43. [51]

    Xiang Yue, Huseyin Inan, Xuechen Li, Girish Kumar, Julia McAnallen, Hoda Sha- jari, Huan Sun, David Levitan, and Robert Sim. 2023. Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe. In Proceedings of the 61st Annual Meeting of the Association f...

    2023

  44. [52]

    Jianqing Zhang, Yang Hua, Hao Wang, Tao Song, Zhengui Xue, Ruhui Ma, and Haibing Guan. 2023. FedALA: Adaptive Local Aggregation for Personalized Fed- erated Learning. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 37. 11237–11244

    2023

  45. [53]

    Barbara Zdrazil, Eloy Felix, Fiona Hunter, Emma J Manners, James Blackshaw, Sybilla Corbett, Marleen de Veij, Harris Ioannidis, David Mendez Lopez, Juan F Mosquera, et al. 2024. The ChEMBL Database in 2023: A Drug Discovery Platform Spanning Multiple Bioactivity Data Types and...

    2024

  46. [54]

    Wenhao Zhao, Shaoyang Song, and Chunlai Zhou. 2024. Generate Synthetic Text Approximating the Private Distribution with Differential Privacy. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence . 6651– 6659

    2024

  47. [55]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2...

    2022 arXiv

  48. [57]

    Tianyuan Zou, Yang Liu, Peng Li, Jianqing Zhang, Jingjing Liu, and Ya-Qin Zhang

  49. [58]

    w Contrastive

    FuseGen: PLM Fusion for Data-generation based Zero-shot Learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). 2172– 2190. Zou, et al. Algorithm 2 WASP for Distributed Feder...

    2024

  50. [59]

    if𝑋 ∼N( 0,𝜎 2 1),𝑌 ∼N( 0,𝜎 2

  51. [60]

    w/o Con” results, and then incorporate con- trastive in-context learning samples into the prompt to achieve the “w/ Con

    are inde- pendent. Therefore, if each private data party adds i.i.d. Gaussian noise followingN( 0,𝜎 2) with𝜎 = 4 √ 2 log(1.25/𝛿) √ 𝑇−1 𝜖 √ 𝐿 , the total noise followsN( 0,𝜎 2 𝑡𝑜𝑡𝑎𝑙) which guaranties(𝜖,𝛿)-DP for the whole WASP process. □ E ADDITIONAL RESULTS E.1 Comparison of S...

    2025

  52. [2024]

    InProceedings of The Twelfth International Conference on Learning Representations

    Knowledge Fusion of Large Language Models. InProceedings of The Twelfth International Conference on Learning Representations . https://openreview.net/ forum?id=jiDsk12qcz

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.