{"id":"74b4e316-4824-4ec3-8236-392f3bc11c16","arxiv_id":"2504.16438","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"POPri uses client similarity scores as RL rewards to DPO-tune an LLM for DP synthetic data generation, outperforming prior private evolution baselines on next-token prediction and classification.","lead":"POPri fine-tunes an LLM with direct preference optimization on client feedback to generate better differentially private synthetic data for federated learning. It improves downstream model accuracy by nearly half of the private-to-nonprivate gap and introduces LargeFedBench, a contamination-controlled federated text benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Privacy accounting in Section 3.1 is inconsistent with Algorithm 1: the aggregated score vector has sensitivity 2/L (or 2 if summed) and noise N(0,sigma^2 I/L), not sensitivity 1 and noise N(0,sigma^2 I); the stated epsilon may be incorrect unless the code is checked.","rationale":"Good-faith reading: the paper's contribution is an empirical one, namely that POPri's DP synthetic data outperforms PE and DP-FL baselines at epsilon=1 and epsilon=7. The method has plausible components (DP preference pairs, DPO fine-tuning, cosine-similarity reward), and the ablations (cosine vs NN histogram, rejected-sample rank, partial participation) are useful. I do not see a reason to doubt the authors' integrity; the issue is a precise mismatch between the stated privacy analysis and the pseudocode. The single most load-bearing concern is the privacy accounting in Section 3.1. The pseudocode averages per-client noised vectors; the text's sensitivity-1/noise-sigma^2 I description matches a sum. These cannot both be correct. If the implementation sums, the replacement-DP sensitivity is 2, so the reported epsilon is about 4x too small in RDP terms; the epsilon=1 experiments would actually be epsilon~4, invalidating the headline 'at epsilon=1' comparison. If the implementation averages, the reported epsilon is an upper bound and the concern is a documentation error rather than a correctness failure, but the text still needs correction because the stated noise and sensitivity are wrong. The released code and a re-accounting can settle this directly. The reader's weakest assumption (embedding-space cosine reward as a proxy for downstream utility) is also legitimate: all model-selection signals (client scores, FID) live in the same embedding space, so a misaligned proxy could in principle inflate POPri's apparent advantage. However, the direct downstream-accuracy evaluations already provide indirect validation, and the paper's ablation shows the cosine ordering is recoverable. The privacy inconsistency is more load-bearing because it concerns whether the central comparisons are made at the claimed privacy level at all. I also agree with the reader's secondary reporting concerns (the 58% gap-closure figure does not match Table 1 arithmetic, and error bars are not tied to a stated number of trials), but these are less fundamental than the privacy accounting.","tokens_in":23477,"tokens_out":15167,"duration_ms":153310,"concrete_test":"Inspect the GitHub repository's aggregation code for the federated POPri loop. Determine (a) whether Scores_t is the average or the sum of the per-client noised vectors, and (b) the exact covariance of the noise after aggregation. Then recompute the privacy accountant for the bioRxiv epsilon=1 and epsilon=7 runs with the correct L2 sensitivity (2/L for the average, or 2 for the sum) and the correct noise scale, using the same RDP accountant and delta=3e-6. If the recomputed epsilon exceeds the reported value by more than a negligible amount, rerun the headline bioRxiv comparison after scaling sigma to hit the reported epsilon; if POPri no longer closes the gap by 40-58%, the central claim fails at the stated privacy level.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1's privacy analysis claims that because each client's score vector is clipped to L2 norm 1, the sensitivity of the aggregated vector is 1, and that per-client noise N(0,sigma^2 I/L) yields aggregate noise N(0,sigma^2 I). But Algorithm 1 (lines 15-20) sends clipped scores plus N(0,sigma^2 I/L) and then computes Scores_t = (1/L) * sum(Scores_{i,t}). Under the paper's own replacement-DP definition, the average has L2 sensitivity 2/L, and the average of L independent Gaussian noises has covariance sigma^2 I/L, not sigma^2 I. If the implementation actually sums (as the prose 'vector given to server has noise N(0,sigma^2 I)' suggests), the sensitivity is 2, not 1, so the RDP/Opacus epsilon reported in Table 3 is underestimated by roughly a factor of 4 (since RDP loss scales as Delta^2 / sigma^2). Because every headline comparison is made at a stated epsilon=1 or epsilon=7, an incorrect sensitivity changes the meaning of those comparisons. This is an internal inconsistency in the core privacy guarantee, not a disagreement about conventions; the released code should resolve which aggregation is actually used.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes POPri, a differentially private federated learning method that generates synthetic training data by fine-tuning a server-side LLM with Direct Preference Optimization (DPO). Client devices score synthetic samples by average cosine similarity to their private data, the scores are clipped, noised, and securely aggregated, and the server uses the resulting preference pairs to iteratively fine-tune the generator. The final generator produces a synthetic corpus used to train a small downstream model. The authors introduce LargeFedBench, a benchmark of bioRxiv and congressional speech data designed to reduce contamination, and report that POPri outperforms DP-FedAvg, DP-FTRL, and Private Evolution baselines, closing up to 58% of the gap between fully private and non-private accuracy.","tokens_in":23757,"tokens_out":2572,"duration_ms":26281,"significance":"If the correctness concerns are resolved, the paper's central claim is significant: it advances the state of the art for private synthetic-data training and demonstrates that preference-optimization techniques can be adapted to DP client feedback. The release of LargeFedBench is a useful community contribution, especially the design goal of periodically updated, contamination-controlled federated text data, and the paper includes ablations (scoring function, rejected-sample rank, partial participation) that support the main design choices. The result would be stronger if the privacy accounting were unambiguous and if the headline gap-closure numbers were internally consistent.","major_comments":[{"comment":"The privacy analysis is internally inconsistent with the pseudocode. Algorithm 1 line 20 computes Scores_t = (1/L) * sum(Scores_{i,t}), so under the paper's replacement-neighbor definition the L2 sensitivity of the aggregated vector is 2/L, not 1. The per-client noise N(0, sigma^2 I / L) averages to N(0, sigma^2 I / L), not N(0, sigma^2 I). If the implementation instead sums the scores before averaging, the sensitivity is 2 rather than 1. Either way, the stated epsilon values computed by the RDP accountant are not justified by the text, and because all headline comparisons are made at epsilon = 1 or 7, the reported privacy levels and the comparison itself are ambiguous. The authors should correct the accounting, rescale the noise or the clipping procedure accordingly, and re-report the privacy parameters.","section":"Section 3.1 / Algorithm 1"},{"comment":"The claimed 58% gap closure for bioRxiv at epsilon = 1 does not match the numbers reported in Table 1. With the table's values, the fully non-private accuracy is 41.5, the epsilon = 0 baseline is 27.9, and POPri reaches 34.8 at epsilon = 1, giving a gap closure of about 50%, not 58%; Private Evolution reaches 31.1, giving about 23%, not the 28% stated in the abstract and Section 5.1. Because this percentage is a headline quantitative claim, the authors should either correct the reported numbers, clarify which setting and which checkpoint selection produced 58%, or restate the claim consistently across the abstract, Figure 1, and Section 5.1.","section":"Section 5.1 and Figure 1"},{"comment":"The paper defines neighboring datasets as differing in at most one client's data but does not specify whether this is replacement, addition/removal, or both. The sensitivity calculation in Section 3.1 is sensitive to this choice: the replacement definition gives an L2 sensitivity of 2/L for the average, while addition/removal with a norm-1 clip gives 1/L. This ambiguity propagates to the claimed epsilon values and should be resolved explicitly in the privacy analysis.","section":"Section 2.1 / Definition of neighboring datasets"}],"minor_comments":[{"comment":"The caption states that all standard deviation error bars are less than 0.5, but Table 1 displays no error bars; the authors should either include the error bars in the table or move the statement to the text with the actual variance values.","section":"Table 1 caption"},{"comment":"Line 24, 'Select positive synthetic sample: Pt[k, 1] = Pt[k, 1]', appears to be a typo; it should presumably assign the highest-scoring response for prompt k, as described in the text.","section":"Algorithm 1, line 24"},{"comment":"Line 18 assigns 'P(b)_t[k, 2] = Pt[k, 2]', but the right-hand side should reference the current class's preference set P(b)_t, not the undeclared global Pt.","section":"Algorithm 3, line 18"},{"comment":"The percentage gap closure attributed to prior synthetic data methods is given as 23% in Figure 1, 28% in the abstract and Section 5.1, and '1-28%' in Section 5.1; these numbers should be reconciled.","section":"Abstract / Figure 1 / Section 5.1"},{"comment":"The text says 'we fine-tune the models for 2 epochs and select the best checkpoint with the lowest FID score relative to the validation dataset,' but elsewhere the method selects checkpoints by validation accuracy; the checkpoint-selection criterion should be stated consistently.","section":"Appendix B.1"},{"comment":"For the PubMed and OpenReview comparisons, POPri uses LLaMA-3-8B while the Aug-PE baselines use Llama-2-7b-chat-hf, so the comparison is not fully controlled for the base generator; this should be acknowledged as a potential confound in the discussion.","section":"Section 5 / Baselines"}],"recommendation":"major_revision","confidential_remarks":"The central empirical claim is compelling but the privacy accounting issue in Section 3.1 is load-bearing and must be resolved before the paper can be accepted. The mismatch in the headline 58% number is also concerning because it appears in the abstract, figure, and text; it is likely fixable but should not be left unexplained. The paper's evaluation relies heavily on the authors' own PrE-Text baseline and LargeFedBench, which is understandable for a new benchmark but worth noting in the editor's assessment of novelty and independence. The code release should be checked during revision to confirm whether the implemented aggregation is an average or a sum."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the core idea is genuinely new—using DP client feedback as preference pairs to DPO-fine-tune the generator—and the downstream accuracy gains over PE and DP-FL look real. But Section 3.1's privacy analysis is inconsistent with Algorithm 1, and until that is fixed, every headline epsilon comparison is on shaky ground.\n\nWhat's good: The reformulation is clean. Prior PE methods used client feedback only for prompt rewriting; POPri turns the same feedback into a reward signal and uses DPO. The cosine-similarity dense scoring is a small change with a solid ablation behind it. LargeFedBench (Congress, bioRxiv) is a useful resource, especially the date filtering for contamination control. The ablations on rejected-sample rank, client sampling, and scoring function are honest and informative. They shipped code and data, which should make verification possible.\n\nThe soft spots: The privacy accounting in Section 3.1 claims sensitivity 1 and aggregate noise N(0, sigma^2 I). Algorithm 1 adds N(0, sigma^2 I/L) per client, then computes Scores_t = (1/L) sum. That average has sensitivity 2/L, and the averaged noise has variance sigma^2/L^2, so the effective Delta-to-sigma ratio is 2/sigma, not 1/sigma. The reported epsilons are roughly twice what they claim (and the RDP loss is off by ~4x). Whether the implementation sums or averages, the accounting needs to be redone before eps=1 vs eps=7 comparisons mean what they say. This is not a convention dispute; it's an internal contradiction in the paper's own equations. Smaller issues: Table 1 has no visible error bars despite the caption claiming they are below 0.5, and the Figure 1 caption's 58% gap closure doesn't match Table 1 arithmetic for bioRxiv at eps=1 (I get about 51%). The PubMed/OpenReview comparisons are imported from prior papers, so they are only as strong as those original runs.\n\nThe reward proxy assumption—cosine similarity in embedding space—is a legitimate concern, but the ablations give indirect support and I don't see it as a fatal flaw, just something to probe further.\n\nBottom line: The idea deserves a serious referee. If the privacy accounting is corrected and the numbers hold, this would be a useful contribution. Right now, treat the accuracy comparisons as provisional. Send it to review, but the authors must fix the accounting and report proper statistics.","headline":"Promising DPO-based twist on DP synthetic data, but the privacy accounting in Section 3.1 contradicts Algorithm 1, so the headline epsilon comparisons are not yet trustworthy.","tokens_in":24335,"tokens_out":8557,"would_cite":false,"duration_ms":71170,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"POPri recasts private federated synthetic-data generation as an LLM policy-optimization problem, using DP-noised client preference scores to fine-tune a generator with Direct Preference Optimization.","keywords":["differential privacy","federated learning","synthetic data","policy optimization","direct preference optimization","LLM fine-tuning","on-device learning","benchmark dataset"],"falsifier":"A decisive test: run POPri on a fixed dataset with its true client scores replaced by scores from a deliberately uninformative embedding (e.g., random vectors or a bag-of-characters embedding) and measure downstream accuracy; if accuracy stays high, the reward is not the carrier. More directly, check across rounds whether the aggregated cosine-similarity reward and downstream next-token accuracy move together — if the reward rises while accuracy falls, as the paper's own FID curves suggest can happen late in training, the proxy is not faithful.","tokens_in":23245,"feed_emoji":"🔒","tokens_out":5862,"duration_ms":55249,"temperature":0.7,"pith_summary":"POPri is a method for training an on-device model from private federated text data without ever sending raw client text to a server. It asks participating clients to score server-generated synthetic samples by how similar they are to the client's own data, adds noise to those scores for differential privacy, and then treats the noisy scores as preference feedback to fine-tune a public LLM with Direct Preference Optimization. The fine-tuned LLM produces a large synthetic corpus, on which a small downstream model is trained and deployed on devices. The paper reports that on its new LargeFedBench benchmark and on two earlier central-DP benchmarks, POPri beats DP-FedAvg, DP-FTRL, and Private Evolution baselines on downstream accuracy, and at $\\epsilon=1$ it closes 40-58% of the gap between fully private and fully non-private next-token-prediction accuracy, compared with 1-28% for Private Evolution. The point of the work is that client feedback can be used as an RL reward rather than only as a prompt-engineering signal, which makes better use of the LLM's capacity.","feed_headline":"Policy optimization turns private client feedback into better synthetic data","feed_subtitle":"A DPO-fine-tuned LLM closes up to 58% of the private-to-nonprivate accuracy gap, beating DP-FL and prior synthetic-data methods.","key_machinery":"The carrier of the argument is the construction of preference pairs from DP-noised client scores, combined with Direct Preference Optimization (DPO), an RLHF-style loss that raises the probability of a chosen response and lowers that of a rejected response without training a separate reward model. The scoring function is average cosine similarity between synthetic and private text embeddings rather than the sparse nearest-neighbor histogram used by Private Evolution; the paper's ablations identify this dense scoring as critical for producing informative preference pairs under DP noise. Secure aggregation combines client score vectors, and per-client clipping plus Gaussian noise gives the user-level $\\epsilon,\\delta$-DP guarantee. The mechanism does the work of turning diffuse client feedback into a concrete training signal that improves the LLM's synthetic-data distribution.","core_discovery":"The paper claims that client feedback collected for private synthetic-data generation can be repurposed as a reinforcement-learning reward, enabling policy optimization to fine-tune the data-generating LLM. In POPri, the server generates J synthetic samples from each of K prompts, clients score every sample by the average cosine similarity between its embedding and their private embeddings, and the server aggregates these clipped, Gaussian-noised scores via secure aggregation. For each prompt, the highest-scoring sample becomes the chosen response and the fifth-highest becomes the rejected response in a DPO preference pair; iterating DPO fine-tuning over T rounds moves the LLM's output distribution toward text that clients judge relevant. The resulting generator produces a final synthetic corpus used to fine-tune a small downstream model. The central empirical claim is that this policy-optimized synthetic data outperforms both DP-FL methods and Private Evolution on next-token prediction and text classification, closing the private-to-nonprivate accuracy gap by up to 58% at $\\epsilon=1$.","pith_inferences":["A direct test of the paper's core premise would track whether the aggregated cosine-similarity reward and downstream next-token accuracy move together across rounds; the paper's FID curves suggest they can diverge late in training, so the correlation is not yet established.","The same preference-optimization loop could in principle accept other client-side signals — labels, loss values, or model gradients — as long as they can be aggregated with DP; the paper does not explore these alternatives.","The paper shows the generator can overfit and drift away from the client distribution after several rounds; an adaptive stopping rule or regularizer tuned to downstream validation performance is a natural extension the authors leave implicit.","The method inherits the public LLM's biases, and the paper explicitly leaves open the problem of clients whose data the LLM cannot represent; adding an abstention or 'none of these' option in the client feedback would be a concrete way to test and address that failure mode."],"forward_implications":["At $\\epsilon=1$, POPri's downstream model recovers 40-58% of the accuracy lost to full privacy, versus 1-28% for Private Evolution and 3% for the tested DP-FL methods.","In the bioRxiv case study, POPri cuts per-client communication to roughly 7 million floats downloaded and 18,000 uploaded per round, compared with 82 million floats each way for FedAvg.","The method removes the need for hand-crafted prompts describing client data; only a generic base prompt and client feedback are required.","Because the output is DP synthetic data, the post-processing property of differential privacy allows the final corpus to be reused freely for downstream training without extra privacy loss.","On the central-DP benchmarks PubMed and OpenReview, POPri outperforms Aug-PE with the same synthetic sample budget of 2,000 samples."],"supporting_citations":[{"why":"Defines PrE-Text, the prior synthetic-data FL method POPri builds on and compares against.","marker":"(Hou et al., 2024)"},{"why":"Supplies Aug-PE, the text Private Evolution baseline, its nearest-neighbor histogram scoring, and the PubMed/OpenReview evaluation results POPri must beat.","marker":"(Xie et al., 2024)"},{"why":"Introduces Direct Preference Optimization, the loss POPri uses to fine-tune the LLM from client preference pairs.","marker":"(Rafailov et al., 2023)"},{"why":"Establishes the Private Evolution framework that POPri adapts by replacing in-context prompting with weight fine-tuning.","marker":"Lin et al. (2023; 2025)"},{"why":"Provides the FedAvg DP-FL baseline and the per-client clipping and Gaussian noise precedent POPri's privacy analysis follows.","marker":"(McMahan et al., 2017b)"},{"why":"Provides the DP-FTRL baseline and its privacy accounting method.","marker":"(Kairouz et al., 2021a)"},{"why":"Supplies the secure aggregation protocol POPri uses to combine client score vectors with reduced noise.","marker":"(Bonawitz et al., 2016)"}],"fun_headline_variants":["Client feedback as RL reward sharpens private synthetic data","DPO fine-tuning on private scores closes 58% of accuracy gap","POPri: Policy-optimized synthetic data beats DP-FL in FL","Private client scores as preference pairs for DPO in FL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"POPri's advantage rests on the assumption that 'how similar a synthetic sample looks to a client's data under a sentence-embedding model' is the right measure of how useful that sample will be for training the downstream model.","fun_headline_variants_meta":{"raw":{"variants":["Client feedback as RL reward sharpens private synthetic data","DPO fine-tuning on private scores closes 58% of accuracy gap","POPri: Policy-optimized synthetic data beats DP-FL in FL","Private client scores as preference pairs for DPO in FL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000195,"raw_usage":{"total_tokens":1408,"prompt_tokens":1047,"completion_tokens":361,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":663,"completion_tokens_details":{"reasoning_tokens":288}},"tokens_in":663,"tokens_out":361,"duration_ms":4369,"temperature":1.0,"reasoning_tokens":288,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:03:38.749887+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test: run POPri on a fixed dataset with its true client scores replaced by scores from a deliberately uninformative embedding (e.g., random vectors or a bag-of-characters embedding) and measure downstream accuracy; if accuracy stays high, the reward is not the carrier. More directly, check across rounds whether the aggregated cosine-similarity reward and downstream next-token accuracy move together — if the reward rises while accuracy falls, as the paper's own FID curves suggest can happen late in training, the proxy is not faithful.","supporting_citations":[{"cited_title":"A., Nori, H., Jiang, H., Zhang, H., Lee, Y","cited_arxiv_id":null,"evidence_quote":"Supplies Aug-PE, the text Private Evolution baseline, its nearest-neighbor histogram scoring, and the PubMed/OpenReview evaluation results POPri must beat."}],"review_version":1}