{"id":"f725cfdf-fc06-44ce-90a0-ef93c6a97e0a","arxiv_id":"2412.16669","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"P3EFT reduces label leakage during API-based LoRA fine-tuning by splitting gradients across servers and mixing multiple adapters while keeping most task accuracy.","lead":"This paper proposes P3EFT, a split-learning protocol that lets clients fine-tune large hosted models with LoRA while obscuring private labels from the server. It tests a practical privacy fix for fine-tuning APIs on DeBERTa, Flan-T5, and LLaMA-2 and reports accuracy close to non-private fine-tuning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-provider trajectory attack defeats private_backprop; P3EFT's label-privacy claim is conditional on non-colluding servers or TEEs, which standard fine-tuning APIs do not provide.","rationale":"The reader's weakest_assumption identifies the right pressure point. Section 3.3's private_backprop is linear in the output gradient, and the protocol's security against a single step assumes the m pieces go to servers that cannot combine them or observe consecutive optimizer states. Section 3.3 explicitly concedes that a server seeing θ_t and θ_{t+1} can recover g_θ by inverting the optimizer, and Section 5 defers long-term interaction attacks. In the paper's own motivating application, one provider operates the fine-tuning API; that provider receives every piece of every backprop call and every adapter state, so it can carry out the conceded attack. The Appendix C analysis does not close this gap: it is a per-step, single-batch white-box argument (m=B for a single server) and says nothing about the multi-step trajectory or the activations' mixing regularizer. I therefore do not see a reason to move the reader's CONDITIONAL verdict: the empirical comparison against DC/PSLF is useful, the code is promised, and the honest limitation statements are present, but the headline privacy claim is conditional on infrastructure that standard APIs do not provide. A single simulation of the single-server trajectory attack is the decisive check.","tokens_in":17044,"tokens_out":10295,"duration_ms":96731,"concrete_test":"Run the released P3EFT code on DeBERTa/SST-2 against a single adversarial server that logs all m backprop inputs/outputs and the successive adapter states θ_t. The server then attempts the optimizer-inversion attack of Section 3.3: for the published optimizer, solve for the aggregate gradient g_t consistent with θ_t → θ_{t+1}, and feed the recovered per-step gradients into the k-means/XGBoost label-inference pipeline used in Section 4.1. If the recovered-label AUC approaches the Regular FT value (≈99.1 in Table 1) rather than the reported P3EFT leak (≈62.6), the central privacy claim is refuted in the standard single-provider setting; if AUC stays near chance, the non-collusion assumption can be relaxed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The decisive weakness is the single-provider trajectory attack. P3EFT's gradient protection (Section 3.3) assumes the m obfuscated backprop pieces are processed by independent servers that never see consecutive client states. The paper itself concedes that a server which sees θ_t and θ_{t+1} can invert the optimizer (e.g., g_θ = (θ_t − θ_{t+1})/η under SGD) and can then propagate back to g_h when the Jacobian is invertible; Section 5 defers long-term client–provider interaction attacks to future work. A standard fine-tuning API is operated by one provider, so that provider receives every piece of every backprop call and every adapter state across the whole trajectory. This re-opens the Section 3.2 gradient-clustering attack that private_backprop was designed to close, regardless of the noise variance used for individual pieces. The Appendix C guarantee covers only a single batch/step under a white-box model and explicitly leaves the general case open; the full P3EFT activation-mixing regularizer (Section 3.4) has no formal privacy proof. Thus the headline claim 'label privacy throughout training' holds only under multi-server non-collusion or TEE deployment, neither of which is the advertised fine-tuning API setting.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies label leakage in split learning / API fine-tuning with parameter-efficient fine-tuning (PEFT). It observes that backpropagation is linear in the output gradient for fixed inputs and parameters, and builds on this to design private_backprop (Algorithm 1), which decomposes the client's gradient into m obfuscated pieces sent to independent servers and exactly recovers the true gradient by linearity. To protect activations, the full P3EFT algorithm (Algorithm 2) trains n LoRA adapter sets with randomized mixing weights (Eq. 4-5) and an adversarial regularizer (Eq. 7). Experiments on DeBERTa-v2-XXLarge, Flan-T5-Large, and LLaMA-2 7B over SST-2, QNLI, and MNLI report that P3EFT retains accuracy close to non-private LoRA fine-tuning while reducing empirical leakage relative to regular fine-tuning and the distance-correlation baseline. Appendix C provides a white-box, per-step analysis of private_backprop, and Appendix B reports an additional PSLF baseline and an ablation on the number of adapters.","tokens_in":17223,"tokens_out":5287,"duration_ms":49415,"significance":"The conditional-linearity observation underlying private_backprop is elegant, and the exact gradient reconstruction is a useful protocol primitive for vertical split learning with PEFT. The experimental coverage is broad: three model families with up to 7B parameters, several GLUE tasks, and a released codebase. The paper is also transparent about several limitations. If the multi-server non-collusion assumption holds, P3EFT provides a practical way to reduce label leakage without replacing the base model, loss, or LoRA adapters. However, the headline claim that labels are kept private 'throughout training' is not supported in the standard single-provider API setting, and the full algorithm has no formal privacy guarantee; the reported leak metrics are attack-based empirical values rather than privacy bounds. The contribution is solid within its stated assumptions, but the claims need to be narrowed and the trajectory-attack gap needs to be addressed.","major_comments":[{"comment":"The paper's headline claim that P3EFT 'can maintain label privacy throughout training' (Abstract and Section 1) is not supported in the primary fine-tuning-API setting it targets. As the text itself concedes in Section 3.3, if the same server receives two consecutive parameter sets θ_t and θ_{t+1}, it can invert the optimizer update (e.g., g_θ = (θ_t − θ_{t+1})/η under SGD) and, when the Jacobian is invertible, recover g_h. A standard fine-tuning API is operated by one provider, so that provider observes every obfuscated backprop piece and every adapter state over the whole trajectory; this re-opens the Section 3.2 gradient-based label-recovery attacks regardless of the per-piece noise variance. The mitigations mentioned in Section 3.3 (non-colluding servers or trusted execution environments) are not analyzed, evaluated, or incorporated into the threat model used for the experiments. To make the central claim defensible, the paper should either restrict the privacy claims to the multi-server/TEE setting with an explicit threat model, or provide and evaluate a concrete mechanism that provably prevents trajectory inversion (for example, securely hidden optimizer statistics) and show that it does not degrade training.","section":"Section 3.3, consecutive-steps attack"},{"comment":"Appendix C analyzes only Algorithm 1 (private_backprop), under a white-box attacker, for a single batch/step, and for binary classification; the full P3EFT algorithm with randomized mixing weights (Eq. 4-5) and the adversarial regularizer (Eq. 7) has no formal privacy analysis. The privacy scores in Tables 1-3 are empirical attack-based metrics (ROC AUC, K-means accuracy), not privacy guarantees, so the abstract's phrase 'keeping the labels private' is stronger than what is demonstrated. I recommend either reframing the claims as resistance to the tested attacks or providing a formal guarantee for the full protocol. Additionally, the single-server case in Appendix C requires m ≥ B pieces per batch, so the communication cost of privacy-preserving backprop grows linearly in batch size; this cost should be stated explicitly in the main text alongside the claim that P3EFT has low overhead.","section":"Section 3.4 and Appendix C"}],"minor_comments":[{"comment":"The return statement in Algorithm 2 refers to an index M that is not defined; it should be n, the number of adapter sets.","section":"Algorithm 2, line 19"},{"comment":"The text contains a broken cross-reference to 'Appendix ??' for the detailed LoRA description; the reference should be resolved.","section":"Section 2.2"},{"comment":"The definition of the mixing weights W in Eq. (5) is hard to parse: W is written as a column of row vectors, while the text uses W_i ⊙ h(x, θ_i). Please define W_i as the i-th row explicitly and make the dimensions of all quantities clear.","section":"Equation (5)"},{"comment":"The abstract uses P$^3$EFT while the body consistently uses P3EFT; please unify the notation.","section":"Abstract and body"},{"comment":"In the private-backprop experiment, the text says 'n = 2 with noise variance set to 1000,' but Algorithm 1 uses the parameter m for the number of shards and z for the noise; please clarify whether n here is m and what distribution z is drawn from.","section":"Section 4.1"},{"comment":"The rows with ε = 0 in the PSLF baseline are described as 'random labels'; since label differential privacy is usually defined with ε > 0, consider presenting these runs as an ablation of the privacy budget rather than as part of the DP grid.","section":"Tables 4 and 7"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about the single-provider trajectory attack is valid and is the main reason for major revision. The paper is honest enough to acknowledge the attack in Section 3.3 and defer it in Section 5, but the abstract and contribution list still make the unqualified claim of label privacy 'throughout training.' The authors should either narrow the claim to the multi-server/TEE setting or provide and evaluate a concrete defense against the trajectory attack. The lack of a formal privacy guarantee for the full P3EFT algorithm is a second concern that should be addressed by reframing or by adding a formal analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper gives a plausible, well-tested protocol for hiding labels in split learning with LoRA, but the headline claim is too strong for the setting it advertises. The privacy guarantee is attack-specific and rests on multiple servers not colluding or seeing consecutive states. For a single-provider fine-tuning API, the trajectory attack described in the paper itself reopens the leak.\n\nWhat is genuinely new and good: the analysis of LoRA label leakage in split learning (Figure 2) is a nice concrete demonstration. The P3EFT construction is clever—using linearity of backprop to split gradients across servers, mixing multiple LoRA adapters to obfuscate activations, and adding an adversarial regularizer to stop individual adapters from leaking. The experiments cover three model families and several GLUE tasks, with accuracy close to non-private fine-tuning and leak metrics well below regular FT and DC. They ship code, and Appendix C gives a real, if restricted, formal analysis of the private backprop protocol.\n\nSoft spots: the abstract says 'keeping labels private' and 'maintain label privacy throughout training.' What is actually measured is resistance to three specific attacks. There is no formal guarantee for the full algorithm, and the single-server trajectory risk (Section 3.3) is a real gap: the same server can sum the m pieces from one step or invert the optimizer across steps. The paper acknowledges this and defers long-term interaction attacks to future work, but that deferral undercuts the API fine-tuning motivation, where one provider sees everything. The main tables also do not report m (the number of obfuscated gradient pieces), and some baselines lack error bars. Minor: the DC baseline is a reimplementation, but the comparison seems fair.\n\nBottom line: this is a serious empirical paper, honest about its limitations, and the core idea is worth knowing. It deserves peer review, not desk rejection. My recommendation: send it out, and ask the authors to scope the privacy claim to the threat model and ideally add an experiment with the trajectory attack.","headline":"A useful, honest empirical protocol for label privacy in split learning with PEFT, but the advertised 'label private' claim holds only under non-colluding servers or TEEs.","tokens_in":17842,"tokens_out":1633,"would_cite":true,"duration_ms":15702,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Split learning can hide training labels behind mixed adapters and masked gradients.","keywords":["label privacy","split learning","LoRA","parameter-efficient fine-tuning","vertical federated learning","gradient obfuscation","adversarial regularization","fine-tuning API"],"falsifier":"Collect the per-server gradient pieces and adapter states from a P3EFT run in which the same provider executes both calls and logs consecutive optimizer states, then train a classifier to recover batch labels from the pieces; if the classifier achieves test AUC far above chance on a balanced set, the non-collusion assumption is doing the work and label privacy fails in single-provider deployments.","tokens_in":1652,"feed_emoji":"🔒","tokens_out":1825,"duration_ms":58448,"temperature":0.7,"pith_summary":"The paper argues that label privacy in API fine-tuning of large models does not require changing the base model, the loss, or the LoRA adapters; it can be obtained by an orchestration protocol that exploits properties of parameter-efficient training. It first shows that ordinary LoRA split learning leaks labels through both communicated gradients and learned activations, as demonstrated by clustering and classifier attacks. The proposed method, P3EFT, hides gradients by splitting each true gradient into random pieces sent to different servers, and hides activations by training several adapters whose outputs are mixed with secret weights while an adversarial regularizer stops individual adapters from encoding the label. On DeBERTa, Flan-T5, and LLaMA-2 across GLUE tasks, the paper reports near-baseline accuracy, such as 96.5% versus 96.9% on SST-2 with DeBERTa, with measured label leakage much lower than standard fine-tuning and competitive with the distance-correlation defense.","feed_headline":"P3EFT keeps labels private in fine-tuning for under one point","feed_subtitle":"Mixing several LoRA adapters and splitting gradients across servers blocks label-recovery attacks with near-baseline accuracy.","key_machinery":"The machinery is the conditional linearity of backpropagation, $\\mathrm{backprop}(x,\\theta,g_h)=g_\\theta$, which holds for fixed $x,\\theta$ even when the model is nonlinear because backprop multiplies $g_h$ by the Jacobian $\\partial h(x,\\theta)/\\partial\\theta$. This turns private gradient computation into a linear secret-sharing problem: the client writes $g_h=\\sum_{i=1}^{m}\\alpha_i\\hat{g}^{(i)}_h$ with large random pieces, sends each piece to a separate server, and recovers $g_\\theta=\\sum_i\\alpha_i\\,\\mathrm{backprop}(x,\\theta,\\hat{g}^{(i)}_h)$ exactly. For activation protection, the client keeps $n$ LoRA adapter sets $\\theta_1,\\dots,\\theta_n$ and merges their outputs with randomly generated mixing weights $W$, initialized so the mixture equals the mean of the individual activations at step zero; an adversarial linear head per adapter serves as a regularizer so no single $h(x,\\theta_i)$ predicts the label.","core_discovery":"The central claim is that label privacy in the two-party split-learning setup can be maintained throughout training with a significantly smaller accuracy drop than previous defenses, and the mechanism is specific to parameter-efficient fine-tuning. Concretely, the paper shows that backpropagation is conditionally linear in the output gradient for fixed inputs and adapter weights, so a client can decompose its gradient into m random pieces, send one piece to each of m non-colluding servers, and recover the exact parameter gradient by a weighted sum without any server seeing the true gradient. For activations, the client keeps n independent LoRA adapter sets and combines their outputs with a secret coordinatewise mixing matrix, so the mixed model matches the standard fine-tuning objective while each individual adapter's activations carry no label information. A client-side adversarial linear head regularizer, applied to each adapter separately, prevents the adapters from learning to leak labels over time. The experiments support that this combination achieves accuracy close to unperturbed LoRA fine-tuning while reducing worst-case privacy leakage on the tested metrics.","pith_inferences":["Editorial extension: if a server can observe many mixed activations across batches, it may attempt blind source separation to estimate the mixing weights $W$; the paper does not analyze this attack, and it would be the natural next test of the activation-hiding component.","Editorial extension: the white-box analysis in the appendix suggests that $m=B$ (one piece per batch element) protects all labels in a single-server batch, implying a concrete scale-up path where the client sends one basis vector per example rather than per gradient.","Editorial extension: the authors' own framing points toward combining P3EFT with input-privacy methods; a practical privacy mode in an API would likely need both, since the input text itself can carry label information.","Editorial extension: if P3EFT were deployed by an API provider using trusted execution environments, the method would give clients label privacy without requiring them to run any part of the model locally, which is a stronger practical guarantee than prior prompt-tuning-only defenses."],"forward_implications":["If P3EFT is correct, any existing LoRA fine-tuning workload can be made label-private by wrapping the API calls in the private-backprop procedure and training several adapter copies, without retraining the base model or changing the loss.","Because the reconstructed parameter gradient is exactly the true gradient, the private-backprop part of P3EFT does not change the training dynamics at all; the entire accuracy cost comes from activation mixing and the adversarial regularizer.","The protocol transfers to other PEFT methods and to any vertical split-learning setting where gradients must propagate through an untrusted party, not only the NLP benchmarks tested.","For single-provider fine-tuning APIs, the paper's privacy guarantee requires trusted execution environments or some other way to make consecutive calls non-linkable; without that, the label-privacy claim does not apply in the most common deployment.","The paper's ablation suggests that even a single adapter set, trained with the private-backprop and regularizer machinery, can be competitive, indicating a simpler deployment path than the full multi-adapter scheme."],"supporting_citations":[{"why":"defines LoRA, the PEFT method whose adapters and hyperparameters all experiments use.","marker":"Hu et al. (2022)"},{"why":"supplies the label-recovery attack methodology and the two-party split-learning setting this work extends to large models.","marker":"Li et al. (2022)"},{"why":"provides the distance-correlation defense that P3EFT is compared against as the main baseline.","marker":"Sun et al. (2022)"},{"why":"provides PSLF, the label-differentially-private baseline that the paper reimplements and finds unstable.","marker":"Wan et al. (2023)"},{"why":"inspires the adversarial linear-head regularizer that keeps individual adapters from encoding labels.","marker":"Ganin & Lempitsky (2015)"},{"why":"supplies the secure-aggregation analogy for splitting a secret gradient into shares that sum to the true value.","marker":"Bonawitz et al. (2017)"}],"fun_headline_variants":["P3EFT hides labels in split fine-tuning at near-zero accuracy cost","Label privacy for large-model fine-tuning: P3EFT delivers","Secret gradient splitting: P3EFT keeps labels private in PEFT","Fine-tune large models privately with P3EFT's LoRA mixing"],"cache_read_input_tokens":19968,"weakest_assumption_plain":"The protocol only protects labels if the servers that receive the scrambled gradient pieces never cooperate, never see consecutive optimizer states, and faithfully run the prescribed computations; a single provider that logs both calls can undo the hiding.","fun_headline_variants_meta":{"raw":{"variants":["P3EFT hides labels in split fine-tuning at near-zero accuracy cost","Label privacy for large-model fine-tuning: P3EFT delivers","Secret gradient splitting: P3EFT keeps labels private in PEFT","Fine-tune large models privately with P3EFT's LoRA mixing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000641,"raw_usage":{"total_tokens":2968,"prompt_tokens":981,"completion_tokens":1987,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":1911}},"tokens_in":597,"tokens_out":1987,"duration_ms":11576,"temperature":1.0,"reasoning_tokens":1911,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:21:55.893486+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect the per-server gradient pieces and adapter states from a P3EFT run in which the same provider executes both calls and logs consecutive optimizer states, then train a classifier to recover batch labels from the pieces; if the classifier achieves test AUC far above chance on a balanced set, the non-collusion assumption is doing the work and label privacy fails in single-provider deployments.","supporting_citations":[{"cited_title":"Unsupervised domain adaptation by backpropagation","cited_arxiv_id":null,"evidence_quote":"inspires the adversarial linear-head regularizer that keeps individual adapters from encoding labels."},{"cited_title":"Practical secure aggregation for privacy-preserving machine learning","cited_arxiv_id":null,"evidence_quote":"supplies the secure-aggregation analogy for splitting a secret gradient into shares that sum to the true value."}],"review_version":1}