Out-of-Distribution Federated Distillation with Domain-Aware Proxy

Jiahao Xiao; Jiangming Liu

REVIEW 3 major objections 4 minor 12 references

Out-of-Distribution Federated Distillation with Domain-Aware Proxy

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Domain-aware proxy selection lets federated distillation keep its accuracy when test data shifts.

desk verdict Proxy data in the main OOD experiments is sampled from the same unseen domains as the test set, so the reported gains measure test-domain adaptation, not OOD generalization; the core selection idea is still salvageable. read the letter →

arxiv 2608.08525 v1 pith:SJIXD2AN submitted 2026-08-09 cs.LG

Jiahao Xiao , Jiangming Liu This is my paper · ORCID

classification cs.LG

keywords federateddistillationout-of-distributionproxydataselectiondomain-awareLLM-generatedvocabularyconstraintsheterogeneousclientsknowledge

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the weak spot in federated distillation under distribution shift is the proxy data used for knowledge transfer, and that fixing it is enough to produce large gains. It proposes DPS-FD, which each round has every client and the server score proxy samples by cosine similarity to their domain centroids, keep the top-K, and distill on the union. When no real proxy data exists, it generates proxy reviews with an LLM constrained to the global vocabulary of real data. On a five-domain Amazon sentiment benchmark with held-out test domains, the method reports 82.9% F1 with proxy data and 80.6% without, beating the FD baselines and the comparison methods DS-FL, MHAT, and FedKD. A sympathetic reading: proxy-data selection and lexical alignment, not new aggregation rules, are what let FD survive domain shift.

What carries the argument

The load-bearing object is the domain-aware proxy set $D_p^* = \bigcup_{k} P_k \cup P_g$, rebuilt each round. Client $k$ picks $P_k$ as the top-$K$ proxy samples with highest cosine similarity between the proxy sample's feature vector and the client's mean feature vector (centroid) over private data; the server picks $P_g$ similarly using the global centroid over proxy data. Distillation then runs on this union. The complementary machinery is the vocabulary-constrained LLM generator: a global vocabulary is built by tokenizing real proxy data, and few-shot prompts force the LLM to produce reviews using only that vocabulary, pulling synthetic text toward the real lexical distribution (shown by TF-IDF projection).

What would settle it

Run DPS-FD and FD-GP with proxy data drawn only from the five client domains (automotive, baby, clothing, health, sport) and evaluate on the five held-out domains; if the F1 gap over FD-LP shrinks or vanishes, the reported 82.9% result is an artifact of target-domain leakage rather than evidence of OOD generalization.

Watch

Extended reading notes

Core claim

On its own terms, DPS-FD establishes two things. First, substituting a domain-aware proxy set, the union of each client's top-K proxy samples closest to its private-data centroid and the server's top-K samples closest to the global centroid, for the raw proxy set improves global and local F1 under domain shift while slightly reducing communication cost (98.3% and 95.9% of the standard FD cost). Second, when proxy data is unavailable, proxy text generated by an LLM under a vocabulary constraint drawn from real data is close enough to real proxy data to keep most of the benefit; constrained generation outperforms unconstrained generation by about 2.9 F1 points. The paper also reports that global-domain proxy data helps more than local-domain proxy data, and that DPS-FD lifts the weakest client domain (baby) from 47.6% F1 to 84.6%.

Load-bearing premise

The entire OOD claim depends on treating proxy data sampled from the test domains as though it still measures out-of-distribution robustness; if that assumption is dropped, the reported gains may only show that training on target-domain proxy data helps, which is expected.

Editorial extensions

If this is right

If proxy selection is the main lever, FD can be made more resistant to domain shift without changing client training or aggregation rules.
Because the selected proxy set is smaller than the full set, communication cost drops while accuracy rises, so the method's benefit does not trade against bandwidth.
The vocabulary-constraint result implies that lexical distribution matching, not just semantic plausibility, is a controllable factor in LLM-generated proxy data.
The global model's held-out accuracy improves with more clients (from 5 to 10), suggesting that broader domain coverage in private data strengthens the global centroid and the selected proxy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the comparison FD-GP uses global proxy data composed of samples from the very domains that are unseen by the clients, so the strong result may be measuring adaptation to the test domains via proxy leakage rather than generalization to truly unseen domains; a stricter protocol with proxy restricted to client domains would separate the two.
Editorial inference: since selection is by similarity to centroids, a testable extension is replacing the top-K cosine rule with uncertainty- or diversity-aware sampling to see whether the gains come from denoising or from retaining hard samples; the paper itself notes that smaller top-K can remove informative near-boundary samples.
Editorial inference: the same selection-plus-constrained-generation recipe could transfer to other token-based modalities such as code or clinical notes, where domain vocabularies are cheap to construct, though the paper only demonstrates sentiment classification.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the central OOD claim doesn't hold as stated. The with-proxy experiments give the server proxy data sampled from the same domains as the OOD test set. Section 5.3 defines FD-GP as "global proxy data composed of samples from all domains that are unseen by the clients," and Section 4.1 defines the OOD test set as domains not seen by the clients. So the global model is distilled on unlabeled data from the test distribution. The reported 82.9 vs 81.4 F1 gain is target-domain adaptation, not out-of-distribution robustness. This is the load-bearing flaw, and it contaminates the main result.

What's actually new: the DPS selection rule (top-K cosine similarity to client and global centroids, then union) is clean and computationally cheap. The vocabulary-constrained LLM generation is a reasonable answer to the no-proxy-data problem for NLP, and the analysis in Figure 4 shows the vocabulary constraint does pull generated data closer to real word statistics. Credit also for releasing code.

The without-proxy arm has a softer but related problem: the vocabulary is built from real proxy data, so "without proxy" is not strictly true; it is "without sharing proxy data for distillation." That may be acceptable if clearly stated, but it's not stated.

Other soft spots: DAFKD (Wang et al., 2023a), a closely related domain-aware FD method, is cited but never compared. There are no error bars or repeated trials; the p<0.001 claim needs a statistical method. The top-K parameters appear to be chosen on the test set (Table 3 reports several values but no held-out selection).

Who is this for? Someone working on federated distillation might still find the selection heuristic worth trying in a properly set OOD benchmark. But as submitted, the evidence does not back the advertised claim. I'd want to see a revision with a clean OOD protocol: proxy data drawn from a disjoint distribution (e.g., a broader pool that does not contain test domains, or a leave-one-domain-out setup), and the without-proxy arm either using no real proxy data at all or clearly labeled as vocabulary-only access. That is a fixable paper, and the core idea is not wrong, just mistested.

Recommendation: I would not reject it outright as nonsense, but I would not accept it as is. A serious referee could help the authors redesign the experiments. If the editor's policy is to desk reject papers with load-bearing design flaws, that's defensible. If they are willing to send it out, the review will be straightforward: point at Section 5.3 and ask for the proper control.

Referee Report

3 major / 4 minor

Summary. The paper proposes DPS-FD, a domain-aware proxy selection framework for federated distillation, in which each client and the server select top-K proxy samples most similar to their feature centroids (Eqs. 6-9) to construct a domain-aware proxy set. When no proxy data is available, the authors introduce a vocabulary-constrained LLM-based generation strategy (Section 4.3) that uses a global vocabulary derived from real proxy data. Experiments on Amazon review sentiment classification with heterogeneous clients report improved global F1 over baselines: 82.9% with proxy data and 80.6% without proxy data in Tables 1 and 4.

Significance. The idea of jointly selecting proxy samples at both client and server levels is coherent, and the method is computationally light and clearly described. The release of code and data is commendable. However, the experimental protocol does not measure out-of-distribution generalization as claimed: the with-proxy baseline FD-GP uses proxy data from the same domains as the OOD test set, and the without-proxy pipeline depends on real proxy data for vocabulary construction. As a result, the central advertised claim that DPS-FD addresses OOD distribution shifts is not supported by the evidence presented.

major comments (3)

[5.3, 5.1, 4.1] The main with-proxy evaluation does not constitute an OOD test. Section 5.3 defines FD-GP as using "global proxy data composed of samples from all domains that are unseen by the clients," and Section 4.1 defines the OOD test set as "sampled from the domains not seen by the clients." Thus the proxy data in the principal comparison is drawn from the same domains as the test set. DPS-FD then selects top-K proxy samples via Eqs. (6)-(9), and the global model is distilled on unlabeled samples from the test distribution. The reported improvement in Table 1 (DPS-FD 82.9 vs FD-GP 81.4 global F1) therefore measures target-domain adaptation, not out-of-distribution robustness. This invalidates the abstract's claim that the model "effectively address[es] the challenges of distribution shifts under OOD."
[4.3] The without-proxy pipeline is not self-contained. Section 4.3 states that "we first construct a vocabulary from tokenizing real proxy data," which presupposes access to proxy data even in the no-proxy scenario. If that real proxy data includes the unseen test domains (as in the with-proxy setup), the generated proxy inherits lexical information from the test distribution, so the without-proxy results in Table 1 also fail to establish OOD generalization. The Limitations section does not acknowledge this dependency on real proxy data for the vocabulary constraint.
[Tables 1-2, 5.4] The captions of Tables 1 and 2 report p<0.001, but the manuscript does not state the statistical test used, the number of random seeds, or any variance measure over runs. Without this information, the small reported gaps (e.g., 82.9 vs 81.4 global F1) cannot be assessed for significance, further weakening the support for the central claim.

minor comments (4)

[1] There is a typo in Section 1: "promblem" should read "problem."
[4.3] The phrase "aVocabulary-Constrained" in Section 4.3 is missing a space; it should read "a Vocabulary-Constrained."
[5.3] In Section 5.3, "DPS-FDis" is missing a space; it should read "DPS-FD is."
[Figure 3] Figure 3's axis labels appear garbled in the manuscript text, with repeated "/uni000000..." strings, which makes the figure caption and the top-K analysis hard to interpret.

Circularity Check

2 steps flagged · score 6.0 of 10

Reported OOD gains rely on proxy data drawn from the same unseen domains as the test set; the no-proxy arm also builds its vocabulary from real proxy data.

self definitional [Section 4.1 (OOD Benchmark Construction) and Section 5.3 (Baselines)]
"we simulate OOD settings by replacing the original test set, which is composed of data from the client domains, with a test set sampled from the domains not seen by the clients. ... FD-GP uses global proxy data composed of samples from all domains that are unseen by the clients. ... DPS-FD is our proposed model, applying DPS on FD with global domain proxy data..."

The OOD test set is sampled from domains not seen by the clients, and the with-proxy baselines and DPS-FD are distilled on proxy data from exactly those same unseen domains. The server therefore transfers knowledge about the test distribution into the global model before evaluation; Table 1's global-distribution F1 measures adaptation to target-domain proxy data, not generalization to truly unseen domains. The claimed OOD setting and the proxy input are the same domain set by construction.
fitted input called prediction [Section 4.3 (LLM-Based Proxy Generation) and Section 5.3 (Baselines)]
"we first construct a vocabulary from tokenizing real proxy data, which serves as a constraint to ensure that generated samples adhere to realistic lexical distributions and domain-specific semantics. ... FD-VC leverages a vocabulary-constrained LLM-based generation strategy to synthesize proxy data without proxy data."

The "without proxy data" arm is evaluated after constructing its generation constraint from real proxy data. The LLM's vocabulary and few-shot demonstrations are fitted to the real proxy distribution, so the generated proxy is anchored to the data that the scenario claims to do without. If the real proxy data includes the unseen target domains (as in the global proxy setting), the no-proxy pipeline inherits target-domain lexical patterns, so the advertised no-proxy OOD result is not independent of proxy data.

full rationale

The central empirical claim is circular in its operationalization. Section 4.1 defines the OOD test set as domains unseen by the clients, while Section 5.3 defines FD-GP, and the with-proxy DPS-FD, as using global proxy data from all domains unseen by the clients. Distilling on unlabeled data from the test domains and then evaluating on those same domains measures target-domain adaptation, not out-of-distribution generalization. The without-proxy arm has a parallel issue: Section 4.3 constructs the vocabulary from real proxy data, so the allegedly proxy-free generation still depends on real proxy text. These are not mathematical tautologies, but the benchmark makes the headline result largely an in-distribution or transductive result. Self-citations such as Xiao and Liu (2025) appear for the non-IID setup and the claim that distillation struggles without proxy data, but they are not the primary source of circularity; even setting them aside, the proxy/test domain overlap is sufficient to invalidate the OOD claim as stated. A design that kept proxy domains disjoint from test domains would have avoided this problem.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the FD objective from prior work, the domain-relevance assumption that cosine similarity to centroids selects useful proxy samples, a clean OOD setup that is in fact violated, and a plausibility assumption about constraining LLM vocabulary. Top-K sizes and the review length are hand-set hyperparameters. No new theoretical entities are introduced.

free parameters (4)

client top-K (per-client selection size) = 25,000
Hand-picked after a sweep in Table 3; final value selected to maximize global F1, no validation split described.
server top-K (server selection size) = 35,000
Hand-picked after the sweep in Table 3; same selection concern as the client top-K.
average generated review length = ~90 words
Prompt template instructs the LLM to generate about 90 words; heuristic, no ablation reported.
softmax temperature tau
Used in soft labels in Eq. (3) but its value is not reported; a hyperparameter that affects distillation.

assumptions (4)

domain assumption The global model and client models produce comparable feature spaces for centroid similarity (Eq. 6-7).
The method assumes cosine similarity between proxy features and centroids is a meaningful domain-relevance signal; this is plausible but unvalidated.
domain assumption The OOD test domains are not represented in the proxy data used for training.
The paper's claim of OOD evaluation relies on this premise, but it is violated by the FD-GP baseline and DPS-FD with-proxy setting (Section 5.3), which use proxy data from unseen domains.
ad hoc to paper The LLM can be constrained to generate only words from a fixed vocabulary.
No enforcement mechanism is described, so this is an unsupported procedural assumption.
standard math Weighted KL aggregation of soft labels in Eq. (4) is a valid global objective.
Standard federated distillation objective from prior work (Lin et al., 2020); accepted as background.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Out-of-Distribution Federated Distillation with Domain-Aware Proxy." pith.science (2026). https://pith.science/paper/SJIXD2AN

@misc{pith2026260808525,
  author       = {Pith},
  title        = {Pith review of: Out-of-Distribution Federated Distillation with Domain-Aware Proxy},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SJIXD2AN}},
  note         = {Machine review of arXiv:2608.08525}
}

read the original abstract

Federated Learning is a distributed machine learning paradigm that trains a global model by aggregating local clients without sharing private data of each client. Federated Distillation (FD) builds upon this paradigm by leveraging knowledge distillation to exchange soft predictions on proxy data instead of model parameters, enabling more efficient communication and supporting heterogeneous model collaboration. However, FD models trained on In-Distribution data are hardly adapted to Out-of-Distribution (OOD) scenarios. In this paper, we propose a domain-aware proxy selection framework to better adopt proxy data for OOD problems. The experimental results show that the proposed models effectively address the challenges of distribution shifts under OOD with and without proxy data by achieving average 82.9\% and 80.6\% over existing works on standard benchmarks. The codes and data are released in https://anonymous.4open.science/r/DPS-FD-8596.

Figures

Figures reproduced from arXiv: 2608.08525 by the authors.

**Figure 1.** DPS-FD. stance and its corresponding label, respectively, C is the number of classes, and Nk is the size of private data. The objective is to train a globally optimal model fg parameterized by θg through aggregating client models fk parameterized by θk in a decentralized manner without exposing private data on the server. To mitigate the limitations of FL based on parameter exchange, FD leverages knowledge distill… view at source ↗

**Figure 2.** LLM-based Proxy Generation with Vocabulary-Constraints. shown in [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** F1 scores varies with the top-K selections of the clients and server Communication Cost To facilitate a fair comparison, we normalize the communication cost of the traditional FD method to 100% as a baseline. As shown in the column cost of [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Word distributions of proxy data generated [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: Domain distributions across different do [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 10 canonical work pages

[1]

Identify the product domains and senti- ment tendencies represented in the exam- ples above, and infer additional potential domains that could reasonably exist

work page
[2]

Based on the examples’ and inferred do- mains, generate exactly realistic and diverse unlabeled review texts

work page
[3]

InProceedings of the ACM Web Conference 2023, pages 3077–3085

Duet: A tuning-free device-cloud collaborative parameters generation framework for efficient device model generalization. InProceedings of the ACM Web Conference 2023, pages 3077–3085. Xinge Ma, Jin Wang, and Xuejie Zhang. 2025. Data- free black-box federated learning via zeroth-order gradient estimation. InProceedings of the AAAI Con- ference on Artifici...

work page 2023
[4]

Do not shorten because you need to output many

Each review should be about 90 words (minimum 40, maximum 180). Do not shorten because you need to output many

work page
[5]

Maintain variability in tone, sentence structure, and vocabulary

The reviews should not be templates or mechanically repeated. Maintain variability in tone, sentence structure, and vocabulary. Output format:

work page
[7]

Each review should be fluent, reflecting the natural style of customer reviews

work page
[10]

Output each reviews on a separate line

work page
[11]

Ensure the quality of each generated sen- tence and do not sacrifice quality for the sake of quantity

work page

Show all 12 references

[12]

Begin now:

Do not add any prefix, numbering, head- ers, or label—only the raw reviews. Begin now:
[2022]

In2022 IEEE International Conference on Trust, Security and Pri- vacy in Computing and Communications (TrustCom), pages 687–694

Flightner: A federated learning approach to lightweight named-entity recognition. In2022 IEEE International Conference on Trust, Security and Pri- vacy in Computing and Communications (TrustCom), pages 687–694. IEEE. Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormand...

2018 arXiv
[2023]

InProceedings of the 31st ACM International Conference on Multimedia, pages 1536–1545

Joint local relational augmentation and global nash equilibrium for federated learning with non-iid data. InProceedings of the 31st ACM International Conference on Multimedia, pages 1536–1545. Xinting Liao, Weiming Liu, Chaochao Chen, Pengyang Zhou, Fengyuan Yu, Huabin Zhu, Bi...

2020 arXiv
[2024]

In 2024 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6

Attentive modeling and distillation for out-of- distribution generalization of federated learning. In 2024 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. Zhuang Qi, Sijin Zhou, Lei Meng, Han Hu, Han Yu, and Xiangxu Meng. 2025. Federated deconfounding a...

2024 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Identify the product domains and senti- ment tendencies represented in the exam- ples above, and infer additional potential domains that could reasonably exist

work page

[2] [2]

Based on the examples’ and inferred do- mains, generate exactly realistic and diverse unlabeled review texts

work page

[3] [3]

InProceedings of the ACM Web Conference 2023, pages 3077–3085

Duet: A tuning-free device-cloud collaborative parameters generation framework for efficient device model generalization. InProceedings of the ACM Web Conference 2023, pages 3077–3085. Xinge Ma, Jin Wang, and Xuejie Zhang. 2025. Data- free black-box federated learning via zeroth-order gradient estimation. InProceedings of the AAAI Con- ference on Artifici...

work page 2023

[4] [4]

Do not shorten because you need to output many

Each review should be about 90 words (minimum 40, maximum 180). Do not shorten because you need to output many

work page

[5] [5]

Maintain variability in tone, sentence structure, and vocabulary

The reviews should not be templates or mechanically repeated. Maintain variability in tone, sentence structure, and vocabulary. Output format:

work page

[6] [7]

Each review should be fluent, reflecting the natural style of customer reviews

work page

[7] [10]

Output each reviews on a separate line

work page

[8] [11]

Ensure the quality of each generated sen- tence and do not sacrifice quality for the sake of quantity

work page