Pith. sign in

REVIEW 4 major objections 4 minor 59 references

Rethinking Personalized Reward Modeling for LLMs under Preference Heterogeneity via Group-Debiased Federated Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single federated reward model, near random before adaptation, out-adapts per-group models after a few local steps under balanced preference groups.

desk verdict A genuinely new empirical observation about FedAvg as a personalization seed, but the strong 'group-specific initialization is unnecessary' claim is only fully supported on a synthetic benchmark; the real-world evidence is thin and the benefit is mostly speed. read the letter →

arxiv 2608.01556 v1 pith:DJERVDEO submitted 2026-08-03 cs.LG cs.AI

Seongyoon Kim , Boryeong Cho , Jihwan Oh , Seokhyun Chung , Se-Young Yun This is my paper · ORCID
classification cs.LGcs.AI
keywords federatedlearningrewardmodelingpreferenceheterogeneitypersonalizationgroupdebiasingflatnessLLMalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Federated learning for LLM reward models usually responds to users who disagree by clustering users and training one reward model per cluster, on the assumption that each group needs its own initialization. This paper argues that assumption is unnecessary: under balanced preference groups, a single model trained with standard federated averaging (FedAvg) starts at near-random accuracy yet surpasses reward models trained separately for each ground-truth group after only ten local fine-tuning steps. The explanation is that averaging cancels conflicting side choices while preserving shared feature detectors, leaving the model in a flat region of the loss near the decision boundary. Group imbalance breaks this cancellation, so the paper proposes FedGD, which discovers latent preference groups during training and uses group-debiased client sampling to keep the shared initialization flat. If correct, the practical implication is that personalized reward modeling should invest in one debiased shared initialization rather than multiple group-specific models.

What carries the argument

The key diagnostic is the Gradient Quotient (GQ), a layer-wise measure of how much the local gradient changes after one optimization step; a small GQ means the loss is flat, so successive local steps stay aligned. The complementary measure, Headroom, records whether one local step moves a held-out sample toward its correct preference. The mechanism that produces low GQ under FedAvg is the two-component decomposition of preference learning: clients share the knowledge of which feature separates the two responses and differ only in which side they prefer, so averaging across clients preserves the shared feature and cancels the opposing side choices. FedGD turns this into an algorithm by discovering latent preference groups with clustered federated learning, letting each client join the expert with the lowest validation loss, and then training a single reward model with group-debiased client sampling plus hierarchical aggregation so each discovered group contributes equally.

What would settle it

Construct a balanced federated preference task in which each client's response pairs are disjoint from every other client's, train a single FedAvg model and per-group models, and check whether FedAvg still has substantially lower GQ and still exceeds per-group accuracy after ten local steps; if either fails, the cancellation mechanism and the central claim collapse.

Watch

Extended reading notes

Core claim

The paper's central claim is that the quality of an initialization for personalized reward modeling is determined by loss flatness, not by pre-adaptation accuracy or by matching a client's group. Under balanced preference groups, a single FedAvg model starts at 50.40% accuracy and reaches 93.45% after ten local steps, surpassing a centralized model trained per ground-truth group (Multi-CL), which starts at 92.20% and ends at 91.45%. The mechanism is that average aggregation preserves the shared representation of the response-distinguishing feature while canceling opposite preference directions, leaving initial reward margins small and local gradients slowly varying. Group imbalance makes the cancellation asymmetric and pushes minority clients' samples deep into saturated margins, so ten local steps cannot recover them. A group-debiased sampling oracle restores the flat initialization, and FedGD realizes this without knowing the groups by discovering them through clustered federated learning and then training one reward model with hierarchical group-debiased aggregation.

Load-bearing premise

The load-bearing premise is that preference learning splits into a shared feature-detection part and a client-specific side-preference part, so that averaging cancels side choices while preserving the shared detector; the paper guarantees this split only in its synthetic dataset, where all groups evaluate the same response pairs.

Editorial extensions

If this is right

  • Under balanced preference groups, a single FedAvg initialization reaches 93.45% accuracy after ten local steps, surpassing a centralized per-group model (91.45%) that starts at 92.20%, so per-group initializations are not needed for fast adaptation.
  • Group imbalance is the operative failure mode: uniform FedAvg under 15/15/5/5 groups starts at 61.80% and stays at 61.75% after ten steps, because minority clients start too far from the decision boundary.
  • Debiasing the sampling restores the property: with group-debiased sampling, accuracy jumps from 54.95% to 92.35% in ten steps, and FedGD achieves this without knowing the true groups.
  • The benefit is adaptation speed rather than a better final optimum; with enough rounds and fine-tuning steps, standard FL catches up to FedGD, so the method's value is the flat initialization.
  • Exact group recovery is unnecessary: FedGD with K=2, K=3, and K=4 all stay within 91.0% to 92.8% accuracy after ten steps, as long as the discovered partition removes the sampling bias.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: if flatness is the operative property, GQ measured at the end of federated training could serve as a cheap, communication-free predictor of how much local fine-tuning will help, letting designers compare initialization strategies before deploying.
  • Editorial extension: the cancellation mechanism predicts that the advantage should shrink when users in different groups rate different response sets; a dataset with no overlapping response pairs across clients would directly test the assumed shared-feature component.
  • Editorial extension: the same shared-feature-versus-opposing-side structure may appear in other subjective judgment tasks, so FedGD's group-debiased sampling is a natural candidate for federated content moderation, style preference, or safety alignment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper studies personalized reward modeling for LLMs under federated learning with preference heterogeneity. It argues that a single FedAvg global model provides a better initialization for local personalization than per-group models, attributing the advantage to a flat loss landscape and cancellation of conflicting preference directions during aggregation. It then identifies group imbalance as the failure mode and proposes FedGD, a two-phase method that discovers latent preference groups and trains a single reward model with group-debiased client sampling. The method is evaluated on a new synthetic style-preference benchmark and on Reddit TL;DR, comparing against centralized training, FedAvg, per-group oracle models, FedBiscuit, and a soft-clustering variant. FedGD achieves the highest post-adaptation accuracy in the reported settings, although the real-world advantage over FedAvg is small and the authors note that FedAvg eventually catches up.

Significance. If confirmed, the paper's central claim would challenge the common practice of training separate initializations for each preference group and would offer a simple, communication-matched alternative: keep one shared model and debias client sampling. The paper has several strengths: the synthetic benchmark is configurable and permits controlled study of group-size imbalance; the diagnostics (GQ and Headroom) are clearly defined; the full algorithm and baselines are given in pseudocode; and the authors are transparent that the benefit of FedGD is largely faster adaptation rather than a better final optimum. However, the load-bearing evidence is weakened by the absence of multiple seeds, by a synthetic construction that mechanically enforces the proposed cancellation mechanism, and by a real-world evaluation whose headline gain is on the order of one accuracy point. The empirical contribution is substantial enough to warrant revision rather than rejection, but the central claim needs stronger support.

major comments (4)
  1. [Section 5, Table 3] All accuracy numbers in Table 3 are from a single seed; the reported standard deviations are across the 34 clients, not across random initializations. The central claim that FedGD beats FL by 1.41 points on real TL;DR at acc240 (64.41 vs 63.00) is therefore not yet distinguishable from run-to-run variation. Please provide means and standard deviations over at least five independent runs and a paired test for the FedGD-versus-FL comparison, and do the same for the synthetic tables.
  2. [Section 2.3 and Section 3.1] The synthetic benchmark is constructed so that every client in a group sees the same style-labeled response pairs, and groups differ only in which endpoint of a style axis is preferred. This guarantees that opposing gradient directions cancel at aggregation, so the 'shared knowledge vs. client-specific side choice' mechanism is built into the data rather than emergent. The real TL;DR data do not have this structure because each annotator rates its own summaries, and Table 3 shows the effect nearly vanishes there: FL improves only from 60.82 to 61.53 after 10 local steps, and Section 5.1 admits that FL catches up after sufficient personalization. The claim that a single shared initialization makes per-group models unnecessary is therefore demonstrated only under the favorable synthetic construction. I ask the authors to either measure how often the same response pair receives conflicting labels in TL;DR, evaluate on a subset of annotators with overlapping response pairs, or explicitly qualify the headline claim to settings where the cancellation mechanism holds.
  3. [Section 2.2, Headroom definition] Headroom is defined as H(x,y+,y-) = -eta <grad_theta m, g_c(theta0)> with the stated approximation H approx delta m, where delta m is the true margin change after one full-batch step. Consequently, the reported H+ fractions in Tables 1, 2, and 5 are essentially the fractions of samples whose margin improves after a single step; they are not independent predictors of whether a few local steps will succeed. To support the mechanistic attribution, please report the actual margin change delta m alongside H and discuss explicitly how much information H adds beyond the one-step accuracy improvement.
  4. [Section 5.1] The paper acknowledges that under the balanced synthetic setting FL at 400 rounds reaches acc80=94.70, slightly above FedGD's 94.30, and that the benefit of FedGD lies in faster adaptation rather than a better final optimum. This is consistent with the abstract's claim being about the speed of adaptation, but the abstract's phrase 'surpasses ... after only a few local optimization steps' should be qualified to the balanced setting and to the synthetic scenario. Please make the scope of the headline claim explicit in the abstract and conclusion.
minor comments (4)
  1. [Tables 1 and 3] Table 1 reports results from 200 global rounds (Section 3) while Table 3 reports 400 rounds, and the same named quantity acc10 for Multi-CL differs between them (91.45 in Table 1 vs. 92.15 in Table 3). Please state the global-round budget in each table caption and reconcile the numbers.
  2. [Section 2.2] The paper defines the retention-set headroom H0 in Section 2.2 but never reports H0 in any table. Either use H0 in the diagnostics or remove the definition to avoid confusion.
  3. [Section 4.2 and Appendix C] The mixing coefficient w=0.6 and group-debiased sample size |G|=5 are free parameters, but no sensitivity analysis for w appears in the main text or appendix. Please add a sentence or a small table showing that the results are robust to w.
  4. [Section 5, Table 4] The real-world comparison across Qwen-1.5B and Gemma-2B is reported without error bars over seeds; the same single-seed concern applies here, so please provide seed-level variability or state clearly that only one seed was used.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FedAvg-vs-Multi-CL comparison is empirical, and the GQ/Headroom diagnostics are post-hoc explanations rather than fitted predictions.

full rationale

The paper's central claim—that a single FedAvg initialization outperforms per-group centralized models after a few local steps—is supported by direct accuracy measurements in Tables 1-4, which do not depend on the GQ or Headroom diagnostics. GQ is adopted from external prior work [7], and Headroom, although defined as the first-order approximation of the one-step margin change (so that positive-Headroom fractions are close to restatements of one-step margin improvement), is used only to explain the independently measured acc10 values, not to generate them. The synthetic benchmark's shared-response-pair construction makes the cancellation mechanism mechanical, but that is a limitation of external validity rather than a circular derivation: the paper does not fit a parameter and then rename it as a prediction. Self-citations (FedBABU, BOIL, FedDr+, FedFN) are used only as precedents for the Body/Head layer split or appear without load-bearing role, and no uniqueness theorem is imported from the authors' own work. The empirical comparisons, including real-world TL;DR results, stand independently of the diagnostics, so no load-bearing step reduces to its own inputs by construction.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper's central claims rest on a set of domain assumptions about preference structure and on hand-chosen hyperparameters, but no fitted constants that are then relabeled as predictions. The synthetic benchmark is constructed to embody the assumed decomposition, so it does not independently test that assumption.

free parameters (3)
  • Group-debiased sample size |G| = 5 (one per group plus one random) = 5
    The number of clients sampled per round and the allocation rule are chosen by hand (Section 3.2, Section 5). No sensitivity analysis is provided for this choice, and it directly affects the debiasing strength.
  • Mixing coefficient w in Phase 1 expert updates = 0.6
    Used in FedGD expert moving average (Algorithm 1, line 17). Chosen without ablation; affects how quickly experts track assigned clients.
  • Number of groups K (default 4) = 4
    Default number of experts/groups. The paper includes a robustness study over K=2,3,4, so this is not load-bearing, but it is still a hand-set hyperparameter.
assumptions (5)
  • domain assumption Preference learning decomposes into a shared feature-detection component and a client-specific side-preference component.
    Stated as 'Our key hypothesis' in Section 1; underpins the flatness/cancellation explanation and the synthetic benchmark design.
  • domain assumption Clients can be partitioned into groups with internally homogeneous preferences, and validation loss on a client's validation split is a reliable signal for group membership.
    Used in FedGD Phase 1 reassignment (Section 4.1, Algorithm 1). Requires validation labels with preferences, which may not exist in all federated settings.
  • standard math The loss landscape around the initialization is smooth enough that the first-order Taylor approximation of the Gradient Quotient is valid.
    Used in Section 2.2 to derive the GQ approximation; assumes the Hessian-gradient product dominates higher-order terms.
  • domain assumption LoRA adapters on a frozen base model are sufficient to represent personalized reward functions.
    Used throughout; no proof that LoRA rank 8 captures per-client preferences, though results suggest it works for tested settings.
  • ad hoc to paper The synthetic dataset construction guarantees that response pairs differ along two style axes, so the shared feature is identifiable and opposing preferences can cancel exactly.
    Section 2.3 describes generating two candidate responses per prompt along style axes. This guarantees the mechanism the paper explains, so it is not an independent test of the mechanism.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Personalized Reward Modeling for LLMs under Preference Heterogeneity via Group-Debiased Federated Learning." pith.science (2026). https://pith.science/paper/DJERVDEO

@misc{pith2026260801556,
  author       = {Pith},
  title        = {Pith review of: Rethinking Personalized Reward Modeling for LLMs under Preference Heterogeneity via Group-Debiased Federated Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DJERVDEO}},
  note         = {Machine review of arXiv:2608.01556}
}
read the original abstract

Large language models are increasingly aligned to human preferences via reward modeling, but user preference data are sensitive and often cannot be centralized. Federated learning keeps such data local while learning a shared initial reward model, which is later personalized for each client through local fine-tuning. Because users often assign opposite labels to the same pair of responses, existing federated methods address preference heterogeneity by clustering similar clients and training one reward model per group, assuming that each group requires its own initialization. We show that this assumption is unnecessary. Under balanced preference groups, a single FedAvg model, despite starting at nearly random accuracy, surpasses reward models trained separately for each ground-truth group after only a few local optimization steps. We attribute this phenomenon to the flatness of the shared initialization: averaging across all clients learns richer shared representations that distinguish responses while canceling conflicting preference directions, leaving the model near a decision boundary that can be rapidly adapted. Group imbalance breaks this effect as the cancellation becomes asymmetric and leaves minority clients too far from the boundary to recover. Motivated by this observation, we propose FedGD (Federated Learning with Group Debiasing), which discovers latent preference groups during federated training and learns a single reward model using group-debiased client sampling. By counteracting the effect of group imbalance, FedGD learns an initialization that remains highly adaptable, enabling effective personalization without prior knowledge of the underlying groups.

Figures

Figures reproduced from arXiv: 2608.01556 by the authors.

Figure 1
Figure 1. Overview of FedGD. Phase 1 discovers the preference groups, which can differ in size. Phase 2 trains a single reward model over them with group-debiased client sampling, which compensates for group-size imbalance during client selection. The resulting model serves as a shared initialization for personalization. Phase 3 personalizes the initial model for each client through a few local optimization steps, yielding Fe… view at source ↗
Figure 2
Figure 2. Group-wise personalized accuracy under the bal [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Group-wise PFL accuracy under the imbalanced [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Phase 1 clustering results under the synthetic im [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Sensitivity to the local optimization budget and learning rate on the balanced synthetic dataset. We compare FL and [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Evolution of client-to-expert assignments during Phase 1 on the synthetic dataset. Each marker denotes the expert [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 37 canonical work pages

  1. [1]

    Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. 2021. A General Language Assistant as a ...

    arXiv 2021

  2. [2]

    Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, K...

    arXiv 2022

  3. [3]

    Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomek Korbak, David Lindner, Pedro Freire, Tony Tong Wang, Samuel Marks, Charbel-Raphael Segerie, Micah Carroll, Andi Peng, Phillip J.K. Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J Michau...

    work page 2023

  4. [4]

    Wenlin Chen, Samuel Horváth, and Peter Richtárik. 2022. Optimal Client Sam- pling for Federated Learning.Transactions on Machine Learning Research(2022). https://openreview.net/forum?id=8GvRCWKHIL

    work page 2022

  5. [5]

    Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. 2021. Exploiting Shared Representations for Personalized Federated Learning. InPro- ceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 2089–2099. https://proceedings.mlr.press/v1...

    work page 2021

  6. [6]

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun

    work page

  7. [7]

    Yann N Dauphin and Samuel Schoenholz. 2019. MetaInit: Initializing learning by learning to initialize. InAdvances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc. https://proceedings.neurips.cc/paper_ files/paper/2019/file/876e8108f87eb61877...

    work page 2019

  8. [8]

    A. P. Dempster, N. M. Laird, and D. B. Rubin. 1977. Maximum likelihood from incomplete data via the EM algorithm.Journal of the Royal Statistical Society: Series B39 (1977), 1–38. http://web.mit.edu/6.435/www/Dempster77.pdf

    work page 1977

Show all 59 references
  1. [9]

    Dinh, Nguyen H

    Canh T. Dinh, Nguyen H. Tran, and Tuan Dung Nguyen. 2020. Personalized federated learning with moreau envelopes. InProceedings of the 34th International Conference on Neural Information Processing Systems(Vancouver, BC, Canada) (NIPS ’20). Curran Associates Inc., Red Hook, NY,...

    2020

  2. [10]

    Xin Dong, Sai Qian Zhang, Ang Li, and H.T. Kung. 2022. SphereFed: Hyper- spherical Federated Learning. InComputer Vision – ECCV 2022: 17th European Conference, Tel A viv, Israel, October 23–27, 2022, Proceedings, Part XXVI(Tel Aviv, Israel). Springer-Verlag, Berlin, Heidelberg...

    2022 doi

  3. [11]

    Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. 2020. Personalized fed- erated learning with theoretical guarantees: a model-agnostic meta-learning approach. InProceedings of the 34th International Conference on Neural Informa- tion Processing Systems(Vancouver, BC, Canad...

    2020

  4. [12]

    Flint Xiaofeng Fan, Cheston Tan, Yew-Soon Ong, Roger Wattenhofer, and Wei- Tsang Ooi. 2025. FedRLHF: A Convergence-Guaranteed Federated Framework for Privacy-Preserving and Personalized RLHF. InProceedings of the 24th International Conference on Autonomous Agents and Multiagen...

    2025

  5. [13]

    Yann Fraboni, Richard Vidal, Laetitia Kameni, and Marco Lorenzi. 2021. Clustered Sampling: Low-Variance and Improved Representativity for Clients Selection in Federated Learning. InProceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Lea...

    2021

  6. [14]

    Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. 2020. An efficient framework for clustered federated learning. InProceedings of the 34th International Conference on Neural Information Processing Systems(Vancouver, BC, Canada)(NIPS ’20). Curran Associates Inc., R...

    2020

  7. [15]

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-Efficient Transfer Learning for NLP. InProceedings of the 36th Interna- tional Conference on Machine Learning (Procee...

    2019

  8. [16]

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. InInternational Conference on Learning Representations. https: //openreview.net/forum?id=nZeVKeeFYf9

    2022

  9. [17]

    Erin Illman and Paul Temple. 2019. California consumer privacy act.The Business Lawyer75, 1 (2019), 1637–1646

    2019

  10. [18]

    Jacobs, Michael I

    Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton

  11. [19]

    Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. 2024. Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging. InAdaptive Foundation Model...

    2024

  12. [20]

    Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Ben- nis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. 2021. Advances and open problems in federated learning. Foundations and trends®in machine learning...

    2021

  13. [21]

    Seongyoon Kim, Minchan Jeong, Sungnyun Kim, Sungwoo Cho, Sumyeong Ahn, and Se-Young Yun. 2025. FedDr+: Stabilizing Dot-regression with Global Feature Distillation for Federated Learning.Transactions on Machine Learning Research (2025). https://openreview.net/forum?id=a6WthNFhL2

    2025

  14. [22]

    Seongyoon Kim, Gihun Lee, Jaehoon Oh, and Se-Young Yun. 2023. FedFN: Feature Normalization for Alleviating Data Heterogeneity Problem in Federated Learning. InInternational Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023. https:/...

    2023

  15. [23]

    Andreas Köpf, Yannic Kilcher, Dimitri Von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, et al. 2023. Openassistant conversations-democratizing large language model alignment.Advances in neural information...

    2023

  16. [24]

    Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. 2021. Ditto: Fair and Robust Federated Learning Through Personalization. InProceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong...

    2021

  17. [25]

    Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. 2020. Federated Optimization in Heterogeneous Networks. In Proceedings of Machine Learning and Systems, I. Dhillon, D. Papailiopoulos, and V. Sze (Eds.), Vol. 2. 429–450. https://proc...

    2020

  18. [26]

    Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. 2020. On the Convergence of FedAvg on Non-IID Data. InInternational Conference on Learning Representations. https://openreview.net/forum?id=HJxNAnVtDS

    2020

  19. [27]

    Guodong Long, Ming Xie, Tao Shen, Tianyi Zhou, Xianzhi Wang, and Jing Jiang

  20. [28]

    Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. InInternational Conference on Learning Representations. https://openreview.net/ 9 forum?id=Bkg6RiCqY7

    2019

  21. [29]

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep net- works from decentralized data. InArtificial intelligence and statistics. PMLR, 1273–1282

    2017

  22. [30]

    Jaehoon Oh, SangMook Kim, and Se-Young Yun. 2022. FedBABU: Toward Enhanced Representation for Federated Image Classification. InInternational Conference on Learning Representations. https://openreview.net/forum?id= HuaYQfggn5u

    2022

  23. [31]

    Jaehoon Oh, Hyungjun Yoo, ChangHwan Kim, and Se-Young Yun. 2021. {BOIL}: Towards Representation Change for Few-shot Learning. InInterna- tional Conference on Learning Representations. https://openreview.net/forum?id= umIdUL8rMH

    2021

  24. [32]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback.Advances in neural information processing systems35 (...

    2022

  25. [33]

    Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. 2024. Personalizing reinforcement learning from human feedback with variational preference learning.Advances in Neural Information Processing Sys- tems37 (2024), 52516–52544

    2024

  26. [34]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model.Advances in neural information processing systems36 (2023), 53728–53741

    2023

  27. [35]

    Shyam Sundhar Ramesh, Yifan Hu, Iason Chaimalas, Viraj Mehta, Pier Giuseppe Sessa, Haitham Bou Ammar, and Ilija Bogunovic. 2024. Group robust preference optimization in reward-free rlhf.Advances in Neural Information Processing Systems37 (2024), 37100–37137

    2024

  28. [36]

    Protection Regulation. 2016. Regulation (EU) 2016/679 of the European Parliament and of the Council.Regulation (eu)679, 2016 (2016), 10–13

    2016

  29. [37]

    Yichen Ruan and Carlee Joe-Wong. 2022. Fedsoft: Soft clustered federated learning with proximal local updating. InProceedings of the AAAI conference on artificial intelligence, Vol. 36. 8124–8131

    2022

  30. [38]

    Felix Sattler, Klaus-Robert Müller, and Wojciech Samek. 2021. Clustered Federated Learning: Model-Agnostic Distributed Multitask Optimization Under Privacy Constraints.IEEE Transactions on Neural Networks and Learning Systems32, 8 (2021), 3710–3722. https://doi.org/10.1109/TNN...

    2021

  31. [39]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

  32. [40]

    Jonathan Scott, Hossein Zakerinia, and Christoph H Lampert. 2024. PeFLL: Personalized Federated Learning by Learning to Learn. InThe Twelfth Interna- tional Conference on Learning Representations. https://openreview.net/forum?id= MrYiwlDRQO

    2024

  33. [41]

    Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. InInternational Conference on Learning Representations. https://openreview.net/foru...

    2017

  34. [42]

    Jingyan Shen, Jiarui Yao, Rui Yang, Yifan Sun, Feng Luo, Rui Pan, Tong Zhang, and Han Zhao. 2025. MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learning. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C...

    2025

  35. [43]

    Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano

    Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020. Learning to sum- marize from human feedback. InProceedings of the 34th International Conference on Neural Information Processing Systems(Van...

    2020

  36. [44]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupati- raju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juli- ette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, ...

    2024 arXiv

  37. [45]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. InProceedings of the 31st International Conference on Neural Information Processing Systems(Long Beach, California, US...

    2017

  38. [46]

    Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. 2017. TL;DR: Mining Reddit to Learn Automatic Summarization. InProceedings of the Workshop on New Frontiers in Summarization, Lu Wang, Jackie Chi Kit Cheung, Giuseppe Carenini, and Fei Liu (Eds.). Association for ...

    2017 doi

  39. [47]

    Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. 2024. Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards. InProceedings of the 62nd Annual Meeting of the Associ...

    2024 doi

  40. [48]

    Lei Wang, Jieming Bian, Letian Zhang, and Jie Xu. 2025. Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems. https://openreview.net/ forum?id=es4TTVGJ9x

    2025

  41. [49]

    Feijie Wu, Xiaoze Liu, Haoyu Wang, Xingchen Wang, Lu Su, and Jing Gao

  42. [50]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Cheng- peng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jin...

    2024 arXiv

  43. [51]

    Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, and Siheng Chen. 2024. OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery...

    2024

  44. [52]

    Liping Yi, Han Yu, Gang Wang, Xiaoguang Liu, and Qinghua Hu. 2026. pFedMoE: Data-Level Personalization With Mixture of Experts in Model-Heterogeneous Personalized Federated Learning.IEEE Transactions on Knowledge and Data Engineering38, 3 (2026), 1905–1918. https://doi.org/10....

    2026

  45. [53]

    Liping Yi, Han Yu, Gang Wang, Xiaoguang Liu, and Xiaoxiao Li. 2024. pFed- LoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuning. arXiv:2310.13283 [cs.LG] https://arxiv.org/abs/2310.13283

    2024 arXiv

  46. [54]

    Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2023. RRHF: Rank Responses to Align Language Models with Human Feedback. InThirty-seventh Conference on Neural Information Processing Systems. https://openreview.net/forum?id=EdIGMCHk4l 10 - Appendi...

    2023

  47. [1991]

    https://doi.org/10.1162/neco.1991.3.1.79

    Adaptive Mixtures of Local Experts.Neural Computation3, 1 (1991), 79–87. https://doi.org/10.1162/neco.1991.3.1.79

    1991 doi

  48. [2017]

    arXiv:1707.06347 [cs.LG] https://arxiv.org/abs/1707.06347

    Proximal Policy Optimization Algorithms. arXiv:1707.06347 [cs.LG] https://arxiv.org/abs/1707.06347

    arXiv

  49. [2023]

    World Wide Web26, 1 (2023), 481–500

    Multi-center federated learning: clients clustering for better personalization. World Wide Web26, 1 (2023), 481–500

    2023

  50. [2024]

    InProceedings of the 41st International Conference on Machine Learning(Vienna, Austria)(ICML’24)

    ULTRAFEEDBACK: boosting language models with scaled AI feedback. InProceedings of the 41st International Conference on Machine Learning(Vienna, Austria)(ICML’24). JMLR.org, Article 384, 23 pages

  51. [2025]

    InThe Thirteenth International Conference on Learning Representations

    Towards Federated RLHF with Aggregated Client Preference for LLMs. InThe Thirteenth International Conference on Learning Representations. https: //openreview.net/forum?id=mqNKiEB6pd

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.