Pith. sign in

REVIEW 5 major objections 5 minor 79 references

Multi-objective Large Language Model Alignment with Hierarchical Experts

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a single frozen-base LLM with hierarchical LoRA and router experts can be steered to any point on the Pareto frontier across up to five objectives, outperforming 15 baselines.

desk verdict A plausible hierarchical MoE for preference steering, but the routing equations don't parse and the 'training-free' claim is overstated; worth reviewing after major revision. read the letter →

arxiv 2505.20925 v1 pith:TWLO4CJQ submitted 2025-05-27 cs.CL cs.AI

Zhuo Li , Guodong Du , Weiyang Guo , Yigeng Zhou , Xiucheng Li , Wenya Wang , Fangming Liu , Yequan Wang
show 3 more authors
Deheng Ye Min Zhang Jing Li
This is my paper
classification cs.CLcs.AI
keywords multi-objectivealignmentlargelanguagemodelsmixtureofexpertsLoRAParetofrontierpreferenceroutingmodelmergingTchebycheffscalarization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that multi-objective alignment does not require retraining or storing many full models: one base LLM with frozen weights can be steered to any requested preference weighting by plugging in hierarchical experts. HoE builds a three-level stack, with LoRA experts extracted from off-the-shelf single-objective models, merged multi-objective LoRA experts, and small trained routers, so that a user's preference vector selects a tailored combination of adapters during inference. If the claim holds, a single deployment can serve arbitrary mixes of objectives such as helpfulness, harmlessness, humor, and math accuracy, and new objectives can be added without retraining existing components. The paper reports results across 14 objectives, 200 preferences, and 6 benchmarks, with Pareto frontiers that dominate 15 recent baselines.

What carries the argument

The load-bearing object is the three-level expert hierarchy: LoRA experts obtained by task-SVD, merged multi-objective LoRA experts, and one-layer router experts, all coordinated by a parameter-free preference-routing layer. The routing layer selects the $N$ expert weightings nearest to the user's $\lambda$ by Euclidean distance, the selected routers produce input-dependent logits weighted by the resulting convex combination, and the final output is the base weight plus a mixture of LoRA adapters. Router experts are trained with Tchebycheff scalarization and online mirror descent inside a PPO loop, which the paper argues keeps non-convex regions of the Pareto frontier reachable and comes with an $O(\log N/T)$ convergence rate.

What would settle it

Run a human-preference evaluation on the same benchmarks and preference grid, comparing HoE with top baselines at intermediate weightings such as $[1/3,1/3,1/3]$; if human judges do not prefer HoE's responses at least as often as the baselines despite HoE's higher reward-model scores, the central claim that HoE accommodates diverse user preferences is not supported.

Watch

Extended reading notes

Core claim

The central claim is that the Pareto frontier of a multi-objective aligned LLM can be assembled point-by-point from specialized, preference-local experts instead of being approximated by one monolithic policy. HoE first converts each off-the-shelf single-objective model into a compact LoRA expert via task-vector singular value decomposition; it then synthesizes additional multi-objective LoRA experts by merging task vectors with preference weights; finally it trains negligible one-layer router experts, each tied to a preference weighting, to select and combine the nearest LoRA experts based on hidden states. A parameter-free preference-routing layer maps any user vector to the closest expert weightings and expresses it as a convex combination, and the paper reports that this hierarchy yields Pareto frontiers that dominate 15 baselines while training only 8M parameters.

Load-bearing premise

The load-bearing premise is that the reward models used to train router experts and to draw the Pareto frontiers faithfully represent the human preferences HoE claims to accommodate; if those reward models are biased or gameable, the reported dominance does not establish preference alignment.

Editorial extensions

If this is right

  • A single frozen-base deployment can be aligned to any preference weighting without retraining; serving different users only changes the routing weights.
  • Adding a new objective requires only a new off-the-shelf single-objective model and its LoRA expert, while existing experts and routers remain valid.
  • The 8M trainable parameters and single-pass inference make the approach practical where per-preference retraining or multi-pass decoding is too expensive.
  • The same hierarchy can blend task behaviors in multi-task learning, not only preference trade-offs within one task.
  • Dominating reward-model Pareto frontiers with fewer stored parameters could shift practical alignment from training many models to managing a small expert library.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because router experts are trained against reward-model feedback, how well HoE satisfies actual human preferences is only as strong as those reward models; a human-judgment study on the same preference grid would test that transfer.
  • Editorial inference: the routing layer is agnostic to what the objectives are, so the same hierarchical decomposition could apply to continual or multi-task learning where new LoRA experts are added without retraining old ones.
  • Editorial inference: if the $O(\log N/T)$ rate holds, the cost of training routers grows only logarithmically with the number of objectives, so at large $N$ the practical bottleneck may shift from training to storing the growing LoRA expert library.
  • Editorial inference: router activation patterns could serve as a diagnostic for objective interference; a router that ignores an expert even when the user's weighting favors it would indicate the merge or SVD step lost that capability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes HoE (Hierarchical Mixture-of-Experts), a multi-objective alignment framework built from three components: LoRA experts extracted from off-the-shelf single-objective models via task-SVD, multi-objective LoRA experts synthesized by model merging, and trained router experts selected by a preference-routing module. The authors claim that HoE enables a single frozen-base LLM to steer along the entire Pareto frontier for arbitrary user preference vectors, without training the base model, and they report experiments on 14 objectives, 200 preferences, 6 benchmarks, and 15 baselines, including GPT-4 win rates and a cost analysis. The central claims are that HoE consistently outperforms all baselines, achieves smooth and convex Pareto frontiers, and enjoys a theoretical convergence rate of O(log N / T).

Significance. If the method works as described, it is a meaningful step toward practical multi-objective alignment: a single architecture with frozen base weights, low-rank adapters, and small routers could traverse the Pareto frontier at inference time, with storage savings relative to per-preference fine-tuned models. The empirical breadth is substantial, and the inclusion of GPT-4 win rates, a cost comparison (Table 3), and ablations (Fig. 6) is a genuine strength. However, the significance is conditional: the routing equations in Section 3.4 are not well defined as written, the theoretical guarantee in Appendix F is not actually proven for the claimed objective, and the training-free claim in the abstract conflicts with the trained router experts. The reported dominance over baselines also depends partly on an evaluation loop that uses the same reward models for router training and Pareto evaluation, so independent validation is needed for the preference-accommodation claim.

major comments (5)
  1. [Section 3.4, Eqs. (10)–(12); Appendix D.2] The preference-to-expert map is not well defined as written. Eq. (10) expresses λ_user as a convex combination of the N nearest expert weightings selected from Λ = {λ_i}_{i∈[L+R]}, yielding a vector ω_r whose dimension is L+R (or a selection over the N chosen indices). Eq. (11) then multiplies ω_r by [r_{λ_i}(x)]_{i∈[R]}, which has length R, so the dimensions are inconsistent. In the two-objective setup of Appendix D.2 (L=3, R=1), take λ_user = [1,0]; the nearest set includes the [1,0] LoRA expert and one router expert, and Eq. (10) gives ω_r with all mass on the LoRA expert. Eq. (11) then assigns zero weight to the only router expert, so no LoRA expert is activated and the output degenerates to the base model. If the intended computation is instead some other aggregation, such as activating the selected LoRA experts directly and using router logits only to modulate among them, then Eqs. (10)–(12) must be rewritten and the precise algorithm used to produce Figs. 3 and 6 must be described. As it stands, the central mechanism of the paper is undefined.
  2. [Section 3.2, Eq. (5); Section 4; Figs. 3–4] The evaluation loop is partly circular. Eq. (5) trains each router expert to maximize R_λ = Σ_i λ_i R_i under the mixture policy, and the Pareto frontiers in Figs. 3–4 are then computed with those same reward models R_i. Under these conditions, HoE's reported dominance may partly reflect fitting the evaluation metric rather than satisfying the human preferences the paper claims to accommodate. The paper does include independent evidence, namely GPT-4 win rates in Fig. 9 and accuracy/over-refusal metrics, but these cover only a subset of the 14 objectives. Please report GPT-4 or other independent evaluations for the two-objective frontiers where the main dominance claims are made, and state explicitly whether any of the evaluation reward checkpoints were used during router training. If they were, a hold-out reward model or a human study is needed to support the preference-accommodation claim.
  3. [Appendix F, Eqs. (21)–(22) and Theorem F.2] The claimed convergence rate is not established in the manuscript. Theorem F.2 states a rate of O(log N / T) 'as proven in [35]', but the preceding derivation defines L(θ|λ) as a linear weighted sum of expected rewards (Eq. 21), not as the Tchebycheff scalarization of Eq. (6); the max-min structure introduced in Eq. (14) is abandoned in the proof. Moreover, Assumption 1 (each f_i(θ) is convex in θ) is asserted without justification for LLM policy parameters, and no argument is given that the feasible objective space is convex. If the convergence claim is to be kept, the authors need to prove the rate for the actual TCH objective used in Eqs. (6)–(8), or explicitly state that the rate applies to a different algorithm and remove the claim from Section 3.2.
  4. [Abstract; Section 1; Section 3.2; Table 3] The abstract and introduction overstate the training-free property. The abstract says HoE 'eliminates the need for model training', and Section 1 repeats 'without training' for the whole approach; however, Section 3.2 optimizes router experts via PPO and Table 3 reports 8M trainable parameters for HoE. This is not just a wording issue: readers cannot tell whether the reported results require the trained routers and, if so, what data and compute were used. Please revise the abstract to state that HoE requires no training of the base model or the LoRA experts, and specify the router training budget (data, steps, hardware) in Section 6.2 or Appendix D.
  5. [Appendix D.1–D.3] Several hyperparameters essential for reproducing the results are missing or appear to be selected on the evaluation signal. The reference point z* in Eq. (14), the step size α in Eq. (18), the number of OMD/TD iterations, and the number and placement of additional LoRA and router experts per task are not reported; Appendix D.2 says the router expert is 'adaptively added based on the evaluation result', which suggests a selection procedure that could leak the test signal. Please publish a full hyperparameter table and specify how each expert count was chosen, ideally on a validation split. Without this information, the empirical results in Figs. 3–5 cannot be reproduced or independently verified.
minor comments (5)
  1. [Throughout] There are several typos and inconsistencies: 'off-the-shell' in Section 3.1, 'Specificaly' in Section 1, 'Mullti-Task' in Appendix B.1, and inconsistent spellings of 'Tchebycheff' versus 'Tschebyscheff' across Sections 3.2 and 6.1.
  2. [Sections 3.3–3.4] The symbol N is used both for the number of objectives and for the number of nearest experts selected in Eqs. (9)–(12); this conflation makes the routing equations harder to parse. Please use a distinct symbol such as K for the number of selected experts.
  3. [Appendix E.5; Abstract] The abstract claims 200 different preferences, but Appendix E.5 reports 11 two-objective points per setup, 13 three-objective points per setup, and a small number of five-objective points; the total is not obviously 200. Please clarify how the 200 preferences are counted across the six benchmarks.
  4. [Appendix F] Equation numbering in Appendix F conflicts with the main text: Eq. (14) in Appendix D.3 duplicates the main-text Eq. (6), and the proof in Appendix F reuses equations (20)–(28) without referencing the main text, making cross-references confusing.
  5. [Figure 3] The two-objective Pareto plots in Figure 3 have small fonts and overlapping curves; the claims of dominance would be easier to verify if the figures showed the preference weightings explicitly and used distinct markers for each baseline.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: HoE is built from constructive, externally grounded components; the shared-reward evaluation loop is a benchmark convention rather than a derived prediction, and the main flagged flaw (Eqs. 10-12) is an internal inconsistency, not a circular reduction.

full rationale

HoE's derivation chain does not reduce to its own inputs. LoRA experts are extracted from off-the-shelf single-objective models via task vectors and task-SVD (Sec. 3.1, Eqs. 2-4), with quality checked against external metrics such as GSM8K Pass@1, OR-Bench safety/helpfulness, and GPT-4 win rates (Sec. 5, Figs. 7 and 9, Tab. 4). The router experts are trained with Eq. 5 to maximize R_lambda, and Sec. 4 evaluates Pareto frontiers with the same RMs; this is a closed evaluation loop, but it is not a derivation-level circularity because every baseline is measured on the same RMs and held-out prompts (Psoups/HelpSteer2, Sec. 5.2), so the reported dominance is not forced by construction. The largest textual problem is Section 3.4: Eq. 10 constructs omega_r over the L+R experts, but Eq. 11 multiplies only the R router experts, and with the paper's own 2-objective setup (App. D.2, L=3, R=1) a one-hot preference can zero the router weight and collapse the output to the base model; this makes the literal derivation undefined, but it is a correctness/consistency bug, not a circularity. Appendix F's claimed O(log N/T) convergence is delegated to [35] rather than proved in the paper, an omitted-support issue. Many self-citations appear (e.g., [9,10,15,29,36,37,50,65,66,70]), but they are background or baseline references, not the sole justification for HoE's central mechanism, so none is load-bearing. Section 7 candidly lists limitations (dependence on off-the-shelf models and on model merging/SVD), further reducing any sense that the approach is definitionally guaranteed. Overall, no predicted quantity is equivalent to a fitted parameter or to a self-citation by construction.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The three hierarchical components are assembled from known modules; no new physical or conceptual entities are introduced. The load-bearing additions are design choices and hyperparameters, which are listed above.

free parameters (6)
  • task-SVD pruning ratio = 40%
    Appendix D.1 prunes 40% of least significant parameters before SVD; no ablation is shown for this ratio.
  • LoRA rank = 128 (r in Eq 2)
    Appendix D.1 selects the top-128 rank-1 matrices; the rank ablation in Fig 6 shows lower ranks degrade performance.
  • rescaling factor for Math LoRA expert = 1.9
    Appendix D.1 states the factor is 'dynamically determined based on test results' to improve GSM8K accuracy.
  • number and placement of additional LoRA and router experts per task = e.g., 3 LoRA + 1 router (2-obj), 4 LoRA + 1 router (3-obj), 6 LoRA + 1 router (5-obj)
    Appendix D.2 chooses these 'adaptively... based on the evaluation result', a post-hoc configuration on the test setup.
  • reference point z* in Tchebycheff scalarization = not specified
    Eq 6 and Eq 14 require a reference point z*; the paper does not state how it is chosen.
  • STCH softmax and TD learning hyperparameters = not specified
    Eq 17-18 involve a softmax and a learning rate alpha; values are not reported.
assumptions (6)
  • domain assumption Off-the-shelf single-objective models exist for every objective and their task vectors capture that objective's behavior
    Assumed throughout Section 3.1; the Limitation section acknowledges dependence on such models.
  • domain assumption Reward models used to train routers faithfully reflect human preferences
    Section 3.2 trains routers on R_lambda built from reward models, and Section 4 evaluates with the same reward models.
  • domain assumption Task-SVD with 40% pruning and rank 128 preserves single-objective policy performance
    Section 3.1 and Appendix D.1 rely on this compression; only the Math expert is validated (GSM8K accuracy).
  • domain assumption Model merging of task vectors yields valid multi-objective experts for arbitrary preference lambda (Eq 4)
    Section 3.1 uses an unspecified Merge function to synthesize intermediate experts; the merge algorithm is not detailed.
  • ad hoc to paper Each objective gap f_i(theta) is convex in theta (Assumption 1, Appendix F)
    Used to claim strict convergence and a unique saddle point; this is false for neural network policies.
  • standard math Boundedness, Lipschitz, and bounded-variance conditions (Assumptions 2-6, Appendix F)
    Standard regularity assumptions for stochastic approximation; stated but not verified for the actual policy class.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-objective Large Language Model Alignment with Hierarchical Experts." pith.science (2026). https://pith.science/paper/TWLO4CJQ

@misc{pith2026250520925,
  author       = {Pith},
  title        = {Pith review of: Multi-objective Large Language Model Alignment with Hierarchical Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TWLO4CJQ}},
  note         = {Machine review of arXiv:2505.20925}
}
read the original abstract

Aligning large language models (LLMs) to simultaneously satisfy multiple objectives remains a significant challenge, especially given the diverse and often conflicting nature of human preferences. Existing alignment methods struggle to balance trade-offs effectively, often requiring costly retraining or yielding suboptimal results across the Pareto frontier of preferences. In this paper, we introduce \textit{HoE}(Hierarchical Mixture-of-Experts), a \textit{lightweight}, \textit{parameter-efficient}, and \textit{plug-and-play} approach that eliminates the need for model training, while enabling LLMs to adapt across the entire Pareto frontier and accommodate diverse user preferences. In particular, \textit{HoE} consists of three hierarchical components: LoRA Experts, Router Experts and Preference Routing, reaching optimal Pareto frontiers and achieving a trade-off between parameter size, training cost, and performance. We evaluate \textit{HoE} across various tasks on 14 objectives and 200 different preferences among 6 benchmarks, demonstrating superior performance over 15 recent baselines. Code is available in the supplementary materials.

Figures

Figures reproduced from arXiv: 2505.20925 by the authors.

Figure 1
Figure 1. (Left) HoE decomposes the multi-objective alignment problem into a series of single-preference sub￾problems, each handled by a specialized expert. (Right) HoE employs hierarchical experts, integrating LoRA and router experts to approach the optimal Pareto frontier. One promising framework for such de￾composition strategy lies in the LoRA￾based Mixture-of-Experts (LoRAMoE) [8, 11, 64, 3] framework, originally devel￾o… view at source ↗
Figure 2
Figure 2. Illustration of our HoE approach. The left side illustrates the application scenario, where the model generates a response aligned with the prompt and given preferences. The bottom-right highlights its three hierarchical components - the LoRA experts, router experts, and a preference routing. The top-right depicts individual components, each serving as an expert for specific weightings, designed for seamless plug-an… view at source ↗
Figure 3
Figure 3. Results of two-objective alignment on HelpAssistant, Reddit Summary and BeaverTails [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Comparison of alignment results with three objectives ( [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Five-objective alignment results on HelpSteer. Prefer [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Ablation studies assessing the impact of expert count (Left), LoRA ranks (Middle), and [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Multi-Task Learning results. Our router experts specialized for “Helpful Assistant” and [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Alignment results with Helpful Assistant task on three-objective. Our approach consistently [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: GPT-4 evaluates all methods on Psoups and Helpsteer2 task, comparing the relative win [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Visualization of Case Study and Selection Ratio of Each LoRA Experts (i.e. router logits [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

79 extracted references · 53 canonical work pages

  1. [35]

    M. Liu, X. Zhang, C. Xie, K. Donahue, and H. Zhao. Online mirror descent for tchebycheff scalarization in multi-objective optimization.arXiv preprint arXiv:2410.21764, 2024

    arXiv 2024

  2. [1]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [2]

    Y . Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. E. Showk, N. Elhage, Z. Hatfield- Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. B. Brown, J. Clark, S. McCandlish, C. Olah, B. Mann, and J. Kapl...

    arXiv 2022

  4. [3]

    E. L. Buehler and M. J. Buehler. X-lora: Mixture of low-rank adapter experts, a flexible framework for large language models with applications in protein mechanics and molecular design.APL Machine Learning, 2(2), 2024

    work page 2024

  5. [4]

    B. Cao, J. Yang, X. Zhou, Z. Kheiri, F. Zahmatkesh, and X. Yang.Fuzzy Relational Mathemat- ical Programming - Linear, Nonlinear and Geometric Programming Models, volume 389 of Studies in Fuzziness and Soft Computing. Springer, 2020. ISBN 978-3-030-33784-1

    work page 2020

  6. [5]

    R. Chen, X. Zhang, M. Luo, W. Chai, and Z. Liu. PAD: personalized alignment at decoding-time. arXiv preprint arXiv:2410.04070, 2024

    arXiv 2024

  7. [6]

    Cobbe, V

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv 2021

  8. [7]

    J. Cui, W. Chiang, I. Stoica, and C. Hsieh. Or-bench: An over-refusal benchmark for large language models.arXiv preprint arXiv:2405.20947, 2024

    arXiv 2024

Show all 79 references
  1. [8]

    S. Dou, E. Zhou, Y . Liu, S. Gao, W. Shen, L. Xiong, Y . Zhou, X. Wang, Z. Xi, X. Fan, S. Pu, J. Zhu, R. Zheng, T. Gui, Q. Zhang, and X. Huang. Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin. InProceedings of the 62nd Annual Meeti...

    1932

  2. [9]

    G. DU, J. Li, H. Liu, R. Jiang, S. Yu, Y . Guo, S. K. Goh, and H.-K. Tang. Knowledge fusion by evolving weights of language models. InFindings of The 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024

    2024

  3. [10]

    G. Du, Z. Fang, J. Li, J. Li, R. Jiang, S. Yu, Y . Guo, Y . Chen, S. K. Goh, H.-K. Tang, D. He, H. Liu, and M. Zhang. Neural parameter search for slimmer fine-tuned models and better transfer. arXiv preprint arXiv:2505.18713, 2025. URLhttps://arxiv.org/abs/2505.18713

    2025 arXiv

  4. [11]

    C. Gao, K. Chen, J. Rao, B. Sun, R. Liu, D. Peng, Y . Zhang, X. Guo, J. Yang, and V . Subrahma- nian. Higher layers need more lora experts.arXiv preprint arXiv:2402.08562, 2024

    2024 arXiv

  5. [12]

    A. A. Gargiulo, D. Crisostomi, M. S. Bucarelli, S. Scardapane, F. Silvestri, and E. Rodolà. Task singular vectors: Reducing task interference in model merging.arXiv preprint arXiv:2412.00081, 2024

    2024 arXiv

  6. [13]

    H. Gu, W. Li, L. Li, Q. Zhu, M. Lee, S. Sun, W. Xue, and Y . Guo. Delta decompression for moe-based llms compression.CoRR, abs/2502.17298, 2025. 10

    2025 arXiv

  7. [14]

    W. Guo, J. Li, Y . Li, W. Wang, D. He, J. Yu, and M. Zhang. Mtsa: Multi-turn safety alignment for llms through multi-round red-teaming.arXiv preprint arXiv:2505.17147, 2025. URL https://arxiv.org/abs/2505.17147

    2025 arXiv

  8. [15]

    Guodong, J

    D. Guodong, J. Lee, J. Li, R. Jiang, Y . Guo, S. Yu, H. Liu, S. K. Goh, H.-K. Tang, D. He, et al. Parameter competition balancing for model merging. InProceedings of the Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS), 2024

    2024

  9. [16]

    N. Hansen. The CMA evolution strategy: A tutorial.arXiv preprint arXiv:1604.00772, 2016

    2016 arXiv

  10. [17]

    He, R.-Z

    S. He, R.-Z. Fan, L. Ding, L. Shen, T. Zhou, and D. Tao. Merging experts into one: Improving computational efficiency of mixture of experts. InProceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024

    2024

  11. [18]

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. InProceedings of the Tenth International Conference on Learning Representations (ICLR), 2022

    2022

  12. [19]

    Huang, S

    X. Huang, S. Li, E. Dobriban, O. Bastani, H. Hassani, and D. Ding. One-shot safety alignment for large language models via optimal dualization.arXiv preprint arXiv:2405.19544, 2024

    2024 arXiv

  13. [20]

    Ilharco, M

    G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi. Editing models with task arithmetic. InProceedings of the Eleventh International Conference on Learning Representations (ICLR), 2023

    2023

  14. [21]

    J. Jang, S. Kim, B. Y . Lin, Y . Wang, J. Hessel, L. Zettlemoyer, H. Hajishirzi, Y . Choi, and P. Ammanabrolu. Personalized soups: Personalized large language model alignment via post- hoc parameter merging.arXiv preprint arXiv:2310.11564, 2023

    arXiv

  15. [22]

    J. Ji, M. Liu, J. Dai, X. Pan, C. Zhang, C. Bian, B. Chen, R. Sun, Y . Wang, and Y . Yang. Beavertails: Towards improved safety alignment of LLM via a human-preference dataset. InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Proc...

    2023

  16. [23]

    J. Ji, B. Chen, H. Lou, D. Hong, B. Zhang, X. Pan, T. Qiu, J. Dai, and Y . Yang. Aligner: Efficient alignment by learning to correct. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS), 2024

    2024

  17. [24]

    X. Jin, X. Ren, D. Preotiuc-Pietro, and P. Cheng. Dataless knowledge fusion by merging weights of language models. InProceedings of the Eleventh International Conference on Learning Representations (ICLR), 2022

    2022

  18. [25]

    Khanov, J

    M. Khanov, J. Burapacheep, and Y . Li. ARGS: alignment as reward-guided search. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR), 2024

    2024

  19. [26]

    Konen, S

    K. Konen, S. Jentzsch, D. Diallo, P. Schütt, O. Bensch, R. E. Baff, D. Opitz, and T. Hecking. Style vectors for steering generative large language models. InProceedings of the Findings of the Association for Computational Linguistics (ACL), pages 782–802, 2024

    2024

  20. [27]

    H. J. Kushner and G. G. Yin.Stochastic Approximation and Recursive Algorithms and Applica- tions. Springer, 2003

    2003

  21. [28]

    J. Lee, Y . Wang, J. Li, and M. Zhang. Multimodal reasoning with multimodal knowledge graph. InThe 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024

    2024

  22. [29]

    J. Lee, G. DU, J. Li, S. K. Goh, W. Wang, Y . Wang, F. Liu, H.-K. Tang, S. Alharbi, D. He, and M. Zhang. Multi-modality expansion and retention for llms through parameter merging and decoupling.arXiv preprint arXiv:2505.17110, 2025. URL https://arxiv.org/abs/2505. 17110

    2025 arXiv

  23. [30]

    K. Li, T. Zhang, and R. Wang. Deep reinforcement learning for multiobjective optimization. IEEE transactions on cybernetics, 51(6):3103–3114, 2020. 11

    2020

  24. [31]

    K. Li, T. Zhang, and R. Wang. Deep reinforcement learning for multiobjective optimization. IEEE Trans. Cybern., 51(6):3103–3114, 2021

    2021

  25. [32]

    X. Li, Y . Zhou, L. Zhao, J. Li, and F. Liu. Impromptu cybercrime euphemism detection. InThe 31st International Conference on Computational Linguistics (COLING), 2025

    2025

  26. [33]

    B. Lin, W. Jiang, Y . Xu, H. Chen, and Y .-C. Chen. Parm: Multi-objective test-time alignment via preference-aware autoregressive reward model.arXiv preprint arXiv:2505.06274, 2025

    2025 arXiv

  27. [34]

    X. Lin, X. Zhang, Z. Yang, F. Liu, Z. Wang, and Q. Zhang. Smooth tchebycheff scalarization for multi-objective optimization. InProceedings of the Forty-first International Conference on Machine Learning (ICML), 2024

    2024

  28. [36]

    Y . Lu, J. Li, Y . Zhou, Y . Zhang, W. Wang, X. Li, M. Zhang, F. Liu, J. Yu, and M. Zhang. Adaptive detoxification: Safeguarding general capabilities of llms through toxicity-aware knowledge editing. InFindings of the 63rd Annual Meeting of the Association for Computational Li...

    2025

  29. [37]

    Y . Lu, Y . Zhou, J. Li, Y . Wang, X. Liu, D. He, and F. L. andMin Zhang. Knowledge editing with dynamic knowledge graphs for multi-hop question answering. InThe Thirty-Ninth AAAI Conference on Artificial Intelligence (AAAI), 2025

    2025

  30. [38]

    X. Ma, J. Li, and M. Zhang. Chain of thought with explicit evidence reasoning for few-shot relation extraction. InFindings of the Association for Computational Linguistics (EMNLP), pages 2334–2352, 2023. URL https://aclanthology.org/2023.findings-emnlp.153

    2023

  31. [39]

    M. S. Matena and C. A. Raffel. Merging models with fisher-weighted averaging. InProceedings of the Advances in Neural Information Processing Systems (NeurIPS), pages 17703–17716, 2022

    2022

  32. [40]

    Mukherjee, A

    S. Mukherjee, A. Lalitha, S. Sengupta, A. Deshmukh, and B. Kveton. Multi-objective alignment of large language models through hypervolume maximization.arXiv preprint arXiv:2412.05469, 2024

    2024 arXiv

  33. [41]

    Paternain, M

    S. Paternain, M. Calvo-Fullana, L. F. O. Chamon, and A. Ribeiro. Safe policies for reinforcement learning via primal-dual methods.IEEE Trans. Autom. Control., 68(3):1321–1336, 2023

    2023

  34. [42]

    B. Ping, S. Wang, H. Wang, X. Han, Y . Xu, Y . Yan, Y . Chen, B. Chang, Z. Liu, and M. Sun. Delta-come: Training-free delta-compression with mixed-precision for large language models. InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Informati...

    2024

  35. [43]

    S. Qiu, D. Zhang, R. Yang, B. Lyu, and T. Zhang. Traversing pareto optimal policies: Provably efficient multi-objective reinforcement learning.arXiv preprint arXiv:2407.17466, 2024

    2024 arXiv

  36. [44]

    A. Ramé, G. Couairon, C. Dancette, J. Gaya, M. Shukor, L. Soulier, and M. Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. InProceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2023

    2023

  37. [45]

    Rimsky, N

    N. Rimsky, N. Gabrieli, J. Schulz, M. Tong, E. Hubinger, and A. M. Turner. Steering llama 2 via contrastive activation addition. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 15504–15522, 2024

    2024

  38. [46]

    Robbins, Herbert, and S. Monro. A stochastic approximation method.The annals of mathemati- cal statistics, pages 400–407, 2023

    2023

  39. [47]

    S. Ryu, S. Seo, and J. Yoo. Efficient storage of fine-tuned models via low-rank approximation of weight residuals.CoRR, abs/2305.18425, 2023. 12

    2023 arXiv

  40. [48]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  41. [49]

    R. Shi, Y . Chen, Y . Hu, A. Liu, H. Hajishirzi, N. A. Smith, and S. S. Du. Decoding-time language model alignment with multiple objectives.arXiv preprint arXiv:2406.18853, 2024

    2024 arXiv

  42. [50]

    Z. Shi, Y . Zhou, J. Li, Y . Jin, Y . LI, D. He, F. Liu, S. Alharbi, J. Yu, and M. Zhang. Safety alignment via constrained knowledge unlearning.arXiv preprint arXiv:2505.18588, 2025. URL https://arxiv.org/abs/2505.18588

    2025 arXiv

  43. [51]

    Stiennon, L

    N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. V oss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize from human feedback.arXiv preprint arXiv:2009.01325, 2020

    2009 arXiv

  44. [52]

    H. Sun, A. Hüyük, and M. van der Schaar. Query-dependent prompt evaluation and optimization with offline inverse rl. InProceedings of the Twelfth International Conference on Learning Representations (ICLR), 2023

    2023

  45. [53]

    Touvron, L

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, and et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  46. [54]

    Vamplew, R

    P. Vamplew, R. Dazeley, C. Foale, S. Firmin, and J. Mummery. Human-aligned artificial intelligence is a multiobjective problem.Ethics and information technology, 20:27–40, 2018

    2018

  47. [55]

    H. Wang, Y . Lin, W. Xiong, R. Yang, S. Diao, S. Qiu, H. Zhao, and T. Zhang. Arithmetic control of llms for diverse user preferences: Directional preference alignment with multi-objective rewards. InProceedings of the 62nd Annual Meeting of the Association for Computational Li...

    2024

  48. [56]

    K. Wang, R. Kidambi, R. Sullivan, A. Agarwal, C. Dann, A. Michi, M. Gelmi, Y . Li, R. Gupta, K. Dubey, et al. Conditional language policy: A general framework for steerable multi-objective finetuning. InProceedings of the Findings of the Association for Computational Linguisti...

    2024

  49. [57]

    X. Wang, Y . Zheng, Z. Wan, and M. Zhang. Svd-llm: Truncation-aware singular value decomposition for large language model compression.arXiv preprint arXiv:2403.07378, 2024

    2024 arXiv

  50. [58]

    Y . Xu, U. M. Sehwag, A. Koppel, S. Zhu, B. An, F. Huang, and S. Ganesh. Genarm: Reward guided generation with autoregressive reward model for test-time alignment. InThe Thirteenth International Conference on Learning Representations, ICLR 2025, 2025

    2025

  51. [59]

    Yadav, D

    P. Yadav, D. Tam, L. Choshen, C. A. Raffel, and M. Bansal. Ties-merging: Resolving interfer- ence when merging models. InProceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  52. [60]

    E. Yang, Z. Wang, L. Shen, S. Liu, G. Guo, X. Wang, and D. Tao. Adamerging: Adaptive model merging for multi-task learning. InProceedings of the Twelfth International Conference on Learning Representations (ICLR), 2024

    2024

  53. [61]

    K. Yang, Z. Liu, Q. Xie, J. Huang, T. Zhang, and S. Ananiadou. Metaaligner: Towards generalizable multi-objective alignment of language models. InProceedings of the Thirty- eighth Annual Conference on Neural Information Processing Systems (NeurIPS), 2024

    2024

  54. [62]

    R. Yang, X. Pan, F. Luo, S. Qiu, H. Zhong, D. Yu, and J. Chen. Rewards-in-context: Multi- objective alignment of foundation models with dynamic preference adjustment. InProceedings of the Forty-first International Conference on Machine Learning (ICML), 2024

    2024

  55. [63]

    Z. Yuan, Y . Shang, Y . Song, Q. Wu, Y . Yan, and G. Sun. ASVD: activation-aware singular value decomposition for compressing large language models.CoRR, abs/2312.05821, 2023

    2023 arXiv

  56. [64]

    Zadouri, A

    T. Zadouri, A. Üstün, A. Ahmadian, B. Ermis, A. Locatelli, and S. Hooker. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. InProceedings of the Twelfth International Conference on Learning Representations (ICLR), 2024. 13

    2024

  57. [65]

    Zhang, B

    L. Zhang, B. Wang, J. Wang, X. Zhao, M. Zhang, H. Yang, M. Zhang, Y . Li, J. Li, J. Yu, and M. Zhang. Function-to-style guidance of llms for code translation. InThe Forty-Second International Conference on Machine Learning (ICML), 2025

    2025

  58. [66]

    Zhang, J

    L. Zhang, J. Wang, M. Zhang, G. Cao, E. Shi, mayuchi, J. Yu, H. LIU, J. Li, and M. Zhang. Speed up your code: Progressive code acceleration through bidirectional tree editing. InThe 63rd Annual Meeting of the Association for Computational Linguistics (ACL), 2025

    2025

  59. [67]

    Zhang and H

    Q. Zhang and H. Li. Moea/d: A multiobjective evolutionary algorithm based on decomposition. IEEE Transactions on evolutionary computation, 11(6):712–731, 2007

    2007

  60. [68]

    Zhang, M

    X. Zhang, M. Burger, and S. J. Osher. A unified primal-dual algorithm framework based on bregman iteration.J. Sci. Comput., 46(1):20–46, 2011

    2011

  61. [69]

    Zheng and H

    S. Zheng and H. Wang. Free-merging: Fourier transform for model merging with lightweight experts.arXiv preprint arXiv.2411.16815, 2024

    2024 arXiv

  62. [70]

    Y . Zhou, W. Li, Y . Lu, J. Li, F. Liu, M. Zhang, Y . Wang, D. He, H. LIU, and M. Zhang. Reflection on knowledge graph for large language models reasoning. InFindings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), 2025

    2025

  63. [71]

    Z. Zhou, J. Liu, J. Shao, X. Yue, C. Yang, W. Ouyang, and Y . Qiao. Beyond one-preference-fits- all alignment: Multi-objective direct preference optimization. InProceedings of Findings of the Association for Computational Linguistics (ACL), pages 10586–10613, 2024

    2024

  64. [72]

    Helpful Assistant

    D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irv- ing. Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593, 2019. 14 A The workflow ofHoE Algorithm 1 show the whole pipeline ofHoE. Algorithm 1The wo...

    1909 arXiv

  65. [73]

    few-experts

    Math Task: Solves math problems from the GSM8K dataset[6]. The first two tasks were evaluated on the over-refusal benchmark[7], the Summary Task was assessed using the average score across three objectives, and the Math Task was evaluated with Pass@1 accuracy on the GSM8K test...

  66. [74]

    Convexity:∀i∈[N],f i(θ)is convex inθ

  67. [75]

    Bounded objectives:∀i∈[N],∀θ∈Θ, f i(θ)≤U

  68. [76]

    Bounded gradients and stochastic gradients: ∀i∈[N],∀θ∈Θ,∥∇f i(θ)∥∞ ≤L,∥δf i(θ)∥∞ ≤ L

  69. [77]

    Bounded feasible region:∀θ∈Θ,∥θ∥ ∞ ≤R θ

  70. [78]

    Policy feasibility: A feasible reference policy π∗ exists such that z∗ is feasible, that is ∃π∈ Π,∀iE x∼D,τ∼π(x) [Ri(τ)] =z ∗ i

  71. [79]

    I think you look beautiful without makeup

    Bounded gradients variance:∀i∈[N],∀θ∈Θ,∥V ar[∇f i(θ)]∥∞ ≤L We define the expected cumulative reward under policy with preferenceλas: V π λ (s) =E τ∼π(x) [ ∞X t=1 γt NX i=1 λiri(st, at)](20) 22 The objective function for Tchebycheff scalarization is given by: L(θ|λ) =E x∼D[V πθ...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.