Pith. sign in

REVIEW 2 major objections 5 minor 113 references

Pointwise constraints cut tail violations in language-model fine-tuning while keeping average performance.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pointwise constrained fine-tuning via sample-wise augmented Lagrangians and learned relaxations reduces tail constraint violations across safety, tool-calling, and re-ranking while preserving average task performance.

T0 review reviewed 2026-07-13 challenge →

load-bearing objection Clean dual recipe for pointwise LM constraints that actually shrinks violation tails on three small-model tasks; ordinary dual-method caveats, not a broken argument. the 2 major comments →

arxiv 2607.08968 v1 pith:CAIULV7A submitted 2026-07-09 eess.SP

Every Sample Counts: Supervised Fine-Tuning of Language Models with Pointwise Constraints

Ignacio Hounie , Ignacio Boero , Alejandro Ribeiro This is my paper
classification eess.SP
keywords pointwise constraintsaugmented Lagrangianlanguage-model fine-tuningsafety alignmentpreference optimizationre-rankingsample-dependent relaxation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Language-model fine-tuning often needs hard requirements (safety, correct tool use, relevance) to hold for every input, not just on average. Average constraints or fixed penalties can still leave large failures on hard or rare samples. This paper formulates fine-tuning as an average-loss objective subject to per-sample inequality constraints, then solves it with an augmented Lagrangian that carries one dual multiplier per training example. When constraints are too strict or data contain outliers, a learned sample-dependent relaxation trades a user-chosen cost against the training objective. Across three small-model tasks—safe instruction following, function-calling preferences, and length-aware re-ranking—the method shrinks the right tail of the violation distribution and yields more even downstream trade-offs than average constraints or common preference losses.

Core claim

Enforcing constraints almost everywhere, rather than only in expectation, and optimizing the resulting dual with sample-wise multipliers and a quadratic augmentation term, reduces both the frequency and severity of large per-sample violations while still attaining competitive average objective values and task metrics.

What carries the argument

The empirical augmented Lagrangian (and its quadratically regularized dual for the relaxed problem) that assigns a nonnegative multiplier to each training sample and is maximized by dual ascent interleaved with a few primal steps on the language-model parameters.

Load-bearing premise

A handful of inexact gradient steps on the model parameters, together with a hand-chosen fixed augmentation strength, is enough for the dual iterates to recover near-feasible solutions that generalize beyond the training samples.

What would settle it

On a held-out test split of any of the three tasks, the empirical CDF of constraint values for the pointwise method fails to dominate the average-constraint and fixed-penalty baselines in the upper tail (high-violation region), or the reported CVaR of violations is no smaller while the average objective is worse.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes a pointwise constrained supervised fine-tuning framework for language models: minimize an average objective subject to almost-everywhere (per-sample) inequality constraints, rather than average constraints. To handle overly tight constraints and outliers it introduces a learned sample-dependent relaxation u with a user-chosen convex cost (instantiated as quadratic β‖u‖²), and solves the resulting problem via an empirical augmented Lagrangian dual with sample-wise multipliers and a fixed augmentation parameter α (Algorithm 1). Strong duality for the empirical relaxed problem is proved under mild assumptions (Prop. 3, App. B). The framework is instantiated on three small-LM tasks—safety refusal in instruction following, preference constraints in tool calling, and length-aware re-ranking—and compared to average constraints, fixed penalties, and task-specific baselines. Empirically, pointwise (and relaxed) methods reduce the tails of the constraint-violation distribution (CDFs, CVaR) while preserving or improving average objective and downstream metrics, often expanding the Pareto frontier (Figs. 1–4, Tables 2, 10–12).

Significance. If the claims hold, the work supplies a clean, reusable dual formulation that directly targets sample-level disparities that average-constrained alignment can leave unaddressed. The combination of pointwise constraints, resilient quadratic relaxation (Prop. 1), and an AL dual tailored to non-convex LM fine-tuning is a natural and useful extension of constrained learning theory to SFT. Strengths include: (i) clean duality derivations and a strong-duality result for the empirical relaxed problem; (ii) three distinct, carefully specified task instantiations that demonstrate flexibility; (iii) transparent empirical reporting (seed means ± std, full CDFs, CVaR, Pareto fronts, ablations on α/β, and explicit negative cases in E.4). The contribution is methods-oriented rather than a new theoretical guarantee for non-convex LMs, but it is a solid and timely addition to constrained alignment.

major comments (2)
  1. §3.3 and Algorithm 1 (and App. C): The central practical claim—that modest inexact primal steps plus a hand-chosen fixed α recover near-feasible, generalizing solutions—rests on the cited AL-COLE theory without finite-sample rates for the non-convex LM setting. The paper already scopes this as a limitation and shows empirical stability to α (App. F.2) and out-of-sample CDFs/CVaR, so it does not overturn the results; still, a short discussion of when the dual may fail to generalize (linking to the E.4 negative cases) would strengthen the load-bearing optimization claim.
  2. §4 / Tables 2, 10–12 and Figs. 1–4: The evidence that tail reduction improves downstream metrics is convincing on the reported small models and datasets, but the paper does not quantify how much of the gain is due to the sample-wise multipliers versus the particular choice of ε and β. A more systematic sensitivity analysis (or a simple ablation that freezes the dual variables after a few epochs) would make H2 more robust; without it the Pareto improvements remain empirical rather than fully attributed.
minor comments (5)
  1. Notation: the same symbol ε is used for different constraint levels across tasks (ε_U, ε_H, ε_win, ε_lose, margin ε); a short table of task-specific tolerances would help.
  2. Fig. 1 and related CDFs: axis labels and the meaning of positive vs. negative constraint values could be stated more explicitly in the caption for readers who skip §4.1.
  3. App. A / related work: the connection to soft-margin SVMs and scenario optimization is useful; a sentence clarifying that the present work targets infinite-dimensional (functional) duals rather than finite-sample scenario bounds would avoid confusion.
  4. Typos / polish: occasional missing spaces after periods and a few long sentences in §3.2–3.3; a light copy-edit would improve readability.
  5. Reproducibility: runtime/memory overhead is discussed (App. F) and is negligible; releasing the dual-ascent implementation or a minimal reference script would further help adoption.

Circularity Check

1 steps flagged

No significant circularity: dual construction and empirical claims are independent of fitted targets; only minor self-citation of authors' AL theory for non-convex guarantees.

specific steps
  1. self citation load bearing [§3.1 (after Eq. 6) and §3.3 / App. C (Alg. 1)]
    "under the conditions studied in [27], an approximate solution for a constrained learning problem can be obtained via the primal minimizer of the empirical augmented Lagrangian at the optimal augmented dual pair [27, Theorem 2.3]. ... convergence can still be guaranteed as long as the primal minimization error decreases sufficiently across iterations [27, Theorem 3.1]."

    The paper invokes its own concurrent AL-COLE preprint for the key non-convex duality-gap and inexact-ascent guarantees that justify using fixed-α dual ascent with few primal steps. This is a self-citation of overlapping authors, but it is not load-bearing for the central empirical claims (tail reduction, Pareto improvements), which are validated independently on held-out CDFs/CVaR/downstream metrics; the citation merely supplies theoretical scaffolding already partially re-derived for the relaxed case in App. B.

full rationale

The derivation chain is standard constrained optimization applied to LM fine-tuning: primal (P) with pointwise inequalities, Lagrangian dual, then augmented Lagrangian (6) to address non-convexity, plus quadratic relaxation whose dual is derived in Prop. 1 (rescaling + regularization of gA). These steps are algebraic identities or classical AL results (Rockafellar, Wierzbicki–Kurcyusz), not self-definitional. Hyperparameters ε, β, α are free knobs whose effects are ablated (Figs. 1, 4; App. F), not fitted then re-labeled as predictions. Downstream metrics (CVaR tails, F1, LC-WR, MRR/LenRank, Pareto) are held-out and distinct from the dual objective, so empirical claims C1–C4/H1–H2 stand independently. The sole self-citation of note is [27] (same authors) for AL strong duality / inexact primal-dual convergence (Thm. 2.3, 3.1); this supplies theoretical motivation but is not load-bearing for the reported reductions in violation tails, which rest on the experiments themselves. No uniqueness theorem forbids alternatives, no ansatz is smuggled, and soft-margin SVM is correctly acknowledged as a classical special case rather than renamed. Score 1 reflects only that minor self-citation; the paper is otherwise self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 1 invented entities

The central empirical claims rest on standard non-convex duality theory, the modeling choice that pointwise almost-everywhere constraints are the right target, and a handful of free dual/relaxation hyper-parameters that are grid-searched rather than derived.

free parameters (4)
  • relaxation cost β
    Controls the quadratic penalty on dual multipliers / primal slacks; swept over orders of magnitude and shown to trade objective vs. constraint tails.
  • augmentation parameter α
    Fixed by hand (often 1 or 1000); ablation shows relative stability but remains a free choice.
  • constraint tolerances ε (ε_U, ε_H, ε_win, ε_lose, margin ε)
    Task-specific thresholds set by the user or grid search; directly define the feasible set.
  • dual learning rate η
    Chosen by grid search; affects stability of the dual ascent.
axioms (3)
  • domain assumption The empirical augmented dual with inexact primal steps converges to a near-optimal dual pair when α is large enough (shifted-penalty theory).
    Invoked in §3.3 and Appendix C; relies on [27,40] without new finite-sample rates for transformers.
  • domain assumption Pointwise almost-everywhere constraints are the appropriate modeling target for safety/preference/length requirements.
    Stated as (C1) and used to define problem (P); alternative high-probability formulations are discussed but not used.
  • standard math Standard continuity, compactness, and bounded-loss assumptions that guarantee strong duality of the empirical relaxed problem.
    Assumptions 1–3 in Appendix B.4; mild and standard for the setting.
invented entities (1)
  • sample-dependent relaxation function u(x,y) with quadratic cost c(u)=β‖u‖² no independent evidence
    purpose: Softens overly restrictive pointwise constraints while remaining dual-regularizable.
    Defined in (PR); classical soft-margin idea lifted to functional dual form; no independent physical existence claimed.

reviewed 2026-07-13 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Every Sample Counts: Supervised Fine-Tuning of Language Models with Pointwise Constraints." pith.science (2026). https://pith.science/paper/CAIULV7A

@misc{pith2026260708968,
  author       = {Pith},
  title        = {Pith review of: Every Sample Counts: Supervised Fine-Tuning of Language Models with Pointwise Constraints},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CAIULV7A}},
  note         = {Machine review of arXiv:2607.08968}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Fine-tuning language models often requires enforcing constraints on individual inputs without compromising downstream performance. Existing constrained alignment methods impose constraints on average, which can induce undesirable disparities across inputs or users. We propose a novel alignment framework that addresses this gap by enforcing per-sample constraints while still minimizing an average loss. To mitigate the impact of overly restrictive constraints and outliers, we introduce a learned, sample-dependent relaxation that minimally adjusts the constraints, trading off a user-defined relaxation cost with the training objective. To address practical duality and optimization challenges, we develop an augmented Lagrangian approach tailored to this formulation. We demonstrate the flexibility of the framework by instantiating it under distinct small language-model fine-tuning tasks and constraints: safety in instruction following, preferences in function calling and length in re-ranking. Across these settings, our approach reduces tail constraint violations while largely preserving the model's performance.

Figures

Figures reproduced from arXiv: 2607.08968 by Alejandro Ribeiro, Ignacio Boero, Ignacio Hounie.

Figure 1
Figure 1. Figure 1: Empirical Cumulative Distribution Function (CDF) of sample-wise constraint values [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Length normalised accuracy, Hallucination Rates and F1 scores for Llama-3.2-1B [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Safety vs Helpfulness trade-off. (Left) length controlled win rates on AlpacaEval [ [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: MRR@10 for ranking the positive passage versus LenRank@10 metric. The base model is [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Empirical CDF of the tokenwise average log likelihood, i.e. [PITH_FULL_IMAGE:figures/full_fig_p034_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Empirical CDF of the tokenwise average log likelihood, i.e. [PITH_FULL_IMAGE:figures/full_fig_p034_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Empirical CDF of the tokenwise average log likelihood, i.e. [PITH_FULL_IMAGE:figures/full_fig_p034_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Empirical CDF of the tokenwise average log likelihood, i.e. [PITH_FULL_IMAGE:figures/full_fig_p035_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Empirical CDF of the tokenwise average log likelihood, i.e. [PITH_FULL_IMAGE:figures/full_fig_p035_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Empirical CDF of the tokenwise average log likelihood, i.e. [PITH_FULL_IMAGE:figures/full_fig_p036_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Empirical CDF of the tokenwise average log likelihood, i.e. [PITH_FULL_IMAGE:figures/full_fig_p036_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Empirical Cumulative Density Function (CDF) of length normalized likelihood for unsafe [PITH_FULL_IMAGE:figures/full_fig_p037_12.png] view at source ↗
Figure 14
Figure 14. Figure 14: The ablation also shows that the augmented Lagrangian is substantially more robust to the choice of dual step size. For the standard Lagrangian, decreasing the dual step size leads to a large degradation in MRR@10, from 0.711 at step size 1.0 to 0.488 at step size 10−3 . In contrast, the augmented Lagrangian remains stable across step sizes 10−3 , 10−2 , and 10−1 , with MRR@10 values between 0.724 and 0.7… view at source ↗
Figure 13
Figure 13. Figure 13: Empirical Cumulative Density Function (CDF) of likelihood ratios for preference data. In [PITH_FULL_IMAGE:figures/full_fig_p039_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Comparison between the standard dual formulation and the augmented dual formulation [PITH_FULL_IMAGE:figures/full_fig_p039_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

113 extracted references · 1 canonical work pages

  1. [1]

    A comprehensive survey on evaluating large language model applications in the medical industry.arXiv preprint arXiv:2404.15777, 2024

    Yining Huang, Keke Tang, Meilian Chen, and Boyuan Wang. A comprehensive survey on evaluating large language model applications in the medical industry.arXiv preprint arXiv:2404.15777, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    On large language models in mission-critical it governance: Are we ready yet?, 2025

    Matteo Esposito, Francesco Palagiano, Valentina Lenarduzzi, and Davide Taibi. On large language models in mission-critical it governance: Are we ready yet?, 2025. URL https: //arxiv.org/abs/2412.11698

    Pith/arXiv arXiv 2025

  3. [3]

    Safe RLHF: Safe reinforcement learning from human feedback

    Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe reinforcement learning from human feedback. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  4. [4]

    One-shot safety alignment for large language models via optimal dualization.Advances in Neural Information Processing Systems, 37:84350–84383, 2024

    Xinmeng Huang, Shuo Li, Edgar Dobriban, Osbert Bastani, Hamed Hassani, and Dongsheng Ding. One-shot safety alignment for large language models via optimal dualization.Advances in Neural Information Processing Systems, 37:84350–84383, 2024

    2024

  5. [5]

    Stepwise alignment for constrained language model policy optimization.Advances in Neural Information Processing Systems, 37:104471–104520, 2024

    Akifumi Wachi, Thien Q Tran, Rei Sato, Takumi Tanabe, and Youhei Akimoto. Stepwise alignment for constrained language model policy optimization.Advances in Neural Information Processing Systems, 37:104471–104520, 2024

    2024

  6. [6]

    Enhancing LLM safety via constrained direct preference optimization.arXiv preprint arXiv:2403.02475, 2024

    Zixuan Liu, Xiaolin Sun, and Zizhan Zheng. Enhancing LLM safety via constrained direct preference optimization.arXiv preprint arXiv:2403.02475, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Enhancing safety in reinforcement learning with human feedback via rectified policy optimization.arXiv preprint arXiv:2410.19933, 2024

    Xiyue Peng, Hengquan Guo, Jiawei Zhang, Dongqing Zou, Ziyu Shao, Honghao Wei, and Xin Liu. Enhancing safety in reinforcement learning with human feedback via rectified policy optimization.arXiv preprint arXiv:2410.19933, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    L3ms–lagrange large language models.arXiv preprint arXiv:2410.21533, 2024

    Guneet S Dhillon, Xingjian Shi, Yee Whye Teh, and Alex Smola. L3ms–lagrange large language models.arXiv preprint arXiv:2410.21533, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Adversarial training for high-stakes reliability.Advances in neural information processing systems, 35: 9274–9286, 2022

    Daniel Ziegler, Seraphina Nix, Lawrence Chan, Tim Bauman, Peter Schmidt-Nielsen, Tao Lin, Adam Scherlis, Noa Nabeshima, Benjamin Weinstein-Raun, Daniel de Haas, et al. Adversarial training for high-stakes reliability.Advances in neural information processing systems, 35: 9274–9286, 2022

    2022

  10. [10]

    Confronting reward model overoptimization with constrained RLHF

    Ted Moskovitz, Aaditya K Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca Dragan, and Stephen Marcus McAleer. Confronting reward model overoptimization with constrained RLHF. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id=gkfUvn0fLU

    2024

  11. [11]

    Large lan- guage models struggle to learn long-tail knowledge

    Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large lan- guage models struggle to learn long-tail knowledge. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors,Proceed- ings of the 40th International Conference on Machine Learning, volume 202 ofProceed- ings of M...

    2023

  12. [12]

    The devil is in the tails: How long-tailed code distributions impact large language models

    Xin Zhou, Kisub Kim, Bowen Xu, Jiakun Liu, DongGyun Han, and David Lo. The devil is in the tails: How long-tailed code distributions impact large language models. In2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 40–52,

  13. [13]

    doi: 10.1109/ASE56229.2023.00157

    doi:10.1109/ase56229.2023.00157 2023

  14. [14]

    The neglected tails in vision-language models

    Shubham Parashar, Zhiqiu Lin, Tian Liu, Xiangjue Dong, Yanan Li, Deva Ramanan, James Caverlee, and Shu Kong. The neglected tails in vision-language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12988–12997, June 2024. 10

    2024

  15. [15]

    The law of knowledge overshadowing: Towards understanding, predicting, and preventing llm hallucination.arXiv preprint arXiv:2502.16143, 2025

    Yuji Zhang, Sha Li, Cheng Qian, Jiateng Liu, Pengfei Yu, Chi Han, Yi R Fung, Kathleen McKeown, Chengxiang Zhai, Manling Li, et al. The law of knowledge overshadowing: Towards understanding, predicting, and preventing llm hallucination.arXiv preprint arXiv:2502.16143, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    Is gpt-oss good? a comprehensive evaluation of openai’s latest open source models.arXiv preprint arXiv:2508.12461, 2025

    Ziqian Bi, Keyu Chen, Chiung-Yi Tseng, Danyang Zhang, Tianyang Wang, Hongying Luo, Lu Chen, Junming Huang, Jibin Guan, Junfeng Hao, et al. Is gpt-oss good? a comprehensive evaluation of openai’s latest open source models.arXiv preprint arXiv:2508.12461, 2025

    arXiv 2025

  17. [17]

    Probably approximately correct constrained learning

    Luiz Chamon and Alejandro Ribeiro. Probably approximately correct constrained learning. Advances in Neural Information Processing Systems, 33:16722–16735, 2020

    2020

  18. [18]

    Achieving zero constraint violation for constrained reinforcement learning via primal-dual approach, 2022

    Qinbo Bai, Amrit Singh Bedi, Mridul Agarwal, Alec Koppel, and Vaneet Aggarwal. Achieving zero constraint violation for constrained reinforcement learning via primal-dual approach, 2022. URLhttps://arxiv.org/abs/2109.06332

    Pith/arXiv arXiv 2022

  19. [19]

    Strong duality relations in nonconvex risk- constrained learning

    Dionysis Kalogerias and Spyridon Pougkakiotis. Strong duality relations in nonconvex risk- constrained learning. In2024 58th Annual Conference on Information Sciences and Systems (CISS), pages 1–6, 2024. doi: 10.1109/CISS59072.2024.10480186

    doi:10.1109/ciss59072.2024.10480186 2024

  20. [20]

    Andrei Manolache, Luiz F. O. Chamon, and Mathias Niepert. Learning (approximately) equivariant networks via constrained optimization, 2025. URL https://arxiv.org/abs/ 2505.13631

    arXiv 2025

  21. [21]

    Constrained sliced wasserstein embedding, 06 2025

    Navid Naderializadeh, Darian Salehi, Xinran Liu, and Soheil Kolouri. Constrained sliced wasserstein embedding, 06 2025

    2025

  22. [22]

    Near-optimal solutions of constrained learning problems.arXiv preprint arXiv:2403.11844, 2024

    Juan Elenter, Luiz FO Chamon, and Alejandro Ribeiro. Near-optimal solutions of constrained learning problems.arXiv preprint arXiv:2403.11844, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    Simultaneous multi-robot motion planning with projected diffusion models.arXiv preprint arXiv:2502.03607, 2025

    Jinhao Liang, Jacob K Christopher, Sven Koenig, and Ferdinando Fioretto. Simultaneous multi-robot motion planning with projected diffusion models.arXiv preprint arXiv:2502.03607, 2025

    Pith/arXiv arXiv 2025

  24. [24]

    Qifeng Hu, Shamsulhaq Basir, and Inanc Senocak. Conditionally adaptive augmented lagrangian method for physics-informed learning of forward and inverse problems using artificial neural networks.arXiv preprint arXiv:2508.15695, 2025

    arXiv 2025

  25. [25]

    Safe: Finding sparse and flat minima to improve pruning.arXiv preprint arXiv:2506.06866, 2025

    Dongyeop Lee, Kwanhee Lee, Jinseok Chung, and Namhoon Lee. Safe: Finding sparse and flat minima to improve pruning.arXiv preprint arXiv:2506.06866, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    Constrained discrete diffusion.arXiv preprint arXiv:2503.09790, 2025

    Michael Cardei, Jacob K Christopher, Thomas Hartvigsen, Bhavya Kailkhura, and Ferdinando Fioretto. Constrained discrete diffusion.arXiv preprint arXiv:2503.09790, 2025

    arXiv 2025

  27. [27]

    Dual optimistic ascent (pi control) is the augmented lagrangian method in disguise.arXiv preprint arXiv:2509.22500, 2025

    Juan Ramirez and Simon Lacoste-Julien. Dual optimistic ascent (pi control) is the augmented lagrangian method in disguise.arXiv preprint arXiv:2509.22500, 2025

    arXiv 2025

  28. [28]

    Al-cole: Augmented lagrangian for constrained learning.arXiv preprint arXiv:2510.20995, 2025

    Ignacio Boero, Ignacio Hounie, and Alejandro Ribeiro. Al-cole: Augmented lagrangian for constrained learning.arXiv preprint arXiv:2510.20995, 2025

    arXiv 2025

  29. [29]

    Xstest: A test suite for identifying exaggerated safety behaviours in large language models

    Paul Röttger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers),...

    2024

  30. [30]

    Safedpo: A simple approach to direct preference optimization with enhanced safety.arXiv preprint arXiv:2505.20065, 2025

    Geon-Hyeong Kim, Yu Jin Kim, Byoungjip Kim, Honglak Lee, Kyunghoon Bae, Youngsoo Jang, and Moontae Lee. Safedpo: A simple approach to direct preference optimization with enhanced safety.arXiv preprint arXiv:2505.20065, 2025

    arXiv 2025

  31. [31]

    Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions

    Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Rottger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions. InThe Twelfth International Conference on Learning Representations. 11

  32. [32]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. InProceedings of the Advances in Neural Information Processing Systems, volume 36, 2024

    2024

  33. [33]

    Simpo: Simple preference optimization with a reference-free reward.Advances in Neural Information Processing Systems, 37:124198–124235, 2024

    Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward.Advances in Neural Information Processing Systems, 37:124198–124235, 2024

    2024

  34. [34]

    Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation.arXiv preprint arXiv:2401.08417, 2024

    Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Ken- ton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation.arXiv preprint arXiv:2401.08417, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Slic- hf: Sequence likelihood calibration with human feedback.arXiv preprint arXiv:2305.10425, 2023

    Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic- hf: Sequence likelihood calibration with human feedback.arXiv preprint arXiv:2305.10425, 2023

    Pith/arXiv arXiv 2023

  36. [36]

    Retrieval-augmented generation for knowledge-intensive nlp tasks.Advances in neural information processing systems, 33:9459–9474, 2020

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks.Advances in neural information processing systems, 33:9459–9474, 2020

    2020

  37. [37]

    Tyrrell Rockafellar

    R. Tyrrell Rockafellar. Augmented lagrange multiplier functions and duality in nonconvex programming.SIAM Journal on Control, 12(2):268–285, 1974. doi: 10.1137/0312021

    doi:10.1137/0312021 1974

  38. [38]

    The augmented lagrangian methods: Overview and recent advances.arXiv preprint arXiv:2510.16827, 2025

    Kangkang Deng, Rui Wang, Zhenyuan Zhu, Junyu Zhang, and Zaiwen Wen. The augmented lagrangian methods: Overview and recent advances.arXiv preprint arXiv:2510.16827, 2025

    arXiv 2025

  39. [39]

    Resilient constrained learning

    Ignacio Hounie, Alejandro Ribeiro, and Luiz FO Chamon. Resilient constrained learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  40. [40]

    Rockafellar.Convex Analysis

    R.T. Rockafellar.Convex Analysis. Princeton Landmarks in Mathematics and Physics. Princeton University Press, 1997. ISBN 9780691015866. URL https://books.google.com/books? id=GV6YDwAAQBAJ

    1997

  41. [41]

    Wierzbicki and Stanislaw Kurcyusz

    Andrzej P. Wierzbicki and Stanislaw Kurcyusz. Projection on a cone, penalty functionals and duality theory for problems with inequaltity constraints in hilbert space.SIAM Journal on Control and Optimization, 15(1):25–56, 1977. doi: 10.1137/0315003. URL https://doi. org/10.1137/0315003

    work page doi:10.1137/0315003 1977

  42. [42]

    Reproducibility in machine-learning-based research: Overview, barriers, and drivers.AI Magazine, 46(2):e70002, 2025

    Harald Semmelrock, Tony Ross-Hellauer, Simone Kopeinik, Dieter Theiler, Armin Haberl, Stefan Thalmann, and Dominik Kowald. Reproducibility in machine-learning-based research: Overview, barriers, and drivers.AI Magazine, 46(2):e70002, 2025

    2025

  43. [43]

    Small language models are the future of agentic ai,

    Peter Belcak, Greg Heinrich, Shizhe Diao, Yonggan Fu, Xin Dong, Saurav Muralidharan, Yingyan Celine Lin, and Pavlo Molchanov. Small language models are the future of agentic ai,

  44. [44]

    URLhttps://arxiv.org/abs/2506.02153

    Pith/arXiv arXiv

  45. [45]

    Fali Wang, Zhiwei Zhang, Xianren Zhang, Zongyu Wu, Tzuhao Mo, Qiuhao Lu, Wanjing Wang, Rui Li, Junjie Xu, Xianfeng Tang, et al. A comprehensive survey of small language models in the era of large language models: Techniques, enhancements, applications, collaboration with llms, and trustworthiness.ACM Transactions on Intelligent Systems and Technology, 16(...

    2025

  46. [46]

    When2call: When (not) to call tools

    Hayley Ross, Ameya Sunil Mahabaleshwarkar, and Yoshi Suhara. When2call: When (not) to call tools. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3391–3409, 2025

    2025

  47. [47]

    The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models

    Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. InForty-second International Conference on Machine Learning, 2025. 12

    2025

  48. [48]

    Apigen: Generative api method recommendation

    Yujia Chen, Cuiyun Gao, Muyijie Zhu, Qing Liao, Yong Wang, and Guoai Xu. Apigen: Generative api method recommendation. In2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 171–182. IEEE, 2024

    2024

  49. [49]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  50. [50]

    Apigen-mt: Agentic pipeline for multi-turn data generation via simulated agent-human interplay.arXiv preprint arXiv:2504.03601, 2025

    Akshara Prabhakar, Zuxin Liu, Ming Zhu, Jianguo Zhang, Tulika Awalgaonkar, Shiyu Wang, Zhiwei Liu, Haolin Chen, Thai Hoang, Juan Carlos Niebles, et al. Apigen-mt: Agentic pipeline for multi-turn data generation via simulated agent-human interplay.arXiv preprint arXiv:2504.03601, 2025

    Pith/arXiv arXiv 2025

  51. [51]

    Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning

    Hao Zhao, Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning. InInternational Conference on Machine Learning, pages 60674–60703. PMLR, 2024

    2024

  52. [52]

    Stanford alpaca: An instruction-following llama model, 2023

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following llama model, 2023

    2023

  53. [53]

    Beavertails: Towards improved safety alignment of LLM via a human-preference dataset

    Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of LLM via a human-preference dataset. InProceedings of the Advances in Neural Information Processing Systems, volume 36, 2024

    2024

  54. [54]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023

    2023

  55. [55]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models, 2023

    2023

  56. [56]

    Ms marco: A human generated machine reading comprehension dataset.arXiv preprint arXiv:1611.09268, 2016

    Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human generated machine reading comprehension dataset.arXiv preprint arXiv:1611.09268, 2016

    Pith/arXiv arXiv 2016

  57. [57]

    Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference

    Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Griffin Thomas Adams, Jeremy Howard, and Iacopo Poli. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. InProceedi...

    2025

  58. [58]

    Multi-stage document ranking with bert.arXiv preprint arXiv:1910.14424, 2019

    Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. Multi-stage document ranking with bert.arXiv preprint arXiv:1910.14424, 2019

    Pith/arXiv arXiv 1910

  59. [59]

    C- pack: Packed resources for general chinese embeddings

    Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. C- pack: Packed resources for general chinese embeddings. InProceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, pages 641–649, 2024

    2024

  60. [60]

    cross-encoder/ms-marco-minilm-l12-v2

    Sentence Transformers. cross-encoder/ms-marco-minilm-l12-v2. https://huggingface. co/cross-encoder/ms-marco-MiniLM-L12-v2 , 2026. Hugging Face model card, accessed 2026-04-21

    2026

  61. [61]

    Pattern recognition using generalized portrait method.Automa- tion and Remote Control, 24:774–780, 1963

    Vladimir Naumovich Vapnik. Pattern recognition using generalized portrait method.Automa- tion and Remote Control, 24:774–780, 1963. URL https://api.semanticscholar.org/ CorpusID:115205884. 13

    1963

  62. [62]

    Boser, Isabelle M

    Bernhard E. Boser, Isabelle M. Guyon, and Vladimir N. Vapnik. A training algorithm for optimal margin classifiers. InProceedings of the Fifth Annual Workshop on Computational Learning Theory, COLT ’92, page 144–152, New York, NY , USA, 1992. Association for Computing Machinery. ISBN 089791497X. doi: 10.1145/130385.130401. URL https: //doi.org/10.1145/1303...

    doi:10.1145/130385.130401 1992

  63. [63]

    Support-vector networks.Machine learning, 20(3): 273–297, 1995

    Corinna Cortes and Vladimir Vapnik. Support-vector networks.Machine learning, 20(3): 273–297, 1995

    1995

  64. [64]

    A theory of the risk for optimization with relaxation and its application to support vector machines.Journal of Machine Learning Research, 22(288): 1–38, 2021

    Marco C Campi and Simone Garatti. A theory of the risk for optimization with relaxation and its application to support vector machines.Journal of Machine Learning Research, 22(288): 1–38, 2021

    2021

  65. [65]

    Non-convex scenario optimization.Mathematical Programming, 209(1):557–608, 2025

    Simone Garatti and Marco C Campi. Non-convex scenario optimization.Mathematical Programming, 209(1):557–608, 2025

    2025

  66. [66]

    Feasible learning, 2025

    Juan Ramirez, Ignacio Hounie, Juan Elenter, Jose Gallego-Posada, Meraj Hashemizadeh, Alejandro Ribeiro, and Simon Lacoste-Julien. Feasible learning, 2025. URL https://arxiv. org/abs/2501.14912

    Pith/arXiv arXiv 2025

  67. [67]

    Reinforce- ment learning with almost sure constraints

    Agustin Castellano, Hancheng Min, Enrique Mallada, and Juan Andrés Bazerque. Reinforce- ment learning with almost sure constraints. In Roya Firoozi, Negar Mehr, Esen Yel, Rika Antonova, Jeannette Bohg, Mac Schwager, and Mykel Kochenderfer, editors,Proceedings of The 4th Annual Learning for Dynamics and Control Conference, volume 168 ofPro- ceedings of Mac...

    2022

  68. [68]

    Anytime-constrained reinforcement learning

    Jeremy McMahan and Xiaojin Zhu. Anytime-constrained reinforcement learning. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors,Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 ofProceedings of Machine Learning Research, pages 4321–4329. PMLR, 02–04 May 2024. URLhttps://proceedings. mlr.press/v...

    2024

  69. [69]

    Sauté rl: Almost surely safe reinforcement learning using state augmentation

    Aivar Sootla, Alexander I Cowen-Rivers, Taher Jafferjee, Ziyan Wang, David H Mguni, Jun Wang, and Haitham Ammar. Sauté rl: Almost surely safe reinforcement learning using state augmentation. InInternational Conference on Machine Learning, pages 20423–20443. PMLR, 2022

    2022

  70. [70]

    On tilted losses in machine learning: Theory and applications.Journal of Machine Learning Research, 24(142):1–79, 2023

    Tian Li, Ahmad Beirami, Maziar Sanjabi, and Virginia Smith. On tilted losses in machine learning: Theory and applications.Journal of Machine Learning Research, 24(142):1–79, 2023. URLhttp://jmlr.org/papers/v24/21-1095.html

    2023

  71. [71]

    Large-scale methods for distributionally robust optimization.Advances in neural information processing systems, 33: 8847–8860, 2020

    Daniel Levy, Yair Carmon, John C Duchi, and Aaron Sidford. Large-scale methods for distributionally robust optimization.Advances in neural information processing systems, 33: 8847–8860, 2020

    2020

  72. [72]

    Probabilistically robust learning: Balancing average and worst-case performance

    Alexander Robey, Luiz Chamon, George J Pappas, and Hamed Hassani. Probabilistically robust learning: Balancing average and worst-case performance. InInternational Conference on Machine Learning, pages 18667–18686. PMLR, 2022

    2022

  73. [73]

    Robust llm alignment via distributionally robust direct preference optimization

    Zaiyan Xu, Sushil Vemuri, Kishan Panaganti, Dileep Kalathil, Rahul Jain, and Deepak Ra- machandran. Robust llm alignment via distributionally robust direct preference optimization. arXiv preprint arXiv:2502.01930, 2025

    arXiv 2025

  74. [74]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. InProceedings of the Advances in Neural Information Processing Systems, volume 33, pages 3008–3021, 2020

    2020

  75. [75]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022. 14

    2022

  76. [76]

    Rrhf: Rank responses to align language models with human feedback without tears.arXiv preprint arXiv:2304.05302, 2023

    Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears.arXiv preprint arXiv:2304.05302, 2023

    Pith/arXiv arXiv 2023

  77. [77]

    Repo: Understanding preference learning through relu-based optimization

    Junkang Wu, Kexin Huang, Xue Wang, Jinyang Gao, Bolin Ding, Jiancan Wu, Xiangnan He, and Xiang Wang. Repo: Understanding preference learning through relu-based optimization. arXiv preprint arXiv:2503.07426, 2025

    arXiv 2025

  78. [78]

    A general theoretical paradigm to understand learn- ing from human preferences

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learn- ing from human preferences. InProceedings of the International Conference on Artificial Intelligence and Statistics, pages 4447–4455, 2024

    2024

  79. [79]

    Cal-dpo: Calibrated direct preference optimization for language model alignment.Advances in Neural Information Processing Systems, 37:114289–114320, 2024

    Teng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li, and Vasant G Honavar. Cal-dpo: Calibrated direct preference optimization for language model alignment.Advances in Neural Information Processing Systems, 37:114289–114320, 2024

    2024

  80. [80]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    Pith/arXiv arXiv 2022

Showing first 80 references.

This paper was first reviewed by grok-4.5 on July 13, 2026.