Pith. sign in

REVIEW 4 major objections 5 minor 45 references

Model Diffusion for Certifiable Few-shot Transfer Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By confining the downstream hypothesis space to a finite set of adapters sampled from a diffusion model, STEEL obtains non-vacuous PAC-Bayes risk certificates in the low-shot regime while matching standard fine-tuning accuracy.

desk verdict A genuinely new combination — diffusion-sampled adapters plus finite-hypothesis PAC-Bayes — that delivers the first non-vacuous low-shot certificates at scale, but the unreported confidence epsilon blocks verification of every reported bound. read the letter →

arxiv 2502.06970 v2 pith:FCMFJUMV submitted 2025-02-10 cs.LG stat.ML

Fady Rezk , Royson Lee , Henry Gouk , Timothy Hospedales , Minyoung Kim This is my paper · ORCID
classification cs.LGstat.ML MSC 68T0768T05
keywords PAC-Bayesboundsfew-shotlearningdiffusionmodelsparameter-efficientfine-tuninggeneralizationcertificatestransferriskmodelzoo
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes STEEL, a transfer-learning method that replaces gradient-based fine-tuning on a new task with a sample-then-evaluate procedure. A diffusion model is trained on the parameter-efficient adapters fitted to upstream tasks; a finite pool of adapter candidates is sampled from it, and the learner picks the candidate with the lowest loss on the low-shot support set. Because the hypothesis space is finite and fixed before the target data is seen, a classic PAC-Bayes bound for finite hypothesis classes yields a computable risk certificate whose penalty grows only as the square root of the log of the pool size. On Flan-T5 with LoRA-XS adapters (LaMP personalization) and CLIP with CoOp prompts (four fine-grained vision datasets), the method reports non-vacuous certificates — guaranteed error below 1, and often below chance level — for most low-shot episodes, while SGD, MeZO, BBPT, Meta-PB, and LoRA-Hub with quantized-parameter bounds remain vacuous or weaker. The practical significance is that a user can certify a minimum accuracy level before deploying a low-data model, not merely hope the fine-tuned model generalizes.

What carries the argument

The carrier is a finite hypothesis set $\Theta$ of PEFT parameters produced by a diffusion model trained on a zoo of upstream adapters. Sampling from the diffusion model defines $\Theta$; selecting by minimum support loss, $\theta^* = \arg\min_{\theta\in\Theta} r(\theta)$, turns learning into a finite search. The certificate is the finite-hypothesis PAC-Bayes bound of Eq. (3), derived with a uniform prior over $\Theta$ and a Dirac-delta posterior, which makes the complexity term $C\sqrt{(\log|\Theta|+\log(1/\epsilon))/(2n)}$; since $|\Theta|$ appears only logarithmically, even tens of thousands of candidates keep the bound tight. Two search procedures are used to select a candidate: exhaustive evaluation when forward passes are cheap (vision), and hierarchical clustering-based search that evaluates cluster medoids first and then the best cluster's members (LLM text generation). Both procedures leave the bound in Eq. (3) valid; the approximate search only risks a slightly higher support-error term.

What would settle it

Take a collection of downstream tasks from a distribution the upstream model zoo does not cover — for instance, classes or user types excluded from the upstream splits — run STEEL, and record the minimum support error $\min_{\theta\in\Theta} r(\theta)$ for each episode. If for a nontrivial fraction of such tasks this minimum exceeds $1 - C\sqrt{(\log|\Theta|+\log(1/\epsilon))/(2n)}$, the certificates are vacuous and the central claim would not survive for those tasks.

Watch

Extended reading notes

Core claim

The central claim is that adaptation to a new task can be replaced by selection, and that this replacement is what makes generalization certifiable. The paper trains a diffusion model on the parameter-efficient adapters fitted to upstream tasks, samples a finite pool $\Theta$ of adapter parameters from it, and chooses $\theta^* = \arg\min_{\theta\in\Theta} r(\theta)$ on the support set. Because $\Theta$ is finite and fixed before seeing the target data, the standard PAC-Bayes finite-hypothesis bound applies: with probability at least $1-\epsilon$, $R(\theta) \le r(\theta) + C\sqrt{(\log|\Theta| + \log(1/\epsilon))/(2n)}$ for every $\theta \in \Theta$. The right-hand side is computable and can be non-vacuous even for $n$ in the single digits, because $|\Theta|$ enters only through a logarithm. The experiments instantiate this with Flan-T5 plus LoRA-XS on LaMP personalization tasks and CLIP plus CoOp prompts on four fine-grained vision datasets; the paper reports that STEEL produces non-vacuous certificates for a large fraction of episodes (99% on LaMP-5 and 97.5–100% on vision datasets at 16 shots) while the continuous-space baselines SGD, MeZO, BBPT, and Meta-PB remain vacuous throughout, and that STEEL's test accuracy is comparable to ordinary fine-tuning.

Load-bearing premise

The load-bearing premise is that the diffusion model trained on upstream tasks proposes at least one adapter that fits the new task's small support set well enough that its measured error plus a small penalty term stays below 1 (or below chance level); if the sampled pool misses the task, the certificate is vacuous no matter how small the complexity term is.

Editorial extensions

If this is right

  • Any downstream task whose support set is drawn i.i.d. from the same task distribution as the upstream tasks can be certified with probability $1-\epsilon$, giving the first reported non-vacuous generalization bounds for low-shot learning in large vision and language models.
  • Because the penalty grows as $\sqrt{\log|\Theta|}$, users can afford large candidate pools; more samples improve the support error without destroying the certificate, as the paper's learning-curve analysis shows.
  • The diffusion generator compresses the upstream model zoo, so certification does not require storing thousands of adapters, and interpolating between upstream adapters improves accuracy over raw model-zoo selection.
  • Increasing the support set beyond the low-shot regime keeps tightening the bound even after the support error saturates: at 64–128 shots on iNaturalist birds, the certified risk is only about 6% above the empirical query error.
  • Compared with the quantized-parameter bound applied to SGD, MeZO, BBPT, or LoRA-Hub, STEEL yields a much higher fraction of non-vacuous episodes and tighter median guarantees while staying within a few points of standard fine-tuning accuracy on most benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper leaves implicit is to quantify how certificate strength depends on the diffusion model's density-estimation error: if $p(\theta)$ is a poor proxy for $p_{\text{true}}(T)$, the support-error term degrades, and measuring that degradation on held-out task families would turn the coverage assumption into a testable quantity.
  • The finite-hypothesis perspective suggests that any generative model over parameters, not just diffusion, could power the same certificate as long as the candidate pool is fixed before seeing the target support set.
  • The bound treats all candidates as independent; exploiting the redundancy or correlation among sampled adapters could reduce the effective size of $\Theta$ and yield tighter certificates, a direction the paper itself flags as future work.
  • Because the diffusion model stores task knowledge in a single generator, the approach could couple with privacy-preserving training of the generator to certify personalized models without releasing per-user adapters; the paper mentions privacy as a broader-impact concern but does not analyze the certificate under such protections.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes STEEL (Sample ThEn Evaluate Learner), a transfer-learning method that trains a diffusion model over PEFT adapter parameters on upstream tasks, then samples a finite set of adapters before seeing the downstream task and selects one by empirical risk on the downstream support set. The central claim is that restricting the hypothesis space to this finite, pre-sampled set yields non-vacuous finite-hypothesis/PAC-Bayes risk certificates in the low-shot regime, unlike standard continuous-parameter fine-tuning. The paper instantiates the bound with FLAN-T5 on three LaMP benchmarks and CLIP+CoOp on four vision benchmarks, reporting that STEEL achieves non-vacuous certificates for a substantial fraction of episodes while retaining accuracy comparable to SGD, MeZO, BBPT, and Meta-PB. An additional iNaturalist experiment with larger support sets shows the certificate tightening under distribution shift between upstream and downstream data.

Significance. If the numerical certificates can be verified, the paper makes a meaningful contribution: it demonstrates that a gradient-free 'sample-then-evaluate' procedure over a task-conditioned finite hypothesis set can produce meaningful generalization guarantees in a low-shot regime where conventional fine-tuning bounds are vacuous. The theoretical derivation is standard and correctly applied: the finite set Θ is sampled before the support set is observed, so the union bound over Θ is valid for the selected adapter. The diffusion-based adapter generator is a natural mechanism for making such a finite set useful, and the paper provides unusually detailed experimental configuration details. The main caveats are the unreported confidence level ε, which affects every number in the paper, and some comparability inconsistencies in the evaluation; these do not invalidate the theoretical argument but currently prevent full verification of the headline claims. The diffusion-coverage assumption identified by the reader is real, but the cross-dataset iNaturalist experiment partially addresses it, and it is best treated as a limitation rather than a correctness error.

major comments (4)
  1. [Sec. 3.4, Eq. (3); Sec. 5.1-5.2, Tables 1-2] The value of the confidence parameter ε in Eq. (3) is never reported. Since log(1/ε) (or log(|Θ|/ε)) appears inside the complexity term, every numerical certificate in Tables 1 and 2, the curves in Figures 1-4, and the non-vacuous ratios depend on a quantity the paper does not disclose. If ε was chosen close to 1, the certificates hold with negligible probability; if a conventional value such as 0.05 or 0.01 was used, entries near the vacuous threshold (e.g., LaMP-3 median bound 0.93 in Table 1 and FGVCAircraft max bound 0.85 in Table 2) could shift by more than their margin to the threshold. The authors should state the exact value of ε used for all methods and recompute all reported certificates with a common, standard confidence level.
  2. [Sec. 5.2, Table 2] The comparison with Meta-PB mixes incomparable risk metrics. The text states that Meta-PB's bound (Appendix A.3, Eq. 23) is on the unbounded cross-entropy loss, while STEEL, SGD, and BBPT are evaluated on bounded 0/1 error; nevertheless Table 2 reports Meta-PB's min/median/max bounds and gap in the same columns and concludes that its certification is 'much worse than STEEL.' A numeric comparison of bounds on different losses is not meaningful. The authors should either remove Meta-PB from the bound comparison in Table 2 or convert the Meta-PB certificate to the same bounded error metric used for the other methods.
  3. [Sec. 5, Fig. 1 and Table 2] The definition of 'non-vacuous' is used inconsistently. Figure 1 shades certificates as vacuous when support error plus complexity is at least 1, while Section 5.1 defines non-vacuous as 'strength above chance-level'; the reported ratios in Table 2 (e.g., FGVCAircraft 97.5% with a max bound of 0.85 for 5-way tasks) are consistent only with a chance-level threshold of 0.8, not with the threshold of 1 used in Figure 1. Since the non-vacuous ratio is the leading certifiability metric, the authors should state a single definition or, if both thresholds are meaningful, define and report them separately.
  4. [Appendix B.2.1, Table 1] The LaMP-5 support error term is not specified as a bounded loss. The text says 'For support errors, we use ... ROUGE-1 for LaMP-5,' but ROUGE-1 is a higher-is-better score, not a loss, and Eq. (3) requires r(θ) ∈ [0,C]. If the implemented quantity is 1 − ROUGE-1, this should be stated explicitly; otherwise the LaMP-5 certificates are not well defined. Similarly, the LaMP-3 metric based on 'absolute error between the labels one-hot encoding and Flan-T5 model logits' should clarify whether the logits are normalized, because raw logits are unbounded and the claim that the metric lies in [0,1] does not follow as written.
minor comments (5)
  1. [Sec. 3.4, Eq. (3)] The complexity term in Eq. (3) is notationally ambiguous; please write it unambiguously as sqrt(log(|Θ|/ε)/(2n)) or the precise intended form.
  2. [Sec. 3.3 and Appendix B.3] The main text should state explicitly that Θ is sampled and fixed before the downstream support set is observed; the validity of Eq. (3) relies on this, and for the vision experiments it is stated only in the appendix.
  3. [Sec. 5.3, Figure 4] Figure 4 and the surrounding text should clarify that the displayed bounds at intermediate sample counts are not used to choose the sample count based on the support set, since the finite-set certificate requires Θ to be fixed before the support is observed.
  4. [Appendix B.2] The sentence 'N is chosen as the minimum Silhouette score' appears to be inverted; silhouette scores should be maximized, not minimized.
  5. [Sec. 5.2 and Appendix C.2] There are minor typos: 'vaccuous' should be 'vacuous' in Section 5.2, and the table header 'FGVVCA' in Appendix C.2 should be 'FGVCAircraft'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the finite-hypothesis certificate is computed from a pre-fixed hypothesis set and measured support error, not fitted to downstream query data.

full rationale

The derivation chain is self-contained and not circular. The central certificate in Eq. (3) is a standard finite-hypothesis PAC-Bayes/union bound instantiated from Alquier's theorem, applied to a hypothesis set Theta sampled from a diffusion model trained only on upstream tasks, before the downstream support set S* is seen. The bound is evaluated from the known, pre-fixed quantities |Theta| and n together with the measured support error r(theta); although the selected theta is chosen after S*, the union bound covers all theta in Theta simultaneously, so the data-dependent selection does not invalidate the certificate. Nothing is fitted to downstream query data and then presented as a prediction: support error is an input, the complexity term is computed from reported hyperparameters (e.g., 10K LoRA-XS samples for LaMP and 20K prompts for vision, fixed before downstream evaluation), and the diffusion model acts as an upstream density estimate rather than a downstream fit. The only self-citation involving the present authors (Ref. [23], cited for general scalability benefits of gradient-free methods) is not load-bearing for any theorem or empirical certificate. The paper's own Appendix D discusses possible future tightness improvements, which further indicates that the certificate is not assumed optimal by construction. The omission of the confidence parameter epsilon in the instantiations is a reproducibility gap, but it does not make the derivation circular: the bound remains a valid high-probability statement for any chosen epsilon, and the numerical values would simply need to be recomputed once epsilon is disclosed.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The ledger is small. The main free parameter is the sample count |Θ|, and the confidence ε is an unreported user choice that directly affects the certificate. The axioms are standard PAC-Bayes background, the i.i.d. task assumption, the representativeness of the diffusion prior, the independence of Θ from S*, and bounded losses. No new physical or theoretical entities are invented.

free parameters (2)
  • Number of diffusion samples |Θ| = 20,000 (vision), 10,000 (LLM)
    Controls the bound's complexity term in Eq. (3) and the coverage of the hypothesis space. Chosen by the authors as a fixed hyperparameter; the learning curve analysis in Section 5.3 shows the certificate is sensitive to this choice.
  • Confidence level ε in Eq. (3) = Not reported
    The certificate value depends on log(1/ε); without a reported ε the numerical bounds in Tables 1-2 cannot be reproduced.
assumptions (5)
  • standard math Finite-hypothesis PAC-Bayes bound (Alquier 2021)
    Used to derive Eq. (3); cited as Sec. 2.1.3 in [1].
  • domain assumption Upstream tasks and downstream task are i.i.d. from a common true task distribution p_true(T).
    Section 3.2 states this setup, inherited from meta-learning and model-zoo literature; needed for the transfer story.
  • ad hoc to paper The diffusion model p(θ) is a good proxy for p_true(T) and its samples include adapters with low loss on the downstream task.
    Section 3.3 assumes 'the estimated p(θ) serves as a proxy for p_true(T)'. If false, support error is high and certificates become vacuous.
  • domain assumption The finite set Θ is sampled before seeing downstream support S* and is independent of S*.
    Required for the finite-hypothesis bound to apply without an additional complexity penalty; the paper follows this procedure.
  • domain assumption Bounded loss l in [0,C].
    Eq. (3) requires a bounded loss with maximum C; used for LaMP and vision metrics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Model Diffusion for Certifiable Few-shot Transfer Learning." pith.science (2026). https://pith.science/paper/FCMFJUMV

@misc{pith2026250206970,
  author       = {Pith},
  title        = {Pith review of: Model Diffusion for Certifiable Few-shot Transfer Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FCMFJUMV}},
  note         = {Machine review of arXiv:2502.06970}
}
read the original abstract

In contemporary deep learning, a prevalent and effective workflow for solving low-data problems is adapting powerful pre-trained foundation models (FMs) to new tasks via parameter-efficient fine-tuning (PEFT). However, while empirically effective, the resulting solutions lack generalisation guarantees to certify their accuracy - which may be required for ethical or legal reasons prior to deployment in high-importance applications. In this paper we develop a novel transfer learning approach that is designed to facilitate non-vacuous learning theoretic generalisation guarantees for downstream tasks, even in the low-shot regime. Specifically, we first use upstream tasks to train a distribution over PEFT parameters. We then learn the downstream task by a sample-and-evaluate procedure -- sampling plausible PEFTs from the trained diffusion model and selecting the one with the highest likelihood on the downstream data. Crucially, this confines our model hypothesis to a finite set of PEFT samples. In contrast to the typical continuous hypothesis spaces of neural network weights, this facilitates tighter risk certificates. We instantiate our bound and show non-trivial generalization guarantees compared to existing learning approaches which lead to vacuous bounds in the low-shot regime.

Figures

Figures reproduced from arXiv: 2502.06970 by the authors.

Figure 1
Figure 1. Generalization bounds for adapting CLIP to novel tasks (5-way classification with 1–16 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Distribution of generalisation guarantees (x-axis, log scale) obtained over few-shot LLM [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Dependence of generalisation guarantee on training set size. Our finite-hypothesis class [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: “Learning curves" illustrating empirical and certified learning dynamics of STEEL with [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Support/query error and certified risk vs. sup￾port set size on iNaturalist birds. As the number of shots increases, support and query errors converge, while the bound continues to tighten. The gap between the certified risk and empirical query error drops to 6% at 128…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 38 canonical work pages

  1. [1]

    P. Alquier. User-friendly introduction to PAC-Bayes bounds. arXiv preprint arXiv:2110.11216, 2021

    arXiv 2021

  2. [2]

    S. M. R. Arnold, P. Mahajan, D. Datta, I. Bunner, and K. S. Zarkias. learn2learn: A library for meta-learning research, 2020

    work page 2020

  3. [3]

    Bałazy, M

    K. Bałazy, M. Banaei, K. Aberer, and J. Tabor. LoRA-XS: Low-rank adaptation with extremely small number of parameters, 2024

    work page 2024

  4. [4]

    H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y . Tay, W. Fedus, Y . Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. S. Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, A. Castro-Ros, M. Pellat, K. Robinson, D. Valter, S. Narang, G. Mishra, A. Yu, V . Y . Zhao, Y . Huang, A. M. Dai, H. Yu, S. Petrov, E. H. Chi, J. Dean, J. Devlin, A. Roberts, D. Zhou, Q. V . ...

    work page 2024

  5. [5]

    Cimpoi, S

    M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014

    2014

  6. [6]

    Y . Du, Z. Xiao, S. Liao, and C. Snoek. ProtoDiff: Learning to Learn Prototypical Networks by Task-Guided Diffusion. In Advances in Neural Information Processing Systems, 2023

    work page 2023

  7. [7]

    G. K. Dziugaite and D. M. Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. UAI, 2017

    work page 2017

  8. [8]

    J. Ho, A. Jain, and P. Abbeel. Denoising Diffusion Probabilistic Models, 2020. In Advances in Neural Information Processing Systems

    work page 2020

Show all 45 references
  1. [9]

    G. V . Horn, O. M. Aodha, Y . Song, Y . Cui, C. Sun, A. Shepard, H. Adam, P. Perona, and S. Belongie. The inaturalist species classification and detection dataset, 2018

    2018

  2. [10]

    T. M. Hospedales, A. Antoniou, P. Micaelli, and A. J. Storkey. Meta-Learning in Neural Networks: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021

    2021

  3. [11]

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  4. [12]

    Huang, Q

    C. Huang, Q. Liu, B. Y . Lin, T. Pang, C. Du, and M. Lin. Lorahub: Efficient cross-task generalization via dynamic loRA composition. In First Conference on Language Modeling, 2024

    2024

  5. [13]

    D. J. Kopiczko, T. Blankevoort, and Y . M. Asano. VeRA: Vector-based Random Matrix Adaptation. In International Conference on Learning Representations, 2024

    2024

  6. [14]

    Langford

    J. Langford. Tutorial on practical prediction theory for classification. Journal of Machine Learning Research, 2005

    2005

  7. [15]

    Lotfi, M

    S. Lotfi, M. A. Finzi, Y . Kuang, T. G. J. Rudner, M. Goldblum, and A. G. Wilson. Non-vacuous generalization bounds for large language models. In Forty-first International Conference on Machine Learning, 2024

    2024

  8. [16]

    S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013

    2013

  9. [17]

    Malladi, T

    S. Malladi, T. Gao, E. Nichani, A. Damian, J. D. Lee, D. Chen, and S. Arora. Fine-tuning language models with just forward passes. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  10. [18]

    Miranda, E

    M. Miranda, E. S. Ruzzetti, A. Santilli, F. M. Zanzotto, S. Bratières, and E. Rodolà. Preserving privacy in large language models: A survey on current threats and solutions, 2024

    2024

  11. [19]

    Mohri, A

    M. Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of Machine Learning. MIT Press, 2018. 10

    2018

  12. [20]

    Nilsback and A

    M.-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, 2008

    2008

  13. [21]

    Perez-Ortiz, O

    M. Perez-Ortiz, O. Rivasplata, J. Shawe-Taylor, and C. Szepesvári. Tighter risk certificates for neural networks. Journal of Machine Learning Research, 2021

    2021

  14. [22]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision, 2021

    2021

  15. [23]

    F. Rezk, A. Antoniou, H. Gouk, and T. Hospedales. Liouna: Biologically plausible learn- ing for efficient pre-training of transferrable deep models. In 2nd Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ICM...

    2024

  16. [24]

    Salemi, S

    A. Salemi, S. Mysore, M. Bendersky, and H. Zamani. Lamp: When large language models meet personalization, 2024

    2024

  17. [25]

    Schürholt, M

    K. Schürholt, M. W. Mahoney, and D. Borth. Towards scalable and versatile weight space learning, 2024

    2024

  18. [26]

    Shalev-Shwartz and S

    S. Shalev-Shwartz and S. Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014

    2014

  19. [27]

    L. N. Smith and N. Topin. Super-convergence: Very fast training of neural networks using large learning rates, 2018

    2018

  20. [28]

    B. Soro, B. Andreis, H. Lee, W. Jeong, S. Chong, F. Hutter, and S. J. Hwang. Diffusion-based neural network weights generation. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  21. [29]

    Staab, M

    R. Staab, M. Vero, M. Balunovic, and M. Vechev. Beyond memorization: Violating privacy via inference with large language models. In The Twelfth International Conference on Learning Representations, 2024

    2024

  22. [30]

    Z. Tan, Z. Liu, and M. Jiang. Personalized pieces: Efficient personalized large language models through collaborative efforts. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024

    2024

  23. [31]

    Triantafillou, T

    E. Triantafillou, T. Zhu, V . Dumoulin, P. Lamblin, U. Evci, K. Xu, R. Goroshin, C. Gelada, K. Swersky, P.-A. Manzagol, and H. Larochelle. Meta-dataset: A dataset of datasets for learning to learn from few examples. In International Conference on Learning Representations, 2020

    2020

  24. [32]

    V . N. Vapnik.The nature of statistical learning theory. Springer-Verlag New York, Inc., 1995

    1995

  25. [33]

    R. A. Waelen. The ethics of computer vision: an overview in terms of power. AI and Ethics, 2023

    2023

  26. [34]

    C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. Cubirds dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011

    2011

  27. [35]

    K. Wang, D. Tang, B. Zeng, Y . Yin, Z. Xu, Y . Zhou, Z. Zang, T. Darrell, Z. Liu, and Y . You. Neural Network Diffusion. arXiv preprint arXiv:2402.13144, 2024

    2024 arXiv

  28. [36]

    Y . Wang, Q. Yao, J. T. Kwok, and L. M. Ni. Generalizing from a few examples: A survey on few-shot learning. ACM Comput. Surv., 2020

    2020

  29. [37]

    Y . You, J. Li, S. Reddi, J. Hseu, S. Kumar, S. Bhojanapalli, X. Song, J. Demmel, K. Keutzer, and C.-J. Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes, 2020

    2020

  30. [38]

    L. Yu, Q. Chen, J. Lin, and L. He. Black-box prompt tuning for vision-language model as a service. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23. International Joint Conferences on Artificial Intelligence Organization, 2023. 11

    2023

  31. [39]

    Zakerinia, A

    H. Zakerinia, A. Behjati, and C. H. Lampert. More Flexible PAC-Bayesian Meta-Learning by Learning Learning Algorithms. In International Conference on Machine Learning, 2024

    2024

  32. [40]

    Zhang, C

    B. Zhang, C. Luo, D. Yu, H. Lin, X. Li, Y . Ye, and B. Zhang. MetaDiff: Meta-Learning with Conditional Diffusion for Few-Shot Learning. 2024

    2024

  33. [41]

    K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Learning to prompt for vision-language models. International Journal of Computer Vision (IJCV), 2022. 12 A Existing Risk Bounds for Deep Models A.1 Vanilla PAC-Bayes Bound This is the vanilla, non-transfer learning bound. As a baseline,...

    2022

  34. [42]

    The paper is about LLM pre-training setup with large training data, and the bound would be vacuous if training data size is not large enough (e.g., ≥ 10K)

  35. [43]

    They derive the same finite hypothesis space PAC-Bayes bound, but replace thelog |H| term by log(1/p(h)) where p(h) is the prior likelihood, and log(1/p(h)) is approximated and upper-bounded by C(h) which is the number of bits for representing the hypothesis h

  36. [44]

    The finite hypothesis space comes from the fixed-size floating point representation for real numbers (e.g., if there ared trainable parameters, then C(h) = d·32), but to reduce it further, they propose what is called the SubLoRA, which is a random subspace representation (i.e....

  37. [45]

    1 N NX i=1 l(A(Si); Si) # (14) ˆE(ρ) = EA∼ρ

    Also, instead of 32 bit for each of d params, they do some clustering to reduce it to shorter coding, more precisely the arithmetic coding. The followings are some details of their bound derivation. With probability at least 1 − ϵ, R(θ) ≤ r(θ) + C · r K(θ) + 2 logK(θ) + log(1/...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.