Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

Do we really have to filter out random noise in pre-training data for language models?

T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Random noise in pre-training data raises next-token loss far less than its share of the corpus, the paper argues, and a local-gradient-matching loss can protect downstream accuracy.

desk verdict The paper's core observation — NTP loss is surprisingly robust to uniform random token noise — is new and probably right in direction, but the headline 1% number is confounded by a token-budget mismatch and the theory leans on post-hoc fitted parameters. read the letter →

arxiv 2502.06604 v2 pith:PLWMQ7BS submitted 2025-02-10 cs.CL

Jinghan Ru , Yuxin Xie , Xianwei Zhuang , Yuguo Yin , Zhihui Guo , Zhiming Liu , Qianli Ren , Yuexian Zou This is my paper · ORCID
classification cs.CL
keywords randomnoisepre-trainingdatanext-tokenpredictionlosslanguagemodelsqualityrobustnesslocalgradientmatchingblack-boxfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether random noise in pre-training corpora—garbage token sequences from decoding errors or unmaintained web pages—really needs aggressive filtering. Training GPT-2 models on a web-scale English corpus mixed with 1%, 5%, and 20% uniformly random tokens, it finds that the clean-validation next-token loss rises only slightly: about 1% at 20% noise, and still small at 2.7B scale with 5% noise. It proposes a theory, built on a disjoint-support assumption between clean and noise distributions, predicting that a model optimizing the mixed distribution is barely affected by random noise when a tradeoff parameter $k$ is large. The paper then shows that downstream-task accuracy can still drop even when the pre-training loss moves little, and introduces a black-box fine-tuning loss, Local Gradient Matching, that aligns gradients on normal and perturbed features to recover and even improve downstream performance across language and vision benchmarks. A sympathetic reader would take away that random noise is not the main enemy of pre-training loss, but downstream robustness is a separate question.

What carries the argument

The load-bearing object is Proposition 1's tradeoff parameter $k$, defined through the model's losses on clean and noise distributions: after training on $\alpha$-noise data the clean loss becomes $-\log(p_c - \epsilon)$ and the noise loss $-\log(p_n + \epsilon/k)$, with $k$ shown to be $\Omega(e^{L_{ntp}(P^n,h)})$. Because random noise is nearly unlearnable, $p_n$ is tiny and $k$ is large, so the excess loss $\epsilon$ on clean data is squeezed to order $\alpha p_c - (1-\alpha) k p_n$. The companion mechanism is the Local Gradient Matching loss, which adds Gaussian perturbation to features and penalizes $\|E\nabla_\theta \ell(g_\theta(t),y) - E\nabla_\theta \ell(g_\theta(\hat{t}),y)\|^2$; Proposition 2 bounds it by smoothness, input flatness, and the cross-entropy term, connecting the method to flat-minima-style generalization.

What would settle it

A concrete falsifier: pre-train a 124M GPT-2 on a clean corpus with 50% uniform random tokens and measure the clean-validation NTP loss against a clean-trained model; Proposition 1 predicts the excess $\epsilon$ remains small when $k$ is large, so if the loss gap grows roughly linearly in $\alpha$ or if a measured $k$ from checkpoints is near 1, the theory's parameterization fails. A cheaper check is to train with 20% Gaussian noise and compare the excess loss to the predicted order $\alpha p_c - (1-\alpha) k p_n$ using $k$ measured from the checkpoints rather than inferred after the fact.

Watch

Extended reading notes

Core claim

The central discovery is that random noise with support disjoint from clean text is almost invisible to next-token prediction at the population level. Concretely, with the mixed distribution $P^m = \alpha P^n + (1-\alpha)P^c$, the paper's Proposition 1 states that when $\alpha$ is small or the noise tradeoff parameter $k$ is large, the global minimizer of the mixed-distribution NTP loss achieves a clean-distribution loss close to the clean-trained model's loss; the noise-induced excess $\epsilon$ is bounded on the order of $\alpha p_c - (1-\alpha) k p_n$. Empirically, 20% uniform random tokens raise the clean validation loss by about 1% at 124M scale, and 5% random noise at 2.7B behaves similarly. The paper also establishes that a lower NTP loss does not guarantee downstream accuracy: Gaussian-noise-trained models can lose 1.5% accuracy, and its Local Gradient Matching loss, which minimizes the difference between gradients on original and noise-perturbed features, consistently improves downstream classification across 8 language and 14 vision benchmarks.

Load-bearing premise

The argument rests on assuming that the effect of noise on the model's probabilities can be captured by a single tradeoff parameter $k$ that is large for random noise and is measured after training rather than predicted; if $k$ is not large for real-world noise, the conclusion that random noise barely affects pre-training loss does not follow.

Editorial extensions

If this is right

  • Random noise up to 20% of tokens can be left in pre-training data without a large NTP-loss penalty, so filtering budgets may be better spent on structured low-quality or synthetic data.
  • Measured NTP loss is not a reliable proxy for downstream task quality when noise is present; a model with lower clean loss can still be 1.5% worse on downstream accuracy.
  • The same disjoint-support analysis explains why multilingual models can train on several languages without one language's data destroying another's loss, and extends to audio/text and visual autoregressive models.
  • Adding the Local Gradient Matching loss to a linear or MLP head improves downstream accuracy on clean pre-trained models too, not only on noisy ones.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the disjoint-support reasoning transfers to real web noise, data curators could rank noise types by learnability rather than by annoyance—hard-to-learn garbage matters less for loss, while learnable distractor distributions matter more.
  • Editorial inference: a direct test the paper leaves open is whether $k$ stays large when noise is adversarial rather than random, such as tokens sampled from a different natural language or from a model's own output; Proposition 1 suggests such noise could behave more like Gaussian noise and hurt more.
  • Editorial inference: the LGM loss's success on clean backbones hints that its benefit is not denoising per se but input-sensitivity regularization; one could test this by comparing LGM to sharpness-aware minimization on the same probes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies the effect of 'random noise' (uniform or Gaussian-distributed random token sequences) in pre-training data on language models. Using GPT-2 124M and 2.7B models trained on OpenWebText with 1%, 5%, and 20% noise, it reports that NTP loss on clean validation increases by only about 1% at 20% noise, much less than the noise proportion. The paper proposes a theoretical explanation (Proposition 1) based on a Huber contamination model with disjoint clean/noise supports and a parameter k capturing how much the model's noise-loss improves relative to clean-loss degradation. It then claims the theory explains multilingual and multimodal pretraining. A separate contribution is a Local Gradient Matching (LGM) loss for black-box fine-tuning of downstream heads, with experiments on 8 language and 14 vision benchmarks.

Significance. The empirical question is timely and the study is one of the first systematic investigations of random (token-level) noise as opposed to low-quality or synthetic data. The release of code, data, and checkpoints enables reproduction. The one genuinely predictive qualitative result—Gaussian noise (higher pn) leading to a smaller clean-validation loss gap than uniform random noise—is a useful falsifiable claim. The LGM method shows consistent gains across many backbones and datasets. However, the headline quantitative claim currently rests on an uncontrolled comparison, and the theory's key parameter k is fitted post hoc; both issues must be addressed before the main conclusions can be accepted.

major comments (3)
  1. [Section 3.3, Figure 2] The headline claim that 20% random noise increases clean validation loss by only about 1% compares models trained for the same number of iterations (300k), so the noisy model has seen only 80% (or 95%/99%) of the clean tokens seen by the clean model. The discrepancy is therefore not attributable to noise alone. Figure 2(c) is labeled 'Calibrated' but the calibration procedure is not specified in the main text or appendices; the caption's phrase 'same number of training iterations on clean OpenWebText data' admits at least two readings (continuing training on clean data vs. matching clean-token counts). A clean-model baseline trained to the same number of clean tokens as each noisy model is needed to support the quantitative claim.
  2. [Sections 3.2.1–3.2.2 and Appendix B.2, Lemma 1] The proof that Lntp(Pm,h)=αLntp(Pn,h)+(1−α)Lntp(Pc,h) relies on supp(Pc)∩supp(Pn)=∅. But the uniform random noise defined by Equation (3) assigns positive probability to every token sequence of length L, so its support is the entire space X×W; any nontrivial clean distribution overlaps with it. The assumption can be salvaged by interpreting Pc and Pn as empirical distributions on the finite training corpora, but the paper does not state this, and Proposition 1 is formulated for the population distributions. This is a load-bearing gap in the theoretical justification for the main experiments.
  3. [Section 3.4, Appendix B.3, Section F] The parameterization Lntp(Pc,h)=−log(pc−ϵ), Lntp(Pn,h)=−log(pn+ϵ/k) is asserted rather than derived, and the key quantity k is measured post hoc from the trained models (Figure 6(a)). The conclusion that k is large for random noise therefore explains the observed small loss increase only by fitting the free parameter to the same experiments it is supposed to explain. The paper should either derive k from a model of the training dynamics/architecture, or explicitly reframe Proposition 1 as a descriptive account with k as an empirical quantity, and report how robust the conclusions are to the measured k. The paper's own Limitations (Section F) disclaim dynamical analysis, so the global-minimum result cannot by itself justify the finite-token SGD comparison used in the experiments.
minor comments (6)
  1. [Section 3.3, Figure 3(a)] The abstract states the finding holds 'even when the model was scaled up to 2.7B,' but the 2.7B experiment uses 5% noise only; please qualify the claim or add a 20% run at this scale.
  2. [Section 3.5] The claim that multilingual models succeed because pc≈pn 'leading to ϵ≈0' is not derived; please state the additional assumptions under which equal losses imply small ϵ in Proposition 1.
  3. [Section 4.2, Proposition 2] The bound in Equation (7) is a valid but loose upper bound; the text should explain more explicitly how minimizing Lgm is expected to reduce generalization error (e.g., via Rρ), rather than treating the bound itself as the explanation.
  4. [Equation (6) and surrounding text] The distribution \hat D is never formally defined; please specify the joint distribution of (t + γδ, y) and the independence assumption on δ.
  5. [Throughout] There are several typos and inconsistencies: 'argut' in Section 3.2.2, 'landsape' in Appendix C, 'Mathcing' in the Figure 4 caption, and inconsistent capitalization of Lgm/LGM.
  6. [References] Reference [124] duplicates [118]; please merge or renumber.

Circularity Check

1 steps flagged · score 5.0 of 10

The headline 1% loss-increase claim is an honest measurement, but the Proposition 1 'why' is partially circular: the tradeoff parameter k is measured from the same noisy checkpoints whose small clean-loss increase it is then used to explain; the one non-circular prediction is the Gaussian ordering.

  1. self definitional [Section 3.4, Proposition 1, and Appendix B.3 (Figure 6(a))]
    "When the model h is trained on a mixed distribution P m which includes noise, it attempts to fit P n, leading to an increase in the loss on the clean distribution P c, such that Lntp(P c, h) = − log(pc − ϵ) and Lntp(P n, h) = − log(pn + ϵ/k) for some ϵ > 0 ... We plot the trend of k in Figure 6(a). We compare checkpoints trained for the same iterations on both P c and P m, where pc is calculated based on the loss of the model trained on P c, and pn is determined by the loss of a model trained for 10,000 iterations on P m when evaluated on P n."

    The parameter k is introduced in Proposition 1 as the ratio connecting the clean-loss increase ϵ to the noise-loss improvement: by construction k = ϵ / (e^{-Lntp(P_n,h)} - pn). Appendix B.3 then computes k from the very P_m-trained checkpoints whose clean-loss behavior is the phenomenon under study, and Section 3.4 concludes 'k is much greater than 1, this implies ϵ is much smaller than αpc'. That conclusion restates the measured losses in new notation rather than providing an independent bound on ϵ; the proposition only rearranges Lemma 1's exact decomposition. The one genuinely non-tautological check is the Gaussian prediction (higher pn implies smaller clean-loss increase), and even that is qualitative.

full rationale

Section 3.3's headline measurements are self-contained: clean and noisy GPT-2 models are trained and evaluated on held-out clean OpenWebText, and the 2.7B scaling check is likewise a direct measurement. The LGM experiments in Section 4 are benchmark comparisons with an independent regularizer, and Proposition 2 is an upper bound rather than a derivation of the observed improvements. Self-citations such as [14, 15, 31] are illustrative examples, not load-bearing evidence. The circularity is confined to the 'Why' story: Proposition 1 parameterizes any model by its clean/noise losses and defines k through the same ϵ that it later 'bounds'. Measuring k from checkpoints of the very models whose small loss increase is to be explained (Appendix B.3) turns the central explanation into a relabeling of the observation. The Gaussian comparison is a real qualitative prediction and prevents the paper from being fully circular, so the empirical 'What' and 'How' contributions stand independently. Section F explicitly disclaims dynamical analysis, reinforcing that the k-based explanation is a post-hoc fit of a global-minimum result to SGD trajectories. The under-specified 'calibrated' panel in Figure 2(c) is a correctness and verifiability gap rather than a circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on Assumption 1 (disjoint supports), the ad hoc epsilon/k tradeoff, and the fitted parameter k; these are measured from the same models whose behavior is explained, which raises the circularity burden. For the LGM method, the free hyperparameters gamma and lambda are tuned per dataset, and the theory only gives a loose upper bound.

free parameters (4)
  • k (probability shift ratio) = measured from trained models, e.g., around 300 for 5% random noise (Figure 6(a))
    Used in Proposition 1 to bound the effect of noise. The condition for negligible noise depends on k being large, and the paper measures k from the losses of the very models the theory explains.
  • gamma (LGM perturbation scale) = 0.01 in Table 1; grid searched in Table 6 (e.g., 0.001, 0.05, 0.1)
    Controls the magnitude of noise added to features in LGM; chosen by hand and tuned per dataset, not derived.
  • lambda (LGM regularization weight) = 0.15 in Table 1; grid searched in Table 6 (0.05, 0.1, 0.15)
    Balances Lgm against cross-entropy; tuned per dataset.
  • sigma (Gaussian noise standard deviation) = 500 in experiments
    Used to generate the 'learnable' Gaussian noise; a design choice for the noise model.
assumptions (4)
  • domain assumption Assumption 1: supp(P_c) ∩ supp(P_n) = ∅ (clean and noise data have disjoint support)
    Formal definition of random noise in Section 3.2.1; load-bearing for Lemma 1 and Proposition 1.
  • ad hoc to paper The epsilon/k parameterization: fitting noise reduces clean probability by epsilon and increases noise probability by epsilon/k, for some epsilon > 0 with k > 1
    Stated in Proposition 1 (Section 3.4) with no derivation from the transformer architecture; it is the core mechanism that makes noise 'cheap' to ignore.
  • domain assumption The model minimizes population loss to a global minimum; no analysis of SGD dynamics
    The theoretical results concern Lntp(P_m, h) for arbitrary h, and the claims about 'the global minimum on P_m' require the training procedure to reach it; the authors note in Limitations F that dynamics are not analyzed.
  • standard math Beta-smoothness and rho-input flatness of the downstream loss
    Used to prove Proposition 2 in Appendix B.4; standard regularity assumptions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do we really have to filter out random noise in pre-training data for language models?." pith.science (2026). https://pith.science/paper/PLWMQ7BS

@misc{pith2026250206604,
  author       = {Pith},
  title        = {Pith review of: Do we really have to filter out random noise in pre-training data for language models?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PLWMQ7BS}},
  note         = {Machine review of arXiv:2502.06604}
}
read the original abstract

Web-scale pre-training datasets are the cornerstone of LLMs' success. However, text data curated from the Internet inevitably contains random noise caused by decoding errors or unregulated web content. In contrast to previous works that focus on low quality or synthetic data, our study \textbf{provides the first systematic investigation of such random noise through a cohesive ``What-Why-How'' framework.} Surprisingly, we observed that the resulting increase in the loss of next-token prediction (NTP) was significantly lower than the proportion of random noise even when the model was scaled up to 2.7B. We provide a theoretical justification for this phenomenon, which also elucidates the success of multilingual models and can be applied to multimodal models. On the other hand, experiments show that the model's performance in downstream tasks is not based solely on the NTP loss, which means that random noise may result in degraded downstream performance. To address the potential adverse effects, we introduce a novel plug-and-play Local Gradient Matching loss, which explicitly enhances the denoising capability of the downstream task head by aligning the gradient of normal and perturbed features without requiring knowledge of the model's parameters. Additional experiments on 8 language and 14 vision benchmarks further validate its effectiveness.

Figures

Figures reproduced from arXiv: 2502.06604 by the authors.

Figure 1
Figure 1. Overview of the study and methodol￾ogy. (a) The common scenario in which a GPT model, pre-trained on filtered data P c , demon￾strates robust performance. (b) When the pre￾training dataset is contaminated with random noise P n, the resultant language model may exhibit un￾predictable behavior. (c) Our approach follows [17] and focuses on the effective fine-tuning of black-box noisy models for downstream tasks P d . O… view at source ↗
Figure 2
Figure 2. Next-token prediction loss on the clean OpenWebText validation set for GPT-2 124M [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Validation experiments with the models trained with 5% random noise. (a) Loss trends on [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Overview of the proposed Local Gradient Mathcing scheme. Let ℓ(ˆy, y) be the loss function(typically cross￾entropy), and Lce(D, gθ) = E(t,y)∼Dℓ(gθ(t), y) be the population-level loss where D represents the joint distribution of downstream features and labels. Due to th…
Figure 6
Figure 6. Figure 6: Visualization of k and Lntp(P m, h∗ ) − Lntp(P m, h). (a) The trend of k as it changes with training, plotted using the model trained on P c as h ∗ . (b) Visualization of Lntp(P m, h) when the parameter settings are consistent with the experiment. Following the same li…
Figure 7
Figure 7. Figure 7: Loss and its difference across different types and levels of noise within the ArXiv and [PITH_FULL_IMAGE:figures/full_fig_p027_7.png]
Figure 8
Figure 8. Figure 8: Visualization of (a) Lntp(P m, h∗ )−Lntp(P m, h) and (b) Lntp(P c , h) with α = 0.55, k = 1,Lntp(P c , h∗ ) = 2.9,Lntp(P n, h∗ ) = 2.8. observe that the performance of models subjected to Gaussian noise varies across different datasets. These observations warrant furth…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A dual-level attention intervention that boosts salient visual-token attention and suppresses text/system attention during decoding reduces hallucination rates in LLaVA, MiniGPT-4, and mPLUG-Owl2 on POPE and CHAIR.

  2. HAD: Hybrid Architecture Distillation Outperforms Teacher in Genomic Sequence Modeling

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A compact hybrid GDN+attention model distilled from Nucleotide Transformer v2 outperforms similarly sized models and, on several tasks, its 500x larger teacher.

Reference graph

Works this paper leans on

138 extracted references · 47 canonical work pages · cited by 2 Pith papers

  1. [1]

    A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity,

    S. Longpre, G. Yauney, E. Reif, K. Lee, A. Roberts, B. Zoph, D. Zhou, J. Wei, K. Robinson, D. Mimno, and D. Ippolito, “A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity,” inProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human La...

    2024

  2. [2]

    What’s in my big data?

    Y . Elazar, A. Bhagia, I. H. Magnusson, A. Ravichander, D. Schwenk, A. Suhr, E. P. Walsh, D. Groeneveld, L. Soldaini, S. Singh, H. Hajishirzi, N. A. Smith, and J. Dodge, “What’s in my big data?” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=RvfPnOkPV4

    2024

  3. [3]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azharet al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  4. [4]

    Physics of language models: Part 3.1, knowledge storage and extraction,

    Z. Allen-Zhu and Y . Li, “Physics of language models: Part 3.1, knowledge storage and extraction,” in Forty-first International Conference on Machine Learning , 2024. [Online]. Available: https://openreview.net/forum?id=5x788rqbcj

    2024

  5. [5]

    Data selection for language models via importance resampling,

    S. M. Xie, S. Santurkar, T. Ma, and P. S. Liang, “Data selection for language models via importance resampling,” in Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. 34 201–34 227. [Online]. Available: https://proceedings.neurips.cc/ pa...

    2023

  6. [6]

    Ai models collapse when trained on recursively generated data,

    I. Shumailov, Z. Shumaylov, Y . Zhao, N. Papernot, R. Anderson, and Y . Gal, “Ai models collapse when trained on recursively generated data,” Nature, vol. 631, no. 8022, pp. 755–759, 2024

    2024

  7. [7]

    How bad is training on synthetic data? a statistical analysis of language model collapse,

    M. E. A. Seddik, S.-W. Chen, S. Hayou, P. Youssef, and M. A. DEBBAH, “How bad is training on synthetic data? a statistical analysis of language model collapse,” in First Conference on Language Modeling, 2024. [Online]. Available: https://openreview.net/forum?id=t3z6UlV09o

    2024

  8. [8]

    Leveraging Web-Crawled Data for High-Quality Fine-Tuning

    J. Zhou, C. Jiang, W. Shen, X. Zhou, and X. He, “Leveraging web-crawled data for high-quality fine-tuning,” arXiv preprint arXiv:2408.08003, 2024

    work page Pith review arXiv 2024

Show all 138 references
  1. [9]

    Wavlm: Large-scale self-supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao et al., “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  2. [10]

    Noise-aware learning from web-crawled image-text data for image captioning,

    W. Kang, J. Mun, S. Lee, and B. Roh, “Noise-aware learning from web-crawled image-text data for image captioning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 2942–2952

    2023

  3. [11]

    A survey on data selection for language models,

    A. Albalak, Y . Elazar, S. M. Xie, S. Longpre, N. Lambert, X. Wang, N. Muennighoff, B. Hou, L. Pan, H. Jeong, C. Raffel, S. Chang, T. Hashimoto, and W. Y . Wang, “A survey on data selection for language models,” Transactions on Machine Learning Research, 2024, survey Certifica...

    2024

  4. [12]

    Dolma: an open corpus of three trillion tokens for language model pretraining research,

    L. Soldaini, R. Kinney, A. Bhagia, D. Schwenk, D. Atkinson, R. Authur, B. Bogin, K. Chandu, J. Dumas, Y . Elazar, V . Hofmann, A. Jha, S. Kumar, L. Lucy, X. Lyu, N. Lambert, I. Magnusson, J. Morrison, N. Muennighoff, A. Naik, C. Nam, M. Peters, A. Ravichander, K. Richardson, Z...

    2024

  5. [13]

    Openwebtext corpus,

    A. Gokaslan, V . Cohen, E. Pavlick, and S. Tellex, “Openwebtext corpus,” 2019

    2019

  6. [14]

    Atri: Mitigating multilingual audio text retrieval inconsistencies by reducing data distribution errors,

    Y . Yin, Y . Xie, W. Yang, D. Yang, J. Ru, X. Zhuang, L. Liang, and Y . Zou, “Atri: Mitigating multilingual audio text retrieval inconsistencies by reducing data distribution errors,”arXiv preprint arXiv:2502.14627, 2025

    2025 arXiv

  7. [15]

    Vargpt: Unified understanding and generation in a visual autoregressive multimodal large language model,

    X. Zhuang, Y . Xie, Y . Deng, L. Liang, J. Ru, Y . Yin, and Y . Zou, “Vargpt: Unified understanding and generation in a visual autoregressive multimodal large language model,”arXiv preprint arXiv:2501.12327, 2025

    2025 arXiv

  8. [16]

    Uniaudio: An audio foundation model toward universal audio generation,

    D. Yang, J. Tian, X. Tan, R. Huang, S. Liu, X. Chang, J. Shi, S. Zhao, J. Bian, X. Wuet al., “Uniaudio: An audio foundation model toward universal audio generation,” arXiv preprint arXiv:2310.00704, 2023

    2023 arXiv

  9. [17]

    Understanding and mitigating the label noise in pre-training on downstream tasks,

    H. Chen, J. Wang, A. Shah, R. Tao, H. Wei, X. Xie, M. Sugiyama, and B. Raj, “Understanding and mitigating the label noise in pre-training on downstream tasks,” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https://openreview.n...

    2024

  10. [18]

    Is out-of-distribution detection learnable?

    Z. Fang, Y . Li, J. Lu, J. Dong, B. Han, and F. Liu, “Is out-of-distribution detection learnable?” in Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 37 199–...

    2022

  11. [19]

    Defweb: Defending user privacy against cache-based website fingerprinting attacks with intelligent noise injection,

    S. Seonghun, D. Debopriya Roy, and G. Berk, “Defweb: Defending user privacy against cache-based website fingerprinting attacks with intelligent noise injection,” in Proceedings of the 39th Annual Computer Security Applications Conference, 2023, pp. 379–393

    2023

  12. [20]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskeveret al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019

    2019

  13. [21]

    The pile: An 800gb dataset of diverse text for language modeling,

    L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima et al., “The pile: An 800gb dataset of diverse text for language modeling,”arXiv preprint arXiv:2101.00027, 2020

    2020 arXiv

  14. [22]

    Shalev-Shwartz and S

    S. Shalev-Shwartz and S. Ben-David, Understanding machine learning: From theory to algorithms. Cambridge university press, 2014

    2014

  15. [23]

    A theory of learning from different domains,

    S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan, “A theory of learning from different domains,” Machine learning, vol. 79, pp. 151–175, 2010

    2010

  16. [24]

    A mathematical exploration of why language models help solve downstream tasks,

    N. Saunshi, S. Malladi, and S. Arora, “A mathematical exploration of why language models help solve downstream tasks,” in International Conference on Learning Representations, 2021. [Online]. Available: https://openreview.net/forum?id=vVjIW3sEc1s

    2021

  17. [25]

    Why do pretrained language models help in downstream tasks? an analysis of head and prompt tuning,

    C. Wei, S. M. Xie, and T. Ma, “Why do pretrained language models help in downstream tasks? an analysis of head and prompt tuning,” in Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y . Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34. Curran ...

    2021

  18. [26]

    Same pre-training loss, better downstream: Implicit bias matters for language models,

    H. Liu, S. M. Xie, Z. Li, and T. Ma, “Same pre-training loss, better downstream: Implicit bias matters for language models,” in Proceedings of the 40th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, A. Krause, E. Brunskill, K. Cho,...

    2023

  19. [27]

    Revisiting discriminative vs. generative classifiers: Theory and implications,

    C. Zheng, G. Wu, F. Bao, Y . Cao, C. Li, and J. Zhu, “Revisiting discriminative vs. generative classifiers: Theory and implications,” in Proceedings of the 40th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, A. Krause, E. Brunskill...

    2023

  20. [28]

    How multilingual is multilingual BERT?

    T. Pires, E. Schlinger, and D. Garrette, “How multilingual is multilingual BERT?” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , A. Korhonen, D. Traum, and L. Màrquez, Eds. Florence, Italy: Association for Computational Linguistics...

    2019

  21. [29]

    Finding universal grammatical relations in multilingual BERT,

    E. A. Chi, J. Hewitt, and C. D. Manning, “Finding universal grammatical relations in multilingual BERT,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, Eds. Online: Association fo...

    2020

  22. [30]

    Zeronlg: Aligning and autoencoding domains for zero-shot multimodal and multilingual natural language generation,

    B. Yang, F. Liu, Y . Zou, X. Wu, Y . Wang, and D. A. Clifton, “Zeronlg: Aligning and autoencoding domains for zero-shot multimodal and multilingual natural language generation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 8, pp. 5712–5724, 2024

    2024

  23. [31]

    Vargpt-v1. 1: Improve visual autoregressive large unified model via iterative instruction tuning and reinforcement learning,

    X. Zhuang, Y . Xie, Y . Deng, D. Yang, L. Liang, J. Ru, Y . Yin, and Y . Zou, “Vargpt-v1. 1: Improve visual autoregressive large unified model via iterative instruction tuning and reinforcement learning,” arXiv preprint arXiv:2504.02949, 2025

    2025 arXiv

  24. [32]

    Almtokenizer: A low-bitrate and semantic-rich audio codec tokenizer for audio language modeling,

    D. Yang, S. Liu, H. Guo, J. Zhao, Y . Wang, H. Wang, Z. Ju, X. Liu, X. Chen, X. Tanet al., “Almtokenizer: A low-bitrate and semantic-rich audio codec tokenizer for audio language modeling,” arXiv preprint arXiv:2504.10344, 2025

    2025 arXiv

  25. [33]

    Stochastic collapse: How gradient noise attracts SGD dynamics towards simpler subnetworks,

    F. Chen, D. Kunin, A. Yamamura, and S. Ganguli, “Stochastic collapse: How gradient noise attracts SGD dynamics towards simpler subnetworks,” in Thirty-seventh Conference on Neural Information Processing Systems , 2023. [Online]. Available: https://openreview.net/forum?id=iFxWrxDekd

    2023

  26. [34]

    Positive-negative momentum: Manipulating stochastic gradient noise to improve generalization,

    Z. Xie, L. Yuan, Z. Zhu, and M. Sugiyama, “Positive-negative momentum: Manipulating stochastic gradient noise to improve generalization,” in Proceedings of the 38th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, M. Meila and T. Zha...

    2021

  27. [35]

    Noise stability regularization for improving BERT fine-tuning,

    H. Hua, X. Li, D. Dou, C. Xu, and J. Luo, “Noise stability regularization for improving BERT fine-tuning,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, K. Toutanova, A. Rumshi...

    2021

  28. [36]

    A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima,

    Z. Xie, I. Sato, and M. Sugiyama, “A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima,” inInternational Conference on Learning Representations, 2021. [Online]. Available: https://openreview.net/forum?id=wXgk_iCiYGo

    2021

  29. [37]

    Unveiling the structure of wide flat minima in neural networks,

    C. Baldassi, C. Lauditi, E. M. Malatesta, G. Perugini, and R. Zecchina, “Unveiling the structure of wide flat minima in neural networks,” Phys. Rev. Lett., vol. 127, p. 278301, Dec

  30. [38]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al., “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  31. [39]

    Le Gall, Measure theory, probability, and stochastic processes

    J.-F. Le Gall, Measure theory, probability, and stochastic processes. Springer, 2022

    2022

  32. [40]

    Structured pruning of large language models,

    Z. Wang, J. Wohlwend, and T. Lei, “Structured pruning of large language models,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , B. Webber, T. Cohn, Y . He, and Y . Liu, Eds. Online: Association for Computational Linguistics,...

    2020

  33. [41]

    The optimal BERT surgeon: Scalable and accurate second-order pruning for large language models,

    E. Kurtic, D. Campos, T. Nguyen, E. Frantar, M. Kurtz, B. Fineran, M. Goin, and D. Alistarh, “The optimal BERT surgeon: Scalable and accurate second-order pruning for large language models,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process...

    2022

  34. [42]

    Plug-and- play: An efficient post-training pruning method for large language models,

    Y . Zhang, H. Bai, H. Lin, J. Zhao, L. Hou, and C. V . Cannistraci, “Plug-and- play: An efficient post-training pruning method for large language models,” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https://openreview.net/fo...

    2024

  35. [43]

    LRQuant: Learnable and robust post-training quantization for large language models,

    J. Zhao, M. Zhang, C. Zeng, M. Wang, X. Liu, and L. Nie, “LRQuant: Learnable and robust post-training quantization for large language models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , L.-W. Ku, A. Mart...

    2024

  36. [44]

    A comprehensive evaluation of quantization strategies for large language models,

    R. Jin, J. Du, W. Huang, W. Liu, J. Luan, B. Wang, and D. Xiong, “A comprehensive evaluation of quantization strategies for large language models,” inFindings of the Association for Computational Linguistics: ACL 2024 , L.-W. Ku, A. Martins, and V . Srikumar, Eds. Bangkok, Tha...

    2024

  37. [45]

    IntactKV: Improving large language model quantization by keeping pivot tokens intact,

    R. Liu, H. Bai, H. Lin, Y . Li, H. Gao, Z. Xu, L. Hou, J. Yao, and C. Yuan, “IntactKV: Improving large language model quantization by keeping pivot tokens intact,” in Findings of the Association for Computational Linguistics: ACL 2024 , L.-W. Ku, A. Martins, and V . Srikumar, ...

    2024

  38. [46]

    Cost-effective distillation of large language models,

    S. Dasgupta, T. Cohn, and T. Baldwin, “Cost-effective distillation of large language models,” in Findings of the Association for Computational Linguistics: ACL 2023, A. Rogers, J. Boyd- Graber, and N. Okazaki, Eds. Toronto, Canada: Association for Computational Linguistics, Ju...

    2023

  39. [47]

    Distilling the knowledge in a neural network,

    G. Hinton, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  40. [48]

    mGPT: Few-shot learners go multilingual,

    O. Shliazhko, A. Fenogenova, M. Tikhonova, A. Kozlova, V . Mikhailov, and T. Shavrina, “mGPT: Few-shot learners go multilingual,” Transactions of the Association for Computational Linguistics , vol. 12, pp. 58–79, 2024. [Online]. Available: https: //aclanthology.org/2024.tacl-1.4

    2024

  41. [49]

    Toward understanding generative data augmentation,

    C. Zheng, G. Wu, and C. LI, “Toward understanding generative data augmentation,” in Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. 54 046–54 060. [Online...

    2023

  42. [50]

    Smoothness, low noise and fast rates,

    N. Srebro, K. Sridharan, and A. Tewari, “Smoothness, low noise and fast rates,” in Advances in Neural Information Processing Systems , J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta, Eds., vol. 23. Curran Associates, Inc., 2010. [Online]. Available: https:...

    2010

  43. [51]

    Diffsound: Discrete diffusion model for text-to-sound generation,

    D. Yang, J. Yu, H. Wang, W. Wang, C. Weng, Y . Zou, and D. Yu, “Diffsound: Discrete diffusion model for text-to-sound generation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 1720–1733, 2023

    2023

  44. [52]

    Towards explainable joint models via information theory for multiple intent detection and slot filling,

    X. Zhuang, X. Cheng, and Y . Zou, “Towards explainable joint models via information theory for multiple intent detection and slot filling,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 17, pp. 19 786–19 794, Mar. 2024. [Online]. Available: https:...

    2024

  45. [53]

    Kdpror: A knowledge-decoupling probabilistic framework for video-text retrieval,

    X. Zhuang, H. Li, X. Cheng, Z. Zhu, Y . Xie, and Y . Zou, “Kdpror: A knowledge-decoupling probabilistic framework for video-text retrieval,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 313–331

    2024

  46. [54]

    Gpa: global and prototype alignment for audio-text retrieval,

    Y . Xie, Z. Zhu, X. Zhuang, L. Liang, Z. Wang, and Y . Zou, “Gpa: global and prototype alignment for audio-text retrieval,” inProc. Interspeech 2024, 2024, pp. 5078–5082

    2024

  47. [55]

    Preparing lessons for progressive training on language models,

    Y . Pan, Y . Yuan, Y . Yin, J. Shi, Z. Xu, M. Zhang, L. Shang, X. Jiang, and Q. Liu, “Preparing lessons for progressive training on language models,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 17, 2024, pp. 18 860–18 868

    2024

  48. [56]

    Do as we do, not as you think: the conformity of large language models,

    Z. Weng, G. Chen, and W. Wang, “Do as we do, not as you think: the conformity of large language models,” arXiv preprint arXiv:2501.13381, 2025

    2025 arXiv

  49. [57]

    Rmt: Retentive networks meet vision transformers,

    Q. Fan, H. Huang, M. Chen, H. Liu, and R. He, “Rmt: Retentive networks meet vision transformers,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 5641–5651

    2024

  50. [58]

    Reusing pretrained models by multi-linear operators for efficient training,

    Y . Pan, Y . Yuan, Y . Yin, Z. Xu, L. Shang, X. Jiang, and Q. Liu, “Reusing pretrained models by multi-linear operators for efficient training,” Advances in Neural Information Processing Systems, vol. 36, pp. 3248–3262, 2023

    2023

  51. [59]

    Semantic equitable clustering: A simple, fast and effective strategy for vision transformer,

    Q. Fan, H. Huang, M. Chen, and R. He, “Semantic equitable clustering: A simple, fast and effective strategy for vision transformer,”arXiv preprint arXiv:2405.13337, 2024

    2024 arXiv

  52. [60]

    Pcad: Towards asr- robust spoken language understanding via prototype calibration and asymmetric decoupling,

    X. Zhuang, X. Cheng, L. Liang, Y . Xie, Z. Wang, Z. Huang, and Y . Zou, “Pcad: Towards asr- robust spoken language understanding via prototype calibration and asymmetric decoupling,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vo...

    2024

  53. [61]

    Macsc: Towards multimodal- augmented pre-trained language models via conceptual prototypes and self-balancing calibra- tion,

    X. Zhuang, Z. Wang, X. Cheng, Y . Xie, L. Liang, and Y . Zou, “Macsc: Towards multimodal- augmented pre-trained language models via conceptual prototypes and self-balancing calibra- tion,” in Proceedings of the 2024 Conference of the North American Chapter of the Association f...

    2024

  54. [62]

    Hifi-codec: Group-residual vector quantization for high fidelity audio codec,

    D. Yang, S. Liu, R. Huang, J. Tian, C. Weng, and Y . Zou, “Hifi-codec: Group-residual vector quantization for high fidelity audio codec,” arXiv preprint arXiv:2305.02765, 2023

    2023 arXiv

  55. [63]

    Uniaudio 1.5: Large language model-driven audio codec is a few-shot audio task learner,

    D. Yang, H. Guo, Y . Wang, R. Huang, X. Li, X. Tan, X. Wu, and H. Meng, “Uniaudio 1.5: Large language model-driven audio codec is a few-shot audio task learner,” arXiv preprint arXiv:2406.10056, 2024

    2024 arXiv

  56. [64]

    Fedzkp: Federated model ownership verification with zero-knowledge proof,

    W. Yang, Y . Yin, G. Zhu, H. Gu, L. Fan, X. Cao, and Q. Yang, “Fedzkp: Federated model ownership verification with zero-knowledge proof,”arXiv preprint arXiv:2305.04507, 2023

    2023 arXiv

  57. [65]

    Fedsov: Federated model secure ownership verification with unforgeable signature,

    W. Yang, G. Zhu, Y . Yin, H. Gu, L. Fan, Q. Yang, and X. Cao, “Fedsov: Federated model secure ownership verification with unforgeable signature,”arXiv preprint arXiv:2305.06085, 2023

    2023 arXiv

  58. [66]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,”Journal of machine learning research, vol. 21, no. 140, pp. 1–67, 2020

    2020

  59. [67]

    Physics of language models: Part 3.3, knowledge capacity scaling laws,

    Z. Allen-Zhu and Y . Li, “Physics of language models: Part 3.3, knowledge capacity scaling laws,” arXiv preprint arXiv:2404.05405, 2024

    2024 arXiv

  60. [68]

    Parameterized algorithms and complexity for the traveling purchaser problem and its variants,

    M. Xiao, J. Zhang, and W. Lin, “Parameterized algorithms and complexity for the traveling purchaser problem and its variants,” Journal of Combinatorial Optimization, pp. 1–17, 2022

    2022

  61. [69]

    Dataset pruning: Reducing training data by examining generalization influence,

    S. Yang, Z. Xie, H. Peng, M. Xu, M. Sun, and P. Li, “Dataset pruning: Reducing training data by examining generalization influence,” inThe Eleventh International Conference on Learning Representations, 2023. [Online]. Available: https://openreview.net/forum?id=4wZiAXD29TQ 14

    2023

  62. [70]

    On training data influence of GPT models,

    Y . Chai, Q. Liu, S. Wang, Y . Sun, Q. Peng, and H. Wu, “On training data influence of GPT models,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y . Al-Onaizan, M. Bansal, and Y .-N. Chen, Eds. Miami, Florida, USA: Association for ...

    2024

  63. [71]

    From quantity to quality: Boosting LLM performance with self-guided data selection for instruction tuning,

    M. Li, Y . Zhang, Z. Li, J. Chen, L. Chen, N. Cheng, J. Wang, T. Zhou, and J. Xiao, “From quantity to quality: Boosting LLM performance with self-guided data selection for instruction tuning,” in Proceedings of the 2024 Conference of the North American Chapter of the Associati...

    2024

  64. [72]

    Doremi: Optimizing data mixtures speeds up language model pretraining,

    S. M. Xie, H. Pham, X. Dong, N. Du, H. Liu, Y . Lu, P. S. Liang, Q. V . Le, T. Ma, and A. W. Yu, “Doremi: Optimizing data mixtures speeds up language model pretraining,” in Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt...

    2023

  65. [73]

    Beyond scale: the diversity coefficient as a data quality metric demonstrates llms are pre-trained on formally diverse data,

    A. Lee, B. Miranda, S. Sundar, and S. Koyejo, “Beyond scale: the diversity coefficient as a data quality metric demonstrates llms are pre-trained on formally diverse data,” arXiv preprint arXiv:2306.13840, 2023

    2023 arXiv

  66. [74]

    Talking nonsense: Probing large language models’ understanding of adversarial gibberish inputs,

    V . Cherepanova and J. Zou, “Talking nonsense: Probing large language models’ understanding of adversarial gibberish inputs,” arXiv preprint arXiv:2404.17120, 2024

    2024 arXiv

  67. [75]

    A comprehensive survey on source-free domain adaptation,

    J. Li, Z. Yu, Z. Du, L. Zhu, and H. T. Shen, “A comprehensive survey on source-free domain adaptation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 8, pp. 5743–5762, 2024

    2024

  68. [76]

    Cross-domain mutual information adversarial maximiza- tion,

    L. Meng, H. Su, C. Lou, and J. Li, “Cross-domain mutual information adversarial maximiza- tion,” Engineering Applications of Artificial Intelligence, vol. 110, p. 104665, 2022

    2022

  69. [77]

    Domain adaptive land-cover classification via local consistency and global diversity,

    A. Ma, C. Zheng, J. Wang, and Y . Zhong, “Domain adaptive land-cover classification via local consistency and global diversity,” IEEE Transactions on Geoscience and Remote Sensing , vol. 61, pp. 1–17, 2023

    2023

  70. [78]

    Diffusion-based probabilistic uncertainty estimation for active domain adaptation,

    Z. Du and J. Li, “Diffusion-based probabilistic uncertainty estimation for active domain adaptation,” in Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. 1...

    2023

  71. [79]

    Online adaptive fault diagnosis with test-time domain adaptation,

    K. Wu, J. Li, L. Meng, F. Li, and K. Lu, “Online adaptive fault diagnosis with test-time domain adaptation,” IEEE Transactions on Industrial Informatics, pp. 1–11, 2024

    2024

  72. [80]

    Divergence-agnostic unsupervised domain adaptation by adversarial attacks,

    J. Li, Z. Du, L. Zhu, Z. Ding, K. Lu, and H. T. Shen, “Divergence-agnostic unsupervised domain adaptation by adversarial attacks,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 11, pp. 8196–8211, 2021

    2021

  73. [81]

    Imbalanced open set domain adaptation via moving-threshold estimation and gradual alignment,

    J. Ru, J. Tian, C. Xiao, J. Li, and H. T. Shen, “Imbalanced open set domain adaptation via moving-threshold estimation and gradual alignment,” IEEE Transactions on Multimedia, vol. 26, pp. 2504–2514, 2024

    2024

  74. [82]

    Learning from noisy labels with deep neural networks: A survey,

    H. Song, M. Kim, D. Park, Y . Shin, and J.-G. Lee, “Learning from noisy labels with deep neural networks: A survey,” IEEE transactions on neural networks and learning systems , vol. 34, no. 11, pp. 8135–8153, 2022

    2022

  75. [83]

    Does label smoothing mitigate label noise?

    M. Lukasik, S. Bhojanapalli, A. Menon, and S. Kumar, “Does label smoothing mitigate label noise?” in Proceedings of the 37th International Conference on Machine Learning , ser. Proceedings of Machine Learning Research, H. D. III and 15 A. Singh, Eds., vol. 119. PMLR, 13–18 Jul...

    2020

  76. [84]

    Smoothgrad: removing noise by adding noise,

    D. Smilkov, N. Thorat, B. Kim, F. Viégas, and M. Wattenberg, “Smoothgrad: removing noise by adding noise,” arXiv preprint arXiv:1706.03825, 2017

    2017 arXiv

  77. [85]

    Pure noise to the rescue of insufficient data: Improving imbalanced classification by training on random noise images,

    S. Zada, I. Benou, and M. Irani, “Pure noise to the rescue of insufficient data: Improving imbalanced classification by training on random noise images,” in Proceedings of the 39th International Conference on Machine Learning , ser. Proceedings of Machine Learning Research, K....

    2022

  78. [86]

    Llm-pcgc: Large language model-based point cloud geometry compres- sion,

    Y . Ye and W. Gao, “Llm-pcgc: Large language model-based point cloud geometry compres- sion,” arXiv preprint arXiv:2408.08682, 2024

    2024 arXiv

  79. [87]

    Vision transformer with sparse scan prior,

    Q. Fan, H. Huang, M. Chen, and R. He, “Vision transformer with sparse scan prior,” arXiv preprint arXiv:2405.13335, 2024

    2024 arXiv

  80. [88]

    Semigmmpoint: Semi-supervised point cloud segmentation based on gaussian mixture models,

    X. Zhuang, H. Wang, X. He, S. Fu, and H. Hu, “Semigmmpoint: Semi-supervised point cloud segmentation based on gaussian mixture models,” Pattern Recognition, vol. 158, p. 111045, 2025

    2025

  81. [89]

    Composed fine-tuning: Freezing pre-trained denoising autoencoders for improved generalization,

    S. M. Xie, T. Ma, and P. Liang, “Composed fine-tuning: Freezing pre-trained denoising autoencoders for improved generalization,” in Proceedings of the 38th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds.,...

    2021

  82. [90]

    Game on tree: Visual hallucination mitigation via coarse-to-fine view tree and game theory,

    X. Zhuang, Z. Zhu, Z. Chen, Y . Xie, L. Liang, and Y . Zou, “Game on tree: Visual hallucination mitigation via coarse-to-fine view tree and game theory,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 17 984–18 003

    2024

  83. [91]

    Not all texts are the same: Dynamically querying texts for scene text detection,

    L. Tang, P. Yi, M. Chen, M. Yang, and D. Liang, “Not all texts are the same: Dynamically querying texts for scene text detection,” inChinese Conference on Pattern Recognition and Computer Vision (PRCV). Springer, 2024, pp. 363–377

    2024

  84. [92]

    Towards multimodal-augmented pre-trained language models via self-balanced expectation-maximization iteration,

    X. Zhuang, X. Cheng, Z. Zhu, Z. Chen, H. Li, and Y . Zou, “Towards multimodal-augmented pre-trained language models via self-balanced expectation-maximization iteration,” in Pro- ceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 4670–4679

    2024

  85. [93]

    Vasparse: Towards efficient visual hallu- cination mitigation for large vision-language model via visual-aware sparsification,

    X. Zhuang, Z. Zhu, Y . Xie, L. Liang, and Y . Zou, “Vasparse: Towards efficient visual hallu- cination mitigation for large vision-language model via visual-aware sparsification,” arXiv preprint arXiv:2501.06553, 2025

    2025 arXiv

  86. [94]

    Improving pretrained language model fine-tuning with noise stability regularization,

    H. Hua, X. Li, D. Dou, C.-Z. Xu, and J. Luo, “Improving pretrained language model fine-tuning with noise stability regularization,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–15, 2023

    2023

  87. [95]

    SMART: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization,

    H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and T. Zhao, “SMART: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. ...

    2020

  88. [96]

    Tip-adapter: Training- free clip-adapter for better vision-language modeling,

    R. Zhang, R. Fang, W. Zhang, P. Gao, K. Li, J. Dai, Y . Qiao, and H. Li, “Tip-adapter: Training- free clip-adapter for better vision-language modeling,” arXiv preprint arXiv:2111.03930 , 2021

    2021 arXiv

  89. [97]

    Learning to prompt for vision-language models,

    K. Zhou, J. Yang, C. C. Loy, and Z. Liu, “Learning to prompt for vision-language models,” International Journal of Computer Vision, vol. 130, no. 9, pp. 2337–2348, 2022. 16

    2022

  90. [98]

    Conditional prompt learning for vision-language models,

    ——, “Conditional prompt learning for vision-language models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 16 816–16 825

    2022

  91. [99]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in Proceedings of the 38th International Conference on Machine L...

    2021

  92. [100]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in International Conference on Learning Representations, 2022. [Online]. Available: https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  93. [101]

    Implicit gradient regularization,

    D. Barrett and B. Dherin, “Implicit gradient regularization,” in International Conference on Learning Representations, 2021. [Online]. Available: https://openreview.net/forum?id= 3q5IqUrkcF

    2021

  94. [102]

    Gradient norm aware minimization seeks first-order flatness and improves generalization,

    X. Zhang, R. Xu, H. Yu, H. Zou, and P. Cui, “Gradient norm aware minimization seeks first-order flatness and improves generalization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 20 247–20 257

    2023

  95. [103]

    Sharpness-aware minimization for efficiently improving generalization,

    P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur, “Sharpness-aware minimization for efficiently improving generalization,” in International Conference on Learning Representations, 2021. [Online]. Available: https://openreview.net/forum?id=6Tm1mposlrM

    2021

  96. [104]

    How sharpness-aware minimization minimizes sharpness?

    K. Wen, T. Ma, and Z. Li, “How sharpness-aware minimization minimizes sharpness?” in The Eleventh International Conference on Learning Representations, 2023. [Online]. Available: https://openreview.net/forum?id=5spDgWmpY6x

    2023

  97. [105]

    On the overlooked pitfalls of weight decay and how to mitigate them: A gradient-norm perspective,

    Z. Xie, Z. Xu, J. Zhang, I. Sato, and M. Sugiyama, “On the overlooked pitfalls of weight decay and how to mitigate them: A gradient-norm perspective,” in Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Ed...

    2023

  98. [106]

    Decoupled weight decay regularization,

    I. Loshchilov, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  99. [107]

    Adaptive inertia: Disentangling the effects of adaptive learning rate and momentum,

    Z. Xie, X. Wang, H. Zhang, I. Sato, and M. Sugiyama, “Adaptive inertia: Disentangling the effects of adaptive learning rate and momentum,” in Proceedings of the 39th International Conference on Machine Learning , ser. Proceedings of Machine Learning Research, K. Chaudhuri, S. ...

    2022

  100. [108]

    Recursive deep models for semantic compositionality over a sentiment treebank,

    R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts, “Recursive deep models for semantic compositionality over a sentiment treebank,” in Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, D. Yarowsky, T. Baldwin, A...

    2013

  101. [109]

    A fast and accurate dependency parser using neural networks,

    D. Chen and C. Manning, “A fast and accurate dependency parser using neural networks,” in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) , A. Moschitti, B. Pang, and W. Daelemans, Eds. Doha, Qatar: Association for Computational L...

    2014

  102. [110]

    Mitigating uncertainty in document classification,

    X. Zhang, F. Chen, C.-T. Lu, and N. Ramakrishnan, “Mitigating uncertainty in document classification,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Pa...

    2019

  103. [111]

    Mining and summarizing customer reviews,

    M. Hu and B. Liu, “Mining and summarizing customer reviews,” inProceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, 2004, pp. 168–177

    2004

  104. [112]

    News sentiment analysis,

    A. Samuels and J. Mcgonical, “News sentiment analysis,” arXiv preprint arXiv:2007.02238, 2020

    2007 arXiv

  105. [113]

    Balanced copa: Countering superficial cues in causal reasoning,

    P. Kavumba, N. Inoue, B. Heinzerling, K. Singh, P. Reisert, and K. Inui, “Balanced copa: Countering superficial cues in causal reasoning,”Association for Natural Language Processing, pp. 1105–1108, 2020

    2020

  106. [114]

    Automatically constructing a corpus of sentential paraphrases,

    B. Dolan and C. Brockett, “Automatically constructing a corpus of sentential paraphrases,” in Third international workshop on paraphrasing (IWP2005), 2005

    2005

  107. [115]

    WiC: the word-in-context dataset for evaluating context-sensitive meaning representations,

    M. T. Pilehvar and J. Camacho-Collados, “WiC: the word-in-context dataset for evaluating context-sensitive meaning representations,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologie...

    2019

  108. [116]

    Sequence to sequence learning with neural networks,

    I. Sutskever, “Sequence to sequence learning with neural networks,” arXiv preprint arXiv:1409.3215, 2014

    2014 arXiv

  109. [117]

    3d object representations for fine-grained categorization,

    J. Krause, M. Stark, J. Deng, and L. Fei-Fei, “3d object representations for fine-grained categorization,” in Proceedings of the IEEE international conference on computer vision workshops, 2013, pp. 554–561

    2013

  110. [119]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky, G. Hinton et al., “Learning multiple layers of features from tiny images,” 2009

    2009

  111. [120]

    Describing textures in the wild,

    M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi, “Describing textures in the wild,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2014, pp. 3606–3613

    2014

  112. [121]

    Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification,

    P. Helber, B. Bischke, A. Dengel, and D. Borth, “Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 12, no. 7, pp. 2217–2226, 2019

    2019

  113. [122]

    Fine-grained visual classification of aircraft,

    S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi, “Fine-grained visual classification of aircraft,” arXiv preprint arXiv:1306.5151, 2013

    2013 arXiv

  114. [123]

    Automated flower classification over a large number of classes,

    M.-E. Nilsback and A. Zisserman, “Automated flower classification over a large number of classes,” in 2008 Sixth Indian conference on computer vision, graphics & image processing. IEEE, 2008, pp. 722–729

    2008

  115. [124]

    Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories,

    L. Fei-Fei, R. Fergus, and P. Perona, “Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories,” in 2004 conference on computer vision and pattern recognition workshop. IEEE, 2004, pp. 178–178

    2004

  116. [125]

    Cats and dogs,

    O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar, “Cats and dogs,” in 2012 IEEE conference on computer vision and pattern recognition. IEEE, 2012, pp. 3498–3505. 18

    2012

  117. [126]

    Rotation equivariant cnns for digital pathology,

    B. S. Veeling, J. Linmans, J. Winkens, T. Cohen, and M. Welling, “Rotation equivariant cnns for digital pathology,” inMedical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part...

    2018

  118. [127]

    Remote sensing image scene classification: Benchmark and state of the art,

    G. Cheng, J. Han, and X. Lu, “Remote sensing image scene classification: Benchmark and state of the art,” Proceedings of the IEEE, vol. 105, no. 10, pp. 1865–1883, 2017

    2017

  119. [128]

    Reading digits in natural images with unsupervised feature learning,

    Y . Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, A. Y . Ng et al., “Reading digits in natural images with unsupervised feature learning,” in NIPS workshop on deep learning and unsupervised feature learning, vol. 2011, no. 2. Granada, 2011, p. 4

    2011

  120. [129]

    EfficientNet: Rethinking model scaling for convolutional neural networks,

    M. Tan and Q. Le, “EfficientNet: Rethinking model scaling for convolutional neural networks,” in Proceedings of the 36th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09–15 J...

    2019

  121. [130]

    Imagenet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009, pp. 248–255

    2009

  122. [131]

    Revisiting unreasonable effectiveness of data in deep learning era,

    C. Sun, A. Shrivastava, S. Singh, and A. Gupta, “Revisiting unreasonable effectiveness of data in deep learning era,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 843–852

    2017

  123. [132]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778

    2016

  124. [133]

    Imagenet-21k pretraining for the masses,

    T. Ridnik, E. Ben-Baruch, A. Noy, and L. Zelnik-Manor, “Imagenet-21k pretraining for the masses,” in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) , 2021. [Online]. Available: https: //openreview.net/forum?id=Zkj_VcZ6ol

    2021

  125. [134]

    Swin transformer: Hierarchical vision transformer using shifted windows,

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10 012–10 022

    2021

  126. [135]

    Convnext v2: Co- designing and scaling convnets with masked autoencoders,

    S. Woo, S. Debnath, R. Hu, X. Chen, Z. Liu, I. S. Kweon, and S. Xie, “Convnext v2: Co- designing and scaling convnets with masked autoencoders,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 16 133–16 142

    2023

  127. [136]

    Laion-5b: An open large-scale dataset for training next generation image-text models,

    C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, P. Schramowski, S. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarczyk, and J. Jitsev, “Laion-5b: An open large-scale dataset for training next generation image-...

    2022

  128. [137]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020. 19 A Notations The commonly used notations and their descriptions are as follows. Notation Description L context length d embedding dimension W ...

    2010 arXiv

  129. [139]

    Besides full-parameter fine-tuning, numerous parameter-efficient fine-tuning algorithms have been extensively studied

    improved model robustness by adding regularization terms. Besides full-parameter fine-tuning, numerous parameter-efficient fine-tuning algorithms have been extensively studied. [96] introduced adapters into the original model architecture, optimizing only these parameters duri...

  130. [2021]

    Available: https://link.aps.org/doi/10.1103/PhysRevLett.127.278301

    [Online]. Available: https://link.aps.org/doi/10.1103/PhysRevLett.127.278301

    doi

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.