Pith. sign in

REVIEW 4 major objections 3 minor 52 references

A Theoretical Framework for OOD Robustness in Transformers using Gevrey Classes

T0 review · 4 major / 3 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A Transformer's prediction map is Gevrey-smooth, so its out-of-distribution mean-squared error is bounded by a sub-exponential term in the Wasserstein-1 shift between the training and test distributions.

desk verdict The paper asks a good question and runs clean experiments, but its main theorem rests on a false tail-bound lemma and a wrong minimization, so the headline sub-exponential bound is not established. read the letter →

arxiv 2504.12991 v2 pith:MNVLB3NC submitted 2025-04-17 cs.LG

Yu Wang , Fu-Chieh Chang , Pei-Yuan Wu This is my paper · ORCID
classification cs.LG
keywords out-of-distributionrobustnessTransformerlanguagemodelsGevreyclassesWasserstein-1distancechain-of-thoughtpromptingin-contextlearninggeneralizationboundssub-exponentialerrordecay
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that when a Transformer is viewed as a recursively composed smooth map, it belongs to a controlled smoothness class called a Gevrey class, and that this regularity alone controls how much its predictions degrade under semantic distribution shift. The main theorem states that if the in-distribution mean-squared error is at most $\varepsilon$ and the test distribution is at Wasserstein-1 distance $d$ from the training distribution, then the out-of-distribution MSE is bounded by a sub-exponential term $\exp(-C d^{-1/(s+1)} \log(1/d))$ plus $\varepsilon$ and a Lipschitz term in $d$. Such a bound would matter because it works even when training and test supports are disjoint, a regime where KL-based bounds are undefined. The authors report controlled arithmetic and chain-of-thought experiments with latent permutations and scalings whose error curves, they argue, track the predicted sub-exponential shape.

What carries the argument

The load-bearing object is the Gevrey class $G^s$, the set of smooth functions whose derivatives satisfy $|\partial^\alpha f(x)| \le C R^{|\alpha|} (|\alpha|!)^s$ on compact sets; it generalizes analytic functions ($s=1$) while still taming derivative growth. The paper models an autoregressive Transformer as a recursive family $F_{n+1}(x_1,\dots,x_{n+1}) = F(x_1,\dots,x_n,F_n(x_1,\dots,x_n))$ and argues each $F_n$ stays Gevrey-regular because Gevrey classes are closed under composition and multiplication. Two supporting lemmas carry the proof: the tail-based Wasserstein shift bound (Lemma B.2) splits the shift error into a dominant threshold term plus an integral over the transport-coupling tail, and the Gevrey modulus bound (Lemma C.1) converts regularity into a pointwise modulus of continuity $\exp(-B \rho^{-1/s} \log(1/\rho))$ for displacement $\rho$. Together they produce the sub-exponential term in the main inequality.

What would settle it

Two concrete checks settle the theorem's quantitative form: minimize $F(N)=(R\rho)^N N^{sN}$ directly for $s>1$ (the stated $N^*=1/(eR\rho)$ is only the $s=1$ minimizer, so the exponent's dependence on $\rho$ needs rederivation), and test Lemma B.2's inequality with a coupling that puts mass $1/2$ at distances $d$ and $3d$ and $\phi(r)=1/r$ at cutoff $r_0=2d$. If either check fails, the sub-exponential term with $\log(1/d)$ does not follow from the lemmas as written.

Watch

Extended reading notes

Core claim

The central claim is Theorem 4.4: for a Gevrey-class prediction map $F$ and a Lipschitz target $G$ on a common compact domain $K$, if the in-distribution error under $P_1$ is at most $\varepsilon$ and $d=W_1(P_1,P_2)$, then $\mathbb{E}_{x\sim P_2}[\|F(x)-G(x)\|^2] \le 6A^2\exp(-C d^{-1/(s+1)} \log(1/d)) + 3\varepsilon + 3L_1^2 d^2$. The paper claims this is the first MSE-style upper bound for Transformer prediction under semantic shift, and that the smoothness index $s$ of the Gevrey class governs how quickly error can grow as the test distribution moves away. It further claims that the empirical curves in Figures 1--3 validate the shape of this bound under permutation and scaling shifts, with all theoretical constants fixed across experiments.

Load-bearing premise

The bound assumes the trained model really is a Gevrey-class function with fixed constants on a single compact domain containing both the training and test inputs.

Editorial extensions

If this is right

  • Test-time MSE under a distribution shift should rise sub-exponentially and super-polynomially in the Wasserstein shift $d$, not merely linearly or quadratically.
  • Models with smaller Gevrey index $s$ should degrade more gracefully under small shifts, because the dominant exponent $-C d^{-1/(s+1)}$ is less aggressive when $s$ is small.
  • Because the bound does not require overlapping support, the same framework can quantify OOD robustness in settings where KL-type divergences are undefined or infinite.
  • The recursive structure of the proof predicts that error compounds with chain-of-thought length, with later reasoning steps suffering larger degradation under the same shift.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the bound holds quantitatively, the $\log(1/d)$ factor in the exponent is a distinctive fingerprint: it predicts a specific concave curvature in log-error versus shift-size plots that could distinguish Gevrey-regular models from merely Lipschitz ones.
  • A testable extension is to train representations under an explicit smoothness penalty designed to lower the effective $s$ and check whether the OOD error curve shifts downward; the paper itself does not run this intervention.
  • One could relax the common compact-domain assumption to sub-Gaussian or heavy-tailed input distributions and predict how the sub-exponential exponent weakens as the tails thicken.
  • In classification settings, the same Gevrey modulus would bound how logits or margins move under shift, linking OOD robustness to adversarial robustness bounds.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper studies out-of-distribution (OOD) robustness of Transformer language models under semantic shifts with disjoint support. It models the prediction map as a Gevrey-class function and measures the shift by Wasserstein-1 distance, deriving an upper bound (Theorem 4.4) of the form MSE <= 6A^2 exp(-C d^{-1/(s+1)} log(1/d)) + 3 epsilon + 3 L1^2 d^2. The authors validate the bound on a synthetic mean-square arithmetic task and on chain-of-thought tasks with latent permutations and scalings, arguing that the empirical degradation tracks the predicted sub-exponential behavior.

Significance. If the main theorem were correct, the paper would provide a novel quantitative link between function regularity and OOD generalization for sequence models, and the controlled experimental design is a reasonable starting point. The paper also ships reproducible code and gives explicit constructions of the Wasserstein-1 upper bounds in Appendix D. However, the central proof rests on a false lemma (Lemma B.2) and a minimization error in Lemma C.1, so the claimed d-dependence is not established. The empirical 'validation' is curve-fitting with undisclosed constants, and the paper's own Limitations section concedes that the bounds are not predictive tools. These issues are load-bearing for both the theoretical and empirical halves of the central claim.

major comments (4)
  1. [B.2] Lemma B.2 is false as stated. The proof converts the Markov tail bound P[||x-y|| >= r] <= d/r into the density-type bound dP[||x-y|| = r] <= (d/r^2) dr, but a tail bound does not determine the distribution of ||x-y||, so this step is unjustified. A concrete counterexample is: let ||x-y|| take the values d and 3d each with probability 1/2, so E||x-y|| = 2d; let phi(r) = 1/r and r0 = 2d. Then the left side is (1/2)(1/d) + (1/2)(1/(3d)) = 2/(3d), while the right side is phi(r0) + ∫_{2d}^{3d} phi(r)(2d/r^2) dr = 1/(2d) + 2d[-1/(2r^2)]_{2d}^{3d} = 23/(36d). Since 2/(3d) = 24/(36d) > 23/(36d), the claimed inequality fails. This lemma is the step in the proof of Theorem B.1 that converts the Wasserstein mean constraint into the exponential tail term, so Theorem 4.4 is not proved.
  2. [C.1] The proof of Lemma C.1 minimizes F(N) = (R rho)^N N^{sN}. Taking the logarithmic derivative, the stationary point satisfies log(R rho) + s(log N + 1) = 0, i.e. N* = e^{-1}(R rho)^{-1/s}. The paper instead uses N* = 1/(e R rho), which is only correct for s = 1; for s > 1, the subsequent algebra leading to the exponent -B rho^{-1/s} log(1/rho) and to B = (s-1)/(eR) log(1/R) does not follow. In addition, the modulus phi(r) = exp(-2B r^{-1/s} log(1/r)) is increasing for sufficiently small r when s > 1 (this follows by differentiating -r^{-1/s} log r), so it does not satisfy the non-increasing hypothesis of Lemma B.2. The distinctive sub-exponential d^{-1/(s+1)} log(1/d) dependence in Theorem 4.4 is therefore unsupported by the supplied algebra.
  3. [4.4] The empirical validation is fitted, not predictive. The theory curves in Figures 1-3 depend on constants A, C, s, epsilon, and L1, whose values are never reported and are described in Sec. 4.4 as 'chosen solely to highlight the asymptotic shape' after seeing the data. No fitting procedure, parameter values, or held-out comparison is provided. Since the bound is an upper bound with free constants, any monotone trend can be matched by such curves, so Figures 1-3 do not 'validate that our theoretical predictions meaningfully track the empirical behavior'. This is further contradicted by the paper's own Limitations section, which states that the bounds 'are therefore best viewed as qualitative guides rather than predictive tools'. The abstract and Sec. 4.4 overstate the empirical support, and the validation claim must be withdrawn or replaced with a falsifiable prediction protocol.
  4. [4.3] The claim that transformer-based autoregressive models are Gevrey-regular is not established. The proof sketch states that ReLU belongs to a Gevrey class, but ReLU is not C^infty and therefore does not satisfy Definition 4.1; layer normalization also fails to be smooth if the variance approaches zero. Moreover, autoregressive generation involves sampling from a softmax distribution, which is not described by a deterministic smooth map F_n: X^n -> X. Since the Gevrey assumption on F is the hypothesis of Theorem 4.4, the applicability of the main theorem to the GPT-2 models used in the experiments is unsupported.
minor comments (3)
  1. [D] Appendix D refers to 'Theorem 4' in several places, but the main theorem is numbered Theorem 4.4; the cross-references should be updated.
  2. [3.2] The description of the six OOD testing sets (i = 0,...,5) and the in-distribution testing set is confusing: Fig. 1 labels both 'Testing Set, ID(i=0)' and 'Testing Set, OOD', and it is unclear whether the i=0 OOD set is the same as the ID testing set. The caption and text should disambiguate these curves.
  3. [C.1] The notation for the Gevrey index s is inconsistent: Lemma C.1 states s >= 1, while Theorem B.1 states s > 1; the paper should state the exact range of s used in each result.

Circularity Check

1 steps flagged · score 6.0 of 10

The main theorem is not circular, but the empirical validation is: the theory curves in Figures 1-3 use free constants chosen after seeing the data, so the claimed agreement is by construction rather than prediction.

  1. fitted input called prediction [Section 4.4 'Connection to Empirical Plots'; Section 5 'Bounding Tightness']
    "All constants used to generate the theory curves are fixed and shared across experiments, and chosen solely to highlight the asymptotic shape of the upper bound. Together, these results validate that our theoretical predictions meaningfully track the empirical behavior of LLMs under structured latent perturbations."

    The theory curves are computed from Theorem 4.4, whose bound contains unspecified constants A, C, the Gevrey index s, the in-distribution error epsilon, and the Lipschitz constant L1. The paper does not fix these from independent measurements or a parameter-free calibration; it states they are 'chosen solely to highlight the asymptotic shape.' Overlaying a bound whose free constants are selected after seeing the empirical MSE values cannot validate the d-dependence: large A makes the bound lie above any monotone error curve, and s can be chosen to match the observed slope on a log plot. The agreement is therefore by construction, not by prediction.

full rationale

The core derivation of Theorem 4.4 is not circular: it is a self-contained argument from Gevrey regularity, a Wasserstein-1 distance assumption, and an in-distribution error assumption, and it does not import a conclusion from the authors' own prior work. There are no load-bearing self-citations in the paper. The suspected mathematical errors in Lemma B.2 and Lemma C.1 are correctness issues, not circularity: a false tail-bound step or a miscalculated minimizer does not make the theorem equivalent to its assumptions by definition. The significant circularity is in Section 4.4: the 'Theory Upper Bound' curves depend on unspecified constants that are chosen after observing the empirical plots, so the statement that the figures 'validate that our theoretical predictions meaningfully track the empirical behavior' reduces to the trivial fact that a hand-parameterized upper-bound curve can be placed above and shaped to resemble the data. This is the fitted-input-called-prediction pattern. The theorem itself retains independent content, so the score reflects partial circularity rather than full collapse of the derivation into its inputs.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The theory rests on a smoothness assumption plus two technical lemmas; neither is externally validated. The experimental 'validation' adds free constants fitted to the displayed curves.

free parameters (4)
  • Gevrey index s = not disclosed
    Used in Theorem 4.4 and in the plotted theory curves; s controls the exponent -1/(s+1) but is never estimated from the model or reported.
  • Constants A and C = not disclosed
    From Theorem 4.4, these are said to be fixed and shared across experiments and 'chosen solely to highlight the asymptotic shape' (Sec. 4.4), i.e., fitted to the plotted curves.
  • In-distribution error epsilon = not reported
    The additive 3 epsilon term in the bound; training losses appear nonzero in Figures 1-3 and no value is given for the theory curves.
  • Lipschitz constant L1 of target G = not reported
    The 3 L1^2 d^2 term; L1 for the mean-square target depends on the domain and is never computed.
assumptions (6)
  • domain assumption The trained transformer prediction map F belongs to a Gevrey class G^s on a fixed compact domain K.
    Invoked in Theorem 4.4; justified by Theorem 4.3, whose proof only covers smooth activations and requires layer-norm denominators bounded away from zero. ReLU is listed as an allowed nonlinearity but is not smooth, and test supports in the mean-square task are not shown to lie inside K.
  • domain assumption The target function G is Lipschitz on K.
    Used in Theorem 4.4 to produce the 3 L1^2 d^2 term; true for the arithmetic target on bounded domains, but L1 is never bounded.
  • domain assumption Both P1 and P2 are supported on the same compact K with finite Wasserstein-1 distance d.
    Requirement of Theorem 4.4; the experimental OOD sets are defined on disjoint supports that are not shown to lie inside a common compact set with uniform Gevrey constants.
  • standard math Gevrey classes are closed under composition, addition, and pointwise multiplication with explicit constant control (Proposition A.4).
    Used repeatedly in Theorem 4.3 and Appendix A; the statement uses constants (e.g., C_h = C_f C_g 2^{s_h}) that do not match the proof's own 2^{|alpha|} factor, though the closure property itself is standard.
  • ad hoc to paper Lemma C.1: Gevrey functions satisfy the modulus bound exp(-B rho^{-1/s} log(1/rho)).
    This is the load-bearing estimate for the theorem's sub-exponential term; its proof contains the wrong minimization (N* valid only for s=1), so the bound is not established as stated.
  • ad hoc to paper Lemma B.2: tail-based Wasserstein shift bound for any non-increasing phi.
    Stated as a general lemma and false; the theorem's Eshift bound depends on it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Theoretical Framework for OOD Robustness in Transformers using Gevrey Classes." pith.science (2026). https://pith.science/paper/MNVLB3NC

@misc{pith2026250412991,
  author       = {Pith},
  title        = {Pith review of: A Theoretical Framework for OOD Robustness in Transformers using Gevrey Classes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MNVLB3NC}},
  note         = {Machine review of arXiv:2504.12991}
}
read the original abstract

We study the robustness of Transformer language models under semantic out-of-distribution (OOD) shifts, where training and test data lie in disjoint latent spaces. Using Wasserstein-1 distance and Gevrey-class smoothness, we derive sub-exponential upper bounds on prediction error. Our theoretical framework explains how smoothness governs generalization under distributional drift. We validate these findings through controlled experiments on arithmetic and Chain-of-Thought tasks with latent permutations and scalings. Results show empirical degradation aligns with our bounds, highlighting the geometric and functional principles underlying OOD generalization in Transformers.

Figures

Figures reproduced from arXiv: 2504.12991 by the authors.

Figure 1
Figure 1. Loss on training and testing sets for the mean [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Test loss on Θe vs. Θ. 0.1 0.2 0.3 0.4 0.5 10−4 10−1 102 δ Ltest(h) Training Set, h = 2 Training Set, h = 1 Testing Set Θ, h = 2 Testing Set Θ, h = 1 Testing Set Θ¯ , p = 1 − δ, h = 2 Testing Set Θ¯ , p = 1 − δ, h = 1 Testing Set Θ¯ , p = 1 + δ, h = 2 Testing Set Θ¯ , p = 1 + δ, h = 1 Theory Upper Bound of Θ¯ , h = 1 Theory Upper Bound of Θ¯ , h = 2 [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Test loss on Θ¯ vs. Θ. impact is plotted in [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 19 canonical work pages

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Invariant risk minimization

    Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez-Paz, D. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019

    arXiv 1907

  3. [3]

    Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J. W. A theory of learning from different domains. Machine learning, 79: 0 151--175, 2010

    2010

  4. [4]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  5. [5]

    Position coupling: Improving length generalization of arithmetic transformers using task structure, 2024

    Cho, H., Cha, J., Awasthi, P., Bhojanapalli, S., Gupta, A., and Yun, C. Position coupling: Improving length generalization of arithmetic transformers using task structure, 2024. URL https://arxiv.org/abs/2405.20671

    arXiv 2024

  6. [6]

    Joint distribution optimal transportation for domain adaptation

    Courty, N., Flamary, R., Habrard, A., and Rakotomamonjy, A. Joint distribution optimal transportation for domain adaptation. Advances in neural information processing systems, 30, 2017

    2017

  7. [7]

    Frogner, C., Zhang, C., Mobahi, H., Araya, M., and Poggio, T. A. Learning with a wasserstein loss. Advances in neural information processing systems, 28, 2015

    work page 2015

  8. [8]

    A pac-bayesian approach for domain adaptation with specialization to linear classifiers

    Germain, P., Habrard, A., Laviolette, F., and Morvant, E. A pac-bayesian approach for domain adaptation with specialization to linear classifiers. In International conference on machine learning, pp.\ 738--746. PMLR, 2013

    work page 2013

Show all 52 references
  1. [9]

    L., Leskovec, J., and Jurafsky, D

    Hamilton, W. L., Leskovec, J., and Jurafsky, D. Diachronic word embeddings reveal statistical laws of semantic change. arXiv preprint arXiv:1605.09096, 2016

    2016 arXiv

  2. [10]

    and Andriushchenko, M

    Hein, M. and Andriushchenko, M. Formal guarantees on the robustness of a classifier against adversarial manipulation. Advances in neural information processing systems, 30, 2017

    2017

  3. [11]

    and Gimpel, K

    Hendrycks, D. and Gimpel, K. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136, 2016

    2016 arXiv

  4. [12]

    Unveiling the statistical foundations of chain-of-thought prompting methods, 2024

    Hu, X., Zhang, F., Chen, S., and Yang, Z. Unveiling the statistical foundations of chain-of-thought prompting methods, 2024. URL https://arxiv.org/abs/2408.14511

    2024 arXiv

  5. [13]

    M., and Schockaert, S

    Khalid, I., Nourollah, A. M., and Schockaert, S. Benchmarking systematic relational reasoning with large language and reasoning models. arXiv preprint arXiv:2503.23487, 2025

    2025 arXiv

  6. [14]

    Detecting change in data streams

    Kifer, D., Ben-David, S., and Gehrke, J. Detecting change in data streams. In VLDB, volume 4, pp.\ 180--191. Toronto, Canada, 2004

    2004

  7. [15]

    and Suzuki, T

    Kim, J. and Suzuki, T. Transformers provably solve parity efficiently with chain of thought. arXiv preprint arXiv:2410.08633, 2024

    2024 arXiv

  8. [16]

    A simple unified framework for detecting out-of-distribution samples and adversarial attacks

    Lee, K., Lee, K., Lee, H., and Shin, J. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018

    2018

  9. [17]

    Training nonlinear transformers for chain-of-thought inference: A theoretical generalization analysis

    Li, H., Wang, M., Lu, S., Cui, X., and Chen, P.-Y. Training nonlinear transformers for chain-of-thought inference: A theoretical generalization analysis. arXiv preprint arXiv:2410.02167, 2024

    2024 arXiv

  10. [18]

    Universal representation learning from multiple domains for few-shot classification

    Li, W.-H., Liu, X., and Bilen, H. Universal representation learning from multiple domains for few-shot classification. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 9526--9535, 2021

    2021

  11. [19]

    Energy-based out-of-distribution detection

    Liu, W., Wang, X., Owens, J., and Li, Y. Energy-based out-of-distribution detection. Advances in neural information processing systems, 33: 0 21464--21475, 2020

    2020

  12. [20]

    Transdrift: Modeling word-embedding drift using transformer

    Madaan, N., Chaudhury, P., Kumar, N., and Bedathur, S. Transdrift: Modeling word-embedding drift using transformer. In Companion Proceedings of the ACM Web Conference 2024, pp.\ 1388--1393, 2024

    2024

  13. [21]

    Domain adaptation with multiple sources

    Mansour, Y., Mohri, M., and Rostamizadeh, A. Domain adaptation with multiple sources. Advances in neural information processing systems, 21, 2008

    2008

  14. [22]

    McAllester, D. A. Pac-bayesian stochastic model selection. Machine Learning, 51 0 (1): 0 5--21, 2003

    2003

  15. [23]

    Rethinking the role of demonstrations: What makes in-context learning work?, 2022

    Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. Rethinking the role of demonstrations: What makes in-context learning work?, 2022. URL https://arxiv.org/abs/2202.12837

    2022 arXiv

  16. [24]

    Towards a statistical theory of learning to learn in-context with transformers

    Mroueh, Y. Towards a statistical theory of learning to learn in-context with transformers. In NeurIPS 2023 Workshop Optimal Transport and Machine Learning, 2023

    2023

  17. [25]

    Opschoor, J. A. and Schwab, C. Exponential expressivity of relu k neural networks on gevrey classes with point singularities. Applications of Mathematics, 69 0 (5): 0 695--724, 2024

    2024

  18. [26]

    Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages

    Qin, L., Chen, Q., Wei, F., Huang, S., and Che, W. Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages. arXiv preprint arXiv:2310.14799, 2023

    2023 arXiv

  19. [27]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. Technical report, OpenAI, 2019

    2019

  20. [28]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  21. [29]

    M., and Zhang, M

    Rausch, A., Sedeh, A. M., and Zhang, M. Autoencoder-based semantic novelty detection: Towards dependable ai-based systems. Applied Sciences, 11 0 (21): 0 9881, 2021

    2021

  22. [30]

    Theoretical analysis of domain adaptation with optimal transport

    Redko, I., Habrard, A., and Sebban, M. Theoretical analysis of domain adaptation with optimal transport. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2017, Skopje, Macedonia, September 18--22, 2017, Proceedings, Part II 10, pp.\ 737-...

    2017

  23. [31]

    J., Fertig, E., Snoek, J., Poplin, R., Depristo, M., Dillon, J., and Lakshminarayanan, B

    Ren, J., Liu, P. J., Fertig, E., Snoek, J., Poplin, R., Depristo, M., Dillon, J., and Lakshminarayanan, B. Likelihood ratios for out-of-distribution detection. Advances in neural information processing systems, 32, 2019

    2019

  24. [32]

    W., Hashimoto, T

    Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731, 2019

    1911 arXiv

  25. [33]

    A., and Ommer, B

    Sanakoyeu, A., Bautista, M. A., and Ommer, B. Deep unsupervised learning of visual similarities. Pattern Recognition, 78: 0 331--343, 2018

    2018

  26. [34]

    J., de Rezende Rocha, A., Sapkota, A., and Boult, T

    Scheirer, W. J., de Rezende Rocha, A., Sapkota, A., and Boult, T. E. Toward open set recognition. IEEE transactions on pattern analysis and machine intelligence, 35 0 (7): 0 1757--1772, 2012

    2012

  27. [35]

    Wasserstein distance guided representation learning for domain adaptation

    Shen, J., Qu, Y., Zhang, W., and Yu, Y. Wasserstein distance guided representation learning for domain adaptation. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  28. [36]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  29. [37]

    C., Murino, V., and Savarese, S

    Volpi, R., Namkoong, H., Sener, O., Duchi, J. C., Murino, V., and Savarese, S. Generalizing to unseen domains via adversarial data augmentation. Advances in neural information processing systems, 31, 2018

    2018

  30. [38]

    Can in-context learning really generalize to out-of-distribution tasks? arXiv preprint arXiv:2410.09695, 2024

    Wang, Q., Wang, Y., Wang, Y., and Ying, X. Can in-context learning really generalize to out-of-distribution tasks? arXiv preprint arXiv:2410.09695, 2024

    2024 arXiv

  31. [39]

    Beyond in-distribution success: Scaling curves of cot granularity for language model generalization

    Wang, R., Huang, W., Song, S., Zhang, H., Iwasawa, Y., Matsuo, Y., and Guo, J. Beyond in-distribution success: Scaling curves of cot granularity for language model generalization. arXiv preprint arXiv:2502.18273, 2025

    2025

  32. [40]

    V., Zhou, D., et al

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  33. [41]

    N., Li, W., Ba, J., Grosse, R

    Wu, Y., Rabe, M. N., Li, W., Ba, J., Grosse, R. B., and Szegedy, C. Lime: Learning inductive bias for primitives of mathematical reasoning. In International Conference on Machine Learning, pp.\ 11251--11262. PMLR, 2021

    2021

  34. [42]

    Zero-shot learning-the good, the bad and the ugly

    Xian, Y., Schiele, B., and Akata, Z. Zero-shot learning-the good, the bad and the ugly. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 4582--4591, 2017

    2017

  35. [43]

    and Liu, B

    Xiao, C. and Liu, B. A theory for length generalization in learning to reason. arXiv preprint arXiv:2404.00560, 2024

    2024 arXiv

  36. [44]

    M., Raghunathan, A., Liang, P., and Ma, T

    Xie, S. M., Raghunathan, A., Liang, P., and Ma, T. An explanation of in-context learning as implicit bayesian inference, 2022. URL https://arxiv.org/abs/2111.02080

    2022 arXiv

  37. [45]

    Chain-of-thought provably enables learning the (otherwise) unlearnable

    Yang, C., Li, Z., and Wipf, D. Chain-of-thought provably enables learning the (otherwise) unlearnable. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  38. [46]

    Glue-x: Evaluating natural language understanding models from an out-of-distribution generalization perspective

    Yang, L., Zhang, S., Qin, L., Li, Y., Wang, Y., Liu, H., Wang, J., Xie, X., and Zhang, Y. Glue-x: Evaluating natural language understanding models from an out-of-distribution generalization perspective. arXiv preprint arXiv:2211.08073, 2022

    2022 arXiv

  39. [47]

    Unveiling the mechanisms of explicit cot training: How chain-of-thought enhances reasoning generalization

    Yao, X., Ren, R., Liao, Y., and Liu, Y. Unveiling the mechanisms of explicit cot training: How chain-of-thought enhances reasoning generalization. arXiv preprint arXiv:2502.04667, 2025

    2025

  40. [48]

    Revisiting out-of-distribution robustness in nlp: Benchmarks, analysis, and llms evaluations

    Yuan, L., Chen, Y., Cui, G., Gao, H., Zou, F., Cheng, X., Ji, H., Liu, Z., and Sun, M. Revisiting out-of-distribution robustness in nlp: Benchmarks, analysis, and llms evaluations. Advances in Neural Information Processing Systems, 36: 0 58478--58507, 2023

    2023

  41. [49]

    Evaluating interpolation and extrapolation performance of neural retrieval models

    Zhan, J., Xie, X., Mao, J., Liu, Y., Guo, J., Zhang, M., and Ma, S. Evaluating interpolation and extrapolation performance of neural retrieval models. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp.\ 2486--2496, 2022

    2022

  42. [50]

    Theoretically principled trade-off between robustness and accuracy

    Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L., and Jordan, M. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pp.\ 7472--7482. PMLR, 2019

    2019

  43. [51]

    V., et al

    Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  44. [52]

    What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization, 2023

    Zhang, Y., Zhang, F., Yang, Z., and Wang, Z. What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization, 2023. URL https://arxiv.org/abs/2305.19420

    2023 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.