Pith. sign in

REVIEW 3 major objections 5 minor 78 references

Blink of an eye: a simple theory for feature localization in generative models

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Critical windows in generative models are generic: the decisive interval is exactly the time between two total-variation thresholds.

desk verdict A clean unifying bound for critical windows in localization samplers, but the 'generic emergence' claim outruns what the theorem actually proves. read the letter →

arxiv 2502.00921 v2 pith:25HTPJLG submitted 2025-02-02 cs.LG

Marvin Li , Aayush Karan , Sitan Chen This is my paper · ORCID
classification cs.LG
keywords criticalwindowsstochasticlocalizationfeaturediffusionmodelsautoregressivelanguageall-or-nothingphenomenontotalvariationdistance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Sudden behavioral shifts in generative models—an LLM abandoning a math problem, a jailbreak turning a safe model harmful, a diffusion image committing to a cat—are all instances of the same localization phenomenon. The paper proves that, in any generative model that is a stochastic localization sampler, the critical window in which a feature is decided is the interval during which the forward degradation process has made a smaller sub-population indistinguishable from a larger one while the larger one still stands apart from the rest of the distribution. When that interval is nonempty, the forward-reverse experiment starting from the smaller sub-population returns a distribution within total-variation distance $\varepsilon(1+\max(1,W)/\sqrt{2})$ of the larger sub-population. The theory needs no log-concavity or Gaussian-specific assumptions, applies to autoregressive and diffusion models alike, removes the dimension factor in earlier diffusion bounds, and ties the phenomenon to all-or-nothing phase transitions. A reader should care because it turns scattered observations about model failures and feature emergence into one testable prediction about when a model commits.

What carries the argument

The central object is a stochastic localization sampler: a generative model obtained by reversing a Markovian degradation process, in which $X \to Y_{t_1} \to \cdots \to Y_{t_k}$ is a Markov chain and the sampler generates $X$ by sampling backwards along this chain. Diffusion (Gaussian noising and denoising) and autoregressive models (right-to-left token masking and next-token prediction) are both special cases. The argument that carries Theorem 2 is the forward-reverse experiment: sample from $p^{S_{\mathrm{init}}}$, degrade to time $\hat T$, then resample with the full model's reverse kernel. The proof uses the Markov property and the data processing inequality twice, plus a Bayes-rule coupling lemma that bounds the reverse discrepancy by the posterior mass on the complement of $S_{\mathrm{target}}$.

What would settle it

Run a stochastic localization sampler on a two-component target whose forward total-variation curve $\mathrm{TV}(p^{S_{\mathrm{init}}}_t, p^{S_{\mathrm{target}}}_t)$ is not monotone, or find $S_{\mathrm{init}} \subset S_{\mathrm{target}}$ with $T^{\,S}_{\mathrm{end}} > T^{\,S}_{\mathrm{start}}$; then the predicted critical window is empty and the theorem no longer constrains the forward-reverse experiment. Concretely, a diffusion on a mixture of Gaussians with different variances is a natural place to look for such a violation.

Watch

Extended reading notes

Core claim

The paper's central claim is Theorem 2, stated on its own terms: for sub-mixtures $S_{\mathrm{init}} \subseteq S_{\mathrm{target}} \subseteq \Theta$, if the forward-reverse time $\hat T$ falls between $T^{\,S}_{\mathrm{end}}$—the time at which the forward laws of $S_{\mathrm{init}}$ and $S_{\mathrm{target}}$ become $\varepsilon$-close—and $T^{\,S}_{\mathrm{start}}$—the time at which the forward law of $S_{\mathrm{target}}$ is still separated from its complement—then $\mathrm{TV}(p^{S_{\mathrm{init}},\hat T}, p^{S_{\mathrm{target}}}) \le \varepsilon (1+\max(1,W)/\sqrt{2})$, where $W$ is the weight ratio of the complement to the target. This is a generic statement about localization, not about particular data models: critical windows emerge whenever the degradation process erases fine distinctions before coarse ones. The paper presents this as a strict improvement over prior diffusion-only theory, which required strong distributional assumptions and produced dimension-dependent bounds.

Load-bearing premise

The load-bearing premise is that the total variation between the forward laws of $S_{\mathrm{init}}$ and $S_{\mathrm{target}}$ is nonincreasing in time and that the two threshold times are ordered $T^{\,S}_{\mathrm{end}} \le T^{\,S}_{\mathrm{start}}$; the paper notes the ordering can fail, and when it does the theorem's interval is empty and the bound is vacuous.

Editorial extensions

If this is right

  • If Theorem 2 is right, the location of a critical window in any localization-based model can be read off from two total-variation curves: where $S_{\mathrm{init}}$ and $S_{\mathrm{target}}$ collapse, and where $S_{\mathrm{target}}$ separates from its complement.
  • The same mechanism explains diffusion feature emergence, LLM reasoning errors, and jailbreaks that occur in the first few generated tokens, with no distribution-specific assumptions.
  • Because the error bound is independent of dimension, critical-window predictions no longer lose polynomial factors in dimension as earlier diffusion analyses did.
  • The connection to all-or-nothing phase transitions means sharp thresholds from statistical inference—like those in sparse linear regression—translate directly into sharp critical windows for in-context learning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implicit testable consequence, not developed in the paper: the theory predicts that samplers whose forward process is not monotone in the relevant total-variation sense will show diffuse or absent critical windows, so the sharpness of the window itself measures how hierarchical the sampler's degradation is.
  • The ordering assumption $T_{\mathrm{end}} \le T_{\mathrm{start}}$ is checked only in examples; a systematic search for natural mixtures that violate it—say, components with unequal variances or non-isotropic geometries—would reveal how often the theorem's interval is actually nonempty.
  • If the bound is tight in practice, a practitioner could localize a model's decisive steps by estimating TV curves rather than by brute-force truncation and resampling sweeps, which would make critical-window monitoring substantially cheaper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a unifying theory of critical windows in stochastic localization samplers, covering both diffusion and autoregressive models. Data is modeled as a mixture over a discrete label set Θ; the paper defines a forward-reverse experiment and, in Theorem 2, shows that if the reverse-start time \hat T falls between the two threshold times T_S_end and T_S_start, then the forward-reverse distribution launched from a smaller sub-mixture Sinit is close in total variation to the larger sub-mixture Starget. The proof is elementary, combining the triangle inequality, the data processing inequality, Bayes' rule, and a bounding lemma for likelihood ratios. The paper computes critical windows for Gaussian mixtures, a discrete masking process, random-walk models of math problem solving, and in-context learning with an all-or-nothing transition; it also introduces an ε-mixture tree hierarchy and reports LLM experiments on structured outputs and chain-of-thought reasoning.

Significance. If the main theorem is stated with its hypotheses made explicit, this is a valuable contribution. The framework is genuinely general across diffusion and autoregressive models, avoids stochastic-calculus and statistical-mechanics machinery, and gives a dimension-independent bound, improving on the comparable result in [LC24]. The connection to the all-or-nothing phenomenon and the hierarchy formalism are promising, and the paper ships code for the experiments. The central proof is a clean and likely correct combination of standard inequalities; the weaknesses are that the advertised 'generic emergence' of critical windows is conditional on an unproven non-vacuity condition, and that the proof of Theorem 6, which is the main new all-or-nothing application, contains internal inconsistencies as written. Both issues are fixable, but they are load-bearing for the paper's central claims.

major comments (3)
  1. [Section 3.1, Theorem 2] The claimed generic emergence of critical windows is not established by Theorem 2, because the theorem is vacuous unless the interval [T_S_end, T_S_start] is nonempty, and the paper proves no general sufficient condition for that ordering. Monotonicity of the individual total-variation curves follows from Definition 1 and the data processing inequality, so the missing condition is not monotonicity but the non-vacuity condition T_S_end(ε) ≤ T_S_start(ε). The text explicitly concedes in Section 3.1 that 'in general it need not be and our bound would be vacuous,' and the Section 4 examples are all constructed so that the ordering holds. The abstract's statement that critical windows 'emerge generically' therefore goes beyond what Theorem 2 actually delivers; a revised version should either state a formal condition on (p, {P→}) guaranteeing nonemptiness, or reframe the main claim as a conditional theorem supported by the computed instantiations.
  2. [Section 4.2.2, Theorem 6] The proof of Theorem 6 applies Theorem 2 outside the stated critical interval. In the second application it asserts 'we trivially have that Tend(ε)=Ns+2' and then invokes Theorem 2 at t=Ns+2−(1−δ2)N*s, which is strictly less than Ns+2; the theorem requires \hat T ≥ T_S_end, so this instance is not covered by the stated assumptions. In the first application, the proof sets Sinit = Starg = Θs;signal, which makes the conclusion TV(pSinit,·,pStarget) ≤ ε trivially true because the two distributions are identical, and does not establish closeness to Sbefore. The proof can likely be repaired by defining T_S_end through the weak-detection threshold and T_S_start through the strong-detection threshold and then proving the resulting interval is nonempty, but as written the all-or-nothing connection is not proved.
  3. [Section 3, Definition 4 and Introduction] The critical window in Definition 4 is defined by the same two total-variation inequalities that Theorem 2 bounds, so the theorem is partly a consistency statement relative to that definition rather than a derivation that critical windows must occur. The genuine content of the paper lies in the non-vacuity of the interval and in the analytic computations in Section 4. The Introduction and abstract present the result as a general 'characterization' of critical windows; the revised manuscript should make the logical structure explicit, namely that Theorem 2 gives a conditional bound and the examples, not the theorem itself, establish the existence and location of critical windows in concrete settings.
minor comments (5)
  1. [Section 3.1] The definitions of T_S_start(ε) and T_S_end(ε) use '∈ sup' and '∈ inf' where equality is clearly intended; additionally, the text should specify what to do if the defining sets are empty or if sup/inf are not attained in discrete index sets.
  2. [Section 4.2.2, Theorem 6] The proof contains multiple notation slips that should be corrected: 'Starg' appears instead of 'Starget', 'Ns+2' and 'N+2' are used inconsistently, and the roles of Sbefore and Safter in the theorem statement appear reversed relative to Definition 4.
  3. [Section 4, Examples 4 and 5] The notation 'ε2' is used throughout for ε², and expressions such as '1−ε2' should be typeset consistently; in Example 5 the constants in T_C_before and the accompanying proof are off by a small additive shift and should be reconciled.
  4. [Section 5, Corollary 1] The proof refers to 'Lemma 15 of [LC24]' and 'Condition 3 on NoiseAmount' without restating either in the present paper; since the hierarchy section is relatively self-contained, adding statements of these two ingredients would improve readability.
  5. [Section 6.2] The operational definition of a critical window in the experiments (a >0.5 jump in probability between consecutive truncation fractions) is not tied back to the ε used in Definition 4; a sentence explaining the relationship, or stating that experiments are only qualitative, would be helpful.

Circularity Check

2 steps flagged · score 4.0 of 10

Theorem 2 is a genuine DPI-based inequality, but the 'critical window' object is defined by the very TV thresholds that Theorem 2 assumes, and the headline LLM example builds the window in by assumption; self-citations to the authors' prior [LC24] carry part of the example computations.

  1. self definitional [Section 3.1, Definition 4 (after Theorem 2)]
    "Define Safter⊂ Sbefore⊂ Θ. For Sbefore, we define T C_before = inf{t∈ I : TV(pSafter_t, pSbefore_t)≤ ε and TV(pSbefore_t, pΘ−Sbefore_t)≥ 1−ε2} (Sinit ≜ Safter; Starget ≜ Sbefore). For Safter, consider T C_after = sup{t∈ I : TV(pSafter_t, pΘ−Safter_t)≥ 1−ε2} (Sinit, Starget ≜ Safter). A critical window is the interval [T C_after, T C_before]."

    The critical window is defined as the set of times at which the forward sub-mixtures satisfy exactly the TV collapse/separation thresholds that Theorem 2 uses as hypotheses (T_S_end and T_S_start). Thus the paper's assertion that critical windows occur where the larger and smaller sub-populations collapse while the larger remains separated from its complement is not an independent prediction: it is true by Definition 4. The theorem still supplies a nontrivial TV bound on the forward-reverse distribution via the data processing inequality, so the reduction is partial. Also, Section 3.1 explicitly admits that T_S_end ≤ T_S_start need not hold ('in general it need not be and our bound would be vacuous'), so the advertised generic emergence is not established by the theorem alone.

  2. self definitional [Section 4.2, Example 4]
    "We assume that these two modes do not differ until some T− T′∈ I. Between T− T′ and T− T′− k, the distributions become nearly disjoint, P_{x∼p_{θharmful}^{T−T′−k}}(x∈ supp(p_{θsafe}^{T−T′−k}))≤ε. ... Then, on component θharmful we have the critical window T C_before = T− T′ and T C_after = T− T′− k."

    In the paper's headline LLM applications (jailbreaks, Yellowstone), the location of the critical window is not derived from Theorem 2; it is imposed by the assumption that the modes agree until T−T′ and become nearly disjoint by T−T′−k. Those assumed times are then named T_C_before and T_C_after. This is the same definitional reduction as Definition 4 applied to a hand-built model, so the example cannot serve as independent evidence that critical windows emerge generically.

full rationale

The central mathematical result, Theorem 2, is not circular by construction: its proof uses the Markov property, the data processing inequality, Bayes' rule, and a coupling argument to bound TV(pSinit,bT,pStarget) in terms of forward-process TV thresholds. That derivation is self-contained given Definition 1 and Lemmas 3-4. However, the paper's announced object of study, the 'critical window', is defined in Definition 4 using the same TV thresholds that appear as the hypotheses of Theorem 2, so the claimed location of critical windows is partly a restatement of the definition. Section 3.1 itself concedes that the interval [T_S_end, T_S_start] can be empty and the bound vacuous; no general condition for nonemptiness is proved, so the abstract's 'emerges generically' overstates what is derived. Example 4 further builds the transition in by assumption rather than predicting it. Some example computations, especially the T_C_before halves in Examples 1 and 2, are imported from the authors' own prior work [LC24] ('The proof for T C_before ... can be found in Appendix B.1 of [LC24]' and 'The proof for T C_before can be found in Section 5.2 of [LC24]'); these are real prior results but are self-citations carrying part of the example evidence. The proof of Theorem 6 also appears to apply Theorem 2 outside its own interval (it sets Tend(ε)=Ns+2 and then uses t=Ns+2−(1−δ2)N*s < Ns+2); this is a correctness gap rather than a circularity, so it is noted but not counted in the circularity score. Overall, the core inequality is independent, but the phenomenon-level claims partially reduce to the definitions and assumptions, and self-citation contributes modestly, giving a score of 4 rather than 0-2.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central theorem introduces no fitted constants; the model parameters in the examples (mu, K, Delta, alpha, inner product of u and v) are explicit inputs to toy data models, not fitted to outcomes. The framework rests on structural axioms: mixture decomposition of features, a fixed Markovian degradation shared by sub-populations, complete observations, exact posterior reverse sampling, monotone TV, and non-vacuity of the window. The non-vacuity condition is acknowledged in the paper but not characterized.

assumptions (5)
  • domain assumption A finite mixture representation of features exists: p = sum w_theta p_theta and each feature is a sub-mixture.
    Section 2.1 introduces this mixture model to 'capture the notion of a feature'. The theorem's Sinit and Starget are arbitrary subsets of Theta, so the characterization is only as meaningful as this decomposition.
  • domain assumption The observation process is Markovian, complete, and becomes less informative as t increases.
    Definition 1 and the surrounding text in Section 2.1. Completeness is used to define the sampler, and decreasing informativeness is used informally to order T_end and T_start.
  • domain assumption The forward degradation kernel is fixed and shared by all sub-populations.
    Section 2.2: 'we will assume that the degradation procedure is fixed'. This is needed to compare pS_t across different S through the same kernel.
  • ad hoc to paper Total variation between forward sub-mixture laws is nonincreasing in t and the interval [T_S_end, T_S_start] is nonempty.
    Used implicitly in the proof of Theorem 2 to move from threshold times to all bT in the interval. Section 3.1 acknowledges that T_S_end <= T_S_start may fail in general, making the theorem vacuous, but monotonicity is never stated as a formal axiom.
  • domain assumption The reverse sampler uses the exact posterior of p; learned score error is ignored.
    Section 2.1: 'we define X ~ p to be the sampler's distribution' and the learning issue is declared orthogonal. For autoregressive models this is exact under next-token prediction, but for diffusion models it idealizes away score approximation error.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Blink of an eye: a simple theory for feature localization in generative models." pith.science (2026). https://pith.science/paper/25HTPJLG

@misc{pith2026250200921,
  author       = {Pith},
  title        = {Pith review of: Blink of an eye: a simple theory for feature localization in generative models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/25HTPJLG}},
  note         = {Machine review of arXiv:2502.00921}
}
read the original abstract

Large language models can exhibit unexpected behavior in the blink of an eye. In a recent computer use demo, a language model switched from coding to Googling pictures of Yellowstone, and these sudden shifts in behavior have also been observed in reasoning patterns and jailbreaks. This phenomenon is not unique to autoregressive models: in diffusion models, key features of the final output are decided in narrow ``critical windows'' of the generation process. In this work we develop a simple, unifying theory to explain this phenomenon using the formalism of stochastic localization samplers. We show that it emerges generically as the generation process localizes to a sub-population of the distribution it models. While critical windows have been studied at length in diffusion models, existing theory heavily relies on strong distributional assumptions and the particulars of Gaussian diffusion. In contrast to existing work our theory (1) applies to autoregressive and diffusion models; (2) makes no distributional assumptions; (3) quantitatively improves previous bounds even when specialized to diffusions; and (4) requires basic tools and no stochastic calculus or statistical-physics-based machinery. We also identify an intriguing connection to the all-or-nothing phenomenon from statistical inference. Finally, we validate our predictions empirically for LLMs and find that critical windows often coincide with failures in problem solving for various math and reasoning benchmarks.

Figures

Figures reproduced from arXiv: 2502.00921 by the authors.

Figure 1
Figure 1. Examples of critical windows for different data modalities and samplers, including reason [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Illustration of a critical window for a cat feature with an image diffusion model. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Intuition with the forward-reverse experiment with [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Structured output plots for LLAMA-3.1-8B-Instruct. [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Probability that the answer is the same as a function of the percentage of the generation [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: Example of critical window for Phi-3-7B-Instruct (bolded). [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]
Figure 7
Figure 7. Figure 7: Examples of critical windows in jailbreaks for LLAMA- [PITH_FULL_IMAGE:figures/full_fig_p038_7.png]
Figure 8
Figure 8. Figure 8: Left-to-right: LLAMA-3.1-8B-Instruct, Phi-3-7B-Instruct, and Qwen-2.5-7B-Instruct barplots of original generation accuracy for generations with (≥ 0.5 jump in probability) and without critical windows (< 0.5) [PITH_FULL_IMAGE:figures/full_fig_p040_8.png]
Figure 9
Figure 9. Figure 9: Probability that the answer is the same as a function of the percentage of the generation re [PITH_FULL_IMAGE:figures/full_fig_p040_9.png]
Figure 10
Figure 10. Figure 10: Probability that the answer is the same as a function of the percentage of the generation [PITH_FULL_IMAGE:figures/full_fig_p041_10.png]
Figure 11
Figure 11. Figure 11: Difference between resampling to obtain the answer versus directly asking the model for the [PITH_FULL_IMAGE:figures/full_fig_p041_11.png]
Figure 12
Figure 12. Figure 12: Temperature versus critical window frequency for LLAMA- [PITH_FULL_IMAGE:figures/full_fig_p047_12.png]
Figure 13
Figure 13. Figure 13: Temperature versus mean critical window size for LLAMA- [PITH_FULL_IMAGE:figures/full_fig_p047_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

78 extracted references · 48 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Hewett, Mojan Javaheripi, Piero Kauffmann, James R

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu,...

    2024

  3. [3]

    Bowman, Ethan Perez, Roger Grosse, and David Duvenaud

    Cem Anil, Esin Durmus, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Nina Rimsky, Meg Tong, Jesse Mu, Daniel Ford, Francesco Mosconi, Rajashree Agrawal, Rylan Schaeffer, Naomi Bashkansky, Samuel Svenningsen, Mike Lambert, Ansh Radhakrishnan, Carson Denison, Evan J Hubinger, Yuntao Bai, Trenton Bricken, Timothy Maxwell, Nicholas Schiefer, Jami...

    work page 2024

  4. [4]

    Aryaman Arora, Dan Jurafsky, Christopher Potts, and Noah D. Goodman. Bayesian scaling laws for in-context learning, 2024

    work page 2024

  5. [5]

    Detecting language model attacks with perplexity, 2023

    Gabriel Alon and Michael Kamfonas. Detecting language model attacks with perplexity, 2023

    work page 2023

  6. [6]

    A latent variable model approach to pmi-based word embeddings, 2019

    Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A latent variable model approach to pmi-based word embeddings, 2019

    work page 2019

  7. [7]

    Sampling from mean-field gibbs measures via diffusion processes

    Ahmed El Alaoui, Andrea Montanari, and Mark Sellke. Sampling from mean-field gibbs measures via diffusion processes. arXiv preprint arXiv:2310.08912 , 2023

    arXiv 2023

  8. [8]

    Developing a computer use model

    Anthropic . Developing a computer use model. https://www.anthropic.com/news/developing-computer-use, 2024

    work page 2024

Show all 78 references
  1. [9]

    What learning algorithm is in-context learning? investigations with linear models, 2023

    Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models, 2023

    2023

  2. [10]

    Dynamical regimes of diffusion models, 2024

    Giulio Biroli, Tony Bonnaire, Valentin de Bortoli, and Marc Mézard. Dynamical regimes of diffusion models, 2024

    2024

  3. [11]

    All-or-nothing statistical and computational phase transitions in sparse spiked matrix estimation, 2020

    Jean Barbier, Nicolas Macris, and Cynthia Rush. All-or-nothing statistical and computational phase transitions in sparse spiked matrix estimation, 2020

    2020

  4. [12]

    Obfuscated activations bypass llm latent-space defenses, 2024

    Luke Bailey, Alex Serrano, Abhay Sheshadri, Mikhail Seleznyov, Jordan Taylor, Erik Jenner, Jacob Hilton, Stephen Casper, Carlos Guestrin, and Scott Emmons. Obfuscated activations bypass llm latent-space defenses, 2024

    2024

  5. [13]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1803.05457, 2018

    2018 arXiv

  6. [14]

    Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions

    Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May...

    2023

  7. [15]

    Localization schemes: A framework for proving mixing bounds for markov chains

    Yuansi Chen and Ronen Eldan. Localization schemes: A framework for proving mixing bounds for markov chains. In 2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS) , pages 110--122. IEEE, 2022

    2022

  8. [16]

    Perception prioritized training of diffusion models

    Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11462--11471, 2022

    2022

  9. [17]

    Statistical and computational phase transitions in group testing

    Amin Coja-Oghlan, Oliver Gebhard, Max Hahn-Klimroth, Alexander S Wein, and Ilias Zadik. Statistical and computational phase transitions in group testing. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory , volume 178 of Proce...

    2022

  10. [18]

    Pappas, and Eric Wong

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries, 2023

    2023

  11. [19]

    Enhancing chat language models by scaling high-quality instructional conversations, 2023

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations, 2023

    2023

  12. [20]

    A survey on in-context learning, 2024

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. A survey on in-context learning, 2024

    2024

  13. [21]

    Sampling from the sherrington-kirkpatrick gibbs measure via algorithmic stochastic localization

    Ahmed El Alaoui, Andrea Montanari, and Mark Sellke. Sampling from the sherrington-kirkpatrick gibbs measure via algorithmic stochastic localization. In 2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS) , pages 323--334. IEEE, 2022

    2022

  14. [22]

    Thin shell implies spectral gap up to polylog via a stochastic localization scheme

    Ronen Eldan. Thin shell implies spectral gap up to polylog via a stochastic localization scheme. Geometric and Functional Analysis , 23(2):532--569, 2013

    2013

  15. [23]

    Taming correlations through entropy-efficient measure decompositions with applications to mean-field approximation

    Ronen Eldan. Taming correlations through entropy-efficient measure decompositions with applications to mean-field approximation. Probability Theory and Related Fields , 176(3-4):737--755, 2020

    2020

  16. [24]

    Challenges and solutions for aging adults

    Gemini . Challenges and solutions for aging adults. https://gemini.google.com/share/6d141b742a13, 2024

    2024

  17. [25]

    Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio

    Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014

    2014

  18. [26]

    Jailbroken llama-3.1-8b-instruct via lora

    grimjim . Jailbroken llama-3.1-8b-instruct via lora. https://huggingface.co/grimjim/Llama-3.1-8B-Instruct-abliterated_via_adapter, 2024

    2024

  19. [27]

    What can transformers learn in-context? a case study of simple function classes, 2023

    Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes, 2023

    2023

  20. [28]

    The journey, not the destination: How data guides diffusion models

    Kristian Georgiev, Joshua Vendrow, Hadi Salman, Sung Min Park, and Aleksander Madry. The journey, not the destination: How data guides diffusion models. arXiv preprint arXiv:2312.06205 , 2023

    2023 arXiv

  21. [29]

    High-dimensional regression with binary coefficients

    David Gamarnik and Ilias Zadik. High-dimensional regression with binary coefficients. estimating squared error and a phase transition, 2019

    2019

  22. [30]

    Automated multi-turn red-teaming with cascade

    Haize Labs . Automated multi-turn red-teaming with cascade. https://blog.haizelabs.com/posts/cascade, 2024

    2024

  23. [31]

    A trivial jailbreak against llama 3

    Haize Labs . A trivial jailbreak against llama 3. https://github.com/haizelabs/llama3-jailbreak, 2024

    2024

  24. [32]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations , 2021

    2021

  25. [33]

    Measuring mathematical problem solving with the math dataset, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021

    2021

  26. [34]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840--6851, 2020

    2020

  27. [35]

    Sampling from spherical spin glasses in total variation via algorithmic stochastic localization

    Brice Huang, Andrea Montanari, and Huy Tuan Pham. Sampling from spherical spin glasses in total variation via algorithmic stochastic localization. arXiv preprint arXiv:2404.15651 , 2024

    2024 arXiv

  28. [36]

    What is in your safe data? identifying benign data that breaks safety, 2024

    Luxi He, Mengzhou Xia, and Peter Henderson. What is in your safe data? identifying benign data that breaks safety, 2024

    2024

  29. [37]

    Critical windows: non-asymptotic theory for feature emergence in diffusion models, 2024

    Marvin Li and Sitan Chen. Critical windows: non-asymptotic theory for feature emergence in diffusion models, 2024

    2024

  30. [38]

    Logiqa: A challenge dataset for machine reading comprehension with logical reasoning

    Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intellig...

    2020

  31. [39]

    Bowman, and Ethan Perez

    Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish...

    2023

  32. [40]

    T ruthful QA : Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. T ruthful QA : Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 3214--3252, Dublin, Ireland, May 2022. Associati...

    2022

  33. [41]

    Llm defenses are not robust to multi-turn human jailbreaks yet, 2024

    Nathaniel Li, Ziwen Han, Ian Steneker, Willow Primack, Riley Goodside, Hugh Zhang, Zifan Wang, Cristina Menghini, and Summer Yue. Llm defenses are not robust to multi-turn human jailbreaks yet, 2024

    2024

  34. [42]

    Let's verify step by step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050 , 2023

    2023 arXiv

  35. [43]

    Critical tokens matter: Token-level contrastive estimation enhances llm's reasoning capability, 2024

    Zicheng Lin, Tian Liang, Jiahao Xu, Xing Wang, Ruilin Luo, Chufan Shi, Siheng Li, Yujiu Yang, and Zhaopeng Tu. Critical tokens matter: Token-level contrastive estimation enhances llm's reasoning capability, 2024

    2024

  36. [44]

    Discrete diffusion modeling by estimating the ratios of the data distribution, 2024

    Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution, 2024

    2024

  37. [45]

    The unlocking spell on base llms: Rethinking alignment via in-context learning, 2023

    Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base llms: Rethinking alignment via in-context learning, 2023

    2023

  38. [46]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. In The Twelfth International Conference on Learning Representations , 2024

    2024

  39. [47]

    Program induction by rationale generation: Learning to solve and explain algebraic word problems

    Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages ...

    2017

  40. [48]

    SDE dit: Guided image synthesis and editing with stochastic differential equations

    Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDE dit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations , 2022

    2022

  41. [49]

    Sharp thresholds in inference of planted subgraphs

    Elchanan Mossel, Jonathan Niles-Weed, Youngtak Sohn, Nike Sun, and Ilias Zadik. Sharp thresholds in inference of planted subgraphs. In Gergely Neu and Lorenzo Rosasco, editors, Proceedings of Thirty Sixth Conference on Learning Theory , volume 195 of Proceedings of Machine Lea...

    2023

  42. [50]

    Sampling, diffusions, and stochastic localization, 2023

    Andrea Montanari. Sampling, diffusions, and stochastic localization, 2023

    2023

  43. [51]

    Sampling, diffusions, and stochastic localization

    Andrea Montanari. Sampling, diffusions, and stochastic localization. arXiv preprint arXiv:2305.10690 , 2023

    2023 arXiv

  44. [52]

    Posterior sampling from the spiked models via diffusion processes

    Andrea Montanari and Yuchen Wu. Posterior sampling from the spiked models via diffusion processes. arXiv preprint arXiv:2304.11449 , 2023

    2023 arXiv

  45. [53]

    Feder Cooper, Daphne Ippolito, Christopher A

    Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models, 2023

    2023

  46. [54]

    The all-or-nothing phenomenon in sparse tensor pca

    Jonathan Niles-Weed and Ilias Zadik. The all-or-nothing phenomenon in sparse tensor pca. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 17674--17684. Curran Associates, Inc., 2020

    2020

  47. [55]

    all" for

    Jonathan Niles-Weed and Ilias Zadik. It was "all" for "nothing": sharp phase transitions for noiseless discrete channels, 2023

    2023

  48. [56]

    Detecting misbehavior in frontier reasoning models

    OpenAI. Detecting misbehavior in frontier reasoning models. OpenAI Research , 2025

    2025

  49. [57]

    How deep neural networks learn compositional data: The random hierarchy model

    Leonardo Petrini, Francesco Cagnetta, Umberto M Tomasini, Alessandro Favero, and Matthieu Wyart. How deep neural networks learn compositional data: The random hierarchy model. arXiv preprint arXiv:2307.02129 , 2023

    2023 arXiv

  50. [58]

    Safety alignment should be made more than just a few tokens deep, 2024

    Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety alignment should be made more than just a few tokens deep, 2024

    2024

  51. [59]

    Spontaneous symmetry breaking in generative diffusion models

    Gabriel Raya and Luca Ambrogioni. Spontaneous symmetry breaking in generative diffusion models. In Thirty-seventh Conference on Neural Information Processing Systems , 2023

    2023

  52. [60]

    Xstest: A test suite for identifying exaggerated safety behaviours in large language models, 2024

    Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models, 2024

    2024

  53. [61]

    B. Roos. Binomial approximation to the poisson binomial distribution: The krawtchouk expansion. Theory of Probability & Its Applications , 45(2):258--272, 2001

    2001

  54. [62]

    The all-or-nothing phenomenon in sparse linear regression, 2019

    Galen Reeves, Jiaming Xu, and Ilias Zadik. The all-or-nothing phenomenon in sparse linear regression, 2019

    2019

  55. [63]

    Limits on support recovery with probabilistic models: An information-theoretic framework, 2016

    Jonathan Scarlett and Volkan Cevher. Limits on support recovery with probabilistic models: An information-theoretic framework, 2016

    2016

  56. [64]

    Consistency models, 2023

    Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models, 2023

    2023

  57. [65]

    Probing the latent hierarchical structure of data via diffusion models, 2024

    Antonio Sclocchi, Alessandro Favero, Noam Itzhak Levi, and Matthieu Wyart. Probing the latent hierarchical structure of data via diffusion models, 2024

    2024

  58. [66]

    A phase transition in diffusion models reveals the hierarchical nature of data

    Antonio Sclocchi, Alessandro Favero, and Matthieu Wyart. A phase transition in diffusion models reveals the hierarchical nature of data. Proceedings of the National Academy of Sciences , 122(1):e2408799121, 2025

    2025

  59. [67]

    A strongreject for empty jailbreaks, 2024

    Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and Sam Toyer. A strongreject for empty jailbreaks, 2024

    2024

  60. [68]

    Truong, Matthew Aldridge, and Jonathan Scarlett

    Lan V. Truong, Matthew Aldridge, and Jonathan Scarlett. On the all-or-nothing behavior of bernoulli group testing, 2021

    2021

  61. [69]

    Truong and Jonathan Scarlett

    Lan V. Truong and Jonathan Scarlett. Support recovery in the phase retrieval model: Information-theoretic fundamental limits, 2020

    2020

  62. [70]

    High-Dimensional Probability: An Introduction with Applications in Data Science

    Roman Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press

  63. [71]

    Probability in high dimension

    Ramon van Handel. Probability in high dimension. 2016

    2016

  64. [72]

    Jailbroken: How does llm safety training fail?, 2023

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail?, 2023

    2023

  65. [73]

    An explanation of in-context learning as implicit bayesian inference, 2022

    Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference, 2022

    2022

  66. [74]

    Improving alignment and robustness with circuit breakers, 2024

    Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, and Dan Hendrycks. Improving alignment and robustness with circuit breakers, 2024

    2024

  67. [75]

    Dissecting learning and forgetting in language model finetuning

    Xiao Zhang and Ji Wu. Dissecting learning and forgetting in language model finetuning. In The Twelfth International Conference on Learning Representations , 2024

    2024

  68. [76]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023

    2023

  69. [77]

    Adversarial contrastive decoding: Boosting safety alignment of large language models via opposite prompt optimization, 2024

    Zhengyue Zhao, Xiaoyun Zhang, Kaidi Xu, Xing Hu, Rui Zhang, Zidong Du, Qi Guo, and Yunji Chen. Adversarial contrastive decoding: Boosting safety alignment of large language models via opposite prompt optimization, 2024

    2024

  70. [78]

    What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization, 2023

    Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang. What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization, 2023

    2023

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.