Pith. sign in

REVIEW 4 major objections 5 minor 16 cited by

Energy-Based Transformers are Scalable Learners and Thinkers

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Energy-Based Transformers out-scale standard models by up to 35%

desk verdict Serious architecture work with public code, but the central scaling-rate claim is not supported because EBTs are compared at 6.66x the FLOPs per step. read the letter →

arxiv 2507.02092 v1 pith:5ZGAYXVI submitted 2025-07-02 cs.LG cs.AIcs.CLcs.CV

Alexi Gladstone , Ganesh Nanduru , Md Mofijul Islam , Peixuan Han , Hyeonjeong Ha , Aman Chadha , Yilun Du , Heng Ji
show 2 more authors
Jundong Li Tariq Iqbal
This is my paper
classification cs.LGcs.AIcs.CLcs.CV
keywords Energy-BasedModelsSystem2ThinkingScalingLawsInference-TimeComputeSelf-VerificationImageDenoisingAutoregressiveModelingUncertaintyEstimation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a Transformer can be trained as an Energy-Based Model to make predictions by minimizing a learned energy, and that doing so turns System 2 thinking—extra computation at inference—into a byproduct of unsupervised pretraining. Concretely, EBTs are reported to scale faster than the standard Transformer++ recipe during pretraining on every axis tested (data, batch size, depth, parameters, FLOPs, and width), with up to a 35% higher scaling rate. At inference, spending more optimization steps or sampling and self-verifying candidates improves language perplexity by up to 29% more than Transformer++, and on image denoising EBTs beat DiTs while using 1% of the forward passes. If these trends hold at larger scale, EBTs would be the first architecture to out-scale the dominant feed-forward recipe while also getting better with thinking.

What carries the argument

The load-bearing object is the learned energy landscape $E_\theta(x,\hat y)$: a scalar field over input-candidate pairs whose minimum, reached by gradient descent, is the prediction. Training unrolls this optimization and backpropagates the final loss through it, which requires Hessian-vector products (linear in model size) rather than full Hessians. Three regularization techniques make the landscape smooth and convex enough to think over: a replay buffer that simulates longer optimization trajectories, Langevin noise for exploration, and randomized step size and number of optimization steps. The decoder-only EBT implementation also solves an engineering problem—each row's predicted next token differs, so the causal attention scores cannot be one matrix multiplication—by appending predicted representations as a superdiagonal in a $2N$-length sequence.

What would settle it

Train an EBT and a Transformer++ of the same size at about 7B parameters on about 1T tokens and fit their validation-loss scaling curves; if EBT's slope advantage over Transformer++ does not remain positive (or flips), the paper's extrapolation to foundation scale is false.

Watch

Extended reading notes

Core claim

The paper's central discovery is that explicitly learning to verify the compatibility between an input and a candidate prediction—rather than learning to generate directly—makes both learning and thinking scale faster. An EBT assigns an energy (unnormalized likelihood) to every context-prediction pair; a prediction is made by starting from random noise and following the gradient of that energy to lower-energy candidates. Because the same network that produces the energy also serves as the verifier, no external reward model or verifier is needed, and the approach works for discrete text and continuous video/image spaces alike. The paper reports that EBTs generalize better than Transformer++ on most downstream tasks even when pretraining perplexity is slightly worse, and that thinking gains grow linearly with out-of-distribution shift.

Load-bearing premise

The central claim collapses if EBTs' faster scaling slope, measured on models up to 708M parameters and roughly 1e21 FLOPs, does not persist when data and model size grow by roughly a factor of a thousand.

Editorial extensions

If this is right

  • If the scaling-rate advantage persists, EBTs trained at 1000x data and model scale would beat Transformer++ even without inference-time thinking.
  • Because thinking gains grow as data shifts out-of-distribution, EBTs offer a mechanism for robust OOD generalization at inference.
  • Self-verification (best-of-N over candidate predictions) improves with training scale, so larger EBTs should benefit more from thinking, not less.
  • In continuous modalities, EBTs provide per-prediction uncertainty from the energy scalar without discretization or auxiliary losses.
  • Image denoising with EBTs reaches better PSNR with 99% fewer forward passes than DiTs, suggesting far cheaper diffusion-style inference.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the energy values are well calibrated, EBTs could decide when to stop thinking, allocating compute per token based on energy convergence rather than a fixed budget; this is testable by comparing energy convergence with downstream correctness.
  • The verification-as-generation view suggests EBTs could serve as a drop-in verifier for outputs of fast System 1 generators, replacing learned reward models; the paper only hints at this complementary use.
  • The convex-landscape training objective explains the reported failure on multimodal data (e.g., text-to-image) as averaging of modes; adding multiple basins or mixture energies would be a direct test of that explanation.
  • The paper's headline extrapolation to foundation scale rests on slopes measured below 708M parameters; a 7B-parameter, 1T-token comparison would either confirm or overturn it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Energy-Based Transformers (EBTs), a Transformer architecture trained as an energy-based model by unrolling a small number of gradient-descent steps on a candidate prediction and backpropagating through them. The authors claim that EBTs out-scale the standard Transformer++ recipe during pretraining across data, batch size, depth, parameters, FLOPs, and embedding width, with up to 35% higher scaling rates, and that at inference EBTs support System 2 Thinking through longer optimization and self-verification, improving language perplexity and image denoising relative to Transformer++ and Diffusion Transformers. The core evidence comes from small models (up to 708M parameters, at most about 10^21 FLOPs) with one training run per configuration. The paper is clearly written and includes extensive appendices with implementation details, pseudocode, and release of code. However, the central scaling-rate claim rests on comparisons that are not FLOP-matched, since EBTs consume substantially more compute per training step than Transformer++ under the authors' own accounting.

Significance. If the scaling-rate advantage were established, this would be a significant result: it would be the first demonstration that an energy-based, optimization-centric architecture out-scales the dominant feed-forward Transformer recipe on multiple axes while also providing a natural mechanism for inference-time verification. The paper's strengths include a concrete architecture with efficient causal attention for input-space predictions, a thoughtful discussion of energy-landscape regularization, a public code release, and a broad set of experiments spanning language, video, and image domains. The FLOP calculations in Section D.5 are unusually transparent, which is commendable but also highlights the central weakness: the claimed out-scaling across FLOPs is not supported by matched-compute comparisons. The thinking results are interesting but the current evidence is mostly qualitative or based on a small set of selected benchmarks, so the significance of the thinking claims is proportionate to the modest scale of the experiments.

major comments (4)
  1. [§D.5, Figures 4 and 5] The central claim that EBTs out-scale Transformer++ during pretraining is not yet supported because the comparisons are not FLOP-matched. As the authors state in Section D.5, an S1 EBT trained with two optimization steps consumes roughly 6.66× the FLOPs of a comparable Transformer++ step on the same tokens and outer steps. For the parameter and FLOP scaling experiments in D.1.1, all models are pretrained for 105k outer steps, so EBT points are systematically shifted to much larger cumulative compute. A steeper loss-vs-tokens curve under a 6.66× compute multiplier does not establish data efficiency or a higher scaling rate; it can simply reflect more compute per token. The separate FLOPs axis in Figure 5b uses these same approximate estimates, so the headline out-scaling across FLOPs is not established until matched-FLOP curves are provided, either by reducing EBT steps or by plotting loss against cumulative FLOPs with comparable total compute.
  2. [§D.1.1, Figures 4 and 5] All scaling rates are computed from a single training run per configuration, with no reported seeds, error bars, or confidence intervals. Scaling-law exponents are known to be fragile to implementation details, as the paper itself notes in D.1.1, and with only a few model sizes (Table D.1), the reported 35% differences could be within run-to-run or hyperparameter sensitivity. Multiple seeds, or at least a sensitivity analysis around the fitted exponents, are needed before claiming that EBTs consistently out-scale Transformer++ across all six axes.
  3. [Table 3 and §4.1.3] The generalization claim that EBTs generalize better than Transformer++ is based on four downstream tasks chosen deliberately for reasoning alignment, with EBT showing better perplexity on three of four tasks despite worse pretraining perplexity. There is no statistical significance testing, no error bars, and no evidence that the result holds across a broader, pre-registered set of benchmarks. The 29% more than Transformer++ thinking improvement in Figure 6a uses the same four tasks, so the selection issue propagates into the headline thinking claim. The paper should either broaden the evaluation or temper the claim to 'on the selected tasks' rather than a general generalization advantage.
  4. [Table 4 and §4.3] The bidirectional image-denosing comparison to DiT, while suggestive, does not yet establish that EBTs are broadly better than diffusion Transformers. The DiT baseline is used with its standard training setup and then evaluated with recursive DDIM denoising up to 300 forward passes, whereas EBTs are trained with a different objective and use only 3 passes. The comparison may conflate architectural differences with differences in training objectives, noising schedules, and inference procedures. At minimum, the paper should report matched-compute curves (e.g., PSNR vs. total FLOPs) for both models and clarify whether DiT was given any opportunity to train with a comparable number of optimization-style steps or a verification loss.
minor comments (5)
  1. [§D.5 vs. §H.3] There is an internal inconsistency in the reported FLOP overhead: Section D.5 says a single second-order optimization step is roughly 3.33× more expensive than a standard Transformer step, while the FAQ in H.3 says a single step makes training about a constant 1.66× as expensive. The authors should reconcile these numbers.
  2. [§4.1.1] The definition of scaling rate is never made explicit. The paper should state whether the reported percentages are slopes in log-log space, and how the fits were performed (e.g., least squares over which points).
  3. [Figure 6b and §B.1] The text says 'increasing up to 10%−14% from 4%−8%' in Figure 6b, but the figure caption and surrounding text do not clearly separate the two series. Please clarify the exact numbers and units in the caption.
  4. [§4.1.2] The claim that Transformer++ cannot improve performance at all from thinking longer is true by construction, since there is no per-token iterative optimization mechanism; this comparison is therefore not an empirical demonstration of EBT superiority but an architectural consequence. The text should present it as such.
  5. [§3.3, Algorithm 2] The self-verification procedure in Algorithm 2 returns the minimum-energy candidate, but the paper does not discuss how the energy scalar is calibrated across different inputs or whether comparing energies across different contexts is valid. A sentence on this would be useful.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: scaling and thinking results are measured against external baselines; self-citations are methodological and not load-bearing.

full rationale

The paper's central claims—higher pretraining scaling rates, System 2 Thinking improvements, and downstream generalization—are empirical measurements against external baselines (Transformer++ and DiT), not derivations from the model definition. The training objective and algorithms define the EBT procedure, but the scaling rates, perplexity values, PSNR/NFE curves, and ablations are evaluated rather than imposed by construction. Citations to prior EBM work by co-author Yilun Du ([48,51,67]) supply the training method and S2 design choices, yet the load-bearing evidence for the paper's scaling and generalization claims is the paper's own controlled experiments and ablations (Figures 4,5,6,12; Tables 2,3,4), with external baselines. The paper's own limitations (Section 7, A.8, D.5) concede that FLOP estimates are approximate and that scales beyond 800M parameters / ~10^21 FLOPs are untested; these are correctness and extrapolation risks, not circularity. The FLOP-matched comparison concern (EBT uses ~6.66x FLOPs per step per D.5) is a validity threat to the FLOP-scaling comparison, but it does not reduce any equation to its inputs: loss values are measured and slopes are fitted, not forced by definition. No step in the derivation chain is self-referential or equivalent to its inputs by construction.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The paper introduces EBT-specific hyperparameters (step count, step size, Langevin noise, replay buffer, learning-rate adjustment) that are fit per task and directly control the optimization procedure that defines the model. There are no new physical or unverified mathematical constants; the main risk is the assumption that the empirical energy landscape, trained with only 2-3 unrolled steps, remains valid for longer chains and larger scales.

free parameters (5)
  • Optimization step count (N) = 2-3 optimization steps
    The number of gradient steps is a hyperparameter that directly controls the depth of unrolled optimization; the paper tunes it per experiment and admits more steps were not stable.
  • Optimization step size (alpha) = 500 (text), 30,000 (video)
    The step size determines the update magnitude of predictions. The paper reports it must be tuned per modality and that poor choices destabilize training.
  • Langevin noise (sigma) = 3 (S2 NLP)
    Noise magnitude is a tuned hyperparameter for landscape regularization; the ablation shows it changes thinking behavior.
  • Replay buffer size = not specified
    Used in S2 training as a landscape regularization technique, but no size is reported.
  • Lower learning rate for EBT video models = factor 3 lower than Transformer++
    The authors report that EBTs required a lower learning rate for training stability; this is a fitted adjustment.
assumptions (4)
  • domain assumption The energy landscape can be made sufficiently convex and smooth via the proposed regularization techniques so that gradient descent converges to good predictions.
    The paper relies on this to justify that energy minimization produces valid predictions and that thinking improvements will scale.
  • ad hoc to paper Backpropagating through a small number of unrolled optimization steps (2-3) is enough to learn a usable energy landscape.
    The paper found more than 2-3 steps unstable, so the entire approach is evaluated only for this short unroll.
  • domain assumption Perplexity is a sufficiently linear metric to extrapolate downstream gains across OOD datasets.
    The paper cites a survey that scaling laws are fragile, but still reports percentage improvements and linear OOD trends based on perplexity.
  • domain assumption The Hessian-vector product FLOP estimate (10N per token) is accurate enough for the FLOP scaling comparison.
    The paper explicitly states these estimates are approximate and that they had no existing library for HVP FLOPs.
invented entities (1)
  • EBT energy scalar as an unnormalized likelihood and verifier
    purpose: Serves as both uncertainty estimate and prediction verifier at each step of inference
    The energy is internally trained and there is no external calibration showing the scalar is a well-calibrated verifier. Figures show correlations, but no quantitative calibration or comparison to human/external verification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Energy-Based Transformers are Scalable Learners and Thinkers." pith.science (2026). https://pith.science/paper/5ZGAYXVI

@misc{pith2026250702092,
  author       = {Pith},
  title        = {Pith review of: Energy-Based Transformers are Scalable Learners and Thinkers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5ZGAYXVI}},
  note         = {Machine review of arXiv:2507.02092}
}
read the original abstract

Inference-time computation techniques, analogous to human System 2 Thinking, have recently become popular for improving model performances. However, most existing approaches suffer from several limitations: they are modality-specific (e.g., working only in text), problem-specific (e.g., verifiable domains like math and coding), or require additional supervision/training on top of unsupervised pretraining (e.g., verifiers or verifiable rewards). In this paper, we ask the question "Is it possible to generalize these System 2 Thinking approaches, and develop models that learn to think solely from unsupervised learning?" Interestingly, we find the answer is yes, by learning to explicitly verify the compatibility between inputs and candidate-predictions, and then re-framing prediction problems as optimization with respect to this verifier. Specifically, we train Energy-Based Transformers (EBTs) -- a new class of Energy-Based Models (EBMs) -- to assign an energy value to every input and candidate-prediction pair, enabling predictions through gradient descent-based energy minimization until convergence. Across both discrete (text) and continuous (visual) modalities, we find EBTs scale faster than the dominant Transformer++ approach during training, achieving an up to 35% higher scaling rate with respect to data, batch size, parameters, FLOPs, and depth. During inference, EBTs improve performance with System 2 Thinking by 29% more than the Transformer++ on language tasks, and EBTs outperform Diffusion Transformers on image denoising while using fewer forward passes. Further, we find that EBTs achieve better results than existing models on most downstream tasks given the same or worse pretraining performance, suggesting that EBTs generalize better than existing approaches. Consequently, EBTs are a promising new paradigm for scaling both the learning and thinking capabilities of models.

Figures

Figures reproduced from arXiv: 2507.02092 by the authors.

Figure 1
Figure 1. Autoregressive Architecture Comparison. (a) Autoregressive (AR) Transformer is the most common, with (b) RNNs becoming more popular recently [22, 23]. (c) Diffusion Transform￾ers [26] (which are often bidirectional but can also be autoregressive) are the most similar to EBT, being able to dynamically allocate computation during inference, but predict the noise rather than the energy [27, 28]. Consequently, diffusion… view at source ↗
Figure 2
Figure 2. EBT for Autoregressive Modeling. Each blue box corresponds to a different prediction based on the current step of the thinking process, where the initial prediction starts as random. At each step, a new prediction is fed into the model, which gives an energy scalar for the prediction’s current compatibility (unnormalized likelihood) with the context (Facets 2 and 3). Then, the gradient of this energy with respect to… view at source ↗
Figure 3
Figure 3. Thinking Process Visualization. A learned energy landscape and its optimization through gradient descent, interpreted as a thinking process. In this example, the model predicts a distribution over text tokens, progressively shifting from an initial random distribution toward the target distribution. At each step, the EBM assigns an energy scalar indicating how compatible the current prediction is with the context, v… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Language Learning Scalability—Data, Batch Size, and Depth. A comparison between the scaling of the Transformer++ recipe [87] and EBTs across data, batch size, and depth during pretraining. On all of these axes, EBTs out-scale the Transformer++ recipe significantly, ind…
Figure 5
Figure 5. Figure 5: Language Learning Scalability—Parameters, FLOPs, and Width. Pretraining scaling comparisons between the Transformer++ recipe [87] and EBTs across model size (parameters), compute (FLOPs), and width (embedding dimension). EBTs slightly out-scale the Transformer++ in FLO…
Figure 6
Figure 6. Figure 6: EBT Thinking Analysis. (a) Mean performance degradation of the standard Transformer++ recipe [87] and the Energy-Based Transformer (EBT) on four Out of Distribution (OOD) datasets. While the Transformer++ cannot reduce perplexity at a per-token level, EBTs can by perfo…
Figure 7
Figure 7. Figure 7: OOD Thinking Performance. As the data becomes more OOD, thinking leads to greater performance improvements, with a roughly linear trend. These findings highlight that EBTs thinking is especially critical for robust generalization to OOD data. Performance is measured on…
Figure 8
Figure 8. Figure 8: Learning Uncertainty on Text Results. EBTs learn to vary uncertainty across text tokens without any explicit supervision. As an example, in both (a) and (b), simple tokens such as “.”, “is”, “a”, “but”, or “the” have lower energies across inference-time optimization (t…
Figure 9
Figure 9. Figure 9: Video Learning Scalability—Width and Parameters. The minimum validation loss achieved on the Something Something V2 (SSV2) dataset. While EBTs achieve higher validation loss than the Transformer++ at smaller scales, the scaling rate is more than 33% higher, suggesting …
Figure 10
Figure 10. Figure 10: Qualitative OOD Image Denoising. EBTs achieve better denoising quality during inference while using one step for every 100 denoising steps of a DiT. The overall image quality of EBT denoised images is less blurry than images denoised by DiT. (OOD) noisier images. The …
Figure 11
Figure 11. Figure 11: Learning Uncertainty on Video Re￾sults. In line with cognitive Facet 2, EBTs learn to express uncertainty across continuous video frames without supervision. At the start of the video, uncertainty is high (high energy) because the frame is mostly empty and the scene i…
Figure 12
Figure 12. Figure 12: Image Denoising Thinking Scala￾bility. A comparison between EBT and DiT on image denoising given a different number of for￾ward passes. EBTs require only 1% of the forward passes used by DiT to achieve comparable or better PSNR. Further, the scaling rate of PSNR impro…

Discussion (0). Sign in to comment.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Text Dictates, Music Decorates: Energy-based Attention for Editable Dance Motion Generation

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    STREAM decouples text and music conditioning in a diffusion transformer via AdaLN for structure and BEAM for beats, plus new Motorica++ dataset and editability metrics, claiming SOTA music alignment with preserved semantics.

  2. One Scale at a Time: Scale-Autoregressive Modeling for Fluid Flow Distributions

    cs.CE 2026-04 conditional novelty 7.0 of 10

    Scale-autoregressive modeling (SAR) samples fluid flow distributions hierarchically from coarse to fine resolutions on meshes, achieving lower distributional error and 2-7x faster runtime than diffusion or flow-matchi...

  3. The Phase Is the Gradient: Equilibrium Propagation for Frequency Learning in Kuramoto Networks

    cs.LG 2026-04 unverdicted novelty 7.0 of 10

    In Kuramoto networks at equilibrium, weak nudging makes phase displacement the exact gradient of loss w.r.t. natural frequencies, enabling frequency learning that beats weight learning and resolves convergence via spe...

  4. Learning 3D Reconstruction with Priors in Test Time

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    Test-time constrained optimization incorporates priors into pre-trained multiview transformers via self-supervised losses and penalty terms to improve 3D reconstruction accuracy.

  5. Training Large Language Models to Reason in a Continuous Latent Space

    cs.CL 2024-12 unverdicted novelty 7.0 of 10

    Coconut lets LLMs perform reasoning directly in continuous latent space by recycling hidden states as inputs, outperforming standard chain-of-thought on search-intensive logical tasks with better accuracy-efficiency t...

  6. Text Dictates, Music Decorates: Energy-based Attention for Editable Dance Motion Generation

    cs.AI 2026-06 conditional novelty 6.5 of 10

    STREAM decouples text (via AdaLN) from music (via energy-based BEAM attention) to generate editable, musically aligned dance motions with a new annotated dataset and editability metric.

  7. Solver-Hard Is Not Model-Hard: A Hardness-Controlled Diagnostic for LLM Constraint Reasoning

    cs.LG 2026-07 conditional novelty 6.0 of 10

    At matched clause density and width, LLM verdict accuracy and token spend do not track CDCL solver conflict counts across proof-hard vs proof-easy Tseitin formulas.

  8. Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    FPRM is a Transformer-based model using fixed-point convergence for adaptive halting in looped architectures, claimed effective on Sudoku, Maze, state-tracking, and ARC-AGI benchmarks.

  9. Revisiting Transformer Layer Parameterization Through Causal Energy Minimization

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    CEM recasts Transformer layers as energy minimization steps, enabling constrained parameterizations like weight sharing and low-rank interactions that match standard baselines in 100M-scale language modeling.

  10. C-voting: Confidence-Based Test-Time Voting without Explicit Energy Functions

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    C-voting improves recurrent reasoning models by selecting among multiple latent trajectories the one with highest average top-1 probability, achieving 4.9% better Sudoku-hard accuracy than energy-based voting and outp...

  11. On the Spatiotemporal Dynamics of Generalization in Neural Networks

    cs.LG 2026-02 unverdicted novelty 6.0 of 10

    Deriving a neural cellular automaton from locality, symmetry, and stability postulates produces 100% accurate addition generalization from 16-digit to 1-million-digit inputs.

  12. Learning Latent Energy-Based Models via Interacting Particle Langevin Dynamics

    stat.ML 2025-10 conditional novelty 6.0 of 10

    A particle Langevin algorithm (EBIPLA) trains latent energy-based models via maximum marginal likelihood, with convergence bounds and competitive image generation.

  13. Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Training on the best of K generated candidates improves image, video, and language generative models, with the reported gains growing with scale and enabling single-pass end-to-end generation.

  14. Memoir: Should a Model Write to Its Memory While It Thinks?

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Writing to fast memory during pondering slows associative-recall learning at a fixed budget, but does not reduce final performance once training is long enough.

  15. Advancing Optimal Subset Oracle via Learning Relaxation of Neural Set Functions

    cs.LG 2026-07 conditional novelty 5.0 of 10

    A learned continuous relaxation of neural set functions can replace Monte Carlo ELBO gradients in optimal-subset oracles, improving efficiency and accuracy under weak submodularity assumptions.

  16. Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models

    cs.AI 2025-03 unverdicted novelty 5.0 of 10

    The paper unifies perspectives on Long CoT in reasoning LLMs by introducing a taxonomy, detailing characteristics of deep reasoning and reflection, and discussing emergence phenomena and future directions.

Reference graph

Works this paper leans on

162 extracted references · 19 canonical work pages · cited by 15 Pith papers

  1. [1]

    Thinking, fast and slow

    Daniel Kahneman. Thinking, fast and slow. macmillan, 2011. 1, 3, 30

    2011

  2. [2]

    Dual-process theories of reasoning: Contemporary issues and developmental applications

    Jonathan St BT Evans. Dual-process theories of reasoning: Contemporary issues and developmental applications. Developmental review, 31(2-3):86–102, 2011

    2011

  3. [3]

    Representativeness revisited: Attribute substitution in intuitive judgment

    Daniel Kahneman, Shane Frederick, et al. Representativeness revisited: Attribute substitution in intuitive judgment. Heuristics and biases: The psychology of intuitive judgment, 49(49-81):74, 2002

    2002

  4. [4]

    Dual-process and dual-system theories of reasoning

    Keith Frankish. Dual-process and dual-system theories of reasoning. Philosophy Compass, 5(10): 914–926, 2010. 1

    2010

  5. [5]

    Dual processing in reasoning: Two systems but one reasoner

    Wim De Neys. Dual processing in reasoning: Two systems but one reasoner. Psychological science, 17 (5):428–433, 2006. 2

    2006

  6. [6]

    Dissociation of mechanisms underlying syllogistic reasoning

    Vinod Goel, Christian Buchel, Chris Frith, and Raymond J Dolan. Dissociation of mechanisms underlying syllogistic reasoning. Neuroimage, 12(5):504–514, 2000. 2

    2000

  7. [7]

    From system 1 to system 2: A survey of reasoning large language models

    Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, et al. From system 1 to system 2: A survey of reasoning large language models. arXiv preprint arXiv:2502.17419, 2025. 2

    arXiv 2025

  8. [8]

    Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models

    Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Fara- jtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05229, 2024. 2 17

    arXiv

Show all 162 references
  1. [9]

    Do phd-level llms truly grasp elementary addition? probing rule learning vs

    Yang Yan, Yu Lu, Renjun Xu, and Zhenzhong Lan. Do phd-level llms truly grasp elementary addition? probing rule learning vs. memorization in large language models. arXiv preprint arXiv:2504.05262, 2025. 6

    2025

  2. [10]

    Escapebench: Pushing language models to think outside the box

    Cheng Qian, Peixuan Han, Qinyu Luo, Bingxiang He, Xiusi Chen, Yuji Zhang, Hongyi Du, Jiarui Yao, Xiaocheng Yang, Denghui Zhang, Yunzhu Li, and Heng Ji. Escapebench: Pushing language models to think outside the box. In arxiv, 2025. 2

    2025

  3. [11]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Hel- yar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024. 2, 16, 30

    2024 arXiv

  4. [12]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. 2, 8, 16

    2025 arXiv

  5. [13]

    Grok 3 Beta — The Age of Reasoning Agents, 2025

    xAI. Grok 3 Beta — The Age of Reasoning Agents, 2025. URL https://x.ai/blog/grok-3. Accessed: 2025-02-21. 2

    2025

  6. [14]

    Claude 3.7 sonnet and claude code, 2025

    Anthropic. Claude 3.7 sonnet and claude code, 2025. URL https://www.anthropic.com/news/ claude-3-7-sonnet . Accessed: 2025-02-21. 2, 16

    2025

  7. [15]

    Learning to reason with llms, 2024

    OpenAI. Learning to reason with llms, 2024. URL https://openai.com/index/ learning-to-reason-with-llms/ . Accessed: 2025-02-21. 2

    2024

  8. [16]

    Expanding rl with verifiable rewards across diverse domains

    Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Expanding rl with verifiable rewards across diverse domains. arXiv preprint arXiv:2503.23829, 2025

    2025 arXiv

  9. [17]

    The illusion of thinking: Understanding the strengths and limitations of reasoning mod- els via the lens of problem complexity, 2025

    Parshin Shojaee*†, Iman Mirzadeh*, Keivan Alizadeh, Maxwell Horton, Samy Bengio, and Mehrdad Farajtabar. The illusion of thinking: Understanding the strengths and limitations of reasoning mod- els via the lens of problem complexity, 2025. URL https://ml-site.cdn-apple.com/pape...

    2025

  10. [18]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025. 2

    2025 arXiv

  11. [19]

    Inference-time scaling for diffusion models beyond scaling denoising steps

    Nanye Ma, Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang, Yandong Li, Tommi Jaakkola, Xuhui Jia, et al. Inference-time scaling for diffusion models beyond scaling denoising steps. arXiv preprint arXiv:2501.09732, 2025. 2, 3, 6, 8, 15, 16, 30, 36, 37

    2025 arXiv

  12. [20]

    Video-t1: Test-time scaling for video generation

    Fangfu Liu, Hanyang Wang, Yimo Cai, Kaiyan Zhang, Xiaohang Zhan, and Yueqi Duan. Video-t1: Test-time scaling for video generation. arXiv preprint arXiv:2503.18942, 2025

    2025 arXiv

  13. [21]

    A general framework for inference-time scaling and steering of diffusion models

    Raghav Singhal, Zachary Horvitz, Ryan Teehan, Mengye Ren, Zhou Yu, Kathleen McKeown, and Rajesh Ranganath. A general framework for inference-time scaling and steering of diffusion models. arXiv preprint arXiv:2501.06848, 2025. 2, 16

    2025 arXiv

  14. [22]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. 2, 8, 9, 16, 27, 33, 34

    2023 arXiv

  15. [23]

    Rwkv: Reinventing rnns for the transformer era

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV , et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048, 2023. 2, 16

    2023 arXiv

  16. [24]

    Long short-term memory

    Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735– 1780, 1997. 2

    1997

  17. [25]

    Scaling up test-time compute with latent reasoning: A recurrent depth approach

    Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint arXiv:2502.05171, 2025. 2, 3, 12, 16, 41

    2025 arXiv

  18. [26]

    Scalable diffusion models with transformers, 2023

    William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023. 2, 4, 8, 13, 33, 34, 35, 43

    2023

  19. [27]

    Autoregressive image generation without vector quantization

    Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. Advances in Neural Information Processing Systems, 37:56424–56445, 2025. 2, 8 18

    2025

  20. [28]

    Causal diffusion transformers for generative modeling

    Chaorui Deng, Deyao Zhu, Kunchang Li, Shi Guang, and Haoqi Fan. Causal diffusion transformers for generative modeling. arXiv preprint arXiv:2412.12095, 2024. 2, 33

    2024 arXiv

  21. [29]

    Training large language models to reason in a continuous latent space

    Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769,

    arXiv

  22. [30]

    Think before you speak: Training language models with pause tokens

    Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. arXiv preprint arXiv:2310.02226, 2023. 3

    2023 arXiv

  23. [31]

    Evidence for time-variant decision making

    Jochen Ditterich. Evidence for time-variant decision making. European Journal of Neuroscience, 24(12): 3628–3641, 2006. 3

    2006

  24. [32]

    Prefrontal cortex and flexible cognitive control: Rules without symbols

    Nicolas P Rougier, David C Noelle, Todd S Braver, Jonathan D Cohen, and Randall C O’Reilly. Prefrontal cortex and flexible cognitive control: Rules without symbols. Proceedings of the National Academy of Sciences, 102(20):7338–7343, 2005. 3

    2005

  25. [33]

    Uncertainty- based abstention in llms improves safety and reduces hallucinations

    Christian Tomani, Kamalika Chaudhuri, Ivan Evtimov, Daniel Cremers, and Mark Ibrahim. Uncertainty- based abstention in llms improves safety and reduces hallucinations. arXiv preprint arXiv:2404.10960,

    arXiv

  26. [34]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017. 3, 15

    2017

  27. [35]

    Auto-encoding variational bayes, 2013

    Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013. 3, 15

    2013

  28. [36]

    Bayesformer: Transformer with uncertainty estimation

    Karthik Abinav Sankararaman, Sinong Wang, and Han Fang. Bayesformer: Transformer with uncertainty estimation. arXiv preprint arXiv:2206.00826, 2022. 3

    2022 arXiv

  29. [37]

    Out-of-distribution detection with a single unconditional diffusion model

    Alvin Heng, Harold Soh, et al. Out-of-distribution detection with a single unconditional diffusion model. Advances in Neural Information Processing Systems, 37:43952–43974, 2024

    2024

  30. [38]

    Do deep generative models know what they don’t know? arXiv preprint arXiv:1810.09136, 2018

    Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don’t know? arXiv preprint arXiv:1810.09136, 2018

    2018 arXiv

  31. [39]

    Input complexity and out-of-distribution detection with likelihood-based generative models

    Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F Núñez, and Jordi Luque. Input complexity and out-of-distribution detection with likelihood-based generative models. arXiv preprint arXiv:1909.11480, 2019. 3

    1909 arXiv

  32. [40]

    Mixture density networks

    Christopher M Bishop. Mixture density networks. 1994. 3

    1994

  33. [41]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020. 3

    2011 arXiv

  34. [42]

    Introduction to latent variable energy-based models: a path toward autonomous machine intelligence

    Anna Dawid and Yann LeCun. Introduction to latent variable energy-based models: a path toward autonomous machine intelligence. Journal of Statistical Mechanics: Theory and Experiment, 2024(10): 104011, 2024. 3, 6, 7, 15, 40

    2024

  35. [43]

    Peters, B

    A. Peters, B. McEwen, and Karl J. Friston. Uncertainty and stress: Why it causes diseases and how it is mastered by the brain. Progress in Neurobiology, 156:164–188, 2017. doi: 10.1016/j.pneurobio.2017.05

    2017 doi

  36. [44]

    Vilares, J

    I. Vilares, J. D. Howard, Hugo L. Fernandes, J. Gottfried, and Konrad Paul Kording. Differential representations of prior and likelihood uncertainty in the human brain. Current Biology, 22:1641–1648,

  37. [45]

    Sarinopoulos, D

    Issidoros C. Sarinopoulos, D. Grupe, Kristen L. Mackiewicz, J. Herrington, M. Lor, E. E. Steege, and J. Nitschke. Uncertainty during anticipation modulates neural responses to aversion in human insula and amygdala. Cerebral cortex, 20 4:929–40, 2010. doi: 10.1093/cercor/bhp155. 3

    2010 doi

  38. [46]

    Paving the way to eureka—introducing “dira” as an experimental paradigm to observe the process of creative problem solving

    Frank Loesche, Jeremy Goslin, and Guido Bugmann. Paving the way to eureka—introducing “dira” as an experimental paradigm to observe the process of creative problem solving. Frontiers in Psychology, 9: 1773, 2018. 3

    2018

  39. [47]

    Using contents and containers to investigate problem solving strategies among toddlers

    Zaid Alkouri. Using contents and containers to investigate problem solving strategies among toddlers

  40. [48]

    Learning iterative reasoning through energy minimization

    Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Learning iterative reasoning through energy minimization. In International Conference on Machine Learning, pages 5570–5582. PMLR, 2022. 3, 6, 7, 13, 16, 36

    2022

  41. [49]

    Mastering the game of go without human knowledge

    David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354–359, 2017. 3

    2017

  42. [50]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The Twelfth International Conference on Learning Representations, 2023. 3, 15

    2023

  43. [51]

    Implicit generation and modeling with energy based models

    Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. Advances in neural information processing systems, 32, 2019. 4, 5, 6, 7, 8, 27, 42

    2019

  44. [52]

    Improved contrastive divergence training of energy based models

    Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Improved contrastive divergence training of energy based models. arXiv preprint arXiv:2012.01316, 2020

    2012 arXiv

  45. [53]

    Learning energy-based models in high-dimensional spaces with multiscale denoising-score matching

    Zengyi Li, Yubei Chen, and Friedrich T Sommer. Learning energy-based models in high-dimensional spaces with multiscale denoising-score matching. Entropy, 25(10):1367, 2023. 4

    2023

  46. [54]

    Generalized energy based models

    Michael Arbel, Liang Zhou, and Arthur Gretton. Generalized energy based models. arXiv preprint arXiv:2003.05033, 2020. 4, 8, 42

    2003 arXiv

  47. [55]

    Improving language understand- ing by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understand- ing by generative pre-training. 2018. 4, 8

    2018

  48. [56]

    Bert: Pre-training of deep bidirectional transformers for language understanding, 2019

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. 4, 8

    2019

  49. [57]

    Visualizing the loss landscape of neural nets

    Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018. 5

    2018

  50. [58]

    The complexity of theorem-proving procedures

    Stephen A Cook. The complexity of theorem-proving procedures. In Logic, automata, and computational complexity: The works of Stephen A. Cook, pages 143–152. 2023. 5

    2023

  51. [59]

    The knowledge complexity of interactive proof- systems

    Shafi Goldwasser, Silvio Micali, and Chales Rackoff. The knowledge complexity of interactive proof- systems. In Providing sound foundations for cryptography: On the work of shafi goldwasser and silvio micali, pages 203–225. 2019. 5

    2019

  52. [60]

    Letter to john von neumann, 1956

    Kurt Gödel. Letter to john von neumann, 1956. URL https://ecommons.cornell.edu/server/ api/core/bitstreams/46aef9c4-288b-457d-ab3e-bb6cb1a4b88e/content . Accessed: 2025- 04-28. 5

    1956

  53. [61]

    A survey on the applications of zero-knowledge proofs

    Ryan Lavin, Xuekai Liu, Hardhik Mohanty, Logan Norman, Giovanni Zaarour, and Bhaskar Krishna- machari. A survey on the applications of zero-knowledge proofs. arXiv preprint arXiv:2408.00243, 2024. 5

    2024 arXiv

  54. [62]

    A method for obtaining digital signatures and public-key cryptosystems

    Ronald L Rivest, Adi Shamir, and Leonard Adleman. A method for obtaining digital signatures and public-key cryptosystems. Communications of the ACM, 21(2):120–126, 1978. 5

    1978

  55. [63]

    Alphacode 2 technical report

    AlphaCode Team. Alphacode 2 technical report. December 2023. 6

    2023

  56. [64]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023. 6

    2023

  57. [65]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:2773...

    2022

  58. [66]

    All roads lead to likelihood: The value of reinforcement learning in fine-tuning

    Gokul Swamy, Sanjiban Choudhury, Wen Sun, Zhiwei Steven Wu, and J Andrew Bagnell. All roads lead to likelihood: The value of reinforcement learning in fine-tuning. arXiv preprint arXiv:2503.01067, 2025. 6

    2025

  59. [67]

    Learning iterative reasoning through energy diffusion

    Yilun Du, Jiayuan Mao, and Joshua B Tenenbaum. Learning iterative reasoning through energy diffusion. arXiv preprint arXiv:2406.11179, 2024. 6, 16, 30, 37

    2024 arXiv

  60. [68]

    what it can create, it may not understand

    Peter West, Ximing Lu, Nouha Dziri, Faeze Brahman, Linjie Li, Jena D Hwang, Liwei Jiang, Jillian Fisher, Abhilasha Ravichander, Khyathi Chandu, et al. The generative ai paradox:" what it can create, it may not understand". arXiv preprint arXiv:2311.00059, 2023. 6 20

    2023 arXiv

  61. [69]

    Commonsense psychology in human infants and machines

    Gala Stojni´c, Kanishk Gandhi, Shannon Yasuda, Brenden M Lake, and Moira R Dillon. Commonsense psychology in human infants and machines. Cognition, 235:105406, 2023. 6

    2023

  62. [70]

    Position: Llms can’t plan, but can help planning in llm-modulo frameworks

    Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Paul Saldyt, and Anil B Murthy. Position: Llms can’t plan, but can help planning in llm-modulo frameworks. In Forty-first International Conference on Machine Learning, 2024. 6

    2024

  63. [71]

    Energy-inspired self-supervised pretraining for vision models

    Ze Wang, Jiang Wang, Zicheng Liu, and Qiang Qiu. Energy-inspired self-supervised pretraining for vision models. arXiv preprint arXiv:2302.01384, 2023. 6, 7, 38, 40, 43

    2023 arXiv

  64. [72]

    A path towards autonomous machine intelligence version 0.9

    Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62, 2022. 6

    2022

  65. [73]

    Generative adversarial nets

    Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014. 7

    2014

  66. [74]

    A neural probabilistic language model

    Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of machine learning research, 3(Feb):1137–1155, 2003. 7

    2003

  67. [75]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 7, 16

    2020

  68. [76]

    How to compute hessian-vector products? In ICLR Blogposts 2024, 2024

    Mathieu Dagréou, Pierre Ablin, Samuel Vaiter, and Thomas Moreau. How to compute hessian-vector products? In ICLR Blogposts 2024, 2024. URL https://iclr-blogposts.github.io/2024/blog/ bench-hvp/. https://iclr-blogposts.github.io/2024/blog/bench-hvp/. 7, 35

    2024

  69. [77]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008–3021, 2020. 8

    2020

  70. [78]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023. 8

    2023

  71. [79]

    Dinov2: Learning robust visual features without supervision, 2023

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabb...

    2023

  72. [80]

    Masked autoencoders are scalable vision learners, 2021

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners, 2021. 14

    2021

  73. [81]

    Audiolm: a language modeling approach to audio generation, 2023

    Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. Audiolm: a language modeling approach to audio generation, 2023. 8

    2023

  74. [82]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. 8, 15, 30

    2019

  75. [83]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 8, 15, 31

    2017

  76. [84]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 8, 9, 10, 11, 12, 28

    2024 arXiv

  77. [85]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. 8, 33, 34

    2001 arXiv

  78. [86]

    (mis) fitting: A survey of scaling laws

    Margaret Li, Sneha Kudugunta, and Luke Zettlemoyer. (mis) fitting: A survey of scaling laws. arXiv preprint arXiv:2502.18969, 2025. 8, 9, 33

    2025 arXiv

  79. [87]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023

  80. [88]

    Revisiting neural scaling laws in language and vision

    Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300–22312, 2022

    2022

  81. [89]

    Scaling laws for autoregressive generative modeling

    Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020

    2010 arXiv

  82. [90]

    Training compute-optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. 8

    2022 arXiv

  83. [91]

    Neural ordinary differential equations

    Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018. 8, 14

    2018

  84. [92]

    Redpajama: an open dataset for training large language models

    Maurice Weber, Dan Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, et al. Redpajama: an open dataset for training large language models. Advances in neural information processing systems, 37:116462–11649...

    2024

  85. [93]

    Redpajama: an open dataset for training large language models, 2023

    Together Computer. Redpajama: an open dataset for training large language models, 2023. URL https://github.com/togethercomputer/RedPajama-Data. 8

    2023

  86. [94]

    Gpt-neox-20b: An open-source autoregressive language model

    Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745, 2022. 8

    2022 arXiv

  87. [95]

    Learning to (learn at test time): Rnns with expressive hidden states

    Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, et al. Learning to (learn at test time): Rnns with expressive hidden states. arXiv preprint arXiv:2407.04620, 2024. 8

    2024 arXiv

  88. [96]

    Physics of language models: Part 2.1, grade- school math and the hidden reasoning process

    Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of language models: Part 2.1, grade- school math and the hidden reasoning process. In The Thirteenth International Conference on Learning Representations, 2024. 9

    2024

  89. [97]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. 9

    arXiv

  90. [98]

    Squad: 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016. 9

    2016 arXiv

  91. [99]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint a...

    arXiv

  92. [100]

    Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems, 36:55565–55581, 2023

    Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems, 36:55565–55581, 2023. 9

    2023

  93. [101]

    Byte latent transformer: Patches scale better than tokens

    Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, et al. Byte latent transformer: Patches scale better than tokens. arXiv preprint arXiv:2412.09871, 2024. 9, 10

    2024 arXiv

  94. [102]

    Language models scale reliably with over-training and on downstream tasks

    Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, et al. Language models scale reliably with over-training and on downstream tasks. arXiv preprint arXiv:2403.08540, 2024. 11

    2024 arXiv

  95. [103]

    Don’t stop pretraining: Adapt language models to domains and tasks

    Suchin Gururangan, Ana Marasovi´c, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don’t stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964, 2020. 22

    2004 arXiv

  96. [104]

    Improving pretraining data using perplexity correlations

    Tristan Thrush, Christopher Potts, and Tatsunori Hashimoto. Improving pretraining data using perplexity correlations. arXiv preprint arXiv:2409.05816, 2024. 11

    2024 arXiv

  97. [105]

    Scaling laws for predicting downstream performance in llms

    Yangyi Chen, Binxuan Huang, Yifan Gao, Zhengyang Wang, Jingfeng Yang, and Heng Ji. Scaling laws for predicting downstream performance in llms. arXiv preprint arXiv:2410.08527, 2024. 11, 12, 34

    2024 arXiv

  98. [106]

    Scaling laws for downstream task performance of large language models

    Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. Scaling laws for downstream task performance of large language models. InICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024. 11, 12

    2024

  99. [107]

    Autoregressive video generation without vector quantization

    Haoge Deng, Ting Pan, Haiwen Diao, Zhengxiong Luo, Yufeng Cui, Huchuan Lu, Shiguang Shan, Yonggang Qi, and Xinlong Wang. Autoregressive video generation without vector quantization. arXiv preprint arXiv:2412.14169, 2024. 12

    2024 arXiv

  100. [108]

    Scaling autoregressive video models

    Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models. arXiv preprint arXiv:1906.02634, 2019

    1906 arXiv

  101. [109]

    Latent video transformer

    Ruslan Rakhimov, Denis V olkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev. Latent video transformer. arXiv preprint arXiv:2006.10704, 2020

    2006 arXiv

  102. [110]

    Video prediction by efficient transformers

    Xi Ye and Guillaume-Alexandre Bilodeau. Video prediction by efficient transformers. Image and Vision Computing, 130:104612, 2023

    2023

  103. [111]

    Long-context autoregressive video modeling with next-frame prediction

    Yuchao Gu, Weijia Mao, and Mike Zheng Shou. Long-context autoregressive video modeling with next-frame prediction. arXiv preprint arXiv:2503.19325, 2025. 12

    2025 arXiv

  104. [112]

    Will we run out of data? an analysis of the limits of scaling datasets in machine learning

    Pablo Villalobos, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and Anson Ho. Will we run out of data? an analysis of the limits of scaling datasets in machine learning. arXiv preprint arXiv:2211.04325, 2022. 12

    2022 arXiv

  105. [113]

    URL https://research.google/blog/data-centric-ml-benchmarking-announcing-dataperfs-2023-challenges/ . 12

    2023

  106. [114]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 12, 16, 33, 35, 40

    2022

  107. [115]

    12, 33, 35

    URL https://huggingface.co/stabilityai/sd-vae-ft-mse . 12, 33, 35

  108. [116]

    something something

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Pr...

    2017

  109. [117]

    Eqa-mx: Embodied question answer- ing using multimodal expression

    Md Mofijul Islam, Alexi Gladstone, Riashat Islam, and Tariq Iqbal. Eqa-mx: Embodied question answer- ing using multimodal expression. In The Twelfth International Conference on Learning Representations,

  110. [118]

    Learning to stop while learning to predict

    Xinshi Chen, Hanjun Dai, Yu Li, Xin Gao, and Le Song. Learning to stop while learning to predict. In International conference on machine learning, pages 1520–1530. PMLR, 2020. 13

    2020

  111. [119]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings...

    2014

  112. [120]

    URL https://huggingface.co/datasets/AbdoTW/COCO_2014. 13, 34

  113. [121]

    Transformer uncertainty estimation with hierarchical stochastic attention

    Jiahuan Pei, Cheng Wang, and György Szarvas. Transformer uncertainty estimation with hierarchical stochastic attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11147–11155, 2022. 14

    2022

  114. [122]

    Imagenet large scale visual recognition challenge

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211–252, 2015. 14

    2015

  115. [123]

    URL https://www.youtube.com/watch?v=6nJZopACRuQ&ab_channel=OpenAI. 15 23

  116. [124]

    Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment

    Audrey Huang, Adam Block, Qinghua Liu, Nan Jiang, Dylan J Foster, and Akshay Krishnamurthy. Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment. arXiv preprint arXiv:2503.21878, 2025. 15

    2025 arXiv

  117. [125]

    Transformers in speech processing: A survey

    Siddique Latif, Aun Zaidi, Heriberto Cuayahuitl, Fahad Shamshad, Moazzam Shoukat, and Junaid Qadir. Transformers in speech processing: A survey. arXiv preprint arXiv:2303.11607, 2023. 15

    2023 arXiv

  118. [126]

    Universal transformers

    Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Universal transformers. arXiv preprint arXiv:1807.03819, 2018. 16

    2018 arXiv

  119. [127]

    Reasoning with latent thoughts: On the power of looped transformers

    Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J Reddi. Reasoning with latent thoughts: On the power of looped transformers. arXiv preprint arXiv:2502.17416, 2025. 16, 36

    2025 arXiv

  120. [128]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022. 16

    2022

  121. [129]

    Implicit reasoning in transformers is reasoning through shortcuts

    Tianhe Lin, Jian Xie, Siyu Yuan, and Deqing Yang. Implicit reasoning in transformers is reasoning through shortcuts. arXiv preprint arXiv:2503.07604, 2025. 16

    2025 arXiv

  122. [130]

    Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting

    Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems, 36:74952–74965, 2023

    2023

  123. [131]

    Faithfulness vs

    Chirag Agarwal, Sree Harsha Tanneru, and Himabindu Lakkaraju. Faithfulness vs. plausibility: On the (un) reliability of explanations from large language models. arXiv preprint arXiv:2402.04614, 2024. 16

    2024 arXiv

  124. [132]

    Diffusion models for video prediction and infilling, 2022

    Tobias Höppe, Arash Mehrjou, Stefan Bauer, Didrik Nielsen, and Andrea Dittadi. Diffusion models for video prediction and infilling, 2022. 16

    2022

  125. [133]

    Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc

    Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Sussman Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc. In International conference o...

  126. [134]

    A tutorial on energy-based learning

    Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, Fujie Huang, et al. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006. 16

    2006

  127. [135]

    a is b" fail to learn

    Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Llms trained on" a is b" fail to learn" b is a". arXiv preprint arXiv:2309.12288, 2023. 26

    2023 arXiv

  128. [136]

    Planning with diffusion for flexible behavior synthesis

    Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.09991, 2022. 26

    2022 arXiv

  129. [137]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273668, 2023

    2023

  130. [138]

    Dino-wm: World models on pre-trained visual features enable zero-shot planning

    Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. Dino-wm: World models on pre-trained visual features enable zero-shot planning. arXiv preprint arXiv:2411.04983, 2024. 26

    2024 arXiv

  131. [139]

    Residual energy-based models for text

    Anton Bakhtin, Yuntian Deng, Sam Gross, Myle Ott, Marc’Aurelio Ranzato, and Arthur Szlam. Residual energy-based models for text. Journal of Machine Learning Research, 22(40):1–41, 2021. 26, 38

    2021

  132. [140]

    Energy-based reranking: Improving neural machine translation using energy-based models

    Sumanta Bhattacharyya, Amirmohammad Rooshenas, Subhajit Naskar, Simeng Sun, Mohit Iyyer, and Andrew McCallum. Energy-based reranking: Improving neural machine translation using energy-based models. arXiv preprint arXiv:2009.13267, 2020. 26, 38

    2009 arXiv

  133. [141]

    Recurrent neuronal circuits in the neocortex

    Rodney J Douglas and Kevan AC Martin. Recurrent neuronal circuits in the neocortex. Current biology, 17(13):R496–R500, 2007. 27

    2007

  134. [142]

    A conceptual introduction to hamiltonian monte carlo

    Michael Betancourt. A conceptual introduction to hamiltonian monte carlo. arXiv preprint arXiv:1701.02434, 2017. 27

    2017 arXiv

  135. [143]

    The fineweb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydlí ˇcek, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro V on Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:30811–30849, 2024. 28 24

    2024

  136. [144]

    Adaptive inference-time compute: Llms can predict if they can do better, even mid-generation

    Rohin Manvi, Anikait Singh, and Stefano Ermon. Adaptive inference-time compute: Llms can predict if they can do better, even mid-generation. arXiv preprint arXiv:2410.02725, 2024. 30

    2024 arXiv

  137. [145]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024. 30

    2024 arXiv

  138. [146]

    Active inference: the free energy principle in mind, brain, and behavior

    Thomas Parr, Giovanni Pezzulo, and Karl J Friston. Active inference: the free energy principle in mind, brain, and behavior. MIT Press, 2022. 30

    2022

  139. [147]

    Pytorch lightning

    William A Falcon. Pytorch lightning. GitHub, 3, 2019. 33

    2019

  140. [148]

    Gpt-neox-20b: An open-source autoregressive language model, 2022

    Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. Gpt-neox-20b: An ope...

    2022

  141. [149]

    Predicting emergent abilities with infinite resolution evaluation

    Shengding Hu, Xin Liu, Xu Han, Xinrong Zhang, Chaoqun He, Weilin Zhao, Yankai Lin, Ning Ding, Zebin Ou, Guoyang Zeng, et al. Predicting emergent abilities with infinite resolution evaluation. arXiv preprint arXiv:2310.03262, 2023. 34

    arXiv

  142. [150]

    Transformer flops

    Adam Casson. Transformer flops. 2023. URL https://adamcasson.com/posts/ transformer-flops. 35

    2023

  143. [151]

    Compositional visual generation with composable diffusion models

    Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision , pages 423–439. Springer, 2022. 36

    2022

  144. [152]

    Energy transformer

    Benjamin Hoover, Yuchen Liang, Bao Pham, Rameswar Panda, Hendrik Strobelt, Duen Horng Chau, Mohammed Zaki, and Dmitry Krotov. Energy transformer. Advances in Neural Information Processing Systems, 36, 2024. 38

    2024

  145. [153]

    Your autoregressive generative model can be better if you treat it as an energy-based one

    Yezhen Wang, Tong Che, Bo Li, Kaitao Song, Hengzhi Pei, Yoshua Bengio, and Dongsheng Li. Your autoregressive generative model can be better if you treat it as an energy-based one. arXiv preprint arXiv:2206.12840, 2022. 38

    2022 arXiv

  146. [154]

    A connection between score matching and denoising autoencoders

    Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661–1674, 2011. 38, 40

    2011

  147. [155]

    Mapping sentence form onto meaning: The syntax–semantic interface

    Angela D Friederici and Jürgen Weissenborn. Mapping sentence form onto meaning: The syntax–semantic interface. Brain research, 1146:50–58, 2007. 38

    2007

  148. [156]

    When can transformers compositionally generalize in-context? arXiv preprint arXiv:2407.12275, 2024

    Seijin Kobayashi, Simon Schug, Yassir Akram, Florian Redhardt, Johannes von Oswald, Razvan Pascanu, Guillaume Lajoie, and João Sacramento. When can transformers compositionally generalize in-context? arXiv preprint arXiv:2407.12275, 2024. 38

    2024 arXiv

  149. [157]

    T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation

    Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36:78723–78747, 2023. 38

    2023

  150. [158]

    Neural networks and physical systems with emergent collective computational abilities

    John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554–2558, 1982. 41

    1982

  151. [159]

    Understanding the difficulty of training deep feedforward neural networks

    Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256. JMLR Workshop and Conference Proceedings, 2010. 42

    2010

  152. [160]

    Glu variants improve transformer, 2020

    Noam Shazeer. Glu variants improve transformer, 2020. 42

    2020

  153. [161]

    The Reversal Curse

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. 42 25 In this appendix, we provide additional insight and details on EBTs. First, we provide more insight...

    2024

  154. [2012]

    doi: 10.1016/j.cub.2012.07.010

    2012 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.