Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Decoding-based Regression

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A decoding-based regression head — an autoregressive transformer that emits numbers as tokens — matches or beats pointwise and histogram heads on regression and density estimation, with a theorem reducing its risk to the classical…

desk verdict Solid empirical findings on decoding heads for regression; the theory is a standard histogram bound under an idealized assumption and doesn't cover the conditional case, so the abstract overstates it. read the letter →

arxiv 2501.19383 v2 pith:4P2RTOU7 submitted 2025-01-31 cs.LG cs.AIcs.CLstat.ML

Xingyou Song , Dara Bahri This is my paper · ORCID
classification cs.LGcs.AIcs.CLstat.ML MSC 62G0762G2068T07
keywords decoding-basedregressionnext-tokenpredictiondensityestimationhistogramestimatorstokenizationofrealnumbersautoregressivetransformerscross-entropylossheads
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a language-model-style decoder head — an autoregressive Transformer that outputs numbers one token at a time and is trained with ordinary next-token cross-entropy loss — can serve as a general-purpose numeric regression head attached to any feature representation. If true, it means regression and density estimation need not be delegated to dedicated pointwise or parametric distribution heads: the same machinery used for text generation can represent a predictive distribution over real numbers. The authors support this with a theorem showing that under a mild capacity condition the decoder's density estimator is exactly a classical histogram estimator, so its risk follows the familiar bias-variance formula, and with experiments on tabular benchmarks, curve fitting, and density estimation showing the decoder is competitive with or better than pointwise, histogram, and Gaussian mixture heads.

What carries the argument

The central object is the decoding head: an autoregressive sequence model, by default a small Transformer, that takes the feature vector $\phi(x)$ as initial context and generates $K$ tokens representing a number, thereby defining $p_\theta(y|x)$ as a product of per-token conditional probabilities. Numeric tokenization maps real $y$ to a base-$B$ digit string, normalized to $[0,1]$ or unnormalized with exponent tokens in the style of IEEE-754. The key identity is the binary-tree interpretation: the first $k$ bits of $y$ identify one of $2^k$ uniform bins of $[0,1]$, so the decoder simultaneously learns $K$ histogram estimators and decoding one more bit refines the estimate. Theorem 1 is what carries the argument: under $K$-bit universality (Definition 1), the cross-entropy-trained decoder's density estimate coincides with the $2^k$-bin histogram of the training data, giving the exact bias-variance risk formula and explaining both the need for more bits to capture curvature and more samples to fill the resulting bins.

What would settle it

Train a small decoder on $N$ i.i.d. samples from a known smooth density $f$ on $[0,1]$ for several bit lengths $K$, and compare its empirical integrated squared error to Theorem 1's formula $R = \frac{2^{-2K}}{12}\int_0^1 f'(y)^2\,dy + \frac{2^K}{N}$. If over many seeds the measured risk departs from the predicted bias-variance tradeoff — for instance, if the optimal $K$ does not shift with $N$ as the formula dictates, or if the decoder cannot fit a uniform distribution over $2^K$ bins (measurable by cross-entropy minus entropy) — then the universality assumption or the histogram equivalence fails and the theorem does not describe the trained model.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that decoding-based regression heads are as performant as standard pointwise heads on standard regression tasks when trained in the usual way (next-token prediction via cross-entropy), while being flexible enough to capture smooth numeric distributions in density estimation. The mechanism is a tokenization of real numbers — normalized base-$B$ expansions on $[0,1]$ or unnormalized sign/exponent/mantissa strings — decoded autoregressively conditioned on the feature vector. Theorem 1 states that if the model class is $K$-bit universal (able to fit any discrete distribution over $K$-bit strings), the mean integrated squared error of the resulting density estimator is exactly $R = \frac{2^{-2k}}{12}\int_0^1 f'(y)^2\,dy + \frac{2^k}{N}$ plus negligible terms: the classical histogram decomposition into squared bias and variance, with $k$ the number of decoded bits. Thus decoding $k$ bits gradually refines a histogram estimate, and the theory transfers the whole classical bias-variance tradeoff to neural decoders.

Load-bearing premise

The theory depends on $K$-bit universality: the decoder model class must be able to fit any discrete distribution over $K$-bit strings exactly and the training procedure must find that fit, an assumption the paper does not verify for the small Transformers used in the experiments.

Editorial extensions

If this is right

  • Decoder heads can serve as drop-in replacements for pointwise heads: on AMLB and OpenML-CTR23 tabular regression benchmarks, the unnormalized decoder head is competitive with and often better than a pointwise head given the same training data.
  • Decoder heads are sample-efficient relative to histogram (Riemann) heads: with the same normalization they outperform the histogram head on most tasks, and the gain is explained by the exponential reduction in effective bins (e.g. $10^3$ bins as 10 tokens over 3 decoding steps instead of 1000 categories).
  • Decoder heads perform flexible density estimation: vanilla temperature sampling from the decoder matches $p(y|x)$ on synthetic shapes and gives consistently low negative log-likelihood on UCI datasets, outperforming both Gaussian mixture heads (high variance) and Riemann heads (consistently worse).
  • Pointwise estimates such as mean, median, and mode can be extracted from the decoder's distribution, and coding-theory-style repetition with majority voting reduces outlier-driven mean squared error when aggregating samples.
  • Unnormalized tokenization avoids $y$-normalization and handles wide or unbounded ranges that break pointwise and Riemann heads on functions with large Lipschitz constants or vertical asymptotes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The experiments suggest an implicit regularization that the theorem does not capture: at low sample sizes and large bit counts, the decoder's empirical risk is lower than the histogram formula predicts, consistent with the model refusing to fit noisy empirical distributions; this could be tested by measuring whether the effect scales with model capacity and data size.
  • The histogram equivalence connects decoding-based regression to distributional reinforcement learning and categorical value-based methods, suggesting that techniques from those literatures (rank-consistency, soft labels, distributional temporal-difference losses) might transfer directly to numeric token decoding.
  • The theoretical framework is not tied to base 2; extending Theorem 1 to other token trees (p-adic expansions, Stern-Brocot, or Hamming-distance representations) would give a principled way to choose tokenizations for robustness, although the paper's own Hamming experiment shows such gains are not automatic.
  • If decoding heads truly match pointwise heads, then generative reward models and LLM-as-a-judge pipelines that already output numeric strings could be used for calibrated predictive distributions rather than just scalar scores, a consequence the paper gestures toward but does not develop.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper formalizes "decoding-based regression," in which an autoregressive sequence model produces a tokenized string representation of a numeric target y given a feature representation φ(x). It proposes normalized and unnormalized tokenization schemes, discusses pointwise estimation from the decoded distribution, and states a theoretical result (Theorem 1) that, under a K-bit universality assumption, the risk of the maximum-likelihood decoder equals the classical histogram mean integrated squared error expansion. The empirical sections compare decoder heads with pointwise, Riemann/histogram, and mixture-density-network heads on synthetic curve fitting, BBOB functions, AMLB and OpenML-CTR23 regression benchmarks, and UCI density-estimation tasks, with additional ablations on decoder size and error-correction by repeated decoding.

Significance. The empirical study is valuable and comparatively well controlled: the authors sweep hyperparameters, average over multiple runs, use standard benchmarks, and ablate the decoder head size and tokenization choices. The formalization of numeric tokenization and the observation that decoding heads can be competitive with pointwise heads are useful contributions to the growing literature on language models for numeric prediction. The theoretical result is a correct and standard histogram bias-variance calculation, but it is limited to unconditional density estimation for a fixed density f on [0,1] and relies on a strong universality assumption. As written, the abstract and conclusion claim more than Theorem 1 supports: the theorem does not involve features x and therefore does not ground the conditional regression or conditional density-estimation claims that are the paper's central experimental subject. The paper would be publishable after the theoretical claims are scoped accurately and the missing experimental details about scalar prediction are supplied.

major comments (3)
  1. [§3.3, Definition 3 and Theorem 1] Theorem 1 is stated for Y_1,...,Y_N i.i.d. draws from a fixed density f on [0,1]; the MLE objective in Definition 3 is over -log p_theta(lambda_K(Y_n)) and contains no feature variable x and no conditional distribution p(y|x). The derived MISE bound is therefore an unconditional density-estimation result for the histogram induced by the decoder's marginal distributions. It does not provide theoretical grounds for the paper's stated subject, namely decoding-based heads as regression heads "given any feature representation" or for estimating p(y|x), as claimed in the abstract and conclusion. I recommend either restricting all theoretical claims to unconditional density estimation or extending the analysis to paired (X,Y) samples and stating the additional assumptions needed for a conditional version.
  2. [§3.3, Definition 1 and Figure 2] The K-bit universality assumption is not verified for the small Transformer decoders used in the experiments, and the paper's own Figure 2 indicates that the premise fails in the relevant regime: at N=1024 and larger K, the trained decoder deviates substantially from the theorem's risk curve, an effect the authors attribute to implicit regularization. Under K-bit universality (with successful optimization), the proof forces p^k_theta* to equal the empirical k-bit histogram, so the observed deviation means the trained model does not satisfy the assumption. Consequently, Theorem 1 describes an idealized limit and cannot be invoked as the explanation for the empirical density-estimation behavior. The authors should either verify universality for the architectures used, weaken the claim to an illustrative idealized setting, or explicitly characterize the discrepancy as an additional inductive-bias effect that is outside the theorem.
  3. [§4.1, §4.2 and Appendix C] The scalar prediction rule used for the decoder heads in the main regression comparisons is not specified. Section 3.2 states that the method for computing pointwise estimates is a hyperparameter, and Figure 10 shows that mean aggregation with unnormalized tokenization can be severely degraded by outlier decodings. Yet the headline regression results in Table 1 and Figure 5 report unnormalized and normalized decoder performance without stating whether predictions were obtained by averaging samples, using the mode, using the median, or some other rule, nor how many decoded samples were used. This is necessary to interpret the central empirical claim that decoder heads are "as performant as" pointwise heads, and it is needed for reproducibility.
minor comments (5)
  1. [§3.3, Theorem 1] The theorem statement says the risk "can be exactly computed," but the right-hand side contains O(2^{-4k} + 1/N) terms; the proof itself concludes with an asymptotic expansion. Please rephrase as "asymptotically equal to" or explicitly describe the remainder as a bounded error term.
  2. [§3.3, Definition 3] The estimator is defined as \hat f_N but Theorem 1 uses the notation f_k*_N; please align these symbols to avoid confusion.
  3. [Appendix C, Normalized Decoder hyperparameters] The length sweep is listed as "K: [4, 8, 6]"; this appears to be a typo, likely intended to be [4, 6, 8] or another increasing order.
  4. [Appendix C] The phrase "we sweeped the encoder" should be "we swept the encoder."
  5. [Figure 5 caption] The caption contains the duplicated phrase "sorted by gap performance gap," and the stacked-bar construction with a y-axis capped at 1.0 is hard to read; side-by-side bars or explicit offsets would be clearer.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation; Theorem 1 reduces to the classical histogram MISE under an explicit, unverified universality assumption, which is a scope limitation rather than a circular step.

full rationale

Theorem 1 is derived from explicit axioms: K-bit universality (Definition 1) and twice-continuous differentiability of f. Under universality, the MLE on truncated bit strings equals the empirical bin distribution, so the estimator f_k*_N(y) = 2^k p^k_theta*(lambda_k(y)) 'lines up exactly as a 2^k-bin histogram estimator.' The subsequent bias-variance computation in Appendix B is the standard histogram MISE decomposition. This is a conditional mathematical reduction, not a fit of the target result: the universality assumption does not assert the risk formula, and the conclusion follows by proof rather than by construction from the paper's own outputs. The empirical NLL and Kendall-Tau results are evaluated on held-out test splits, and Figure 2 explicitly compares the theoretical curve with measured decoder risk and reports deviation at low N, so no fitted parameter is relabeled as a prediction. Self-citations (Song et al. 2024; Tang et al. 2024; Nguyen et al. 2024) are used for motivation and related work, not to justify the theorem or to forbid alternatives; no uniqueness theorem is imported from the authors' prior work. The main weaknesses are scope and support rather than circularity: Theorem 1 concerns an unconditional density f on [0,1] with i.i.d. draws and contains no feature x, so it does not by itself ground conditional regression p(y|x), and K-bit universality is not verified for the small Transformers used in the experiments. These are correctness/robustness limitations, not circular steps.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central theoretical result depends on two explicit axioms: K-bit universality, a strong realizability condition, and twice continuous differentiability of the true density. The empirical claims rely on a set of swept hyperparameters such as tokenization lengths, base, transformer size, bin counts, and mixture counts that are tuned per task. No new physical entities are introduced; the decoding head is a model component.

free parameters (3)
  • K (normalized token length) = swept over [4, 8, 6]
    Resolution of the base-B expansion; the theorem shows an optimal K for given N, and the paper tunes it per task.
  • Unnormalized tokenization (B, E, M) = B in [4,8,10], E in [1,2,4], M in [2,4,8]
    Base and digit counts for the IEEE-754-like representation; chosen by hand and swept in experiments, affecting range and granularity.
  • Decoder transformer size = 1 layer, 32 units, 1 head or 3 layers, 128 units, 4 heads
    Model capacity is a hyperparameter; the main text uses the small size, the appendix sweeps larger sizes.
assumptions (3)
  • domain assumption K-bit universality (Definition 1): for every discrete distribution p on K-bit strings, min_theta H(p, p_theta) = H(p,p).
    This is the key premise of Theorem 1; it is a strong capacity assumption that the small Transformers used in experiments are not shown to satisfy, and it reduces the estimator to the empirical histogram.
  • domain assumption The true density f is twice continuously differentiable on [0,1].
    Used in the bias expansion in the proof of Theorem 1; reasonable for smooth distributions but not verified for the real datasets.
  • standard math Samples Y_1,...,Y_N are i.i.d. draws from f.
    Standard statistical assumption in the density estimation setting.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Decoding-based Regression." pith.science (2026). https://pith.science/paper/4P2RTOU7

@misc{pith2026250119383,
  author       = {Pith},
  title        = {Pith review of: Decoding-based Regression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4P2RTOU7}},
  note         = {Machine review of arXiv:2501.19383}
}
read the original abstract

Language models have recently been shown capable of performing regression wherein numeric predictions are represented as decoded strings. In this work, we provide theoretical grounds for this capability and furthermore investigate the utility of causal sequence decoding models as numeric regression heads given any feature representation. We find that, despite being trained in the usual way - for next-token prediction via cross-entropy loss - decoder-based heads are as performant as standard pointwise heads when benchmarked over standard regression tasks, while being flexible enough to capture smooth numeric distributions, such as in the task of density estimation.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings

    cs.LG 2025-06 conditional novelty 5.0 of 10

    A string-based, metadata-guided language model framework for universal offline black-box optimization, with two variants and two embedding regularizations.

Reference graph

Works this paper leans on

54 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Reyes, Grant C

    Yash Akhauri, Bryan Lewandowski, Cheng-Hsi Lin, Adrian N. Reyes, Grant C. Forbes, Arissa Wongpanich, Bangding Yang, Mohamed S. Abdelfattah, Sagi Perel, and Xingyou Song. Performance prediction for large systems via text-to-text regression, 2025

    work page 2025

  3. [3]

    Bowman, Zac Hatfield - Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran - Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse,...

    arXiv doi:10.48550/arxiv.2212.08073 2022

  4. [4]

    Bellemare, Will Dabney, and R \' e mi Munos

    Marc G. Bellemare, Will Dabney, and R \' e mi Munos. A distributional perspective on reinforcement learning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , volume 70 of Proceedings of Machine Learning Research, pp.\ 449--458. PMLR , 2017

    work page 2017

  5. [5]

    Christopher M. Bishop. Mixture density networks. Technical report, Aston University, 1994

    1994

  6. [6]

    Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952. ISSN 00063444, 14643510

    1952

  7. [7]

    Rank consistent ordinal regression for neural networks with application to age estimation

    Wenzhi Cao, Vahid Mirjalili, and Sebastian Raschka. Rank consistent ordinal regression for neural networks with application to age estimation. Pattern Recognit. Lett., 140: 0 325--331, 2020. doi:10.1016/J.PATREC.2020.11.008

    doi:10.1016/j.patrec.2020.11.008 2020

  8. [8]

    Linear algebra with transformers

    Fran c ois Charton. Linear algebra with transformers. Trans. Mach. Learn. Res., 2022, 2022

    work page 2022

Show all 54 references
  1. [9]

    Towards learning universal hyperparameter optimizers with transformers

    Yutian Chen, Xingyou Song, Chansoo Lee, Zi Wang, Richard Zhang, David Dohan, Kazuya Kawakami, Greg Kochanski, Arnaud Doucet, Marc'Aurelio Ranzato, Sagi Perel, and Nando de Freitas. Towards learning universal hyperparameter optimizers with transformers. In Sanmi Koyejo, S. Moha...

    2022

  2. [10]

    TRACT: regression-aware fine-tuning meets chain-of-thought reasoning for llm-as-a-judge

    Cheng - Han Chiang, Hung - yi Lee, and Michal Lukasik. TRACT: regression-aware fine-tuning meets chain-of-thought reasoning for llm-as-a-judge. CoRR, abs/2503.04381, 2025. doi:10.48550/ARXIV.2503.04381

    2025 doi arXiv

  3. [11]

    Deep symbolic regression for recurrent sequences

    St \' e phane d'Ascoli, Pierre - Alexandre Kamienny, Guillaume Lample, and Fran c ois Charton. Deep symbolic regression for recurrent sequences. CoRR, abs/2201.04600, 2022

    2022 arXiv

  4. [12]

    Soft labels for ordinal regression

    Raul Diaz and Amit Marathe. Soft labels for ordinal regression. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019 , pp.\ 4738--4747. Computer Vision Foundation / IEEE , 2019. doi:10.1109/CVPR.2019.00487

    2019

  5. [13]

    UCI machine learning repository, 2017

    Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http://archive.ics.uci.edu/ml

    2017

  6. [14]

    Coco: the large scale black-box optimization benchmarking (bbob-largescale) test suite

    Ouassim Elhara, Konstantinos Varelas, Duc Nguyen, Tea Tusar, Dimo Brockhoff, Nikolaus Hansen, and Anne Auger. Coco: the large scale black-box optimization benchmarking (bbob-largescale) test suite. arXiv preprint arXiv:1903.06396, 2019

    1903 arXiv

  7. [15]

    Angela Fan, Mike Lewis, and Yann N. Dauphin. Hierarchical neural story generation. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Lo...

    2018 doi

  8. [16]

    Open ML - CTR 23 a curated tabular regression benchmarking suite

    Sebastian Felix Fischer, Liana Harutyunyan Matthias Feurer, and Bernd Bischl. Open ML - CTR 23 a curated tabular regression benchmarking suite. In AutoML Conference 2023 (Workshop), 2023

    2023

  9. [17]

    Pieter Gijsbers, Marcos L. P. Bueno, Stefan Coors, Erin LeDell, S \' e bastien Poirier, Janek Thomas, Bernd Bischl, and Joaquin Vanschoren. AMLB: an automl benchmark. J. Mach. Learn. Res., 25: 0 101:1--101:65, 2024

    2024

  10. [18]

    Bayesian Neural Networks: An Introduction and Survey

    Ethan Goan and Clinton Fookes. Bayesian Neural Networks: An Introduction and Survey. Springer International Publishing, Cham, 2020. ISBN 978-3-030-42553-1. doi:10.1007/978-3-030-42553-1_3

    2020 doi

  11. [19]

    Sequence transduction with recurrent neural networks

    Alex Graves. Sequence transduction with recurrent neural networks. CoRR, abs/1211.3711, 2012

    2012 arXiv

  12. [20]

    Gustafsson, Martin Danelljan, Goutam Bhat, and Thomas B

    Fredrik K. Gustafsson, Martin Danelljan, Goutam Bhat, and Thomas B. Sch \" o n. Energy-based models for deep probabilistic regression. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan - Michael Frahm (eds.), Computer Vision - ECCV 2020 - 16th European Conference, Glasgow...

    2020 doi

  13. [21]

    Harrell and C

    Frank E. Harrell and C. E. Davis. A new distribution-free quantile estimator. Biometrika, 69 0 (3): 0 635--640, 1982. ISSN 00063444, 14643510

    1982

  14. [22]

    u ller, Lennart Purucker, Arjun Krishnakumar, Max K \

    Noah Hollmann, Samuel M \" u ller, Lennart Purucker, Arjun Krishnakumar, Max K \" o rfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nat., 637 0 (8044): 0 319--326, 2025. doi:10.1038/S41586-024-08328-6

    2025 doi

  15. [23]

    The curious case of neural text degeneration

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020

    2020

  16. [24]

    Ieee standard for floating-point arithmetic

    IEEE. Ieee standard for floating-point arithmetic. IEEE Std 754-2019 (Revision of IEEE 754-2008), pp.\ 1--84, 2019. doi:10.1109/IEEESTD.2019.8766229

    2019

  17. [25]

    Improving regression performance with distributional losses

    Ehsan Imani and Martha White. Improving regression performance with distributional losses. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm \" a ssan, Stockholm, Sweden, July 10-15, 2018 ,...

    2018

  18. [26]

    Kingma and Max Welling

    Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , 2014

    2014

  19. [27]

    Bayesian approach for neural networks--review and case studies

    Jouko Lampinen and Aki Vehtari. Bayesian approach for neural networks--review and case studies. Neural Networks, 14 0 (3): 0 257--274, 2001. doi:10.1016/S0893-6080(00)00098-8

    2001 doi

  20. [28]

    L.E. Lehmann. Theory of Point Estimation. A Wiley publication in mathematical statistics. Wiley, 1983

    1983

  21. [29]

    Energy-based models for deep probabilistic regression

    Xixi Liu, Che-Tsung Lin, and Christopher Zach. Energy-based models for deep probabilistic regression. In 2022 26th International Conference on Pattern Recognition (ICPR), pp.\ 2693--2699, 2022. doi:10.1109/ICPR56361.2022.9955636

    2022

  22. [30]

    Regression aware inference with llms

    Michal Lukasik, Harikrishna Narasimhan, Aditya Krishna Menon, Felix Yu, and Sanjiv Kumar. Regression aware inference with llms. In Yaser Al - Onaizan, Mohit Bansal, and Yun - Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florid...

    2024

  23. [31]

    Yu, and Sanjiv Kumar

    Michal Lukasik, Zhao Meng, Harikrishna Narasimhan, Aditya Krishna Menon, Yin Wen Chang, Felix X. Yu, and Sanjiv Kumar. Better autoregressive regression with LLM s. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, Singapore, April 24...

    2025

  24. [32]

    Generative reward models

    Dakota Mahan, Duy Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan - Philipp Fr \" a nken, Chelsea Finn, and Alon Albalak. Generative reward models. CoRR, abs/2410.12832, 2024. doi:10.48550/ARXIV.2410.12832

    2024 doi arXiv

  25. [33]

    Radford M. Neal. Connectionist learning of belief networks. Artif. Intell., 56 0 (1): 0 71--113, 1992. doi:10.1016/0004-3702(92)90065-6

    1992 doi

  26. [34]

    Predicting from strings: Language model embeddings for bayesian optimization

    Tung Nguyen, Qiuyi Zhang, Bangding Yang, Chansoo Lee, Jorg Bornschein, Yingjie Miao, Sagi Perel, Yutian Chen, and Xingyou Song. Predicting from strings: Language model embeddings for bayesian optimization. CoRR, abs/2410.10190, 2024. doi:10.48550/ARXIV.2410.10190

    doi

  27. [35]

    Investigating the limitations of the transformers with simple arithmetic tasks

    Rodrigo Frassetto Nogueira, Zhiying Jiang, and Jimmy Lin. Investigating the limitations of the transformers with simple arithmetic tasks. CoRR, abs/2102.13019, 2021

    2021 arXiv

  28. [36]

    On Estimation of a Probability Density Function and Mode

    Emanuel Parzen. On Estimation of a Probability Density Function and Mode . The Annals of Mathematical Statistics, 33 0 (3): 0 1065 -- 1076, 1962. doi:10.1214/aoms/1177704472

    1962

  29. [37]

    Hamming-distance-based binary representation of numbers

    Minghai Qin. Hamming-distance-based binary representation of numbers. In 2018 IEEE International Symposium on Information Theory (ISIT), pp.\ 2202--2205, 2018. doi:10.1109/ISIT.2018.8437644

    2018

  30. [38]

    The infinite gaussian mixture model

    Carl Edward Rasmussen. The infinite gaussian mixture model. In Sara A. Solla, Todd K. Leen, and Klaus - Robert M \" u ller (eds.), Advances in Neural Information Processing Systems 12, [NIPS Conference, Denver, Colorado, USA, November 29 - December 4, 1999] , pp.\ 554--560. Th...

    1999

  31. [39]

    Remarks on Some Nonparametric Estimates of a Density Function

    Murray Rosenblatt. Remarks on Some Nonparametric Estimates of a Density Function . The Annals of Mathematical Statistics, 27 0 (3): 0 832 -- 837, 1956. doi:10.1214/aoms/1177728190

    1956

  32. [40]

    Omnipred: Language models as universal regressors

    Xingyou Song, Oscar Li, Chansoo Lee, Bangding Yang, Daiyi Peng, Sagi Perel, and Yutian Chen. Omnipred: Language models as universal regressors. CoRR, abs/2402.14547, 2024

    2024 arXiv

  33. [41]

    Understanding LLM embeddings for regression

    Eric Tang, Bangding Yang, and Xingyou Song. Understanding LLM embeddings for regression. CoRR, abs/2411.14708, 2024. doi:10.48550/ARXIV.2411.14708

    2024 doi arXiv

  34. [42]

    Learning stochastic feedforward neural networks

    Yichuan Tang and Ruslan Salakhutdinov. Learning stochastic feedforward neural networks. In Christopher J. C. Burges, L \' e on Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger (eds.), Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Inf...

    2013

  35. [43]

    Yee Whye Teh, Max Welling, Simon Osindero, and Geoffrey E. Hinton. Energy-based models for sparse overcomplete representations. J. Mach. Learn. Res., 4: 0 1235--1260, 2003

    2003

  36. [44]

    D. M. Titterington. Bayesian Methods for Neural Networks and Related Models . Statistical Science, 19 0 (1): 0 128 -- 139, 2004. doi:10.1214/088342304000000099

    2004 doi

  37. [46]

    van Rijn, Bernd Bischl, and Lu \' s Torgo

    Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Lu \' s Torgo. Openml: networked science in machine learning. SIGKDD Explor. , 15 0 (2): 0 49--60, 2013. doi:10.1145/2641190.2641198

    2013

  38. [47]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (...

    2017

  39. [48]

    A no-regret generalization of hierarchical softmax to extreme multi-label classification

    Marek Wydmuch, Kalina Jasinska, Mikhail Kuznetsov, R \' o bert Busa - Fekete, and Krzysztof Dembczynski. A no-regret generalization of hierarchical softmax to extreme multi-label classification. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicol \` o Ces...

    2018

  40. [49]

    Generative verifiers: Reward modeling as next-token prediction

    Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. CoRR, abs/2408.15240, 2024. doi:10.48550/ARXIV.2408.15240

    2024 doi arXiv

  41. [50]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei - Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Proce...

    2023

  42. [51]

    Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B

    Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. CoRR, abs/1909.08593, 2019

    1909 arXiv

  43. [52]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  44. [53]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  45. [54]

    Dua, Dheeru and Graff, Casey

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2024 arXiv

  46. [55]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.