Early Stopping Tabular In-Context Learning

Frank Hutter; Jaris K\"uken; Lennart Purucker

REVIEW 4 major objections 6 minor 1 cited by

Early Stopping Tabular In-Context Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Early exiting a tabular foundation model at intermediate layers speeds inference up to 2.2x while keeping accuracy nearly intact.

desk verdict Prior-trained layer-wise decoders make TabPFN early exits work without fine-tuning, but the speedup claims rest on missing baseline timings and an unvalidated entropy threshold. read the letter →

arxiv 2506.21387 v2 pith:NXXFSHQL submitted 2025-06-26 cs.LG

Jaris Küken , Lennart Purucker , Frank Hutter This is my paper · ORCID

classification cs.LG

keywords earlyexitingtabularfoundationmodelsin-contextlearninginferenceaccelerationentropy-basedstoppingTransformerefficiencyTabPFNclassification

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a tabular foundation model can cut inference cost by stopping its forward pass before the last Transformer layer. The proposed mechanism attaches a small, pre-trained decoder to each layer and exits when the average softmax entropy of the test-set predictions falls below a threshold, with no task-specific finetuning. On 34 small classification tasks this speeds up inference by up to 1.3x with negligible accuracy loss, and on five larger tasks by up to 2.2x. If the claim holds, early exiting becomes a practical, cheap efficiency lever for in-context tabular learning, which matters because self-attention cost grows quadratically with context size.

What carries the argument

The central mechanism is a set of layer-specific decoders plus an entropy-based stopping rule. Each decoder shares the architecture of the final TabPFN decoder, is trained independently on roughly 820,000 synthetic datasets from the prior with the Transformer backbone frozen, and converts an intermediate layer's representation into class probabilities. At inference, after each encoder layer the decoder is applied to all test samples, the average softmax entropy of those predictions is computed, and if it falls below a dataset-specific threshold $\tau$ the forward pass stops early. This makes the decoders the read-out device and entropy the control signal, combining to trade a small amount of accuracy for skipped computation.

What would settle it

Take a classification dataset with strong label imbalance or class overlap, run the method with $\tau = 0.4$, and compare accuracy when exiting at the chosen layer against full-model accuracy; if low-entropy early exits systematically underperform the full model by more than the paper's reported bounds, the entropy criterion is not a trustworthy correctness signal.

Watch

Extended reading notes

Core claim

The core discovery is that intermediate Transformer layers of TabPFN already carry enough predictive structure to classify many test sets correctly, as long as each layer is paired with its own decoder pre-trained on synthetic data drawn from the prior. Averaged softmax entropy across the test set works as an exit signal: when it dips below a threshold $\tau$, the model returns that layer's predictions instead of running all twelve layers. With this rule, average exit layers drop to roughly 5–7 for moderate thresholds while ROC AUC stays within a few points of the full forward pass. The authors interpret this as evidence that in-context learning converges incrementally, with confident predictions available well before the final layer.

Load-bearing premise

The method assumes that when the average softmax entropy of test-set predictions at an intermediate layer is low, those predictions are actually correct, and that this trust survives across datasets, dataset sizes, and a fixed threshold grid.

Editorial extensions

If this is right

Any layered tabular foundation model could be accelerated by adding pre-trained exit decoders, with no downstream finetuning.
Speedups should grow with context size since self-attention scales quadratically; the 2.2x gains on five larger tasks point in this direction.
Decoders trained only on synthetic datasets of about 1,000 samples transfer to datasets up to 5,000 samples, suggesting the prior captures structure that scales.
The threshold $\tau$ provides a single interpretable knob for trading accuracy against latency in deployed systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A per-sample variant that exits each test point individually rather than by average entropy would likely recover more speed on mixed datasets, since average statistics can mask confident subgroups.
Choosing $\tau$ per dataset by a small grid search is the main practical friction; a validation-free or prior-based calibration rule is an implicit next step the paper does not develop.
The same decoder-plus-entropy recipe could be tested on other in-context tabular learners such as TabICL or TabDPT, which the paper mentions but does not evaluate.
Because the decoders are cheap and pre-trained, the approach could be stacked with other efficiency tricks like input-conditional layer skipping to push beyond the reported 2.2x.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing to know: this is a competent engineering paper whose qualitative result is honest but whose headline numbers don't survive close reading. The genuinely new piece is applying early-exit (entropy thresholds + layer-wise decoders) to TabPFN, with the decoders pre-trained on the synthetic prior rather than fine-tuned per task. That preserves TabPFN's zero-shot property, and I don't know of prior early-exit work for tabular in-context learning.

What it does well: the method is simple, clearly described, and well motivated. Figure 2 convincingly shows intermediate TabPFN layers carry decodable signal, and the evaluation is reasonably extensive: 34 small plus 5 larger datasets, 10-fold CV, plus a real test of generalization since the decoders were trained only on synthetic data up to ~1k samples yet transfer to 5k-sample tasks. No circularity problem worth raising—training on the prior, testing on real benchmarks.

The soft spots are in the quantitative reporting, and they're substantial. No absolute wall-clock baselines anywhere—Table 1 gives only runtime deltas in seconds—so the claimed x1.3 and x2.2 speedups can't be verified. The "up to" framing means these are maxima over datasets and thresholds; averaged deltas imply more modest mean gains. Most load-bearing: no tuning rule for tau. Section 3 calls it "dataset specific, tunable," but no validation-based selection is described, so the thresholds appear to be scanned on test data. As reported, the speedups are retrospective, not predictive. And the per-dataset appendix contradicts the "negligible" framing: phoneme falls from 0.969 to 0.877 ROC AUC at tau=0.4, parity5 from 1.0 to 0.725 at tau=0.5; the averaged tables hide this variance. No code or decoder checkpoints are released either.

Minor but telling: Table 8 is headed "Small Datasets" while reporting the large ones, Appendix D has a dangling "Figure??", and the decoder prior is attributed to Muller et al. (2023) in Section 3 but Hollmann et al. (2023) in Appendix A. Cosmetic, but signs of a rushed final pass.

The central qualitative claim holds up; the quantitative claims need verification artifacts and a principled tuning procedure, both fixable. This is for people working on tabular foundation model efficiency or deploying TabPFN in latency-sensitive settings. It deserves a serious referee, not a desk reject; the right outcome is major revision demanding absolute timings, per-dataset speedup ratios, and a validation-based tau rule.

Referee Report

4 major / 6 minor

Summary. The paper proposes an early-exit mechanism for tabular in-context learning with TabPFN. Layer-wise decoders are pre-trained on synthetic data from the same prior used for TabPFN, and at inference time the average softmax entropy of the test-set predictions at each layer is compared with a threshold tau to decide whether to stop the forward pass early. Experiments on 34 small PMLB-mini classification tasks and five larger datasets report speedups of up to 1.3x and 2.2x respectively, with claimed negligible or small accuracy loss. The contribution is presented as a finetuning-free efficiency lever for tabular foundation models.

Significance. If the reported speedups are reproducible and the threshold can be set without test labels, this is a practically useful and inexpensive addition to TabPFN-style in-context learners. The method has clear strengths: it requires no task-specific finetuning, the decoders are pre-trained once on synthetic data, and the paper includes per-dataset results in Appendix E, which is good transparency. The significance is currently tempered by three unresolved issues: the threshold selection protocol is unspecified, absolute baseline runtimes are never reported, and the 'negligible degradation' claim is contradicted by per-dataset results in Appendix E. These are addressable, which is why I view the paper as a major revision rather than a rejection.

major comments (4)

[Section 3 and Section 4] Section 3 calls tau a 'dataset specific, tunable hyperparameter,' but no tuning procedure is given; Section 4 simply evaluates five fixed thresholds, and the headline 'up to' speedups appear to be maxima over datasets and thresholds. On a new dataset, a user cannot choose tau without test labels. Please provide a validation-based protocol (e.g., choose tau on a held-out validation fold using the same rule for every dataset) and report results under that protocol, or clearly restrict the claims to the retrospective per-threshold analysis.
[Table 1] Table 1 reports only 'Runtime Delta in (s)' relative to the full TabPFN forward pass and never reports absolute baseline wall-clock times or per-dataset speedup ratios. Since the claimed speedups (x1.3 and x2.2) are defined as baseline time divided by early-exit time, the reported deltas alone are insufficient to verify the speedup factors. Please report absolute runtimes for both baseline and early-exit inference, together with per-dataset speedup ratios and standard errors.
[Abstract, Section 5, and Appendix E] The claim of 'negligible degradation' is not supported by the per-dataset results in Appendix E. For phoneme, the ROC AUC drops from 0.969 at tau=0.1 to 0.877 at tau=0.4, an absolute drop of 0.092; for parity5, the ROC AUC drops from 1.000 to 0.725 at tau=0.5. These large per-dataset losses are hidden by the averaged Table 1, which shows only a decrease from 0.960 to 0.920 for large datasets. Please report worst-case and per-dataset degradation, and define 'negligible' with a pre-specified tolerance rather than relying on averages.
[Algorithm 1 and Section 3] Algorithm 1 computes the entropy H of a single prediction p and appears to decide per sample, whereas Section 3 states that the entropies are averaged across the test set and the average is compared with tau. This is not a cosmetic discrepancy: a global exit decision for the entire test set is a different mechanism from per-sample early exiting, with different implications for the speedup and for the interpretation of the entropy signal. Please clarify whether exits are per-sample or per test set and align the pseudocode with the actual implementation.

minor comments (6)

[Section 3] The sentence 'we decide whether to exist early based on prediction entropy' should read 'whether to exit early.'
[Abstract] The phrase 'Experiments across 34 small classification tasks size' is missing a word and should be rephrased, for example 'tasks of increasing size' or simply 'tasks.'
[Table 1 caption] The caption says 'across 34 small and 4 large classification datasets,' but the text and Appendix E list five large datasets (ada, churn, phoneme, Satellite, sylvine).
[Table 8 caption] The caption for Table 8 says 'for Small Datasets' but the table reports results for the large datasets.
[Appendix D] The reference 'Figure??' is unresolved and should be replaced with the correct figure number.
[Section 4 and Section 5] The notation '5.000 samples' and 'up to 5.000 samples' uses a decimal point as a thousands separator; use '5,000' or '5000' for consistency with standard English convention.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the speedup and accuracy claims are empirical, evaluated on external benchmarks, and no prediction reduces to a fitted parameter or to a load-bearing self-citation.

full rationale

The paper's central claim—that entropy-based early exiting speeds up TabPFN by up to x1.3 on small tasks and x2.2 on larger tasks with small accuracy loss—is an empirical result measured on PMLB-mini datasets and five larger external datasets. The layer-wise decoders are pretrained on synthetic data from TabPFN's prior with the Transformer frozen, and the method is then evaluated on real-world benchmarks not used for decoder training, so the reported predictive performance is not definitionally forced. The threshold tau is presented as a user-set, dataset-specific hyperparameter and is swept over five fixed values with per-threshold results reported; it is neither fitted to the test folds nor defined in terms of the reported speedup, so there is no fitted input being renamed as a prediction. The entropy criterion is a standard early-exit heuristic borrowed from BranchyNet and DeeBERT, not an equation whose output equals its input. Self-citations to TabPFN (Hollmann et al.) are background context for the base model and do not carry the efficiency argument; no uniqueness theorem or ansatz is imported from the same authors to forbid alternatives. The absence of a validation-based rule for choosing tau and the reporting of runtime deltas without absolute baseline wall-clock times are legitimate correctness and reproducibility concerns, but they are not circularity: nothing in the claimed derivation reduces by construction to a fitted value, an input label, or a self-citation chain.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim's only tunable quantity is the entropy threshold tau, which is scanned rather than derived. The method rests on three domain assumptions (entropy calibration, prior-to-benchmark transfer, intermediate-layer decodability) plus standard transformer math. No new physical or conceptual entities are introduced; the layer-wise decoders are trained architectural components, not postulates.

free parameters (1)

Entropy threshold tau = 0.1 to 0.5 (grid of five values)
Hand-set hyperparameter controlling early exit. Section 3 calls it 'a dataset specific, tunable hyperparameter'; reported speedups depend on choosing favorable threshold values, and no validation-based selection is described.

assumptions (4)

domain assumption Average softmax entropy of unlabeled test predictions is a reliable early-exit signal.
Invoked in Section 3 and Algorithm 1: the forward pass stops when average entropy falls below tau. The paper provides no calibration analysis, so this is an unverified assumption.
domain assumption Decoders pre-trained on synthetic data from the TabPFN prior transfer to real benchmark tasks.
Section 3 trains each decoder on synthetic datasets from the prior, then Section 5 evaluates on real PMLB-mini and TabPFNv2 benchmarks, assuming out-of-distribution transfer.
domain assumption Intermediate transformer layers of TabPFN contain enough predictive information for a task-agnostic decoder.
Stated as motivation in Section 3 and tested empirically in Figure 2, but the entire method depends on this decodability property.
standard math Standard transformer forward pass and softmax entropy computation behave as specified.
Used without proof in Algorithm 1; standard background.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Early Stopping Tabular In-Context Learning." pith.science (2026). https://pith.science/paper/NXXFSHQL

@misc{pith2026250621387,
  author       = {Pith},
  title        = {Pith review of: Early Stopping Tabular In-Context Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NXXFSHQL}},
  note         = {Machine review of arXiv:2506.21387}
}

read the original abstract

Tabular foundation models have shown strong performance across various tabular learning tasks via in-context learning, offering robust generalization without any downstream finetuning. However, their inference-time costs remain high, particularly for larger datasets. To address this, we propose early-stopping the in-context learning process. We achieve this by dynamically evaluating whether to stop in-context learning after each Transformer encoder layer. Once stopped, we decode the embedding using a pre-trained layer-wise decoder. Experiments across 34 small classification tasks size show that early stopping in-context learning accelerates inference by up to x1.3 with negligible degradation in predictive performance. To assess scalability, we further evaluate our method on five larger classification tasks, achieving speedups of up to x2.2. Our results demonstrate the potential of early exiting as an effective and practical strategy for improving the efficiency of tabular in-context learning.

Figures

Figures reproduced from arXiv: 2506.21387 by the authors.

**Figure 1.** Early Exit Strategy for TabPFN. We extend TabPFN with an early-exit mechanism for in-context learning. For each Transformer layer, we pre-train a dedicated decoder on synthetically generated data from the prior (blue) while keeping the Transformer layers (green) and the final decoder frozen. During inference, all test samples are passed through each decoder in sequence. If the average prediction entropy falls below … view at source ↗

**Figure 2.** shows the classification accuracy obtained when exiting TabPFN at different Transformer layers using their corresponding pre-trained decoders, compared to using only the final decoder. Results are averaged over 34 small-scale datasets from the PMLB-mini (Knauer et al., 2024) benchmark. We observe that by pre-training individual decoders for each layer stabilizes performance across layers, with several intermediate … view at source ↗

**Figure 3.** Tradeoff between Improved Runtime and Decrease in Predictive Performance. We report the relative improvements in runtime as well as the relative decrease in ROC AUC score averaged over 5 large datasets when early stopping based on entropy for 5 different entropy thresholds. It is trivial to see that a higher threshold τ leads to higher improvements in runtime, but also leads to a stronger decrease in terms of predi… view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Tradeoff between Improved Runtime and Decrease in Predictive Performance. We report the relative improvements in runtime as well as the relative decrease in ROC AUC score averaged over 34 small datasets when early stopping based on entropy for 5 different entropy thres…

**Figure 5.** Figure 5: Performance per Layer over 5 Large Datasets with Individual Decoders. We present the performance average ROC AUC score over 5 large datasets with up to 5k samples averaged over 10 folds as well as the 95% confidence interval. Additionally we present the baseline of Tab…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Algorithmic Recourse of In-Context Learning for Tabular Data
cs.LG 2026-05 unverdicted novelty 7.0 of 10

The paper delivers the first theoretical analysis and practical zeroth-order framework for algorithmic recourse under in-context learning for tabular prediction.

Reference graph

Works this paper leans on

19 extracted references · 9 canonical work pages · cited by 1 Pith paper

[3]

We train all decoders on a single NVIDIA RTX 2080 GPU. Table

work page
[8]

doi: 10.18653/v1/2020.acl-main

Association for Com- putational Linguistics. doi: 10.18653/v1/2020.acl-main

doi:10.18653/v1/2020.acl-main 2020
[10]

arXiv:2410.18164 [cs]

URL http://arxiv.org/ abs/2410.18164. arXiv:2410.18164 [cs]. M¨uller, A., Curino, C., and Ramakrishnan, R. Mothernet: A foundational hypernetwork for tabular classification. arXiv:2312.08598 [cs.LG],

arXiv
[12]

arXiv:2501.16825 [cs]

URL http://arxiv.org/ abs/2501.16825. arXiv:2501.16825 [cs]. Sanh, V ., Debut, L., Chaumond, J., and Wolf, T. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter, March

arXiv
[13]

arXiv:1910.01108 [cs]

URL http://arxiv.org/ abs/1910.01108. arXiv:1910.01108 [cs]. Shen, S., Zhen, D., Ye, J., Ma, L., Yao, Z., Gholami, A., Mahoney, M., and Keutzer, K. Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT. Proceedings of the AAAI Conference on Artificial Intelligence , 34: 8815–8821, April

arXiv 1910
[14]

Teerapittayanon, S., McDanel, B., and Kung, H

doi: 10.1609/aaai.v34i05.6409. Teerapittayanon, S., McDanel, B., and Kung, H. BranchyNet: Fast inference via early exiting from deep neural networks. In 2016 23rd International Conference on Pattern Recognition (ICPR) , pp. 2464– 2469, December

work page doi:10.1609/aaai.v34i05.6409 2016
[15]

doi: 10.1109/ICPR.2016. 7900006. URL https://ieeexplore.ieee. org/document/7900006. van Breugel, B. and van der Schaar, M. Why tabular foun- dation models should be a research priority. In Salakhut- dinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Machin...

work page arXiv 2016
[16]

Attention is all you need

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A., Kaiser, L., and Polosukhin, I. Attention is all you need. In Guyon et al. (2017). Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., and Zhou, M. MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Trans- formers. In Advances in Neural Information Pro...

work page 2017

Show all 19 references

[17]

neurips.cc/paper/2020/hash/ 3f5ee243547dee91fbd053c1c4a845aa-Abstract

URL https://proceedings. neurips.cc/paper/2020/hash/ 3f5ee243547dee91fbd053c1c4a845aa-Abstract. html. Xie, S. M., Raghunathan, A., Liang, P., and Ma, T. An Ex- planation of In-context Learning as Implicit Bayesian In- ference. October

2020
[18]

doi: 10.18653/v1/2020.acl-main.204

Association for Computational Linguis- tics. doi: 10.18653/v1/2020.acl-main.204. URL https: //aclanthology.org/2020.acl-main.204/. Zafrir, O., Boudoukh, G., Izsak, P., and Wasserblat, M. Q8BERT: Quantized 8Bit BERT. In2019 Fifth Workshop on Energy Efficient Machine Learning an...

2020 doi
[537]

acl-main.537/

URL https://aclanthology.org/2020. acl-main.537/. Ma, J., Thomas, V ., Hosseinzadeh, R., Kamkari, H., Labach, A., Cresswell, J. C., Golestan, K., Yu, G., V olkovs, M., and Caterini, A. L. TabDPT: Scaling Tabular Foundation Models, October

2020
[2015]

arXiv: 1503.02531

URL http://arxiv.org/ abs/1503.02531. arXiv: 1503.02531. Hollmann, N., M ¨uller, S., Eggensperger, K., and Hutter, F. TabPFN: A transformer that solves small tabular classification problems in a second. In The Eleventh International Conference on Learning Representations (ICLR...

arXiv
[2016]

BERT: Pre-training of deep bidirectional transformers for lan- guage understanding

Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for lan- guage understanding. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Com...

2019
[2019]

doi: 10.1109/EMC2-NIPS53020.2019. 00016. URL https://ieeexplore.ieee.org/ abstract/document/9463531. 6 Early Stopping Tabular In-Context Learning A. Decoder Pre-Training. We pre-train each decoder independently on synthetic datasets sampled from the prior, following the prior ...

2019
[2020]

neurips.cc/paper/2020/hash/ 6f5216f8d89b086c18298e043bfe48ed-Abstract

URL https://proceedings. neurips.cc/paper/2020/hash/ 6f5216f8d89b086c18298e043bfe48ed-Abstract. html. Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.-Y . Lightgbm: A highly efficient gradient boosting decision tree. In Guyon et al. (2017). Kim, S....

2017
[2021]

doi: 10.18653/v1/2021

Association for Computational Linguistics. doi: 10.18653/v1/2021. findings-emnlp.43. URL https://aclanthology. org/2021.findings-emnlp.43/. 5 Early Stopping Tabular In-Context Learning Liu, W., Zhou, P., Wang, Z., Zhao, Z., Deng, H., and Ju, Q. FastBERT: a Self-distilling BERT...

2021 doi
[2022]

Language models are few-shot learners

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-V oss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin...

1901
[2024]

org/abs/2409.01635

URL http://arxiv. org/abs/2409.01635. arXiv:2409.01635 [cs]. Li, L., Lin, Y ., Chen, D., Ren, S., Li, P., Zhou, J., and Sun, X. CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cas- cade. In Moens, M.-F., Huang, X., Specia, L., ...

2021 arXiv
[2025]

arXiv:2502.05564 [cs]

URL http:// arxiv.org/abs/2502.05564. arXiv:2502.05564 [cs]. Reuter, A., Rudner, T. G. J., Fortuin, V ., and R ¨ugamer, D. Can Transformers Learn Full Bayesian Inference in Context?, January

arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [3]

We train all decoders on a single NVIDIA RTX 2080 GPU. Table

work page

[2] [8]

doi: 10.18653/v1/2020.acl-main

Association for Com- putational Linguistics. doi: 10.18653/v1/2020.acl-main

doi:10.18653/v1/2020.acl-main 2020

[3] [10]

arXiv:2410.18164 [cs]

URL http://arxiv.org/ abs/2410.18164. arXiv:2410.18164 [cs]. M¨uller, A., Curino, C., and Ramakrishnan, R. Mothernet: A foundational hypernetwork for tabular classification. arXiv:2312.08598 [cs.LG],

arXiv

[4] [12]

arXiv:2501.16825 [cs]

URL http://arxiv.org/ abs/2501.16825. arXiv:2501.16825 [cs]. Sanh, V ., Debut, L., Chaumond, J., and Wolf, T. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter, March

arXiv

[5] [13]

arXiv:1910.01108 [cs]

URL http://arxiv.org/ abs/1910.01108. arXiv:1910.01108 [cs]. Shen, S., Zhen, D., Ye, J., Ma, L., Yao, Z., Gholami, A., Mahoney, M., and Keutzer, K. Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT. Proceedings of the AAAI Conference on Artificial Intelligence , 34: 8815–8821, April

arXiv 1910

[6] [14]

Teerapittayanon, S., McDanel, B., and Kung, H

doi: 10.1609/aaai.v34i05.6409. Teerapittayanon, S., McDanel, B., and Kung, H. BranchyNet: Fast inference via early exiting from deep neural networks. In 2016 23rd International Conference on Pattern Recognition (ICPR) , pp. 2464– 2469, December

work page doi:10.1609/aaai.v34i05.6409 2016

[7] [15]

doi: 10.1109/ICPR.2016. 7900006. URL https://ieeexplore.ieee. org/document/7900006. van Breugel, B. and van der Schaar, M. Why tabular foun- dation models should be a research priority. In Salakhut- dinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Machin...

work page arXiv 2016

[8] [16]

Attention is all you need

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A., Kaiser, L., and Polosukhin, I. Attention is all you need. In Guyon et al. (2017). Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., and Zhou, M. MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Trans- formers. In Advances in Neural Information Pro...

work page 2017