Pith. sign in

REVIEW 2 major objections 5 minor 51 references

Test-time augmentation improves efficiency in conformal prediction

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Test-time augmentation shrinks conformal prediction sets by 10-14% while preserving coverage.

desk verdict A solid, honestly reported empirical result on TTA for conformal prediction; the coverage guarantee needs one missing detail—where RAPS hyperparameters are tuned—resolved before it's fully clean. read the letter →

arxiv 2505.22764 v1 pith:MT4XAVNL submitted 2025-05-28 cs.LG cs.CV

Divya Shanmugam , Helen Lu , Swami Sankaranarayanan , John Guttag This is my paper · ORCID
classification cs.LGcs.CV
keywords conformalpredictiontest-timeaugmentationsetefficiencyexchangeabilitydistributionshiftimageclassificationadaptivesets
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that test-time augmentation (TTA), which aggregates a classifier's predictions over transformed copies of an input, can be inserted into split conformal prediction to produce smaller prediction sets at the same nominal coverage. It claims this is a general efficiency boost: it works with two conformal scores (APS and RAPS), three architectures, three image datasets, and four corruption shifts, with average set-size reductions of about 10% in-distribution and 14% under shift, and no loss of coverage. This matters because conformal classifiers often produce uninformatively large sets, and the proposed fix requires no retraining of the base model and no extra base classifiers, only a learned aggregation of augmentations plus a fraction of the already-labeled data.

What carries the argument

The load-bearing object is the learned TTA aggregation $g(x; f, A, \theta) = \sigma(\theta^\top A(f, A, x))$, in which $A(f, A, x)$ is the $M \times K$ matrix of logits the pretrained classifier produces on $M$ augmented views of $x$, and $\theta$ is a vector of augmentation weights learned by cross-entropy on a dedicated split $\mathcal{D}_{\mathrm{TTA}}$. It does two jobs: it induces useful invariances into the probability vector that feeds the conformal score, and, because $\theta$ is learned on examples disjoint from the calibration set, it remains a deterministic transformation applied equally to calibration and test points, preserving the exchangeability on which the coverage guarantee rests.

What would settle it

Learn the TTA weights on the calibration set itself and check whether empirical coverage falls below $1-\alpha$ on a held-out test set; or apply the expanded policy to a dataset like MNIST, where inverting colors changes digit labels, and check whether set sizes grow and coverage degrades.

Watch

Extended reading notes

Core claim

The central claim is that replacing the classifier's raw probability vector $f(x)$ in the conformal score with a TTA-aggregated vector $g(x; f, A, \theta) = \sigma(\theta^\top A(f, A, x))$, where $A$ stacks predicted logits over $m$ augmentations and $\theta$ is learned to maximize cross-entropy on a separate labeled split, reduces the size of split-conformal prediction sets at level $1-\alpha$ while preserving coverage. The paper shows that the effect is not mainly about fixing top-1 errors (only about 3% of shrinking sets come from corrected top-1 classification) but about promoting the true class's rank in the predicted-probability ordering, so the cumulative-probability conformal scores (APS and RAPS) need to include fewer wrong classes. It reports average set-size reductions of about 10% in-distribution and 14% under distribution shift, with no coverage loss, and finds that classes with the largest sets and hardest classes benefit most.

Load-bearing premise

The coverage guarantee rests on the assumption that the labeled split used to learn the augmentation weights comes from the same distribution as future test examples, and that the augmentations preserve each image's true class; if either fails, the smaller sets may come without the advertised coverage.

Editorial extensions

If this is right

  • Across three image datasets and three ResNet architectures, learned TTA shrinks average prediction sets by about 10% in-distribution and 14% under ImageNet-C corruptions at nominal coverage levels of 90%, 95%, and 99%.
  • The gains appear with both APS and RAPS scoring, so the method sits on top of the conformal score rather than replacing it.
  • TTA can narrow the gap between base classifiers: ResNet-101 with learned TTA yields smaller sets than ResNet-152 without it at $\alpha = 0.01$.
  • The extra data cost is modest: learning the aggregation on 20% of the validation set already gives most of the benefit, and the approach needs no retraining of the base model.
  • Under four corruption shifts, TTA-Learned keeps coverage at least as high as plain RAPS while producing smaller sets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The rank-promotion mechanism suggests TTA will help most in high-cardinality label spaces and at low $\alpha$; a direct test would vary the number of classes and coverage level while holding base accuracy fixed.
  • Because the aggregation is a single learned linear combination of logits, per-class or per-example weighting of augmentations is a natural extension that might improve on the reported 10-14%.
  • The top-$k$ diagnostic the paper uses could serve as a cheap pre-screening test: estimate the $k$ required for top-$k$ coverage on the validation split before committing to the extra forward passes.
  • The dependence on label-preserving augmentations implies a clear boundary: on domains where the transformation menu contains few label-preserving members, learned weights cannot manufacture invariances that are not there.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes test-time-augmented conformal prediction (TTA-CP): for a fixed augmentation policy A, a set of aggregation weights θ is learned on a labeled split D_TTA, and the classifier's output probabilities in Eq. (4) are replaced by the TTA-aggregated probabilities g(x; f, A, θ). These transformed probabilities are then fed into standard split-conformal classifiers (APS and RAPS). The authors claim that this reduces average prediction-set size by 10%–14% while maintaining nominal coverage, and they support the claim with experiments on ImageNet, iNaturalist, and CUB-Birds using ResNet-50/101/152 and MobileNetV2, two conformal scores, three α levels, and ten calibration/test splits, including four corruption-based distribution shifts. The key theoretical safeguard is the division of the validation set into D_TTA (used to learn θ) and D_cal (used to compute the conformal threshold), which preserves conditional exchangeability.

Significance. If the central claim holds, the paper makes a useful and practical contribution: it reduces conformal set sizes without retraining the base classifier, works with any conformal score, and is computationally cheap. The paper has real strengths: the D_TTA split is a standard and sound way to preserve exchangeability; the experimental evaluation is broad and includes significance testing; the authors honestly report that gains are small or absent on CUB-Birds; and the analysis of why TTA helps (promoting the rank of the true class) is informative. However, two load-bearing issues must be resolved before the coverage guarantee and the headline reductions can be accepted: the manuscript does not disclose which split is used to select the RAPS hyperparameters k_reg and λ, and the supplementary text contains a direct contradiction of the D_TTA safeguard. These issues affect the formal validity of the coverage claim and the interpretation of the empirical reductions.

major comments (2)
  1. [Section 5, Baselines] The text says 'automatically select hyperparameters kreg and λ to minimize set size' but does not state which data are used for this selection. If k_reg and λ are chosen on D_cal, then the score function is not fixed before calibration, the calibration scores are not exchangeable with the test score in the sense required by the split-conformal argument, and the nominal coverage guarantee is void; in that case the reported set-size reductions could be a tuning artifact. The empirical coverage tables (e.g., Table S8) cannot rule this out, because with n on the order of 20,000 a modest tuning bias would be within the reported standard errors. Please disclose the exact split used for hyperparameter selection; if necessary, fix k_reg and λ on a split disjoint from D_cal, and confirm that the baseline RAPS hyperparameters are selected in the same way.
  2. [Supplementary S1.2] Supplementary S1.2 ('Learning aggregation function') states: 'We learn ˆg by minimizing the cross-entropy loss with respect to the true labels on the calibration set.' This directly contradicts Section 4 ('Preserving exchangeability'), which motivates the D_TTA split precisely to avoid using calibration labels when learning the TTA transformation. If θ was in fact fit on D_cal, the coverage guarantee and the claimed 10%–14% reductions are unsupported. This wording must be corrected to specify the disjoint split, or the experiments must be rerun with the correct split.
minor comments (5)
  1. [Table S5 caption] The caption contains an unfinished placeholder reading 'FILL IN THE REST, explain how TTA’s improvement to Top-1 accuracy alone is small...' This must be completed before the manuscript is publishable.
  2. [Section 5, Baselines] Equation numbering is inconsistent: the text says APS is 'described in Eqn. 4', but Eq. (4) defines the TTA-transformed probabilities; the APS score is defined in Eqs. (1)–(3). Please renumber or correct the cross-reference.
  3. [Section 6.1 and Table S8] The claim 'TTA-Learned never decreases the coverage achieved by RAPS alone' is not literally supported by the table; for example, at α=0.10 with the simple augmentation policy on CUB-Birds, RAPS+TTA-Learned reports 0.913±0.011 versus 0.919±0.014 for RAPS. If the intended claim is that there is no statistically significant decrease, please state it that way.
  4. [Tables 1 and 2] The main-text tables should state explicitly that the baseline RAPS uses the full validation set for calibration, while TTA variants use only the 80% remaining after setting aside D_TTA; the trade-off is studied in Figure S3, but the reader should not have to infer it from the supplement.
  5. [Section 5, Evaluation] The text says 'Code to reproduce all experiments will be made publicly available' but no repository or release is provided. Please include a link or state how the code can be obtained for review.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the TTA-transformed scores are fit on a separate split and set-size reductions are held-out measurements, not derived from fitted values.

full rationale

The central claim is an empirical efficiency improvement, not a quantity derived from its own input. Section 4 learns the aggregation weights on a dedicated split D_TTA ('We learn the aggregation weights θ using a portion of the validation set, D(T T A), distinct from calibration set used to identify the conformal threshold'), and the 'Preserving exchangeability' paragraph gives the standard conditional-exchangeability argument: conditional on D_TTA, D_cal and the test point remain exchangeable, so the split-conformal quantile argument applies. Prediction set sizes in Tables 1, 2, and the supplementary tables are measured on held-out test splits across 10 random calibration/test splits, so the 10%-14% reduction is an empirical comparison rather than a fitted parameter renamed as a prediction. The mechanism analysis (top-K accuracy and true-class rank, Section 6.5) is measured independently and is not the same quantity as conformal set size. I flag three manuscript-level concerns that are not circularity. First, Supplementary S1.2 says the aggregation function is learned 'with respect to the true labels on the calibration set', which contradicts Section 4's distinct D_TTA split; if taken literally, this breaks the exchangeability argument and voids the formal coverage guarantee, but that is a validity/correctness risk, not a derivation that reduces to its inputs by construction. Second, Section 5 states RAPS hyperparameters k_reg and λ are 'automatically select[ed] ... to minimize set size' without specifying the data split; if selected on D_cal this could bias coverage, but the paper does not expose the reduction needed to call this circular. Third, the Table S5 caption contains an unfinished placeholder ('FILL IN THE REST...'), an incompleteness unrelated to circularity. None of these issues makes the paper's central derivation self-referential or equivalent to its own inputs.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method introduces no new theoretical entities. Its trained component is the augmentation weight vector theta, fit on a dedicated split, plus the standard RAPS regularization hyperparameters and the split fraction beta. The validity argument relies on exchangeability of the data splits and on the standard result that deterministic transformations preserve exchangeability. No equation in the paper reduces to a fitted parameter; efficiency numbers are measured on held-out test data.

free parameters (3)
  • Aggregation weights theta = Learned vector, one entry per augmentation, via SGD on D_TTA
    This is the method's main trained component, fit by minimizing cross-entropy on a dedicated labeled split (Section 4, Equation 5).
  • Validation split fraction beta = 0.2
    Chosen for all experiments based on Figure S3; the paper reports performance is insensitive to the choice.
  • RAPS hyperparameters k_reg and lambda = Auto-selected to minimize prediction set size
    Selected per experiment on calibration or validation data (Section 5, Baselines); affects both the baseline and the proposed method.
assumptions (3)
  • domain assumption Calibration, D_TTA, and test examples are exchangeable draws from the same data distribution.
    Required for split conformal validity; invoked in Section 4 'Preserving exchangeability'.
  • standard math A deterministic transformation applied equally to all calibration and test scores preserves exchangeability.
    Used to justify coverage after TTA; cited to Kuchibhotla [24] in Section 4.
  • domain assumption The hand-selected augmentation functions are approximately label-preserving for the image domains considered.
    If augmentations systematically destroy label information, the aggregated probabilities would be biased and set-size reductions could come at the cost of validity; the paper argues label preservation is why certain augmentations receive zero weight (Section 6.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Test-time augmentation improves efficiency in conformal prediction." pith.science (2026). https://pith.science/paper/MT4XAVNL

@misc{pith2026250522764,
  author       = {Pith},
  title        = {Pith review of: Test-time augmentation improves efficiency in conformal prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MT4XAVNL}},
  note         = {Machine review of arXiv:2505.22764}
}
read the original abstract

A conformal classifier produces a set of predicted classes and provides a probabilistic guarantee that the set includes the true class. Unfortunately, it is often the case that conformal classifiers produce uninformatively large sets. In this work, we show that test-time augmentation (TTA)--a technique that introduces inductive biases during inference--reduces the size of the sets produced by conformal classifiers. Our approach is flexible, computationally efficient, and effective. It can be combined with any conformal score, requires no model retraining, and reduces prediction set sizes by 10%-14% on average. We conduct an evaluation of the approach spanning three datasets, three models, two established conformal scoring methods, different guarantee strengths, and several distribution shifts to show when and why test-time augmentation is a useful addition to the conformal pipeline.

Figures

Figures reproduced from arXiv: 2505.22764 by the authors.

Figure 1
Figure 1. We illustrate the addition of test-time augmentation to conformal calibration in green (left) and provide a snapshot of the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Robustness to distribution shift. We compare average prediction set size achieved by RAPS (yellow) to average prediction set size achieved when combining RAPS with TTA-Learned (green). Results reflect the distribution of average prediction set size across 10 runs using ImageNet and ResNet50. We evaluate performance on different corruptions (x-axis) and different coverage guarantees (left, middle, right). RAPS+TTA-Le… view at source ↗
Figure 3
Figure 3. (A) Class-conditional prediction set sizes. We plot the distribution of class-conditional prediction set sizes, for ImageNet and ResNet-50 with α = .01. RAPS+TTA-Learned (green) produces a noticeable reduction in class-conditional prediction set sizes. (B, C) Relationship between TTA improvements and original class set sizes and class difficulty. TTA introduces the largest improvements for classes with the largest o… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: (A) Effect of TTA-Learned on optimal Top-K: TTA-Learned significantly lowers the value of k required for Top-k prediction sets to achieve coverage on ImageNet and iNaturalist, but not on CUB-Birds. (B,C) Effect of TTA-Learned on rank of true class: TTA-Learned improves…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 37 canonical work pages

  1. [1]

    Anastasios Angelopoulos, Stephen Bates, Jitendra Malik, and Michael I. Jordan. Uncertainty Sets for Image Classi- fiers using Conformal Prediction, 2022. arXiv:2009.14193 [cs, math, stat]. 2, 4, 11

    arXiv 2022

  2. [2]

    Test-time Data Augmentation for Estimation of Heteroscedastic Aleatoric Uncertainty in Deep Neural Networks

    Murat Sec ¸kin Ayhan and Philipp Berens. Test-time Data Augmentation for Estimation of Heteroscedastic Aleatoric Uncertainty in Deep Neural Networks. 2018. 1, 2

    work page 2018

  3. [3]

    On the Utility of Prediction Sets in Human-AI Teams, 2022

    Varun Babbar, Umang Bhatt, and Adrian Weller. On the Utility of Prediction Sets in Human-AI Teams, 2022. arXiv:2205.01411 [cs]. 1

    arXiv 2022

  4. [4]

    Optimized conformal classification using gradient descent approximation, 2021

    Anthony Bellotti. Optimized conformal classification using gradient descent approximation, 2021. arXiv:2105.11255 [cs]. 2

    arXiv 2021

  5. [5]

    Aggregated Conformal Prediction

    Lars Carlsson, Martin Eklund, and Ulf Norinder. Aggregated Conformal Prediction. In Artificial Intelligence Applications and Innovations, pages 231–240, Berlin, Heidelberg, 2014. Springer. 2

    work page 2014

  6. [6]

    Cyclic test time augmentation with entropy weight method

    Sewhan Chun, Jae Young Lee, and Junmo Kim. Cyclic test time augmentation with entropy weight method. In Proceed- ings of the Thirty-Eighth Conference on Uncertainty in Ar- tificial Intelligence , pages 433–442. PMLR, 2022. ISSN: 2640-3498. 2

    work page 2022

  7. [7]

    Simple post-training robust- ness using test time augmentations and random forest

    Gilad Cohen and Raja Giryes. Simple post-training robust- ness using test time augmentations and random forest. In Proceedings of the IEEE/CVF Winter Conference on Appli- cations of Computer Vision, pages 3996–4006, 2024. 1

    work page 2024

  8. [8]

    ICU Survival Prediction Incorporating Test- Time Augmentation to Improve the Accuracy of Ensemble- Based Models

    Seffi Cohen, Noa Dagan, Nurit Cohen-Inger, Dan Ofer, and Lior Rokach. ICU Survival Prediction Incorporating Test- Time Augmentation to Improve the Accuracy of Ensemble- Based Models. IEEE Access, 9:91584–91592, 2021. Con- ference Name: IEEE Access. 2

    work page 2021

Show all 51 references
  1. [9]

    Lopes, Cristiano Preme- bida, and Urbano J

    Pedro Conde, Tiago Barros, Rui L. Lopes, Cristiano Preme- bida, and Urbano J. Nunes. Approaching Test Time Aug- mentation in the Context of Uncertainty Calibration for Deep Neural Networks, 2023. arXiv:2304.05104 [cs]. 2

    2023 arXiv

  2. [10]

    Conformal prediction sets improve human decision mak- ing

    Jesse C Cresswell, Yi Sui, Bhargava Kumar, and No¨el V ouit- sis. Conformal prediction sets improve human decision mak- ing. arXiv preprint arXiv:2401.13744, 2024. 1

    2024 arXiv

  3. [11]

    Cubuk, Barret Zoph, Dandelion Mane, Vijay Va- sudevan, and Quoc V

    Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Va- sudevan, and Quoc V . Le. AutoAugment: Learning Aug- mentation Policies from Data, 2019. arXiv:1805.09501 [cs, stat]. 11

    2019 arXiv

  4. [12]

    ImageNet: A large-scale hierarchical im- age database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical im- age database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009. ISSN: 1063-

    2009

  5. [13]

    Angelopoulos, Stephen Bates, Michael I

    Tiffany Ding, Anastasios N. Angelopoulos, Stephen Bates, Michael I. Jordan, and Ryan J. Tibshirani. Class- Conditional Conformal Prediction With Many Classes, 2023. arXiv:2306.09335 [cs, stat]. 2

    2023 arXiv

  6. [14]

    Training Uncertainty-Aware Classifiers with Conformalized Deep Learning

    Bat-Sheva Einbinder, Yaniv Romano, Matteo Sesia, and Yanfei Zhou. Training Uncertainty-Aware Classifiers with Conformalized Deep Learning. 15

  7. [15]

    Dynamic Test-Time Augmentation via Differentiable Functions, 2023

    Shohei Enomoto, Monikka Roslianna Busto, and Takeharu Eda. Dynamic Test-Time Augmentation via Differentiable Functions, 2023. arXiv:2212.04681 [cs]. 2

    2023 arXiv

  8. [16]

    Gammerman, V

    A. Gammerman, V . V ovk, and V . Vapnik. Learning by trans- duction. In Proceedings of the Fourteenth conference on Un- certainty in artificial intelligence, pages 148–155, San Fran- cisco, CA, USA, 1998. Morgan Kaufmann Publishers Inc. 2

    1998

  9. [17]

    Deep Residual Learning for Image Recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 770–778, Las Vegas, NV , USA, 2016. IEEE. 4

    2016

  10. [18]

    Brinker, and Florian Buettner

    Achim Hekler, Titus J. Brinker, and Florian Buettner. Test Time Augmentation Meets Post-hoc Calibration: Uncer- tainty Quantification under Real-World Conditions. Pro- ceedings of the AAAI Conference on Artificial Intelligence , 37(12):14856–14864, 2023. Number: 12. 1, 2

    2023

  11. [19]

    Benchmarking neu- ral network robustness to common corruptions and perturba- tions

    Dan Hendrycks and Thomas Dietterich. Benchmarking neu- ral network robustness to common corruptions and perturba- tions. arXiv preprint arXiv:1903.12261, 2019. 6

    1903 arXiv

  12. [20]

    Ensemble Conformalized Quantile Regression for Probabilistic Time Series Forecasting.IEEE Transactions on Neural Networks and Learning Systems , pages 1–12, 2022

    Vilde Jensen, Filippo Maria Bianchi, and Stian Normann An- finsen. Ensemble Conformalized Quantile Regression for Probabilistic Time Series Forecasting.IEEE Transactions on Neural Networks and Learning Systems , pages 1–12, 2022. Conference Name: IEEE Transactions on Neural Ne...

    2022

  13. [21]

    Learning Loss for Test-Time Augmentation

    Ildoo Kim, Younghoon Kim, and Sungwoong Kim. Learning Loss for Test-Time Augmentation. In Advances in Neural Information Processing Systems , pages 4163–4174. Curran Associates, Inc., 2020. 2

    2020

  14. [22]

    Wilds: A benchmark of in-the- wild distribution shifts

    Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubra- mani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the- wild distribution shifts. In International conference on machine learning...

    2021

  15. [23]

    Im- ageNet Classification with Deep Convolutional Neural Net- works

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Im- ageNet Classification with Deep Convolutional Neural Net- works. In Advances in Neural Information Processing Sys- tems. Curran Associates, Inc., 2012. 11

    2012

  16. [24]

    Exchangeability, conformal pre- diction, and rank tests

    Arun Kumar Kuchibhotla. Exchangeability, conformal pre- diction, and rank tests. arXiv preprint arXiv:2005.06095 ,

    2005 arXiv

  17. [25]

    On the Calibration of Aggregated Conformal Predictors

    Henrik Linusson, Ulf Norinder, Henrik Bostr¨om, Ulf Johans- son, and Tuve L ¨ofstr¨om. On the Calibration of Aggregated Conformal Predictors. In Proceedings of the Sixth Work- shop on Conformal and Probabilistic Prediction and Appli- cations, pages 154–173. PMLR, 2017. ISSN: 2...

    2017

  18. [26]

    Linusson, U

    H. Linusson, U. Johansson, and H. Bostr ¨om. Efficient con- formal predictor ensembles. Neurocomputing, 397:266–278,

  19. [27]

    Improved Text Classification via Test-Time Augmenta- tion, 2022

    Helen Lu, Divya Shanmugam, Harini Suresh, and John Gut- tag. Improved Text Classification via Test-Time Augmenta- tion, 2022. arXiv:2206.13607 [cs]. 2

    2022 arXiv

  20. [28]

    Greedy Policy Search: A Simple Baseline for Learnable Test-Time Aug- mentation

    Alexander Lyzhov, Yuliya Molchanova, Arsenii Ashukha, Dmitry Molchanov, and Dmitry Vetrov. Greedy Policy Search: A Simple Baseline for Learnable Test-Time Aug- mentation. In Proceedings of the 36th Conference on Un- certainty in Artificial Intelligence (UAI) , pages 1308–1317....

    2020

  21. [29]

    Rizzo, Valentino Peluso, Andrea Calimera, and Enrico Macii

    Luca Mocerino, Roberto G. Rizzo, Valentino Peluso, Andrea Calimera, and Enrico Macii. Adaptive Test-Time Augmen- tation for Low-Power CPU, 2021. arXiv:2105.06183 [cs, eess]. 2

    2021 arXiv

  22. [30]

    iNaturalist competition, 2021

    Niers, Tom. iNaturalist competition, 2021. original-date: 2021-12-10T10:56:46Z. 4

    2021

  23. [31]

    PyTorch: An Imperative Style, High-Performance Deep Learning Library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Rai- son, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, L...

    2019

  24. [32]

    Perez, Motasem Alfarra, Guillaume Jeanneret, Laura Rueda, Ali Thabet, Bernard Ghanem, and Pablo Arbelaez

    Juan C. Perez, Motasem Alfarra, Guillaume Jeanneret, Laura Rueda, Ali Thabet, Bernard Ghanem, and Pablo Arbelaez. Enhancing Adversarial Robustness via Test-time Transfor- mation Ensembling. In 2021 IEEE/CVF International Con- ference on Computer Vision Workshops (ICCVW), pages...

    2021

  25. [33]

    JAWS: Au- diting Predictive Uncertainty Under Covariate Shift, 2022

    Drew Prinster, Anqi Liu, and Suchi Saria. JAWS: Au- diting Predictive Uncertainty Under Covariate Shift, 2022. arXiv:2207.10716 [cs, stat]. 2

    2022 arXiv

  26. [34]

    Cand `es

    Yaniv Romano, Matteo Sesia, and Emmanuel J. Cand `es. Classification with Valid and Adaptive Coverage, 2020. arXiv:2006.02544 [stat]. 2, 3, 4

    2020 arXiv

  27. [35]

    Transduction with Confi- dence and Credibility

    C Saunders and Royal Holloway. Transduction with Confi- dence and Credibility. 1999. 2

    1999

  28. [36]

    A Tutorial on Conformal Prediction

    Glenn Shafer and Vladimir V ovk. A Tutorial on Conformal Prediction. 2008. 1, 3

    2008

  29. [37]

    Better Aggregation in Test-Time Augmen- tation

    Divya Shanmugam, Davis Blalock, Guha Balakrishnan, and John Guttag. Better Aggregation in Test-Time Augmen- tation. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1194–1203, Montreal, QC, Canada, 2021. IEEE. 1, 2

    2021

  30. [38]

    Learning Optimal Conformal Classi- fiers, 2022

    David Stutz, Krishnamurthy, Dvijotham, Ali Taylan Cemgil, and Arnaud Doucet. Learning Optimal Conformal Classi- fiers, 2022. arXiv:2110.09192 [cs, stat]. 2

    2022 arXiv

  31. [39]

    Intriguing properties of neural networks

    Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013. 1

    2013 arXiv

  32. [40]

    Conformal Prediction Under Covari- ate Shift

    Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal Prediction Under Covari- ate Shift. In Advances in Neural Information Processing Sys- tems. Curran Associates, Inc., 2019. 2

    2019

  33. [41]

    Benchmarking Rep- resentation Learning for Natural World Image Collections,

    Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. Benchmarking Rep- resentation Learning for Natural World Image Collections,

  34. [42]

    Algorithmic Learning in a Random World

    Vladmir V ovk. Algorithmic Learning in a Random World . Springer-Verlag, New York, 2005. 2

    2005

  35. [43]

    Cross-conformal predictors, 2012

    Vladimir V ovk. Cross-conformal predictors, 2012. arXiv:1208.0806 [cs, stat]. 2

    2012 arXiv

  36. [44]

    Cross-conformal predictors

    Vladimir V ovk. Cross-conformal predictors. Annals of Mathematics and Artificial Intelligence, 74(1-2):9–28, 2015. 2

    2015

  37. [45]

    The caltech-ucsd birds-200-2011 dataset

    Catherine Wah, Steve Branson, Peter Welinder, Pietro Per- ona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011. 4

    2011

  38. [46]

    Bonferroni correction

    Eric W Weisstein. Bonferroni correction. https://mathworld. wolfram. com/, 2004. 4

    2004

  39. [47]

    Evaluating the utility of conformal pre- diction sets for ai-advised image labeling

    Dongping Zhang, Angelos Chatzimparmpas, Negar Kamali, and Jessica Hullman. Evaluating the utility of conformal pre- diction sets for ai-advised image labeling. In Proceedings of the CHI Conference on Human Factors in Computing Sys- tems, pages 1–19, 2024. 1

    2024

  40. [48]

    MEMO: Test Time Robustness via Adaptation and Augmentation,

    Marvin Zhang, Sergey Levine, and Chelsea Finn. MEMO: Test Time Robustness via Adaptation and Augmentation,

  41. [51]

    augmentations are unlikely to be label-preserving. In particular, we exclude CutOut (because it is clearly not label-preserving in many domains) and exclude brightness, contrast, saturation, and color for their overlap with color-jitter. We also exclude contrast, because it is...

  42. [2021]

    arXiv:2103.16483 [cs]. 4

    arXiv

  43. [2022]

    2 Supplementary Material S1

    arXiv:2110.09506 [cs]. 2 Supplementary Material S1. Experimental Details S1.1. Augmentations The simple augmentation policy consists of a random crop and a horizontal flip, drawn from a widely used test-time augmentation policy in image classification [23]. The random crop pad...

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.