The minimax rate of estimating second-order calibration error is Õ(1/√n) with a matching Ω(1/√n) lower bound, enabled by analyticity from the sech kernel and yielding the first finite-sample guarantee for second-order Platt scaling.
hub Mixed citations
Bayesian Active Learning for Classification and Preference Learning
Mixed citation behavior. Most common role is background (67%).
45
Pith papers citing it
Background
67% of classified citations
abstract
Information theoretic active learning has been widely studied for probabilistic models. For simple regression an optimal myopic policy is easily tractable. However, for other tasks and with more complex models, such as classification with nonparametric models, the optimal solution is harder to compute. Current approaches make approximations to achieve tractability. We propose an approach that expresses information gain in terms of predictive entropies, and apply this method to the Gaussian Process Classifier (GPC). Our approach makes minimal approximations to the full information theoretic objective. Our experimental performance compares favourably to many popular active learning algorithms, and has equal or lower computational complexity. We compare well to decision theoretic approaches also, which are privy to more information and require much more computational time. Secondly, by developing further a reformulation of binary preference learning to a classification problem, we extend our algorithm to Gaussian Process preference learning.
hub tools
citation-role summary
background 5
method 1
citation-polarity summary
representative citing papers
Proves CLT for stochastic gradient non-reversible Langevin Monte Carlo and sufficient condition for variance reduction via anti-symmetric perturbation relative to reversible baseline.
FairBED quantifies dataset fairness as uninformative about sensitive attributes and uses fairness-aware BED to gather data yielding better fairness-accuracy trade-offs than random or standard BED acquisition.
ICL-derived intrinsic rewards are biased in general MDPs but asymptotically match true learning progress in non-temporal settings, with supporting experiments.
SymQNet uses offline RL to amortize acquisition for adaptive Hamiltonian learning, delivering 47-72x lower decision latency than online Bayesian baselines on Ising models while keeping posterior updates.
ATLAS uses active learning with disentangled RNN ensembles to design experiments that recover RL agent models from bandit behavior 5-10x more efficiently than random or expert baselines in simulations.
OVR is a new one-step lookahead BO algorithm with Monte Carlo approximations that achieves a vanishing Bayesian expected simple regret upper bound.
Introduces an active learning method using LOT and Gaussian Processes to select optimal timepoints for inferring continuous probability paths from sparse distributional snapshots.
Introduces LoRA-Curve parameterization to link independent LoRA optima via low-loss valleys, yielding higher predictive mutual information on reasoning and classification tasks with Qwen2.5 7B.
STAP reduces training data costs for PDE surrogates by selectively acquiring key time steps per trajectory instead of full simulations.
AB-SID-iVAR enables Gaussian process active learning for self-induced Boltzmann distributions by closed-form approximation of the target, with high-probability error vanishing guarantees and empirical gains on PES and drug discovery tasks.
LG-CoTrain, an LLM-guided co-training method, outperforms classical semi-supervised baselines for crisis tweet classification in low-resource settings with 5-25 labeled examples per class.
InfoChess proposes a symmetric adversarial game focused purely on information control and probabilistic king-location inference, with RL agents outperforming heuristic baselines and gameplay dissected via belief entropy, cross-entropy, and predictive scores.
A bias-aware Bayesian model with judge-specific covariates and a top-k membership uncertainty acquisition rule recovers accurate top-k rankings from noisy LLM judges using fewer comparisons than naive aggregation or standard active learning.
POLAR uses pretrained predictive foundation models as fixed belief-state encoders and trains only a lightweight policy head on top for amortised Bayesian experimental design, optimisation, and active learning.
Extends VBI to QR and CR for analytic ELBO and predictive densities in multi-modal regression, outperforming baselines and enabling active learning.
Introduces BAG, an open-source benchmark suite with baselines, datasets, and tasks for assessing Bayesian low-rank adaptation of multi-modal language models on calibration, robustness, and decision-making under uncertainty.
QUEST measures uncertainty via the Lebesgue volume of highest-density regions of a distribution's support, evaluated at robustness parameter alpha, and claims to satisfy UQ axioms while outperforming variance and differential entropy on selective prediction tasks.
Surprise-Guided MergeSort uses a VLM-based composite surprise scorer to prioritize human comparisons inside a MergeSort scheduler, skipping up to 535 pairs per session and raising Kendall's τ by 6-12 points over Active Elo at fixed budget across six benchmarks.
The mutual-information measure of epistemic uncertainty is not reducible by additional data, requiring a split into aleatoric, sample-reducible epistemic, and mechanism-reducible epistemic uncertainty.
ShaplEIG adaptively selects coalitions for Shapley value estimation via expected information gain from a Gaussian process surrogate, with closed-form computation and polynomial complexity, showing improved sample efficiency in low-budget regimes.
Active learning strategies for preference-based MPC objective learning achieve better closed-loop alignment with human preferences using fewer queries than random sampling in numerical tests.
Adaptive Prompt Elicitation (APE) uses an information-theoretic framework to generate visual queries that elicit and compile user intent into better prompts for text-to-image models, showing improved alignment in benchmarks and a user study.
UG-TTT adds epistemic uncertainty measured by adapter disagreement as an exploration bonus in RL for LLMs, raising maximum reward and diversity on scientific discovery benchmarks.
citing papers explorer
-
The Minimax Rate of Second-Order Calibration
The minimax rate of estimating second-order calibration error is Õ(1/√n) with a matching Ω(1/√n) lower bound, enabled by analyticity from the sech kernel and yielding the first finite-sample guarantee for second-order Platt scaling.
-
Variance Reduction for Stochastic Gradient Generalized Non-reversible Langevin Monte Carlo Algorithms
Proves CLT for stochastic gradient non-reversible Langevin Monte Carlo and sufficient condition for variance reduction via anti-symmetric perturbation relative to reversible baseline.
-
FairBED: A Bayesian Experimental Design Approach to Gathering Fairer Data
FairBED quantifies dataset fairness as uninformative about sensitive attributes and uses fairness-aware BED to gather data yielding better fairness-accuracy trade-offs than random or standard BED acquisition.
-
Can In-Context Learning Support Intrinsic Curiosity?
ICL-derived intrinsic rewards are biased in general MDPs but asymptotically match true learning progress in non-temporal settings, with supporting experiments.
-
SymQNet: Amortized Acquisition for Low-Latency Adaptive Hamiltonian Learning
SymQNet uses offline RL to amortize acquisition for adaptive Hamiltonian learning, delivering 47-72x lower decision latency than online Bayesian baselines on Ising models while keeping posterior updates.
-
ATLAS: Active Theory Learning for Automated Science
ATLAS uses active learning with disentangled RNN ensembles to design experiments that recover RL agent models from bandit behavior 5-10x more efficiently than random or expert baselines in simulations.
-
Optimal-Point Variance Reduction For Bayesian Optimization With Regret Guarantee
OVR is a new one-step lookahead BO algorithm with Monte Carlo approximations that achieves a vanishing Bayesian expected simple regret upper bound.
-
Active Timepoint Selection for Learning Measure-Valued Trajectories
Introduces an active learning method using LOT and Gaussian Processes to select optimal timepoints for inferring continuous probability paths from sparse distributional snapshots.
-
On the Construction and Implications of Low-Loss Valleys in LoRA-based Bayesian Inference
Introduces LoRA-Curve parameterization to link independent LoRA optima via low-loss valleys, yielding higher predictive mutual information on reasoning and classification tasks with Qwen2.5 7B.
-
Active Learning with Selective Time-Step Acquisition for PDEs
STAP reduces training data costs for PDE surrogates by selectively acquiring key time steps per trajectory instead of full simulations.
-
Active Learning for Gaussian Process Regression Under Self-Induced Boltzmann Weights
AB-SID-iVAR enables Gaussian process active learning for self-induced Boltzmann distributions by closed-form approximation of the target, with high-probability error vanishing guarantees and empirical gains on PES and drug discovery tasks.
-
LLM-guided Semi-Supervised Approaches for Social Media Crisis Data Classification
LG-CoTrain, an LLM-guided co-training method, outperforms classical semi-supervised baselines for crisis tweet classification in low-resource settings with 5-25 labeled examples per class.
-
InfoChess: A Game of Adversarial Inference and a Laboratory for Quantifiable Information Control
InfoChess proposes a symmetric adversarial game focused purely on information control and probabilistic king-location inference, with RL agents outperforming heuristic baselines and gameplay dissected via belief entropy, cross-entropy, and predictive scores.
-
Ask the Right Comparison:Bias-Aware Bayesian Active Top-$k$ Ranking with LLM Judges
A bias-aware Bayesian model with judge-specific covariates and a top-k membership uncertainty acquisition rule recovers accurate top-k rankings from noisy LLM judges using fewer comparisons than naive aggregation or standard active learning.
-
Efficient Adaptive Data Acquisition via Pretrained Belief Representations
POLAR uses pretrained predictive foundation models as fixed belief-state encoders and trains only a lightweight policy head on top for amortised Bayesian experimental design, optimisation, and active learning.
-
Efficient Analytic Uncertainty Quantification for Multi-Modal Regression
Extends VBI to QR and CR for analytic ELBO and predictive densities in multi-modal regression, outperforming baselines and enabling active learning.
-
Bayesian Adaptation Gym: A Benchmark for the Bayesian Low-Rank Adaptation of Multi-Modal Language Models
Introduces BAG, an open-source benchmark suite with baselines, datasets, and tasks for assessing Bayesian low-rank adaptation of multi-modal language models on calibration, robustness, and decision-making under uncertainty.
-
On the QUEST for Uncertainty Quantification via Highest Density Regions
QUEST measures uncertainty via the Lebesgue volume of highest-density regions of a distribution's support, evaluated at robustness parameter alpha, and claims to satisfy UQ axioms while outperforming variance and differential entropy on selective prediction tasks.
-
Surprise-Guided MergeSort: Budget-Efficient Human-in-the-Loop Ranking via Adaptive Comparison Scheduling
Surprise-Guided MergeSort uses a VLM-based composite surprise scorer to prioritize human comparisons inside a MergeSort scheduler, skipping up to 535 pairs per session and raising Kendall's τ by 6-12 points over Active Elo at fixed budget across six benchmarks.
-
Epistemic Uncertainty Is Not the Reducible Kind
The mutual-information measure of epistemic uncertainty is not reducible by additional data, requiring a split into aleatoric, sample-reducible epistemic, and mechanism-reducible epistemic uncertainty.
-
ShaplEIG: Bayesian Experimental Design for Shapley Value Estimation
ShaplEIG adaptively selects coalitions for Shapley value estimation via expected information gain from a Gaussian process surrogate, with closed-form computation and polynomial complexity, showing improved sample efficiency in low-budget regimes.
-
Active Learning MPC Objective Functions from Preferences
Active learning strategies for preference-based MPC objective learning achieve better closed-loop alignment with human preferences using fewer queries than random sampling in numerical tests.
-
Adaptive Prompt Elicitation for Text-to-Image Generation
Adaptive Prompt Elicitation (APE) uses an information-theoretic framework to generate visual queries that elicit and compile user intent into better prompts for text-to-image models, showing improved alignment in benchmarks and a user study.
-
Epistemic Uncertainty for Test-Time Discovery
UG-TTT adds epistemic uncertainty measured by adapter disagreement as an exploration bonus in RL for LLMs, raising maximum reward and diversity on scientific discovery benchmarks.
-
Decoupled PFNs: Identifiable Epistemic-Aleatoric Decomposition via Structured Synthetic Priors
Decoupled PFNs use controllable synthetic priors to train separate latent-signal and noise heads, making epistemic-aleatoric decomposition identifiable and improving acquisition in noisy settings.
-
Active Learning for Communication Structure Optimization in LLM-Based Multi-Agent Systems
An ensemble-based information-theoretic active learning method using ensemble Kalman inversion selects valuable tasks to optimize communication structures in LLM multi-agent systems more reliably than random sampling under limited training budgets.
-
ProEval: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluation
Bayesian deep learning method rankings are unreliable under data scarcity, reversing across datasets and sample sizes, and a hierarchical Bayesian framework with predictive detectability curves is needed to assess evaluation sufficiency.
-
Vibrotactile Preference Learning: Uncertainty-Aware Preference Learning for Personalized Vibration Feedback
VPL learns individualized vibrotactile preferences efficiently via uncertainty-aware Gaussian process models and active query selection in a 13-participant user study on an Xbox controller.
-
Boundary-Centric Active Learning for Temporal Action Segmentation
B-ACT improves label efficiency in temporal action segmentation by selecting only boundary frames for annotation via a two-stage uncertainty-driven process that fuses neighborhood uncertainty, class ambiguity, and temporal dynamics.
-
Agentic Discovery with Active Hypothesis Exploration for Visual Recognition
HypoExplore uses LLMs for hypothesis-driven evolutionary search with a Trajectory Tree and Hypothesis Memory Bank to discover lightweight vision architectures, reaching 94.11% accuracy on CIFAR-10 from an 18.91% baseline and generalizing to other datasets including state-of-the-art on MedMNIST.
-
Re-mixing Embeddings for Patient Augmentation in Data Scarce Multiple Instance Learning
A GMM-based embedding remix technique generates augmented patients for data-scarce medical MIL, improving performance in missing-class and low-data regimes.
-
Active Continual Learning with Metaplastic Binary Bayesian Neural Networks
BiMU derives a non-saturating posterior for binary Bayesian NNs via bounded-memory variational inference, enabling buffer-free active continual learning with up to 32x label savings on long task sequences and imbalanced data.
-
Soft Specialists: $\alpha$-R\'enyi Ensembles for Uncertainty-Aware LLM Post-Training
An α-Rényi variational ensemble method learns distributions over LoRA adapter parameters for uncertainty-aware LLM post-training, balancing individual model plausibility with complementary specialization.
-
ALIEN: Aligned Entropy Head for Improving Uncertainty Estimation of LLMs
ALIEN trains a lightweight uncertainty head initialized to model entropy and refined via supervised regularization to improve detection of incorrect predictions and calibration on classification and NER tasks.
-
Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measure
Negative log-likelihood of the greedy-decoded most likely sequence (G-NLL) is a principled single-sequence uncertainty measure for LLMs that achieves state-of-the-art results.
-
Test-Time Alignment via Hypothesis Reweighting
HyRe personalizes reward models at test time by reweighting an ensemble of heads trained on aggregate preferences, using few target examples to outperform uniform averaging and prior methods on RewardBench and 32 tasks.
-
RADS: Reinforcement Learning-Based Sample Selection Improves Transfer Learning in Low-resource and Imbalanced Clinical Settings
RADS applies reinforcement learning to pick informative samples for transfer learning, improving performance over uncertainty and diversity sampling in low-resource imbalanced clinical settings.
-
When Active Learning Falls Short: An Empirical Study on Chemical Reaction Extraction
Active learning for chemical reaction extraction frequently produces non-monotonic learning curves and fails to deliver stable gains over random sampling because of strong pretraining, structured CRF decoding, and label sparsity.
-
Testing the Assumptions of Active Learning for Translation Tasks with Few Samples
Informativeness and diversity of samples selected by active learning show no correlation with test performance on translation tasks using few samples; ordering and pre-training effects dominate instead.
-
Active Learning for Optimal Experimental Design in Machine Learning-Based Building Energy System Identification
Active learning for optimal experimental design in ML-based building energy system identification yields up to 54% lower RMSE than passive random sampling on the BOPTEST simulator across neural network and Gaussian process models.
-
Active Learning for Manifold Gaussian Process Regression
A joint optimization of neural manifold learning and active-learning-guided Gaussian process regression in latent space outperforms random sampling on synthetic data for complex functions.
-
Feature Weighting Improves Pool-Based Sequential Active Learning for Regression
Feature weighting derived from ridge regression coefficients improves sample selection in pool-based sequential active learning for both single-task and multi-task regression.
-
A tutorial on learning from preferences and choices with Gaussian Processes
Tutorial on a GP-based framework for preference and choice learning that unifies random utility models, limits of discernment, and multi-utility scenarios via customized likelihoods for object and label preferences.
-
Active Learning Solution on Distributed Edge Computing
A hybrid approach applies active learning at edge devices and federated learning at fog nodes to reduce training data volume and communication cost for image classification in distributed edge-fog setups.
- Do We Still Need Humans in the Loop? Comparing Human and LLM Annotation in Active Learning for Hostility Detection