Awesome Uncertainty In Deep learning | Learning Resources & Collection

# Awesome Uncertainty in Deep learning

This repository contains a collection of surveys, datasets, papers, and codes, for predictive uncertainty estimation in deep learning models.

A Collection of AWESOME papers, codes, books, and blogs about Uncertainty and Deep learning. Feel free to star and fork.

## Table of Contents

- Awesome Uncertainty in Deep learning
- Papers
- Surveys
- Theory
- Bayesian-Methods
- Ensemble-Methods
- Sampling/Dropout-based-Methods
- Auxiliary-Methods/Learning-loss-distributions
- Data-augmentation/Generation-based-methods
- Dirichlet-networks/Evidential-deep-learning
- Deterministic-Uncertainty-Methods
- Quantile-Regression/Predicted-Intervals
- Calibration
- Applications
- Classification and Semantic-Segmentation
- Regression
- Anomaly-detection and Out-of-Distribution-Detection
- Object detection
- Domain adaptation
- Natural Language Processing
- Datasets and Benchmarks
- Libraries
- Lectures and tutorials
- Books

# Papers

## Surveys

**Conference**

- A Comparison of Uncertainty Estimation Approaches in Deep Learning Components for Autonomous Vehicle Applications [AISafety Workshop 2020]

**Journal**

- Ensemble deep learning: A review [Engineering Applications of AI 2021]
- Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods [Machine Learning 2021]
- Predictive inference with the jackknife+ [The Annals of Statistics 2021]
- A review of uncertainty quantification in deep learning: Techniques, applications and challenges [Information Fusion 2021]
- A Survey on Uncertainty Estimation in Deep Learning Classification Systems from a Bayesian Perspective [ACM 2021]
- Uncertainty in big data analytics: survey, opportunities, and challenges [Journal of Big Data 2019]

**Arxiv**

- A Survey on Uncertainty Reasoning and Quantification for Decision Making: Belief Theory Meets Deep Learning [arXiv2022]
- A survey of uncertainty in deep neural networks [arXiv2021] - [GitHub]
- A Survey on Evidential Deep Learning For Single-Pass Uncertainty Estimation [arXiv2021]

## Theory

**Conference**

- Probabilistic Contrastive Learning Recovers the Correct Aleatoric Uncertainty of Ambiguous Inputs [ICML2023] - [PyTorch]
- On Second-Order Scoring Rules for Epistemic Uncertainty Quantification [ICML2023]
- Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning [ICLR2023]
- Unmasking the Lottery Ticket Hypothesis: What’s Encoded in a Winning Ticket’s Mask? [ICLR2023]
- Top-label calibration and multiclass-to-binary reductions [ICLR2022]
- Bayesian Model Selection, the Marginal Likelihood, and Generalization [ICML2022]
- Neural Variational Gradient Descent [AABI2022]
- Repulsive Deep Ensembles are Bayesian [NeurIPS2021] - [PyTorch]
- Bayesian Optimization with High-Dimensional Outputs [NeurIPS2021]
- Residual Pathway Priors for Soft Equivariance Constraints [NeurIPS2021]
- Dangers of Bayesian Model Averaging under Covariate Shift [NeurIPS2021] - [TensorFlow]
- With malice towards none: Assessing uncertainty via equalized coverage [AIES 2021]
- A Mathematical Analysis of Learning Loss for Active Learning in Regression [CVPR Workshop2021]
- Uncertainty in Gradient Boosting via Ensembles [ICLR2021] - [PyTorch]
- Deep Convolutional Networks as shallow Gaussian Processes [ICLR2019]
- On the accuracy of influence functions for measuring group effects [NeurIPS2018]
- To Trust Or Not To Trust A Classifier [NeurIPS2018] - [Python]
- Understanding Measures of Uncertainty for Adversarial Example Detection [UAI2018]

**Journal**

- Testing for Outliers with Conformal p-values [Ann. Statist. 2023] [Python]
- Multivariate Uncertainty in Deep Learning [TNNLS2021]
- A General Framework for Uncertainty Estimation in Deep Learning [RAL2020]
- Adaptive nonparametric confidence sets [Ann. Statist. 2006]

**Arxiv**

- Ensembles for Uncertainty Estimation: Benefits of Prior Functions and Bootstrapping [arXiv2022]
- Efficient Gaussian Neural Processes for Regression [arXiv2021]
- Dense Uncertainty Estimation [arXiv2021] - [PyTorch]
- A higher-order swiss army infinitesimal jackknife [arXiv2019]

## Bayesian-Methods

**Conference**

- Gradient-based Uncertainty Attribution for Explainable Bayesian Deep Learning [CVPR2023]
- Robustness to corruption in pre-trained Bayesian neural networks [ICLR2023]
- Uncertainty Estimation for Multi-view Data: The Power of Seeing the Whole Picture [NeurIPS2022]
- Activation-level uncertainty in deep neural networks [ICLR2021]
- On the Effects of Quantisation on Model Uncertainty in Bayesian Neural Networks [UAI2021]
- Learnable uncertainty under Laplace approximations [UAI2021]
- Bayesian Deep Learning and a Probabilistic Perspective of Generalization [NeurIPS2020]
- On Batch Normalisation for Approximate Bayesian Inference [AABI2021]
- How Good is the Bayes Posterior in Deep Neural Networks Really? [ICML2020]
- Bayesian Uncertainty Estimation for Batch Normalized Deep Networks [ICML2020]
- Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors [ICML2020] - [TensorFlow]
- Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks [ICML2020]
- TRADI: Tracking deep neural network weight distributions for uncertainty estimation [ECCV2020] - [PyTorch]
- A Simple Baseline for Bayesian Uncertainty in Deep Learning [NeurIPS2019] - [PyTorch]
- Lightweight Probabilistic Deep Networks [CVPR2018] - [PyTorch]
- A Scalable Laplace Approximation for Neural Networks [ICLR2018] - [Theano]
- Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning [ICML2018]
- Weight Uncertainty in Neural Networks [ICML2015]

**Journal**

- Bayesian modeling of uncertainty in low-level vision [IJCV1990]

**Arxiv**

- Density Uncertainty Layers for Reliable Uncertainty Estimation [arXiv2023]
- Encoding the latent posterior of Bayesian Neural Networks for uncertainty quantification [arXiv2020] - [PyTorch]
- Bayesian Neural Networks with Soft Evidence [arXiv2020] - [PyTorch]
- Bayesian neural network via stochastic gradient descent [arXiv2020]

## Ensemble-Methods

**Conference**

- Weighted Ensemble Self-Supervised Learning [ICLR2023]
- Agree to Disagree: Diversity through Disagreement for Better Transferability [ICLR2023] - [PyTorch]
- Packed-Ensembles for Efficient Uncertainty Estimation [ICLR2023] - [PyTorch]
- Normalizing Flow Ensembles for Rich Aleatoric and Epistemic Uncertainty Modeling [AAAI2023]
- Deep Ensembles Work, But Are They Necessary? [NeurIPS2022]
- FiLM-Ensemble: Probabilistic Deep Learning via Feature-wise Linear Modulation [NeurIPS2022]
- Prune and Tune Ensembles: Low-Cost Ensemble Learning With Sparse Independent Subnetworks [AAAI2022]
- Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity [ICLR2022] - [PyTorch]
- Robustness via Cross-Domain Ensembles [ICCV2021] - [PyTorch]
- Masksembles for Uncertainty Estimation [CVPR2021] - [PyTorch/TensorFlow]
- Uncertainty Quantification and Deep Ensembles [NeurIPS2021]
- Uncertainty in Gradient Boosting via Ensembles [ICLR2021] - [PyTorch]
- Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning [ICLR2020] - [PyTorch]
- Maximizing Overall Diversity for Improved Uncertainty Estimates in Deep Ensembles [AAAI2020]
- Hyperparameter Ensembles for Robustness and Uncertainty Quantification [NeurIPS2020]
- Bayesian Deep Ensembles via the Neural Tangent Kernel [NeurIPS2020]
- BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning [ICLR2020] - [TensorFlow] - [PyTorch]
- Uncertainty in Neural Networks: Approximately Bayesian Ensembling [AISTATS 2020]
- Accurate Uncertainty Estimation and Decomposition in Ensemble Learning [NeurIPS2019]
- Diversity with Cooperation: Ensemble Methods for Few-Shot Classification [ICCV2019]
- High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach [ICML2018] - [TensorFlow]
- Simple and scalable predictive uncertainty estimation using deep ensembles [NeurIPS2017]

**Journal**

- One Versus all for deep Neural Network for uncertaInty (OVNNI) quantification [IEEE Access2021]

**Arxiv**

- On the Usefulness of Deep Ensemble Diversity for Out-of-Distribution Detection [arXiv2022]
- Deep Ensemble as a Gaussian Process Approximate Posterior [arXiv2022]
- Sequential Bayesian Neural Subnetwork Ensembles [arXiv2022]
- Confident Neural Network Regression with Bootstrapped Deep Ensembles [arXiv2022] - [TensorFlow]
- Dense Uncertainty Estimation via an Ensemble-based Conditional Latent Variable Model [arXiv2021]
- Deep Ensembles: A Loss Landscape Perspective [arXiv2019]

## Sampling/Dropout-based-Methods

**Conference**

- Efficient Bayesian Uncertainty Estimation for nnU-Net [MICCAI2022]
- Training-Free Uncertainty Estimation for Dense Regression: Sensitivity as a Surrogate [AAAI2022]
- Dropout Sampling for Robust Object Detection in Open-Set Conditions [ICRA2018]
- Test-time data augmentation for estimation of heteroscedastic aleatoric uncertainty in deep neural networks [MIDL2018]
- Concrete Dropout [NeurIPS2017]
- Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning [ICML2016]

**Journal**

- A General Framework for Uncertainty Estimation in Deep Learning [Robotics and Automation Letters2020]

**Arxiv**

- SoftDropConnect (SDC) – Effective and Efficient Quantification of the Network Uncertainty in Deep MR Image Analysis [arXiv2022]
- Wasserstein Dropout [arXiv2021] - [PyTorch]

## Auxiliary-Methods/Learning-loss-distributions

**Conference**

- Post-hoc Uncertainty Learning using a Dirichlet Meta-Model [AAAI2023] - [PyTorch]
- Improving the reliability for confidence estimation [ECCV2022]
- Gradient-based Uncertainty for Monocular Depth Estimation [ECCV2022] - [PyTorch]
- BayesCap: Bayesian Identity Cap for Calibrated Uncertainty in Frozen Neural Networks [ECCV2022] - [PyTorch]
- Detecting Misclassification Errors in Neural Networks with a Gaussian Process Model [AAAI2022]
- Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisation [NeurIPS2022]
- Learning Structured Gaussians to Approximate Deep Ensembles [CVPR2022]
- Learning Uncertainty For Safety-Oriented Semantic Segmentation In Autonomous Driving [ICIP2022]
- SLURP: Side Learning Uncertainty for Regression Problems [BMVC2021] - [PyTorch]
- Learning to Predict Error for MRI Reconstruction [MICCAI2021]
- Triggering Failures: Out-Of-Distribution detection by learning from local adversarial attacks in Semantic Segmentation [ICCV2021] - [PyTorch]
- A Mathematical Analysis of Learning Loss for Active Learning in Regression [CVPR Workshop2021]
- Real-time uncertainty estimation in computer vision via uncertainty-aware distribution distillation [WACV2021]
- Quantifying Point-Prediction Uncertainty in Neural Networks via Residual Estimation with an I/O Kernel [ICLR2020] - [TensorFlow]
- Gradients as a Measure of Uncertainty in Neural Networks [ICIP2020]
- Learning Loss for Test-Time Augmentation [NeurIPS2020]
- On the uncertainty of self-supervised monocular depth estimation [CVPR2020] - [PyTorch]
- Addressing failure prediction by learning model confidence [NeurIPS2019] - [PyTorch]
- Learning loss for active learning [CVPR2019] - [PyTorch] (unofficial codes)
- Structured Uncertainty Prediction Networks [CVPR2018] - [TensorFlow]
- Uncertainty estimates and multi-hypotheses networks for optical flow [ECCV2018] - [TensorFlow]
- Classification uncertainty of deep neural networks based on gradient information [IAPR Workshop2018]
- What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? [NeurIPS2017]
- Estimating the Mean and Variance of the Target Probability Distribution [(ICNN94)]

**Journal**

- Confidence Estimation via Auxiliary Models [TPAMI2021]

**Arxiv**

- Instance-Aware Observer Network for Out-of-Distribution Object Segmentation [arXiv2022]
- DEUP: Direct Epistemic Uncertainty Prediction [arXiv2020]
- Learning Confidence for Out-of-Distribution Detection in Neural Networks [arXiv2018]

## Data-augmentation/Generation-based-methods

**Conference**

- On the Pitfall of Mixup for Uncertainty Calibration [CVPR2023]
- Diverse, Global and Amortised Counterfactual Explanations for Uncertainty Estimates [AAAI2022]
- PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures [CVPR2022]
- Towards efficient feature sharing in MIMO architectures [CVPR Workshop2022]
- Robust Semantic Segmentation with Superpixel-Mix [BMVC2021] - [PyTorch]
- MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks [ICCV2021] - [PyTorch]
- Training independent subnetworks for robust prediction [ICLR2021]
- Uncertainty-aware GAN with Adaptive Loss for Robust MRI Image Enhancement [ICCV Workshop2021]
- Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning [ICML2020]
- Uncertainty-Aware Deep Classifiers using Generative Models [AAAI2020]
- Synthesize then Compare: Detecting Failures and Anomalies for Semantic Segmentation [ECCV2020] - [PyTorch]
- Detecting the Unexpected via Image Resynthesis [ICCV2019] - [PyTorch]
- On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks [NeurIPS2019]
- Deep Anomaly Detection with Outlier Exposure [ICLR2019]

**Arxiv**

- ZigZag: Universal Sampling-free Uncertainty Estimation Through Two-Step Inference [arXiv2022]
- Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness [arXiv2021]
- Quantifying uncertainty with GAN-based priors [arXiv2019]

## Dirichlet-networks/Evidential-deep-learning

**Conference**

- Uncertainty Estimation by Fisher Information-based Evidential Deep Learning [ICML2023]
- Exploring and Exploiting Uncertainty for Incomplete Multi-View Classification [CVPR2023]
- Fast Predictive Uncertainty for Classification with Bayesian Deep Networks [UAI2022] - [PyTorch]
- An Evidential Neural Network Model for Regression Based on Random Fuzzy Numbers [BELIEF2022]
- Natural Posterior Network: Deep Bayesian Uncertainty for Exponential Family Distributions [ICLR2022] - [PyTorch]
- Improving Evidential Deep Learning via Multi-task Learning [AAAI2022]
- Trustworthy multimodal regression with mixture of normal-inverse gamma distributions [NeurIPS2021]
- Misclassification Risk and Uncertainty Quantification in Deep Classifiers [WACV2021]
- Evaluating robustness of predictive uncertainty estimation: Are Dirichlet-based models reliable? [ICML2021]
- Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts [NeurIPS2020] - [PyTorch]
- Being Bayesian about Categorical Probability [ICML2020]
- Ensemble Distribution Distillation [ICLR2020]
- Conservative Uncertainty Estimation By Fitting Prior Networks [ICLR2020]
- Noise Contrastive Priors for Functional Uncertainty [UAI2020]
- Deep Evidential Regression [NeurIPS2020] - [TensorFlow]
- Towards Maximizing the Representation Gap between In-Domain & Out-of-Distribution Examples [NeurIPS Workshop2020]
- Uncertainty on Asynchronous Time Event Prediction [NeurIPS2019] - [TensorFlow]
- Reverse KL-Divergence Training of Prior Networks: Improved Uncertainty and Adversarial Robustness [NeurIPS2019]
- Quantifying Classification Uncertainty using Regularized Evidential Neural Networks [AAAI FSS2019]
- Evidential Deep Learning to Quantify Classification Uncertainty [NeurIPS2018] - [PyTorch]
- Predictive uncertainty estimation via prior networks [NeurIPS2018]

**Journal**

- Region-Based Evidential Deep Learning to Quantify Uncertainty and Improve Robustness of Brain Tumor Segmentation [NCA2022]
- An evidential classifier based on Dempster-Shafer theory and deep learning [Neurocomputing2021] - [TensorFlow]
- Evidential fully convolutional network for semantic segmentation [AppliedIntelligence2021] - [TensorFlow]
- Information Aware max-norm Dirichlet networks for predictive uncertainty estimation [NeuralNetworks2021]
- A neural network classifier based on Dempster-Shafer theory [IEEETransSMC2000]

**Arxiv**

- Uncertainty Estimation by Fisher Information-based Evidential Deep Learning [arXiv2023]
- The Unreasonable Effectiveness of Deep Evidential Regression [arXiv2022]
- Effective Uncertainty Estimation with Evidential Models for Open-World Recognition [arXiv2022]
- Multivariate Deep Evidential Regression [arXiv2022]
- A Survey on Evidential Deep Learning For Single-Pass Uncertainty Estimation [arXiv2021]
- Regression Prior Networks [arXiv2020]
- A Variational Dirichlet Framework for Out-of-Distribution Detection [arXiv2019]
- Uncertainty estimation in deep learning with application to spoken language assessment[PhDThesis2019]
- Inhibited softmax for uncertainty estimation in neural networks [arXiv2018].
- Quantifying Intrinsic Uncertainty in Classification via Deep Dirichlet Mixture Networks [arXiv2018]

## Deterministic-Uncertainty-Methods

**Conference**

- Deep Deterministic Uncertainty: A Simple Baseline [CVPR2023] - [PyTorch]
- Training, Architecture, and Prior for Deterministic Uncertainty Methods [ICLR Workshop2023] - [PyTorch]
- Latent Discriminant deterministic Uncertainty [ECCV2022] - [PyTorch]
- Improving Deterministic Uncertainty Estimation in Deep Learning for Classification and Regression [CoRR2021]
- Training normalizing flows with the information bottleneck for competitive generative classification [NeurIPS2020]
- Simple and principled uncertainty estimation with deterministic deep learning via distance awareness [NeurIPS2020]
- Uncertainty Estimation Using a Single Deep Deterministic Neural Network [ICML2020] - [PyTorch]
- Single-Model Uncertainties for Deep Learning [NeurIPS2019] - [PyTorch]
- Sampling-Free Epistemic Uncertainty Estimation Using Approximated Variance Propagation [ICCV2019] - [PyTorch]

**Journal**

- Density estimation in representation space [EDSMLS2020]

**Arxiv**

- On the Practicality of Deterministic Epistemic Uncertainty [arXiv2021]
- The Hidden Uncertainty in a Neural Network’s Activations [arXiv2020]
- A simple framework for uncertainty in contrastive learning [arXiv2020]
- Distance-based Confidence Score for Neural Network Classifiers [arXiv2017]

## Quantile-Regression/Predicted-Intervals

**Conference**

- Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imaging [ICML2022] - [PyTorch]
- Prediction Intervals: Split Normal Mixture from Quality-Driven Deep Ensembles [UAI2020] - [PyTorch]
- Classification with Valid and Adaptive Coverage [NeurIPS2020]
- Conformal Prediction Under Covariate Shift [NeurIPS2019]
- Conformalized Quantile Regression [NeurIPS2019]
- Single-Model Uncertainties for Deep Learning [NeurIPS2019] - [PyTorch]
- High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach [ICML2018] - [TensorFlow]

**Journal**

- Scalable Uncertainty Quantification for Deep Operator Networks using Randomized Priors [CMAME2022]
- Exploring uncertainty in regression neural networks for construction of prediction intervals [Neurocomputing2022]

**Arxiv**

- Interval Neural Networks: Uncertainty Scores [arXiv2020]
- Tight Prediction Intervals Using Expanded Interval Minimization [arXiv2018]

## Calibration

**Conference**

- Beyond calibration: estimating the grouping loss of modern neural networks [ICLR2023]
- Rethinking Confidence Calibration for Failure Prediction [ECCV2022] - [PyTorch]
- The Devil is in the Margin: Margin-based Label Smoothing for Network Calibration [CVPR2022] - [PyTorch]
- Calibrating Deep Neural Networks by Pairwise Constraints [CVPR2022]
- Top-label calibration and multiclass-to-binary reductions [ICLR2022]
- Meta-Calibration: Learning of Model Calibration Using Differentiable Expected Calibration Error [ICML Workshop2021] - [PyTorch]
- Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification [NeurIPS2021]
- Diagnostic Uncertainty Calibration: Towards Reliable Machine Predictions in Medical Domain [AIStats2021]
- From label smoothing to label relaxation [AAAI2021]
- Calibrating Deep Neural Networks using Focal Loss [NeurIPS2020] - [PyTorch]
- Stationary activations for uncertainty calibration in deep learning [NeurIPS2020]
- Confidence-Aware Learning for Deep Neural Networks [ICML2020] - [PyTorch]
- Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning [ICML2020]
- Regularization via structural label smoothing [ICML2020]
- Well-Calibrated Regression Uncertainty in Medical Imaging with Deep Learning [MIDL2020] - [PyTorch]
- Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision [CVPR Workshop2020] - [PyTorch]
- Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration [NeurIPS2019] - [GitHub]
- When does label smoothing help? [NeurIPS2019]
- Verified Uncertainty Calibration [NeurIPS2019]
- Generalized zero-shot learning with deep calibration network [NeurIPS2018]
- Measuring Calibration in Deep Learning [CVPR Workshop2019]
- Accurate Uncertainties for Deep Learning Using Calibrated Regression [ICML2018]
- On calibration of modern neural networks [ICML2017]
- On Fairness and Calibration [NeurIPS2017]
- Obtaining Well Calibrated Probabilities Using Bayesian Binning [AAAI2015]

**Journal**

- Evaluating and Calibrating Uncertainty Prediction in Regression Tasks [Sensors2022]
- Calibrated Prediction Intervals for Neural Network Regressors [IEEE Access 2018] - [Python]

**Arxiv**

- Towards Understanding Label Smoothing [arXiv2020]
- An Investigation of how Label Smoothing Affects Generalization [arXiv2020]

## Applications

### Classification and Semantic-Segmentation

**Conference**

- CRISP - Reliable Uncertainty Estimation for Medical Image Segmentation [MICCAI2022]
- TBraTS: Trusted Brain Tumor Segmentation [MICCAI2022] - [PyTorch]
- Anytime Dense Prediction with Confidence Adaptivity [ICLR2022] - [PyTorch]
- Robust Semantic Segmentation with Superpixel-Mix [BMVC2021] - [PyTorch]
- Classification with Valid and Adaptive Coverage [NeurIPS2020]
- DEAL: Difficulty-aware Active Learning for Semantic Segmentation [ACCV2020]
- Human Uncertainty Makes Classification More Robust [ICCV2019]
- Classification uncertainty of deep neural networks based on gradient information [IAPR Workshop2018]
- Guided Curriculum Model Adaptation and Uncertainty-Aware Evaluation for Semantic Nighttime Image Segmentation [ICCV2019]
- Uncertainty-aware self-ensembling model for semi-supervised 3D left atrium segmentation [MICCAI2019] - [PyTorch]
- A Probabilistic U-Net for Segmentation of Ambiguous Images [NeurIPS2018] - [PyTorch]
- Evidential Deep Learning to Quantify Classification Uncertainty [NeurIPS2018] - [PyTorch]
- Lightweight Probabilistic Deep Networks [CVPR2018] - [PyTorch]
- To Trust Or Not To Trust A Classifier [NeurIPS2018]
- Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding [BMVC2017]

**Journal**

- Explainable machine learning in image classification models: An uncertainty quantification perspective." [KnowledgeBased2022]
- Region-Based Evidential Deep Learning to Quantify Uncertainty and Improve Robustness of Brain Tumor Segmentation [NCA2022]

**Arxiv**

- Deep Deterministic Uncertainty for Semantic Segmentation [arXiv2021]
- Evaluating Bayesian Deep Learning Methods for Semantic Segmentation [arXiv2018]

### Regression

**Conference**

- Learning the Distribution of Errors in Stereo Matching for Joint Disparity and Uncertainty Estimation [CVPR2023] - [PyTorch]
- Variational Depth Networks: Uncertainty-Aware Monocular Self-supervised Depth Estimation [ECCV Workshop2022]
- Uncertainty Quantification in Depth Estimation via Constrained Ordinal Regression [ECCV2022]
- On Monocular Depth Estimation and Uncertainty Quantification using Classification Approaches for Regression [ICIP2022]
- Anytime Dense Prediction with Confidence Adaptivity [ICLR2022] - [PyTorch]
- Learning Structured Gaussians to Approximate Deep Ensembles [CVPR2022]
- Training-Free Uncertainty Estimation for Dense Regression: Sensitivity as a Surrogate [AAAI2022]
- Robustness via Cross-Domain Ensembles [ICCV2021] - [PyTorch]
- SLURP: Side Learning Uncertainty for Regression Problems [BMVC2021] - [PyTorch]
- Learning to Predict Error for MRI Reconstruction [MICCAI2021]
- Deep Evidential Regression [NeurIPS2020] - [TensorFlow]
- Quantifying Point-Prediction Uncertainty in Neural Networks via Residual Estimation with an I/O Kernel [ICLR2020] - [TensorFlow]
- Well-Calibrated Regression Uncertainty in Medical Imaging with Deep Learning [MIDL2020] - [PyTorch]
- On the uncertainty of self-supervised monocular depth estimation [CVPR2020] - [PyTorch]
- Fast Uncertainty Estimation for Deep Learning Based Optical Flow [IROS2020]
- Inferring Distributions Over Depth from a Single Image [IROS2019] - [TensorFlow]
- Multi-Task Learning based on Separable Formulation of Depth Estimation and its Uncertainty [CVPR Workshop2019]
- Lightweight Probabilistic Deep Networks [CVPR2018] - [PyTorch]
- Uncertainty estimates and multi-hypotheses networks for optical flow [ECCV2018] - [TensorFlow]
- Accurate Uncertainties for Deep Learning Using Calibrated Regression [ICML2018]
- Structured Uncertainty Prediction Networks [CVPR2018] - [TensorFlow]

**Journal**

- Evaluating and Calibrating Uncertainty Prediction in Regression Tasks [Sensors2022]
- Exploring uncertainty in regression neural networks for construction of prediction intervals [Neurocomputing2022]
- Wasserstein Dropout [Machine Learning 2022] - [PyTorch]
- Deep Distribution Regression [Computational Statistics & Data Analysis2021]
- Calibrated Prediction Intervals for Neural Network Regressors [IEEE Access 2018] - [Python]
- Learning a Confidence Measure for Optical Flow [TPAMI2013]

**Arxiv**

- How Reliable is Your Regression Model’s Uncertainty Under Real-World Distribution Shifts? [arXiv2023] - [PyTorch]
- UncertaINR: Uncertainty Quantification of End-to-End Implicit Neural Representations for Computed Tomographaphy [arXiv2022]
- Efficient Gaussian Neural Processes for Regression [arXiv2021]

### Anomaly-detection and Out-of-Distribution-Detection

**Conference**

- Uncertainty-Aware Optimal Transport for Semantically Coherent Out-of-Distribution Detection [CVPR2023] - [PyTorch]
- Modeling the Distributional Uncertainty for Salient Object Detection Models [CVPR2023] - [PyTorch]
- SQUID: Deep Feature In-Painting for Unsupervised Anomaly Detection [CVPR2023] - [PyTorch]
- How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection? [ICLR2023] - [PyTorch]
- Can CNNs Be More Robust Than Transformers? [ICLR2023]
- Modeling the Data-Generating Process is Necessary for Out-of-Distribution Generalization [ICLR2023]
- A framework for benchmarking class-out-of-distribution detection and its application to ImageNet [ICLR2023]
- Augmenting Softmax Information for Selective Classification with Out-of-Distribution Data [ACCV2022]
- Detecting Misclassification Errors in Neural Networks with a Gaussian Process Model [AAAI2022]
- VOS: Learning What You Don’t Know by Virtual Outlier Synthesis [ICLR2022] - [PyTorch]
- Anomaly Detection via Reverse Distillation from One-Class Embedding [CVPR2022]
- Towards Total Recall in Industrial Anomaly Detection [CVPR2022] - [PyTorch]
- Fully Convolutional Cross-Scale-Flows for Image-based Defect Detection [WACV2022] - [PyTorch]
- Out-of-Distribution Detection Using Union of 1-Dimensional Subspaces [CVPR2021] - [PyTorch]
- On the Importance of Gradients for Detecting Distributional Shifts in the Wild [NeurIPS2021]
- Exploring the Limits of Out-of-Distribution Detection [NeurIPS2021]
- NAS-OoD: Neural Architecture Search for Out-of-Distribution Generalization [ICCV2021]
- NADS: Neural Architecture Distribution Search for Uncertainty Awareness [ICML2020]
- Energy-based Out-of-distribution Detection [NeurIPS2020]
- PaDiM: a Patch Distribution Modeling Framework for Anomaly Detection and Localization [ICPR2020] - [PyTorch]
- Detecting out-of-distribution image without learning from out-of-distribution data. [CVPR2020]
- Learning Open Set Network with Discriminative Reciprocal Points [ECCV2020]
- Synthesize then Compare: Detecting Failures and Anomalies for Semantic Segmentation [ECCV2020] - [PyTorch]
- Towards Maximizing the Representation Gap between In-Domain & Out-of-Distribution Examples [NeurIPS Workshop2020]
- Memorizing Normality to Detect Anomaly: Memory-Augmented Deep Autoencoder for Unsupervised Anomaly Detection [ICCV2019] - [PyTorch]
- Detecting the Unexpected via Image Resynthesis [ICCV2019] - [PyTorch]
- Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks [ICLR2018]
- A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks [ICLR2017] - [TensorFlow]

**Journal**

- One Versus all for deep Neural Network for uncertaInty (OVNNI) quantification [IEEE Access2021]

**Arxiv**

- Neuron Activation Coverage: Rethinking Out-of-distribution Detection and Generalization [arXiv2023] - [PyTorch]
- A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection [arXiv2021]
- Generalized out-of-distribution detection: A survey [arXiv2021]
- Do We Really Need to Learn Representations from In-domain Data for Outlier Detection? [arXiv2021]
- DATE: Detecting Anomalies in Text via Self-Supervision of Transformers [arXiv2021]
- Frequentist uncertainty estimates for deep learning [arXiv2018]

### Object detection

**Conference**

- Bridging Precision and Confidence: A Train-Time Loss for Calibrating Object Detection [CVPR2023]
- Estimating and Evaluating Regression Predictive Uncertainty in Deep Object Detectors [ICLR2021]
- Gaussian YOLOv3: An Accurate and Fast Object Detector Using Localization Uncertainty for Autonomous Driving [ICCV2019] - [CUDA] - [PyTorch] - [Keras]

### Domain adaptation

**Conference**

- Guiding Pseudo-labels with Uncertainty Estimation for Source-free Unsupervised Domain Adaptation [CVPR2023] - [PyTorch]
- Uncertainty-guided Source-free Domain Adaptation [ECCV2022] - [PyTorch]

### Natural Language Processing

- R-U-SURE? Uncertainty-Aware Code Suggestions By Maximizing Utility Across Random User Intents [ICML2023] - [GitHub]

# Datasets and Benchmarks

- SHIFT: A Synthetic Driving Dataset for Continuous Multi-Task Domain Adaptation [CVPR2022]
- MUAD: Multiple Uncertainties for Autonomous Driving, a benchmark for multiple uncertainty types and tasks [BMVC2022] - [PyTorch]
- ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding [ICCV2021]
- The MVTec Anomaly Detection Dataset: A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection [IJCV2021]
- SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation [NeurIPS2021]
- Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning [arXiv2021] - [TensorFlow]
- Curriculum Model Adaptation with Synthetic and Real Data for Semantic Foggy Scene Understanding [IJCV2020]
- Benchmarking the Robustness of Semantic Segmentation Models [CVPR2020]
- Fishyscapes: A Benchmark for Safe Semantic Segmentation in Autonomous Driving [ICCV Workshop2019]
- Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming [NeurIPS Workshop2019] - [GitHub]
- Semantic Foggy Scene Understanding with Synthetic Data [IJCV2018]
- Lost and Found: Detecting Small Road Hazards for Self-Driving Vehicles [IROS2016]

# Libraries

- Fortuna [GitHub - JAX]
- Bayesian Torch [GitHub]
- A Bayesian Neural Network library for PyTorch [GitHub]
- TensorFlow Probability [Website]
- Uncertainty Toolbox [GitHub]
- Mixture Density Networks (MDN) for distribution and uncertainty estimation [GitHub]
- OpenOOD: Benchmarking Generalized OOD Detection [GitHub]

# Lectures and tutorials

- Uncertainty and Robustness in Deep Learning Workshop in ICML (2020, 2021) [SlidesLive]
- Yarin Gal: Bayesian Deep Learning 101 [Website]
- MIT 6.S191: Evidential Deep Learning and Uncertainty (2021) [Youtube]
- Hands-on Bayesian Neural Networks - a Tutorial for Deep Learning Users [IEEE Computational Intelligence Magazine]

# Books

- The “Probabilistic Machine-Learning” book series by Kevin Murphy [Book]

Source: GitHub & OneHack.Us

**ENJOY & HAPPY LEARNING!**

**ENJOY & HAPPY LEARNING!**

`Appreciate the share & feedback! don’t be cheap!`