probly

probly is an open-source Python library for uncertainty quantification in machine learning. It unifies methods that typically live in separate codebases under a single framework-agnostic pipeline of transformation, representation, quantification, and evaluation, with native support for PyTorch, JAX, Flax, and scikit-learn. The library covers second-order uncertainty representations — Bayesian inference, deep ensembles, evidential models, and credal sets — alongside calibration, conformal prediction, and uncertainty quantification for large language models. These methods can be readily applied and benchmarked across common downstream tasks such as selective prediction, out-of-distribution detection, and active learning. The library additionally provides support for first-order data, including curated datasets and evaluation metrics tailored to uncertainty quantification.

Github: https://github.com/pwhofman/probly.
Documentation: https://pwhofman.github.io/probly/.