Skip to content

Changelog

All notable changes to this project will be documented in this file.

Format: Keep a Changelog; versioning: SemVer.

Unreleased

0.1.1 - 2026-08-08

Added

  • probcal.metrics.kernel: squared kernel calibration error skce (estimators uq/ul/biased, Laplacian/Gaussian kernels, deterministic median-heuristic bandwidth with strided-subsample and tie fallbacks, probability/logit kernel scale, seeded ul pairing) and skce_test (Arcones–Giné centered bootstrap per the paper's Appendix G and the O(n) asymptotic-normal linear method, one-sided, with the distribution-free p_value_bound), after Widmann et al. (2019); report-only — deliberately excluded from evaluate() and the selector (DECISIONS 53); 21 tests incl. hand-computed anchors, brute-force references, bootstrap-identity check, and level/power studies
  • Documentation: SKCE section and selection-table row in the metrics chapter, probcal.metrics.kernel API page, namesake-disambiguation FAQ entry (the unaffiliated R package probcal and the ECAI 2025 research codebase), and a README feature-matrix column for the R package
  • Tooling: mypy targets Python 3.12 in config and CI (numpy ≥ 2 stubs use PEP 695 type statements that mypy only parses for target ≥ 3.12; DECISIONS 54); runtime 3.11 support unchanged
  • Visualization: annotated plot_reliability (stats box via the new probcal.metrics.reliability_summary aggregate, deterministic event/non-event rug thinned to ≤ 1000 marks per class), plot_ecce cumulative-drift walks with the pointwise ±2 SD envelope (on the new probcal.curves.ecce_curve, whose stat_max agrees exactly with metrics.ecce), plot_grade_backtest traffic-light chart with 90% display intervals (new ci_low/ci_high fields on both grade results, Jeffreys central / Clopper–Pearson, powered by the new numpy-only probcal._math.beta_ppf), and plot_offset_audit for fitted LogitOffset stages; seeded docs/scripts/generate_figures.py regenerates every documentation figure deterministically; DECISIONS 55; 16 tests incl. hand anchors, a scipy beta.ppf reference, and rug/annotation determinism checks

Changed

  • All plots now style themselves via a per-call rc_context house style (muted palette, no top/right spines, light grid) — global matplotlib rcParams are never touched, verified by test
  • plot_reliability: the twin-axis count-bar margin is now opt-in (counts=False default) — the rug replaces it as the density view
  • All documentation figures regenerated under the house style by the new figure script (the logit-scale reliability figure is now the annotated variant)

0.1.0 - 2026-08-07

First public release on PyPI.

Fixed

  • reliability_binned: Wilson interval bounds are now forced to contain the point estimate (floating-point noise at zero-event bins could push ci_low above a 0.0 event rate, breaking error-bar rendering)
  • plot_reliability(scale="logit"): bins with an event rate of exactly 0 or 1 (no finite logit) are omitted from the point layer instead of rendering at the clipping floor and crushing the axis; they remain visible in the count margin

Added

  • Packaging and CI: PEP 639 license metadata (SPDX license = "MIT" plus license-files, replacing the deprecated table form), Python 3.13 classifier, Development Status :: 4 - Beta, a tag-vs-version consistency check in the publish workflow, and a ci.yml matrix running lint, type-check, and the test suite on Python 3.11/3.12/3.13

  • Documentation: generated example figures in the visualization chapter, "In probcal" code snippets in every concept chapter, and a probcal-vs-netcal guidance note in the FAQ

  • Polish: README quickstart with real printed output, feature matrix vs scikit-learn/netcal/single-method packages, performance note on the deliberate absence of Rust acceleration, docs cross-linking pass

  • Datasets, tutorial, and user docs: make_pd_portfolio (beta-family generative miscalibration with exact event-rate anchoring), executed tutorial notebook pd_calibration_walkthrough.ipynb (diagnose → select → fit → re-anchor → backtest → threshold translation, incl. the counterfactual-engine interop recipe), getting-started.md with a runnable quickstart and its printed output, how-it-works.md pipeline walkthrough with schema, three-page mkdocstrings API reference, and the FAQ (inverse-map protocol, Target.probability trap, selection rules)
  • Automatic selection: CalibratorSelector — default 8-candidate menu per spec, inner stratified seeded K-fold with structurally out-of-fold-only scoring, criteria log_loss/brier/ici/smooth_ece/ece_sweep (plain ECE and Hosmer–Lemeshow refused), guardrail flags on pooled out-of-fold predictions, one-standard-error parsimony tie-break, winner refit on the full set, ranked SelectionReport; 9 tests incl. a spy-calibrator structural no-leakage check and the parsimony-on-calibrated-data behavior
  • Wrapper flows: CalibratedModel — prefit flow (score + calibrate, the credit-risk canon), cv flow with duck-typed cloning (sklearn clone if installed, else deepcopy) and stratified seeded folds, ensemble=False pooled default (one calibrator on out-of-fold scores, final model refit) vs ensemble=True fold averaging; offset_to appending inspectable LogitOffset stages; composed interval_inverse and affine_logit_coeffs_; predict_proba/predict_proba_2d; structural no-leakage tests with a clone-surviving spy model (out-of-fold scoring disjoint from training rows, each row scored exactly once)
  • Inverse maps and thresholds: interval_inverse(lo, hi, *, space, buffer_logit) implemented across the catalog — closed forms (Platt, temperature, LogitOffset), block-structure searchsorted (isotonic, CIR, histogram, scaling-binning), monotone bisection (beta, spline, Venn–Abers); UnattainableTargetError instead of silent clamping; non-monotone calibrators refuse with an explanatory NotImplementedError; thresholds.py with calibrated_interval_to_raw and the masterscale calibrated_bands_to_raw; 16 tests covering the full spec §10 list (round-trips, block-edge semantics, offset −δ shift, buffer monotonicity, space consistency)
  • Attribution adjustment: adjust_attributions → AdjustedAttribution — affine-exact mode (exact composed Shapley values for Platt/temperature/offset via affine_logit_coeffs_) and Aumann–Shapley mode (exact additivity for any calibrator, central-difference fallback on degenerate rows); logit and probability scales; shap.Explanation duck-typing without a shap import; 10 tests incl. affine/AS 1e-12 equivalence and sign/rank invariance
  • Logit offset: LogitOffset — explicit-delta and target-mean modes (unique bisection root, unit-tested against a brute-force grid), audit trail (delta_, pre_mean_, post_mean_, timestamp), audit_report() with pre/post guardrails, interpret() with odds-factor and central-tendency readings, affine logit coefficients for attribution composition
  • Curves, belt, and plots: curves.py — reliability_binned (Wilson CIs, both scales), reliability_loess, reliability_spline, calibration_belt (GiViTI-style: forward LR degree selection ≤4, information-matrix pointwise bands, identity-test p-value; reimplemented from the Nattino papers); plots.py ([viz]-guarded) — plot_reliability with logit-scale probability-labeled ticks and count margin, plot_belt, plot_comparison, plot_interval, plot_selection; SmoothReliabilityCurve result dataclass; 11 tests incl. a hand-checked Wilson interval and belt null/alternative behavior
  • Metrics catalog: probcal.metrics complete — proper scores (log_loss, brier_score, brier_skill_score, Murphy decomposition with optional bias correction, log-loss calibration/refinement split), binned estimators (ece l1/l2/max, ece_debiased, ece_sweep, adaptive_ece, hosmer_lemeshow), binning-free estimators (smooth_ece with self-consistent bandwidth, ecce, ici/e50/e90/emax, spiegelhalter_z), the recalibration-regression framework (calibration_intercept/_slope/_test, calibration_guardrails), per-grade backtests (binomial_grade_test, jeffreys_grade_test with traffic lights), and evaluate() with seeded bootstrap percentile CIs; 38 tests incl. hand-computed cases and sklearn/scipy references
  • Spline calibrator: SplineCalibrator — natural cubic basis on the logit scale, penalized IRLS with second-difference roughness penalty, λ by stratified K-fold CV on log loss, effective degrees of freedom reported, post-fit monotonicity check with warning; completes the spec §6 calibrator catalog (11 of 11)
  • Binning + Bayesian calibrators: HistogramBinningCalibrator (equal-mass/equal-width, Jeffreys shrinkage, empty-bin fallback, post-fit monotonicity flag), ScalingBinningCalibrator (Platt stage + equal-mass binning of fitted values), BBQCalibrator (Beta–Binomial marginal likelihood over a B-grid, Jeffreys prior, top-3 reporting), ENIRCalibrator (nearly-isotonic mPAVA path from raw data to the isotonic fit, BIC-weighted ensemble, is_monotone_ = False); 20 tests incl. hand-computed bin rates and the path-endpoint-equals-PAVA anchor
  • Isotonic family + Venn–Abers: IsotonicCalibrator (PAVA step map, tie pooling, clamping, optional linear interpolation, block-structure attributes), CenteredIsotonicCalibrator (CIR through weight-centered block points), VennAbersCalibrator (IVAP with predict_interval(), log-loss-minimax scalarization, width reporting in interpret()), CrossVennAbersCalibrator (stratified seeded folds, geometric-mean merge, conservative envelope interval); 19 tests incl. direct-refit agreement and width-shrinkage checks
  • Base API + parametric calibrators: BaseCalibrator (fit/predict_proba/predict_proba_2d/interpret, manual get_params/set_params, is_monotone_, affine_logit_coeffs_, interval_inverse contract stub), PlattCalibrator (Lin–Lin–Weng target smoothing), TemperatureCalibrator (safeguarded 1-D NLL solve), BetaCalibrator (variants abm/ab/a, betacal negative-coefficient refit, constraint_active_); public exports incl. logit/expit; 30 tests (identity/distortion recovery, monotonicity, small-sample stability, interpret contract)
  • Numerical core: _math.py (overflow-safe logit/expit, weighted PAVA with block structure, IRLS logistic regression with ridge stabilization and separation detection, safeguarded 1-D Newton and bisection, vectorized lgamma/erf, regularized incomplete beta and lower incomplete gamma, chi-square and normal quantiles, tricube LOESS, natural cubic spline basis), _validation.py (score/target/weight validation with 1e-12 clipping), _results.py (frozen result dataclasses with as_dict() and aligned-table reprs); 49 unit tests plus 9 reference tests against scipy/scikit-learn/statsmodels (betainc and gammainc within 1e-12, IRLS within rtol 1e-8)
  • Repository scaffold: src layout with docstring-only modules, tooling (uv, ruff, black, mypy, pytest), CI workflows (docs deploy, PyPI publish — tag-gated), documentation skeleton (mkdocs-material + MathJax), MIT license, citation metadata
  • Reference verification: all 12 ⚠ references of the spec verified against primary sources; completed bibliographic records logged in docs/DECISIONS.md entries 10–21
  • Theory guidebook, chunk 1: concepts/why-calibration.md (definitions, sources of miscalibration, decisioning and regulatory consequences, proper-scoring-rule lens) and concepts/methods-parametric.md (Platt, temperature, beta: derivations, parameter interpretation, worked reading)
  • Theory guidebook, chunk 4 (completes the guidebook — ~16,400 words total): concepts/shap-calibration.md (identifiability obstacle, affine-exact class, Aumann–Shapley mode, invariance properties), concepts/inverse-maps.md (preimage identity, generalized inverses, attainability, buffer_logit drift robustness, masterscale workflow), concepts/auto-selection.md (selector protocol, structural no-leakage, report reading), concepts/visualization.md (reliability constructions, logit-scale rationale, calibration belt)
  • Theory guidebook, chunk 3: concepts/metrics.md (full metric catalog with formulas and pathologies, bootstrap-CI protocol, report-reading order, selection-suitability table), concepts/data-splitting.md (prefit vs cv flows, ensemble vs pooled, calibration-set sizing, nested selection), concepts/offset.md (uniqueness of the bisection root, King–Zeng/Elkan/Tasche equivalences, worked re-anchoring, audit practice)
  • Theory guidebook, chunk 2: concepts/methods-nonparametric.md (PAVA with worked micro-example, CIR, histogram binning, scaling-binning sample-complexity argument, BBQ, ENIR, spline calibration, properties table) and concepts/methods-distribution-free.md (IVAP construction, validity guarantee scope, scalarization caveat, exchangeability limits, CVAP geometric-mean merge)