Skip to content

Changelog

All notable changes to this project will be documented in this file.

Format: Keep a Changelog; versioning: SemVer. Each release opens with a short summary; the itemized entries sit under Details.

Unreleased

0.4.0 - 2026-10-17

A cleanup release. The aliases deprecated in 0.3.4 are gone, payloads written before 0.3.4 no longer load, and the code is about 850 lines shorter. Numerical results are bit-identical to 0.3.5, and so are the payloads written. Two bugs are fixed: adjust_attributions on a CalibratedScorecard, and calibrate_scorecard with a Chain prototype.

Details

Removed

  • Deprecated names (warned since 0.3.4):
  • metrics.adaptive_ece: use ece(strategy="mass").
  • hosmer_lemeshow(g=): use n_bins=.
  • ecce(presorted=).
  • evaluate(seed=): use random_state=.
  • level= on reliability_smooth, corp_reliability and jeffreys_upper_bands: use confidence=.
  • BBQCalibrator.weights_ and ENIRCalibrator.weights_: use model_weights_.
  • BeltResult.lower_80, upper_80, lower_95 and upper_95: use BeltResult.bands.
  • Positional arguments to CalibratedModel.offset_to and to LogitOffset.fit's sample_weight: pass them by keyword.
  • predict_proba_2d on calibrators and CalibratedModel. Use np.column_stack([1 - p, p]), or the sklearn adapters, whose predict_proba returns the (n, 2) matrix.
  • Payloads written by probcal < 0.3.4 no longer load. Payloads written by 0.3.4 and 0.3.5 load unchanged, and the golden test files are now written by 0.3.4. The removed cases:
  • Monitor steps without log_e_increment, and the "drift onset unavailable" path. MonitorStep.log_e_increment and grade_delta_ci are now required fields.
  • Selection reports without CORP columns. SelectionReport.mcb, dsc and unc are now required fields.
  • Old state names in BBQ, ENIR, IVAP, CVAP and Spline payloads.
  • Monitor grade keys written as "1.0".
  • The CSV-based scorecard fingerprint.
  • Private helpers. Nothing outside the package used these:
  • _math.norm_cdf, _math.gammainc_lower and BaseCalibrator._clone;
  • metrics._common.binary_y and metrics._common.effective_weights;
  • the metrics._deprecation module;
  • CalibrationMonitor._cs_grid and _cs_max;
  • _steps.migrate_state;
  • the _prep and _resolve_grades re-exports;
  • other unused helpers, unused parameters and stale type: ignore/noqa comments.

Fixed

  • adjust_attributions accepts any wrapper that exposes the calibrator protocol: CalibratedModel, CalibratorSelector, the sklearn adapters and CalibratedScorecard. With a non-affine calibrator, a scorecard used to raise IndexError.
  • calibrate_scorecard clones a Chain prototype correctly. It used to raise TypeError.
  • smooth_ece works on numpy 1.26, the declared minimum. It had used np.trapezoid, which requires numpy 2.0.
  • The HTML report escapes & in the CORP note.
  • The hl_e_test docstring and the metrics page describe its weights as scale-invariant exponents, which is how the code treats them.

Changed

  • CalibratedModel and CalibratorSelector take interval_inverse, point_inverse and (selector) interpret from the shared calibrator protocol. Results are unchanged.
  • For the ensemble flow, the inverses raise NotImplementedError with a pointer to ensemble=False.
  • A selector candidate whose class only shares a registered name is rejected when saving.
  • CenteredIsotonicCalibrator().interpolation reads "linear". The CIR map is a linear interpolation through block centres, so the attribute now says what the map does.
  • Validation. n_boot and n_resamples must be integers; reliability_smooth used to truncate 2.7 to 2. Level, alpha and count arguments share one validator, which changes some error messages. Examples: "confidence must satisfy 0 < confidence < 1", and "y must not be empty" for an empty monitor batch.
  • CalibratedModel(flow="cv") decides once, before the fold fits, whether the model accepts sample_weight, and warns once.
  • Internals. Code that did the same job several times now has one shared version:
  • one bisection for every numeric inverse;
  • one weighted Bernoulli log-likelihood;
  • one within-bin variance correction;
  • one prototype encoder and decoder for SegmentedCalibrator and CalibratorSelector.

Other internal cleanups: - CIR reuses the isotonic linear path. - SegmentedCalibrator looks up segments with vectorised code. - pava expands blocks without a Python loop. - Dead branches and redundant docstrings are removed. - Tooling. A CI job runs the test suite on numpy 1.26, the declared minimum, and mypy reports unused type: ignore comments.

0.3.5 - 2026-10-12

A correctness release that finishes the work started in 0.3.4. 0.3.4 used Kish's effective n for weighted tests, and that was wrong whenever weights vary with the per-row variance. Every weighted statistic now uses its exact variance and still does not depend on the scale of the weights. The monitor is anytime-valid for any sample weights. Step calibrators no longer give a high PD below a tied minimum score. Raw band edges agree with Masterscale.assign to the last floating-point step (ulp). Hosmer–Lemeshow now uses the correct out-of-sample reference distribution by default. Unweighted and unit-weight results are bit-identical to 0.3.4, except where the release notes below say otherwise. Every saved artifact still loads.

Details

Fixed

  • Weighted metrics. Statistics now use exact weighted variances and stay invariant to rescaling all weights. In 0.3.4 they used Kish-rescaled weights.
  • spiegelhalter_z uses the null variance sum(w^2 (1-2p)^2 p(1-p)). When weights correlate with p, its type-I rate was 0.14 at alpha = 0.05; it is now 0.05.
  • The ECCE null envelope uses the exact prefix SD sqrt(cumsum(w^2 p(1-p))) / sum(w).
  • ece_debiased and murphy_decomposition(bias_corrected=True) use each bin's own effective count, (sum_b w)^2 / sum_b w^2.
  • hosmer_lemeshow uses the per-group variance sum_g(w^2) p~(1-p~).
  • calibration_test refers the weighted LR to a Rao–Scott corrected chi-square. CalibrationTestResult gains a field df, which is fractional when weighted.
  • calibration_belt uses weighted ML with sandwich bands and a Rao–Scott p-value.
  • The monitor's residual-LR diagnostic uses a Rao–Scott correction.
  • estimate_offset gives se = sqrt(sum w^2 q(1-q)) / sum w q(1-q). The old 1 / sqrt(sum w q(1-q)) read weights as counts, so with a constant weight of 10 the 95% interval covered 48% of the time. SegmentedCalibrator shrinkage (se_, tau2_, shrink_) and its predictions no longer depend on the scale of sample_weight. The monitor's apply_recommendation audit se is fixed by the same change.
  • Monitoring.
  • Sample weights enter as exponents min(1, w / mean(w)) per factor, which keeps every factor an e-value for any weights. Before, a constant weight of 10 raised a null alarm in 87% of runs with no warning. Constant weights now reproduce the unweighted monitor, and the "non-uniform weights" warning is gone. metrics.hl_e_test follows the same rule.
  • Payloads written by 0.3.4 are no longer re-keyed on load. String grades such as "1.0" used to become "1".
  • The onset estimate never falls after alarm_at, ties resolve to the earliest batch, and without plug-in evidence the window is the whole history instead of the last batch.
  • Vectorised grade keys match grade_key: mixed lists, float32 and large integers.
  • moc_offset_from_counts no longer depends on the scale of the weights.
  • A rejected batch no longer stores its masterscale fingerprint. An empty mixture_grid and a delta_ci_grid that does not bracket 0 are rejected at construction.
  • slope_ci handles windows with constant logits.
  • Calibrators and inverse maps.
  • When more than 1/B of the calibration mass is tied at the minimum score, equal-mass binning no longer leaves an empty first bin, so scores below the tie no longer get the prior 0.5 (BBQ) or the global rate (Histogram). In the example: 0.065 becomes 0.0025.
  • Empty bins under strategy="width" take their neighbours' rate.
  • BBQ on constant scores predicts a constant.
  • Every left and right inverse, both the affine closed form and the bisection path, is corrected to the exact ulp. calibrated_bands_to_raw and calibrated_interval_to_raw now partition the raw axis exactly like Masterscale.assign for Platt, Temperature, Beta, ScalingBinning, linear Isotonic and CIR. Before, up to about 9 ulp could disagree at shared edges.
  • Metrics, other.
  • ICI, E50, E90 and logloss_calibration_refinement no longer depend on the row order of tied p (LOESS fits on tie-block means). With constant p, ICI is 0, and the evaluate bootstrap CI contains the point estimate.
  • The weighted quantile is continuous in the weights and equals np.quantile for equal weights. Nearly-equal weights used to switch rules and move E90 and bin edges.
  • jeffreys_upper_bands no longer produces zero-width bands. A grade whose bound would not exceed the running maximum is merged into the preceding band (key "A+B", with a UserWarning), so every band is strictly increasing and usable by Masterscale.
  • When logit(p) is constant, calibration_slope returns NaN with a warning instead of a ridge value, and calibration_test tests the intercept only (df = 1).
  • BeltResult, BinomialGradeResult, JeffreysGradeResult and EcceCurve compare by value. The 0.3.4 fix had missed these four.
  • Wrappers and adapters.
  • CalibratedModel(flow="cv") with a Pipeline and sample_weight warns and fits the model unweighted, like CalibratedClassifier. Before, it raised.
  • offset_to(target_mean=) honours the calibration weights, and takes sample_weight= together with X.
  • A model loaded from JSON without its estimator keeps its model_ref and fingerprint when saved again.
  • Prefit CalibratedModel checks that the model's classes_ are {0, 1}. A model trained on string labels used to calibrate the wrong column silently.
  • CalibratorSelector and CalibratedModel(ensemble=True) require at least cv events and non-events, with an actionable error. Before, they failed inside a fold.
  • clone_unfitted clones a Chain, and nested candidates, deeply.
  • adjust_attributions accepts SklearnCalibrator and CalibratedClassifier.
  • Masterscale and reports.
  • assign, index, table and build_masterscale use p without clipping. p = 0 or 1 near an edge used to land in the wrong grade, and mass at p = 1 produced an edge at 0.999999999999.
  • A -0.0 bound gives the same fingerprint as 0.0.
  • interpret() describes the half-open band inversion.
  • corp_reliability(n_resamples=0) and validation_report(n_boot=0) skip the bands instead of crashing.
  • Plots.
  • plot_reliability(by=, stats=MetricReport) raises, because it used to print pooled metrics on every group panel. A GroupedMetricReport fills each panel.
  • Group labels 1 and "1" raise, as in evaluate.
  • No figure leaks on any error, and an unknown scale raises.
  • Docs.
  • Corrected stale outputs (optbinning, offset, cutoffs, conservatism), the logit-axis claim in the README, a reference to a non-existent probcal.load(), the SECURITY I/O list, and the pandas note.
  • OffsetEstimate is in the API reference.
  • Doc snippets' # -> expected comments are now checked against actual output.

Added

  • BBQCalibrator(monotone=False). Set to True, it isotonizes each candidate binning with weighted PAVA before averaging, so the map is monotone and invertible. Default BBQ fits come out non-monotone on about 90% of realistic PD portfolios. The option leaves held-out accuracy unchanged or slightly better. With the default, predictions and payload fingerprints are bit-identical. A payload saved with monotone=True does not load in 0.3.4.
  • hosmer_lemeshow(ddof=); see below.

Changed (numerically visible)

  • hosmer_lemeshow refers to chi2(groups) by default (new keyword ddof=0). Out of sample, the classical groups - 2 rejected about 12% of calibrated portfolios at a nominal 5%. With ddof=0 the rate is about 5%. Every p-value is now larger and df grows by 2. ddof=2 restores 0.3.4 exactly, and is correct for a logistic model fitted on the same data.
  • Weighted statistics. With unit or constant weights nothing moves. With weights independent of p (U(0.5, 2), n = 3000), values move by under 2%. With weights correlated with p:
  • Spiegelhalter z by about 2%.
  • Hosmer-Lemeshow by 9-16%.
  • calibration_test by about 6%.
  • ece_debiased by 5-17%.
  • Murphy reliability by 8-12%.
  • The ECCE envelope by about 10%.
  • Belt bands by up to 42% narrower; Kish had overstated their variance.
  • estimate_offset SE with weights: about 2x larger for lognormal or p-correlated weights, and 3.16x larger for a constant weight of 10. Weights with mean below 1 now give a smaller SE. SegmentedCalibrator shrinkage can change substantially: in one example, the largest segment's shrink_ moved from 0.93 to 0.51. Integer weights still give the same delta as duplicated rows, but no longer the same SE.
  • Monitor with non-constant weights. E-values, confidence sequences, plug-ins and onset increments all change. Null alarm rates drop to at most alpha, and the cost is power with skewed weights. Onset labels can move earlier.
  • Weighted equal-mass bins (Histogram, ScalingBinning, BBQ) move by up to about 3 inter-observation gaps under the new weighted quantile. With U(0.5, 2) weights, the median prediction change is about 4e-5, and boundary rows can jump.
  • Binned calibrators with ties at the minimum score.
  • They have fewer bins (see Fixed).
  • ScalingBinning stores one bin fewer, but its predictions are unchanged.
  • Inverse-map bounds move by at most about 9 ulp.
  • Tied p. ICI, E90 and the log-loss calibration term move by at most 4e-15 relative.
  • Monitor grades from float32 data are keyed as the exact double, matching grade_key.
  • hl_e_test with integer weights no longer equals the e-value of duplicated rows: each row is one draw, and its weight is an exponent min(1, w / mean(w)) per grade.

Changed

  • Validation errors.
  • A prefit CalibratedModel whose model has classes_ other than {0, 1} raises.
  • A model whose fit accepts only **kwargs is fitted unweighted with a warning.
  • A hand-made Masterscale whose outer edge lies within 1e-12 of 0 or 1 raises for p = 0 or 1. Clipping used to hide this.
  • Read-only and unhashable results.
  • BeltResult.bands is a read-only mapping.
  • The grade and ECCE result classes are unhashable, like the other array-holding results.
  • New public names.
  • report.py and integrations/optbinning.py define __all__.
  • CalibratorProtocolMixin moved to probcal._protocol.

0.3.4 - 2026-10-03

A correctness release from a full-package review. Inverse maps are exact and agree with Masterscale (step calibrators no longer give overlapping raw bands), is_monotone_ is truthful, the monitor's global alarm keeps its anytime-valid guarantee when grades appear mid-stream, metric weights follow one convention, and tail p-values no longer round to 0. Serialization output is strict JSON. Every schema-1 artifact still loads. Deprecated names warn and are removed in 0.4.0; numerically visible changes are listed with their size below.

Details

Fixed

  • Inverse maps and thresholds.
  • calibrated_bands_to_raw returns half-open raw intervals that partition the raw axis exactly like Masterscale.assign; plateaus of isotonic/histogram maps used to land in two bands.
  • IsotonicCalibrator(interpolation="linear").interval_inverse inverted the step map, not the interpolated one.
  • ScalingBinningCalibrator never set is_monotone_ (a decreasing Platt stage gave lo > hi), and its right bound could fall outside the preimage.
  • is_monotone_ for BBQ and Spline is checked exactly over the whole domain (a fixed probe grid missed low-PD regions); Segmented and Selector keep it across JSON.
  • CalibratorSelector delegates the full protocol (point_inverse, exact interval_inverse, affine_logit_coeffs_, complexity_rank) to its winner.
  • Monitoring.
  • The global e-mixture has a fixed component set. Declared grades= are in it from the first batch and undeclared labels raise; with grades=None per-grade processes are reported but kept out of the alarm (warned once). Previously a grade entering mid-stream joined at e = 1, which could raise e_global without evidence.
  • Empty batches, plug_in_window=0 and a negative min_history are rejected; grade labels 1 and 1.0 are one grade; the onset series counts only active components; report() no longer prints nan.
  • apply_recommendation(target=CalibratedModel) works on a model loaded from JSON.
  • Metrics.
  • ecce depended on the row order of tied scores.
  • jeffreys_upper_bands could lower an upper bound and produce a zero-width band (now a running maximum, like pluto_tasche).
  • p-values below ~1e-16 no longer round to 0 (Hosmer-Lemeshow, calibration_test, belt, Spiegelhalter, SKCE, binomial grade test).
  • spiegelhalter_z with constant p = 0.5, hosmer_lemeshow with fewer than 3 groups (was a fake df = 1), evaluate(n_boot=0), and grade tests with mismatched lengths now give NaN or a clear error.
  • reliability_smooth reports NaN in data gaps instead of a rate of 0 with a [0, 0] band.
  • evaluate(by=) no longer merges the labels 1 and "1".
  • emax(sample_weight=) and reliability_loess(sample_weight=) were silently ignored.
  • Wrappers and adapters.
  • CalibratedModel and CalibratedClassifier pass X to the model untouched (DataFrames, string features, ColumnTransformer); CalibratedModel validates cv and forwards sample_weight to model.fit in the cv flow.
  • CalibratedClassifier(stratify=False) really uses shuffled KFold (it stratified and ignored random_state); cv accepts any splitter; prefit checks classes_.
  • The sklearn adapters reject negative sample_weight instead of dropping those rows.
  • adjust_attributions accepts a CalibratedModel; the affine target is exact.
  • Objects and serialization.
  • Masterscale supports copy.deepcopy and pickle; its provenance is deep-frozen.
  • Result dataclasses holding arrays compare by value instead of raising.
  • to_json writes strict JSON: non-finite floats are tagged {"__float__": "inf"} instead of Infinity/NaN tokens. Constructor parameters given as lists stay lists.
  • A Segmented model with a custom-candidate Selector base serializes.
  • Numerics. irls_logistic no longer reports convergence after a stalled line search under quasi-separation; Spline uses the same safeguarded solver.
  • Other.
  • make_pd_portfolio(intercept=) had no effect, and the identity case ignored event_rate. The default portfolio is unchanged.
  • validation_report validates y/p up front and orders every grade table by mean PD.
  • LogitOffset with an unattainable target_mean raises UnattainableTargetError naming the attainable range.
  • Negative buffer_logit and unknown space raise everywhere.
  • Plots:
    • plot_comparison legend colours;
    • plot_reliability(by=) forwards stats/annotate/counts/rug;
    • logit scale with no finite bin no longer crashes;
    • plot_interval handles unsorted scores;
    • plot_murphy leaks no figure on error;
    • the grades panel stays inside the figure;
    • a caller-supplied ax gets the same styling.

Changed (numerically visible)

  • Venn-Abers pools tied calibration scores, so predictions no longer depend on row order. Only tied scores or queries equal to a calibration score move: up to 2.5e-2 in-sample, mean 5.6e-4 on 3-dp tied data. Payloads saved before 0.3.4 keep their old tables (ties_pooled_=False).
  • Equal-mass bins honour sample_weight (Histogram, ScalingBinning, BBQ). Weighted fits move by up to ~6e-2; unweighted fits are bit-identical.
  • Weights are relative frequency weights. Sample-size-dependent statistics use Kish's effective n: Spiegelhalter, ece_debiased, the Murphy bias correction, Hosmer-Lemeshow, calibration_test, the calibration belt, and the ECCE null envelope. Unit weights give identical numbers. With U(0.5, 2) weights, HL and LR statistics scale by about 0.72.
  • ece_sweep follows Roelofs et al. (2022) and stops at the first non-monotone bin count. The chosen count changed in about half of synthetic test sets, with values moving ≤ 0.012 at n = 500 and ≤ 0.003 at n = 1e4. It is about 5x faster. rule="largest" restores 0.3.x.
  • ECCE reads the cumulative walk at the end of tied-score blocks and weights stat_mean. Results are unchanged without ties.
  • skce_test bootstrap replicates use the observed statistic's scale.
  • calibration_belt fits on centred, scaled logits (bands move ≤ 3e-13). It is documented as a simultaneous Wald approximation whose p-value ignores degree selection.
  • CalibratedClassifier(stratify=False) fold assignment changed (see Fixed).

Changed

  • Masterscale equality and hash include provenance, so two scales are equal exactly when their fingerprints match. interpret() lists lo/hi per grade.
  • plot_reliability(by=) panels follow the single-panel defaults; ax=/smooth= with by= raise. plot_belt draws from BeltResult.bands.
  • Internals:
  • One JSON-IO mixin and payload envelope for every serializable object.
  • Shared step-function, binning and fold helpers (_steps, _validation).
  • metrics/__init__ is split into private modules (__all__ unchanged).
  • The monitor is split into _processes (ConfidenceSequence), _diagnostics and _actions.
  • The sklearn adapters share base and protocol mixins; the estimator-check tables live in probcal.sklearn._xfail.
  • bern_log_lr/logsumexp live in _math, so metrics no longer import the monitor.
  • Dead private helpers are removed (newton_1d, norm_ppf, erf_vec, the scalar LOESS path).
  • Tooling:
  • The version is single-sourced from probcal.__version__.
  • black is replaced by ruff format, enforced in CI.
  • CI runs slow tests once and mypy on 3.11 and 3.12; docs deploy only after green CI; the release workflow runs the full CI first.
  • Plot regression tests compare structural snapshots instead of pixel hashes.

Added

  • BeltResult.levels and BeltResult.bands ({level: (lower, upper)}).
  • CalibratedModel.with_offset(offset) and CalibratedModel.point_inverse.
  • Masterscale.from_edges(..., provenance=).
  • ece_sweep(rule=, max_bins=) and order= on binomial_grade_test/jeffreys_grade_test.
  • BinomialGradeResult.p_value, CalibrationTestResult.intercept/.slope.
  • evaluate(random_state=), hosmer_lemeshow(n_bins=), and confidence= on jeffreys_upper_bands, reliability_smooth and corp_reliability.
  • SplineCalibrator.converged_/knots_/theta_; model_weights_ on BBQ and ENIR; scores_/ties_pooled_ on Venn-Abers; CrossVennAbersCalibrator.ivaps_.
  • SklearnCalibrator exposes the calibrator protocol and JSON surface.
  • CalibratedScorecard.from_json(..., scorecard=). The scorecard fingerprint no longer depends on pandas CSV formatting; old fingerprints still verify.
  • validate_binary_y(require_both_classes=).

Deprecated (removed in 0.4.0)

  • adaptive_ece → ece(strategy="mass").
  • ecce(presorted=).
  • evaluate(seed=) → random_state=.
  • hosmer_lemeshow(g=) → n_bins=.
  • level= on jeffreys_upper_bands, reliability_smooth and corp_reliability → confidence=.
  • BeltResult.lower_80/upper_80/lower_95/upper_95 → bands[level].
  • BBQCalibrator.weights_ and ENIRCalibrator.weights_ → model_weights_.
  • Positional arguments to CalibratedModel.offset_to (use keywords).
  • Positional sample_weight in LogitOffset.fit.

0.3.3 - 2026-09-08

A masterscale is one object. Masterscale holds the grade ladder, assigns grades under one boundary convention (lo <= p < hi, top band closed), and serializes with a fingerprint; every grade-taking function, the monitor, the report, and the band translators accept it, and build_masterscale designs one from data. Every example now builds grades through it. No breaking changes: label arrays and band dicts behave exactly as before.

Details

Added

  • Masterscale: from {name: (lo, hi)} or from_edges(edges, names=None, lo=0.0, hi=1.0); validation names the offending band. assign/index (half-open, top band closed, out-of-range raises), bands, edges, names, table(y, p) -> GradeTable, interpret(), JSON round trip and fingerprint() (schema 1, golden file added).
  • Accepted in place of a label array by binomial_grade_test, jeffreys_grade_test, hl_e_test (grades then best to worst, present grades only), jeffreys_upper_bands, pluto_tasche_from_arrays (new keyword-only p=; order defaults to the scale's), CalibrationMonitor.update(grade=) (fingerprint recorded as masterscale_fingerprint_; a different scale later raises), validation_report(grades=) (grade table plus fingerprint in the header); read directly by calibrated_bands_to_raw and CalibratedScorecard.masterscale.
  • build_masterscale(y, p, *, n_grades, min_count=0, min_events=0, objective="likelihood", target_shares=None, prebins=512, sample_weight=None, names=None): exact dynamic programming over equal-mass pre-bins, likelihood or target-share objective under per-grade floors; infeasibility names the binding floor; provenance travels with the scale. Pinned against brute force.

Documentation

  • Set cutoffs and invert maps gains "Your own masterscale" and "Design the grades"; the conservatism chapter, catalog, glossary, and API stability name the object.
  • Every example, script, and notebook builds grades through Masterscale; the repository previously mixed (lo, hi] and [lo, hi). A docs test fails on hand-rolled searchsorted/digitize grade assignment. The snippet vocabulary gains ms. Regenerated outputs are unchanged apart from timestamps.
  • Every page whose snippets draw on the held-out data vocabulary (s_cal, y_cal, ...) now opens with a collapsed box defining those names, each block names the ones it reads in its first comment, and a docs test enforces both. The monitoring guide states the assumptions behind the anytime-valid guarantee; the serialization chapter states what "bit-identical" covers and that metric estimates are not promised across releases. API stability states the two-tier change policy (API-breaking versus numerically visible) up front and drops the per-release symbol lists, which the changelog already carries; the status paragraph is one shared snippet. The calibrator catalog is split into a properties table and a guidance table. Every scikit-learn version floor links to the support matrix.

0.3.2 - 2026-09-07

Documentation only; no code changes. The README on PyPI now matches the docs site.

Details

Documentation

  • README leads with what scikit-learn does not produce (logit-scale guardrails, the auditable offset stage, per-grade backtests) and carries a measured table against scikit-learn and netcal on the Taiwan credit-card default dataset; the calibrator table moved to guide/choosing.md.
  • The comparison benchmark gains that dataset (scorecard base model with integer points, an 8-band masterscale), PlattCalibrator and IsotonicCalibrator rows, and per-method grade-size tables.
  • Both quickstarts fit on one synthetic portfolio and report on a held-out draw; the previous versions scored the fitting rows. Getting started defines every name its blocks use.
  • API reference adds Chain, probcal.monitor, expit/logit, and an Integrations page (probcal.sklearn, probcal.integrations.optbinning); a docs test fails if a public symbol is missing from the rendered reference.
  • New Glossary page; one Tutorials section for both notebooks; plain phrasing on reader pages.
  • This changelog opens each release with a summary and collapses the itemized entries.

0.3.1 - 2026-09-05

A predict_proba matrix now goes straight into any calibrator, metric, or offset; no column slice first. Chain can be fitted from unfitted stages, not only assembled from fitted ones, and works inside sklearn pipelines. SklearnOffset adds the offset as its own pipeline step with its own lifecycle. No breaking changes: every previously accepted input and object behaves identically.

Details

Added

  • Two-column probability-matrix input, core-wide. Every place scores enter the package (calibrator fit/predict_proba/point_inverse, LogitOffset, Chain, the whole metric catalog, probcal.sklearn's probability-input mode) now accepts an (n, 2) array whose rows sum to 1 within 1e-6 and uses column 1, the shape every predict_proba returns. One shared validator, one rejection message.
  • Fittable Chain. Chain(stages) accepts unfitted stages and gains Chain.fit(s, y, sample_weight=None), which fits the head calibrator and then each offset in turn on the running calibrated probabilities. Every stage is always refit; composing already-fitted objects is how a stage stays frozen. stages is stored verbatim, calibrator_/offsets_ are read-only properties, and every reading method raises the standard not-fitted RuntimeError before a fit. get_params/set_params gain stages__i__param nesting (LogitOffset gained get_params/set_params to support it). The serialization payload is unchanged. Both sklearn adapters clone the calibrator prototype via sklearn.base.clone, so SklearnCalibrator(calibrator=Chain([...])) and CalibratedClassifier(calibrator=Chain([...])) work with an unfitted chain.
  • SklearnOffset. probcal.sklearn.SklearnOffset(delta=None, target_mean=None, *, positive_column=1), a TransformerMixin/BaseEstimator: fit(X, y=None, sample_weight=None) fits an inner LogitOffset as offset_; transform returns the shifted (n, 1) column, predict_proba the (n, 2) matrix, to_dict the fitted offset's payload. Runs the sklearn estimator checks with its own inapplicability table.
  • positive_column. Keyword-only on SklearnCalibrator and SklearnOffset, naming which column of a two-column input is the positive class. At fit time a calibrator (which sees y) warns once if the selected column's event mean is below the non-event mean, naming the flip as the likely fix; probcal warns and never auto-flips. input="logit" stays single-column. n_features_in_ (1 or 2) is recorded at fit and enforced at predict.
  • LogitOffset.fit(y=). fit accepts an optional keyword-only y=None, ignored in both modes, so Chain.fit has one call shape across calibrator and offset stages.

Fixed

  • A bare (n, 2) matrix from model.predict_proba(X) raised in every fit/predict_proba call, forcing a manual column slice before every calibration step. The two-column input rule removes that friction end to end, pinned against a real LGBMClassifier matrix flowing through SklearnCalibrator and SklearnOffset in one Pipeline.

Documentation

  • Reader-oriented reorganization. The nav now answers "who is arriving": Learn (install, how it works, the 20-minute walkthrough), Guide (task-ordered along calibrate → evaluate → invert/decide → monitor → act → report, with the integrations beside it), Concepts (regrouped theory). The front page states that lifecycle once, linking each stage to its guide page; no page changed URL. Three assembled guide pages are new: guide/choosing.md (the calibrator catalog, every factual column pinned by a test), guide/cutoffs.md (policy PD → raw cutoff → masterscale → recourse), guide/auditability.md (artifact → what it proves → how to verify it, plus a worked reviewer session). Every Python block in the docs is executed page by page in CI against a fixed data vocabulary (convention in docs/README.md). Every function in probcal.plots has a committed, alt-texted figure on a prose page. guide/report.md shows a full sample validation report.
  • guide/sklearn.md: each tier states its input shape ((n,), (n, 1), or a predict_proba matrix (n, 2)), covers positive_column and the orientation warning, and gains "Stacking an offset": one Chain step, or a separate SklearnOffset step swapped in from apply_recommendation's output without touching the calibrator. guide/monitoring.md gains "Re-offsetting inside a pipeline". concepts/offset.md describes Chain.fit's sequential fitting: stages fit on the same calibration data, the offset anchoring the calibrator's in-sample output exactly as CalibratedModel.offset_to does; no cross-fitting, no automatic MLE offset.

0.3.0 - 2026-08-29

The governance release. Validation reports, grouped evaluation, segmented calibration with empirical-Bayes shrinkage, and the monitoring loop closed: apply_recommendation, a drift-onset estimate, per-grade confidence sequences, and margin-of-conservatism offsets. Pluto–Tasche most-prudent PDs and Jeffreys masterscale bands for low-default portfolios. CORP reliability diagrams, the MCB–DSC plane, attributes and Murphy diagrams. evaluate runs 3.5× faster with bit-identical point estimates. No breaking API changes; every 0.2.0 call and serialized file keeps working. One default changed: the monitor's post-alarm diagnostic window now starts at the estimated drift onset (recommendation_window="trailing" restores 0.2.0 behaviour). One numerical fix: a weighted PlattCalibrator fit differs from 0.2.0; every unweighted fit, and hence every default and golden file, is unchanged.

Details

Added

  • Grouped evaluation. metrics.evaluate(..., by=None) and GroupedMetricReport(pooled, groups, reports, counts) (re-exported from probcal.metrics). by=None is the existing report, byte-identical to 0.2.0; given per-observation group labels, each sorted group gets its own report (seed seed + 1000 * i), identical to calling evaluate() by hand on that slice, plus a pooled report. A single-outcome-class group raises ValueError naming the group. to_frame() returns a list of {group, metric, value, ci_low, ci_high} dicts, or a pandas DataFrame when pandas is importable. Side-by-side reporting only: no cross-group test or multiplicity correction. plots.plot_reliability(..., by=) draws the matching faceted grid (pooled panel first) and returns the Figure. New chapter guide/groups.md.
  • Validation reports. probcal.report.validation_report(y, p, *, calibrator=None, monitor=None, grades=None, by=None, title=None, path=None, format="html", n_boot=200, seed=42) -> str: a self-contained validation document assembled from the existing public API. HTML is a single file with figures embedded as data URIs and no scripts or external links; format="markdown" writes GFM tables plus PNG files to <path stem>_figures/. Sections are omitted when their input is absent: reliability (CORP and smooth) and the metric table always render; the rating-grades section (Jeffreys backtest, Pluto–Tasche, Jeffreys upper bands) with grades; grouped evaluation with by; monitoring with monitor; a collapsible calibrator appendix (to_json() plus interpret()) with calibrator. n_boot/seed drive every resampling site, so two builds from the same inputs are byte-identical apart from the timestamp line. Labels are HTML- or GFM-escaped. import probcal.report stays free of matplotlib until a figure renders; calling it needs probcal[viz]. Not re-exported from top-level probcal. New chapter guide/report.md.
  • Segmented calibration. probcal.SegmentedCalibrator(base=None, *, unseen="global"): empirical-Bayes shrunken per-segment logit offsets on top of a shared base map (default BetaCalibrator()). Per segment, an offset-only logistic MLE against the base map's predictions; between-segment variance by the DerSimonian–Laird estimator; each segment's offset shrunk by tau2/(tau2+se**2), so a noisy small segment shrinks toward the base map and tau2=0 recovers complete pooling. fit/predict_proba take keyword-only segments=; an unseen label at predict time is delta=0 (unseen="global") or a ValueError (unseen="raise"). interval_inverse/point_inverse take segment=. Chain has no segments= slot, so Chain([seg, ...]) always predicts through the global map (documented limitation). Fully serializable with its own golden file. On a 6-segment simulation with sizes 30 to 3,000, mean MSE 0.124 for empirical Bayes versus 0.235 (no pooling) and 0.168 (complete pooling). New chapter concepts/segmented.md.
  • Closing the monitoring loop. CalibrationMonitor.apply_recommendation(target=None) -> AppliedAction (monitor.AppliedAction, re-exported from probcal.monitor): for kind="re-offset", estimates the log-odds shift by maximum likelihood on the same window report()'s diagnostics use, composes the fitted LogitOffset onto target (a Chain or a CalibratedModel; target is never mutated), and returns a fresh CalibrationMonitor with the same constructor parameters, because the old e-process's guarantee no longer holds once the pipeline changes. kind="re-fit"/"none" return no offset; automatic re-fitting stays out of scope. AppliedAction carries kind, offset, composed, monitor, the batch window, and an audit dict of fingerprints plus delta/se, and is itself serializable.
  • Drift-onset localization. Each MonitorStep records log_e_increment, the batch's additive log-LR contribution; monitor._onset.estimate_onset(increments) -> int is a backward-CUSUM argmax (an estimate, not a change-point test). report() runs it after an alarm and exposes MonitorReport.onset_label: str | None, with a matching sentence in reasoning. Steps loaded from a pre-0.3 payload carry log_e_increment = None; such a monitor reports onset_label = None and falls back to the trailing window. Median localization error 1 batch on a 24-batch simulation with drift at batch 12.
  • Per-grade confidence sequences. Each grade in the grade array gets its own time-uniform confidence sequence for its own offset, surfaced as MonitorStep.grade_delta_ci: dict[str, tuple[float, float] | None] (empty without grades). plots.plot_e_process(report, *, grades_panel=False, ax=None): grades_panel=True adds a second axes with every grade's band; the default renders pixel-identically to 0.2.0. Serialization is additive; a 0.2.0 monitor payload still loads and continues.
  • Per-grade e-test audit. metrics.hl_e_test(y, p, grades, *, mixture_grid=(0.1, 0.25, 0.5, 1.0), sample_weight=None) -> HlEResult: a fixed-sample mixture-LR e-value per rating grade (the construction CalibrationMonitor's offset e-process uses, without the predictable plug-in), multiplied across grades; e_value, p_value = min(1, 1/e_value), per-grade e_grade, construction="mixture-lr". Named a safe Hosmer–Lemeshow analogue: Henzi, Puke, Dimitriadis & Ziegel (2024) motivated it and is not the implemented construction. Simulated type-I P(e >= 20) = 0.0025. Re-exported from probcal.metrics.grade.
  • Audited offset MLE. probcal.offset.estimate_offset(y, p, *, sample_weight=None) -> OffsetEstimate (delta, se, n, events, weight_sum) and offset_from_estimate(est, p) -> LogitOffset, both re-exported from probcal: the offset-only logistic MLE with a Fisher-information standard error, the companion to LogitOffset's policy-driven target_mean mode. The MLE and LogitOffset(target_mean=mean_w(y)) share one root-finder and agree bit-for-bit.
  • Margin-of-conservatism offsets. probcal.monitor.moc_offset(monitor_or_report, *, level=None) -> LogitOffset reads the upper end of the monitor's confidence sequence for the current offset and returns a fitted LogitOffset(delta=hi) (a level needs the live monitor; a frozen MonitorReport raises TypeError). moc_offset_from_counts(y, p, *, level=0.9, sample_weight=None) re-anchors p's mean at the one-sided Jeffreys posterior upper quantile of the observed event rate. Both re-exported from probcal.monitor and probcal.
  • Jeffreys masterscale bands. metrics.jeffreys_upper_bands(y, p, grades, *, level=0.9, order=None) -> dict[str, tuple[float, float]]: the per-grade Jeffreys upper bound packaged as the contiguous {grade: (lo, hi)} table thresholds.calibrated_bands_to_raw consumes; a zero-default grade still gets a strictly positive hi. Bounds are monotonized by weighted isotonic regression in order (default: grades sorted by mean p, best first), with a UserWarning only when something changed.
  • Pluto–Tasche most-prudent PDs. metrics.pluto_tasche(grade_n, grade_d, *, confidence=0.9, grades=None) -> PlutoTascheResult and pluto_tasche_from_arrays(grades, y, *, order, confidence=0.9, sample_weight=None): the one-period most-prudent PD for ordered grades, pooling each grade with every worse grade and taking the one-sided Clopper–Pearson upper bound. Accepts an all-zero y. pd_upper is replaced by its cumulative maximum best to worst; PlutoTascheResult.monotonized reports whether that fired. Simulated minimum per-grade coverage 0.92 at confidence 0.9. New chapter concepts/conservatism.md.
  • CORP reliability diagrams. curves.corp_reliability(y, p, *, sample_weight=None, bands="consistency", level=0.9, n_resamples=200, random_state=42) -> CorpResult: the PAV recalibration map of Dimitriadis, Gneiting & Jordan (2021), weighted throughout, plus the exact score == mcb - dsc + unc decomposition of both Brier score and log loss. bands="consistency" resamples under the calibration null, "confidence" bootstraps the data, None skips bands; both are pointwise, not simultaneous. plots.plot_corp(result, *, scale="probability", show_decomposition=True, ax=None). New chapter concepts/corp.md.
  • MCB–DSC plane. plots.plot_mcb_dsc(candidates, *, score="brier", ax=None): each candidate at (DSC, MCB) with iso-score diagonals, from a {name: (y, p)} mapping or a fitted CalibratorSelector's report_. SelectionReport gains mcb, dsc, unc columns (None on pre-0.3 reports).
  • Kernel reliability curves. curves.reliability_smooth (KernelReliabilityCurve): a reliability construction whose bandwidth is metrics.smooth_ece's fixed-point sigma_star, so curve.smooth_ece reproduces the metric bit-for-bit; Nadaraya–Watson event rate and density at that bandwidth, with a seeded bootstrap ribbon (n_boot, default 100; 0 disables).
  • Reliability plot upgrades. plots.plot_reliability renders a KernelReliabilityCurve passed as smooth as a density-weighted variable-width curve with the miscalibration area shaded and an smECE readout; new keyword-only stats (bool | MetricReport) swaps the classic box for n, events, intercept, slope, ICI, smECE, Brier, or the report's values with CIs; new risk_dist ("rug" | "split" | None) selects the rug or a 30-bin spike histogram. The pre-existing default call renders pixel-identically to 0.2.0.
  • Attributes diagram. plots.plot_attributes(y, p, *, method="binned", n_bins=10, scale="probability", sample_weight=None, ax=None): identity, climatology lines, the no-skill line, and the shaded positive-Brier-skill region; method="corp" overlays the CORP PAV fit.
  • Murphy diagrams. metrics.murphy_curve(y, p, *, thresholds=513, sample_weight=None) -> MurphyCurve: the Ehm, Gneiting, Jordan & Krüger (2016) elementary-score curve; 2 * integral equals the Brier score. plots.plot_murphy(curves, *, diff=False, n_boot=200, random_state=42, ax=None); diff=True on exactly two (y, p) pairs draws the pointwise difference with a paired-bootstrap band.
  • sklearn duck typing on the bare core. BaseCalibrator gains __sklearn_is_fitted__ and __sklearn_tags__; Chain, LogitOffset, CalibratedModel gain __sklearn_is_fitted__. On sklearn ≥ 1.6 a bare probcal calibrator works in clone, get_tags, check_is_fitted, and cross_val_score(BetaCalibrator(), s.reshape(-1, 1), y, ...) with no adapter import; contexts needing the (n, 2)/classes_ convention still go through probcal.sklearn. The sklearn import lives inside __sklearn_tags__, so import probcal stays numpy + stdlib.
  • Mirror conformance suite for the sklearn adapter. Every generic estimator check declared inapplicable to a score-level estimator is re-implemented on valid probability data: fit idempotence, no input mutation, pickle and JSON round trips, clone-then-fit equality, subset and order invariance, and integer sample_weight equal to row duplication over all 14 BaseCalibrator subclasses, each at a measured tolerance (exact for centered isotonic, ENIR, isotonic, temperature, Venn–Abers; structural deviations documented for the binning, spline, BBQ, CVAP, and selector classes).

Changed

  • treecf guide: the cross-repo work list is documented as implemented in treecf 0.2.4; the probcal[treecf]/all floor rises to treecf>=0.2.4.
  • CalibrationMonitor.report() computes its post-alarm trailing-window diagnostics on batches from the estimated drift onset onward instead of the full history (recommendation_window="since_onset", the new default); with plug_in_window also set, the window starts at the later of the two. recommendation_window="trailing" restores 0.2.0 behaviour exactly for those diagnostic inputs; onset_label is populated under both.
  • Single-column score input. A (n, 1) column vector is accepted everywhere scores enter the package and ravelled first; results are bit-identical to passing the 1-D array. Every other 2-D shape still raises ("expected 1-D scores (or a single column); got shape ..."); validate_binary_y stays 1-D only.
  • Per-check applicability reasons for the sklearn estimator checks. Each of the 36 declared checks carries its own reason naming the data it generates and which part of the score-level contract it violates; the declared set is unchanged. docs/api-stability.md gains a "scikit-learn estimator checks" section stating the position.
  • guide/sklearn.md rewritten around three tiers: bare duck typing (sklearn ≥ 1.6), SklearnCalibrator (pipelines, VotingClassifier, stacking), CalibratedClassifier (out-of-fold orchestration), plus the prefit recipe via FrozenEstimator. Every code block is included from an executed test file, so the guide cannot drift. CalibratedClassifier.to_json(path=None, *, indent=2) now delegates to calibrator_.to_json(...).

Fixed

  • PlattCalibrator's smoothing prior now reads weighted class mass. The Lin–Lin–Weng targets used raw row counts while the IRLS was weighted, so integer weights did not equal row duplication (2.0e-2 apart). They now use weighted class totals: every unweighted fit, default, and golden file is bit-identical, and weighting a row matches repeating it to 2.5e-16. ScalingBinningCalibrator keeps its 8.5e-2 deviation, now entirely from the equal-mass bins on the Platt output.
  • Sample-weight delivery into CalibratedClassifier's inner fits. fit(X, y, sample_weight=w) raised TypeError when the base estimator's fit had no sample_weight parameter and recovered only through a silent unweighted refit. It now checks first (sklearn's own has_fit_parameter precedent, widened for metadata routing), warns once naming the class when the base cannot take weights, runs the fold fits and the refit unweighted, and fits the calibrator with the weights. With routing on, an undeclared request raises sklearn's UnsetMetadataPassedError instead of dropping the weights. Verified on scikit-learn 1.4.2, 1.6.1, and 1.9.0.

Performance

  • metrics.evaluate's bootstrap loop sorts each replicate by prediction once and shares that order: _math.loess and metrics.ecce gain a keyword-only presorted flag (default False, so direct calls are unchanged); ece, ece_debiased, and mce share one binning pass; ece_sweep's scan reads per-bin sums off prefix sums; and the LOESS anchor evaluation is vectorized in blocks. Point estimates are unchanged bit-for-bit. Bootstrap percentile CI bounds may move in their last bits (at most 3.9e-11 relative on a n=10,000/n_boot=1000 run). The one intended numeric difference (tricube weights cubed by multiplication rather than ** 3) is at most 2.3e-16 relative on a well-conditioned window; the rank-deficient corner is discussed in concepts/metrics.md under "Computational cost". Measured: evaluate(n=10,000, n_boot=1000) over the full catalog fell from 304.2s to 86.9s (3.5×); a full-catalog replicate costs 0.089s, of which the ICI family is 58%, so metrics= excluding ici/e50/e90/emax remains the largest lever. docs/scripts/benchmarks.py gains per-metric rows.

Documentation

  • concepts/conservatism.md consolidated into one chapter for all four conservatism mechanisms, with a "When each mechanism applies" table, a "Margin of conservatism" section with a runnable Chain([calibrator, moc_offset(mon)]) example, and an explicit warning that every bound on the page is a one-period estimator that understates uncertainty under cross-obligor correlation. concepts/offset.md gains "Estimating delta with a standard error". The end-to-end notebook gains a "Conservative margins" section. docs/api-stability.md gains an "Added in 0.3.0" running list.
  • concepts/monitoring.md reorganized in build order: e-process and confidence-sequence theory, per-grade CS, the drift-onset estimate and its since-onset window, apply_recommendation, margin-of-conservatism offsets, the fixed-sample hl_e_test audit, and the simulation tables (per-grade CS coverage 1.000 for both drifted and stable grades; onset localization median error 1 batch). The end-to-end notebook gains a "Monitoring closed loop" section.

0.2.0 - 2026-08-23

Serialization, monitoring, and the sklearn adapter. Every fitted object round-trips through versioned JSON with fingerprints, never pickle, under a golden-file promise that every 0.x release reads schema 1. probcal.monitor watches deployed calibration with an e-process whose alarm keeps its type-I guarantee at every look. probcal.sklearn wraps calibrators as estimators and offers a CalibratedClassifierCV drop-in. Chain composes a calibrator with offsets for recourse engines; probcal.integrations.optbinning calibrates scorecards with exact point cut-offs. Three inverse-map fixes at the probability boundary. The measured comparison page and the end-to-end notebook arrive with this release. smooth_ece values for n <= 8192 may differ from 0.1.3 at the ~1e-4 level (bins=None recovers them).

Details

Fixed

  • Step calibrators' right generalized inverse (IsotonicCalibrator, HistogramBinningCalibrator) returned the next block's left edge, a supremum not in the preimage, so a consumer treating bounds as closed landed one block too high; it now returns the largest float below the boundary, which is attained.
  • point_inverse at the probability boundary: p ∈ {0, 1} was clipped to [1e-12, 1 − 1e-12] before inversion, and a certified root with |z| > logit(1 − 1e-12) ≈ 27.63 was returned as σ(z), which rounds to 0.0/1.0 and cannot round-trip. Both now raise UnattainableTargetError: boundary targets with the offending values named; over-range roots only for space="probability", pointing at space="logit", where the answer is exact.
  • LogitOffset.interval_inverse returned raw bounds below the 1e-12 clip that transform cannot round-trip; bounds beyond the representable output range now collapse to the full-range sentinels (0/1, ±inf) as in BaseCalibrator.interval_inverse, and an interval entirely outside that range raises UnattainableTargetError.

Changed

  • smooth_ece's lattice path engages for every call with a non-degenerate logit range instead of only n > bins, removing the size cliff at typical calibration-set sizes (0.1.3: ~1s per call at n=4000). Path selection: bin at bins (default 8192), accept when sigma* >= 8·width, refine adaptively once, fall back to exact only when refinement is infeasible. Values for n <= bins may differ from the 0.1.2/0.1.3 exact grid at the ~1e-4 level on typical portfolios (measured ≤ 2.4e-4), larger on wide clipped-logit-range data where the lattice value is the better one; bins=None recovers old values bit-for-bit.

Added

  • Versioned JSON serialization on every fitted object: to_dict/from_dict/to_json/from_json with a schema-checked envelope (probcal_schema 1, probcal_version, class name, params, state, fit metadata) and a class registry for dispatch; fingerprint() (SHA-256, version- and timestamp-blind) plus fit_meta["data_fingerprint"] (permutation-invariant hash of the training triple). Covers all calibrators, LogitOffset, CalibratorSelector, and CalibratedModel (which stores a model reference, never the model object; reattach via from_dict(d, model=...); the ensemble flow refuses serialization). Golden files verified in CI. New chapter concepts/serialization.md.
  • CalibratedModel(model_id=...) names the wrapped model in the serialized reference.
  • probcal.monitor (numpy + stdlib): CalibrationMonitor, anytime-valid calibration monitoring by e-processes (alarm at E >= 1/alpha has type-I error <= alpha at every stopping time). Components: offset plug-in with a ±{0.1, 0.25, 0.5, 1.0} mixture, Cox shape plug-in, per-grade offset processes; a time-uniform confidence sequence for the current offset; sticky alarm, anytime p-value, MonitorStep/MonitorReport, a diagnostic re-offset/re-fit recommendation, JSON serialization with bit-exact resume, and plots.plot_e_process. Simulated at 2000 runs × 24 batches × n=2000: worst type-I 0.0275 at alpha 0.05, median detection delay 2 batches for delta=0.4, CS coverage 0.992+. New chapter concepts/monitoring.md.
  • probcal.sklearn (extra probcal[sklearn], scikit-learn ≥ 1.4): SklearnCalibrator, a probcal calibrator as an sklearn estimator over a single score column (transform ends a Pipeline; input="logit" accepts raw margins; the fitted object is calibrator_), and CalibratedClassifier, the CalibratedClassifierCV(ensemble=False) drop-in on probcal calibrators (OOF via cross_val_predict or cv="prefit", one map, full refit; method="decision_function" maps through expit; binary only), which also exposes the probcal calibrator protocol by delegation. Estimator checks run in CI on sklearn 1.4 and latest with inapplicable checks declared. New chapter guide/sklearn.md.
  • probcal.Chain: model-free composition of a fitted calibrator with LogitOffset stages, exposing the full calibrator protocol (exact inverses through every stage, composed affine coefficients, concatenated interpret(), JSON, fingerprints); CalibratedModel.chain_ returns the equivalent chain.
  • probcal.integrations.optbinning (extra probcal[optbinning], ≥ 0.21): calibrate_scorecard → CalibratedScorecard, calibrated PD over an untouched scorecard with points_affine_coeffs_ recovered to machine precision (rounding=True warns and falls back to interval_inverse), an exact masterscale from PD bands to point cut-offs, and provenance JSON binding the calibration layer to the scorecard by fingerprint. New chapters guide/optbinning.md and guide/treecf.md; probcal[treecf] extra.
  • Protocol conformance: every inverse-capable object accepts interval_inverse(lo, hi, space="logit", buffer_logit=...) by keyword with lo=0/hi=1 → ∓inf and buffered-empty refusal; the step-calibrator generalized-inverse contract is spelled out in the inverse-maps chapter.
  • CalibratorSelector is now a BaseCalibrator subclass: same surface plus inherited get_params/set_params, serialization, and inverse maps delegating to the refitted winner; is_monotone_ mirrors the winner.
  • ENIRCalibrator.fit emits a single UserWarning above 50,000 unique scores stating the expected fit minutes.
  • README "Calibrators at a glance" table: one-line scaling note per method.

Documentation

  • End-to-end notebook on a rare-event portfolio (docs/notebooks/pd_end_to_end.ipynb): GBM baseline → logit-scale reliability → selection with CIs → Jeffreys backtests → auditable offset → policy-to-raw translation and a treecf counterfactual → monitoring with an injected drift → JSON round-trips. Home Credit via kagglehub with a documented synthetic fallback that executes in CI.
  • Comparison benchmark (docs/benchmarks/comparison.md, docs/scripts/comparison.py under probcal[bench]): probcal vs sklearn sigmoid/isotonic, netcal BBQ/ENIR/beta, and betacal on five OpenML datasets at 1.5–30% event rates, with bootstrap CIs, Jeffreys grade pass rates, and fit times, including where probcal loses.
  • Docs restructured into Guide, Concepts, Tutorials, Benchmarks, and API reference, plus docs/api-stability.md (public surface, deprecation policy, support matrix). The internal decision log is not published.
  • README front page rewritten around the wedge with the end-to-end notebook above the fold.

Performance

  • smooth_ece measures 1.0–3.6ms for n ∈ [64, 8192], 3–6ms at n=10⁴–10⁵, and ~43ms at n=10⁶. evaluate(n=6000, n_boot=1000) measures 198.9s (0.1.3: ≈10min).

0.1.3 - 2026-08-16

The performance release. The ICI family fits its LOESS smoother at 512 anchors instead of every observation (ici at n=50,000: 192s → 1.2s), smooth_ece smooths a pre-binned measure (seconds → milliseconds), and ENIR's memory is bounded at any size. grid_size=None and bins=None recover the exact pre-0.1.3 values and cost. point_inverse gives an exact single-point preimage for affine and beta maps. evaluate's bootstrap now stratifies by class (stratify=False restores i.i.d.), and four silent defects in weighting, bootstrap degeneracy, and selector parsimony are fixed.

Details

Fixed

  • ENIRCalibrator could exhaust memory well before m=50,000 because path_solutions_ retained every breakpoint's full-length solution; the path solver now retains only the lowest-BIC max_solutions (default 256) breakpoints. A pruning bound that crashed on sub-unit total sample_weight is now disabled in that case instead of raising.
  • sample_weight was accepted but silently had no effect on e50/e90 and on reliability_summary's internal e90; both now honor weights via a Hazen-position weighted_quantile.
  • evaluate's bootstrap silently substituted the point estimate for degenerate single-class resamples, narrowing the reported CI; a degenerate draw is now redrawn (up to 100 times, then RuntimeError).
  • CalibratorSelector's parsimony tie-break ranked every user-supplied candidate last regardless of complexity; it now reads complexity_rank off the candidate instance.
  • smooth_ece's binned path evaluated the smoothed measure on the exact path's 257-point grid, which aliased against the bin lattice and tripped the small-bandwidth guard on every call, so every reported value was correct but cost ~5.7–6.1s regardless of n. The binned path now evaluates natively on its own lattice with one adaptively sized refinement; measured n=10⁴: ~5.7s → ~3ms, n=10⁵: 19.7s → ~10ms, same fixed point.

Changed

  • ici/e50/e90/emax and reliability_summary fit their LOESS smoother at 512 equal-mass anchors by default (loess's grid_size, interpolated between anchors, the device of R's stats::lowess delta); measured |Δici| ≈ 1.3e-6 on make_pd_portfolio(n=5000). The window search differs from the old core only at exact distance ties.
  • smooth_ece smooths a pre-binned (default bins=8192 equal-width logit bins) residual measure, with a small-bandwidth guard that refines once and then falls back to exact.
  • ENIRCalibrator.path_solutions_ is now shaped (K, m) over kept_breakpoints_; path_lambdas_ still records every breakpoint.
  • evaluate's bootstrap resamples negative and positive classes separately by default (stratify=True); stratify=False restores i.i.d. resampling.
  • VennAbersCalibrator predicts from precomputed F0_/F1_ sweeps fit once instead of refitting two PAVA passes per query.

Added

  • grid_size= on loess/ici/e50/e90/emax/reliability_summary and bins= on smooth_ece; None recovers the exact pre-0.1.3 values and cost bit-for-bit.
  • evaluate(metrics=): bootstrap only the requested subset of the catalog; unknown names raise ValueError.
  • docs/scripts/benchmarks.py: deterministic wall-time benchmarks for ici, smooth_ece, evaluate, ENIR, IVAP, and the selector, with matching slow-marked regression ceilings.
  • ENIRCalibrator(max_solutions=) (default 256), plus dropped_weight_ and kept_breakpoints_.
  • VennAbersCalibrator fitted attributes F0_/F1_; evaluate(stratify=); probcal._math.weighted_quantile; BaseCalibrator.complexity_rank (default 100.0), overridden by every built-in.
  • point_inverse(p, *, space): an exact single-point preimage alongside interval_inverse. On BaseCalibrator, any affine-logit map (PlattCalibrator, TemperatureCalibrator, BetaCalibrator's "a"/"ab" variants) in closed form; BetaCalibrator overrides it for all three variants including the non-affine "abm" map, via a seed refined by up to 4 certified Halley steps; LogitOffset in closed form. Non-affine monotone and step calibrators raise NotImplementedError naming interval_inverse.

Performance

  • ici at n=50,000: 192.2s (0.1.2) → 1.2s. loess(grid_size=512) fits n=1,000,000 points in under 30s.
  • ENIR fit at m=10,000: 68.6s / 783MB → 1.6s / 40.8MB peak. IVAP fit at n=100,000: 0.91s; predict_interval at m=100,000: 0.016s. CalibratorSelector()'s default menu: 34.5s at n=4,000 (0.1.2) → 7.9s at n=100,000.
  • Corrected attribution: evaluate's pre-fix cost was the smooth_ece aliasing defect above, not ece_sweep. Post-fix, ici is the largest single per-call contributor (0.24s at n=10⁴); evaluate(n_boot=100) at n=10⁴ measures ~44s. metrics= subsetting is the lever.

0.1.2 - 2026-08-12

Platt and beta fits on wide-score data no longer hit a false "separation" abort, and both calibrators now surface their IRLS convergence status instead of swallowing it.

Details

Fixed

  • irls_logistic no longer aborts at max|eta| > 30, a false separation that biased Platt/beta fits on wide-score data (z ~ N(0, 8) with true slope 1.5: 0.1.1 returned a ≈ 1.18 plus a spurious warning). Newton now step-halves on an overflow-safe softplus objective; separation is detected only for the quasi-separation signature; the ridge fallback converges. calibration_belt may now select higher polynomial degrees on wide-score data where the old cap silently truncated the search.
  • PlattCalibrator and BetaCalibrator no longer swallow IRLS convergence status: an unconverged fit raises a distinct UserWarning and is recorded by interpret(); calibration_belt stops forward degree extension at a separated fit.

Added

  • IrlsResult.nll, the final penalized objective value.
  • converged_ on PlattCalibrator/BetaCalibrator and separation_fallback_ on BetaCalibrator, with matching interpret() audit lines; docs section "Separation, steep maps, and convergence".

0.1.1 - 2026-08-08

The kernel calibration error and its test (SKCE, after Widmann et al. 2019) join the metric catalog as report-only quantities. Four plots arrive: the annotated reliability diagram with a stats box and rug, ECCE drift walks, the per-grade traffic-light chart, and the offset audit chart, all under a per-call house style that never touches global matplotlib settings.

Details

Added

  • probcal.metrics.kernel: skce (estimators uq/ul/biased, Laplacian/Gaussian kernels, deterministic median-heuristic bandwidth, probability/logit kernel scale) and skce_test (Arcones–Giné centered bootstrap and the O(n) asymptotic-normal method, one-sided, with the distribution-free p_value_bound). Report-only: excluded from evaluate() and the selector.
  • Documentation: SKCE section and selection-table row in the metrics chapter, the kernel API page, and the namesake-disambiguation FAQ entry (the unaffiliated R package probcal and the ECAI 2025 research codebase).
  • Tooling: mypy targets Python 3.12 (numpy ≥ 2 stubs use PEP 695 syntax); runtime 3.11 support unchanged.
  • Visualization: annotated plot_reliability (stats box via the new metrics.reliability_summary, deterministic rug thinned to ≤ 1000 marks per class), plot_ecce cumulative-drift walks with the pointwise ±2 SD envelope (on the new curves.ecce_curve), plot_grade_backtest traffic-light chart with 90% display intervals (new ci_low/ci_high fields on both grade results, powered by the numpy-only _math.beta_ppf), and plot_offset_audit. docs/scripts/generate_figures.py regenerates every documentation figure deterministically.

Changed

  • All plots style themselves via a per-call rc_context house style; global matplotlib rcParams are never touched.
  • plot_reliability: the twin-axis count-bar margin is opt-in (counts=False default); the rug is the density view.
  • All documentation figures regenerated under the house style.

0.1.0 - 2026-08-07

First public release on PyPI: thirteen binary calibrators from Platt to Venn–Abers and spline, the full metric catalog with bootstrap CIs and per-grade backtests, exact inverse maps from a policy PD back to a raw score, the auditable LogitOffset stage, SHAP attribution repair, CalibratedModel prefit and cross-validation flows, CalibratorSelector under nested validation, and a four-chunk theory guidebook.

Details

Fixed

  • reliability_binned: Wilson interval bounds are forced to contain the point estimate (floating-point noise at zero-event bins could push ci_low above a 0.0 event rate).
  • plot_reliability(scale="logit"): bins with an event rate of exactly 0 or 1 are omitted from the point layer instead of rendering at the clipping floor; they remain visible in the count margin.

Added

  • Packaging and CI: PEP 639 license metadata, Python 3.13 classifier, beta status, a tag-vs-version check in the publish workflow, and a CI matrix on Python 3.11/3.12/3.13.
  • Documentation: generated figures in the visualization chapter, "In probcal" snippets in every concept chapter, a probcal-vs-netcal note in the FAQ, README quickstart with printed output and a feature matrix, docs cross-linking.
  • Datasets and tutorial: make_pd_portfolio (beta-family generative miscalibration with exact event-rate anchoring), the executed pd_calibration_walkthrough.ipynb, getting-started.md, how-it-works.md, the three-page API reference, and the FAQ.
  • Automatic selection: CalibratorSelector, default 8-candidate menu, inner stratified seeded K-fold with structurally out-of-fold scoring, criteria log_loss/brier/ici/smooth_ece/ece_sweep (plain ECE and Hosmer–Lemeshow refused), guardrail flags, one-standard-error parsimony tie-break, winner refit, ranked SelectionReport.
  • Wrapper flows: CalibratedModel, prefit flow and cv flow with duck-typed cloning and stratified seeded folds, ensemble=False pooled default vs ensemble=True fold averaging, offset_to appending inspectable LogitOffset stages, composed interval_inverse and affine_logit_coeffs_, predict_proba/predict_proba_2d.
  • Inverse maps and thresholds: interval_inverse(lo, hi, *, space, buffer_logit) across the catalog (closed forms for Platt, temperature, LogitOffset; block-structure search for isotonic, CIR, histogram, scaling-binning; monotone bisection for beta, spline, Venn–Abers); UnattainableTargetError instead of silent clamping; non-monotone calibrators refuse with NotImplementedError; thresholds.calibrated_interval_to_raw and the masterscale calibrated_bands_to_raw.
  • Attribution adjustment: adjust_attributions → AdjustedAttribution, affine-exact mode (exact composed Shapley values via affine_logit_coeffs_) and Aumann–Shapley mode (exact additivity for any calibrator), logit and probability scales, shap.Explanation duck-typing without a shap import.
  • Logit offset: LogitOffset with explicit-delta and target-mean modes, audit trail (delta_, pre_mean_, post_mean_, timestamp), audit_report() with pre/post guardrails, interpret() with odds-factor and central-tendency readings.
  • Curves, belt, and plots: reliability_binned (Wilson CIs, both scales), reliability_loess, reliability_spline, calibration_belt (GiViTI-style, forward degree selection ≤ 4, pointwise bands, identity-test p-value); plots.py ([viz]-guarded): plot_reliability with logit-scale probability-labeled ticks, plot_belt, plot_comparison, plot_interval, plot_selection.
  • Metrics catalog: proper scores (log_loss, brier_score, brier_skill_score, Murphy decomposition, log-loss calibration/refinement split), binned estimators (ece l1/l2/max, ece_debiased, ece_sweep, adaptive_ece, hosmer_lemeshow), binning-free estimators (smooth_ece, ecce, ici/e50/e90/emax, spiegelhalter_z), the recalibration-regression framework (calibration_intercept/_slope/_test, calibration_guardrails), per-grade backtests (binomial_grade_test, jeffreys_grade_test with traffic lights), and evaluate() with seeded bootstrap percentile CIs.
  • Calibrators: PlattCalibrator (Lin–Lin–Weng smoothing), TemperatureCalibrator, BetaCalibrator (variants abm/ab/a, constraint_active_), IsotonicCalibrator (PAVA, tie pooling, block-structure attributes), CenteredIsotonicCalibrator, HistogramBinningCalibrator (equal-mass/equal-width, Jeffreys shrinkage), ScalingBinningCalibrator, BBQCalibrator, ENIRCalibrator (is_monotone_ = False), VennAbersCalibrator (IVAP with predict_interval()), CrossVennAbersCalibrator, SplineCalibrator (natural cubic basis on the logit scale, penalized IRLS, λ by CV, monotonicity check); BaseCalibrator with fit/predict_proba/interpret, get_params/set_params, is_monotone_, affine_logit_coeffs_; public logit/expit.
  • Numerical core: overflow-safe logit/expit, weighted PAVA, IRLS logistic regression with ridge stabilization and separation detection, safeguarded 1-D Newton and bisection, regularized incomplete beta and gamma, chi-square and normal quantiles, tricube LOESS, natural cubic spline basis; validation with 1e-12 clipping; frozen result dataclasses with as_dict(). Reference tests against scipy/scikit-learn/statsmodels.
  • Repository scaffold: src layout, uv/ruff/black/mypy/pytest, CI workflows (docs deploy, tag-gated PyPI publish), mkdocs-material + MathJax, MIT license, citation metadata; all 12 flagged references verified against primary sources.
  • Theory guidebook (~16,400 words): concepts/why-calibration.md, methods-parametric.md, methods-nonparametric.md, methods-distribution-free.md, metrics.md, data-splitting.md, offset.md, shap-calibration.md, inverse-maps.md, auto-selection.md, visualization.md.