Choose a calibrator¶

Data these snippets assume

Blocks on this page use a fixed set of names for held-out data instead of re-deriving it each time. To run them, define the names first:

# docs: no-run — the definitions the snippets on this page assume
import numpy as np
from probcal import BetaCalibrator, Masterscale, make_pd_portfolio
from probcal.monitor import CalibrationMonitor

cal_set, new_set = make_pd_portfolio(n=3000, random_state=0), make_pd_portfolio(n=1000, random_state=1)
s_cal, y_cal = cal_set.scores, cal_set.y          # held-out calibration scores and outcomes
w_cal = np.ones_like(y_cal)                       # uniform sample weights
s_new = new_set.scores                            # scores of new obligors
ms = Masterscale.from_edges([0.01, 0.05], names=["G1", "G2", "G3"])   # the masterscale
p_cal = BetaCalibrator().fit(s_cal, y_cal).predict_proba(s_cal)         # calibrated PDs
grades = ms.assign(p_cal)                                              # rating labels
segments = np.array(["seg-a", "seg-b", "seg-c"])[np.arange(len(s_cal)) % 3]  # segment labels
model = ...   # any object with predict_proba(X); the docs use a stub that reads X[:, 0]
mon = CalibrationMonitor(alpha=0.05)              # with a few batches of a calibrated forecast applied

Every block also names, in its first comment line, which of these it uses.

The catalog, as a table you can decide from. Every column that is a property of the code (monotonicity, which inverse exists, what serializes) is read off the fitted object and pinned by a test, so a column that stops being true breaks a build rather than misleading a reader.

The catalog is two tables sharing the class column: what each object is (parameters, monotonicity, inverses, serialization, fit cost), then when to reach for it. Two fit-cost entries change decisions rather than schedules: ENIR is quadratic in unique scores (its fit warns above ~50,000) and Venn–Abers stores its calibration set, so its JSON grows with n.

The catalog¶

Properties¶

Class	Fitted parameters	Monotone?	Exact `point_inverse`?	`interval_inverse`?	Serialization	Fit cost
`TemperatureCalibrator`	`T_` (one)	by construction (`T > 0` always)	yes: affine, `(1/T, 0)`	yes, closed form	JSON, golden-pinned	O(n) per IRLS iteration
`PlattCalibrator`	`a_`, `b_`, `converged_`	checked at fit (`is_monotone_ = a_ > 0`)	yes: affine, `(a_, b_)`	yes, closed form	JSON, golden-pinned	O(n) per IRLS iteration
`BetaCalibrator` (`"abm"`)	`a_`, `b_`, `c_`, `constraint_active_`	by construction (`a_, b_ >= 0` enforced)	yes: its own exact construction (seed + certified Halley)	yes, bisection	JSON, golden-pinned	O(n) per IRLS iteration
`BetaCalibrator(variant="ab"/"a")`	same attributes, tied exponents	by construction	yes: affine (`affine_logit_coeffs_` is not `None`)	yes	JSON, golden-pinned	O(n) per IRLS iteration
`IsotonicCalibrator`	`block_mean_`, `block_first_s_`, `block_last_s_`, `block_center_s_`, `n_blocks_`	by construction (PAVA)	no: `NotImplementedError` naming `interval_inverse`	yes, `searchsorted` on blocks; plateau semantics	JSON, golden-pinned	O(n log n) fit (sort + PAVA)
`CenteredIsotonicCalibrator`	as isotonic (interpolates between block centers)	by construction	no	yes: a point preimage, not a plateau	JSON, golden-pinned	O(n log n) fit
`HistogramBinningCalibrator`	`edges_`, `bin_rate_`, `bin_weight_`, `is_monotone_`	checked at fit: bin rates need not increase, and often do not	no	only when the fit came out monotone; otherwise `NotImplementedError`	JSON, golden-pinned	O(n log n) fit
`ScalingBinningCalibrator`	`platt_` (nested), `edges_`, `bin_value_`	declared (class default; the bin values inherit the Platt stage's order and are not re-checked)	no	yes: pulls the bin edge back through the Platt stage	JSON, golden-pinned (nested Platt envelope)	O(n log n) fit
`BBQCalibrator`	`bins_grid_`, `weights_`, `is_monotone_`	checked at fit on a probe grid	no	only when the fit came out monotone	JSON, golden-pinned	O(n log n) fit per candidate binning
`ENIRCalibrator`	`path_lambdas_`, `path_solutions_`, `kept_breakpoints_`, `weights_`, `dropped_weight_`	never: `is_monotone_ = False` at class level	no	no: `NotImplementedError`, the preimage may be a union of intervals	JSON, golden-pinned	quadratic in unique scores; intended for m ≲ 50,000 (`fit` warns above)
`SplineCalibrator`	`n_knots_`, `lambda_`, `edof_`, spline coefficients, `is_monotone_`	checked at fit on a probe grid (warns when it fails)	no	yes when monotone, by bisection	JSON, golden-pinned	O(n · k) per IRLS iteration (k knots), repeated over the inner CV for `lambda`
`VennAbersCalibrator` (IVAP)	the sorted calibration set plus both sweeps `F0_`, `F1_`	by construction	no	yes, by bisection	JSON, golden-pinned, but O(n): the calibration set is the fitted map	O(n log n) fit, O(log n) per prediction
`CrossVennAbersCalibrator` (CVAP)	`_ivaps`, one fitted IVAP per fold	by construction	no	yes, inherited bisection	JSON, golden-pinned; O(n) for the same reason	one IVAP fit per fold
`SegmentedCalibrator`	`base_` (nested), `segments_`, `delta_hat_`, `se_`, `delta_tilde_`, `shrink_`, `tau2_`	inherited from `base_`	yes whenever `base_` has one (default base: beta), taking `segment=`	yes, taking `segment=`, inverting through `base_` plus that segment's shrunk offset	JSON, golden-pinned (nested base envelope)	base cost, plus O(n) for the per-segment offsets
`CalibratorSelector`	`best_name_`, `best_calibrator_` (nested), `report_`, `is_monotone_`	inherited from the winner	no: the selector exposes no affine coefficients even when the winner is affine; reach for `sel.best_calibrator_.point_inverse`	yes, by bisection through the winner	JSON, golden-pinned (winner envelope plus the ranked report)	every candidate fitted K times (inner CV), plus the winner's refit

Guidance¶

Class	Data appetite	Use when	Avoid when
`TemperatureCalibrator`	~a dozen events	Confidence too soft or too sharp with the base rate already right	cannot move the level at all: `s = 0.5` is a fixed point
`PlattCalibrator`	a few dozen events	Level and slope both off, and you want two auditable numbers	curvature that a straight logit line cannot follow
`BetaCalibrator` (`"abm"`)	a few dozen events	Asymmetric tail distortion, the low-PD default	nothing to gain when `a_ ≈ b_`, where the tie-break drops you to Platt anyway
`BetaCalibrator(variant="ab"/"a")`	a few dozen events	You want the beta family with fewer parameters to defend	the tail asymmetry is the whole problem
`IsotonicCalibrator`	hundreds of events	Visible curvature and events to fund it	recourse downstream, where plateaus and moving block edges make counterfactuals fragile
`CenteredIsotonicCalibrator`	hundreds of events	Isotonic's fit without its step function, the nonparametric choice when recourse is in scope	very few blocks, where interpolation barely differs
`HistogramBinningCalibrator`	hundreds of events, per bin	Grade-shaped output that mirrors a masterscale	thin bins: a non-monotone fit also loses every inverse
`ScalingBinningCalibrator`	between Platt's and binning's: a Platt fit plus a bin count	You want binned output whose calibration error is measurable at `O(1/ε² + B)` samples	you need a continuous map
`BBQCalibrator`	hundreds of events; strong on small samples (see the benchmarks)	Small portfolio where a single binning is a coin flip	thresholding or recourse, where a non-monotone average refuses to invert
`ENIRCalibrator`	hundreds of events; quadratic in unique scores	Pure predictive accuracy with no downstream inversion	anything on this page's cutoff and recourse paths
`SplineCalibrator`	hundreds of events (a penalized basis plus CV over `lambda`)	Smooth curvature you want to show a validator as a curve, not a staircase	small samples, where the penalty is chosen on very little data
`VennAbersCalibrator` (IVAP)	hundreds of events	You need a distribution-free interval, not just a number	file size and load time matter
`CrossVennAbersCalibrator` (CVAP)	hundreds of events, spent across folds instead of carved out	The calibration sample is too small to carve and you still want the VA guarantee	the fold ensemble is harder to describe in a model document
`SegmentedCalibrator`	the base's appetite, plus ~a dozen events per segment	Segments that genuinely differ in level and a shrinkage story you can defend	segments that differ in slope, which is a re-fit, not an offset
`CalibratorSelector`	enough for an inner stratified K-fold (default `cv=5`)	You want the choice made on out-of-fold evidence and written down	you already know the answer and the extra machinery only adds a layer to explain

Three readings of the monotone column, and the difference matters: by construction means the fit cannot produce a decreasing map; checked at fit means is_monotone_ is derived from this fit's parameters, so the same class can invert on one portfolio and refuse on the next; declared means the class default stands and nothing re-derives it. Only is_monotone_ = True buys you interval_inverse; everything else raises NotImplementedError, which is the whole reason the column is in the table. LogitOffset is not a calibrator and so is not a row here, but it carries the same protocol (closed-form both inverses) and composes onto any of the above through Chain, which is not a row either, having no fit of its own to choose: it inherits every column from the stages you put in it.

Count events, not rows: "hundreds of events" means events, not rows, and 500 observations at a 3% base rate is fifteen. The reasoning is in Data splitting; head-to-head evidence at event rates from 1.5% to 30% is in Benchmarks.

Reading the columns off your own fit¶

The table is a summary of what the objects report. On your data, ask them:

# s_cal, y_cal: held-out calibration scores and outcomes
from probcal import BetaCalibrator, HistogramBinningCalibrator

for cal in (BetaCalibrator().fit(s_cal, y_cal),
            HistogramBinningCalibrator(n_bins=10).fit(s_cal, y_cal)):
    print(type(cal).__name__,
          "monotone:", cal.is_monotone_,
          "affine:", cal.affine_logit_coeffs_ is not None,
          "rank:", cal.complexity_rank)

A binning fit that prints monotone: False has just told you it cannot serve a cutoff or a counterfactual on this portfolio, before anyone builds a policy on it. That check belongs in the same script as the fit.

The decision path¶

Start from the diagnosis, not the catalog. A reliability curve on the logit scale plus the guardrails (slope, intercept, Spiegelhalter) says which failure you have; Why calibration and Metrics and tests cover the reading.

Pure level error, and few events. One parameter is all the data can fund: LogitOffset, anchored to a long-run central tendency. This is the disciplined retreat, not a failure.
Slope error. The parametric family: temperature if the base rate is already right, Platt when the level moved too.
Curvature, with events in the hundreds. Isotonic, CIR, or the spline. Prefer CIR or the spline where a cutoff or recourse consumes the output: step functions turn a decision boundary into a plateau edge that moves at every refit (see Set cutoffs).
You need an interval, not a point. Venn–Abers: predict_interval() carries a distribution-free guarantee under exchangeability; CVAP when the sample is too small to carve a calibration split.
The output feeds a masterscale or grade table. Histogram binning or scaling-binning produce grade-shaped output natively, but check is_monotone_ before promising anyone a grade-to-score translation.
Unsure, the usual case. Hand the decision to CalibratorSelector, which scores every candidate on out-of-fold data only and writes down the contest. On a small portfolio give it a restricted menu: every extra candidate spends selection power.

# s_cal, y_cal: held-out calibration scores, outcomes
from probcal import (
    BetaCalibrator,
    CalibratorSelector,
    PlattCalibrator,
    TemperatureCalibrator,
)

sel = CalibratorSelector(
    candidates={"temperature": TemperatureCalibrator(),
                "platt": PlattCalibrator(),
                "beta": BetaCalibrator()},
    scoring="log_loss",
    cv=5,
).fit(s_cal, y_cal)

print(sel.best_name_)
print(sel.report_)          # ranked table, fold sd, guardrail flags
print(sel.interpret())      # the winner's parameters, in words

Read the runner-up's margin in units of the printed fold standard deviations: inside one, the contest was a tie and parsimony already adjudicated it through complexity_rank.

What the choice commits you to¶

Picking a calibrator picks a downstream contract, which is why the inverse columns sit in the same table as the fit quality:

Cutoffs, masterscale bands, and counterfactual targets all run through the inverse: the how-to is Set cutoffs and invert maps, the theory Inverse maps. A masterscale is a Masterscale, one object for assignment, translation, and the report.
Whatever you fit, its parameters, provenance, and reproduction are evidence you can hand to a validator: see Auditability and Build a validation report.
After deployment the map is watched, not assumed: Monitor and act.