API: calibrators¶
base
¶
BaseCalibrator: the common fit / predict_proba / interpret contract.
UnattainableTargetError
¶
Bases: ValueError
The requested calibrated target is unattainable.
Raised instead of silently clamping when an interval does
not intersect the calibrator's output range (or was emptied by
buffer_logit), when a point-inverse target lies outside the open
interval (0, 1), or when a probability-space point-inverse result
would round to 0.0/1.0 (raw logit beyond logit(1 - 1e-12)).
BaseCalibrator
¶
Bases: JsonIO, ABC
Common contract for all probcal calibrators.
Subclasses implement _fit (estimation on validated arrays),
_predict (the fitted map on clipped scores), and interpret.
Everything else — validation, sklearn-style parameter handling without an
sklearn import, the 2-D probability helper — lives here.
| ATTRIBUTE | DESCRIPTION |
|---|---|
is_monotone_ |
Whether the fitted map is guaranteed non-decreasing. Class-level
default
TYPE:
|
fitted_ |
Set by :meth:
TYPE:
|
affine_logit_coeffs_
property
¶
affine_logit_coeffs_: tuple[float, float] | None
Coefficients (a, b) of logit g(s) = a * logit(s) + b, if affine.
None for calibrators that are not affine on the logit scale.
Consumed by the attribution adjustment.
complexity_rank
property
¶
complexity_rank: float
Parsimony rank for selector tie-breaks; lower wins a tie.
Default 100.0 means "unknown — override in subclasses". Custom calibrators declare their place in the tie-break by overriding this property.
fit
¶
fit(s: object, y: object, sample_weight: object = None) -> Self
Fit the calibration map on scores and binary outcomes.
| PARAMETER | DESCRIPTION |
|---|---|
s
|
Raw scores/probabilities in
TYPE:
|
y
|
Binary outcomes in
TYPE:
|
sample_weight
|
Positive observation weights.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
Self
|
The fitted calibrator. |
Source code in src/probcal/base.py
194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 | |
predict_proba
¶
predict_proba(s: object) -> ndarray
Calibrated probabilities P(y = 1) for new scores.
| PARAMETER | DESCRIPTION |
|---|---|
s
|
Raw scores/probabilities in
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
numpy.ndarray of shape (n,)
|
Calibrated probabilities. |
Source code in src/probcal/base.py
237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 | |
__sklearn_is_fitted__
¶
__sklearn_is_fitted__() -> bool
Fitted state for sklearn's check_is_fitted (sklearn >= 1.6).
| RETURNS | DESCRIPTION |
|---|---|
bool
|
|
Source code in src/probcal/base.py
263 264 265 266 267 268 269 270 271 | |
__sklearn_tags__
¶
__sklearn_tags__() -> Tags
Estimator tags for sklearn >= 1.6, built from the public constructors.
sklearn is imported inside the body, never at module or class level:
import probcal stays numpy-only and this hook costs nothing until
sklearn itself calls it. Only the fields that are actually true of a
calibrator are set — it takes 1-D scores, not a 2-D feature matrix,
it is neither a classifier nor a regressor, and it must be fitted.
| RETURNS | DESCRIPTION |
|---|---|
Tags
|
The tag object sklearn's |
| RAISES | DESCRIPTION |
|---|---|
ImportError
|
If sklearn is not installed (only reachable by calling the hook by hand). |
Source code in src/probcal/base.py
273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 | |
interpret
abstractmethod
¶
interpret() -> Interpretation
Fitted parameters with a plain-language, domain-aware reading.
Source code in src/probcal/base.py
304 305 306 | |
interval_inverse
¶
interval_inverse(lo: float, hi: float, *, space: str = 'probability', buffer_logit: float = 0.0) -> tuple[float, float]
Generalized-inverse preimage (raw_lo, raw_hi) of a calibrated interval.
For a non-decreasing fitted map g:
raw_lo = inf{s : g(s) >= lo} and raw_hi = sup{s : g(s) <= hi}.
| PARAMETER | DESCRIPTION |
|---|---|
lo
|
Calibrated-probability bounds;
TYPE:
|
hi
|
Calibrated-probability bounds;
TYPE:
|
space
|
Scale of the returned raw bounds.
TYPE:
|
buffer_logit
|
Shrink the calibrated interval by this margin in logit space before inverting — robustness against future recalibration drift (a central-tendency update of magnitude <= buffer cannot invalidate the result).
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
tuple of float
|
|
| RAISES | DESCRIPTION |
|---|---|
UnattainableTargetError
|
If the (buffered) interval does not intersect the output range — never silently clamped. |
NotImplementedError
|
For non-monotone calibrators ( |
Source code in src/probcal/base.py
326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 | |
point_inverse
¶
point_inverse(p: object, *, space: str = 'probability') -> ndarray
Raw scores whose calibrated probabilities equal p (exact preimage).
Defined only for strictly monotone calibrators with an exact
inverse: affine-logit maps (logit g(s) = a * logit(s) + b)
invert in closed form here, covering Platt scaling, temperature
scaling, and the tied Beta variants ("a", "ab");
BetaCalibrator overrides this method with its own exact
construction for the full "abm" variant. Others
raise NotImplementedError and should use :meth:interval_inverse
instead.
| PARAMETER | DESCRIPTION |
|---|---|
p
|
Calibrated probabilities strictly inside
TYPE:
|
space
|
Scale of the returned raw values.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
ndarray
|
Raw scores (or logits, if |
| RAISES | DESCRIPTION |
|---|---|
RuntimeError
|
If not yet fitted. |
ValueError
|
If |
NotImplementedError
|
If the calibrator is not monotone ( |
UnattainableTargetError
|
If any element of |
Source code in src/probcal/base.py
392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 | |
get_params
¶
get_params(deep: bool = True) -> dict[str, object]
Constructor parameters as a dict (manual sklearn-compatible clone info).
Source code in src/probcal/base.py
513 514 515 516 517 518 519 520 | |
set_params
¶
set_params(**params: object) -> Self
Set constructor parameters; unknown names raise ValueError.
Source code in src/probcal/base.py
522 523 524 525 526 527 528 529 530 531 | |
to_dict
¶
to_dict() -> dict[str, object]
Versioned JSON-native snapshot of the fitted object.
| RETURNS | DESCRIPTION |
|---|---|
dict
|
|
| RAISES | DESCRIPTION |
|---|---|
RuntimeError
|
If not yet fitted. |
Source code in src/probcal/base.py
556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 | |
from_dict
classmethod
¶
from_dict(d: dict) -> BaseCalibrator
Rebuild a fitted object from :meth:to_dict output.
Called on :class:BaseCalibrator itself, dispatches through the
class registry to whatever class wrote d; called on a subclass,
requires d["class"] to match that subclass.
| PARAMETER | DESCRIPTION |
|---|---|
d
|
Output of :meth:
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
BaseCalibrator
|
A fitted instance. |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If the schema version is unknown (naming the writing version),
the class is not registered, or |
Source code in src/probcal/base.py
585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 | |
shrink_interval
¶
shrink_interval(lo: float, hi: float, buffer_logit: float) -> tuple[float, float]
Shrink [lo, hi] by buffer_logit on the logit scale (0/1 ends stay put).
Shared by every interval_inverse: robustness against future
recalibration drift of magnitude <= buffer_logit.
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If |
UnattainableTargetError
|
If the buffer empties the interval. |
Source code in src/probcal/base.py
89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 | |
validate_interval
¶
validate_interval(lo: float, hi: float) -> None
0 <= lo <= hi <= 1.
Source code in src/probcal/base.py
118 119 120 121 | |
clone_unfitted
¶
clone_unfitted(obj: object) -> object
A fresh, unfitted copy built from obj.get_params(deep=False).
Parameters that are themselves estimators (anything with
get_params) — directly or inside a list, tuple, or dict, such as a
:class:probcal.Chain's stages or a selector's candidates — are cloned
recursively, so the copy never shares a sub-object with obj.
Source code in src/probcal/base.py
135 136 137 138 139 140 141 142 143 144 145 146 147 | |
parametric
¶
Parametric calibrators: Platt, temperature, and beta calibration.
Theory, derivations, and parameter interpretation: docs/concepts/methods-parametric.md.
References
Platt (1999); Lin, Lin & Weng (2007); Guo et al. (2017); Kull, Silva Filho & Flach (2017, AISTATS and EJS) — full records in the documentation.
PlattCalibrator
¶
Bases: BaseCalibrator
Logistic recalibration on the logit scale (Platt scaling).
Fits logit g(s) = a * logit(s) + b by IRLS with Lin–Lin–Weng smoothed
targets (N+ + 1)/(N+ + 2) and 1/(N- + 2) for stability on small
samples, where N+/N- are the weighted class masses (row counts
under unit weights), so that integer weights match row duplication. The
identity map is (a, b) = (1, 0).
| ATTRIBUTE | DESCRIPTION |
|---|---|
a_ |
Fitted slope — spread correction:
TYPE:
|
b_ |
Fitted intercept — calibration-in-the-large shift in log-odds.
TYPE:
|
converged_ |
Whether IRLS converged; if
TYPE:
|
References
Platt (1999); Lin, Lin & Weng (2007). The logistic family fitted on raw SVM outputs (Platt's original setting) does not contain the identity; on logits it does — see the parametric-methods chapter.
affine_logit_coeffs_
property
¶
affine_logit_coeffs_: tuple[float, float] | None
(a, b): Platt scaling is affine on the logit scale.
complexity_rank
property
¶
complexity_rank: float
Parsimony rank 2.0: a two-parameter map, simpler than the nonparametric methods.
interpret
¶
interpret() -> Interpretation
Read the fitted slope and intercept against the identity (1, 0).
If IRLS did not converge at fit time (a UserWarning was raised),
the messages include a note not to trust the coefficients.
Source code in src/probcal/parametric.py
189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 | |
TemperatureCalibrator
¶
Bases: BaseCalibrator
Temperature scaling: g(s) = sigma(logit(s) / T).
T minimizes the calibration-set negative log-likelihood: the score in
u = 1/T is monotone, so its root is found by bisection on
u in [1e-6, 1e6]; without an interior minimum the nearer boundary is
used and a UserWarning is raised.
| ATTRIBUTE | DESCRIPTION |
|---|---|
T_ |
Fitted temperature.
TYPE:
|
References
Guo, Pleiss, Sun & Weinberger (2017).
affine_logit_coeffs_
property
¶
affine_logit_coeffs_: tuple[float, float] | None
(1/T, 0): temperature scaling is affine on the logit scale.
complexity_rank
property
¶
complexity_rank: float
Parsimony rank 1.0: the simplest map, a single parameter.
interpret
¶
interpret() -> Interpretation
Read the fitted temperature against the identity T = 1.
Source code in src/probcal/parametric.py
270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 | |
BetaCalibrator
¶
BetaCalibrator(variant: str = 'abm')
Bases: BaseCalibrator
Beta calibration: logit g(s) = a·ln s − b·ln(1 − s) + c.
Variants: "abm" fits (a, b, c);
"ab" ties a = b (equivalent to Platt scaling on logits); "a"
additionally fixes c = 0 (a single-parameter map, the temperature
family in a different parameterization). The monotonicity constraint
a, b >= 0 is enforced by the betacal refit strategy: a negative
exponent drops its feature and refits. :meth:point_inverse is exact for
every variant, including the non-affine "abm" (see
:meth:_point_inverse_logit).
| PARAMETER | DESCRIPTION |
|---|---|
variant
|
TYPE:
|
| ATTRIBUTE | DESCRIPTION |
|---|---|
a_ |
Sensitivity near
TYPE:
|
b_ |
Sensitivity near
TYPE:
|
c_ |
Base-rate shift in log-odds.
TYPE:
|
constraint_active_ |
Whether the
TYPE:
|
converged_ |
Whether the fit whose coefficients survive converged (
TYPE:
|
separation_fallback_ |
Whether any IRLS call during fitting detected separation and fell
back to the ridge-regularized fit; recorded by
TYPE:
|
References
Kull, Silva Filho & Flach (2017), AISTATS 54 and EJS 11(2). The identity
is (a, b, c) = (1, 1, 0): beta calibration cannot un-calibrate an
already calibrated model. a != b captures asymmetric tail distortion;
temperature is the special case a = b = 1/T, c = 0.
Source code in src/probcal/parametric.py
350 351 | |
affine_logit_coeffs_
property
¶
affine_logit_coeffs_: tuple[float, float] | None
(a, c) for the tied variants; None for "abm".
With a = b the map reduces to logit g = a * logit(s) + c,
which is affine on the logit scale; the full three-parameter map
is not (see the shap-calibration chapter).
complexity_rank
property
¶
complexity_rank: float
Parsimony rank by variant: 1.5 ("a"), 2.5 ("ab"), 3.0 ("abm").
.get with a fallback because variant is validated only in
_fit; the property must not raise pre-fit.
interpret
¶
interpret() -> Interpretation
Read the fitted exponents and intercept against the identity (1, 1, 0).
If IRLS did not converge at fit time (a UserWarning was raised),
the messages include a note not to trust the coefficients.
Source code in src/probcal/parametric.py
496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 | |
isotonic
¶
Isotonic calibrators: PAVA-based isotonic and centered isotonic regression (CIR).
Theory and worked example: docs/concepts/methods-nonparametric.md.
References
Barlow, Bartholomew, Bremner & Brunk (1972); Zadrozny & Elkan (2002); Oron & Flournoy (2017) — full records in the documentation.
IsotonicCalibrator
¶
IsotonicCalibrator(interpolation: str = 'none')
Bases: BaseCalibrator
Isotonic calibration: the PAVA step function.
Fits the least-squares non-decreasing map of outcomes on scores. The
fitted map is a right-continuous step function with one level per pooled
block; scores outside the calibration range clamp to the first/last
level. interpolation="linear" instead joins block midpoints, removing
the discontinuities.
| PARAMETER | DESCRIPTION |
|---|---|
interpolation
|
TYPE:
|
| ATTRIBUTE | DESCRIPTION |
|---|---|
n_blocks_ |
Number of pooled blocks — the effective complexity estimated from the data.
TYPE:
|
block_mean_ |
Event rate of each pooled block (the step levels).
TYPE:
|
block_first_s_, block_last_s_ |
Score range covered by each block.
TYPE:
|
block_center_s_ |
Weight-centered score coordinate of each block (used by CIR).
TYPE:
|
References
Barlow et al. (1972) for PAVA; Zadrozny & Elkan (2002) for its use in classifier calibration.
Source code in src/probcal/isotonic.py
68 69 | |
complexity_rank
property
¶
complexity_rank: float
Parsimony rank 50.0: nonparametric, data-driven block count.
interpret
¶
interpret() -> Interpretation
Read the block structure as effective complexity and local event rates.
Source code in src/probcal/isotonic.py
115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 | |
CenteredIsotonicCalibrator
¶
CenteredIsotonicCalibrator()
Bases: IsotonicCalibrator
Centered isotonic regression (CIR): strictly increasing where data permit.
Post-processes the PAVA solution by collapsing each block to its weight-centered score coordinate and interpolating linearly through the points (Oron & Flournoy, 2017). Removes the step function's tied predictions — preferred when downstream ranking must be strict.
| ATTRIBUTE | DESCRIPTION |
|---|---|
n_blocks_ |
Number of pooled blocks — the effective complexity estimated from the data. Inherited from the PAVA fit.
TYPE:
|
block_mean_ |
Event rate of each pooled block (the interpolation y-values).
TYPE:
|
block_first_s_, block_last_s_ |
Score range covered by each block (inherited; not used for
prediction, which interpolates through
TYPE:
|
block_center_s_ |
Weight-centered score coordinate of each block — the interpolation x-values that make CIR strictly increasing.
TYPE:
|
References
Oron & Flournoy (2017).
Source code in src/probcal/isotonic.py
173 174 | |
interpret
¶
interpret() -> Interpretation
Isotonic reading plus the strictness property CIR adds.
Source code in src/probcal/isotonic.py
180 181 182 183 184 185 186 187 188 189 190 191 192 | |
binning
¶
Binning calibrators: histogram binning and scaling-binning.
Theory: docs/concepts/methods-nonparametric.md.
References
Zadrozny & Elkan (2001); Kumar, Liang & Ma (2019) — full records in the documentation.
HistogramBinningCalibrator
¶
HistogramBinningCalibrator(n_bins: int = 10, strategy: str = 'mass', shrinkage: str | None = 'jeffreys')
Bases: BaseCalibrator
Histogram binning: per-bin event rates with optional Jeffreys shrinkage.
| PARAMETER | DESCRIPTION |
|---|---|
n_bins
|
Requested number of bins
TYPE:
|
strategy
|
TYPE:
|
shrinkage
|
TYPE:
|
| ATTRIBUTE | DESCRIPTION |
|---|---|
bin_rate_ |
Calibrated value per (non-degenerate) bin.
TYPE:
|
is_monotone_ |
Computed after fitting: binning does not assume monotonicity, so the flag reports whether the fitted rates happen to be non-decreasing.
TYPE:
|
References
Zadrozny & Elkan (2001).
Source code in src/probcal/binning.py
60 61 62 63 64 65 | |
complexity_rank
property
¶
complexity_rank: float
Parsimony rank 10.0, for either strategy ("mass" or "width").
interpret
¶
interpret() -> Interpretation
Read bin rates as local event frequencies and B as the complexity dial.
Source code in src/probcal/binning.py
111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 | |
ScalingBinningCalibrator
¶
ScalingBinningCalibrator(n_bins: int = 10)
Bases: BaseCalibrator
Scaling-binning (Kumar–Liang–Ma): Platt stage, then bin the fitted values.
Fits Platt scaling first, then forms equal-mass bins of the fitted function values and outputs the mean of the fitted values within each bin. Achieves measurable calibration error with O(1/eps^2 + B) samples versus O(B/eps^2) for histogram binning.
| PARAMETER | DESCRIPTION |
|---|---|
n_bins
|
Requested number of equal-mass bins of the Platt-fitted values (equal
weight under
TYPE:
|
| ATTRIBUTE | DESCRIPTION |
|---|---|
platt_ |
The fitted first-stage Platt calibrator.
TYPE:
|
edges_ |
Interior quantile edges of the Platt-fitted values.
TYPE:
|
bin_value_ |
Mean Platt-fitted value per bin (the calibrated output for that bin).
TYPE:
|
is_monotone_ |
Derived from the fitted state:
TYPE:
|
References
Kumar, Liang & Ma (2019).
Source code in src/probcal/binning.py
172 173 | |
is_monotone_
property
¶
is_monotone_: bool
True iff the Platt stage is increasing (see class docstring).
complexity_rank
property
¶
complexity_rank: float
Parsimony rank 4.0: a Platt stage plus a bin count, still lightweight.
interpret
¶
interpret() -> Interpretation
Two-stage reading: Platt map, then the error-measurability discretization.
Source code in src/probcal/binning.py
226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 | |
bayesian
¶
Bayesian-ensemble calibrators: BBQ and ENIR.
Theory: docs/concepts/methods-nonparametric.md.
References
Naeini, Cooper & Hauskrecht (2015); Naeini & Cooper (2016); Tibshirani, Hoefling & Tibshirani (2011) — full records in the documentation.
BBQCalibrator
¶
BBQCalibrator(min_bins: int | None = None, max_bins: int | None = None, monotone: bool = False)
Bases: BaseCalibrator
Bayesian Binning into Quantiles: model averaging over equal-mass binnings.
Considers equal-mass binning models over a range of bin counts, scores each by its Beta-Binomial log marginal likelihood under a per-bin Jeffreys Beta(1/2, 1/2) prior, and predicts with the posterior-weighted average of the models' (posterior-mean) bin rates.
Nothing in the model forces the bin rates to increase with the score, and
on low-default data a sparse bin often dips below its neighbours, so the
averaged map is usually not monotone (is_monotone_ = False) and
refuses to invert. monotone=True projects each candidate's bin rates
onto the non-decreasing cone (weighted PAVA, weights = posterior counts)
before averaging; a non-negative mixture of non-decreasing step functions
is non-decreasing, so the averaged map is then monotone by construction.
The model weights are unchanged (they score the unconstrained binnings).
| PARAMETER | DESCRIPTION |
|---|---|
min_bins
|
Range of candidate bin counts; defaults to
TYPE:
|
max_bins
|
Range of candidate bin counts; defaults to
TYPE:
|
monotone
|
Isotonize each candidate's bin rates before averaging, so the map is
guaranteed non-decreasing and
TYPE:
|
| ATTRIBUTE | DESCRIPTION |
|---|---|
bins_grid_ |
Candidate bin counts.
TYPE:
|
model_weights_ |
Posterior weights over the candidates (sum to 1).
TYPE:
|
model_edges_, model_rates_ |
Interior edges and posterior-mean bin rates of each candidate binning
(isotonized when
TYPE:
|
is_monotone_ |
Whether the averaged step map is non-decreasing over the whole score domain — checked exactly at every bin edge of every candidate.
TYPE:
|
References
Naeini, Cooper & Hauskrecht (2015).
Source code in src/probcal/bayesian.py
78 79 80 81 82 83 | |
complexity_rank
property
¶
complexity_rank: float
Parsimony rank 40.0: a Bayesian model average over binnings.
interpret
¶
interpret() -> Interpretation
Read the posterior weights as uncertainty about the data's resolution.
Source code in src/probcal/bayesian.py
163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 | |
ENIRCalibrator
¶
ENIRCalibrator(max_solutions: int | None = 256)
Bases: BaseCalibrator
Ensemble of near-isotonic regressions (ENIR).
Computes the full nearly-isotonic solution path (modified PAVA of
Tibshirani, Hoefling & Tibshirani, 2011) from the raw data (lambda = 0)
to the fully isotonic fit, then averages the breakpoint solutions with
BIC weights. The combined map may be non-monotone: is_monotone_ is
False and consumers requiring order preservation should prefer a
monotone calibrator. Fitting is quadratic in the number of unique scores
and intended for m <= 50,000; above that, fit emits a single
UserWarning stating the expected minutes.
| PARAMETER | DESCRIPTION |
|---|---|
max_solutions
|
Number of path solutions to keep for the ensemble, chosen by best
(lowest) BIC;
TYPE:
|
| ATTRIBUTE | DESCRIPTION |
|---|---|
path_lambdas_ |
Breakpoints of the penalty parameter, starting at 0. All breakpoints are recorded, whether or not their solution is retained.
TYPE:
|
path_solutions_ |
Fitted values on the tie-aggregated score grid at the retained
breakpoints, in breakpoint order.
TYPE:
|
kept_breakpoints_ |
Indices into
TYPE:
|
model_weights_ |
BIC weights over the retained solutions, renormalized to sum to 1.
TYPE:
|
scores_ |
Distinct calibration scores: the grid
TYPE:
|
dropped_weight_ |
BIC weight lost to retention — the weight of scored solutions that the
TYPE:
|
References
Naeini & Cooper (2016); Tibshirani, Hoefling & Tibshirani (2011).
Source code in src/probcal/bayesian.py
245 246 | |
complexity_rank
property
¶
complexity_rank: float
Parsimony rank 80.0: an ensemble over the full near-isotonic path.
interpret
¶
interpret() -> Interpretation
Read the path length and BIC weights; warn about non-monotonicity.
Source code in src/probcal/bayesian.py
423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 | |
vennabers
¶
Venn–Abers calibrators: inductive (IVAP) and cross (CVAP).
Theory, validity guarantee scope, and the scalarization caveat:
docs/concepts/methods-distribution-free.md. The guarantee attaches to the
interval returned by :meth:VennAbersCalibrator.predict_interval; the scalar
from predict_proba is the log-loss-minimax merger and is not itself covered
by the validity theorem.
References
Vovk & Petej (2014) — full record in the documentation.
VennAbersCalibrator
¶
Bases: BaseCalibrator
Inductive Venn–Abers predictor (IVAP).
For a query score, two isotonic fits on the calibration set augmented
with the query labeled 0 (resp. 1) yield the interval [p0, p1];
predict_proba scalarizes it as p1 / (1 - p0 + p1).
Both fits are precomputed at fit time by the Vovk & Petej (2014) cumulative-
sum-diagram sweep, so prediction is a searchsorted gather rather than a
pair of PAVA refits per unique query score.
Tied calibration scores are pooled into one point (weights summed), as
isotonic regression requires, and a query whose score equals a calibration
score joins that point; the output therefore does not depend on row
order. A query labeled 1 (resp. 0) strictly between two calibration scores
has the same fitted value as when merged into the next (resp. previous)
point, so the two tables are read at searchsorted "left" (F1_)
and "right" (F0_).
| ATTRIBUTE | DESCRIPTION |
|---|---|
scores_ |
Distinct calibration scores, sorted.
TYPE:
|
F0_, F1_ |
Fitted probabilities for a unit-weight query labeled 0 (resp. 1)
inserted at each of the m+1 gaps between the distinct calibration
scores. Both are non-decreasing, and
TYPE:
|
ties_pooled_ |
TYPE:
|
Notes
With non-unit sample weights the query still enters at weight 1, which is the
natural generalization but sits outside the validity theorem as proved; see
the scope note in docs/concepts/methods-distribution-free.md.
complexity_rank
property
¶
complexity_rank: float
Parsimony rank 60.0: a distribution-free interval predictor.
predict_interval
¶
predict_interval(s: object) -> ndarray
Venn–Abers intervals [p0, p1] for new scores.
| PARAMETER | DESCRIPTION |
|---|---|
s
|
Raw scores/probabilities in
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
numpy.ndarray of shape (n, 2)
|
Columns |
Source code in src/probcal/vennabers.py
147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 | |
interpret
¶
interpret() -> Interpretation
Report interval widths over the calibration scores — where to trust the map.
Source code in src/probcal/vennabers.py
175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 | |
CrossVennAbersCalibrator
¶
CrossVennAbersCalibrator(cv: int = 5, random_state: int = 42)
Bases: BaseCalibrator
Cross Venn–Abers predictor (CVAP): fold-wise IVAPs, geometric-mean merge.
Splits the calibration data into cv stratified folds; each fold's
IVAP is fitted on the remaining folds. The scalar output merges the
fold-wise pairs by the log-loss rule of Vovk & Petej:
GM(p1) / (GM(1 - p0) + GM(p1)). predict_interval returns the
conservative envelope [min_k p0_k, max_k p1_k] (a probcal design
choice — the paper defines only the scalar merge).
| PARAMETER | DESCRIPTION |
|---|---|
cv
|
Number of stratified folds; an integer
TYPE:
|
random_state
|
Seed for the fold assignment.
TYPE:
|
| ATTRIBUTE | DESCRIPTION |
|---|---|
ivaps_ |
One fitted IVAP per fold, each trained on the other
TYPE:
|
References
Vovk & Petej (2014).
Source code in src/probcal/vennabers.py
225 226 227 | |
complexity_rank
property
¶
complexity_rank: float
Parsimony rank 60.0: same tier as IVAP, folded across cv splits.
predict_interval
¶
predict_interval(s: object) -> ndarray
Conservative fold envelope [min_k p0_k, max_k p1_k] for new scores.
| PARAMETER | DESCRIPTION |
|---|---|
s
|
Raw scores/probabilities in
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
numpy.ndarray of shape (n, 2)
|
Columns |
Source code in src/probcal/vennabers.py
252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 | |
interpret
¶
interpret() -> Interpretation
Report fold count and envelope widths over a probe grid.
Source code in src/probcal/vennabers.py
271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 | |
spline
¶
Spline calibrator: penalized natural cubic splines on the logit scale.
Theory: docs/concepts/methods-nonparametric.md.
References
Lucena (2018); Hastie, Tibshirani & Friedman (2009), §5.2.1 — full records in the documentation.
SplineCalibrator
¶
SplineCalibrator(n_knots: int | None = None, lambdas: object = None, cv: int = 5, random_state: int = 42)
Bases: BaseCalibrator
Natural cubic spline calibration on the logit scale.
Models logit g(s) = sum_k theta_k N_k(logit s) with the natural cubic
basis (linear beyond the boundary knots), fitted by penalized IRLS with a
second-difference roughness penalty — the package's safeguarded Newton
solver (:func:probcal._math.irls_logistic with a penalty matrix:
step-halving, full-step convergence test, separation fallback). The
penalty weight is chosen by K-fold cross-validated log loss within the
calibration set.
| PARAMETER | DESCRIPTION |
|---|---|
n_knots
|
Number of knots (placed at equally spaced quantiles of the logit
scores); defaults to
TYPE:
|
lambdas
|
Candidate penalty weights (finite, non-negative); defaults to
TYPE:
|
cv
|
Inner fold count for the lambda search; an integer
TYPE:
|
random_state
|
Seed for the stratified fold assignment.
TYPE:
|
| ATTRIBUTE | DESCRIPTION |
|---|---|
lambda_ |
Selected penalty weight.
TYPE:
|
edof_ |
Effective degrees of freedom — trace of the smoother matrix at the fitted solution; the honest complexity measure.
TYPE:
|
n_knots_ |
Number of knots actually used.
TYPE:
|
knots_ |
Knot locations on the logit scale.
TYPE:
|
theta_ |
Spline coefficients on the natural cubic basis.
TYPE:
|
converged_ |
Whether the final penalized IRLS fit converged; if
TYPE:
|
is_monotone_ |
Checked exactly after fitting: the logit-scale slope is evaluated at every knot (which also gives both linear tails) and at each interior extremum of the piecewise-quadratic slope. The penalty does not enforce monotonicity, and a non-monotone fit is flagged with a warning.
TYPE:
|
References
Lucena (2018); Hastie, Tibshirani & Friedman (2009), §5.2.1.
Source code in src/probcal/spline.py
99 100 101 102 103 104 105 106 107 108 109 | |
complexity_rank
property
¶
complexity_rank: float
Parsimony rank 12.0: a penalized basis expansion, more flexible than binning.
interpret
¶
interpret() -> Interpretation
Read effective degrees of freedom as the honest complexity measure.
Source code in src/probcal/spline.py
205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 | |
segmented
¶
SegmentedCalibrator: empirical-Bayes shrunken per-segment logit offsets.
Theory and the DerSimonian-Laird method-of-moments derivation:
docs/concepts/segmented.md.
References
DerSimonian & Laird (1986) — random-effects meta-analysis method-of-moments heterogeneity estimator, reused here across segments instead of studies.
SegmentedCalibrator
¶
SegmentedCalibrator(base: BaseCalibrator | None = None, *, unseen: str = 'global')
Bases: BaseCalibrator
Per-segment logit offsets on top of a shared base map, empirical-Bayes shrunk.
Fits one shared base calibrator on all data, then an offset-only
logistic MLE (:func:probcal.offset.estimate_offset) of each segment's
residual log-odds shift against the base map's predictions. Segments
with few observations have a noisy, high-variance delta_hat; rather
than use it directly (no pooling — overfits small segments) or discard
it (complete pooling — ignores real heterogeneity), each segment's
estimate is shrunk toward the across-segment mean by the classic
empirical-Bayes/random-effects factor tau2 / (tau2 + se**2), where
tau2 is the between-segment heterogeneity variance estimated by the
DerSimonian-Laird (1986) method of moments. A small, noisy segment
(large se) shrinks toward 0 (the base map); a large, precise segment
(small se) keeps most of its own estimate.
Segments with only one outcome class have no offset MLE
(estimate_offset raises); they are recorded as delta_hat=0.0,
se=inf — fully shrunk, since an infinite-variance estimate carries
no weight in the DerSimonian-Laird pooling and tau2 / (tau2 + inf)
= 0.
fit and predict_proba add a keyword-only segments argument
on top of the base signature (segments=None degrades to a single
segment "__all__" at fit time, and to the plain base map — no
segment-specific offset — at predict time), so the zero-argument
protocol calls (SegmentedCalibrator().fit(s, y),
cal.predict_proba(s)) still work. Because :class:~probcal.chain.Chain
has no segments= slot, Chain([seg, ...]) always predicts through
seg's global map (segments=None, delta=0) — the per-segment
shift is not baked into a Chain; use SegmentedCalibrator directly
(with segments=) when the per-segment offset must apply.
Segment labels are compared as strings (_coerce_segments calls
.astype(str)): fit-time labels 0, 1 (int) become "0",
"1", but predict-time labels 0.0, 1.0 (float) become
"0.0", "1.0" — a mismatch that never raises (every label
looks "unseen") and, under unseen="global", silently falls back
to the base map for every row. Pass segments with the same
representation (e.g. cast to str yourself) at fit and predict
time. When every row of a predict_proba/inverse call is unseen
and unseen="global", a UserWarning is raised naming the
fitted segments_ — a partial overlap (some rows match, some are
genuinely new segments) stays silent.
| PARAMETER | DESCRIPTION |
|---|---|
base
|
Unfitted calibrator cloned (:func:
TYPE:
|
unseen
|
Policy for a segment label at predict/inverse time that was not
seen at fit time.
TYPE:
|
| ATTRIBUTE | DESCRIPTION |
|---|---|
base_ |
The fitted clone of
TYPE:
|
segments_ |
Segment labels seen at fit time, sorted.
TYPE:
|
n_, events_ |
Per-segment observation count and weighted event count, aligned
with
TYPE:
|
delta_hat_, se_ |
Per-segment offset MLE and its standard error
(:func:
TYPE:
|
tau2_ |
Between-segment heterogeneity variance (DerSimonian-Laird
method-of-moments estimate); 0.0 when fewer than two segments have
a finite
TYPE:
|
shrink_ |
Per-segment shrinkage factor
TYPE:
|
delta_tilde_ |
Per-segment shrunk offset,
TYPE:
|
is_monotone_ |
TYPE:
|
Examples:
>>> import numpy as np
>>> from probcal import SegmentedCalibrator, make_pd_portfolio
>>> d = make_pd_portfolio(n=900, random_state=0)
>>> segments = np.array(["a", "b", "c"])[np.arange(900) % 3]
>>> cal = SegmentedCalibrator().fit(d.scores, d.y, segments=segments)
>>> cal.segments_
('a', 'b', 'c')
>>> p_global = cal.predict_proba(d.scores) # segments=None: the base map
>>> p_seg = cal.predict_proba(d.scores, segments=segments)
>>> p_global.shape == p_seg.shape == d.scores.shape
True
Source code in src/probcal/segmented.py
145 146 147 | |
is_monotone_
property
¶
is_monotone_: bool
The fitted base map's is_monotone_ (True before fitting).
A per-segment logit shift is increasing, so it cannot break monotonicity; the base map alone decides.
complexity_rank
property
¶
complexity_rank: float
The base map's parsimony rank (fitted base_, else the prototype).
affine_logit_coeffs_
property
¶
affine_logit_coeffs_: tuple[float, float] | None
(a, b + delta_tilde) only for a single fitted segment; else None.
With more than one segment the map is segment-dependent (there is
no single affine map on the logit scale that fits every segment),
so :meth:point_inverse (which relies on this property) is
unavailable then — use :meth:interval_inverse with segment=.
fit
¶
fit(s: object, y: object, sample_weight: object = None, *, segments: object = None) -> SegmentedCalibrator
Fit the shared base map, then per-segment shrunk offsets.
| PARAMETER | DESCRIPTION |
|---|---|
s
|
Raw scores/probabilities in
TYPE:
|
y
|
Binary outcomes in
TYPE:
|
sample_weight
|
Positive observation weights.
TYPE:
|
segments
|
Segment label per observation, same length as
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
SegmentedCalibrator
|
The fitted calibrator. |
Source code in src/probcal/segmented.py
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 | |
predict_proba
¶
predict_proba(s: object, *, segments: object = None) -> ndarray
Calibrated probabilities, with an optional per-observation segment offset.
| PARAMETER | DESCRIPTION |
|---|---|
s
|
Raw scores/probabilities in
TYPE:
|
segments
|
Segment label per observation, same length as
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
numpy.ndarray of shape (n,)
|
Calibrated probabilities. |
Source code in src/probcal/segmented.py
263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 | |
interval_inverse
¶
interval_inverse(lo: float, hi: float, *, space: str = 'probability', buffer_logit: float = 0.0, segment: object = None) -> tuple[float, float]
Preimage of a calibrated interval, optionally for one fitted segment.
segment=None (default) uses the global map (delta=0),
matching :meth:predict_proba's segments=None convention;
otherwise the preimage is through base_ composed with that
segment's delta_tilde (Chain([base_, LogitOffset(delta=...)])).
| PARAMETER | DESCRIPTION |
|---|---|
lo
|
Calibrated-probability bounds.
TYPE:
|
hi
|
Calibrated-probability bounds.
TYPE:
|
space
|
Scale of the returned raw bounds.
TYPE:
|
buffer_logit
|
Logit-space shrinkage applied before inverting.
TYPE:
|
segment
|
Segment label to invert through;
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
tuple of float
|
|
Source code in src/probcal/segmented.py
337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 | |
point_inverse
¶
point_inverse(p: object, *, space: str = 'probability', segment: object = None) -> ndarray
Exact preimage of calibrated probabilities, optionally for one segment.
Same segment= convention as :meth:interval_inverse: inverts
through base_ (composed with the segment's delta_tilde when
segment is given), so this works for any number of fitted
segments as long as base_ itself has an exact point inverse
(base_.affine_logit_coeffs_ is not None) — unlike
:attr:affine_logit_coeffs_ on self, which is only defined for
a single fitted segment.
| PARAMETER | DESCRIPTION |
|---|---|
p
|
Calibrated probabilities strictly inside
TYPE:
|
space
|
Scale of the returned raw values.
TYPE:
|
segment
|
Segment label to invert through;
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
ndarray
|
Raw scores (or logits) whose calibrated probability equals |
Source code in src/probcal/segmented.py
373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 | |
interpret
¶
interpret() -> Interpretation
Per-segment shrinkage table plus the fitted heterogeneity variance.
Source code in src/probcal/segmented.py
404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 | |