Skip to content

Calibration

calibration

PD calibration diagnostics and recalibration for FlagGAM.

This module is an ORIGINAL ADDITION and is not part of Zhao & Welsch (arXiv:2605.31189): the paper evaluates only ranking metrics and never assesses probability calibration. Design notes in docs/DECISIONS.md entry 19.

CalibratedFlagGAM 

CalibratedFlagGAM(estimator: Any, method: str = 'platt', cv: int | str = 5, target_rate: float | None = None)

Recalibration wrapper: platt / isotonic / base_rate over a FlagGAM classifier.

cv=k cross-fits k clones to obtain leak-free out-of-fold predictions for the single pooled calibrator, then refits the estimator on all data; cv="prefit" treats estimator as already fitted and (X, y) in fit() as pure calibration data. Binary classification only.

Source code in src/flaggam/calibration.py 
111
112
113
114
115
116
117
118
119
120
121
def __init__(
    self,
    estimator: Any,
    method: str = "platt",
    cv: int | str = 5,
    target_rate: float | None = None,
) -> None:
    self.estimator = estimator
    self.method = method
    self.cv = cv
    self.target_rate = target_rate

reliability_curve 

reliability_curve(y_true: Any, y_prob: Any, n_bins: int = 10, strategy: str = 'uniform') -> DataFrame

Binned reliability table (empty bins dropped).

Source code in src/flaggam/calibration.py 
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
def reliability_curve(
    y_true: Any, y_prob: Any, n_bins: int = 10, strategy: str = "uniform"
) -> pd.DataFrame:
    """Binned reliability table (empty bins dropped)."""
    y, p = _validate_binary(y_true, y_prob)
    edges = _bin_edges(p, n_bins, strategy)
    idx = np.clip(np.digitize(p, edges[1:-1], right=False), 0, len(edges) - 2)
    rows = []
    for b in range(len(edges) - 1):
        mask = idx == b
        if not mask.any():
            continue
        rows.append(
            {
                "bin_lower": edges[b],
                "bin_upper": edges[b + 1],
                "mean_predicted": float(p[mask].mean()),
                "fraction_positive": float(y[mask].mean()),
                "count": int(mask.sum()),
            }
        )
    return pd.DataFrame(rows)