probcal validation report — sample portfolio

Metric report

metric	value	ci_low	ci_high
log_loss	0.160907	0.159401	0.163031
brier	0.0388998	0.0387471	0.03911
brier_skill	0.0324737	0.0272463	0.0362735
ece	0.0119041	0.011457	0.0125929
ece_debiased	0.0108996	0.0104738	0.0117169
mce	0.0348202	0.0252814	0.0469443
ece_sweep	0.0116966	0.011391	0.0122127
smooth_ece	0.0135576	0.0142289	0.0155732
ecce_max	0.0117344	0.0114285	0.012073
ecce_mean	0.00338949	0.00290404	0.00394186
ici	0.0115624	0.0112129	0.0119149
e50	0.0101385	0.0080486	0.0126861
e90	0.0211354	0.0165918	0.0279877
emax	0.121316	0.0383552	0.196534
spiegelhalter_z	11.9996	11.5809	12.4356
spiegelhalter_p	0	0	0
intercept	0.350861	0.340164	0.360986
slope	0.985415	0.915111	1.04092

score	value	mcb	dsc	unc
brier	0.0388998	0.000341671	0.00164729	0.0402055
log_loss	0.160907	0.00305088	0.0162889	0.174145

Rating grades

Grades ordered best to worst by mean predicted probability: G1, G2, G3, G4, G5.

Per-grade backtest (Jeffreys)

grade	n	k	pd	p_value	light	ci_low	ci_high
G1	2192	14	0.00343442	0.0152669	amber	0.00404361	0.00969011
G2	4568	41	0.00750796	0.126736	green	0.00690304	0.0115069
G3	7712	155	0.0146753	8.30434e-05	red	0.0176012	0.0228618
G4	10395	490	0.0318961	6.19636e-17	red	0.0438115	0.0506507
G5	5133	559	0.0821237	1.11519e-11	red	0.101914	0.116218

Pluto-Tasche most-prudent PD

grade	n	d	n_pooled	d_pooled	pd_upper
G1	2192	14	30000	1259	0.0434891
G2	4568	41	27808	1245	0.0464023
G3	7712	155	23240	1204	0.0537199
G4	10395	490	15528	1049	0.0702092
G5	5133	559	5133	559	0.114684

Jeffreys upper masterscale bands

grade	lo	hi
G1	0	0.00890364
G2	0.00890364	0.0109213
G3	0.0109213	0.0222365
G4	0.0222365	0.0498645
G5	0.0498645	0.114584

Grouped evaluation

group	metric	value	ci_low	ci_high
pooled	log_loss	0.160907	0.159401	0.163031
pooled	brier	0.0388998	0.0387471	0.03911
pooled	brier_skill	0.0324737	0.0272463	0.0362735
pooled	ece	0.0119041	0.011457	0.0125929
pooled	ece_debiased	0.0108996	0.0104738	0.0117169
pooled	mce	0.0348202	0.0252814	0.0469443
pooled	ece_sweep	0.0116966	0.011391	0.0122127
pooled	smooth_ece	0.0135576	0.0142289	0.0155732
pooled	ecce_max	0.0117344	0.0114285	0.012073
pooled	ecce_mean	0.00338949	0.00290404	0.00394186
pooled	ici	0.0115624	0.0112129	0.0119149
pooled	e50	0.0101385	0.0080486	0.0126861
pooled	e90	0.0211354	0.0165918	0.0279877
pooled	emax	0.121316	0.0383552	0.196534
pooled	spiegelhalter_z	11.9996	11.5809	12.4356
pooled	spiegelhalter_p	0	0	0
pooled	intercept	0.350861	0.340164	0.360986
pooled	slope	0.985415	0.915111	1.04092
corporate	log_loss	0.214833	0.210846	0.218641
corporate	brier	0.0542141	0.0537831	0.0545943
corporate	brier_skill	0.0259834	0.0191536	0.0337272
corporate	ece	0.0289987	0.0284989	0.0300633
corporate	ece_debiased	0.0274479	0.0268107	0.0285614
corporate	mce	0.101106	0.0758866	0.135339
corporate	ece_sweep	0.0289186	0.0284004	0.0294975
corporate	smooth_ece	0.029487	0.0294839	0.0311959
corporate	ecce_max	0.0290063	0.0284769	0.0294661
corporate	ecce_mean	0.00827158	0.00725576	0.00928493
corporate	ici	0.0287678	0.0281801	0.0295181
corporate	e50	0.0239435	0.0197083	0.0286114
corporate	e90	0.0577127	0.0474466	0.0689833
corporate	emax	0.2256	0.104827	0.367017
corporate	spiegelhalter_z	17.1059	16.6221	17.5922
corporate	spiegelhalter_p	0	0	0
corporate	intercept	0.730651	0.712931	0.747035
corporate	slope	0.99641	0.906163	1.0913
retail	log_loss	0.117664	0.114893	0.120026
retail	brier	0.026647	0.0263922	0.0268791
retail	brier_skill	0.0297639	0.0213127	0.0390429
retail	ece	0.00512908	0.00474311	0.00884493
retail	ece_debiased	0.00277337	0.00189653	0.00663166
retail	mce	0.0153094	0.0104133	0.0300122
retail	ece_sweep	0.00314332	0.00207228	0.0055622
retail	smooth_ece	0.0129473	0.0146531	0.0171585
retail	ecce_max	0.00252572	0.00206808	0.00403459
retail	ecce_mean	0.000743098	0.000454809	0.00144969
retail	ici	0.00226073	0.00204495	0.00436172
retail	e50	0.00141651	0.000395251	0.00337377
retail	e90	0.00624267	0.0034123	0.0130909
retail	emax	0.0174431	0.00711085	0.102299
retail	spiegelhalter_z	-1.32544	-1.67913	-0.927809
retail	spiegelhalter_p	0.185026	0.093127	0.353507
retail	intercept	-0.0786055	-0.0975029	-0.0580631
retail	slope	1.0097	0.890044	1.16424
sme	log_loss	0.150549	0.147404	0.153586
sme	brier	0.0359309	0.0355848	0.0362606
sme	brier_skill	0.0313008	0.0224138	0.040632
sme	ece	0.00854812	0.00842314	0.0113342
sme	ece_debiased	0.00539417	0.0053002	0.00908197
sme	mce	0.0247769	0.0211202	0.0459276
sme	ece_sweep	0.00845266	0.00805975	0.00972799
sme	smooth_ece	0.0152054	0.0164871	0.0189925
sme	ecce_max	0.00853209	0.00804564	0.00980573
sme	ecce_mean	0.00252037	0.00175171	0.00349682
sme	ici	0.00825457	0.00780451	0.0092684
sme	e50	0.0080245	0.00398537	0.00975059
sme	e90	0.0160931	0.0113262	0.0226448
sme	emax	0.0801554	0.0194387	0.188873
sme	spiegelhalter_z	5.05507	4.60121	5.42155
sme	spiegelhalter_p	4.30232e-07	5.90933e-08	4.20059e-06
sme	intercept	0.264697	0.246891	0.281646
sme	slope	0.976116	0.862444	1.10226

Appendix: calibrator

Fitted state (JSON)

{
  "probcal_schema": 1,
  "probcal_version": "0.3.0",
  "class": "BetaCalibrator",
  "params": {
    "variant": "abm"
  },
  "state": {
    "a_": 0.9230181543085806,
    "b_": 0.14248519964715528,
    "c_": -0.859897385009572,
    "constraint_active_": false,
    "converged_": true,
    "separation_fallback_": false
  },
  "fit_meta": {
    "n_obs": 8000,
    "n_events": 244.0,
    "weight_sum": 8000.0,
    "fitted_at_utc": "2026-08-30T00:20:29+00:00",
    "data_fingerprint": "d5570bacd0fba6b0e94aaa2ebe909cafbcab48760bbcbd70bb10b7e27e9d9149",
    "converged_": true,
    "separation_fallback_": false
  }
}

a = 0.923: sensitivity near s -> 0; a < 1 raises the smallest probabilities (model was overconfident in the low tail), a > 1 deepens them
b = 0.142: sensitivity near s -> 1; the mirrored reading for the high tail
c = -0.860: base-rate shift of -0.860 log-odds, odds factor 0.423
identity map corresponds to (a, b, c) = (1, 1, 0)
a != b (gap +0.781): asymmetric tail distortion that no symmetric (Platt/temperature) map could express

probcal validation report — sample portfolio

Portfolio summary

Reliability

CORP (PAV) reliability diagram

smECE-consistent kernel reliability curve

Metric report

CORP decomposition

Rating grades

Per-grade backtest (Jeffreys)

Pluto-Tasche most-prudent PD

Jeffreys upper masterscale bands

Grouped evaluation

Calibration monitoring

Appendix: calibrator