Testing

A raw SWIFT score has no natural threshold for "significant". Stage 5 calibrates it against the data itself with a permutation test, then applies multiple testing correction across features to produce the final drift decisions.

from swift.threshold import permutation_test, bootstrap_threshold, correct_pvalues
from swift.types import CorrectionMethod

pvalues = permutation_test(X_ref, X_mon, bucket_sets, order=1, n_permutations=1000)
# Returns: dict[str, float] — feature name → p-value in (0, 1]

decisions = correct_pvalues(pvalues, CorrectionMethod.BH, alpha=0.05)
# Returns: dict[str, bool] — True = flagged as drifted

SWIFTMonitor.test() runs both steps (after computing the observed scores) and packs the outcome into a SWIFTResult.

The permutation test

Under the null hypothesis of no drift, reference and monitoring observations are exchangeable: any observation could equally well have appeared in either window. The test exploits this directly, with the observed SWIFT score as the test statistic:

1. Pool the reference and monitoring observations.
2. Randomly split the pool into two groups of the original sizes (\(n_{\text{ref}}\) and \(n_{\text{mon}}\)).
3. Compute the Wasserstein distance between the two random groups.
4. Repeat n_permutations (\(B\)) times to build a null distribution of scores.

The p-value uses the conservative formula

\[ p_j \;=\; \frac{1 + \#\bigl\{\,b : \text{SWIFT}_j^{(b)} \ge \text{SWIFT}_j^{\text{obs}}\bigr\}}{1 + B}, \]

with the \(+1\) in numerator and denominator counting the observed split as one more permutation — so p-values live in \((0, 1]\) and the smallest attainable value is \(1/(1+B)\) (about 0.001 at the default \(B = 1000\)). Raise n_permutations when you need finer resolution; see Configuration.

Two performance-critical mechanics:

• \(\sigma_j\) is never recomputed inside the loop. The pooled data is transformed once per feature before the loop; each permutation only shuffles indices into the pre-transformed arrays and recomputes the distance. SHAP is not touched.
• Subsampling via max_samples. When set and the pool exceeds it, both samples are randomly subsampled (without replacement, preserving the reference/monitoring ratio) before the loop — a significant speedup on large datasets with negligible impact on statistical power.

Multiple testing correction

Every feature is tested simultaneously — with 50 features at \(\alpha = 0.05\), a few false alarms per run would be expected without correction. correct_pvalues supports two methods:

Method	Strings accepted	Controls	Decision rule
Benjamini–Hochberg	"benjamini-hochberg", "bh", "fdr"	FDR	sort p-values ascending; find the largest \(k\) with \(p_{(k)} \le k\alpha/p\); flag all features of rank \(\le k\)
Bonferroni	"bonferroni", "bonf"	FWER	flag feature \(j\) when \(p_j < \alpha/p\) (strict inequality)

Strings are case-insensitive; enum members from swift.types.CorrectionMethod also work, and anything else raises ValueError listing the valid values. Benjamini–Hochberg is the recommended default — a good balance between power and false positive control; Bonferroni is more conservative — use it when false positives are costly.

Alternative: bootstrap thresholds

An alternative calibration also ships: instead of p-values, bootstrap_threshold produces a per-feature absolute score threshold from the reference data alone — a principled alternative to PSI's ad-hoc 0.10 / 0.25 cutoffs.

from swift.threshold import bootstrap_threshold

thresholds = bootstrap_threshold(
    X_ref, bucket_sets, n_mon=len(X_mon), order=1, alpha=0.05, n_bootstrap=1000
)
# Returns: dict[str, float] — feature name → score threshold

It draws n_bootstrap samples of the monitoring size from the reference (with replacement), scores each against the reference, and returns the \((1-\alpha)\) quantile of those null scores per feature: an observed score above the threshold is larger than what sampling noise alone produces at level \(\alpha\). This path is not wired into SWIFTMonitor.test() — call it directly when you want fixed thresholds (e.g. to draw a threshold line in plot_swift_scores).

---

Next stage: per-feature decisions roll up to a model-level verdict in Aggregation. Full signatures: API reference — Threshold.