How SWIFT detects drift¶

Classical monitoring answers "did this feature's distribution change?". SWIFT answers a sharper question: did it change somewhere the model actually looks? The idea is to stop comparing raw feature values and instead compare what those values mean to the model — each observation is mapped to the SHAP impact of the region it falls in, and the drift test runs on those SHAP-transformed distributions.

This article walks the five stages end to end: how the model's own split thresholds define the comparison grid, how SHAP turns that grid into an impact scale, how the Wasserstein distance measures drift on it, and how a permutation test with multiple testing correction turns distances into defensible drift decisions. Each stage links to a concept page that covers it in depth.

Running example

A credit-risk team runs a LightGBM probability-of-default model in production. The reference window X_ref is the validation sample the model was signed off on; every month, the newest application window X_mon is tested against it. Features include utilization, income, n_delinquencies, and months_since_last_delinq — which is NaN for applicants with no delinquency on record. The team's question is not "did income shift?" (it always does, a little) but "did anything shift in a way the model's score actually responds to?" — and, because dozens of features are tested every month, "how do we keep false alarms from burying the real ones?"

Stages 1–3 run once, during fit(), and learn the reference distribution; stages 4–5 run during every test() call. Each stage lives in its own module, documented in the API reference.

flowchart TD
    subgraph fit ["fit(X_ref) — once per reference sample"]
        M["trained model<br/>LightGBM / XGBoost Booster"] -->|"Stage 1: extraction"| DP["decision points<br/>unique split thresholds per feature"]
        DP -->|"Stage 2: bucketing"| B["buckets<br/>null bucket + threshold intervals"]
        M --> SH["SHAP values on X_ref<br/>shap.TreeExplainer"]
        REF["reference data X_ref"] --> SH
        SH -->|"Stage 3: normalization"| SIG["&sigma;&#8342;: mean SHAP per bucket<br/>(synthetic fill for empty buckets)"]
        B --> SIG
    end
    subgraph test ["test(X_mon) — per monitoring sample"]
        MON["monitoring data X_mon"] --> T["transform both samples with &sigma;&#8342;"]
        SIG --> T
        T -->|"Stage 4"| W["Wasserstein distance per feature<br/>= SWIFT score"]
        W -->|"Stage 5"| P["permutation test<br/>p-value per feature"]
        P --> MTC["multiple testing correction<br/>Benjamini-Hochberg / Bonferroni"]
    end
    MTC --> R["SWIFTResult<br/>drifted_features, swift_max, swift_mean"]

Why compare in SHAP space at all?¶

Traditional drift detection (KS test, PSI, …) compares raw feature distributions and flags any statistical shift, regardless of whether the model cares. That produces two failure modes in practice. False alarms: income inflates by a few percent across the board, the KS test fires, but the whole shift happens inside regions where the model's response is flat — predictions do not move. Missed drift dressed as noise: a modest shift concentrated exactly at a threshold the model splits hard on can move scores materially while looking unremarkable in raw-value space.

SWIFT's construction removes the gap by measuring drift on the model's own response scale: every feature value is replaced by the mean SHAP contribution of the region it falls in, and only then are the distributions compared. A shift the model cannot see transforms to nothing; a shift the model responds to is weighted by exactly how much it responds. Regions the model ignores cannot raise an alarm — by construction, not by post-hoc filtering.

Stage 1: the model says where to look¶

The pipeline starts by extracting decision points — the split thresholds — from the trained tree ensemble. These are the feature values where the model's trees make split decisions: the boundaries the model considers meaningful. For each feature, every threshold used across all trees is collected, deduplicated, and sorted; a feature that never appears in any split gets an empty set.

Extraction reads the serialized trees directly — dump_model() for a LightGBM Booster, get_dump(dump_format="json") for an XGBoost Booster — and dispatches on the model type. Anything else raises TypeError; for the scikit-learn wrappers, pass the underlying booster (e.g. clf.booster_ for LightGBM — see the FAQ). In LightGBM trees, only numeric split thresholds are collected; categorical splits (whose threshold is a string like "0||1||3") are skipped.

For the running example: if the ensemble splits utilization at 0.42 in one tree and 0.65 in another, those two numbers — not any quantile of the data — become the boundaries every later comparison is built on.

Models and decision points covers the extraction per library, feature-name resolution, and the unsupported-model error in depth.

Stage 2: buckets, not bins¶

Decision points partition each feature's domain into buckets. For sorted thresholds \(t_1 < t_2 < \dots < t_m\), the bucket set is

bucket 0 — a null bucket that catches missing values (NaN),
bucket 1 — \((-\infty, t_1)\),
buckets 2 … m — \([t_1, t_2), \dots, [t_{m-1}, t_m)\),
bucket \(m{+}1\) — \([t_m, +\infty)\),

for \(m + 2\) buckets in total. A feature with no decision points gets a single catch-all bucket \((-\infty, +\infty)\) next to its null bucket.

The point of this construction: within one bucket, every tree in the ensemble routes the value the same way. The buckets are exactly the resolution at which the model can distinguish feature values — a shift that stays inside a bucket is invisible to the model, and SWIFT will treat it that way.

This is why the boundaries come from the model rather than from quantiles of the data. Quantile bins cut where the data is dense, which both splits regions the model treats identically (manufacturing resolution the model does not have) and merges regions the model separates (hiding resolution it does have). Model-derived buckets make the bucketing lossless with respect to model behavior — nothing finer exists to measure.

Why every feature gets a null bucket — even without missing values

The null bucket always sits at index 0, whether or not the reference sample contains any NaN. In the running example, months_since_last_delinq being NaN means something — no delinquency record — and a change in the share of NaN applicants is a real distribution shift. Because NaN maps to its own bucket with its own mean-SHAP level, that shift moves mass between SHAP levels and is visible to the test, rather than being dropped or imputed away. A null bucket that is empty in the reference gets its level from the synthetic fill (Stage 3).

One consequence follows immediately: a feature with no splits has a single catch-all bucket, every non-missing value transforms to the same mean SHAP, both distributions become identical after transformation, and the feature's SWIFT score is 0 — no matter how dramatically its raw distribution shifts. A feature the model ignores cannot raise a drift alarm; if you need to monitor such features anyway, that is a job for classical input monitoring, not for SWIFT.

Buckets details the interval semantics, the data model (Bucket, BucketSet, BucketType), and a worked example.

Stage 3: SHAP normalization¶

For each bucket, SWIFT computes the mean SHAP value of all reference observations falling in that bucket, using SHAP values from shap.TreeExplainer on the reference data. This defines the transformation

\[ \sigma_j(x_{ij}) = \overline{\text{SHAP}}_j^{\,\text{bucket}(x_{ij})}, \]

which maps any value of feature \(j\) to the mean SHAP of its bucket. After fit(), the reference distribution is fully characterized: each feature has a set of buckets with associated mean SHAP values — for the PD model, each bucket of utilization now carries the average log-odds contribution the model assigns to applicants in that range.

Two details matter in production:

Empty buckets. A bucket with zero reference observations gets a synthetic fill: n_synthetic observations are created by sampling real reference rows and placing the feature value inside the empty bucket, and SHAP is computed on those synthetic rows. (Without a model available, the bucket falls back to mean_shap = 0.0 with a warning.)
Missing values. NaN values map to the null bucket, so missingness has its own SHAP level and a shift in the missing-value rate is visible to the test.

How the synthetic fill works

The model learned a split, but the reference sample has no observation on one side of it — common for the null bucket of a fully populated feature, or for extreme intervals. Skipping the bucket is not an option: a monitoring observation may land there later and needs a defined SHAP level. So SWIFT manufactures one. It samples n_synthetic real reference rows (default 10, with replacement, from the seeded generator), overwrites only the one feature with a value inside the empty bucket — NaN for the null bucket; for numeric intervals the midpoint of a bounded interval, or one unit inside a half-bounded one — and runs shap.TreeExplainer on those rows. The mean of the feature's SHAP column becomes the bucket's level. Keeping all other columns real keeps the synthetic rows near the data distribution; if SHAP computation fails, the bucket falls back to mean_shap = 0.0 with a warning.

transform() applies \(\sigma_j\) to every column — vectorized via np.searchsorted on the sorted decision points, in \(O(n \log k)\) per feature — and this is the crucial invariant of the whole pipeline: \(\sigma_j\) is computed once, from the reference sample, and applied identically to every sample afterwards. It is never recomputed for monitoring data or inside the permutation loop. Recomputing it per window would let the yardstick drift along with the data — the comparison is only meaningful because both samples are measured against the reference model-view.

Normalization covers the transformation, the synthetic fill, and the vectorized assignment in depth.

Stage 4: Wasserstein distance in SHAP space¶

Drift is measured as the Wasserstein distance between the SHAP-transformed reference distribution and the SHAP-transformed monitoring distribution — this per-feature distance is the SWIFT score. Both orders are supported via the sorted-quantile formula

\[ W_p^p(P, Q) \;=\; \frac{1}{N} \sum_i \bigl| F_P^{-1}(i/N) - F_Q^{-1}(i/N) \bigr|^p , \]

so unequal sample sizes are handled correctly (for equal-size samples and \(p=1\) there is a fast path: the mean absolute difference of sorted values).

W1 (order=1) is the earth mover's distance — the \(L_1\) area between CDFs, robust and interpretable as the average shift in SHAP space.
W2 (order=2) is the root of the integral of squared CDF differences — more sensitive to variance changes and large deviations.

Why a transport distance rather than a bin-count divergence like PSI? The transformed values live on a meaningful scale — SHAP units, i.e. model-output contribution — and the Wasserstein distance uses that scale: it weights how far mass moved, not just that it moved between bins. Mass shifting between two buckets with nearly equal mean SHAP contributes almost nothing; the same mass crossing a large SHAP gap contributes a lot. A bin-count divergence would treat both identically.

Unequal sample sizes and the quantile grid

Reference and monitoring windows rarely have the same size. For the general case, the empirical quantile functions of both samples are evaluated on the merged grid of their CDF levels, and the quantile differences are integrated with the grid spacings as weights — an exact evaluation of the distance between the two empirical distributions, with no resampling and no binning. The equal-size \(W_1\) fast path (mean absolute difference of sorted values) is a special case of the same formula.

Because the comparison happens after the SHAP transformation, a large shift in a region where the model's SHAP response is flat produces a small distance, while a small shift across a bucket boundary with a big SHAP jump produces a large one. That is the "model-aware" part of SWIFT in one sentence. In the running example: income inflating uniformly within flat-response regions barely moves the score; a modest migration of utilization across the 0.42 split the model leans on shows up immediately.

SWIFT score covers the formula, both computation paths, and how to read the scores.

Stage 5: permutation test and correction¶

A raw Wasserstein distance has no natural threshold for "significant". SWIFT calibrates it with a permutation test: under the null hypothesis of no drift, reference and monitoring observations are exchangeable, so the pooled sample is repeatedly split at random into two groups of the original sizes and the SWIFT score recomputed each time. The observed score is then ranked against this null distribution. The p-value uses the conservative formula

\[ p_j \;=\; \frac{1 + \#\bigl\{\,b : \text{SWIFT}_j^{(b)} \ge \text{SWIFT}_j^{\text{obs}}\bigr\}}{1 + B}, \]

where \(B\) is n_permutations — so p-values live in \((0, 1]\) and can never be exactly zero.

A permutation test fits this statistic where a parametric test would not: the null distribution of a Wasserstein distance between SHAP-transformed samples — a discrete distribution over bucket levels, with sizes and bucket layouts varying per feature — has no usable closed form. Simulating the null by re-splitting the pooled data sidesteps distributional assumptions while conditioning on the actual data at hand.

What the permutation loop does — and does not — recompute

Each permutation recomputes only the Wasserstein distance. The pooled data is transformed with \(\sigma_j\) once before the loop; a permutation is just a shuffled index split into the pre-transformed arrays. SHAP is never recomputed — the transformation is part of the fitted reference view, not of the sample being tested. When max_samples is set and the pool exceeds it, both samples are randomly subsampled (preserving the reference/monitoring ratio) before the loop — a significant speedup on large datasets with negligible impact on statistical power.

Because every feature is tested simultaneously, multiple testing correction is applied to the \(p\) per-feature p-values. With dozens of features at \(\alpha = 0.05\), uncorrected testing would hand the running example's team a steady stream of false monthly alarms:

Benjamini–Hochberg ("benjamini-hochberg", aliases "bh", "fdr") controls the false discovery rate: sort the p-values, find the largest \(k\) with \(p_{(k)} \le k\alpha/p\), and flag all features of rank \(\le k\). Recommended default.
Bonferroni ("bonferroni", alias "bonf") controls the family-wise error rate: flag feature \(j\) when \(p_j < \alpha/p\). More conservative.

An alternative calibration also ships: bootstrap_threshold draws bootstrap samples of the monitoring size from the reference, computes SWIFT scores against the reference, and returns each feature's \((1-\alpha)\) quantile as an absolute score threshold — a principled alternative to PSI's ad-hoc 0.10 / 0.25 cutoffs.

Testing covers the test mechanics, both correction rules, and the bootstrap alternative.

Aggregation and the result¶

Per-feature scores are finally aggregated to model level: SWIFT_max (the most-drifted feature) and SWIFT_mean (average feature drift), with an optional importance-weighted SWIFT_weighted available through aggregate_scores and compute_importance_weights (weights are normalized mean \(|\text{SHAP}|\) per feature).

Everything lands in a SWIFTResult — the object the monthly monitoring job actually inspects:

Field / property	Meaning
`feature_results`	tuple of per-feature `FeatureSWIFTResult` records — each with `feature_name`, `swift_score`, `wasserstein_order`, `p_value`, `is_drifted`, `num_buckets`
`swift_max`	maximum SWIFT score across features
`swift_mean`	mean SWIFT score across features
`swift_weighted`	importance-weighted score (`None` unless computed with explicit weights)
`alpha`	significance level used for testing
`correction_method`	the `CorrectionMethod` applied
`num_features`	number of features monitored
`num_drifted`	number of features flagged as drifted
`drifted_features`	names of the flagged features

The getting started page shows how to read it, and Aggregation covers the roll-up and both result types in depth.

Where to go next¶

Models and decision points — Stage 1: extraction per library, unsupported models, unused features.
Buckets — Stage 2: interval semantics, the null bucket, the data model.
Normalization — Stage 3: the transformation and the synthetic fill.
SWIFT score — Stage 4: the distance and how to read it.
Testing — Stage 5: permutation test, corrections, bootstrap thresholds.
Missing values — the null bucket end to end: what a NaN is worth and why missingness shifts register as drift.
Aggregation — the model-level verdict and the result object.
Quickstart tutorial — the whole pipeline as a runnable notebook.
Configuration — every SWIFTMonitor parameter and how to tune the permutation test.
scikit-learn integration — estimator/transformer API and sample-vs-sample comparison.
Visualization — bucket profiles and score plots.
API reference.