Skip to content

Normalization

Stage 3 is where SWIFT becomes model-aware. Buckets say where the model distinguishes values; this stage says how much each region matters: every bucket gets the mean SHAP value of the reference observations falling in it, and each feature value is then replaced by its bucket's mean SHAP. Observations in regions where the model relies heavily on a feature get large (absolute) transformed values; regions the model barely uses collapse toward zero.

from swift.normalization import compute_bucket_shap, transform_feature

bucket_sets = compute_bucket_shap(
    bucket_sets, X_ref, shap_values, model=model, n_synthetic=10
)
# Returns: dict[str, BucketSet] with mean_shap populated on each bucket

transformed = transform_feature(X[feature], bucket_sets[feature])
# Returns: 1-D float64 array — each value replaced by its bucket's mean SHAP

The transformation

For feature \(j\) and bucket \(k\), the mean SHAP is computed on the reference sample:

\[ \overline{\text{SHAP}}_j^{\,k} \;=\; \operatorname{mean}\bigl\{\, \text{SHAP}_j(x_i) : x_{ij} \in \text{bucket } k \,\bigr\}, \]

which defines the transformation

\[ \sigma_j(x_{ij}) = \overline{\text{SHAP}}_j^{\,\text{bucket}(x_{ij})}. \]

The SHAP values come from shap.TreeExplainer run on the reference data — during SWIFTMonitor.fit(), the explainer is built from the model you passed in and shap_values(X_ref) is computed once, then handed to compute_bucket_shap.

After this stage, the reference distribution is fully characterized: each feature has a set of buckets with associated mean SHAP values. This is the crucial invariant of the whole pipeline: \(\sigma_j\) is computed once, from the reference sample, and applied identically to every sample afterwards. It is never recomputed for monitoring data or inside the permutation loop.

Empty buckets and the synthetic fill

A bucket can be empty: the model learned a split at a threshold, but the reference sample happens to contain no observation in one of the resulting intervals (the null bucket of a feature with no missing values is the everyday case). An empty bucket has no SHAP values to average — yet a monitoring observation may land in it later, so it needs a level.

When a model is available (it always is inside SWIFTMonitor.fit()), SWIFT fills the bucket synthetically:

  1. Sample n_synthetic rows (default 10) from the reference data, with replacement, using the pipeline's seeded random generator.
  2. In each sampled row, overwrite the feature's value with one that falls inside the empty bucket:
    • null bucket → NaN;
    • numeric bucket → a representative point: the midpoint for a bounded interval \([l, u)\), \(u - 1\) for \((-\infty, u)\), \(l + 1\) for \([l, +\infty)\), and \(0.0\) for the unbounded catch-all;
    • categorical bucket → the first category of the bucket (not produced by the standard pipeline — see Buckets).
  3. Run shap.TreeExplainer(model).shap_values(...) on the synthetic rows and take the mean of the feature's SHAP column as the bucket's mean_shap.

Because the other columns are real reference rows, the synthetic observations stay close to the data distribution; only the one feature is forced into the empty region. If no model is provided, or SHAP computation on the synthetic rows fails, the bucket falls back to mean_shap = 0.0 with a warning.

Applying the transformation

transform_feature (and SWIFTMonitor.transform(), which applies it column by column) is vectorized:

  • NaN positions are mapped to the null bucket;
  • numeric values are assigned to buckets via np.searchsorted on the sorted decision points — \(O(n \log k)\) per feature for \(k\) thresholds;
  • the bucket index is then a table lookup into the per-bucket mean SHAP values (a bucket whose mean_shap was never populated contributes 0.0);
  • features with categorical buckets fall back to an element-wise path.

The output has the same length as the input — transform() on a DataFrame preserves shape and column names, so the SHAP-transformed data can be inspected, plotted, or fed to downstream tooling directly.

Next stage: the transformed reference and monitoring distributions are compared in SWIFT score. Full signatures: API reference — Normalization.