Why Attribution Stability Matters More Than Attribution Accuracy

Enterprise teams treat SHAP and LIME explanations as ground truth. They aren't — they're samples from a distribution that varies with background data, sampling strategy, and runtime conditions. For regulated AI, the property that actually matters under audit isn't whether the explanation is right. It's whether it's stable enough to defend. A credit risk team at a UK retail bank deploys an imbalance-aware fraud detection model. The model uses TreeSHAP to generate feature attributions for every flagged transaction, attached to the case file for compliance review. Six months later, a regulator challenges a specific decision. The compliance team pulls the original case file, presents the attribution: "the decision was driven by transaction velocity, geographic anomaly, and merchant category code." The regulator asks the team to rerun the attribution. They do, using the same model, same input, same TreeSHAP implementation. The ranked feature importances come back in a slightly different order. Transaction velocity drops two ranks. Merchant category jumps three. The team scrambles. The model didn't change. The input didn't change. What changed? The background sample. TreeSHAP — like all SHAP variants — computes attributions as conditional expectations over a reference distribution. That reference distribution is sampled from the training data. Sample a different background, get a different attribution. The model's actual decision didn't shift; the explanation of the decision did. The team has produced an audit defense artifact that another draw of the same procedure can contradict. This is the gap that enterprise AI governance is about to discover, the hard way. What attribution actually measures The conceptual foundation of SHAP attribution is cooperative game theory — specifically, Shapley values, which fairly distribute the contribution of each player to a team's outcome. Applied to ML, "players" are features and "outcome" is the prediction. SHAP attributes how much each feature contributed to the difference between the model's prediction for this specific input and the model's expected prediction over some reference distribution. That last clause matters more than it appears. The reference distribution is the background sample — typically a few hundred observations drawn from the training data. SHAP doesn't tell you "feature X matters this much." It tells you "feature X contributed this much relative to what the model would have predicted on average over this specific background sample. " Different background sample, different baseline, different attribution. For deterministic ML models with stable feature distributions, this dependency is small enough to be tolerable. For models trained under class imbalance — where the minority class has been synthetically oversampled with SMOTE or similar techniques — the dependency becomes substantial. The training distribution and the deployment distribution diverge; the background sample's representativeness becomes unreliable; the attribution variance grows. The σ_SHAP measurement In my recent work on imbalance-aware financial distress prediction (preprint at arXiv:2605.14067), I formalised this property as a measurable quantity. I call it σ_SHAP — the across-repeat variance of mean-absolute SHAP attribution under rotated background sampling. Formally: σ_SHAP = √(m⁻¹ Σⱼ Varₖ(|φⱼ,ₖ|)) where φⱼ,ₖ is the attribution for feature j computed against the kᵗʰ background-sample draw, and the variance is taken across K independent background draws. The square root produces a scalar measure of attribution instability in the same units as the attributions themselves. The methodology in the paper measures this across K = 50 rotated background samples of 100 observations each, on the public Taiwan Bankruptcy Prediction benchmark (UCI ID 572, CC BY 4.0). All random states are explicitly seeded. The full reproducibility infrastructure is on Zenodo (DOI 10.5281/zenodo.20454212) — any third party can run the exact protocol and reproduce the numerical results bit-for-bit. What the measurement reveals is consequential. The variance of SHAP attributions across rotated backgrounds is substantial — far from negligible. And — this is the finding that matters most for production deployments — SMOTE-based imbalance handling materially amplifies the instability. Models trained on SMOTE-oversampled data produce attributions whose variance under rotated backgrounds is higher than models trained on the original imbalanced data. The standard production response to class imbalance — synthetic minority oversampling — degrades the audit-defense property of the resulting model. This is not a theoretical concern. It's a measured engineering reality on a public benchmark, with a public reproducibility package, that any team can verify against their own production pipeline. Why this matters under regulatory challenge Compliance teams reading the UK PRA's Supervisory Statement SS1/23 on model risk management notice the requirement for "sufficient validation evidence" and "effective challenge" of model decisions. They typically interpret this as a documentation requirement — generate explanations, attach them to case files, retain them for audit windows. Engineering teams implementing this interpret it as "run SHAP once per decision and store the output." Both interpretations miss what's coming. The PRA's enforcement mechanism is the right of supervisory challenge on specific decisions. A regulator selects a case, requests the firm to demonstrate the validation evidence, and tests whether the explanation is defensible. If the firm reruns the attribution and gets a materially different answer, the defense collapses — not because the model is wrong, but because the explanation was a point estimate from a distribution that the firm never measured. The European Banking Authority's machine learning guidelines apply similar pressure, requiring documented reasoning behind individual model decisions in credit risk applications. The EU AI Act's high-risk system requirements call out interpretability and reproducibility as compliance baselines. The FCA's emerging AI-in-financial-services guidance emphasizes decisions being reconstructible on challenge. None of these instruments specify "measure attribution variance under rotated backgrounds." They specify outcomes — and attribution variance is the property that determines whether those outcomes can actually be delivered. The same logic applies in ESG and climate-risk reporting domains. In a companion arXiv preprint on auditable ESG validation (arXiv:2606.02604), I developed a synthetic ESG validation benchmark covering six explicit governance failure modes — provenance_conflict, stale_reporting, climate_mismatch, null_inflation, transition_divergence, audit_inconsistency — calibrated against publicly reported characteristics of the GHG Protocol, PCAF, and ISSB reporting standards. The benchmark uses TreeSHAP-based interpretability for the validation pipeline. The same attribution-stability question applies: under UK Sustainability Disclosure Requirements (SDR) compliance review, will the firm's explanation of why a specific record was flagged as a provenance_conflict reproduce on re-running with a different background sample? What stable attribution actually looks like The honest engineering implication isn't that SHAP is broken or shouldn't be used. It's that single-shot attribution as currently deployed in production is constructing audit artifacts that the underlying methodology doesn't support. Stable attribution requires three changes to the production deployment pattern: 1. Measurement under variance, not as point estimate. Production attribution pipelines should compute attributions across multiple background samples — typically 30 to 100 independent draws — and report not just the mean attribution but the variance across draws. A regulator should be able to see "feature X contributes this much, with this much uncertainty" — not just a single number. 2. Fixed background protocol for audit-defense artifacts. If a specific attribution is being attached to a case file for compliance retention, the background sample used to generate it should be itself versioned, persisted, and reproducible. This converts attribution from a stochastic process into a deterministic one for audit purposes — the firm commits to a specific background, records its identity, and any re-run uses the same background. This is engineering discipline, not technical sophistication. 3. Attribution stability as a model-selection criterion. When evaluating candidate models, attribution stability under runtime conditions should be an evaluation dimension — alongside predictive accuracy, calibration, and fairness. Models that produce stable attributions under realistic deployment-time variation are operationally more valuable than models that produce slightly more accurate but materially less stable attributions. This trade-off is currently invisible in most model selection pipelines because nobody measures the stability dimension systematically. In the imbalance-aware distress modeling work, the model selection trade-off was concrete: SMOTE-augmented ensemble gradient boosting achieved the highest aggregate predictive performance, but produced the least stable attributions. Models trained on the original imbalanced data without SMOTE achieved slightly lower predictive performance but materially better attribution stability. For a regulated deployment where audit-defense is binding, the latter is the operationally correct choice — even though it would lose a standard model-selection contest scored on F1 alone. The engineering primitives Practical implementation of stable attribution requires modest infrastructure but explicit discipline. The primitives: Seeded background sampling. Every background draw uses an explicitly seeded random state, recorded in the deployment fingerprint. Reruns use the same seed. Rotated background measurement at evaluation time. The model is evaluated not just on accuracy and calibration but on σ_SHAP — the variance of attributions across K rotated background draws. The evaluation protocol fixes K, fixes the background size, fixes the rotation seed schedule. Persisted background identity at runtime. When attribution is computed for a production decision, the background sample identity (hash of the actual observations used) is persisted alongside the attribution, the model version, and the input. Audit reruns use the persisted background. When a regulator challenges a specific decision, the firm reruns SHAP against the persisted background — not a fresh random sample. The attribution reproduces exactly. None of these are computationally expensive. None require novel research. All of them require deliberate engineering choices that most production deployments currently aren't making. What this means for governance platforms The current generation of LLM governance platforms — Bifrost, Kong AI Gateway, Cloudflare AI Gateway, and similar — handle access control, policy enforcement, and runtime telemetry effectively. Where they fall short is on the measurement side: most platforms produce attribution outputs as point estimates, without variance characterisation, without persisted background identity, without the engineering primitives that would make those attributions defensible under regulatory challenge. This isn't a critique of the platforms. It's an observation about the gap between current governance-platform capabilities and the audit-defense requirements that regulated industries will face. Closing the gap requires the platforms to expose σ_SHAP-style stability measurements as first-class outputs, and requires deploying teams to use those measurements rather than ignore them. The firms that get there first — by measuring attribution stability at deployment time, persisting background identities at runtime, and treating attribution as a measured property rather than a generated artifact — will have audit-defense capabilities the rest of the industry will be retrofitting in five years. The trade-off is real: production deployments will report attribution alongside its uncertainty, and the uncertainty will sometimes be uncomfortably large. The honest engineering response is to measure the discomfort, report it, and design around it — not to suppress it by returning a point estimate that the underlying method doesn't actually support. Audit defense isn't about producing better explanations. It's about producing explanations whose stability you can measure and defend. That's an engineering problem, not a governance problem. And it's solvable — if you measure what's actually happening, rather than what you wish were happening.