SHAP’s Interpretive Boundary: From Model Attribution to Causal Identification

An earlier note used two papers from Nature/Science-family journals to discuss the interpretive boundary of SHAP. It prompted a useful amount of discussion, including criticism, extensions, and republication by other channels. That response made one point clearer: whether SHAP is being over-interpreted is not a problem confined to a few papers. It is a recurring epistemic problem in current scientific practice.

After rereading the comments, I became less satisfied with framing the issue simply as “misuse.” The more useful question is how to classify the claim being made. This essay therefore recasts the boundary of SHAP in three progressively stronger levels: model-based attribution, process-informed interpretation, and causality-informed identification.

1 · Model-based attribution

This is SHAP's most basic and most defensible level. SHAP decomposes model output f(x), not the real process itself. It provides local and global attributions of input features to a prediction; it does not measure the ontological effect of those variables in the world.

At this level the object of explanation is representation, not phenomenon; model structure, not world structure.

Used within that boundary, SHAP is a strong interpretive tool. It can tell us which inputs the model relies on, under what conditions its prediction changes, and which features have larger marginal contributions in the fitted model.

The language should therefore remain at the same epistemic level: “In the model trained for this study, variable X makes a large contribution to the prediction.” It should not be rewritten automatically as “X is a key mechanism of system change” or “X drives Y” (Lundberg, SHAP Documentation, 2018).

2 · Process-informed interpretation

In geoscience, ecology, and hydrology, SHAP is rarely used in a complete absence of theory. Researchers often enter the analysis with process knowledge, regime distinctions, and domain constraints. They may already know which variables are plausible candidate drivers, which dependence structures change under different regimes, and which apparent relations are more likely to represent coupling or confounding.

Under those conditions, model attribution can be interpreted within an established process framework. One can ask whether the attribution is consistent with known mechanisms, whether it shows theoretically plausible patterns within a specific regime, and whether it helps refine current process understanding. Interpretable machine learning can in this sense help reveal how inputs interact in a model and can support process understanding, hypothesis generation, and model evaluation (Jiang et al., Earth's Future, 2024).

But process-informed interpretation is still not causal discovery.

It remains an attribution derived from a predictive model fitted to observational data. Domain knowledge supplies mechanistic plausibility; process theory imposes interpretive discipline. Those constraints can improve credibility and scientific readability, but they do not automatically rule out confounding, proxy dependence, selection effects, or shortcut learning.

Process-informed interpretation is therefore theory-calibrated support, not an explicit identification strategy.

3 · Causality-informed identification

The third level begins when the question changes from “What has the model learned under the observed distribution?” to “What would happen to Y if X were intervened upon?” Predictive modelling addresses observational conditional probabilities; causal analysis addresses interventional probabilities. The epistemic and methodological tasks are different.

SHAP's own documentation states the issue plainly: SHAP can make correlations learned by predictive models transparent, but “making correlations transparent does not make them causal.” Standard SHAP on a predictive model therefore does not perform causal identification.

Stronger causal claims require additional structure: inputs free of relevant unobserved confounding, a structural causal model, a quasi-experimental design, double machine learning under defensible assumptions, causal-discovery procedures, or another explicit identification framework.

The Nature Plants paper When and where soil dryness matters to ecosystem photosynthesis, suggested by several readers, is a good example because it uses causal SHAP. The important point is that the causality does not come from SHAP itself. It comes from a causal structure introduced before the attribution step: the distinction between water-limited and energy-limited regimes, causal-chain graphs among variables, and attribution defined with respect to interventional conditional probabilities in the language of Pearl's do-calculus.

SHAP is therefore not the source of the causal conclusion. It is a device for expressing and decomposing effect attribution inside a causal framework that has already made substantive commitments.

The burden in causality-informed identification lies in those commitments. Is the causal graph defensible? Is the regime partition reliable? Is the intervention object well defined? Has confounding actually been treated structurally? The methodological value of such work is that it does not quietly assign causality to SHAP. It makes the causal assumptions explicit. The methodological cost is that the paper must then be accountable for them.

Figure 1 · The causal structure is not supplied by SHAP. The figure shows regime-specific causal relations between GPP and major drivers, together with threshold responses linking soil moisture and ecosystem functioning (Liu et al., Nature Plants).

Closing note

The point of revisiting SHAP is not to argue against its use. It is to preserve the distinctions among these three levels.

SHAP is fully valid as model-based attribution. It can also be highly useful for process-informed interpretation, provided the claim remains model-consistent support rather than mechanism validation. Only under causality-informed identification can a study discuss intervention-relevant effects, and even then causality comes from explicit causal graphs, identification assumptions, and structural commitments—not from SHAP itself.

The real error is collapsing these levels into one another. Once that happens, contribution becomes “driver,” driver becomes “effect,” and model attribution is quietly read as world causation. That is not merely a wording problem. It is a boundary of scientific inference.

References

• Jiang, S., Sweet, L.-B., Blougouras, G., Brenning, A., Li, W., Reichstein, M., Denzler, J., Shangguan, W., Yu, G., Huang, F., & Zscheischler, J. How interpretable machine learning can benefit process understanding in the geosciences. Earth's Future, 12, e2024EF004540 (2024).
• Lundberg, S. SHAP Documentation (2018).
• Liu, J. et al. When and where soil dryness matters to ecosystem photosynthesis. Nature Plants.