Explainability: knowing why a model said what it said

A model that predicts next-week returns with a respectable out-of-sample R² is a candidate, not an answer, and the next question a careful researcher asks is not “how good is it” but “why does it think what it thinks.” Getting that second question wrong — trusting a plausible-sounding explanation that the model’s actual mechanics do not support — is a common way an otherwise sound piece of research turns into an unfounded story. This article covers the methods available for answering it honestly, computed out of sample, and where each one runs out of road.

Coefficients: the honest baseline

For a linear model, the coefficient on each feature is itself an explanation, in the narrowest and most literal sense: it states how much the predicted value changes for a one-unit change in that feature, holding the others constant. A logistic regression predicting the probability of a positive next-day return with a coefficient of 0.42 on a momentum feature says exactly that — nothing more, and nothing about causation.

The value of coefficients is their transparency: there is no approximation step between the model and the explanation, because the coefficient is the model. The limitation is that this transparency does not survive contact with correlated features. If two momentum features built over slightly different lookback windows are highly correlated, the model can split credit between them arbitrarily — one run assigns most of the coefficient weight to a 20-day window, a nearly identical run on a resampled dataset assigns it to a 25-day window, and neither coefficient individually reflects the feature’s true relationship to the target. Coefficients also say nothing at all about non-linear relationships, which is most of what a tree-based or neural model exists to capture.

Permutation importance

Permutation importance works with any model, including ones with no interpretable internal structure at all — random forests, gradient boosting, neural networks. The method is direct: take a trained model, measure its performance on a held-out test set, then shuffle one feature’s values across the test rows (breaking whatever relationship that feature had to the target while leaving every other feature and the target untouched) and measure performance again. The drop in performance is the feature’s importance — a feature the model relies on heavily produces a large drop when scrambled, and a feature the model ignores produces close to none.

A worked example. A classifier achieves 0.68 ROC AUC on held-out data. Shuffling a 20-day momentum feature drops that to 0.61 — a fall of 0.07. Shuffling a trading-volume feature drops it only to 0.665 — a fall of 0.015. The momentum feature is roughly 4.5 times as important to this model’s predictions as the volume feature, measured by how much the model’s actual out-of-sample performance depends on it.

Its main caveat mirrors the one for coefficients: correlated features share importance in ways that can understate both. If two features carry near-identical information, shuffling either one alone barely hurts performance, because the model can lean on the other — each individually looks unimportant even though the pair, together, may be doing most of the work. Permutation importance also measures association with predictive performance, not causal influence on the underlying market process; a feature can rank highly because it happens to correlate with the true driver of the target in the test window, not because it is that driver.

Partial dependence

Where permutation importance answers “how much does the model rely on this feature,” partial dependence answers a different question: “what does the model predict as this feature varies, holding the distribution of every other feature fixed?” The method computes the model’s average prediction across the dataset for a grid of values of one feature, with every other feature left at its observed values — effectively simulating “what if this one input had been different, all else equal.”

A worked example. A regression model’s partial dependence on a volatility feature might show predicted next-week return declining smoothly from +0.3% at low volatility levels to -0.1% at high volatility levels, with a visible kink around the feature’s 80th percentile — a pattern consistent with the model having learned some form of volatility-regime sensitivity, and a genuinely useful diagnostic for whether that sensitivity is monotonic, threshold-like, or noisy.

Partial dependence assumes the feature being varied is not strongly correlated with the others being held fixed. When it is — a volatility feature and a drawdown feature that move together in practice — the method evaluates the model at combinations of feature values that never actually occur in real data (a hypothetical “very high volatility, very low drawdown” observation, for instance), and the resulting curve can describe model behaviour on inputs the model was never meaningfully trained to handle in that combination.

What these methods cannot tell you

Permutation importance and partial dependence are both model-agnostic and both computed on the same pipeline the model was trained and validated on — that consistency is the point, and it is why they are preferred over methods that require inspecting a model’s internal architecture. No SHAP or LIME-style additive attribution method is part of this toolkit; the explanations available are limited to these two global, out-of-sample diagnostics and the model’s own coefficients where they exist.

This distinction matters most when explainability output is used to justify a decision rather than to audit one. A feature ranking highly by permutation importance is a reason to keep investigating it — check whether it holds up on a different time period, whether it survives the multiple-testing corrections discussed alongside overfitting detection, and whether its apparent importance decays once correlated features are addressed — not a reason to conclude the market genuinely responds to it. Explainability output should be read together with the risk-adjusted performance metrics covered in risk and return metrics that matter: a feature can look important to a model that is not actually predicting anything useful, if the model’s overall out-of-sample skill is close to a naive baseline in the first place.

Explainability also has nothing to say about whether a strategy built on the model is fit to trade. A feature can be genuinely important to a model’s predictions and the resulting strategy can still fail once transaction costs, slippage, and realistic position sizing are applied — questions covered separately once an allocation exists, as in portfolio optimisation from first principles. This workspace does not place trades on the basis of a model’s output, explainable or otherwise: explainability is a research and analysis tool for understanding a model that has already been trained, not a mechanism that acts on that understanding.

Key takeaways

Linear coefficients are the most transparent explanation available, but they split credit arbitrarily among correlated features and say nothing about non-linear relationships.
Permutation importance measures how much a model’s out-of-sample performance depends on a feature by shuffling it and observing the resulting performance drop, always computed on held-out data.
Partial dependence shows how a model’s average prediction changes as one feature varies, holding others fixed, but can describe model behaviour on unrealistic feature combinations when features are correlated.
No SHAP or LIME-style attribution method is available; explainability here means permutation importance, partial dependence and, where applicable, a linear model’s own coefficients.
Neither method establishes causation — both describe what the trained model does with historical data, and high importance is a reason to investigate further, not a conclusion in itself.