What β Coefficients Don’t Tell You — Explaining Models with SHAP
1. Where Regression Analysis Falls Short
Regression analysis has long been one of the most trusted tools in social science research.
Using β coefficients to assess the direction and magnitude of variable effects, and p-values to confirm statistical significance — this workflow remains standard for most researchers.
But as data grows more complex, regression runs into problems that are hard to work around.
- Too many variables: multicollinearity can distort or even reverse coefficient signs.
- Nonlinear relationships: regression models struggle to capture the true pattern.
- Explaining individual cases: regression only shows “average effects” — it cannot go further than that.
In response, many recent studies have turned to machine learning models like Random Forest and XGBoost to uncover relationships and patterns in complex data. The problem is that these models are difficult to interpret — their internal logic is opaque. This is the well-known black-box problem.
Explainable AI (XAI) starts precisely here. Among XAI methods, SHAP (SHapley Additive exPlanations) is currently the most widely used in academic research. It quantifies how much each variable contributed to a given prediction, and expresses that contribution as a numerical value.