01 — Causality

Causality and ML-2

Key terms

  1. Adjusting - A key operation in causal inference. Adjusting a variable means fixed values of that variable are maintained so that the causal effects of other variables can be measured.
  2. Backdoor path - A backdoor would open a spurious path between two variables, showing a causal effect between two variables which is not there in the real world. These paths must be closed through ‘adjusting’ operations. Here, I’m using the term ‘two variables’ as a path is normally defined between two endpoints.
  3. Frontdoor path - Frontdoor path indicates a true causal effect between two variables. There can be other, non-blocking variables in the middle of this path which are called mediators. These paths must be open.
  4. Collider path - This path is naturally closed/blocked.

Causal roles of variables

  1. Confounder - confounders are variables that create backdoor paths. Adjusting it will block this path - which is essential in measuring causal effects.
  2. Mediator - mediators create frontdoor paths. These variables should not be adjusted as it would hide the true causal effects.
  3. Collider - colliders create neither. They are causal outcomes. Adjusting them would create a backdoor path.

A nice graphical explanation can be found in [1].

Correlation is not Causation

This is a famous truism or sometimes a cliche. In predictive machine learning paradigms, the relationship between features and outcomes is often correlational (unfortunately). Let us take an example of supervised learing where the model predicts labels (classification task). A predictive model often involves many variables without accounting for their causal roles. Such involvement can lead to false interpretations of the cause-effect relationships between the variables and outcomes. For example, including a mediator variable ($M$) in a linear regression model would be misleading as it may imply that the mediator has causal effect on the outcome more than it deserves (indicated by its weight) as the true effect comes from $X$.

$Y=\alpha_{1}X+\alpha_{2}M$

Let us try to understand the issue with standard predictive models at a deeper level. The objective of these models is ‘empirical risk minimization’, or simply to find a model (e.g., a set of weights) that would minimize a certain ‘loss’ for a given dataset.

$min_{\theta}~\mathop{E}[\mathcal{l}(f_{\theta}(X),Y)]$

This prefers variables that would help the model to lower the loss efficiently, but they are not guaranteed to be the true causal variables. In other words, predictive models learn the conditional probability of a given dataset such that,

$P(Y X)$ where $X$ can be $(X_1,X_2,X_3,X_4,…)$

This learning does not account for the cause-effect relationships and rely on the correlations (whichever variable that correlates with the outcome). Such correlations may change with environment, time or some other external factors which end up the model breaking under out-of-distribution scenarios. If we can build a model that learns about the mechanisms of a data generating process (in terms of cause-effect relationships) then they would be robust to out-of-dsitribution scenarios, as now the model is grounded on cause-effect relationships and not just correlations. Still, there should be an assumption which is that the underlying mechanism should remain the same.

Simpson’s Paradox

A good motivation to look beyond traditional predictive ML models comes from the Simpson’s paradox. In simple terms, the Simpson’s paradox shows that paradoxes emerging in the same dataset, when combining certain groups. It shows that adjusting for confounding variables can provide wrong insights about the data. In other words, neglecting the causal structure would make the patterns all wrong, thus misleading the ones who use them.

N.B. - I later found out this amazing resource (https://alexdeng.github.io/causal/simpson.html) which has thoroughly explained this and other related concepts.

Invariant Risk Minmization (IRM)

IRM is a recent paradigm for ML model learning that accounts for causal mechanisms (in contrast to empirical risk minimization) [2]. The main goal of IRM is to achieve generalization for out-of-distribution data.

Causal Models

Rubin

Pearl

Reference

[1] - Control of Confounding and Reporting of Results in Causal Inference Studies. Guidance for Authors from Editors of Respiratory, Sleep, and Critical Care Journals, Leiderer et al., https://pubmed.ncbi.nlm.nih.gov/30230362/ [2] -