Benchmark Leaderboard#

This notebook is the leaderboard: every number the docs publish is produced by running it end to end, so the table cannot drift from something nobody can reproduce.

The estimators are scored on three families of data, and the family decides which metric exists:

family	ground truth	metrics
simulated (`synthetic_data`)	per-unit `tau`	PEHE, root PEHE, `ate_error`
semi-synthetic (IHDP)	simulated `mu0` / `mu1`	PEHE, root PEHE, `ate_error`
experiment (LaLonde)	experimental ATE only	`ate_error` against that estimate

A ranking on simulated outcomes is a statement about that simulation, not about which estimator is best on your data. The point of the table is to make the comparison reproducible, not to crown a winner.

The simulated section needs no download. The IHDP and LaLonde sections fetch from their original sources and skip themselves cleanly when that is unavailable — see Benchmark Datasets for where each comes from.

[1]:

import warnings

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from xgboost import XGBRegressor

from importlib.metadata import version

from causalml.dataset import fetch_ihdp, fetch_lalonde, synthetic_data
from causalml.inference.meta import BaseSRegressor, BaseTRegressor, BaseXRegressor
from causalml.inference.tree import CausalTreeRegressor
from causalml.metrics import ate_error, pehe

warnings.filterwarnings("ignore")

SEED = 42
print("causalml", version("causalml"))

/Users/jeong/dev/causalml/.venv/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
Failed to import duecredit due to No module named 'duecredit'

causalml 0.17.0

The estimators#

One representative of each family, all with the same base learner so the comparison is between the meta-learner designs rather than between boosting configurations. CausalTreeRegressor is included as the tree-based contrast.

[2]:

def make_learners(seed=SEED):
    """A fresh, unfitted estimator per name -- fitted models must not be reused."""
    base = lambda: XGBRegressor(n_estimators=100, max_depth=3, random_state=seed, verbosity=0)
    return {
        "S-learner": BaseSRegressor(learner=base()),
        "T-learner": BaseTRegressor(learner=base()),
        "X-learner": BaseXRegressor(learner=base()),
        "Causal tree": CausalTreeRegressor(control_name=0, random_state=seed),
    }


def cate(learner, X_train, t_train, y_train, X_test):
    """Fit on the training rows and predict the effect on held-out rows."""
    learner.fit(X=X_train, treatment=t_train, y=y_train)
    return np.asarray(learner.predict(X_test)).ravel()

Simulated data#

synthetic_data returns the individual effect tau, so PEHE is defined. Every estimator is fitted on half the draw and scored on the other half, over several seeds, and the table reports the mean.

[3]:

def simulated_scores(mode=1, n=2000, sigma=1.0, seeds=range(5)):
    rows = []
    for seed in seeds:
        np.random.seed(seed)
        y, X, w, tau, _, _ = synthetic_data(mode=mode, n=n, p=5, sigma=sigma)
        X_tr, X_te, w_tr, w_te, y_tr, y_te, _, tau_te = train_test_split(
            X, w, y, tau, test_size=0.5, random_state=seed
        )
        for name, learner in make_learners(seed).items():
            tau_hat = cate(learner, X_tr, w_tr, y_tr, X_te)
            rows.append(
                {
                    "estimator": name,
                    "seed": seed,
                    "root PEHE": pehe(tau_te, tau_hat, squared=False),
                    "ate_error": ate_error(tau_te, tau_hat),
                }
            )
    return pd.DataFrame(rows)


simulated = simulated_scores()
simulated.groupby("estimator")[["root PEHE", "ate_error"]].mean().sort_values("root PEHE").round(4)

[3]:

	root PEHE	ate_error
estimator
S-learner	0.2974	0.1275
Causal tree	0.4158	0.3081
X-learner	0.6537	0.0814
T-learner	0.8035	0.1254

IHDP#

100 replications of the same 747 units, each with its own simulated outcomes and its own train/test split. The literature reports a mean and standard error across replications, so a single one is not comparable to a published number; the loop below is the unit of comparison.

Ten replications are used here to keep the notebook runnable. Published tables use all 100 — raise N_REPLICATIONS to reproduce one.

[4]:

N_REPLICATIONS = 10

def ihdp_scores(n_replications=N_REPLICATIONS):
    rows = []
    for replication in range(n_replications):
        train = fetch_ihdp(replication=replication, split="train")
        test = fetch_ihdp(replication=replication, split="test")
        for name, learner in make_learners(replication).items():
            tau_hat = cate(learner, train.data, train.treatment, train.target, test.data)
            rows.append(
                {
                    "estimator": name,
                    "replication": replication,
                    "root PEHE": pehe(test.tau, tau_hat, squared=False),
                    "ate_error": ate_error(test.tau, tau_hat),
                }
            )
    return pd.DataFrame(rows)


try:
    ihdp = ihdp_scores()
    summary = ihdp.groupby("estimator")[["root PEHE", "ate_error"]].agg(["mean", "sem"])
    display(summary.round(4))
except OSError as exc:
    print(f"IHDP unavailable, skipping: {exc}")

	root PEHE		ate_error
	mean	sem	mean	sem
estimator
Causal tree	5.4714	3.3433	0.4125	0.1641
S-learner	2.9277	1.9351	0.2899	0.1653
T-learner	2.6523	1.5387	0.3119	0.1772
X-learner	3.0879	1.9151	0.2576	0.1141

LaLonde#

A randomized experiment with no per-unit ground truth. What it licenses is a comparison against the experimental difference in means, about $1,794: an estimator run on the same experimental sample should recover it.

This is the weakest of the three tests — it says nothing about the individual effects — but it is the only one here computed on real outcomes.

[5]:

try:
    lalonde = fetch_lalonde()
    experimental_ate = (
        lalonde.target[lalonde.treatment == 1].mean()
        - lalonde.target[lalonde.treatment == 0].mean()
    )
    print(f"experimental ATE: {experimental_ate:,.2f}")

    rows = []
    for name, learner in make_learners().items():
        learner.fit(X=lalonde.data, treatment=lalonde.treatment, y=lalonde.target)
        tau_hat = np.asarray(learner.predict(lalonde.data)).ravel()
        rows.append(
            {
                "estimator": name,
                "ATE": tau_hat.mean(),
                "ate_error": ate_error(experimental_ate, tau_hat.mean()),
            }
        )
    display(pd.DataFrame(rows).sort_values("ate_error").round(2))
except OSError as exc:
    print(f"LaLonde unavailable, skipping: {exc}")

experimental ATE: 1,794.34

	estimator	ATE	ate_error
2	X-learner	1630.16	164.18
1	T-learner	1611.82	182.52
0	S-learner	1151.27	643.07
3	Causal tree	775.59	1018.75

Reading this table#

Three things it does not say.

It does not say which estimator is best. Every ranking above is conditional on one data-generating process, one base learner and one hyperparameter setting; IHDP in particular is a single simulation design that the literature has tuned against for a decade.

It does not say the differences are significant. The IHDP section reports a standard error across replications; the simulated section reports a mean over five seeds and no more.

And it says nothing about datasets with no ground truth, which is most real work. There, AUUC, Qini and RATE are what the library offers, and Validation covers them.

Regenerating#

Run this notebook top to bottom. It is refreshed each minor release and whenever an estimator’s defaults change — honesty=True becoming the causal-tree default moved every number in the tree row, which is exactly the kind of change that makes a stale table wrong rather than merely old.