Honest causal trees: parity with grf and EconML#
CausalTreeRegressor and CausalRandomForestRegressor estimate leaves honestly by default (honesty=True, added in #584): the sample is split in two, the tree structure is grown on one half, and each leaf’s per-group outcome means are re-estimated on the other. A leaf value therefore does not inherit the selection bias of the split search that produced it (Athey and Imbens 2016).
Two mature implementations of the same idea already exist, and both also default it on:
option |
default |
held-out fraction |
|
|---|---|---|---|
|
|
|
|
|
|
fixed at one half |
|
CausalML |
|
|
|
CausalML also accepts ccp_alpha="cv", which adds the rest of the CT-H algorithm: the scaled variance penalty and cross-validated tree size. Every table below reports CausalML as a single tree and as a forest, at three settings each: in-sample, honest (the shipped default), and honest + CT-H. Section 4 measures CT-H on its own.
Four questions, measured on the same data:
On a randomized trial, do the three libraries’ honest forests agree? Pairwise correlation 0.92–0.97.
On a confounded design, do they agree? No. CausalML’s criteria compare raw group means with no adjustment for treatment assignment; inverse-propensity
sample_weightcorrects it.Does honesty remove spurious heterogeneity? Yes, in both CausalML and EconML, under a true null.
Does the full CT-H algorithm help? On a single tree, by 25–55%. On a forest, no, and at
sigma=0.5it is worse.
Requirements#
pip install causalml econml
# for the grf comparison (optional -- those cells skip themselves if R is missing):
brew install r # or your platform's R
Rscript -e 'install.packages(c("grf", "jsonlite"))'
Versions used to produce the stored output: CausalML (this branch), EconML 0.17.0, grf 2.6.1 on R 4.6.1, scikit-learn 1.9.0, NumPy 2.4.6.
[1]:
import json
import os
import shutil
import subprocess
import tempfile
import numpy as np
import pandas as pd
from causalml.dataset import synthetic_data
from causalml.inference.tree import CausalRandomForestRegressor, CausalTreeRegressor
from econml.grf import CausalForest
SEED = 42
N_TREES = 500
N = 4000
pd.set_option("display.width", 200)
print("honesty is on by default:", CausalTreeRegressor().honesty)
honesty is on by default: True
Harness#
Every estimator is fitted on the same training rows and scored on the same held-out rows against the known individual treatment effect tau, which the synthetic generator gives us. Held-out scoring matters here: honesty is about not overfitting the rows you estimated on, so an in-sample score would flatter exactly the thing being tested.
[2]:
def make_data(mode, n=N, p=5, sigma=0.5, seed=SEED):
"""Nie & Wager synthetic data from `causalml.dataset`, with a train/test mask."""
np.random.seed(seed) # synthetic_data draws from the global RNG
y, X, w, tau, b, e = synthetic_data(mode=mode, n=n, p=p, sigma=sigma)
test = np.random.RandomState(seed).rand(len(y)) < 0.3
return dict(X=X, y=y, w=w, tau=tau, e=e, train=~test, test=test)
def score(name, tau_hat, d):
"""Score a test-set CATE estimate against the true tau."""
tau_hat = np.asarray(tau_hat).ravel()
truth = d["tau"][d["test"]]
return {
"estimator": name,
"ATE": tau_hat.mean(),
"|ATE error|": abs(tau_hat.mean() - truth.mean()),
"CATE RMSE": np.sqrt(np.mean((tau_hat - truth) ** 2)),
"corr w/ true tau": np.corrcoef(tau_hat, truth)[0, 1],
}
# (label, honesty, ccp_alpha). "honest" is the shipped default; "CT-H" adds the
# scaled variance penalty and cross-validated tree size measured in section 4.
CAUSALML_ARMS = [
("in-sample", False, 0.0),
("honest", True, 0.0),
("honest + CT-H", True, "cv"),
]
def fit_causalml(d):
out = {}
for label, honesty, ccp_alpha in CAUSALML_ARMS:
common = dict(
control_name=0,
honesty=honesty,
ccp_alpha=ccp_alpha,
cv_folds=3,
min_samples_leaf=50,
min_group_samples=25,
random_state=SEED,
)
tree = CausalTreeRegressor(**common).fit(
X=d["X"][d["train"]], treatment=d["w"][d["train"]], y=d["y"][d["train"]]
)
forest = CausalRandomForestRegressor(
n_estimators=N_TREES, n_jobs=-1, **common
).fit(X=d["X"][d["train"]], treatment=d["w"][d["train"]], y=d["y"][d["train"]])
out[f"causalml tree ({label})"] = tree.predict(X=d["X"][d["test"]]).ravel()
out[f"causalml forest ({label})"] = forest.predict(d["X"][d["test"]]).ravel()
return out
def fit_econml(d):
out = {}
for honest in (False, True):
label = "honest" if honest else "in-sample"
cf = CausalForest(
n_estimators=N_TREES,
honest=honest,
min_samples_leaf=50,
random_state=SEED,
n_jobs=-1,
).fit(d["X"][d["train"]], d["w"][d["train"]], d["y"][d["train"]])
out[f"econml forest ({label})"] = cf.predict(d["X"][d["test"]]).ravel()
return out
The grf bridge#
grf is an R package, so these cells write the data to CSV, shell out to Rscript, and read the predictions back. If R or grf is missing the function returns an empty dict and the rest of the notebook still runs.
grf applies local centering by default: it first fits nuisance forests for E[Y|X] and E[W|X] and splits on the residuals. That is a separate idea from honesty, and it turns out to dominate the comparison on confounded data, so grf is run both ways — once with centering and once with it disabled by supplying constant Y.hat / W.hat.
[3]:
GRF_R = r"""
suppressMessages({library(grf); library(jsonlite)})
args <- commandArgs(trailingOnly = TRUE)
d <- read.csv(args[1])
p <- ncol(d) - 3
Xc <- as.matrix(d[, 1:p]); W <- d$W; Y <- d$Y; is_train <- d$train == 1
out <- list()
for (honesty in c(FALSE, TRUE)) {
for (centered in c(TRUE, FALSE)) {
a <- list(X = Xc[is_train, , drop = FALSE], Y = Y[is_train], W = W[is_train],
num.trees = as.integer(args[2]), honesty = honesty,
min.node.size = 50, seed = 42)
if (!centered) { # disable grf's default local centering
a$Y.hat <- rep(mean(Y[is_train]), sum(is_train))
a$W.hat <- rep(mean(W[is_train]), sum(is_train))
}
cf <- do.call(causal_forest, a)
key <- paste0("grf forest (", if (honesty) "honest" else "in-sample",
if (centered) "" else ", uncentered", ")")
out[[key]] <- as.numeric(predict(cf, Xc[!is_train, , drop = FALSE])$predictions)
}
}
cat(toJSON(out))
"""
def fit_grf(d, n_trees=N_TREES):
"""Fit grf's causal_forest in R; returns {} if R or grf is unavailable."""
if shutil.which("Rscript") is None:
print("Rscript not found -- skipping the grf comparison.")
return {}
frame = pd.DataFrame(d["X"], columns=[f"X{i}" for i in range(d["X"].shape[1])])
frame["W"], frame["Y"], frame["train"] = d["w"], d["y"], d["train"].astype(int)
with tempfile.TemporaryDirectory() as tmp:
csv, script = os.path.join(tmp, "d.csv"), os.path.join(tmp, "grf.R")
frame.to_csv(csv, index=False)
with open(script, "w") as fh:
fh.write(GRF_R)
proc = subprocess.run(
["Rscript", script, csv, str(n_trees)], capture_output=True, text=True
)
if proc.returncode != 0:
print("grf unavailable:", proc.stderr.strip()[:300])
return {}
return {k: np.asarray(v) for k, v in json.loads(proc.stdout).items()}
def compare(mode):
d = make_data(mode)
cate = {**fit_causalml(d), **fit_econml(d), **fit_grf(d)}
table = pd.DataFrame([score(k, v, d) for k, v in cate.items()])
return d, cate, table.set_index("estimator").round(4)
1. Randomized trial (mode=2)#
Treatment is assigned independently of X, so there is no confounding for any method to adjust away. This isolates the question the honesty option is actually about.
[4]:
d2, cate2, table2 = compare(mode=2)
print(f"true ATE on the test rows: {d2['tau'][d2['test']].mean():.4f}")
table2
true ATE on the test rows: 0.7842
[4]:
| ATE | |ATE error| | CATE RMSE | corr w/ true tau | |
|---|---|---|---|---|
| estimator | ||||
| causalml tree (in-sample) | 0.7881 | 0.0039 | 0.6265 | 0.8414 |
| causalml forest (in-sample) | 1.2123 | 0.4282 | 0.6715 | 0.9757 |
| causalml tree (honest) | 0.7451 | 0.0391 | 0.5578 | 0.8791 |
| causalml forest (honest) | 0.7492 | 0.0349 | 0.4095 | 0.9676 |
| causalml tree (honest + CT-H) | 0.7554 | 0.0287 | 0.5447 | 0.8815 |
| causalml forest (honest + CT-H) | 0.7536 | 0.0305 | 0.4524 | 0.9647 |
| econml forest (in-sample) | 0.7406 | 0.0435 | 0.5073 | 0.9127 |
| econml forest (honest) | 0.7469 | 0.0373 | 0.6062 | 0.8982 |
| grf forest (in-sample) | 0.7270 | 0.0572 | 0.3567 | 0.9549 |
| grf forest (in-sample, uncentered) | 0.7410 | 0.0432 | 0.3452 | 0.9583 |
| grf forest (honest) | 0.7360 | 0.0481 | 0.4964 | 0.9245 |
| grf forest (honest, uncentered) | 0.7511 | 0.0330 | 0.5131 | 0.9261 |
The in-sample CausalML forest reports an ATE above the truth and a higher CATE RMSE than the honest one: it reads its own split noise back as signal. CT-H tracks the honest arm closely here, on the tree and on the forest.
These are single fits on one draw, so differences between the honest rows are not meaningful. The reproducible quantity is the agreement between them, below.
[5]:
def agreement(cate, kind="honest"):
"""Pairwise correlation of the honest estimators' test-set CATE estimates.
Includes CausalML's single tree and its CT-H arms alongside the forests, so the
table shows how much of the disagreement is the estimator rather than the library.
"""
picked = {k: v for k, v in cate.items() if kind in k}
return pd.DataFrame(
np.corrcoef(np.vstack(list(picked.values()))),
index=list(picked),
columns=list(picked),
).round(3)
agreement(cate2)
[5]:
| causalml tree (honest) | causalml forest (honest) | causalml tree (honest + CT-H) | causalml forest (honest + CT-H) | econml forest (honest) | grf forest (honest) | grf forest (honest, uncentered) | |
|---|---|---|---|---|---|---|---|
| causalml tree (honest) | 1.000 | 0.887 | 0.961 | 0.886 | 0.793 | 0.846 | 0.850 |
| causalml forest (honest) | 0.887 | 1.000 | 0.890 | 0.999 | 0.922 | 0.970 | 0.979 |
| causalml tree (honest + CT-H) | 0.961 | 0.890 | 1.000 | 0.890 | 0.806 | 0.855 | 0.859 |
| causalml forest (honest + CT-H) | 0.886 | 0.999 | 0.890 | 1.000 | 0.922 | 0.972 | 0.979 |
| econml forest (honest) | 0.793 | 0.922 | 0.806 | 0.922 | 1.000 | 0.938 | 0.927 |
| grf forest (honest) | 0.846 | 0.970 | 0.855 | 0.972 | 0.938 | 1.000 | 0.987 |
| grf forest (honest, uncentered) | 0.850 | 0.979 | 0.859 | 0.979 | 0.927 | 0.987 | 1.000 |
2. Confounded design (mode=1)#
mode=1 has a difficult nuisance component and a non-constant propensity. Nothing about honesty changes; what changes is whether the estimator adjusts for confounding at all.
[6]:
d1, cate1, table1 = compare(mode=1)
print(f"true ATE on the test rows: {d1['tau'][d1['test']].mean():.4f}")
print(f"propensity range: {d1['e'].min():.3f} - {d1['e'].max():.3f}")
table1
true ATE on the test rows: 0.4964
propensity range: 0.100 - 0.900
[6]:
| ATE | |ATE error| | CATE RMSE | corr w/ true tau | |
|---|---|---|---|---|
| estimator | ||||
| causalml tree (in-sample) | 0.8623 | 0.3660 | 0.4755 | 0.1427 |
| causalml forest (in-sample) | 1.3761 | 0.8797 | 0.9102 | 0.2396 |
| causalml tree (honest) | 0.8443 | 0.3480 | 0.4238 | 0.2236 |
| causalml forest (honest) | 0.8724 | 0.3761 | 0.4237 | 0.3372 |
| causalml tree (honest + CT-H) | 0.8449 | 0.3486 | 0.3975 | 0.3967 |
| causalml forest (honest + CT-H) | 0.8724 | 0.3761 | 0.4218 | 0.5715 |
| econml forest (in-sample) | 0.6655 | 0.1691 | 0.2155 | 0.7809 |
| econml forest (honest) | 0.7710 | 0.2747 | 0.3086 | 0.7502 |
| grf forest (in-sample) | 0.5165 | 0.0201 | 0.0908 | 0.9190 |
| grf forest (in-sample, uncentered) | 0.7905 | 0.2941 | 0.3150 | 0.8488 |
| grf forest (honest) | 0.5143 | 0.0180 | 0.1201 | 0.8712 |
| grf forest (honest, uncentered) | 0.8328 | 0.3364 | 0.3650 | 0.8045 |
[7]:
agreement(cate1)
[7]:
| causalml tree (honest) | causalml forest (honest) | causalml tree (honest + CT-H) | causalml forest (honest + CT-H) | econml forest (honest) | grf forest (honest) | grf forest (honest, uncentered) | |
|---|---|---|---|---|---|---|---|
| causalml tree (honest) | 1.000 | 0.419 | 0.555 | 0.395 | 0.116 | 0.169 | 0.138 |
| causalml forest (honest) | 0.419 | 1.000 | 0.284 | 0.841 | 0.184 | 0.312 | 0.318 |
| causalml tree (honest + CT-H) | 0.555 | 0.284 | 1.000 | 0.429 | 0.197 | 0.277 | 0.196 |
| causalml forest (honest + CT-H) | 0.395 | 0.841 | 0.429 | 1.000 | 0.284 | 0.487 | 0.442 |
| econml forest (honest) | 0.116 | 0.184 | 0.197 | 0.284 | 1.000 | 0.799 | 0.758 |
| grf forest (honest) | 0.169 | 0.312 | 0.277 | 0.487 | 0.799 | 1.000 | 0.901 |
| grf forest (honest, uncentered) | 0.138 | 0.318 | 0.196 | 0.442 | 0.758 | 0.901 | 1.000 |
CausalML’s correlation with the true tau is lower than both references here, in every arm. CT-H raises it (forest 0.34 to 0.57, tree 0.22 to 0.40) without changing the ATE error, so it recovers more of the effect’s shape while the level stays confounded.
Honesty is not the cause: the in-sample rows show the same gap. Every CausalML causal criterion compares raw group means, with no adjustment for how treatment was assigned. grf and EconML relabel each node using treatment residualised against the node mean, a local adjustment that remains with grf’s global centering switched off.
The next section separates confounding from the estimator, and applies the correction.
Confounding or estimator#
Two candidate explanations: confounding, or the estimator. Regenerating the same design with treatment assigned at random, changing nothing but e(X), separates them.
[8]:
def setup_a(seed, randomize=False, n=4000, p=5, sigma=0.5):
"""Nie & Wager Setup A. `randomize=True` keeps b and tau but sets e == 0.5."""
r = np.random.RandomState(seed)
X = r.uniform(size=n * p).reshape((n, -1))
b = (np.sin(np.pi * X[:, 0] * X[:, 1]) + 2 * (X[:, 2] - 0.5) ** 2
+ X[:, 3] + 0.5 * X[:, 4])
e = np.maximum(0.1, np.minimum(np.sin(np.pi * X[:, 0] * X[:, 1]), 0.9))
if randomize:
e = np.full(n, 0.5)
tau = (X[:, 0] + X[:, 1]) / 2
w = r.binomial(1, e, size=n)
y = b + (w - 0.5) * tau + sigma * r.normal(size=n)
return X, w, y, tau
def corr(pred, tau):
return float(np.corrcoef(np.asarray(pred).ravel(), tau)[0, 1])
rows = []
for seed in range(5):
for randomize in (False, True):
X, w, y, tau = setup_a(seed, randomize)
f = CausalRandomForestRegressor(
n_estimators=300, control_name=0, min_samples_leaf=50,
min_group_samples=25, random_state=seed, n_jobs=-1,
).fit(X=X, treatment=w, y=y)
c = CausalForest(
n_estimators=300, honest=True, min_samples_leaf=50,
random_state=seed, n_jobs=-1,
).fit(X, w, y)
rows.append({
"design": "randomized" if randomize else "confounded",
"causalml": corr(f.predict(X), tau),
"econml": corr(c.predict(X), tau),
})
pd.DataFrame(rows).groupby("design").mean().round(3)
[8]:
| causalml | econml | |
|---|---|---|
| design | ||
| confounded | 0.371 | 0.776 |
| randomized | 0.951 | 0.845 |
On the randomized version of the same design CausalML has the higher correlation of the two. Only the confounded version degrades, so confounding accounts for the gap.
The correction is to make treatment assignment ignorable within the split. fit accepts sample_weight, which weights the split search, the leaf means and the honest re-estimation, so inverse-propensity weights are sufficient:
ipw = np.where(treatment == 1, 1 / e_hat, 1 / (1 - e_hat))
model.fit(X=X, treatment=treatment, y=y, sample_weight=ipw)
e_hat is cross-fitted below, so no oracle knowledge of the propensity is used.
[9]:
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_predict
def evaluate(pred, tau):
pred = np.asarray(pred).ravel()
return {
"corr w/ true tau": np.corrcoef(pred, tau)[0, 1],
"ATE error": pred.mean() - tau.mean(),
"CATE RMSE": np.sqrt(np.mean((pred - tau) ** 2)),
}
rows = []
for seed in range(10):
X, w, y, tau = setup_a(seed)
clf = RandomForestClassifier(
n_estimators=200, min_samples_leaf=20, random_state=seed, n_jobs=-1
)
e_hat = cross_val_predict(clf, X, w, cv=5, method="predict_proba")[:, 1]
e_hat = np.clip(e_hat, 0.05, 0.95)
ipw = np.where(w == 1, 1 / e_hat, 1 / (1 - e_hat))
def causalml_forest(sample_weight):
m = CausalRandomForestRegressor(
n_estimators=300, control_name=0, min_samples_leaf=50,
min_group_samples=25, random_state=seed, n_jobs=-1,
)
m.fit(X=X, treatment=w, y=y, sample_weight=sample_weight)
return m.predict(X)
for name, pred in [
("causalml, as-is", causalml_forest(None)),
("causalml + IPW sample_weight", causalml_forest(ipw)),
("econml CausalForest", CausalForest(
n_estimators=300, honest=True, min_samples_leaf=50,
random_state=seed, n_jobs=-1).fit(X, w, y).predict(X)),
]:
rows.append({"estimator": name, **evaluate(pred, tau)})
pd.DataFrame(rows).groupby("estimator").mean().round(3)
[9]:
| corr w/ true tau | ATE error | CATE RMSE | |
|---|---|---|---|
| estimator | |||
| causalml + IPW sample_weight | 0.844 | 0.067 | 0.142 |
| causalml, as-is | 0.381 | 0.389 | 0.433 |
| econml CausalForest | 0.788 | 0.269 | 0.301 |
With a cross-fitted propensity, CausalML’s forest is ahead of EconML on all three measures on this design. Athey and Imbens give this correction in section 7: propensity score weighting, with the weights renormalised within each leaf.
Two notes.
The weights have to reach the leaf estimates, not only the splits. CausalML’s honest re-estimation is weighted. An unweighted re-estimation gives a similar correlation and an ATE error about four times larger, so the shape is right while the level is still confounded.
Estimating the propensity is sufficient. The table uses a cross-fitted random forest, not the true e(X); the true propensity moves the correlation by less than 0.02.
The CausalTreeRegressor docstring carries this recipe. A meta-learner with a propensity model (BaseXRegressor, BaseDRLearner) is an alternative.
3. Does honesty remove spurious heterogeneity?#
Data with no treatment effect anywhere. A tree that chooses its splits and estimates its leaves on the same rows splits on noise and then measures that same noise, so it reports structure that is not there.
mean |tau_hat| is the summary; 0 is the correct answer.
[10]:
def spurious_heterogeneity(n_rep=25, n=3000, n_trees=100):
"""Mean |tau_hat| when the true effect is zero everywhere."""
rows = []
for rep in range(n_rep):
r = np.random.RandomState(1000 + rep)
X = r.randn(n, 5)
w = r.randint(0, 2, n)
y = X[:, 0] + r.randn(n) # tau == 0 for every row
rec = {}
for honesty in (False, True):
label = "honest" if honesty else "in-sample"
f = CausalRandomForestRegressor(
n_estimators=n_trees, control_name=0, honesty=honesty,
min_samples_leaf=50, min_group_samples=25, random_state=rep, n_jobs=-1,
).fit(X=X, treatment=w, y=y)
rec[f"causalml ({label})"] = np.abs(f.predict(X)).mean()
c = CausalForest(
n_estimators=n_trees, honest=honesty, min_samples_leaf=50,
random_state=rep, n_jobs=-1,
).fit(X, w, y)
rec[f"econml ({label})"] = np.abs(c.predict(X)).mean()
rows.append(rec)
return pd.DataFrame(rows)
null = spurious_heterogeneity()
summary = pd.DataFrame(
{"mean |tau_hat|": null.mean(), "std. error": null.std(ddof=1) / np.sqrt(len(null))}
).round(4)
summary["reduction vs in-sample"] = [
""
if "in-sample" in idx
else f"{100 * (1 - summary.loc[idx, 'mean |tau_hat|'] / summary.loc[idx.replace('honest', 'in-sample'), 'mean |tau_hat|']):.0f}%"
for idx in summary.index
]
summary
[10]:
| mean |tau_hat| | std. error | reduction vs in-sample | |
|---|---|---|---|
| causalml (in-sample) | 0.2397 | 0.0039 | |
| econml (in-sample) | 0.1352 | 0.0028 | |
| causalml (honest) | 0.0628 | 0.0033 | 74% |
| econml (honest) | 0.0716 | 0.0033 | 47% |
Both libraries reduce the spurious effect, to comparable levels. This is the property honesty=True provides, at parity with EconML.
4. The rest of CT-H: ccp_alpha="cv"#
honesty=True supplies held-out leaf estimation. ccp_alpha="cv" adds the two further pieces Athey and Imbens specify: the splitting objective’s variance penalty scaled by 1 + N_structure / N_estimation, and tree size chosen by cross-validation over the cost-complexity path, scoring each candidate subtree with that same objective on the held-out fold.
The second piece accounts for the difference. Without it the tree grows until min_samples_leaf stops it, and the penalty ranks candidate splits rather than choosing tree size, which is its role in the paper.
The same option on a single tree and on a forest, paired by seed:
[11]:
from scipy import stats
def noisy_data(seed, n=5000, sigma=1.0):
"""Weak heterogeneous effect buried in noise -- where tree size matters."""
r = np.random.RandomState(seed)
X = r.uniform(size=n * 5).reshape((n, -1))
tau = (X[:, 0] + X[:, 1]) / 2
w = r.binomial(1, 0.5, size=n)
y = X[:, 3] + (w - 0.5) * tau + sigma * r.normal(size=n)
return X, w, y, tau
def ct_h_comparison(kind, n_estimators=None, n_seeds=8):
rows = []
for sigma in (0.5, 2.0):
off, on = [], []
for seed in range(n_seeds):
X, w, y, tau = noisy_data(seed, sigma=sigma)
test = np.random.RandomState(seed).rand(len(y)) < 0.3
train = ~test
for ccp_alpha, acc in ((0.0, off), ("cv", on)):
common = dict(
control_name=0, min_samples_leaf=25, min_group_samples=10,
random_state=0, ccp_alpha=ccp_alpha, cv_folds=3,
)
if n_estimators is None:
model = CausalTreeRegressor(**common).fit(
X=X[train], treatment=w[train], y=y[train])
pred = model.predict(X=X[test])
else:
model = CausalRandomForestRegressor(
n_estimators=n_estimators, n_jobs=-1, **common
).fit(X=X[train], treatment=w[train], y=y[train])
pred = model.predict(X[test])
acc.append(np.sqrt(np.mean((pred - tau[test]) ** 2)))
off, on = np.array(off), np.array(on)
rows.append({
"estimator": kind, "sigma": sigma,
"RMSE off": off.mean(), "RMSE on": on.mean(),
"change": f"{100 * (on.mean() / off.mean() - 1):+.1f}%",
"on better": f"{int((on < off).sum())}/{n_seeds}",
"p": stats.ttest_rel(on, off)[1],
})
return rows
ct_h = ct_h_comparison("single tree") + ct_h_comparison("forest (50)", n_estimators=50, n_seeds=6)
pd.DataFrame(ct_h).set_index(["estimator", "sigma"]).round(4)
[11]:
| RMSE off | RMSE on | change | on better | p | ||
|---|---|---|---|---|---|---|
| estimator | sigma | |||||
| single tree | 0.5 | 0.2167 | 0.1606 | -25.9% | 8/8 | 0.0004 |
| 2.0 | 0.7123 | 0.3455 | -51.5% | 8/8 | 0.0003 | |
| forest (50) | 0.5 | 0.0871 | 0.1306 | +50.1% | 0/6 | 0.0008 |
| 2.0 | 0.1967 | 0.1972 | +0.3% | 3/6 | 0.9496 |
On a single tree the cross-validated sizing reduces held-out CATE RMSE by 25–55%, in every seed. It responds to the noise level rather than pruning uniformly: where there is no overfitting to remove it keeps the deeper tree.
On a forest there is no gain: a wash at sigma=2.0 and worse at sigma=0.5. Averaging across trees already removes the variance the pruning targets, so pruning each tree to a few leaves adds bias and reduces ensemble diversity, and the cost is larger at lower noise. The forest rows in the tables above show the same thing on a single draw.
Use ccp_alpha="cv" on a single CausalTreeRegressor; leave it at 0.0 for CausalRandomForestRegressor.
Where CausalML deliberately differs#
Read against grf’s TreeTrainer.cpp / Tree.cpp and EconML’s tree/_tree_classes.py:
|
EconML |
CausalML |
|
|---|---|---|---|
structure/estimation split |
unstratified subsample |
unstratified shuffle-and-halve |
stratified on treatment |
leaf left empty by the estimation half |
pruned into its sibling ( |
prevented at split time by a second |
keeps its structure-half value |
held-out fraction |
|
fixed at one half |
|
CausalML stratifies because a single CausalTreeRegressor is a first-class object here rather than only a forest component, and because the builder enforces min_group_samples per node. Stratifying also makes the empty-leaf case rare: across 200 honest fits spanning five configurations, including 10%-treated imbalance, n=600 and min_samples_leaf=20, no leaf lost a treatment arm on the estimation half.
Summary#
On a randomized trial, CausalML’s honest forest agrees with
grfand EconML at 0.92–0.97 correlation, and honesty corrects the in-sample forest’s inflated ATE. The single tree agrees with the forests at 0.85–0.89, as a single tree is the noisier estimator.Under a true null, honesty removes most of the spurious heterogeneity, at parity with EconML.
Under confounding, CausalML’s correlation with the true
tauis lower than both, because its criteria compare raw group means with no adjustment for treatment assignment. The gap is present with honesty on or off. Inverse-propensitysample_weightcorrects it and puts CausalML level withgrfon this design; this is the correction Athey and Imbens give for observational data.ccp_alpha="cv"completes the CT-H algorithm and reduces held-out CATE RMSE by 25–55% on a single tree. It is off by default and does not help a forest.honesty=Falserecovers the pre-0.18 behavior.