When partial homogeneity helps, and when it does not

The estimators in this package buy precision by pooling cohort-time effects that share a common value. Whether that trade is worth making depends entirely on whether the distinct effects are separated enough to be told apart. This vignette reproduces, at reduced scale, the Monte Carlo of the paper’s Section 4, which is where that boundary is mapped.

The design

des <- ph_design(N = 2000, T = 10, cohorts = c(0, 3, 5, 7))
des
#> <ph_design>  simulated staggered panel
#> 
#>   N = 2000 units, T = 10 periods, 20000 observations
#>   Cohorts: 0, 3, 5, 7  (0 = never treated)
#>   K = 18 post-treatment cohort-time cells
#>   sigma = 1;  mean flexible SE = 0.0544
#> 
#>   Separation delta is measured in units of that flexible SE,
#>   so ph_truth(des, m_star = 6, delta = 6) puts adjacent group means
#>   six flexible standard errors apart.

Four cohorts, one never treated, giving 18 post-treatment cells. The unit and time fixed effects are removed by the within transformation, so the estimators’ sampling behaviour is driven entirely by the errors.

The two dials

The truth is parameterised by two numbers.

m_star is the true number of distinct effects: 1 is fully homogeneous, where pooling is correct; K is fully heterogeneous, where the flexible estimator is correct; anything in between is the partial homogeneity that motivates the method.

delta is the separation: the distance between adjacent group means in standard errors of the flexible estimates. It is unit-free, and it is the variable that decides whether the partition can be recovered at all.

truth <- ph_truth(des, m_star = 6, delta = 6)
table(truth$labels)
#> 
#> 1 2 3 4 5 6 
#> 3 3 3 3 3 3
round(sort(unique(truth$tau)), 3)
#> [1] -0.816 -0.490 -0.163  0.163  0.490  0.816

A single draw is a ph_data object like any other, so every estimator in the package applies to it:

d <- ph_sample(des, truth)
l0_ph(d)$m          # groups recovered
#> [1] 6
adjusted_rand(l0_ph(d)$partition, truth$labels)
#> [1] 1

Precision under partial homogeneity

The headline claim is that at partial homogeneity the feasible estimators cut the sampling variance of the cohort-time effects substantially relative to flexible TWFE while remaining essentially unbiased.

The Bayesian estimator dominates the run time, since each replication runs a full chain. The paper uses 500 replications with long chains; this vignette uses a handful with short ones so that it builds in seconds rather than minutes. Expect the numbers to be noisy – read the ordering between methods, not the individual values. The section at the end gives the settings that reproduce a published row.

res <- sim_study(
  des, m_star = 6, delta = 6, R = 8,
  bayes_args = list(iters = 250, burn = 50),
  seed = 11, progress = FALSE
)
res[, c("method", "var_ratio", "abs_bias", "ari", "cover_CATT", "cover_ATT")]
#>     method  var_ratio   abs_bias       ari cover_CATT cover_ATT
#> 1   pooled 0.09231643 0.48989795 0.0000000  0.0625000     0.000
#> 2 flexible 1.00000000 0.01964105 0.0000000  0.9513889     1.000
#> 3   oracle 0.35619278 0.01133915 1.0000000  0.9791667     1.000
#> 4       l0 0.79106549 0.01595371 0.9423599  0.9027778     1.000
#> 5    bayes 1.02018512 0.01905387 0.9186008  0.9027778     0.875

Three patterns to look for, all of which the full-scale experiment shows sharply.

Pooled TWFE is always the most precise and always unusable for cohort-time effects. Its variance ratio sits near 0.14 regardless of the truth, but under any heterogeneity it is severely biased, and its intervals – short and centred on the wrong estimand – cover at around 0.04.

At partial homogeneity the feasible estimators approach the infeasible oracle. In the paper’s full experiment they cut the CATT sampling variance by 26-52% relative to flexible TWFE while remaining essentially unbiased, and recover the partition well.

They degrade correctly at the extremes. At m_star = 1 they shrink toward pooling; at m_star = K any pooling hurts and flexible is preferred.

The separation boundary

This is the part that matters most in practice. Fix the number of distinct effects and vary how far apart they are.

grid <- do.call(rbind, lapply(c(3, 6, 12), function(delta) {
  sim_study(des, m_star = 6, delta = delta, R = 6,
            methods = c("flexible", "oracle", "l0"),
            seed = 100 + delta, progress = FALSE)
}))
grid[, c("method", "delta", "var_ratio", "ari")]
#>     method delta var_ratio       ari
#> 1 flexible     3 1.0000000 0.0000000
#> 2   oracle     3 0.4151840 1.0000000
#> 3       l0     3 1.3698881 0.5973508
#> 4 flexible     6 1.0000000 0.0000000
#> 5   oracle     6 0.3103993 1.0000000
#> 6       l0     6 0.6475739 0.9487643
#> 7 flexible    12 1.0000000 0.0000000
#> 8   oracle    12 0.2852459 1.0000000
#> 9       l0    12 0.2852459 1.0000000

At delta = 3 the partition cannot be recovered reliably, and the selection noise erases – indeed reverses – the variance advantage: the feasible estimators are less precise than flexible TWFE. As separation grows the gap to the oracle closes, and by delta = 12 the l0 estimator essentially attains the oracle variance reduction with near-perfect recovery.

The practical message is that these methods deliver their promised gains when heterogeneity is “clumpy” – a modest number of well-separated effect levels – and should not be expected to when the effects form a near-continuum.

Where the two estimators genuinely differ

Not on variance, but on inference. When the partition is uncertain, the l0 plug-in interval treats a possibly-wrong grouping as known, and its coverage collapses; the posterior averages over partitions and degrades gracefully.

cov_grid <- do.call(rbind, lapply(c(3, 12), function(delta) {
  sim_study(des, m_star = 6, delta = delta, R = 6,
            methods = c("flexible", "l0", "bayes"),
            bayes_args = list(iters = 250, burn = 50),
            seed = 200 + delta, progress = FALSE)
}))
cov_grid[, c("method", "delta", "cover_CATT", "len_CATT", "cover_ATT")]
#>     method delta cover_CATT  len_CATT cover_ATT
#> 1 flexible     3  0.9444444 0.2696941 0.8333333
#> 2       l0     3  0.6296296 0.1567238 0.5000000
#> 3    bayes     3  0.8611111 0.2516032 0.6666667
#> 4 flexible    12  0.9629630 0.2681790 1.0000000
#> 5       l0    12  1.0000000 0.1691567 1.0000000
#> 6    bayes    12  1.0000000 0.1688739 1.0000000

In the paper’s full experiment, at low separation the l0 plug-in CATT coverage falls to 0.57-0.62 while the Bayes-PH credible intervals hold 0.79-0.81; where the effects are separable both attain near-nominal coverage.

Two further observations from that experiment are worth carrying into applied work. The overall ATT is well calibrated everywhere for both estimators, because aggregation averages out the cell-level errors – it is the cohort-time intervals where the regime bites. And honest uncertainty under partition selection comes not from adjusting the variance but from averaging over the partition, which is what the posterior does and what no standard-error correction to the l0 interval can replicate. Sample splitting does not rescue it either: halving the sample degrades recovery, and a wrongly merged partition injects a specification bias that a correctly sized standard error cannot undo.